Re: [xml-dev] Testing XML don't use xUnit

[Ihe Onwuka <ihe.onwuka@gmail.com>]

On Thu, Apr 11, 2013 at 11:37 AM, Stephen D Green
<stephengreenubl@gmail.com> wrote:
>
> I also do a lot of work with testers writing tests for HTML pages using the
> very well-known integration test tool Selenium IDE (for Firefox) and that
> makes heavy use of XPath even though the target is HTML. Here we end
> up with many tests which include XPaths which break every time a
> significant change is made to the HTML. So we have to keep mending the
> XPath expressions (even though we try to write them with //, etc, to avoid
> this). So I know the problem. It doesn't stop people favouring Selenium
> IDE for website integration tests though. It's well worth the hassle to
> many.
> I guess it might be different if we were using XPath for unit tests but I
> don't know about that. I guess modern unit tests get associated with TDD
> and Agile, so maybe having separate test assertions (written in prose)
> would be a 'no no' for many unit test writers as it might be seen as going
> against Agile mantras. You'd have to update the assertions less frequently
> than the unit tests but having to update something other than the tests
> and the XML / code would be just too much for many Agile teams. So
> if you wanted to inut test your XML you'd probably be stuck with having to
> keep revising a load of XPath expressions. It goes with the territory.
>
> Best regards
>
> Steve
>

Very very interesting.

Sitting on my hard drive is an advanced draft of an unpublished paper
(Declarative Automated XML Testing)  and 40 odd pages of slides for an
ungiven talk/training course (Schematron for Testers) that I had
forgotten I had ever written.

Both were prepared in 2 years ago and targeted at the testing
community and the Selenium experience was exactly what was at the
forefront of my mind. Very enlightening summarisation of the reception
I could have expected.

Some excerpts from the slides (hence the formatting)

Imagine a test tool that afforded you the ability to specify and
cross-reference XML/web test cases to requirements as well as
automatically check the result of executing them WITHOUT having to
program.
Imagine a free test tool  based on a familiar technology (XML) that
    obviated the need to learn a proprietary language.
     Required  minimal prerequisites knowledge to use.
    Did not require familiarisation with a proprietary user interface.
    Afforded the ability to issue customised  user-friendly
diagnostics  that pinpoint the  problem area with exactitude.
    Afforded test case reuse by allowing you to define templates for
commonly occurring patterns that can be parameterised and
instantiated into actual test cases.
    Afforded test case reuse with include mechanisms.
If such a tool existed would you be interested in using it ?

Such a tool does exist. A Schematron schema is a validation
specification that enables natural language assertions  to be made and
validated against the content of an XML document. It is not designed
to be a test tool but can be used as one.
No new programming  user interface to master. A Schematron schema is
plain XML  specified using elements and attributes from the Schematron
namespace.
A Schematron schema is a declarative specification  from which an XSLT
stylesheet is automatically generated that will navigate an XML
instance and check the constraints it specifies.
The power of Schematron can also be applied to database/file testing
if data is first converted to an XML format.
It can be embedded directly into test/functional specifications
 You don't have to learn how to program to use it.

”Let me start by saying that writing GUI-based tests for Web apps is
no fun, no matter what your testing tool is. You need to navigate
through pages, fill and submit forms, and verify that certain elements
are present on the pages. Doing all this manually can quickly become
tedious and kill whatever joy you may find in testing.”

Many tools attack this problem with record and playback. One of the
many drawbacks of this is that it generates very inefficient, long and
brittle XPath because all expressions navigate from the root. This
means that tests are vulnerable to structural changes in the XML.
Schematron automatically takes care of navigating the XML instance. If
you specify test cases on say the account element, it will work out
for itself how to locate the relevant nodes in the instance.
There is no need for Record and Playback at the test verification
stage and test cases are better insulated against changes to the data
structures.

An extract from the paper

Several automated test tools support XPath but using any of these tools entails
accidental temporal cohesion that forces business rule testing to be tightly
coupled with GUI test. Their ability to auto-generate XPath to retrieve specific
content following a capture replay session may seem like a boon but the XPath
generated is brittle (and inefficient) because it is coded using absolute paths
and thus vulnerable to structural changes in the XML. Non-trivial business tests
will inevitably entail comparisons with data in other XML documents or databases
so one cannot be entirely reliant on autogenerated XPath. Once learnt, you can
convert database or other formats data to XML and thereby gain leverage for your
XPath interrogation skills.

XPath has a natural habitat - XSLT - and that is the vehicle advocated in this
article for housing your XML tests. The reason for this outlandish suggestion is
the availability of Schematron, a tool that supports the specification of test
cases and will generate the XSLT to validate the test cases for you. Schematron
only requires you to familiarise yourself with 4 elements to use it. If you are
willing to learn more of the elements you have an even more powerful tool and
because it generates XSLT and provides hooks that support customisations the
full power of XSLT is an option at your disposal. Let us first clarify that the
testing scenario under contemplation is one where the test data has been created
(whether by executing a test or derivation) and exists in an XML document
waiting to be validated against expected results.