Re: [xml-dev] Testing XML don't use xUnit

[Michael Sokolov <msokolov@safaribooksonline.com>]

On 4/10/2013 10:27 AM, Michael Kay wrote:
>>> It sounds like you are suggesting it's bad to have xpaths in junit
>>> tests?  ...and to use Schematron instead?
>>>
>> Yes. Because changes to the structure of the xml will break the JUnit
>> test whereas Schematron assertions are resilient to that.
>>
> I'm completely bemused. I thought Schematron assertions, like XMLUnit assertions, were both XPath. Why is one better than the other?
>
> Unit testing has always been exposed to the problem that when things change, tests break. My response to that has always been to have very little internal testing - the vast majority of tests are "black box" tests working against stable product interfaces rather than against internal components, because internal components change much more than external product interfaces. But XPath has proved a very good language for writing the assertions, and I've always thought that is why both Schematron and XMLUnit use it - as do the latest generation of W3C test suites, with considerable success.
There is an art to writing robust assertions, whatever language or 
framework is used.  We use XPath throughout our testing suites; in 
selenium integration tests that examine HTML markup, and in java unit 
tests that just look at more isolated XSLT-driven processes. The key is 
ensuring that the assertion tests what it is supposed to, and not a 
coincidental feature of the implementation.  That's not always easy, and 
brittle tests aren't helpful.  I often favor looser tests for this 
reason: ie just make sure the identifier appears somewhere in the 
markup, or use a "//" as was alluded to earlier, rather than a full path.

Re: integration vs. unit tests; they both have merits.   It's hard to 
generalize about kinds of tests since software systems vary in which 
parts are stable or unstable. Highly granular focused tests (usually 
called unit tests) are helpful since when they fail they pinpoint the 
cause more specifically.  We often find that our version of black-box 
testing: integration tests, in a web application, test too many 
different things at once though, and in fact are unstable because they 
are sensitive to page layout, which tends to change a lot in 
uninteresting ways.  So it's not just an internal/external dynamic, I 
think, but it's definitely important to test things that are important, 
that are actual intended behaviors.

I think this issue actually relates somewhat to the whole schema 
brouhaha that is going on in a parallel thread, since schemas can be 
thought of as a kind of testing assertion, but I haven't used them 
enough to say anything intelligent about it :)  We tend to just make our 
assertions in custom code (java unit tests).

-Mike