Re: [xml-dev] How many unit tests should I create for my XMLapplication?

[Amelia A Lewis <amyzing@talsever.com>]

On Sat, 24 Nov 2012 18:29:27 +0000, Costello, Roger L. wrote:
> How many unit tests should I write for my XML application?

As many as are useful.

The metric that you provide is not useful, unless you've got pointy 
hair and a corner office. It isn't even meaningful:

<xs:element name="example" type="xs:string" />

vs.

<xs:element
    name="example"
    type="xs:string">
</xs:element>

If you're going to use the usually-meaningless tests-per-thing metric, 
then don't use "lines" in XML (or anything else, like C), use 
"declarations" or something equivalent (perhaps "statements" in C, or 
perhaps "functions"--programming by contract suggests testing 
per-function). But that's still rampant silliness, simply counting the 
brush strokes used to paint the bike shed.

A more reasonable way to pose the question is: what is "coverage" in 
the context of XML validation? In [my experience of] 
specification-land, "coverage" means "test every assertion". But what 
do you test? Well, you might start by testing common cases for 
compliance (but you can't feasibly test every possible value for 
xs:float; you can't even test every possible value for xs:string). You 
can test failure, too, but again: that universe is infinite. And then 
the combination of infinities makes a larger infinity. You can't cover 
all the possibilities.

You cover the common cases first, and add tests as needed. "Common 
cases" include common mistakes. That's going to require knowledge of 
the domain. At the simplest level, documents created algorithmically 
are apt to fail differently than documents created by humans. The same 
algorithm in different languages/environments is apt to fail in 
different ways as well.

*Shouldn't* you be able to take for granted that all the things tested 
for in the W3C XML Validation Test Suite are ... well, already covered? 
If not, are you going to test that incoming documents really are XML, 
in the encoding each (often implicitly) asserts? If it's not XML, do 
you *care* whether it's a streaming video or a text file? If it's not 
well-formed, are you obligated to decide whether it's because the 
encoding is inaccurate or because a closing '>' has been dropped, or an 
attribute value is missing its close-quote? If it's not valid, are you 
obliged to provide any information beyond what the validator supplied?

If the schema is associated with a specification, then perhaps it's 
actually written with testing in mind, and expands upon the relatively 
simple rules of validation by providing testable assertions, and 
instructions on how to behave for given forms of invalid content. If 
that's so, base testing on the assertions in the specification. In that 
case, if something is invalid, then there are likely to be rules 
specifying how that is signaled, and what information is provided to 
the generator of the invalid document. Perhaps there are categories of 
invalidity (one attribute has an out-of-range value versus a martian 
document, in which the root element is unknown). A good specification 
also indicates error conditions for otherwise valid documents, if the 
specification mandates additional constraints (particularly 
co-constraints that can't be expressed in the chosen schema language).

I don't know if there's a literature on testing for XML schemas. There 
certainly is a literature on testing, and test coverage. It's a better 
place to find rules for generating metrics than comparing the lines in 
schema-for-schema with the number of discrete tests in XSVTS.

Amy!
-- 
Amelia A. Lewis                    amyzing {at} talsever.com
People help their friends. That's what holds it all together. If people 
didn't
help their friends, then everything would fall apart. I'm in favor of 
holding
things together, so I help my friends.
                -- Miri Robertson Tiazan [Sharon Lee, "Agent of Change"]