Re: XML 1.0 Conformance Test Results

[David Brownell <david-b@pacbell.net>]

> > BTW is the "patch" you referred to actually the 2nd edition of the test suite,
> > or is it in fact a patch _to_ the 2nd edition.

Patch to the 2nd draft nist/oasis suite ... I posted the URL a while back:

http://xmlconf.sourceforge.net/xml/?selected=resources

Calling it "2nd edition" is troublesome, it suggests correspondence to
the "XML spec 2nd edition".  Additions are needed in that patch;
there's been some discussion about putting the test database into CVS,
and using the SourceForge issue tracker to make help ensure issues
don't get lost or mis-resolved.


> > their hands in the real world of implementation.  IMHO it would be much better
> > if the W3C ditched the idea of "specs" altogether in favour of  "test suites".

Or rather, insisted on having both.  Test suites are done in reference
to some specification ... and specifications need to be written with
testability in mind.  Language which defeats testability is bad language
for specifications; it's language which promotes non-conformance.

Specs have a role in explaining what's being specified, and the intent
of the spec (essential for future evolution).  Test suite can't supplant
those roles.


> [1] If the W3C published test suites instead of specs, errors in the
> test suite would become part of the spec, wouldn't they? 

Bugs exist ... which is why my original comment was about process.
Good spec evolution processes address issues such as those which
testing (or use, or implementation) turn up.

Testing is one of those "keep the suppliers honest" issues.  In this
case, the supplier is W3C.  Giving it complete control over all tests
can seem much like a fox-and-henhouse issue, however, given the
processes that have been bewailed at length on this list!


> [2] Specs precede implementation better than test packs do - it is
> easier to define a complete spec than it is to define a complete test

Gary touched on completeness ... I think it's fair to say that a REC
should be testable, and the best way to show that's so is to have a
test suite.  But again, I think a conformance test suite should be a
separate thing.  More or less co-equal with the spec.  Having 80%
of the testable parts of a spec actually have tests is pretty darn good,
but the remaining parts of the spec (testable and otherwise) may still
be important.

- Dave