xml-dev - Re: Test suite suggestion

["David Brownell" <david-b@pacbell.net>]

Sounds like a fine idea to me; it raises the prospect of having a
better way to standardize the test reporting than "what Dave does"
(!) by XSLT stylesheets.  The existing report templating setup is
more at the quick'n'dirty level, though it works just fine.

I'd see four sections to such a document:

    - parser description ... name, version, type (validating, which
      of the four non-validating ones and maybe "standalone" modes)

    - test suite version (there's a handy field for it)

    - runtime environment description (e.g. OS and version, JVM
      version info, etc) ... since this can interact with the items
      above.  (Java is pretty good about not having such problems,
      at least on production level JVMs.)

    - results for each test (no need to repeat content as below),
      including diagnostics.

You mentioned all of those except the runtime environment.

Tricky bits include the fact that the results likely vary based on
the particular API being tested.  Example, SAX conformance demands
that errors be reported in particular ways, but other parser APIs
likely work a bit differently.  That also affects what bits of the
runtime environment matter ... C/C++ parsers are unlikely to care
about the JVM that's installed.

- Dave


----- Original Message -----
From: Richard Tobin <richard@cogsci.ed.ac.uk>
To: <xml-dev@xml.org>
Sent: Friday, May 19, 2000 3:50 PM
Subject: Test suite suggestion


> David Brownell recently published some new results from the Oasis test
> suite(http://www.xml.com/pub/2000/05/10/conformance/conformance.html).
> This relied on a harness he had written for Java SAX parsers and
> MSXML.
>
> To allow a wider range of parsers to be tested, I suggest that we (or
> Oasis) define a format (XML of course) for reporting the raw test
> results.  Parser implementors would provide test harnesses to produce
> data in this format, and summaries and comparative surveys could be
> produced idependently by use of stylesheets.
>
> I already do this in a rather minimal way for testing RXP; the output
> looks like this:
>
>  <RESULTS VALIDATING="yes">
> [...]
>  <RESULT ID="invalid--002" TYPE="invalid" SECTIONS="3.2.1"
STATUS="invalid">
>  <DESCRIPTION>
>     Tests the "Proper Group/PE Nesting" validity constraint by
>     fragmenting a content model between two parameter entities.
>  </DESCRIPTION>
>  <DIAGNOSTIC>
>  002.ent:2:19: warning: Content particle ends in different entity from
that in which it starts
>  </DIAGNOSTIC>
>  </RESULT>
> [...]
>
> It needs a little elaboration - it should indicate the version of the
> parser and of the tests, for example.
>
> -- Richard
>


***************************************************************************
This is xml-dev, the mailing list for XML developers.
To unsubscribe, mailto:majordomo@xml.org&BODY=unsubscribe%20xml-dev
List archives are available at http://xml.org/archives/xml-dev/
***************************************************************************