[
Lists Home |
Date Index |
Thread Index
]
Thanks for the response. I've passed the message on to Andrew Eisenberg who
runs the testing task force. (I haven't been involved in producing tests
myself but have put a fair bit of effort into debugging them!)
I'm sure the team will be interested in any feedback you can give on the
issues that make test results difficult to reproduce, such as whitespace
issues and numeric precision, and of course bug reports on specific tests.
Michael Kay
http://www.saxonica.com/
> -----Original Message-----
> From: Martin Probst [mailto:martin@x-hive.com]
> Sent: 20 October 2005 14:10
> To: Michael Kay
> Cc: 'Jeroen van Rotterdam'; xml-dev@lists.xml.org
> Subject: RE: [xml-dev] ANN: X-Hive/DB 7.0 Released
>
> Hi Michael,
>
> > Congratulations on publishing your test results: you have
> certainly set a
> > high standard for others to match, and I hope others won't
> be discouraged
> > from doing so because they don't come up to the same level.
>
> Well, thanks a lot. It has been very helpful for us to go through the
> test cases and correct bugs, even though it took quite a lot of time.
>
> > But I'm curious to know how you achieved such a clean
> slate. I've been
> > running the XQTS 0.7.0 tests over the last few days and
> have reported bugs
> > in over 160 of the 9000-odd tests [1]. Many of these errors
> were presumably
> > present also in the 0.6.0 tests. How did you pass these
> tests? Did you
> > correct them as you went along, or apply your own judgement
> as to whether
> > your results were correct, or what?
>
> The results you see are indeed from the XQTS 0.6.0 drop (the reports
> file itself doesn't include a version number, that should maybe be
> fixed) but I will start working on 0.7 very soon. I tried to refrain
> from changing the actual tests as much as possible, the only things I
> have changed indeed are trivial fixes - like the line break
> in that one
> result document (which is already fixed in the W3C CVS too).
>
> Attached you will find a Zip file containing two XSL files
> and a diff of
> the official XQTS 0.6.0 distribution. The diff includes all changes we
> made to the official test suite except for converting windows
> \r\n to \n
> and removing that ugly byte order mark. Most of them are fixing the
> comparison type ("XML" --> "Fragment").
>
> The two XSL files are used to transform the original test suite files
> into Java JUnit source code. First "transformXQtests.xsl"
> transforms the
> original document into our custom XML test specification format, then
> XQueryTests.xsl is used to generate the Java code. Please
> excuse my poor
> XSL skills ;-)
>
> To save you from looking at that ugly code: basically we .trim() every
> expected result before comparison. XML and Fragment results a
> serialized, then parsed again and compared node-by-node ignoring
> 'ignorable whitespace'
> (DocumentBuilderFactory.setIgnoringElementContentWhitespace(true);).
>
> So we ignore trailing whitespace in "normal" tests for atomic
> values and
> ignore element content whitespace. I think this doesn't skew
> the results
> much in our favor.
>
> Also keep in mind we are still failing ~350 tests of the 7395 we are
> running (making those 95.25% passing tests). A big number
> (100?) of them
> because we don't support "import schema", but that still
> leaves over 250
> failures.
>
> > I also found a large number of tests where I had to apply
> much looser rules
> > to comparison of test results than those prescribed, for example by
> > stripping XML declarations at the start of a file or
> newlines at the end.
> > Again, I'd be interested whether you had the same experience.
>
> See above.
>
> There are some tests which seem more or less impossible to meet to me,
> which are mainly rounding problems. E.g. in some places xs:decimal is
> required to have 18 digits after the colon, and in some 20.
>
> Also in some cases we won't be able to match some tests, e.g.
> the X-Hive
> XQuery implementation is based on our DOM Level3 implementation. DOM
> forbids some node operations which are allowed in the XQuery
> Data Model.
>
> Overall I think the test suite is of a very high quality and
> a real help
> for implementers, so thanks a lot for that to the XQTS group.
>
> Regards,
> Martin Probst
> X-Hive Corporation
>
|
