RE: [xml-dev] ANN: X-Hive/DB 7.0 Released

[Martin Probst <martin@x-hive.com>]

Hi Michael,

> Congratulations on publishing your test results: you have certainly set a
> high standard for others to match, and I hope others won't be discouraged
> from doing so because they don't come up to the same level.

Well, thanks a lot. It has been very helpful for us to go through the
test cases and correct bugs, even though it took quite a lot of time.

> But I'm curious to know how you achieved such a clean slate. I've been
> running the XQTS 0.7.0 tests over the last few days and have reported bugs
> in over 160 of the 9000-odd tests [1]. Many of these errors were presumably
> present also in the 0.6.0 tests. How did you pass these tests? Did you
> correct them as you went along, or apply your own judgement as to whether
> your results were correct, or what?

The results you see are indeed from the XQTS 0.6.0 drop (the reports
file itself doesn't include a version number, that should maybe be
fixed) but I will start working on 0.7 very soon. I tried to refrain
from changing the actual tests as much as possible, the only things I
have changed indeed are trivial fixes - like the line break in that one
result document (which is already fixed in the W3C CVS too).

Attached you will find a Zip file containing two XSL files and a diff of
the official XQTS 0.6.0 distribution. The diff includes all changes we
made to the official test suite except for converting windows \r\n to \n
and removing that ugly byte order mark. Most of them are fixing the
comparison type ("XML" --> "Fragment").

The two XSL files are used to transform the original test suite files
into Java JUnit source code. First "transformXQtests.xsl" transforms the
original document into our custom XML test specification format, then
XQueryTests.xsl is used to generate the Java code. Please excuse my poor
XSL skills ;-)

To save you from looking at that ugly code: basically we .trim() every
expected result before comparison. XML and Fragment results a
serialized, then parsed again and compared node-by-node ignoring
'ignorable whitespace'
(DocumentBuilderFactory.setIgnoringElementContentWhitespace(true);).

So we ignore trailing whitespace in "normal" tests for atomic values and
ignore element content whitespace. I think this doesn't skew the results
much in our favor.

Also keep in mind we are still failing ~350 tests of the 7395 we are
running (making those 95.25% passing tests). A big number (100?) of them
because we don't support "import schema", but that still leaves over 250
failures.

> I also found a large number of tests where I had to apply much looser rules
> to comparison of test results than those prescribed, for example by
> stripping XML declarations at the start of a file or newlines at the end.
> Again, I'd be interested whether you had the same experience.

See above.

There are some tests which seem more or less impossible to meet to me,
which are mainly rounding problems. E.g. in some places xs:decimal is
required to have 18 digits after the colon, and in some 20.

Also in some cases we won't be able to match some tests, e.g. the X-Hive
XQuery implementation is based on our DOM Level3 implementation. DOM
forbids some node operations which are allowed in the XQuery Data Model.

Overall I think the test suite is of a very high quality and a real help
for implementers, so thanks a lot for that to the XQTS group.

Regards,
Martin Probst
X-Hive Corporation

xhive-xqts.zip