[
Lists Home |
Date Index |
Thread Index
]
Len, et al,
At 8/12/2005 08:21 AM, Bullard, Claude L (Len) wrote:
>Without digging into the details, I welcome this
>approach. XML Query is a major addition to the toolkit
>and providing test suites before fielding gets too
>far along is the best way to ensure interoperability
>even if it slows down development and fielding.
Thanks (speaking as a co-chair of the XML Query WG).
>Do you think the test cases cover the right sources
>of problems? I would hope 7000 cases would suffice.
There are important reasons why 7000 don't suffice. First, the
requirements for progressing beyond Candidate Recommendation status include
the ability to ensure that the spec is correct, which is (best?) done by
ensuring that every "feature" is tested --- but, what's a feature? There
are many definitions.
Second, in a largish, somewhat complex language, there are a great many
"things" that everybody will agree to be a "feature", but each such thing
has a number of aspects (edge and corner cases, decision points, etc.) and
thus requires more than a single test.
Third, because the number of such things is so large, it becomes awfully
burdensome for some person to sit down and determine the "best" number of
tests for each thing, so broad brushstroke rules end up determining how
many tests are needed (e.g., 20 per feature). In some cases, 20 is far too
many, while in others it's far too few. It's unlikely that the number will
be reduced for the "far too many" cases, but likely that it will be raised
for the "far too few" cases, as they are discovered.
My fellow co-chair, Andrew Eisenberg, is chair of the testing task force
and has a very believable prediction that we'll need about 12,000 tests at
a minimum (I think I have that number right), although he had hoped to see
about 50% more than that.
Which leads me to the conclusion: 7,000 isn't enough because the existing
7,000 tests don't cover all of the features. One might claim, and get
support for the claim, that 7,000 would be enough if spread appropriately
across all of the features, but few would claim that 7,000 is "large
enough" that the remaining features should go completely untested ;^)
Hope this helps,
Jim
========================================================================
Jim Melton --- Editor of ISO/IEC 9075-* (SQL) Phone: +1.801.942.0144
Co-Chair, W3C XML Query WG; F&O (etc.) editor Fax : +1.801.942.3345
Oracle Corporation Oracle Email: jim dot melton at oracle dot com
1930 Viscounti Drive Standards email: jim dot melton at acm dot org
Sandy, UT 84093-1063 USA Personal email: jim at melton dot name
========================================================================
= Facts are facts. But any opinions expressed are the opinions =
= only of myself and may or may not reflect the opinions of anybody =
= else with whom I may or may not have discussed the issues at hand. =
========================================================================
|
