[ebxml-iic-conform] handling the"spec coverage"

Hi 
all:

 

Welcome to the 
"conformance" list...

 

Here is a proposal 
about the notion of specification coverage, that we discussed this 
morning,

and that is very 
specific to conformance testing (but not specific to any ebXML spec in 
particular):

 

By spec coverage, we 
mean the degree to which our tests adress the

conformance to the 
specification requirements, for each feature of the spec.

 

By Test 
Requirements, we mean a test-oriented rephrasing of the

spec features, 
something between the spec narrative and more detailed test case 
definition.

(This is in other 
words our "contract" with the spec owners, that we both have to agree 
on

before doing the 
real work of test cases design.)

 

1. For each 
feature of the target specification (e.g. each sub-section of its document), 

we 
need to tell people how well these are covered by our set of  
Test Requirement items. 

Indeed, test 
reqs may not cover fully a spec feature due to the constraints associated 

with testing, 
difficulty to reproduce all 
situations, 
etc.

So 
Mike and I started to do that with 
by associating this coverage measure with the 

Test Reqs 
themselves - but the point is, that does not 
work well in some cases where

 there is not a 1-to-1 mapping between test Reqs and 
spec features. 

(e.g. the same spec sub-section may be fully 
covered by three Test Reqs, yet each 
would 

participate only partially into 
the verification of the features inside the 
sub-section. 

Or, the same Test Req might actually address two 
different spec features, differently.)

  

2. One 
solution is to  associate this 
coverage indicator 
with  
the spec features 
themselves. 

(Please see in the latest version of 
ebXMLTestFramework doc 

that we posted on our site, 
the description in Section 4.3 "Specification Coverage".)

If we agree on this, 
that means:

 

- we should remove 
the "coverage" field in current Test Reqs document, and

instead provide a 
list of all sections/subsections numbers of the MS spec,

and for each, 
specify the degree of coverage we estimate our Test Reqs 
provide.

 

- so, when we communicate our Test Requirements to the spec 
owners (or spec originators)

for review,  we provide a document with two sections: 

(a) Test Reqs 
(without coverage element), 

(b) List of spec sections / sub-sections 
reference numbers with, for each , 

(b1) the list of Test Reqs that cover it, 
and  (b2) how well 
the section spec requirements 

are covered by these (full, partial, 
none).

 

 Let us know if any problem with 
that.

 

Jacques