Mike:
sounds
good. See inline.
Jacques
-----Original Message-----
From: Michael Kass
[mailto:]
Sent: Tuesday, July 16, 2002 10:13
AM
To: Jacques Durand; 'Matthew MacKenzie'
Cc:
''
Subject: Re:
[ebxml-iic-conform] Coverage: last attempt
Jacques,
I agree that it is a useful measuring tool for
adjusting expectations of the MS TC on how well
we will cover the
specification. The key will be in understanding the wording of the
requirements and determining
just how well they will cover that spec
item.
I will keep the current annotation and their
qualifications of "full", "partial" and "none". I want to go over those
qualifications again however, based upon the criteria you described below,
as my interpretation was different at the time
I created them. I will
post an updated annotated spec and identify any potential changes for
review.
[Jacques Durand] coverage annotation you already have will
probably not change much I guess . Cases where the MSH
specified
behavior is "internal" and cannot be observed as you mentioned
(e.g. persistence of messages) will get either NONE coverage,
or
possibly PARTIAL in case we can use some black-box test reqs that can
still reveal the right behavior in specific
situations
(I
believe we had suchh a test req where MSH is shut down and restarted,
revealing that some persistence was used.).
I will also annotate possible "new" items in the annotated spec that could be
translated into a test requirement. There are a few
items that I saw
that are candidates for inclusion.
[Jacques Durand] in case you are
concerned it would take more time to cover these, we can annotate
with something like "TBC"
(to
be covered) so that
its clear we know they need be addressed (later).
In
addition I will annotate some spec items that are NOT annotated that are
already covered in test requirements. For example, out test harness will
be "validating" all incoming messages from a candidate MSH. Many parts
of the spec ( rather redundantly ) describe "valid" XML message
format.
[Jacques Durand] in case this mention of conformance is only
incidental (not central to the spec statement where you find it
)
and
has more clearly been mentioned (and covered) before, I would not
bother...
These items are not currently annotated (
giving the impression that we do not cover them in our test requirements..
when in fact
we do, with test requirement R0.1.1 .. syntax/schema
validation). Annotating these areas would give the TC folks the
confidence that the specification is
well covered. To oversimplify..
the more annotations the TC team sees, the more confidence they will have that
we "covered" their spec.
Comments?
I would like to avoid
"cluttering" the annotated spec with additional new "to be added" items if
possible. I feel that it gives the impression that we
are not
really finished with defining test requirements. I think that I can add
anything "new" by the end of the week to the test requirements
documents
and the annotated spec.
[Jacques Durand] OK. Note that in case
the lacking test reqs are confined to some optional spec modules, for
which
lots
of efforts would still be needed, we
can postpone the coverage of this spec module
.
Regarding abstract test cases, I believe that we
could submit a list of abstract Test Cases, each corresponding ( via ID ) to a
matching Test Requirement.
Abstract test cases are "formalized"
representation of the test cases and test steps, in a language that clearly
defines the steps and actions to be
taken, but not coded in the language of
the actual test. Some abstract test cases are highly formalized ( e.g.
for network protocol testing), while others are less formal.
[Jacques
Durand] sounds good.( Incidentally, I guess it's OK for a test case to
cover more than one test req item?)
An example
of a less formal abstract test suite is: ( a test suite for interactive TV
standard )
http://xw2k.sdct.itl.nist.gov/koo/trigger/
We
could use the formalized notation you describe for a test case to populate an
Abstract Test Suite:
Test Steps:
· Step 1:
Test Driver sends an initial message M0 to the Configurator action of
the Test Service, containing expected configuration data.
o Message
Template: mtpl_1 (M-Header: mhdr_1, M-Payload: mpld_config).
· Step 2:
Test Driver receives within time limit a response message M1 from
Configurator action. Correlation: (M1.RefToMessageId = M0.MessageId). M1
reports that the expected configuration is in effect.
o Message
Template: mtpl_1 (M-Header: mhdr_1, M-Payload: mpld_response
(result=OK)).
· Step 3:
Test Driver sends a sample message M2 to the Mute action of the Test
Service.
o Message
Template: mtpl_1 (M-Header: mhdr_1, M-Payload: mpld_1).
o Suggested
Conversation ID: 11250.
o M-Header
updates: PartiId (both): no type attr, no URI content.
· Step 4:
Test Driver receives error message M3 within time limit. Correlation:
(M3.RefToMessageID = M2.MessageId).
Step 5: Verification. Test Case succeeds if: (Step 2 successful)
AND (Step 4 successful) AND (M3 Error: severity="Error",
code="Inconsistent".)
[Jacques Durand] Agree that level of
detail is sufficient for now. In a future version, we can add the XML
scripting for these
once
we have written enough of these cases... (We may introduce such scripts for a
few test cases, as examples).
Cheers,
Jacques
Mike
At 05:31 PM
7/15/2002 -0700, Jacques Durand wrote:
Michael, Matt:
About "specification coverage" attribute:
my last attempt at making sense of this attribute in
the annotated spec:
(I still
believe that may "adjust" expectations from MS TC reviewers, and clarify our
"contract"
with users about
the results of our test suite ...)
COVERAGE:
FULL: The test requirement(s) that address this
specification item, are a good indicator of conformance
to the specification item, i.e. if an
MSH passes a test case that implements properly this test requirement(s),
there is very strong
indication that it will behave similarly in all situations identified by the
spec item.
PARTIAL: The test requirement(s) that address this specification
item, are only a partial indicator of conformance
to the
specification item, i.e. if an MSH passes a test case that implements
properly this test requirement(s),
this indicates that it will behave
similarly in only a subset of all situations identified by the spec
item.
Possible reason maybe:
- the pre-condition(s) of the test
requirement(s) only identify on purpose a subset of these situations,
that can be reasonably tested.
-
the occurrence of situations
that match the pre-condition of a Test Requirementit is under control of the
MSH
(e.g. implementation-dependent) and out of the control of the
testbed. I.e. the test will be verified only in case
the situation
occurs during testing, which we cannot control.
NONE:
this specification item cannot be tested even partially, at least with the
Test Framework on which this
Conformance test suite is to be implemented,
and under the test conditions that are assumed.
Regards,
Jacques
