Title: call tomorrow Wed 8th, 3pm PT
Jacques,
I modified
section 7.1 based on comments from last conference call (changed
structure/design of logic flow HTML table.
Plus I modified the “pass”, “fail”, “undetermined” description at the
end. Please see attached document (turn on
"tracking" to see changes).
It
seems that the last sticking point is the “default” behavior of a Test Driver when encountering a “failed”
TestAssertion. Please see my
comments below:
[Durand]
It is annoying
that the interpretation of an assertion failure (what to do if fails) , is
context-dependent (presence of a step behind or not…) Shouldn’t we say that an
assertion failure just aborts the thread, in the absence of any other explicit
exit statement?
[MIKE] – Why can’t there be a “default” behavior for a
failed TestAssertion. The
fundamental meaning of TestAssertion is just that.. testing an assertion. Generally, if such an assertion is not
met, this means the implementation failed.
However, we have created a more flexible meaning..saying that one can now
“continue” from a failed TestAssertion..or conceivably even “pass” after failing
a TestAsssertion operation. I
believe that the meaning has been “watered down” to the point where it is now
nothing more than a logic flow operator. I believe that the default behavior
should be “fail” for a “failed” TestAssertion. One can still “override” it using the
<Return> instruction ( or <Abort>… although I think that you
suggested that we not use <Abort>).
I am in favor of trying to keep close to the meaning of
<TestAssertion> as we originally defined it.
[Durand]
This thread may be or-joined later and
the entire test case may still succeed thanks to other concurrent threads that
have successfully completed.
[MIKE] – I’m OK with that. I just do not think that
<Abort> should be the default Test Driver behavior when a TestAssertion
fails. In our 216 ebMS Test Cases,
that would result in 216 “undetermined” Test Case results.
Can’t we
say that an aborted test case wil be by default interpreted as
“Undetermined”?
[MIKE] – An “aborted” Test Case should result in an “undetermined” result. But I do not believe that a “false” TestAssertion result should not
(by default) <Abort> a Test Case.
Below are the rules added to the end of section 7.1 for
your review
A
“default” Test Case state of "continue" exists after a TestAssertion operation
returns a boolean result of "true" , or a “Return” instruction is
encountered by the Test Driver, indicating that the current Thread MUST be
terminated, but that logic flow will continue within the “calling”
Thread.
A final Test Case state of
"pass" occurs when the Test
Driver encounters an explicit "exit/pass" instruction from within a
TestAssertion, OR logical testing execution proceeds from beginning to end
without:
a) An explicit "exit/fail" or
"exit/undetermined"
b) A system "exception"
condition
c) A failed Join that precludes further
testing execution beyond the Join
A final Test Case state of
"fail" is given to a TestCase where :
a) A TestAssertion boolean operation
returns a result of "false" (default behavior) (i.e. it is assumed that a
TestAssertion is a meaningful measurement of conformance/interoperability that
MUST pass.. unless explcitly overriden by the test writer in such cases where a
TestAssertion verifies a "precondition" to further testing, in which case the
test writer may wish to "exit" with a state of
"undetermined". Additionally, the test writer may with to "continue"
if the failed result of the TestAssertion is used to alter the flow of Test Case
execution.
b) The
Test Driver encounters an
explicit "exit/fail" instruction within a TestAssertion
operation.
A final Test Case state of
"undetermined" occurs if:
1) An explicit "exit/undetermined"
instruction occurs from a TestAssertion
operation
2) Testing logic flow
cannot proceed beyond a Join due to failed andJoin or orJoin
condition
Comments?
Mike
----- Original Message -----
From:
Jacques
Durand
To: ebXML IIC - main list (E-mail)
(E-mail)
Sent: Tuesday, September 07, 2004 5:27
PM
Subject: [ebxml-iic] call tomorrow Wed
8th, 3pm PT
We'll have exceptionally a meeting tomorrow Wednesday at
3pm PT.
Primary objective is to close on Test Framework spec
remaining points being discussed.
I'll mostly give
a quick update on #2, #3 in agenda below.
We'll
likely postpone #4 to Monday, unless Patrick is on the call.
Jacques
Time: Wed 8th, 3pm-4pm PT
Host: Fujitsu
Toll only : 1-512-225-3050
Participant code: 385218#
tentative Agenda :
1. Test Framework 1.1 technical:
- use cases , BPSS relevance.
- Threads: abort semantics, return
feature.
- Monica comments.
- levels of conformance, if any?
- any "binding" to consider? (optional,
but normative)
2. ETSI follow-up:
- CD status for the test plan
- feedback on test suites: few bugs detected,
updates needed.
3. Request from ebXML Asia for a n Interop
test suite without:
- sync
messaging (bus responses only?)
-
empty messages.
4. Others:
- follow-up/update on Patrick Hogan / Linux
plans.
section7.1-JD1_MK_JD2_MK2.doc
