RE: [ebxml-iic-framework] Test Framework Comments

Eric:

see 
inline,

Thx 
for timely fedback...

 

Jacques

  
-----Original Message-----
From: Eric VanLydegraf 
  [mailto:]
Sent: Friday, January 24, 2003 2:06 
  PM
To: ''
Subject: 
  [ebxml-iic-framework] Test Framework Comments

  
Well I feel like 
  I'm jumping in the middle of a herd of racing elephants - lots of changes guys 
  and tracking them down and resolving is challenging for 
me.

  
But hopefully my 
  comments will apply or have been take care of and my confusions easy to 
  clarify :)

  
 

  
I will say good 
  job on working on so many fronts of the document in rapid sucession. At some 
  point we need to roll back up to the overall document and verify all the 
  sections work together. I think once it's all folded in and change tracking 
  removed it will look quite nice.

  
 

  
I've just read 
  over and reviewed Version 0.92 (Jan 20, 2002) - apologies if I've missed 
  updates since.

  
 

  
Some overall 
  comments:

  
 

  
To driver-mode or 
  non-driver mode ?

  
----------------------------------------------------

  
The "driver-mode" 
  for Test Driver and Test Service is confusing to me.

  
 

  
Test Service 
  Driver Mode - whether it notifies the Test Driver of test events or not is 
  clear enough and sound to me.

  
 

  
Test Driver - 
  driver mode is confusing - since you would have a TestService in driver mode 
  and could have a Test Driver in non-driver mode.

  
I suggest using 
  conformance mode and interop mode instead and make no mention of 
  driver/non-driver mode for the Test Driver. The role of the test driver is 
  clearer

  
and we don't 
  re-use the drive mode term from the Test Service which has a quite different 
  semantic meaning.
[Jacques Durand] mistake here: when Test Driver is coupled 
  with Test Service: we say the T.D is in "service" mode, while the T.S. 
  is

  
in "Driver" 
  mode. 

  
 

  
Though I'm 
  still trying to resolve this with Connection and Service mode - I don't know 
  if they are the same thing or orthogonal and speak more of the communication 
  between Test Driver and Test Service rather the type of testing being 
  done.
[Jacques 
  Durand] let us try to summarize:

  
- When T.D. generates (and 
  received) messages via the T.S. (action 
  "Initiator"): then T.D. mode = "service" (and automatically, T.S. is in 
  "Driver" mode)

  
- When T.D. generates (and receives) 
  messages on the wire : then T.D.mode= "connection" (acting as a communication node). 
  And here, normally there is no T.S. involved on T.D. 
  side.

  
- When T.S. used without any 
  connection with a local T.D.: Then T.S. mode = "non-driver" (its actions are 
  invoked only via received

  
messages, and respond only via 
  messages).

  
- A possible oddity: {T.D. + T.S.} on 
  one host , where T.D. generates/receives messages on the wire 
  ("connection" mode) , yet

  
the T.S. also can send/receive via an 
  MSH and notify the T.D. (T.S.mode= "driver")

  
 

  
So yes they are orthogonal to some 
  degree : T.D. (connection/service) and T.S.(driver/non-driver), 
  

  
as they concern different 
  components.

  
 

  
Are we missing a 
  section ?

  
----------------------------------------

  
We talk about 
  General Methodology and we talk about Test Framework components and we talk 
  about Test Suite chemas and instances but we have no 
  description

  
of the runtime of 
  the testing.

  
 

  
What I mean 
  is how to trigger the testing to begin with - we have figure 4 we show a 
  "Control Interface" but I can't find a description for it.
[Jacques Durand] Right. I 
  will remove the COntrol interface: the "Actions" interfaces should be 
  sufficient: all what we need to say is 
  

  
is that these actions may be triggered 
  either by an MSH, or by the Test Driver.

  
In other 
  sections we talk about Configuration messages and Initiator messages but we 
  don't described how that gets called. I guess I'm looking for the ignition key 
  to the this whole engine.
[Jacques Durand] Mikes gave recently the schema for such 
  messages. Such messages [payloads] would be generated by the Test 
  Driver,

  
from the putMessage() operation of test 
  cases.

  
 

  
 

  
 

  
Should we break up 
  the actions and messages into 2 channels - driver control 
  versus testing ?

  
=====================================================================

  
It seems to me 
  there are 2 channels in the test framework.

  
Control: trigger, 
  configuration, test result reporting, notifications, .. 
etc

  
Test: 
  messages for the purpose of asserting test requirements.
[Jacques Durand] maybe in a 
  future version... the distinction is not very clear-cut. And messages are 
  mostly

  
for test assertions, alsways for 
  Service/Action. 

  
Configuration messages may be used but 
  that is technically also going to a 
  Service/Action.

  
 

  
 

  
++++++++++++++++++++++++++++++++++++++++++

  
No onto the 
  nit picky stuff :)
[Jacques Durand] yes we will fix that... some of these were 
  already underway. 

  
 

  
Header: date says 
  April 2002 (not matching document date)

  
 

  
101 - I think we 
  should come up with a URI placeholder - so when we do an errata sheet the 
  first thing we don't need to do is update this document :D 
  (Catch-22)

  
we can always have 
  a page at the URL that states no errata has been created yet but in the future 
  one could be.

  
 

  
194 - brackets 
  around the MIME,XML,SOAP 

  
I believe SOAP 
  with Attachements is usally written [SOAPATTACH]

  
 

  
Page 10,11 
  (367-393) all end with periods or none (consistency)

  
 

  
466-473 all the 
  bullets are related to 464-465 - either indent them one more tab or change 
  464-465 into a non-bullet phrase so the rest match up to 
  it.

  
 

  
Page 17 (Fig 
  5)suggest breaking them up into Fig 5a 5b or 5 & 6 and then refering them 
  by that in 499--508

  
 

  
will make it a lot 
  easier to read - I also suggest moving those 2 figures above the paragraphs 
  that describe them.

  
 

  
545-551 hints 
  a triggering from a message but not how the message itself is sent. (goes to 
  my overall point above)

  
 

  
612 elts => 
  elements

  
 

  
671-675 
  verificationstatus implies that the Action is doing the verifying -  
  where really the Test Driver will be doing that - so really it just needs to 
  send the error it trapped back to the test driver. The Driver will know if 
  that is correct or not. 

  
How about 
  <testDriverMessage> or <testNotification> or 
  <actionResult>

  
 

  
same for 697-701, 
  727-734

  
 

  
829 level and 
  expected to be general enough to [allow configuration of] most 
  implementations, ...

  
 

  
907 fig number is 
  out of sequence 4 should be 7, fix all subsequent figures

  
 

  
1032-1033 Delete 
  [The ebXML MS Conformance Test Requirements instance file can be found in 
  Appendix E]

  
Reason: Appendix E 
  is firstly Test Report Schema and secondly the Test requirements is in a 
  separate doc

  
 

  
1156 This document 
  will drive the Test Harness ...

  
reason: fix 
  grammar

  
 

  
1186 a Test Case 
  are either large XML files or non-XML message payloads.

  
 

  
1194-1224 missing 
  figure number

  
 

  
1258 ... in a Test 
  Case MUST evaluate to "true" for a Test Case result to 
  "pass".

  
 

  
1301-1304 Test 
  Driver as driver/non-driver with interop and conformance in paranethesis - 
  seems clearerd if we get rid of dirver/non-driver terms

  
 

  
1351 delete "the" 
  between "parsing" and "a"[after parsing a simple Message 
  Declaration]

  
 

  
1626 - 1640 XML 
  attributes are Uper Cased (e.g. Id, Type, Algorithm) which is different than 
  used every where else.

  
suggest consistent 
  UpperCase for Elements, camel (sp?) lowerCase first Word upper Case remaining 
  for attributes

  
 

  
1800 Content-ID 
  change to ContentId 

  
        Content-Location 
  ContentLocation

  
keep it consistent 
  with previous Id usage

  
remove dashes to 
  match MimeHeader and in general XML elements

  
 

  
1806 Dsign => 
  DSign

  
 

  
DSign needs 
  attributes camelCased as well

  
 

  
1948 
  requirement

  
An enumerated list 
  of requirement types of either MUST or MAY

  
 

  
2118 - 
  2135 CPPA=> CPA

  
 

  
we aren't talking 
  the specification we are talking the actual document

  
 

  
btw what is 
  [BASECPPA] ?

  
 

  
3578 [ebTESTREQ] 
  refers to CPPA spec ?

  
3584 
  [ebTESTSUITE] refers to BPSS spec ? 

  
 

  
3607 [ebCPP] = 
  either [CPPA] or if we stick with [ebCPPA] style than we have to search and 
  replace [CPPA] in document and use the reference type here - I guess the 
  prefered way is a good ebXML JC question eh ?

  
 

  
Good job 
  folks

  
 

  
Now onto the 
  latest interop doc :D