Re: [sca-assembly] Issue 101: Complete the Conformance Section - Commentson Proposal

Anish,

Reply inline...

Yours,  Mike.

Strategist - Emerging Technologies, SCA & SDO.

Co Chair OASIS SCA Assembly TC.

IBM Hursley Park, Mail Point 146, Winchester, SO21 2JN, Great Britain.

Phone & FAX: +44-1962-818014    Mobile: +44-7802-467431  

Email:  

Anish Karmarkar <>
wrote on 03/03/2009 07:46:07:

> Re: [sca-assembly] Issue 101: Complete the Conformance Section - 

> Comments on Proposal

> Mike Edwards wrote:

> > 

> > Folks,

> > 

> > Some comments on the proposal:

> > 

> > 

> > 1)  I think that making a series of SCA-related documents
into 

> > conformance points, as per section 12.1,  is unnecessary
and unwise.

> > 

> > The only point of making these documents into conformance points
is if 

> > it is intended to write testcase(s) that will validate those

> > documents.  I do not believe that we have the resources
to write such 

> > testcases and as a result, the conformance demands made

> > here are a waste of time and effort.

> > 

> 

> I would like to understand what additional tests will have to be written.

> There are already requirements on SCA runtime regarding rejecting
what I 

> would call non-conformant SCA documents. To test that we would have
to 

> create non-conformant documents, use it with a runtime and ensure
that 

> the runtime rejects them. What additional test artifact would have
to be 

> created wrt document conformance? I.e., we have a set of documents
that 

> we assert are non-conformant. We of course would have a set of 

> conformant documents (for the positive test for a runtime).

> 

> I can see us not wanting to create non-conformant (or conformant)
SCA 

> documents for every little feature and element/attribute that schema

> allows. But we are now talking about how detailed our test suite is

> going to be. Given that there is a requirement for runtimes to reject

> invalid SCA documents, this is already an issue. I don't think we
need 

> to Turing complete.

>

The tests you describe above are simply tests of an
SCA runtime - and we have

these already, including "invalid documents".
 However, they only check on

the valid behaviour of an SCA runtime.

If we make claims about documents themselves being
valid or invalid, it is

my opinion that such claims are pretty worthless unless
there are tests to

back them up.  Such tests would actually have
to be able to accept some

random document and evaluate whether that document
was a valid document or

not.  I even heard one person on a previous call
ask for a tool of exactly

this kind.

At the moment I don't think it is worth the time and
effort of building

tests that could validate documents.  As a result,
I take the view that

there is little to be gained in having documents as
conformance points.

The specification concentrates on what an SCA runtime
does with a document

and we have the tests to check out that the runtimes
are doing the right

things.

Back to answer Martin's question of whether the runtime
is right or whether 

the document is right when some failure occurs.  If
the SCA runtime passes

our tests then the first reaction should be to say
that the runtime is

right and the document is incorrect.  If this
isn't the case, it actually

shows a hole in our SCA runtime tests - go plug the
hole!!

 

> -Anish

> --

> 

<snip>

Unless stated otherwise above:

IBM United Kingdom Limited - Registered in England and Wales with number
741598. 

Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU