RE: [oic] Re: Idea about how to create conformance testing documents

Title: RE: [oic] Re: Idea about how to create conformance testing documents

Hello Mingfei,

Thanks for this great mail !

* About creating test documents

Personally, I think we'll end up with both a few fairly "large" documents and

a whole lot of specific test documents.

Meeting in the middle, like the test scenarios that Dennis and Stephen are

working on, is a very good way to get things started. Once we start testing them,

we'll find issues and sometimes we might want to isolate this issue in order to

better understand it.

Meaning: creating a small, specific test document for it.

After doing that, or if someone has created separate, small test documents for

some other feature(s), we could try to combine 2 or 3 documents in 1 larger one

and retest it. This retesting is important, because combining documents (well,

actually, features) may expose issues in unexpected ways.

For example:

- the last digit of a page numbers may get truncated after 100 pages

- a colored table might go wrong on a page with an image as background

- some apparently unrelated features may crash an application when combined

(I am not making this up, the vendor correctly identified and fixed this by

the way, but my point is that we *might* encounter strange behavior)

Sometimes we have no choice but to create a small test file.

For instance: a document cannot be encrypted and not encrypted at the same time,

so we have to create at small test document (well: at least 3, .odt, .odp, .ods)

for that.

* Regarding "feature unit"

The easiest way would be to take a look at the spec and look for numbered

parts. For example: 17.4 MIME Type stream

Although I'm not sure if this will always lead to the "unit"

For example: is 17.7.3 File Entry a unit ?

Or should we consider the paragraphs like "Full Path" as a unit ?

* About the process

I like your draft proposal, although I'm not 100% sure about 4.3

"The test document shall pass the semantic verification. That is the behavior

are consistent when the test document is opened and rendered by different

existed ODF applications. Once having confliction, TC vote to make a decision

which behavior is correct by majority voting numbers."

Maybe it's included in the "semantic verification", but the ODF spec itself

must be used.

For example: if the spec says that a thumbnail must be 128x128 pixels PNG, but

the majority of the OIC would decide JPEG is just fine, it's still not correct

according to the spec. Perhaps the spec itself must be changed, but it would

be unfair to decide that not following this version of the spec is OK.

On the other hand, if the spec is NOT clear about it, the OIC could test and

see what is common practice and indeed vote for having things included it in

a profile or a "best practice" guide, or as an improvement for ODF-Next

* Meta data

I don't completely understand your question here:

"A question here, I know test meta data is important, but I am not clear now

where we test metat data is appropriate to use for test document creation.

I understand test meta data should be used for interoperability testing to

record test case, right?"

Do you mean "how could metadata help us for interoperability testing" ?

We could indeed use some kind of metadata (xml) to record the results of the

test cases in various implementations (detailing what part of the spec is to

be tested, and what the result is). Non-members could also send in these test

results as an XML attachment in a mail to the oic-comment list.

Members could also do that using JIRA, so perhaps that's is a more practical

approach:

- providing the test documents + an overview what is to be tested (as an HTML

page) + a very simple text template.

- members and non-members could use this very simple text template to send in

results

- volunteers from the OIC could verify and record the results in JIRA

Best regards,

Bart