Actually my use of the phrase "ultimate form of reference" is incorrect. The test cases are not used quite in that way. From what I have seen during the interop a combination of documents are used to determine whether something was conforming or not.
But what I was trying to communicate was that prose alone is insufficient to define conformance because prose can typically lead to multiple interpretations.
Here is a recent example of a new ECMA document :: ECMA-404 (http://www.ecma-international.org/publications/standards/Ecma-404.htm) which defines JSON.
This is basically the same thing that can be found at www.json.org
It is a simple enough specification and uses a degree of formal definition.
However, this simple definition has several interpretations. For example, can JSON start with a top level array element []? The JSON inventor and others created JsonML which leads one to believe that the answer is yes, yet others say no. What the document needs are some examples of what is allowed. (There is also some confusion about escaping characters a simple query on google will show you that). The result are several commercially available non-conforming JSON parsers depending on who you talk to.
Best Regards,
Mark Joseph, Ph.D.
President
P6R, Inc
408-205-0361
Skype: markjoseph_sc
http://www.linkedin.com/pub/mark-joseph/0/752/4b4
Patrick Durusau <> , 4/1/2014 7:07 AM:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
I think you "hit the nail on the head" when you say:
*****
What I really liked about KMIP was the conformance test details. My
company P6R ships a KMIP client. And those conformance test details
where extremely helpful in both building our client and testing with
KMIP servers. When there is a disagreement about what should go over
the wire, as there always is, those detailed test details are the
ultimate form of reference for both server and client.
*****
The "ultimate form of reference" caught my eye.
Yes, test cases are very important but test cases only test the set of
values specified. That is if a range of values is allowed, a test case
only tests one or more of those values, not all of them.
There is no question that test cases are great for "testing" but
"testing" isn't conformance, which is an entirely different matter.
Many applications may conform but only some of their capabilities are
tested by any set of test cases. Things of test cases as subsets of
the permissible values.
Is supporting a subset of permissible values conformance? It's
certainly a sign of possible conformance but what of the values not
tested?
That to me seems to be the nub of the test cases issue, well, along
with the notion that what isn't specified in the prose is covered by
the test cases.
The prose really should cover all that needs to be said and then use
test cases to their best advantage as you say, for testing one aspect
of conformance.
Hope you are having a great day!
Patrick
On 03/31/2014 05:55 PM, Mark Joseph wrote:
>
>
>
> My second related beef, is that the text of the profiles should be
> the authoritative source for what compliance actually means (i.e.,
> the normative text). If you have to look at the conformance test
> details, which in ISO would simply be "informative" text, you've
> potentially lost the battle to ensure interoperability with KMIP.
>
>
> Hi Eric, I want to make sure I understand what you are saying here.
> The "conformance test details" you are referring to are the
> detailed test case documents?
>
> I have done a lot of work with IETF standards, for example the SSH
> RFCs. I got to tell you they are terrible. There are so many
> different interpretations that when I was building my company's
> version of an SSH server (from scratch) the only way I got it to
> work was to test it against the SSH server on linux. There is no
> other way that would work. I have also worked on several IETF
> email protocols and even though those specs are better the only way
> to get an implementation to work is again by testing it against
> many other implementations. I have been doing this protocol
> development for too long.
>
>
> What I really liked about KMIP was the conformance test details.
> My company P6R ships a KMIP client. And those conformance test
> details where extremely helpful in both building our client and
> testing with KMIP servers. When there is a disagreement about
> what should go over the wire, as there always is, those detailed
> test details are the ultimate form of reference for both server and
> client. I only wish that the SSH IETF documents had something
> similar.
>
>
> Best Regards, Mark Joseph, PhD President, P6R Inc.
>
>
>
- --
Patrick Durusau
Technical Advisory Board, OASIS (TAB)
Co-Chair, OpenDocument Format TC (OASIS)
Editor, OpenDocument Format TC, Project Editor ISO/IEC 26300
Former Chair, V1 - US TAG to JTC 1/SC 34
Convener, JTC 1/SC 34/WG 3 (Topic Maps)
Co-Editor, ISO 13250-5 (Topic Maps)
Another Word For It (blog): http://tm.durusau.net
Homepage: http://www.durusau.net
Twitter: patrickDurusau
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTOsgCAAoJEAudyeI2QFGolHwQAIsDOZEcrqott9415a/u/SjY
v4RsWaoN8L8g+8zMEMFGfDzF7WsYWqQrq8QY9h+zMf/RSn/B0K+M0VP0NwbKhM7X
a36YF0yXiaewssk3XkeCldhIobxV7gqH7hDxKHGXHMLcfXfP2fWgYaMRiZaZtQ1P
uvXUGJpg6szdKI6/5wX6r6QXVPAjgCqli+nRyaOhTVRYIQC7HKuho0sgzpAco0Vw
io4UHyLUyeuYHKMoEKeOU3HwubaGWXpT69qsgVQfdfs5fKZoEqCik7pKAmK5ZqJw
Rd4INROdoz/ngDZ6dJn7xuFoSAhc+1WQAGwS0mspH8GBI18ttPgYBgf35/kapctN
6tKWjov7TjKeg7lim0uMyNNgv4xcBQLbWqfRG3EAWu+7+BiS9eHtYHch9LoOJk+x
8/RUJZ1aH2wugu9SsvO4wXjYUmlpoDqFoCK/SmH79NN6bNtvbGLyYlfqHIylizBy
lIUf1UK8r096WVNFgVl90oL8X354FRvMM4SvAySb5XwsEJ5rwIr9PzXsY+/foQlt
0Jo+C4qJCTxMP6B6mG81f0puLEgjCjwl18gf4Anb/9PV0ANtHxpvaOQH1V3EWnPi
O1DN5GYF/UGVl4xinZnnXeHxmGewRMkdOQ4rRnsjzudkoHQ2tbQv90WZCCGfNU+0
O6XE2tfvB2L07FnQqqDv
=t5nz
-----END PGP SIGNATURE-----
