conformance — archive

[lynne rosenthal]

Hello All Attached is a new version of the Conformance Requirements Guidelines. This document will be discussed at our next meeting - F2F in Orlando, THURSDAY DEC 13, 9-11:30. YOUR HELP is needed. 1. Please read the document 2. The document needs more and better examples - can you provide some based on your expertise and experience? 3. What else should be included in the document (or removed) Thanks. lynne Title: Conformance Requirements Conformance Requirements Guideline (alternate title? Principles of Conformance Requirements) Version 0.3 November 21, 2001 Editors : Lynne Rosenthal (lynne.rosenthal@nist.gov) Contributors: Mark Skall ( mark.skall@nist.gov ) Lofton Henderson (lofton@rockynet.com) David Martson (david_marston@lotus.com) Abstract Document identifying the conformance requirements that need to be included or addressed in specifications.  Target audience is primarily specification developers, followed by conformance test suite developers and implementation developers Status of this Document Pre-Note -- this document will be formatted as a W3C Note. As such it will serve as a contribution towards the QA Specification Framework, addressing some but not all of the topics of the latter.  It is not intended to be the Specification Framework document but to remain as a Note, providing supplemental information that is focused conformance section and statements in a specification. --> Second Committee Draft Document Version History 21 Nov 2001 updated based on input from QAWG Oct 2001 input from OASIS Conformance TC --> 21 Nov 2001 updated based on input from QAWG 25 Oct 2001 updated based on Sept 13 Telecon 22 Aug 2001 updated based on Aug 16 Telecon. 2 Aug 2001 initial draft Reference Documents ISO Guide 2: Standardization and related activities -- General vocabulary. ISO/IEC Directives Part 3: Rules for the structure and drafting of International Standards ebXML Technical Architecture Specification , Conformance Clause W3C WAI Guidelines UNICODE 1. Introduction The objective of this document is to identify the conformance requirements that shall be included or addressed in specifications. Conformance requirements are the expression, in the form of a statement, which conveys the criteria to be fulfilled [ISO Guide 2]. The conformance requirements are stated in a conformance clause or statements within the specification. Note that additional conformance information and guidance may be contained in supplemental documentation. This document describes the purpose and scope of a conformance clause as well as the associated issues that a conformance clause should address. Where ever possible, sample text and examples will be given. The information contained is produced as the result of extensive experience in the development and implementation of conformance clauses and test suites for consensus standards and specifications. It is based on the principles and requirements prescribed by international standards (e.g., ISO/IEC and IEEE) as well as extractions from ebXML, OASIS and W3C specifications. 2. Scope and Audience This document specifies the general requirements and definitions concerning conformance and related issues. It is intended to contribute fundamentally towards mutual understanding amongst developers of specifications and conformance test suites and tools. It is also intended to provide a suitable source for teaching and for reference, briefly covering basic theoretical and practical principles of conformance. This document will not define specific conformance requirements for any specific specification -- this is the responsibility of committees chartered to develop functional specifications. This document is intended primarily for the developers of specifications to help enable them to develop conformance requirements within their specification and to create a testable, unambiguous specification. Secondary audiences include, but are not limited to: developers of conformance test suites, software implementers, international standards bodies, and other industry organizations. 3. Conformance to this document In order to conform to this Conformance Requirements document, a specification shall contain a conformance clause. It is recommended that the conformance clause exist as a separate section within the specification, so that it is clearly identifiable. A specification that conforms to this document shall: use the conformance keywords (section 7.2) address all issues (topics) in Section 8 and indicate the applicability and means for achieving conformance to each issue, examine the issues in Section 9, determine if each issue is applicable and define the conformance requirements for applicable items. For each issue the specification shall be clear whether or not the issue is addressed and its disposition. For example, if a specification does not contain levels it should be clear to the reader that levels are not supported. One method to ensure this clarity is to explicitly state that levels are not supported. [Editor note: we need to make sure that this conformance clause conforms to this document - add what is needed to address all issues in Section 8, etc. 4. Normative references The following normative documents contain provisions, which through reference in this text, constitute provisions of this document. At the time of publications, the editions indicated below were valid. All standards are subject to revision, and parties to agreements based on this document are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. ISO/IEC Guide 2: Standardization and related activities -- General vocabulary ISO/IEC Directives Part 3: Rules for the structure and drafting of International Standards. RFC 2119 : Keywords for use in RFC's to Indicate Requirement Levels UNICODE 5. Informative references World Wide Web Consortium Process Document W3C Publication Rules W3C Conformance --> 6. Terms and definitions For the purposes of this document, the following relevant terms and definitions apply: Certification -- the acknowledgement that a validation has been completed and the criteria established by the certifying organization has been met. Conformance -- the fulfillment of a product, process, or service of specified requirements. Conformance Testing -- a method of verifying implementations of a specification to determine whether or not deviations from the specification exist. Implementation -- the realization of a specification -- it can be a software product, system, program, protocol, application, or document instance. Validation -- the process of testing software for conformance to a specific specification. 7. Conformance Clause The conformance clause is a section of a specification that defines the requirements, criteria, or conditions that must be satisfied to claim conformance. The conformance clause identifies what must conform and how conformance can be met. Typically it is a high-level description of what is required of implementers and applications. It may refer to other parts of the standard. It may specify sets of functions, which may take the form of profiles, levels, or other structures. Additionally, it may specify minimal requirements for certain functions and for implementation-dependent values.It may also specify the permissibility of extensions, options, and alternative approaches and how they are to be handled. Every specification shall contain a conformance clause. 7.1. Rationale for a conformance clause A conformance clause: Promotes a common understanding of conformance and what is required to claim conformance to a specification, Facilitates consistent addressing of conformance within a specification, Facilitates consistent addressing of conformance across related specifications, Promotes interoperability and open interchange, Encourages the use of applicable conformance test suites Promotes uniformity in the development of conformance test suites. 7.2. Conformance keywords There are specific words that are used throughout the specification to denote whether or not requirements are mandatory, optional, or suggested.; Using these keywords help to identify the testable statements in a specification.; Although the keywords used within the ISO/IEC community differ from the keywords used within the IETF communities, they achieve the same results.; Use of these keywords should be consistent (i.e., use the ISO keywords or the IETF keywords, but do not use both). ISO Keywords: SHALL to indicate requirements strictly to be followed in order to conform to the standard and in which no deviation is permitted.; Equivalent expressions include: is to, is required to, has to, it is necessary. Do not use MUST as an alternative for shall. SHALL NOT converse of SHALL. SHOULD to indicate that among several possibilities one is recommended as particularly suitable, without mentioning or excluding others.; SHOULD NOT converse of SHOULD. MAY to indicate a course of action permissible within the limits of the standard.;; Equivalent expressions include: is permitted, is allowed. NEED NOT to indicate a course of action is not required. CAN statement of possibility and capability, whether material, physical or causal.; Equivalent expressions include: be able to, it is possible to. CANNOT converse of CAN. IETF Keywords ( [RCF2119] ): MUST the requirement is an absolute requirement of the specification. MUST NOT the requirement is an absolute prohibition of the specification. REQUIRED see MUST. SHALL see MUST. SHALL NOT see MUST NOT. SHOULD there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a difference course. SHOULD NOT there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label. RECOMMENDED see SHOULD. MAY One vendor may choose to include the item because a particular marketplace requires it or because the vendor feels that it enhances the product while another vendor may omit the same item.  An implementation that does not include a particular option MUST be prepared to interoperate with another implementation that does include the option, though perhaps with reduced functionality. In the same vein an implementation, which does include a particular option MUST be prepared to interoperate with another implementation that does not include the option (except, of course, for the feature the option provides.) Additional keywords include: NORMATIVE the prescriptive parts of the specification, providing that which is necessary in order to be able to claim conformance to the specification. Note: the conformance scheme of a specification can allow claimants to exempt certain normative provisions as long as the claim discloses the exemption. INFORMATIVE (NON-NORMATIVE) statements provided for informational purposes, intended to assist the understanding or use of the specification and shall not contain provisions that are required for conformance. 7.3. General principles An objective of any conformance clause and its related conformance statements is to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional. To achieve this objective, normative and informative sections shall be evident and if necessary, labeled accordingly, uniformity of structure, of style, and terminology shall be maintained within the specification, identical wording shall be used to express identical provisions and analogous wording shall be used to express analogous  provisions. 8. What to Address in a Conformance Clause 8.1. What needs to conform The conformance clause identifies the "class of products" (i.e., object of the claim) that will be developed, where "class of product" may be an implementation, application, service, and/or protocol (e.g., content, user agent, authoring tool). Additionally, the clause specifies the conditions that shall be met in order to claim conformance for that class of product (i.e., make a valid claim). It may also specify that which is not a requirement. There may be several classes of products that are identified, each with its own conformance statement or set of conformance criteria. For example, the W3C DOM Recommendation defines a DOM implementation or host implementation (e.g., a browser) and a DOM application or client application (e.g., a script which runs in a browser). 8.1.1. Modularity of (software) A class of product may consist of several integrated components rather than a single piece of software (e.g., browser). Conformance may be defined in terms of the integrated software (system) and/or for each component. Any restrictions or constraints on the number or types of components that make up the "subject of a conformance claim" shall be specified. For systems that are comprised of several components, it may be sufficient to state that conformance to the system is equivalent to conformance of all the required components considered individually, and the system satisfies at least the minimum conformance requirements for each of those components. For example, the conformance clause in the ebXML Technical Architecture states, "ebXML conformance is defined as conformance to an ebXML system that is comprised of all the architectural components of the ebXML infrastructure and satisfies at least the minimum conformance requirements for each of the ebXML technical specifications." 8.1.2. Specifying conformance claims A specification may apply and group requirements into various conformance levels. These levels may relate to functionality, impact and/or incremental degrees of implementation. The conformance clause shall identify and define all conformance levels. For example: The W3C Web Accessibility Guidelines define three levels of conformance (Level A, Double-A and Triple A) based on the checkpoint priority levels satisfied. Conformance Level A: all Priority 1 checkpoints are satisfied; Conformance Level Double-A: all Priority 1 and 2 checkpoints are satisfied; and Conformance Level Triple-A: all Priority 1, 2, and 3 checkpoints are satisfied. The specification may provide the specific wording of the claim (appendix A provides sample conformance claims). It may also require specific information be contained in the claim, such as name/date/version of the specification, test suite, and tested product. The specification shall impose no restrictions about who can make a conformance claim (e.g., vendor, user, third party) or where the claims may be published. It may provide additional information regarding the responsibility of claimants. 8.2. Profiles and Levels Often implementations do not use all the features within a specification. In order to accommodate these implementations it may be desirable to divide a specification into sets of functions. Implementors would be allowed to implement one or more of these sets rather than the entire standard. These sets are commonly implemented as profiles or levels. Profiles are used as a method for defining subsets of a specification by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users. Specifications that explicitly recognize profiles should provide rules for profile creation, maintenance, registration and applicability. Appendix A provides additional information on profiles. Levels are used to indicate nested subsets of functionality, ranging from minimal or core requirements to full or complete functionality. Typically, level 1 is the minimal or core of the specification that must be implemented by all products. Level 2 includes all of level 1 and also additional functionality. This nesting continues until level n, which consists of the entire specification. It is possible for a specification to have both profiles and levels. If profiles and/or levels are defined, the conformance clause specifies which (if any) of these profiles and/or levels is mandatory. Additionally, any conditions associated with a particular profile, level or combination of these needs to be specified. Experience has shown that having too many profiles and/or levels can inhibit interoperability as well as add confusion to the marketplace. If profiles and/or levels exist, the specification shall indicate the conditions for claiming conformance to a specific profile and/or level. In particular, consider whether or not a claim of conformance to a particular profile/level can include functionality or features of a higher profile/level. Typically, implementations that purport to conform to a specific level n of a specification may include functionality defined within one of the higher levels n+1. The specification requirements for extensions (Section 8.2.2) are applicable for profiles/levels and should be adhered to. 8.3. Extensions An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Allowing extensions affects how conformance is defined as well as what conformance claims could be made. Care should be exercised in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whether extensions should be allowed in conforming content or implementations. It may be better to stipulate that the presence of extensions is non-conforming, than to have content or implementations which cannot interoperate but which may be claimed as conforming to the specification. Appendix C provides additional information about extensions. 8.2.1. Disallow Extensions If a specification disallows extensions, then the conformance clause shall specify that extensions are not allowed and that implementations of the specification shall precisely implement the complete specification. This is called strict conformance. Strict conformance is often imposed on applications (i.e., content) of a specification. For example, a software program or XML document instance. Strict conformance may also be imposed on implementations (e.g., as in Ada). Note, that this prohibition of extensions could be applied to a specific profile or level rather than to the entire specification. 8.3.2. Allow Extensions If specification allows extensions, then the conformance clause shall state the conditions under which extensions are allowed, the applicability of the extensions, their affect on conformance claims, and any limitations or restrictions on the use of the extension. The conformance clause should require that each implementation shall fully support all required functionality of the specification exactly as specified and that the use of extensions shall not contradict nor cause the non-conformance of functionality defined in the specification. Additional requirements may include: Extensions shall not re-define semantics for existing functions, Extensions shall not cause standard-conforming functions (i.e., functions that do not use the extensions) to execute incorrectly, Extensions shall follow the principles and guidelines of the specification they extend, that is, the specifications must be extended in a standard manner (see section below), For implementations and/or applications that contain extensions, extensions shall be clearly described in supporting documentation and the extensions shall be marked as such within the implementation/application, For implementations that contain extensions, there shall be a mode under which the implementation can be directed to produce only conformant files (documents) or to operate in a strictly conformant manner. 8.4. Discretionary Items 8.4.1. Implementation defined values In some instances, it may not be possible to define the behavior or values of a function. Implementation dependent means that an implementation may determine the effect (rather than having the effect mandated by the specification). However, such effects shall be consistent within a single implementation (e.g., a browser's rendering of ??? shall be the same for every invocation regardless of the document instance). Details in a specification may deliberately be omitted (i.e., not specified), so as to provide freedom to adapt implementations to different environments and different requirements. In general this is not a recommended practice. Caution should be exercised if details are omitted and used only in a limited number of instances. Specifications shall indicate implementation dependencies and where applicable, address allowable differences between implementations including, Implementation dependent ranges, data, minimum or maximum values, etc., Values that may be different for different conforming implementations of the standard, Environmental values (i.e., language and local settings) 8.4.2. Alternate approaches Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats, protocols, or codes). In such a case, the conformance clause shall specify under what conditions an implementation is considered to be conformant. Some possible ways to define conformance include mandating that an implementation shall: implement only one approach, implement every approach, be allowed to implement none of the approaches. Note: if the specification doesn't describe the different approaches, this becomes an implementation detail irrelevant to conformance. 8.5. Internationalization – Languages and Character sets Every specification shall identify, either by default or explicitly, a single natural language or a more formal specification language (e.g., IDL, UML) edition as the normative version. Every specification shall specify whether it permits multiple or alternative natural languages, language bindings and/or character encodings. If it does, it shall specify the languages and encodings that must be supported by conforming implementations. Additionally, the error conditions and/or behavior to handle situations in which unsupported languages or encodings are encountered shall be defined. When specifying characters, the Unicode Standard [ISO 10646] shall be used. 9. Additional Issues to Address 9.1Implementation conformance statement (questionnaire) A specification may include an Implementation Conformance Statement (ICS) or questionnaire and require its completion as part of a conformance claim. An ICS is useful in clarifying and declaring optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against the relevant requirements and against those requirements only. The ICS is also helpful in describing the expected interoperability to be achieved with other implementations or applications of the specification. If an ICS proforma is included as part of the specification, it shall be explicitly identified as either a normative or informative part of the specification. Anything to add from the XSLT work? 9.2. Test Assertions A specification may include test assertions as part of the specification. A test assertion is a statement of behavior, action or condition that can be tested or measured. It is derived from the specification requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for a requirement in the specification. Each test assertion results in one or more test cases. Including test assertions as part of the specification facilitates and promotes the development of conformance test suites and tools. Specific benefits include: Developing test assertions in parallel with the specification improves the specification by helping to uncover inconsistencies, ambiguities, holes, and non-testable statements, Developing test assertions in parallel with the specification will ensure consistency between the specification and assertions, Allowing test assertions to be reviewed and accepted by the specification developers and the public, Providing a common set of assertions (and thus interpretation of the requirements) from which test developers can develop conformance tests, Encouraging the early development of conformance tests that can be used by implementers during the development of their implementation, Achieving comparability between the results of corresponding tests developed by different organizations, Achieving confidence in the resulting tests as a measure of conformance. Including test assertions as part of the specification has been done is several IEEE and ISO standards including IEEE POSIX and ISO 10303 (STEP). 9.3. Specify a testing methodology or program A specification may provide a test framework, methodology and/or procedures for testing to the specification. This type of information ensures consistency between testing programs and organizations, and provides confidence in those testing programs. If any of this information is provided, it shall be explicitly identified as either normative or informative guidelines. The test method may describe the conformance testing approach – the use of methods involving rigorous proofs of correctness in which conformance can be conclusively and exhaustively demonstrated (e.g., the syntactic validators for HTML, CSS, WAI content) or the use of methods involving falsification testing. The test methodology may describe the different types of conformance tests and tools that need to be developed, the type of test materials that need to accompany the tests, and the type of information that must be contained in a test report. The procedures for testing may describe the organizational structure, activities and responsibilities for external organizations that establish and operate a testing service for the specification. The procedures for testing may prescribe how testing is conducted, e.g., self-declaration or third party testing laboratories. It may also provide a step-by-step guide for using the tests or tools correctly so that the results may be repeatable and reproducible. This type of information is provided as normative sections in several standards, e.g., ISO 10303 (STEP) and ISO 15046 (Geographic Information), and as part of several consortia specifications. Appendix A: Sample Conformance Claims (Informative) In general, a conformance claim should contain the name and version of the tested implementation, the specification, name and version of the test suite, date testing was completed, conformance level (or profile) satisfied, and the results of the testing. For example: Name of Implementation and version has been tested for Level L conformance to Name of Specification using the Name of Test suite , ver X.X on YY-MM-DD and no nonconformities were found. This Name of Implementation (fully specified) has been tested for conformance to Name of Specification, in accordance with the XXX Validation Procedures using the Test Suite and testing environment listed below: -Name of Certificate Holder: -Implementation Identification: -Testing Environment (hardware/software): -Test Suite name and version -Level of Conformance: -Nonconformities: -Test Report: provide a URI Specific Examples The Web Content Accessibility Guideline requires a claim to contain the title of the guidelines document, its URI, the conformance level satisfied, and the scope covered by the claim (e.g., page, site), for example: This page conforms to W3C's "Web Content Accessibility Guidelines 1.0", available at http://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505, level Double-A Appendix B:  Profiles (Informative ) The following is extracted from ISO 8632 Computer Graphics Metafile Standard A profile of a specification defines the options, elements, and parameters necessary to accomplish a particular function and maximize the probability of interchange between systems implementing the profile.  Profiles are defined to meet the requirements of application constituencies who are asked to adhere to the same subset of the specification.  A profile may be a subset of a single specification or may be part of the set of interrelated standards and profiles assembled for the purpose of accomplishing a larger functional purpose. A profile shall not specify any requirement that would contradict or cause non-conformance to its specification. A profile may: Give the meaning of implementation dependent semantics of some elements, Enforce common resolution of ambiguous semantics, Ensure that identical use of identical elements and parameter values have the same meaning, Specify subsets or groupings of publicly defined extensions, Prohibit undefined or ill-defined elements or parameter values. Profiles provide a means to: Improve interoperability between implementations by inhibiting the proliferation of private subsets of a specification, Provide a foundation for testing and promote uniformity of conformance tests, Enhance the availability of consistent implementations of a profile. Appendix C: Extensions An extension may be private (often vendor specific) or may be public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations.  Public extensions are extensions in which the syntax, semantics, identifiers, etc are defined and published allowing anyone to implement the extended functionality. Mechanism to allow extensions – [Ed NOTE – need to clean up this section] One mechanism to allow extensions within a specification is to provide a standard way of defining the extension.  Basically, this provides a "standard way of being non-standard". This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate action (i.e., to ignore or to implement).  The nature of the extension may dictate the method for defining the extension.  It may be possible to define a function that indicates external (from the specification) functionality. This external function may take the form of an escape or control character or be an identifier, which whenever invoked indicates an extension follows. Another method, especially when extending a list of numeric parameters is to use a scheme where positive values represent standardized values and negative values are reserved for private use. For example, the ISO Computer Graphics Metafile (CGM) is the standard on which the W3C WebCGM Recommendation is based.  It provides both a standard function (GDP element) for defining private graphics functionality, as well as the use of negative values to define private values. Another mechanism that minimizes interoperability problems when extensions are allowed is to have a register for extensions.  This document, distinct from the official specification, contains a list of recognized extensions to the standard.  See section below. Registration of implementer extensions or defined values Registration is a procedure that allows extensions to be acknowledged and made available to the public.  The procedures provide for the extension to have some rigor and technical review.  Typically, the committee developing the specification is responsible for processing the registration of an extension, thus ensuring adequate quality of a proposed extension and a technical description sufficient to be uniformly implementable. Often registered extensions may migrate into a later version of the specification