How to implement validation rules in an environment where some datarequi

["Costello, Roger L." <costello@mitre.org>]

Hi Folks,

Are you in this situation: 

a.  You create some XSD schemas, build an application around them, and deploy the application. 
b.  A short time later some higher-up authority tells you, "Well, you need to incorporate this additional data." 
c.  So you update your XSD schemas, rebuild your application, redeploy. 
d.  Then, a short time after that the higher-up authority tells you, "We've made some changes and additions so please incorporate them." 
e.  So you update your XSD schemas again, rebuild your application again, redeploy again. 
f.  And this goes on over and over. 
g.  You think to yourself, "There has got to be a better way. Most of my XML is unchanging; how can I partition off the static part and then dynamically incorporate the new data requirements?"

If you are in this situation, then this discussion will be of keen interest to you.

Below I have tried to describe the issue as best I can. Many questions remain. I'd appreciate hearing your thoughts on this issue.   /Roger

-------
Issue 
-------
    How should validation rules be implemented in an 
    environment where some data requirements change 
    frequently while others are unchanging? 

    Can validation rules be designed in a way that 
    changes in data requirements can be seamlessly
    integrated without breaking the implementation
    of the unchanging data requirements?

-----------------
Terminology
-----------------
    For purposes of discussion let's assume "validation 
    rules" are implemented using a combination of
    XSD schemas and Schematron schemas.

------
Fact 
------
    XSD schemas tend to be compiled into code, rather 
    than dynamically used.  So a change to the XSD schema 
    can be a big deal, especially after deployment. An XSD 
    schema change often means a new release of the 
    application or database.

---------------
Implication
---------------
    Place the validation rules for those data requirements
    that frequently change into a separate layer from the
    XSD schemas.

--------------------------------------------------------
Categories of Data that Frequently Change
--------------------------------------------------------
    1. Enumeration list changes: the list of valid values for an element or attribute frequently changes. 

    2. Metadata changes: the set of metadata frequently changes.

Example of enumeration list changes:  the list of book publishers frequently changes -- the set of valid values of the <Publisher> element changes frequently.

Example of metadata changes: one day the higher-up authority says, "Add to each <Book> element an indication of whether it is in-stock," another day the higher-up authority says, "Add to each <Book> element an indication of its average user rating," and so on, endlessly adding and changing metadata.

-----------
Solution
-----------
Schematron schemas can be used for the category "enumeration list changes." For example, in the XSD schema declare the <Publisher> element of type xs:string" and then in the Schematron schema have assertions that specify the list of today's valid publishers.

-------------
Questions
-------------
1. Rick Jelliffe says, "the assertions should be dynamically loaded." What does that mean? How does one dynamically load Schematron?

2. How would "metadata changes" be implemented? For example, how would we add to each <Book> element an indication of whether it is in-stock?

3. I listed two "Categories of Data that Frequently Change." Are there other categories? If yes, would you provide an example please?