One answer is to use a schema language with an ability to distinguish between the strengths or natures of different constraints.
You need a schema language that helps you ask and specify, when you have some constraint, questions like WHY have that constraint, HOW should it be reported, and WHAT are the processing ramifications? If your schema language treats documents as existing outside the SDLC and where there is no separation of concerns between different constraints and their rationales, then over time, why would you expect not to be mugged by reality?
For example, in Schematron,
- every assertion (or rule, or pattern) can have an attribute "@role" which lets you classify the constraint;
- you can attach extra information giving boilerplate (or dynamic, specific) further information for the humans in "diagnostic" elements;
- you can attach properties to drive subsequent computer-oriented processing using "properties", which may be dynamic and contain any foreign elements;
- you can use the @flag attribute to raise a flag on the document (flags are raised on an assertion, but are true for the whole document) for dispatching.
<sch:rule context="book">
<sch:assert id="r1a1" test="count(author) =< 100" role="warning" diagnostic="as-of-2022" flag="non-2022"
>A book is expected to have no more than 100 authors.</sch:assert>
...
</sch:rule>
...
<sch:diagnostics>
<sch:diagnostic id="as-of-2021">This constraint is a processing capacity required for
systems developed starting 2021; documents that do not meet this constraint may need
special attention.</sch:diagnostic>
<sch:diagnostic id="as-of-2022">This constraint is a processing capacity required forsystems developed starting 2022; documents that do not meet this constraint may need
special attention</sch:diagnostic>
...
</sch:diagnostics>
<sch:properties>
<sch:property id="book-special-treatment">
<my:handle-specially type="book" />
<sch:span class="title"><sch:value-of select="@id"/></sch:span>
</sch:property>
</sch:properties>
This would typically produce an SVRL (Schematron Validation Report Language) result like this:
<svrl:failed-assert
location="/book[1]"
id="r1a1"
test="count(book) =< 100"
flag="non-2022"
role="warning">
<svrl:text>A book is expected to have no more than 100 authors.</svrl:text>
<svrl:diagnostic-reference id="as-of-2022">
<svrl:text>This constraint is a processing capacity required for systems developed starting 2022; documents that do not meet this constraint
may need special attention</svrl:text>
</svrl:diagnostic-reference>
<svrl:property-reference id="book-special-treatment">
<my:handle-specially type="book"/>
<svrl:text>
<svrl:span class="title">NCD Risk Factor Collaboration</svrl:span>
</svrl:text>
</svrl:property-reference>
</svrl:failed-assert>
You can see that there is a separation of concerns allowing
1) The constraint information provided by SMEs to be expressed in a way that notional users
can understand. (The constraint text.)
2) Extra metadata to be attached to say the severity or nature of that constraint (@role="warning") which can be used for triage, GUI, notifications and pipeline dispatch,
3) Information to categorize the whole document accordingly (@flag="non-2022")
4) Extra information to convey to humans (e.g. whoever is first inline to fix the issue) to explain why this is a problem (the text in the diagnostics element)
5) Extra information for subsequent handling systems to route and process this document (here, an element in a foreign namespace, and the title of the document)
So we see that in this schema, there were also some constraints for system for 2021 and some for 2022: perhaps the number of authors was upgraded. It is possible to model this in Schematron using the phases mechanism: you could have all the patterns that are applicable for systems developed in 2022 which those systems can use to validate, and another for systems developed in 2021 which those systems can use to validate. (Or you could have a phase which lets you detect whether a document requires a 2021 or a 2022 system, and validate that first, then forward the document to the appropriate system or to triage.)
Lets face it, it was common practise by the late 1970s to classify log messages by criteria (such as fatal, error, warning, info). Indeed it still is. But the grammar-based schemas do not provide any means to classify their constraints in useful ways: which means that they are only useful for the most rudimentary kinds of validation, for constraints that are likely true at all stages in the SDLC and document pipeline, and have equal significance. Consider this: some schema languages provide mechanisms for one schema to build on another in some way, but again they fail to provide simple annotations to convey to humans or systems why the change is needed or how it applies.
To put it another way, constraints themselves need to be "typed" or, rather, support standard and ubiquitous metadata annotations that progress into the PSVI or equivalent (in Schematron, the PSVI is the SVRL linking into the original document(s) being validated in a session.) The particular scheme used for those values may be specific to the use-cases and schema, but the standard elements or attributes need to be first-class parts of the schema language. (Note also that the XPaths themselves are not adequate to express anything useful for humans and systems in the rest of the SDLC or document pipeline.)
The problem of describing what goes in a single
instance of a document is, in a sense, trivial (e.g. a schema to allow
some objects to be serialized, transmitted, parsed and bound as objects
at the other end.) There is no time, no change, no gotchas, no intent, no systems, no humans.
The problem of how to do software engineering with
documents, where your schema needs to fit into a dynamic, evolving,
multi-party, non-uniform web, is much more challenging, I think. And violently practical. I think those challenges demand first-class support in the schema language (and because the future is unpredictable, it may well demand an approach based on mix-ins rather than inheritance: I don't think that is too controversial to suggest...) I think Schematron shows that it is entirely practical to have systems that handle those challenges pretty well, and consequently entirely reasonable to require that schema languages should support them.
Regards
Rick
Hi Folks,
Scenario: You are designing an XML Schema for validating XML instances that contain Book data. Each Book element contains Title, Author, Date of Publication, and ISBN. Some Books have multiple Authors. In your
current environment, in your current worldview, no Book has more than 10 Authors. So you constrain the Author element to maxOccurs="10":
<xs:element name="Author" maxOccurs="10" type="…"/>
But what if in the future there are Books with 100 Authors, then XML instances will fail validation. Should you set maxOccurs to unbounded?
No!
Here’s why:
You want to be informed when the world has changed, when the choices you made are no longer relevant. A world in which Books contain 10 times more Authors than you thought they would is a different world. You want
to be informed of this. Your XML Schema was originally written with one world view, if validation starts breaking that means you have got to rethink the initial stuff.
There is an analogous situation in programming. Should you constrain the size of an array or make it variable length? Here’s what John Carmack says: (https://youtu.be/I845O57ZSy4?t=4005)
I'm kind of fond in a lot of cases of static array size declarations. I went through this period where we should just make everything variable length because
I had this history in the early days where Doom had some fixed limits on it and then everybody started making crazier and crazier things and they kept bumping up the different limits -- this many lines, this many sectors -- and it seemed like a good idea that
we should just make it completely generic so it can go up to whatever. There are cases where that's the right thing to do, but the other aspect of the world changing around you is it's good to be informed when the world has changed more than you thought it
would. If you've got a continuously growing collection, you're never going to find out. You might have this quadratic slowdown on something where you thought “Oh, I'm only ever going to have a handful of these,” but something changes and there's a new design
style and all of a sudden you've got 10,000 of them. So I kind of like in many cases picking a number, some nice round power of two, and setting it up in there and having an assert saying “Hey, if you hit this limit, I need to know.” When that occurs, you
should probably think: “Are the choices that I've made around all of this still relevant if somebody's using 10 times more than I thought they would? This code was originally written with this kind of world view, with this kind of set of constraints, and I
was thinking of the world in this way.” If something breaks that means I’ve got to rethink the initial stuff.
