Re: [xml-dev] A single, all-encompassing data validation language - good

["Anthony B. Coates (XML-Dev)" <abcoatesecure-xmldev@yahoo.co.uk>]

There isn't really a single "marketplace" as I see it.  It's a reasonable
generalisation to say that W3C XML Schema is the usual choice for people
who think of XML more as data, and RELAX NG has a lot of support from
people who use XML for marking up human-readable documents.  I'm not sure
how many people, if any, would be commissioning new DTDs from scratch for
anything these days, although people are still doing work maintaining the
existing DTDs rather than converting them.

I don't think anyone would complain if there was a single,
all-encompansing data validation language that had enough implementations
with enough performance, etc.  However, I don't see us getting close to
this in the short to medium term.  W3C XML Schema and RELAX NG are both
good for checking structural and datatyping contraints on XML documents,
but they *both* fall woefully short of being able to do all of the data
validations that people need to do.

Schematron is a useful complement, but in the past, I've always found that
it, too, was too limited for what I needed, and so I found myself using
XSLT instead.  Recently I've been using XQuery, which I think has a lot to
recommend it as a language for encoding a layer of data validation that is
at a level above what the schema languages do.  The integration of both
XSLT 2.0 and XQuery (in some implementations) with W3C XML Schema means
that you can do this level of data validation in the same single step as
the Schema validation, which can be helpful.

Beyond that, you can always write code in Java/C#/Python/etc. to check any
other rules that aren't convenient to do with Schematron/XSLT/XQuery.  One
way or another, your applications probably have this kind of code, but I
think it's useful to pull it out into a separate validation layer, as that
greatly reduced the complexity (e.g. number of try/catch blocks) in your
core application code, which improves readability and maintainability.
The one obvious exception to this is if you do "just in time" validation,
i.e. if you sometimes are able to avoid some particular computationally
expensive data validation depending on the application state.  In that
case, it may not be performant to separate out the validation code.
However, that does tend to be the exception, not the rule.

Above and beyond any question of which language(s) you use for data
validation is the question of who checks that the validation rules have
been implemented correctly?  A not uncommon situation is that a business
analyst is responsible for writing (some of the) validation rules in some
format, and a programmer is responsible for writing an implementation.
However, this leads to the problem of who is capable of checking that the
programmer has correctly implemented what the business analyst asked for.
Neither is fully qualified for the job.

For that reason, there is a lot of interest in whether data validation
languages, at least and the Schematron/XSLT/XQuery and higher level of
data validation, can be designed in a way that makes them readable and
accessible to people who are less technically oriented and more business
oriented.  At this level, you can't really separate languages from tools;
what would win in this space is good combination of language and tool that
makes it easy to write rules so that there workings and their impact can
be understood by people who aren't hard core programmers.  I haven't seen
anyone do this successfully enough yet.  I think we really need to crack
this problem before we can start imagining that we could genuinely build
an all-encompassing data validation language.

Cheers, Tony.

On Fri, 03 Aug 2007 11:14:34 +0100, Andrew Welch
<andrew.j.welch@gmail.com> wrote:

> On 8/2/07, Costello, Roger L. <costello@mitre.org> wrote:
>> 4. Is it in the best interest of the marketplace to have a single,
>> all-encompassing data validation language, or is it better to have
>> multiple data validation languages that work together?
>
> Do you have any information on the current state of the marketplace?
>
> Are you sure that XML Schema isn't already lined up as the single, all
> encompassing validation language once people move on from DTD?  (as
> in, the only market share it will be gaining is from DTD, as the rest
> is negligible)
>
> I've asked this before (so I'll ask it again at the likelihood of
> being flamed) - where are the Schematron implemenations?  As far as I
> can see, it's a specification and a couple of stylesheets...  A while
> back I really wanted to augment my XSD with XPath and looked at
> Schematron, but there was nothing there - just a new language to learn
> and stylesheet to ultimately turn that language into another
> stylesheet - not exactly enough to warrant the time investment to
> learn the language when most people know XPath already (and writing
> the XPath directly you're not constrained by limitations of the
> "Skeleton" transform).
>
> To be honest I was underwhelmed and annoyed that there was so much
> apparent positivity for something that didn't have any substance.  I
> don't know what its like now, hence my question - does Schematron have
> any market share for XML Schema to take when 1.1 is released?

-- 
Anthony B. Coates
Senior Partner
Miley Watts LLP
Experts In Data
UK: +44 (20) 8816 7700, US: +1 (239) 344 7700
Mobile/Cell: +44 (79) 0543 9026
Data standards participant: genericode, ISO 20022 (ISO 15022 XML),  
UN/CEFACT, MDDL, FpML, UBL.
http://www.mileywatts.com/