Re: [xml-dev] Here's how to process XML documents written in German

[Michael Kay <mike@saxonica.com>]

Tony prefaced his advice with "if you can't trust...". You (Roger) left 
that bit out.

Perhaps you did this on the basis that you should never trust anything. 
But if you don't trust anything, why are you processing the XML at all?

Michael Kay
Saxonica

On 30/01/2013 21:54, Costello, Roger L. wrote:
> Tony Graham wrote:
>
>> if you can't trust the encoding or normalization
>> form of the XML that you receive, then normalise
>> it as soon as you receive it so all of your XML is
>> consistent
> Is that a Best Practice? That is:
>
>      Best Practice: before processing any XML
>      document, apply an identity transformation
>      to it which converts the entire XML document
>      into a Unicode normalized form. For example,
>      each combining character sequence is converted
>      into a precomposed character.
>
> Thoughts?
>
> /Roger
>
> -----Original Message-----
> From: Tony Graham [mailto:tgraham@mentea.net]
> Sent: Wednesday, January 30, 2013 4:31 PM
> To: xml-dev@lists.xml.org
> Subject: Re: [xml-dev] Here's how to process XML documents written in German
>
> On Wed, January 30, 2013 6:47 pm, Costello, Roger L. wrote:
> ...
>> This XPath expression does the job:
>>
>> sum(//Posten[@*[normalize-unicode(name(.)) eq
>> normalize-unicode('währung')][. eq 'EUR']])
>>
>> The normalize-unicode() function converts an attribute name into a
>> standard, canonical form.
>>
>> Lesson Learned:
>>
>> When processing markup with diacritical marks, beware that two characters
>> may visually appear the same but inside the computer they are represented
>> very differently. Design XPath expressions accordingly -- use
>> normalize-unicode() to convert markup into canonical form.
> The truism "validate at trust boundaries" comes to mind: if you can't
> trust the encoding or normalization form of the XML that you receive, then
> normalise it as soon as you receive it so all of your XML is consistent
> and you don't have to make your XPaths unreadable.
>
> Your example is much like the example in Section 3.1.1, "Why do we need
> character normalization?" [1] of "Character Model for the World Wide Web
> 1.0: Normalization".  That document discusses the advantages of early or
> late normalization as well as more aspects of normalization that most of
> us could think of on our own.  Unfortunately its recommendations are in
> flux (and have been since May last year), but your scenario would best be
> handled by 'late normalization' where you normalize the data after it's
> transmitted to you.
>
> Regards,
>
>
> Tony Graham                                   tgraham@mentea.net
> Consultant                                 http://www.mentea.net
> Mentea       13 Kelly's Bay Beach, Skerries, Co. Dublin, Ireland
>   --  --  --  --  --  --  --  --  --  --  --  --  --  --  --  --
>      XML, XSL-FO and XSLT consulting, training and programming
>
> [1] http://www.w3.org/TR/charmod-norm/#sec-WhyNormalization
>
> _______________________________________________________________________
>
> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
> to support XML implementation and development. To minimize
> spam in the archives, you must subscribe before posting.
>
> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
> subscribe: xml-dev-subscribe@lists.xml.org
> List archive: http://lists.xml.org/archives/xml-dev/
> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>
>
> _______________________________________________________________________
>
> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
> to support XML implementation and development. To minimize
> spam in the archives, you must subscribe before posting.
>
> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
> subscribe: xml-dev-subscribe@lists.xml.org
> List archive: http://lists.xml.org/archives/xml-dev/
> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>
>