Re: [xml-dev] Here's how to process XML documents written in German

[Olivier JEULIN <olivier.jeulin@gmail.com>]

Like Roger, this is a news to me; I thought that elements and
attributes names were normalized (and frankly what I read here
highlights the fact that XML was made by experts for experts —no
offense).

So, just to be sure that I understand:

http://www.w3.org/TR/REC-xml/#sec-suggested-names (J Suggestions for
XML Names (Non-Normative))
suggests:

1. The first character of any name should have a Unicode property of
ID_Start, or else be '_' #x5F.

2. Characters other than the first should have a Unicode property of
ID_Continue, or be one of the characters listed in the table entitled
"Characters for Natural Language Identifiers" in UAX #31, with the
exception of "'" #x27 and "’" #x2019.

3. Characters in names should be expressed using Normalization Form C
as defined in [UnicodeNormal].

So according to 3., one should use the NFC for characters.

In UAX #31 (http://www.unicode.org/reports/tr31/#Combining_Marks):

Combining marks are accounted for in identifier syntax: a composed
character sequence consisting of a base character followed by any
number of combining marks is valid in an identifier. Combining marks
are required in the representation of many languages, and the
conformance rules in Chapter 3, Conformance, of [Unicode] require the
interpretation of canonical-equivalent character sequences. The
simplest way to do this is to require identifiers in the NFC format
(or transform them into that format); see Section 5, Normalization and
Case.

If “The simplest way to do this is to require identifiers in the NFC
format (or transform them into that format)”, how to do that?
M. Kay says "use a schema". Well, sometimes we don't (and won't) use one.
So the correct way to be sure that the input XML is normalized (and,
*yikes*, the XSLT) would be to use M. Kay's XSLT:
http://www.oxygenxml.com/archives/xsl-list/201104/msg00265.html
or uconv -f utf8 -t utf8 -x nfc

Is that correct?

2013/1/31 Michael Kay <mike@saxonica.com>:
> Tony prefaced his advice with "if you can't trust...". You (Roger) left that
> bit out.
>
> Perhaps you did this on the basis that you should never trust anything. But
> if you don't trust anything, why are you processing the XML at all?
>
> Michael Kay
> Saxonica
>
>
> On 30/01/2013 21:54, Costello, Roger L. wrote:
>>
>> Tony Graham wrote:
>>
>>> if you can't trust the encoding or normalization
>>> form of the XML that you receive, then normalise
>>> it as soon as you receive it so all of your XML is
>>> consistent
>>
>> Is that a Best Practice? That is:
>>
>>      Best Practice: before processing any XML
>>      document, apply an identity transformation
>>      to it which converts the entire XML document
>>      into a Unicode normalized form. For example,
>>      each combining character sequence is converted
>>      into a precomposed character.
>>
>> Thoughts?
>>
>> /Roger
>>
>> -----Original Message-----
>> From: Tony Graham [mailto:tgraham@mentea.net]
>> Sent: Wednesday, January 30, 2013 4:31 PM
>> To: xml-dev@lists.xml.org
>> Subject: Re: [xml-dev] Here's how to process XML documents written in
>> German
>>
>> On Wed, January 30, 2013 6:47 pm, Costello, Roger L. wrote:
>> ...
>>>
>>> This XPath expression does the job:
>>>
>>> sum(//Posten[@*[normalize-unicode(name(.)) eq
>>> normalize-unicode('währung')][. eq 'EUR']])
>>>
>>>
>>> The normalize-unicode() function converts an attribute name into a
>>> standard, canonical form.
>>>
>>> Lesson Learned:
>>>
>>> When processing markup with diacritical marks, beware that two characters
>>> may visually appear the same but inside the computer they are represented
>>> very differently. Design XPath expressions accordingly -- use
>>> normalize-unicode() to convert markup into canonical form.
>>
>> The truism "validate at trust boundaries" comes to mind: if you can't
>> trust the encoding or normalization form of the XML that you receive, then
>> normalise it as soon as you receive it so all of your XML is consistent
>> and you don't have to make your XPaths unreadable.
>>
>> Your example is much like the example in Section 3.1.1, "Why do we need
>> character normalization?" [1] of "Character Model for the World Wide Web
>> 1.0: Normalization".  That document discusses the advantages of early or
>> late normalization as well as more aspects of normalization that most of
>> us could think of on our own.  Unfortunately its recommendations are in
>> flux (and have been since May last year), but your scenario would best be
>> handled by 'late normalization' where you normalize the data after it's
>> transmitted to you.
>>
>> Regards,
>>
>>
>> Tony Graham                                   tgraham@mentea.net
>> Consultant                                 http://www.mentea.net
>> Mentea       13 Kelly's Bay Beach, Skerries, Co. Dublin, Ireland
>>   --  --  --  --  --  --  --  --  --  --  --  --  --  --  --  --
>>      XML, XSL-FO and XSLT consulting, training and programming
>>
>> [1] http://www.w3.org/TR/charmod-norm/#sec-WhyNormalization
>>
>> _______________________________________________________________________
>>
>> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
>> to support XML implementation and development. To minimize
>> spam in the archives, you must subscribe before posting.
>>
>> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
>> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
>> subscribe: xml-dev-subscribe@lists.xml.org
>> List archive: http://lists.xml.org/archives/xml-dev/
>> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>>
>>
>> _______________________________________________________________________
>>
>> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
>> to support XML implementation and development. To minimize
>> spam in the archives, you must subscribe before posting.
>>
>> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
>> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
>> subscribe: xml-dev-subscribe@lists.xml.org
>> List archive: http://lists.xml.org/archives/xml-dev/
>> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>>
>>
>
>
> _______________________________________________________________________
>
> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
> to support XML implementation and development. To minimize
> spam in the archives, you must subscribe before posting.
>
> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
> subscribe: xml-dev-subscribe@lists.xml.org
> List archive: http://lists.xml.org/archives/xml-dev/
> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>



-- 

O. J.