Re: [xml-dev] There is a serious amount of character encoding conversion

[Michael Kay <mike@saxonica.com>]

Well, I do remember that in the Computer Science Tripos at Cambridge 
around 1972 there was a open-ended question "Why is character coding so 
complicated?", and it hasn't become any simpler in the intervening 40 years.

But yes, of course XML parsers decode characters and applications see 
them and compare them in their decoded form. And most of the time it all 
happens seamlessly, and we don't think about it until either something 
goes wrong, or Roger decides to tell us how wonderful it all is.

Michael Kay
Saxonica

On 28/12/2012 14:01, Costello, Roger L. wrote:
> Hi Folks,
>
> All of the characters in this XML file are encoded in iso-8859-1:
>
> <?xml version="1.0" encoding="iso-8859-1"?>
> <Name>López</Name>
>
> The following problem occurred to me:
>
>      Suppose I search that XML document for the string López,
>      where the characters in López are encoded in UTF-8. Will the
>      search application find a match?
>
> I did the search and it found a match.
>
> Amazing!
>
> How did it find a match?
>
> The underlying byte sequence for the iso-8859-1 López is: 4C F3 70 65 7A (one byte -- F3 -- is used to encode ó).
>
> The underlying byte sequence for the UTF-8 López is: 4C C3 B3 70 65 7A (two bytes -- C3 B3 -- are used to encode ó).
>
> The search application cannot be doing a byte-for-byte match, else it would find no match.
>
> The codepoint for the UTF-8 ó character is F3.
>
> Hey, iso-8859-1 uses F3 to encode ó.
>
> So perhaps the search application is converting the UTF-8 bytes to codepoints and then comparing those codepoints to the iso-8859-1 bytes. That would result in a match.
>
> I am told that each search application may do things differently.
>
> In any case, there's a whole lot of encoding conversions occurring ... transparently ... without our knowing.
>
> A second problem occurred to me: Suppose I copy the iso-8859-1 López, and paste it into Google. Will Google only return those web pages that contain iso-8859-1-encoded López strings?
>
> Wow!
>
> That is a mind-boggling thought.
>
> If Google only returns iso-8859-1-encoded López strings then that would force users to perform searches using many different encodings -- search for  iso-8859-1-encoded López and then search for UTF-8-encoded López, and so forth.
>
> That would be a nightmare!
>
> Clearly we don't do that. Google must be converting all search strings and all Web pages to ... what? ... Unicode codepoints?
>
> A third problem occurred to me: Suppose I write an XSLT program and encode all its characters in iso-8859-1:
>
> --------------------------------------------------------
> <?xml version="1.0" encoding="iso-8859-1"?>
> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"; version="1.0">
>
>      <xsl:output method="text"/>
>
>      <xsl:template match="/">
>          <xsl:text>López</xsl:text>
>      </xsl:template>
>
> </xsl:stylesheet>
> --------------------------------------------------------
>
> If I apply that XSLT program to an XML document, also encoded in iso-8859-1, what will be the encoding of the result?
>
> I did it and the XSLT processor output the result in UTF-8. I wonder why?
>
> This character encoding stuff is fascinating.
>
> Some mighty smart fellows figured this character encoding stuff out long ago and now it is buried so deep in the fabric of our computers and the Web that we are completely oblivious to all the encoding conversions that are happening.
>
> /Roger
>
>
> _______________________________________________________________________
>
> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
> to support XML implementation and development. To minimize
> spam in the archives, you must subscribe before posting.
>
> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
> subscribe: xml-dev-subscribe@lists.xml.org
> List archive: http://lists.xml.org/archives/xml-dev/
> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>
>