There is a serious amount of character encoding conversionsoccurring ins

["Costello, Roger L." <costello@mitre.org>]

Hi Folks,

All of the characters in this XML file are encoded in iso-8859-1:

<?xml version="1.0" encoding="iso-8859-1"?>
<Name>López</Name>

The following problem occurred to me: 

    Suppose I search that XML document for the string López, 
    where the characters in López are encoded in UTF-8. Will the 
    search application find a match?

I did the search and it found a match.

Amazing!

How did it find a match?

The underlying byte sequence for the iso-8859-1 López is: 4C F3 70 65 7A (one byte -- F3 -- is used to encode ó).

The underlying byte sequence for the UTF-8 López is: 4C C3 B3 70 65 7A (two bytes -- C3 B3 -- are used to encode ó).

The search application cannot be doing a byte-for-byte match, else it would find no match.

The codepoint for the UTF-8 ó character is F3. 

Hey, iso-8859-1 uses F3 to encode ó.

So perhaps the search application is converting the UTF-8 bytes to codepoints and then comparing those codepoints to the iso-8859-1 bytes. That would result in a match.

I am told that each search application may do things differently.

In any case, there's a whole lot of encoding conversions occurring ... transparently ... without our knowing.

A second problem occurred to me: Suppose I copy the iso-8859-1 López, and paste it into Google. Will Google only return those web pages that contain iso-8859-1-encoded López strings? 

Wow!

That is a mind-boggling thought.

If Google only returns iso-8859-1-encoded López strings then that would force users to perform searches using many different encodings -- search for  iso-8859-1-encoded López and then search for UTF-8-encoded López, and so forth.

That would be a nightmare!

Clearly we don't do that. Google must be converting all search strings and all Web pages to ... what? ... Unicode codepoints? 

A third problem occurred to me: Suppose I write an XSLT program and encode all its characters in iso-8859-1:

--------------------------------------------------------
<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"; version="1.0">

    <xsl:output method="text"/>

    <xsl:template match="/">
        <xsl:text>López</xsl:text>
    </xsl:template>

</xsl:stylesheet>
--------------------------------------------------------

If I apply that XSLT program to an XML document, also encoded in iso-8859-1, what will be the encoding of the result? 

I did it and the XSLT processor output the result in UTF-8. I wonder why?

This character encoding stuff is fascinating.

Some mighty smart fellows figured this character encoding stuff out long ago and now it is buried so deep in the fabric of our computers and the Web that we are completely oblivious to all the encoding conversions that are happening.

/Roger