Re: [xml-dev] Tool converts records to XML

[Roger L Costello <costello@mitre.org>]

Michael Kay wrote:

> the "Barnes & Noble" problem. The number #1 blunder 
> when writing XML is not to bother escaping `<` and `&` 
> if they happen to occur in your input.

Ouch!

You are right Michael.

Upon reflection, I realized that there is an even nastier problem lurking than the problem of converting & and < in the input record data into &amp; and &lt; in the output XML. 

My actual record data is encoded in Windows 1252. The records contain many occurrences of the degree symbol. In Windows 1252 the degree symbol is a single byte (hex B0) but in UTF-8 the degree symbol is two bytes (hex C2 B0). My AWK program doesn't convert the one byte Windows 1252 degree symbol to the two byte UTF-8 degree symbol. In fact, to be correct every Windows 1252 character with an encoding above hex 0F must be converted to two bytes, and my AWK program doesn't do any of that.

Eeeeeeek!

For fun, I also wrote an XSLT program to convert the records to XML. My program uses the unparsed-text() function, which does all character conversions behind-the-scene (i.e., you have no idea that all the Windows 1252 characters above hex 0F are being converted to two byte UTF-8 characters).

To implement the character conversions in AWK would be a monumental task.

Eeeeeeek!

Lesson Learned: Don't use AWK to convert records to XML.

Bummer!

/Roger