Re: [xml-dev] [Summary] Why is Encoding Metadata (e.g. encoding="UTF-8")

[richard@inf.ed.ac.uk (Richard Tobin)]

In article <DB39101C50EE1A43908F7A57377B437E01D68D24@IMCSRV5.MITRE.ORG> you write:
>So we know that the first character in an xml document must be <.
>Which has the ASCII value of 60.
>
>So a parser will keep reading in bytes until it gets up to 60.  
>
>ASCII is 00111100 
>UCS-2 is 00000000 00111100
>
>So with ASCII (or UTF-8), we encounter 60 which is in the first byte.
>After that characters will be considered to be one-byte long until we
>read in the correct encoding attribute.
>
>With UCS-2, read up to 60, see that it took two bytes, and now all
>characters are two-bytes long.

No, you can't do this.  For example, with little-endian UTF-16 the first
byte will be 60, but the second is zero and characters are two bytes wide.

A natural approach is to read the first four bytes.  Look and see if
they match any BOM (even for UTF-8, where byte order doesn't apply,
the BOM can be used as an encoding indicator).  Otherwise see if the
first 2 or 4 bytes are the values for <? in ASCII or UTF-16 in either
byte order.  If you're feeling enthusiastic, try EBCDIC too.

If it's none of these, the document presumablhy doesn't have an
XML declaration, so assume UTF-8.  (If there's no XML declaration,
the file can start with white space, an element, a PI, or a DOCTYPE
statement, so you might as well just start parsing.)

In most cases, what you decide based on the first few bytes may be
modified by the encoding declaration.  If it looked like ASCII, it
might be ASCII, UTF-8, Latin-N, or some other ASCII superset like
shift-JIS or a Microsoft code page.  If it looked like EBCDIC, there
are several code pages.

-- Richard
-- 
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.