Re: UTF-8 BOM

[David Brownell <david-b@pacbell.net>]

I hooked it up to the XML conformance test DTD using
the attached "bomtests.xml" and ran it through its paces
that way.  (Easy way to handle any parser that can use
those 2000+ tests meaningfully ...)

Note that the enclosed file is written as if each of those test
cases is well-formed, though I don't think they are.  (While
UTF-16 specifies a BOM as part of the encoding, to be
removed, UTF-8 doesn't -- it's part of the encoded data.)

Results:

- Both AElfred2 (current) and Crimson 1.1 rejected "doc",
  "docx", "entx", "ent2x"; and accepted "ent", "ent2".

- Xerces 1.4 rejected only "ent2x", and accepted the
  others.

This issue should perhaps become part of "XML Blueberry"
if it leads to a spec revision, I'd suggest ... as with NEL on
mainframes, accepting it would suggest that the idea is that
the standards should dance to the tune of certain vendors,
rather than vendors dancing to the tune of standards!

- Dave


----- Original Message ----- 
From: "Richard Tobin" <richard@cogsci.ed.ac.uk>
To: <xml-dev@lists.xml.org>
Sent: Thursday, June 14, 2001 4:23 AM
Subject: UTF-8 BOM


> The W3C XML Core WG is considering the question of whether a UTF-8
> byte-order make (BOM) is allowed at the start of an XML entity.  This
> question was raised a few weeks ago in a thread on comp.text.xml
> starting at article
> 
>   <180520011620538217%andreas.prilop@altavista.net>
> 
> We would like to determine how existing parsers handle the byte
> sequence #xEF #xBB #xBF when it appears at the start of an XML
> document or other entity.  Is it treated as a BOM (and not part
> of the text of the entity) or as a zero-width non-breaking space
> character?
> 
> We have placed a number of test cases at
> 
>   http://www.cogsci.ed.ac.uk/~richard/bomtest/
> 
> and would be grateful for feedback on how parsers handle them.  Please
> post results here in xml-dev to avoid unnecessary duplication.
> 
> We would also like to know of any editors (or similar tools) that
> generate XML documents starting with a UTF-8 BOM.
> 
> -- Richard (on behalf of the XML Core WG)
> 
> ------------------------------------------------------------------
> The xml-dev list is sponsored by XML.org, an initiative of OASIS
> <http://www.oasis-open.org>
> 
> The list archives are at http://lists.xml.org/archives/xml-dev/
> 
> To unsubscribe from this elist send a message with the single word
> "unsubscribe" in the body to: xml-dev-request@lists.xml.org

<TESTCASES PROFILE="W3C BOM tests (via Richard Tobin)">

    <TEST ID="bom-doc" URI="doc.xml" TYPE="invalid"
    		SECTIONS="tbd">
	XML document with possible UTF-8 BOM, no XML declaration
	</TEST>

    <TEST ID="bom-docx" URI="docx.xml" TYPE="invalid"
    		SECTIONS="tbd">
	XML document with possible UTF-8 BOM, followed by XML declaration
	</TEST>

    <TEST ID="bom-ent" URI="ent.xml" TYPE="invalid" ENTITIES="general"
	    SECTIONS="tbd">
	XML document referring to external entity 
	with possible UTF-8 BOM, no XML declaration
	</TEST>

    <TEST ID="bom-entx" URI="entx.xml" TYPE="invalid" ENTITIES="general"
	    SECTIONS="tbd">
	XML document referring to external entity 
	with possible UTF-8 BOM, followed by XML declaration
	</TEST>

    <TEST ID="bom-ent2" URI="ent2.xml" TYPE="invalid" ENTITIES="general"
	    SECTIONS="tbd">
	XML document referring to external entity
	with possible UTF-8 BOM followed by ZWNBSP, 
	no XML declaration
	</TEST>

    <TEST ID="bom-ent2x" URI="ent2x.xml" TYPE="invalid" ENTITIES="general"
	    SECTIONS="tbd">
	XML document referring to external entity
	with possible UTF-8 BOM followed by ZWNBSP, 
	followed by misplaced XML declaration
	</TEST>

</TESTCASES>