RE: Principle #1 in validating XML documents

[David Lee <dlee@calldei.com>]

IMHO, depends on what you want to do , no single answer.
If I am exchanging XML with a partner I would 

1) Validate 
--> If any errors reject the document and report back an error

If I am trying to slurp in as many docs as possible and have a high tollerence for mistakes then
I would think about deleting invalid unicode chars ... but this would be RARE ...
It depends on your needs.  An example I do this  is twitter feeds.
I hit about .1% of twitter posts contain non-XML valid unicode chars.  I cant do a thing about this 
(I cant go back and ask the person to retweet it right :)  So I either drop the document or delete or replace the bad char.

But in general if there is an invalid unicode char, then to me that means the XML was invalid.
I am not a great fan of fixing up XML ... every kind of fixup is a *guess* as to what was meant.
I dont like to guess about data ...  Hmm was that a balance transfer of $100000?0000 ?  I will just guess.
But sometimes you need to guess to get anything ... 



----------------------------------------
David A. Lee
dlee@calldei.com
http://www.xmlsh.org

-----Original Message-----
From: Costello, Roger L. [mailto:costello@mitre.org] 
Sent: Sunday, March 10, 2013 8:52 AM
To: xml-dev@lists.xml.org
Subject: [xml-dev] Principle #1 in validating XML documents

Hi Folks,

Suppose you deploy a web service and it operates like this:

1. Receive inbound XML.

2. Validate XML.

3. Application processes XML.

Suppose the application performs several conversions on the XML before processing it:

a. Unicode normalization: converts the XML to Unicode Normalization Form C (NFC). This is the standard form for data exchanges on the web.

b. Noncharacter code points: deletes all noncharacter code points from the XML. See [1] for a description of noncharacter code points.

c. Other conversions ...

Here is principle #1 in validating XML documents:

	Validate AFTER conversions.

Normalize the XML and delete noncharacter code points (and any other conversions), and then validate.

Why? Because attacks that are resident in the XML are exposed after conversions. Validating before doing the conversions may not reveal the attacks. (See [2] for a description of various attacks.)

Revise the web service:

1. Receive inbound XML.

2. Perform all conversions on the XML.

2. Validate the converted XML.

3. Application processes the converted XML.

Comments?

/Roger

[1] http://www.unicode.org/reports/tr36/#Deletion_of_Noncharacters

[2] http://www.unicode.org/reports/tr36/ 

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php