Re: [xml-dev] newline/form feed valid as attribute value?

[Michael Kay <mike@saxonica.com>]

On 02/07/2012 21:43, Dan Shelton wrote:
> On 2 July 2012 22:17, Michael Kay<mike@saxonica.com>  wrote:
>> It's theoretically impossible to write an XML parser using regular
>> expressions alone, because XML is not a regular language.
> So what's wrong with the following regex pattern?

This regex pattern is typical of the kind of thing that regex 
enthusiasts produce. It works 99% of the time, and that's usually all 
they care about. It might even work 100% of the time for the flavour of 
XML that's present in the sample they test it against, or even 100% of 
the live data that it has to handle in the first five years of 
operation. But it's wrong. It's trivially easy to find examples of bad 
XML that it doesn't reject (a comment containing "--" for example), or 
good XML that it does reject (whitespace in end tags?) .

Using [:alnum:] as an approximation for the set of characters allowed in 
names, for example, is in my book just insufferably shoddy programming, 
even though it may be years before the program actually produces an 
incorrect result. (The actual meaning of [:alnum:] depends on the regex 
dialect you are using, but it almost certainly doesn't match the XML 
definition.)

There's another reason that this code is bad. If it does reject XML as 
ill-formed, it doesn't tell you why. It's perfectly conformant for an 
XML parser to reject ill-formed XML with the message "This is not XML", 
but it's very unfriendly to the user.

As David points out, my comment about theoretical limitations applies to 
the use of a regex for distinguishing well-formed XML from non-XML (you 
talked of "an XML parser using regex alone"); it doesn't apply to the 
use of a regex for tokenizing prior to parsing. Use of a regex for 
tokenizing is certainly possible, but it needs a lot more care than has 
gone into this example (in fact, XML syntax doesn't really lend itself 
well to the usual separation of tokenizing from syntax analysis, because 
the lexical rules are so context-sensitive, particularly in DTDs.)

Michael Kay
Saxonica

> It was passed around
> by Roland Mainz in David Korn's ksh93 mailing list a few weeks ago and
> is used as a *core* (there's more prep and postprocess code, but the
> parsing alone is done by repeatedly applying the regex to a character
> stream) for a xml fragment parser (brackets not postfixed with ?:
> capture data and are stored in the 2D array .sh.match):
> ---------------
> dummy="${xmltext//~(Ex-p)(?:
> 	(<!--.+-->)+?|	# xml comments
> 	(<[:_[:alnum:]-]+
> 		(?: # attributes
> 			[[:space:]]+
> 			(?: # four different types of name=value syntax
> 				(?:[:_[:alnum:]-]+=[^\"\'[:space:]]+?)|	#x='foo=bar huz=123'
> 				(?:[:_[:alnum:]-]+=\"[^\"]*?\")|		#x='foo="ba=r o" huz=123'
> 				(?:[:_[:alnum:]-]+=\'[^\']*?\')|		#x="foox huz=123"
> 				(?:[:_[:alnum:]-]+)				#x="foox huz=123"
> 			)
> 		)*
> 		[[:space:]]*
> 		\/?	# start tags which are end tags, too (like<foo\/>)
> 	>)+?|				# xml start tags
> 	(<\/[:_[:alnum:]-]+>)+?|	# xml end tags
> 	([^><]+)			# xml text
> 	)/D}"
> ---------------
>