What is "large" versus "huge"? Here's the answer ....

[Roger L Costello <costello@mitre.org>]

Hi Folks,

Here's the number that separates large from huge:

	8K

Allow me to explain.

An XML lexer breaks up into tokens the stream of characters comprising an XML document. For example, this stream of characters that we recognize as an XML document:

<Document>Hello, world</Document>

might be broken up into these tokens:

'<'
Document
'>'
Hello, world
'</'
Document
'>'

All of those tokens are very short. But there are some XML documents with very long tokens.

How long should your tokens be? What are large tokens? What are huge tokens?

Practically speaking, if an XML element contains a string, how long should you allow the string to be?

The answer is 8K. 

Tokens less than 8K are large. They are of acceptable length and won't incur performance penalties in the lexer. 

Tokens greater than 8K are huge. They will incur performance penalties in the lexer. 

Don't create XML documents with elements containing strings longer than 8K.

"Flex" is a tool that auto-generates lexers. Page 49 of the Flex manual [1] says this:

Another final note regarding performance: as mentioned in Chapter 7 [Matching],
page 14, dynamically resizing yytext to accommodate huge tokens is a slow process
because it presently requires that the (huge) token be rescanned from the beginning. Thus
if performance is vital, you should attempt to match "large" quantities of text but not
"huge" quantities, where the cutoff between the two is at about 8K characters per token.

There you have it, the numerical definition of "large" and "huge".

/Roger

[1] https://epaperpress.com/lexandyacc/download/flex.pdf