Re: [xml-dev] Whitespace handling when compressing XML with WInZIP

[Eric Bohlman <ebohlman@earthlink.net>]

9/28/01 10:15:39 PM, Mike_Leditschke@nemmco.com.au wrote:

>I have an XML file with most of the data at a tag nesting level of 5-7,
>and am compressing it with WINZip 8.
>
>While not touching the markup, I've found the whitespace used
>to "step out" the tags has a big impact on the compression ratio
>achievable.
>
>
>Using 4 chars per nesting level, the file is 6.7 MB and compresses
>to 154Kb.
>
>Using  1 tab per nesting level, the file is 4.2 MB and compresses
>to 98 Kb.
>
>While the change in the input file is what I'd anticipated, I would have
>expected the zip file to be roughly the same size. Wouldn't
>repeated runs of the same character  be replaced with a single token,
>whether they be spaces or tabs?
>
>Am I missing something obvious?  Does anyone understand the
>internals of WINZIP enough to explain the descrepancy? The ratio
>of the compressed files suggests its encoding the multiple spaces
>with multiple tokens rather than a single token.

If your data contains repeated occurrences of, say

</end>
<start>

all but the first such occurrence will be replaced with a single token.  But if it contains

</end>
<start>

  </end>
  <start>

    </end>
  <start>

and the like, it will need a different token for each of them and things will start to pile up.  
Remember that the compression algorithm doesn't just squash repeated runs of single characters; it 
tries to write "shorthand" for entire "phrases" that repeat in the document.  The longer the 
repeated "phrases," the better the compression.  Lots of "phrases" that are almost identical 
except in the middle don't compress as well, and that's what varying levels of whitespace give 
you.