Re: [xml-dev] Whitespace handling when compressing XML with WInZIP

[Mike_Leditschke@nemmco.com.au]

Thanks for replying. The file I have is very regular in
terms of whitespace and repeating patterns.

I guess its not the absolute number
that has me puzzled. Its the effect of taking an XML
document and replacing every occurrence of four
spaces with one tab, then compressing the two.

The compressed size should be the same in each case,
since the tokenised sequences (except that
where I had spaces, I now have the equivalent
number of tabs), and the numbers of
occurrences of them,  should be the same .  Surely the
compressed size shouldn't change that much, except in
regards to the storage of the values corresponding to the tokens?

Is there anything available to intelligently dump the contents
of a gzip file in terms of its structure and what tokens it has
identified?

Regards
Michael


                                                                                                                                         
                    Eric Bohlman                                                                                                         
                    <ebohlman@eart        To:     xml-dev@lists.xml.org                                                                  
                    hlink.net>            cc:                                                                                            
                                          Subject:     Re: [xml-dev] Whitespace handling when compressing XML with WInZIP                
                    30/09/2001                                                                                                           
                    03:13 AM                                                                                                             
                                                                                                                                         
                                                                                                                                         




9/28/01 10:15:39 PM, Mike_Leditschke@nemmco.com.au wrote:

>I have an XML file with most of the data at a tag nesting level of 5-7,
>and am compressing it with WINZip 8.
>
>While not touching the markup, I've found the whitespace used
>to "step out" the tags has a big impact on the compression ratio
>achievable.
>
>
>Using 4 chars per nesting level, the file is 6.7 MB and compresses
>to 154Kb.
>
>Using  1 tab per nesting level, the file is 4.2 MB and compresses
>to 98 Kb.
>
>While the change in the input file is what I'd anticipated, I would have
>expected the zip file to be roughly the same size. Wouldn't
>repeated runs of the same character  be replaced with a single token,
>whether they be spaces or tabs?
>
>Am I missing something obvious?  Does anyone understand the
>internals of WINZIP enough to explain the descrepancy? The ratio
>of the compressed files suggests its encoding the multiple spaces
>with multiple tokens rather than a single token.

If your data contains repeated occurrences of, say

</end>
<start>

all but the first such occurrence will be replaced with a single token.
But if it contains

</end>
<start>

  </end>
  <start>

    </end>
  <start>

and the like, it will need a different token for each of them and things
will start to pile up.
Remember that the compression algorithm doesn't just squash repeated runs
of single characters; it
tries to write "shorthand" for entire "phrases" that repeat in the
document.  The longer the
repeated "phrases," the better the compression.  Lots of "phrases" that are
almost identical
except in the middle don't compress as well, and that's what varying levels
of whitespace give
you.



-----------------------------------------------------------------
The xml-dev list is sponsored by XML.org <http://www.xml.org>, an
initiative of OASIS <http://www.oasis-open.org>

The list archives are at http://lists.xml.org/archives/xml-dev/

To subscribe or unsubscribe from this elist use the subscription
manager: <http://lists.xml.org/ob/adm.pl>






------------------------------------------
This e-mail is confidential.  If you are not the intended recipient, any use, disclosure or copying of this document is unauthorised and prohibited.  If you have received this document in error, please delete the email and notify me by return email or by phoning the NEMMCO Helpdesk on 1300 300 295.