[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [xml-dev] Whitespace handling when compressing XML with WInZIP
- From: Mike_Leditschke@nemmco.com.au
- To: Eric Bohlman <firstname.lastname@example.org>
- Date: Sun, 30 Sep 2001 20:58:09 +1000
Thanks for replying. The file I have is very regular in
terms of whitespace and repeating patterns.
I guess its not the absolute number
that has me puzzled. Its the effect of taking an XML
document and replacing every occurrence of four
spaces with one tab, then compressing the two.
The compressed size should be the same in each case,
since the tokenised sequences (except that
where I had spaces, I now have the equivalent
number of tabs), and the numbers of
occurrences of them, should be the same . Surely the
compressed size shouldn't change that much, except in
regards to the storage of the values corresponding to the tokens?
Is there anything available to intelligently dump the contents
of a gzip file in terms of its structure and what tokens it has
<ebohlman@eart To: email@example.com
Subject: Re: [xml-dev] Whitespace handling when compressing XML with WInZIP
9/28/01 10:15:39 PM, Mike_Leditschke@nemmco.com.au wrote:
>I have an XML file with most of the data at a tag nesting level of 5-7,
>and am compressing it with WINZip 8.
>While not touching the markup, I've found the whitespace used
>to "step out" the tags has a big impact on the compression ratio
>Using 4 chars per nesting level, the file is 6.7 MB and compresses
>Using 1 tab per nesting level, the file is 4.2 MB and compresses
>to 98 Kb.
>While the change in the input file is what I'd anticipated, I would have
>expected the zip file to be roughly the same size. Wouldn't
>repeated runs of the same character be replaced with a single token,
>whether they be spaces or tabs?
>Am I missing something obvious? Does anyone understand the
>internals of WINZIP enough to explain the descrepancy? The ratio
>of the compressed files suggests its encoding the multiple spaces
>with multiple tokens rather than a single token.
If your data contains repeated occurrences of, say
all but the first such occurrence will be replaced with a single token.
But if it contains
and the like, it will need a different token for each of them and things
will start to pile up.
Remember that the compression algorithm doesn't just squash repeated runs
of single characters; it
tries to write "shorthand" for entire "phrases" that repeat in the
document. The longer the
repeated "phrases," the better the compression. Lots of "phrases" that are
except in the middle don't compress as well, and that's what varying levels
of whitespace give
The xml-dev list is sponsored by XML.org <http://www.xml.org>, an
initiative of OASIS <http://www.oasis-open.org>
The list archives are at http://lists.xml.org/archives/xml-dev/
To subscribe or unsubscribe from this elist use the subscription
This e-mail is confidential. If you are not the intended recipient, any use, disclosure or copying of this document is unauthorised and prohibited. If you have received this document in error, please delete the email and notify me by return email or by phoning the NEMMCO Helpdesk on 1300 300 295.