[
Lists Home |
Date Index |
Thread Index
]
Rick Jelliffe wrote:
> Also, it would interesting to see binary people use Chinese (Japanese or
> Korean) text
> and markup for their test data. Compressing or packing ASCII is quite
> different to
> compressing or packing UTF-16 Chinese, which has a more random-seeming
> distribution
> of byte values. It is not dishonest to make the case for binary using
> data that
> is most compressible; but businesses who are looking at compression
> strategies
> for world-wide use need to factor in CJK compressability into their
> evaluations.
That only makes a difference if you're actually compressing the text
fields - most binary interchange formats will just write the text in
UTF-8 and leave it at that; however lower-level byte sequence
compressors will just see the text as bytes rather than as characters.
I've yet to see an implementation of the deflate algorithm (as used by
gzip) for UCS-4 codepoints rather than just bytes, but it could be done
and would be very interesting (but if you use a wide range of characters
in the input, your Huffman tree will be a bit memory-intensive! :-)
> Cheers
> Rick Jelliffe
ABS
|
