Apples with apples (was Re: "Binary XML" proposals)

[Rick Jelliffe <ricko@allette.com.au>]

From: Al Snell <alaric@alaric-snell.com>

>One of the things being considered over in xml-bin now is formats with
>indices on, so a DOM implementation can pull stuff from the file as needed
>and no sooner.

To compare apples with apples, I suppose that a comparison of a non-text
tree format including indexes with XML would be more meaningful if the XML
had
indexes too.

For example, one index system for XML could be a PI immediately at the start
of the document (after any XML header) such as
<?index x1 633 x2 8444?>
where the tokens are pairs of ID and character counts into the normalized
form of the document. I note that for space, the first token would occupy
the same as a binary format and, if the binary-format's index used 4-byte
offsets then there is no space saving for indexes into the first 999
characters (i.e. 3 bytes UTF-8 + 1 space).

And if the binary format was sent over email which converted it to some
bin64 encoding, there is no improvement in space taken until indexing past
the first 99,999 characters (or is it more?), which is, for XML document,
effectively never.  (Of course, it would be better to allow either straight
offsets and deltas if size was really so important, but a binary format and
a PI would both be advantaged by that.)

Of course, there is the overhead of a streaming canonicalizer (and any
uncompresser), but if the document is already sent in a moderately canonical
format, I don't know how much overhead that would actually be, given that
detection that things are OK has a smaller performance cost than fixing
things.

Cheers
Rick Jelliffe