OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help



   RE: [xml-dev] Xqueeze: Compact XML Alternative

[ Lists Home | Date Index | Thread Index ]

Note the copied email from Don Brutzman.  There 
are application solutions, in this case, one for the X3D 
language.   Also note as has been noted on 
this list before, the encumbered MPEG solution 
which relies on the Schema as well.


From: Mike Champion [mailto:mc@xegesis.org]

So, I agree that nobody has shown a good solution here, but I don't agree 
that it means there there's no problem.  There may not BE a good solution 
outside a given application (another point I believe Sean has made) because 
the parameters and constraints on them (message size, markup overhead, 
whether or not the vocabulary is known in advance, whether the "SOAP 
profile" of XML is in use or whether entity expansion must be supported, 
what kind of bandwidth and processor speed can be assumed, etc.) are so 
variable. But that doesn't mean that possible solutions shouldn't be 
explored, IMHO.


NPS has been working for a few years on a binary encoding of general
XML documents/languages, indended for rapid prototyping of network protocols
in support of large-scale virtual environments.  This message summarizes
the XFSP technical approach and proposes we adapt it to support an 
extensible binary encoding for X3D.

Originally called Dynamic Behavior Protocol, our approach was to define
a packet-description language in XML.  Such a language might easily be
used to define the packet payloads used by network protocols, then
parsed by a network handler which did the necessary work of reading &
writing protocols conforming to the format.  We got this working last

An important insight was then realized by Andrzej Kapolka, Ekrem Serin
and Don McGregor of NPS.  It turns out that XML Schema already supports 
most of the data-structure information needed for a packet description 
language.  We mapped out an approach for doing so, renaming this work
the Cross-Format Schema Protocol (XFSP).  It was implemented in
November and demonstrated at the IITSEC conference in December 2002.
Ekrem performed a valuable test when he took the IEEE DIS Specification,
encoded the Entity State PDU (ESPDU) as an XML schema, and created an
XFSP reader/writer all within one day.  Such DIS implementation work 
took about a year, originally.

A brief XFSP description and detailed slideset are available at

The current version of XFSP source and Ekrem's draft thesis chapter 
describing this work is on Sourceforge.  Detailed finite state machines
on serialization/deserialization are included.  Comments are welcome.

It appears that this work is suitable for generation of a binary protocol
(and hence binary file format) for X3D.  We're proposing it as such a 
basis.  After some scrutiny, we can consider whether how to reopen the 
deferred RFP for X3D compression.

_X3D Binary Encoding based on XFSP_

- smaller X3D files
- faster loading
- streaming
- all other points in the deferred X3D Binary Requirements and RFP

Basic principles:
- XFSP reads an XML schema for an arbitrary tagset
- XFSP thereby produces a packet reader/writer which can handle
  any valid document corresponding to the XML schema
- Packets are in binary form, using tokens for elements/attributes
  and serialization of payload data
- Unambigous efficient tokenization/serialization/deserialization
- This algorithm can be tuned for X3D binary encoding.

Further/future benefits:
- Since the documents are in XML, they are well structured and suitable
  for further compression.  http://www.research.att.com/sw/tools/xmill 
  has some interesting studies on this topic.  Arbitrary documents may
  grow when converted to XML, but then compress better than the gzipped
- XML format allows simulataneous use of XML Encryption and XML Signature
  (i.e. Authentication), both in W3C Recommendation (i.e. approved) status.
  http://www.w3.org/Encryption   http://www.w3.org/Signature
- XML Encryption/Signature provide default royalty-free (RF) algorithms
  but also allow specification of alternates.  This technique might again
  be applied in an X3D binary encoding to allow both open and extensible
  algorithms for geometry compression, quantization, encryption etc.
- XML format allows integration of other metadata, such as Resource 
  Description Framework (RDF) which is of likely interest to CAD3D group
  and others.  http://www.w3.org/RDF
- Properly adapted, the binary encoding appears to be suitable both as a
  file format and also for network delivery (e.g. scene streaming, 
  progressive rendering, event delivery, etc.)
- Major performance speedups are expected over gzip during binary scene
  loading, since the binary structures are directly ready for immediate 
  insertion into scene-graph data structures in memory.  No further
  character-based parsing is needed for numeric fields.
- Fixing the token set values assigned to X3D nodes and fields permits 
  immediate streamed deserialization upon commencing loading/receipt.
- Suitable for implementation in any network- or file-capable programming
  language.  Example XFSP implementation offered is in Java.  Even better
  performance might be achieved through a tuned X3D-binary implementation,
  rather than an XFSP implementation capable of general XML.
- Tuned X3D-XML implementations might be adapted to handle VRML encoding
  directly.  Alternatively, 1-1 translation to X3D's XML encoding
  allows corresponding usage by VRML encoding also.
- Offered by NPS under royaltee-free terms in accordance with the Web3D
  Consortium Intellectual Property Rights (IPR) Policy

Design approach:
- XML elements are represented both by open-tag and close-tag tokens.
  This allows rapid regeneration of the original tree-like document
- XML attributes represented by single tokens, in same namespace as 
- Payload data for attributes and element content is type dependent,
  occupies predictable lengths between tokens.  MF array types are
  preceded by field-length integers.
- Payload compression is type specific, e.g. long integers, floats,
  doubles, etc. 
- Further typing may be applied for special geometry-related types 
  like Color, Normal, coordinates, indices, pixelTexture, enumerations,
  UUIDs (e.g. for CAD3D), etc.  This would support specialty compression
  components on a geometric-type by geometric-type basis.
- Quantization tables will likely be allowed.
- Separate namespaces for prototype/field names and DEF/IDs.
- Use of XML validation and Canonical XML form can ensure that
  scenes are compressed consistently, e.g. independent of arbitrary
  whitespace, apostrophe/quotation mark attribute delimitations, etc.

- Need to investigate whether to support just field payloads, or 
  also stream nodes.
- Multiple modes may be necessary for uncorrupted updates of field data:  
  (a) direct replacement
  (b) progressive, appending to arrays 
      (e.g. pass continuous motion-capture data)
  (c) buffer until delivered, then swap
- Event delivery likely contains regular binary segment consisting of
  node ID, field token and payload data, along with (optionally honored)

So, there is a lot here but it appears that a compatible composition of
all known X3D binary encoding goals might be achieved.  

Thanks for discussion and review by Alan Hudson and Justin Couch of 
Yumetech.  http://www.yumetech.com  http://www.xj3d.org
Thanks too for long-running stellar efforts by Don, Andrzej and Ekrem
of the NPS MOVES Institute.   http://www.movesinstitute.org

Comments welcome.  Primary discussion forum is x3d-contributors@web3D.org


News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 2001 XML.org. This site is hosted by OASIS