incremental XML outputting and handling of huge DOM trees

[Jens Lauer <jens.lauer@mail.nicsys.de>]

Hi,

as a newbee in this group I try to place a posting regarding a 
programming problem we are currently in.
These probleme I already tried to explain in the XMC4C forum but got no 
answer until now - so I try to post it to a much larger auditorium...

We have made an application, which collects data frome some HW and 
transformes it via XML from a lower up to a higher layer whithin the 
software where it will be displayed to the user.

Unfortunately we have done some design errors:

1) we took the msxml3.dll and so
a) we have to register it after SW install at a new machine
b) for doing this we need admin-rights
c) we cannot debug in the XML-part because it is a COM-object
d) by some reasons we cannot resolve the SW is faily instable under W98
c) it is MS

2) we collect - and this is the main problem - all incoming data from 
startup of the program in a DOM tree in memory. After the user has given 
the save command we finalize the data and print it into an XML file.
The huge und unrestricted DOM tree is a nasty bug. The finalyzing und 
printing may easily takes some minutes  and collecting data in RAM (DOM) 
canot be the final whisdom...
We cannot log the DOM data with every newly created node because 
printing requires finalyzing and adds the XML header every time.

My idea after finding the XML4C library was to evaluate it if it would 
be better suited for our purpose because it seems to catch all the 
disadvantages of the MSXML dll and it is REALY GREAT structured and 
READABLE.

I looked at the printing methods for finding a way to bypass the 
creation of XML headers.
I found the feature 'DOMPrintFilter' and tried to play with it it that 
sense to find the way for printing only the contents of a DOM node 
without headers, but I failed.

I tried to use the DomPrint example but even playing with the settings 
of the DOMPrintFilter for printing out only the comments ( myFilter = 
new DOMPrintFilter( DOMNodeFilter::SHOW_COMMENT ) ) don't brings success.

Now my questions:

Is there any experience with such a SW-structure like we have created ( 
collecting huge DOM-trees)?
How do this people handle printing out to XML files?
Do this people mark some nodes for printing, print them (maybe initiated 
by another thread) and erase them, help keeping the DOM size restricted?
Is it possible to suppress the printing of XML headers every time the 
DOMwriter writes (using some standard ways without patching the software)?
What library/parser/dll  are recommendable if (like we have to) Wintel 
architectures have to be used?

much thanks in advance

--jens