Re: [xml-dev] Large xml processing into SQL Server

[Christopher Milton <cmiltonperl@yahoo.com>]

What libraries do you use to process the XML? JDOM?
Do you make use of DTDs or XML Schema Documents? XSLT?
How large is the XML file in bytes or megabytes?

--- william.j.boxall@accenture.com wrote:
> Hello, all.. I just joined this list and have a question that may have been
> answered by someone before, but here it is.
>  
> At work, we're working on a BEA Weblogic project (Java), and part of it is I
> wrote an interface that takes an xml file, processes it with a validation
> routine, and then stores it in a database.  I have found, though, that when
> they send me a large xml file, I can't handle it.  I get a transaction
> timeout, which is set by Weblogic, but that can be increased.  But then I run
> into out of memory issues.  I've done a lot of research, and the best
> solution appears to be to figure out a way to bulk load the xml document into
> SQL Server and then do my data manipulation.  For some reason, even when I
> trim down the operations so that all I'm doing is a straight insert into the
> database, it still takes way too long and times out.
>  
> I am not an xml guru at all, and am getting a bit of an education in the
> process of trying to find a workable solution.  Is there a bulk loader I can
> use?  If so, where and how do I use it?  Should I use an embedded xml
> database for processing?  If so, are there detailed instructions, and will
> that alleviate in any way storing the xml records into SQL Server?
>  
> I cannot load the entire xml file into memory, it's too big.  Is there some
> sort of streaming operation I can take advantage of?  
>  
> One solution I was messing with was to take the schema and map it to the
> database fields through using sql:relation, etc.. But the problem there is
> the schema is built from the xml file, which is hierarchical in nature.  I
> need to flatten out the elements in the hierarchy in order to do the mapping
> to database fields.  I've searched all over the net for a way to do this, and
> have found some cryptic solutions using xslt, which I am totally unfamiliar
> with.  Again, any help with that would be gratefully appreciated.
>  
> Thank you in advance!
> Bill
> 
> 
> This message is for the designated recipient only and may contain privileged,
> proprietary, or otherwise private information.  If you have received it in
> error, please notify the sender immediately and delete the original.  Any
> other use of the email by you is prohibited.
> 
> -----------------------------------------------------------------
> The xml-dev list is sponsored by XML.org <http://www.xml.org>, an
> initiative of OASIS <http://www.oasis-open.org>
> 
> The list archives are at http://lists.xml.org/archives/xml-dev/
> 
> To subscribe or unsubscribe from this list use the subscription
> manager: <http://www.oasis-open.org/mlmanage/index.php>
> 
>