OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

 


 

   RE: [xml-dev] Large xml processing into SQL Server

[ Lists Home | Date Index | Thread Index ]

Last time someone told me they had a big XML file it turned out to be only
500Kb, so my advice was all wrong. What do you mean by "big"?

Michael Kay
http://www.saxonica.com/ 

> -----Original Message-----
> From: william.j.boxall@accenture.com 
> [mailto:william.j.boxall@accenture.com] 
> Sent: 07 January 2005 17:01
> To: xml-dev@lists.xml.org
> Subject: [xml-dev] Large xml processing into SQL Server
> 
> Hello, all.. I just joined this list and have a question that 
> may have been answered by someone before, but here it is.
>  
> At work, we're working on a BEA Weblogic project (Java), and 
> part of it is I wrote an interface that takes an xml file, 
> processes it with a validation routine, and then stores it in 
> a database.  I have found, though, that when they send me a 
> large xml file, I can't handle it.  I get a transaction 
> timeout, which is set by Weblogic, but that can be increased. 
>  But then I run into out of memory issues.  I've done a lot 
> of research, and the best solution appears to be to figure 
> out a way to bulk load the xml document into SQL Server and 
> then do my data manipulation.  For some reason, even when I 
> trim down the operations so that all I'm doing is a straight 
> insert into the database, it still takes way too long and times out.
>  
> I am not an xml guru at all, and am getting a bit of an 
> education in the process of trying to find a workable 
> solution.  Is there a bulk loader I can use?  If so, where 
> and how do I use it?  Should I use an embedded xml database 
> for processing?  If so, are there detailed instructions, and 
> will that alleviate in any way storing the xml records into 
> SQL Server?
>  
> I cannot load the entire xml file into memory, it's too big.  
> Is there some sort of streaming operation I can take advantage of?  
>  
> One solution I was messing with was to take the schema and 
> map it to the database fields through using sql:relation, 
> etc.. But the problem there is the schema is built from the 
> xml file, which is hierarchical in nature.  I need to flatten 
> out the elements in the hierarchy in order to do the mapping 
> to database fields.  I've searched all over the net for a way 
> to do this, and have found some cryptic solutions using xslt, 
> which I am totally unfamiliar with.  Again, any help with 
> that would be gratefully appreciated.
>  
> Thank you in advance!
> Bill
> 
> 
> This message is for the designated recipient only and may 
> contain privileged, proprietary, or otherwise private 
> information.  If you have received it in error, please notify 
> the sender immediately and delete the original.  Any other 
> use of the email by you is prohibited.
> 
> -----------------------------------------------------------------
> The xml-dev list is sponsored by XML.org <http://www.xml.org>, an
> initiative of OASIS <http://www.oasis-open.org>
> 
> The list archives are at http://lists.xml.org/archives/xml-dev/
> 
> To subscribe or unsubscribe from this list use the subscription
> manager: <http://www.oasis-open.org/mlmanage/index.php>
> 
> 





 

News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 2001 XML.org. This site is hosted by OASIS