RE: [xml-dev] Out of memory error with XML and Java ResultSet

["Brown, Jason B." <Jason.B.Brown@ibx.com>]

Dennis,

One row of XML text is approximately 230 characters, including tag name
and attribute specifiers.  I thought my implementation was streaming the
information to the destination.  Here is the code for the XML writing
class:

	/**
	 * Constructor
	 */
	public CisXmlWriterSAX(StreamResult xmlOutput,
	                       String dtdFile) {
		try {
			transFact = (SAXTransformerFactory)
SAXTransformerFactory.newInstance();
			
			handler = transFact.newTransformerHandler();
			
			serializer = handler.getTransformer();

			serializer.setOutputProperty(OutputKeys.METHOD,
"XML");
			serializer.setOutputProperty(OutputKeys.INDENT,
"yes");
	
serializer.setOutputProperty(OutputKeys.DOCTYPE_SYSTEM, dtdFile);
	
serializer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
			
			handler.setResult(xmlOutput);

/*
Old code ----------
			of = new OutputFormat(outputType,
outputEncoding, true);
			of.setIndent(1);
			of.setIndenting(true);
			of.setDoctype(null, dtdFile);

			// Might be able to use StringWriter
			serializer = new XMLSerializer(xmlOutput, of);

			handler = serializer.asContentHandler();
Old code ----------
*/

		} 
        catch (TransformerConfigurationException tce) {
      	    CDebug.print("Error in CisXmlWriterSAX: " + tce.toString());
			if (CDebug.isDebug()) {
				tce.printStackTrace();
			}
        }

	}

If I stream the information directly to the destination, then that means
the destination would have to receive the information and process it.
Currently, I am saving the information in a StringBuffer object which
gets returned to the calling object.  But, that is still storing the
information in memory, which can be costly.

Thanks.

Jason B. Brown
CIS IS Claims
215-241-4609
x2-4609 (internal), 11th Floor
mailto:jason.b.brown@ibx.com


-----Original Message-----
From: Dennis Sosnoski [mailto:dms@sosnoski.com] 
Sent: Monday, September 08, 2003 12:17 PM
To: Brown, Jason B.
Cc: xml-dev@lists.xml.org
Subject: Re: [xml-dev] Out of memory error with XML and Java ResultSet


Hi Jason,

Do you know the typical size of the XML text generated from each row? 
Also, do you need to have the full document buffered in memory or do you

really want it as a stream that's being sent somewhere else?

In general, you're going to want to avoid any approach (such as DOM) 
that builds a representation of the XML structure in memory. Even just 
buffering the output text in memory is probably going to cause problems 
if this gets longer than a few MB. Your best bet is probably just to 
stream it out, either directly to the destination or to a file on the 
local system.

  - Dennis

Brown, Jason B. wrote:

>Greetings.
>
>I am new to this mailing list.  I was wondering if someone could help 
>me with a small problem I am having trying to create an XML buffer 
>using data from a Java ResultSet.  The ResultSet object contains 
>approximately 300,000 rows, which I want to place in a XML buffer.  I 
>managed to develop an XML implementation that uses JAXP 1.1.  The logic

>for the processing is this: I execute the SQL query, obtain the 
>ResultSet, loop through the result set and as each row in the result 
>set is encountered, I use getLong methods, etc to get the data and 
>write it to a StreamResult object.
>
>Here is a code snippet:
>			rsMetaData = rs.getMetaData();
>
>			// Provide the System.out object as the output
>stream for now
>			xmlWrtr = new CisXmlWriterSAX(new
>StreamResult(xmlData), 
>							((String)
>CUtil.getEnv("AdhocDataDtd")));
>
>			numOfColumns = rsMetaData.getColumnCount();
>
>			// 
>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>			// Start XML document
>			xmlWrtr.startXMLDocument();
>
>			// Set root element
>			xmlWrtr.addXMLElement("AdhocClaims");
>
>			System.out.println("<<>> Building XML file...");
>
>			while (rs.next()) {
>				// Calculate number of rows in result
>set
>				numOfRows++;
>				
>				CDebug.print("" + numOfRows);
>
>				// Create the Row parent element (child
>element to AdhocClaims element)
>				xmlWrtr.addXMLElement("Row");
>
>				for (int colIndex = 1; colIndex <=
>numOfColumns; colIndex++) {
>
>				.....
>					tempStr = 
>crossRef.getReadableName(rsMetaData.getColumnName(colIndex));
>
>					// Create the attributes for the
>element
>					xmlWrtr.addElementAttr("name",
>"CDATA", tempStr);
>
>					// Add Column element
>					xmlWrtr.addXMLElement("Column");
>
>					// Add the Value element and a
>value for the Value element
>					xmlWrtr.addElementValue(null,
>"");
>
>					// End Column element
>					xmlWrtr.endXMLElement("Column");
>				}
>				.....
>			}
>
>			xmlWrtr.endXMLElement("AdhocClaims");
>			
>			// This is done when finished with adding all
>elements within root element
>			xmlWrtr.endXMLDocument();
>			......
>
>xmlWrtr is a class that contains the following objects:
>	private SAXTransformerFactory transFact = null;
>
>	/**
>	* Contains where the resulting XML will be placed/contained
>	*/
>	private StreamResult resultStream = null;
>	
>	private TransformerHandler handler = null;
>
>	private Transformer serializer = null;
>
>	private AttributesImpl attr = null;
>
>My main goal with this posting is to obtain a memory-friendly solution 
>for taking information from a Java ResultSet and placing it in a XML 
>buffer.
>
>Thanks.
>
>Jason B. Brown
>CIS IS Claims
>215-241-4609
>x2-4609 (internal), 11th Floor
>mailto:jason.b.brown@ibx.com
>
>
>
>CONFIDENTIALITY NOTICE: This E-Mail is intended only
>for the use of the individual or entity to which it is addressed and
may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If you have received this communication
in error, please do not distribute and delete the original message.
Please notify the sender by E-Mail at the address shown. Thank you for
your compliance..
>
>
>-----------------------------------------------------------------
>The xml-dev list is sponsored by XML.org <http://www.xml.org>, an 
>initiative of OASIS <http://www.oasis-open.org>
>
>The list archives are at http://lists.xml.org/archives/xml-dev/
>
>To subscribe or unsubscribe from this list use the subscription
>manager: <http://lists.xml.org/ob/adm.pl>
>
>  
>


CONFIDENTIALITY NOTICE: This E-Mail is intended only 
for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you have received this communication in error, please do not distribute and delete the original message.  Please notify the sender by E-Mail at the address shown. Thank you for your compliance.