Re: [xml-dev] I processed a 3GB XML file ... using XSLT streaming

From
Ivan Pedruzzi <>
To
"Costello, Roger L." <>
Date
2013-09-13T20:30:43Z
ID
<>
Thread
Re: [xml-dev] I processed a 3GB XML file ... using XSLT streaming

DataDirect XQuery, a commercial 1.0 implementation, executes the following query in ~ 100 seconds over the 2.7 GB input file, using the default Java memory heap size.

Both streaming and projection are automatically deployed by the processor when possible.

<schools>{
  for $node in doc('massachusetts.osm')/osm/node
  where $node/tag/@k = 'amenity' and $node/tag/@v = 'school'
    return 
  <school>{$node/tag[@k eq 'name']/@v}</school>
}</schools>


Ivan






On Fri, Sep 13, 2013 at 12:03 PM, Costello, Roger L. <[email protected]> wrote:
Martin Honnen wrote:

> Where do you generate that list? The streaming version of
> your program seems to just output a 'count' element but
> not 'Schools' nor 'school' elements.

Oops! I guess that I sent an old version of my program. Sorry about that. Thanks for catching this Martin!

Here is the correct version:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                         xmlns:xs="http://www.w3.org/2001/XMLSchema"
                         exclude-result-prefixes="#all"
                        version="3.0">

    <xsl:output method="xml" />

    <xsl:template match="/">
        <xsl:stream href="../huge-file/massachusetts.xml">
            <Schools>
                <xsl:for-each select="osm">
                    <xsl:iterate select="node">
                        <xsl:param name="count" select="1" as="xs:decimal"/>
                        <xsl:variable name="thisNode" select="copy-of(.)"/>
                        <xsl:choose>
                            <xsl:when test="$thisNode/tag[(@k eq 'amenity') and (@v eq 'school')]">
                                <school>
                                    <xsl:value-of select="$count" />
                                    <xsl:text>. </xsl:text>
                                    <xsl:value-of select="$thisNode/tag[@k eq 'name']/@v" />
                                </school>
                                <xsl:next-iteration>
                                    <xsl:with-param name="count" select="$count+1"/>
                                </xsl:next-iteration>
                            </xsl:when>
                            <xsl:otherwise>
                                <xsl:next-iteration>
                                    <xsl:with-param name="count" select="$count"/>
                                </xsl:next-iteration>
                            </xsl:otherwise>
                        </xsl:choose>
                    </xsl:iterate>
                </xsl:for-each>
            </Schools>
        </xsl:stream>
    </xsl:template>

</xsl:stylesheet>

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: [email protected]
subscribe: [email protected]
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php