Re: [xml-dev] I processed a 3GB XML file ... using XSLT streaming

[Ivan Pedruzzi <ivanpedruzzi@gmail.com>]

DataDirect XQuery, a commercial 1.0 implementation, executes the following query in ~ 100 seconds over the 2.7 GB input file, using the default Java memory heap size.

Both streaming and projection are automatically deployed by the processor when possible.

<schools>{

  for $node in doc('massachusetts.osm')/osm/node

  where $node/tag/@k = 'amenity' and $node/tag/@v = 'school'

    return 

  <school>{$node/tag[@k eq 'name']/@v}</school>

}</schools>

Ivan

On Fri, Sep 13, 2013 at 12:03 PM, Costello, Roger L. <costello@mitre.org> wrote:

Martin Honnen wrote:

> Where do you generate that list? The streaming version of
> your program seems to just output a 'count' element but
> not 'Schools' nor 'school' elements.

Oops! I guess that I sent an old version of my program. Sorry about that. Thanks for catching this Martin!

Here is the correct version:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                         xmlns:xs="http://www.w3.org/2001/XMLSchema"
                         exclude-result-prefixes="#all"
                        version="3.0">

    <xsl:output method="xml" />

    <xsl:template match="/">
        <xsl:stream href=""../huge-file/massachusetts.xml">

            <Schools>

                <xsl:for-each select="osm">
                    <xsl:iterate select="node">

                        <xsl:param name="count" select="1" as="xs:decimal"/>

                        <xsl:variable name="thisNode" select="copy-of(.)"/>
                        <xsl:choose>
                            <xsl:when test="$thisNode/tag[(@k eq 'amenity') and (@v eq 'school')]">
                                <school>

                                    <xsl:value-of select="$count" />

                                    <xsl:text>. </xsl:text>
                                    <xsl:value-of select="$thisNode/tag[@k eq 'name']/@v" />
                                </school>

                                <xsl:next-iteration>
                                    <xsl:with-param name="count" select="$count+1"/>
                                </xsl:next-iteration>

                            </xsl:when>
                            <xsl:otherwise>
                                <xsl:next-iteration>
                                    <xsl:with-param name="count" select="$count"/>
                                </xsl:next-iteration>
                            </xsl:otherwise>
                        </xsl:choose>
                    </xsl:iterate>
                </xsl:for-each>
            </Schools>

        </xsl:stream>
    </xsl:template>

</xsl:stylesheet>

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php