Re: [xml-dev] I processed a 3GB XML file ... using XSLT streaming

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]
From: Hermann Stamm-Wilbrandt <STAMMW@de.ibm.com>
To: "Costello, Roger L." <costello@mitre.org>
Date: Fri, 13 Sep 2013 20:53:10 +0200
I did give your non-streaming stylesheet (B) a try (A).
Slight modifications were necessary to get back to XSLT 1.0
(eq -> = , doc -> document).
16GB of memory were used (17851470K-1067694K), (D).

(A)
[stammw@dp0-l3 osm]$ time curl http://firestar:2101 -s > out.xml

real	0m43.679s
user	0m0.004s
sys	0m0.003s
[stammw@dp0-l3 osm]$ xpath++ "count(/*/*)" out.xml
6009
[stammw@dp0-l3 osm]$


(B)
[stammw@dp0-l3 osm]$ cat Roger.xsl
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
                         version="1.0">

    <xsl:output method="xml" />

    <xsl:variable name="MA" select="document('massachusetts.osm')" />

    <xsl:template match="/">
        <xsl:apply-templates select="$MA/osm" />
    </xsl:template>

    <xsl:template match="osm">
        <Schools>
            <xsl:for-each select="node">
                <xsl:if test="tag[(@k = 'amenity') and (@v = 'school')]">
                    <school>
                        <xsl:value-of select="tag[@k = 'name']/@v" />
                    </school>
                </xsl:if>
            </xsl:for-each>
        </Schools>
    </xsl:template>

</xsl:stylesheet>
[stammw@dp0-l3 osm]$

(D)
...
xi52# show mem

    Memory Usage: 21 %
    Total Memory: 82333842 kilobytes
     Used Memory: 17851470 kilobytes
     Free Memory: 64482372 kilobytes
Requested Memory: 17877976 kilobytes
     Hold Memory: 26506 kilobytes
 Reserved Memory: 16863558 kilobytes
Installed Memory: 99197400 kilobytes


xi52# show mem

    Memory Usage: 1 %
    Total Memory: 82333842 kilobytes
     Used Memory: 1067694 kilobytes
     Free Memory: 81266148 kilobytes
Requested Memory: 1195560 kilobytes
     Hold Memory: 127866 kilobytes
 Reserved Memory: 16863558 kilobytes
Installed Memory: 99197400 kilobytes


xi52#

Mit besten Gruessen / Best wishes,

Hermann Stamm-Wilbrandt
Level 3 support for XML Compiler team and Fixpack team lead
WebSphere DataPower SOA Appliances
https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/
https://twitter.com/HermannSW/     http://www.stamm-wilbrandt.de/ce/
----------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"Costello, Roger L." <costello@mitre.org>                                                                                                         |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"xml-dev@lists.xml.org" <xml-dev@lists.xml.org>,                                                                                                  |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |09/13/2013 05:39 PM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |[xml-dev] I processed a 3GB XML file ... using XSLT streaming                                                                                     |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





Hi Folks,

I processed this 3GB Open Street Map (OSM) XML file for the state of
Massachusetts:

http://downloads.cloudmade.com/americas/northern_america/united_states/massachusetts/massachusetts.osm.bz2


An OSM file consists of <node> elements (a node represents a point), <way>
elements (a way represents a line or area), and <relation> elements (a
relationship represents a relationship between other elements). This page
describes the elements:

http://wiki.openstreetmap.org/wiki/Tags

Here is a snippet of the XML file; it shows the data for one of the schools
in Massachusetts (a school is identified by @k="amenity" @v="school"):

<osm version="0.6" generator="osm-extract.pl">
    ...
    <node id="358264143" version="1" timestamp="2009-03-10T04:54:34Z"
		   uid="4732" user="iandees" changeset="774950"
lat="42.2017681"
		   lon="-70.7561527">
        <tag k="gnis:created" v="08/27/2002"/>
        <tag k="gnis:county_id" v="023"/>
        <tag k="name" v="Scituate Center Central School"/>
        <tag k="amenity" v="school"/>
        <tag k="gnis:feature_id" v="602607"/>
        <tag k="gnis:state_id" v="25"/>
        <tag k="ele" v="34"/>
    </node>
    ...
</osm>

Problem: What are all the schools in the state of Massachusetts?

I was able to answer that problem using the new streaming capability in
XSLT 3.0.

Below are two versions of an XSLT program. The first version (non-streaming
version) is how one might try to solve the problem prior to XSLT streaming.
I ran it on the XML file and it quickly halted with an "Out of memory"
error message. The second version uses XSLT streaming and it solved the
problem. There are over 6000 schools in Massachusetts! At the bottom of
this message I show some of the schools.

How long did it take for the streaming XSLT program to process the 3GB XML
file? Answer: 134 seconds.

I ran my streaming XSLT program using oXygen XML, which invoked SAXON. I
ran it on my laptop running Windows 7.

--------------------------------------------------
    Non-streaming approach
--------------------------------------------------
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
                         version="3.0">

    <xsl:output method="xml" />

    <xsl:variable name="MA" select="doc
('../huge-file/massachusetts.xml')" />

    <xsl:template match="/">
        <xsl:apply-templates select="$MA/osm" />
    </xsl:template>

    <xsl:template match="osm">
        <Schools>
            <xsl:for-each select="node">
                <xsl:if test="tag[(@k eq 'amenity') and (@v eq 'school')]">
                    <school>
                        <xsl:value-of select="tag[@k eq 'name']/@v" />
                    </school>
                </xsl:if>
            </xsl:for-each>
        </Schools>
    </xsl:template>

</xsl:stylesheet>

--------------------------------------------------
    Streaming approach
--------------------------------------------------
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
                         xmlns:xs="http://www.w3.org/2001/XMLSchema";
                         exclude-result-prefixes="#all"
                         version="3.0">

    <xsl:output method="xml" />

    <xsl:template match="/">
        <xsl:stream href="../huge-file/massachusetts.xml">
            <count>
                <xsl:for-each select="osm">
                    <xsl:iterate select="node">
                        <xsl:param name="count" select="0"
as="xs:decimal"/>
                        <xsl:next-iteration>
                            <xsl:with-param name="count" select="$count
+1"/>
                        </xsl:next-iteration>
                        <xsl:on-completion>
                            <xsl:value-of select="$count"/>
                        </xsl:on-completion>
                    </xsl:iterate>
                </xsl:for-each>
            </count>
        </xsl:stream>
    </xsl:template>

</xsl:stylesheet>

--------------------------------------------------
    Schools in Massachusetts
--------------------------------------------------
<Schools>
    <school>1. Wayland Middle School</school>
    <school>2. Walnut Hill School for the Arts</school>
    <school>3. Library</school>
    <school>4. Resource Center</school>
    <school>5. Deerfield Academy</school>
    <school>6. Arthur W Coolidge Middle</school>
    <school>7. Lilliput School</school>
    <school>8. L H Coffin</school>
    <school>9. St Joseph Central High</school>
    <school>10. Stoneham High School</school>
    <school>11. St Louis</school>
    <school>12. Franklin</school>
    <school>13. East Falmouth Elem</school>
    <school>14. May Institute (Woburn)</school>
    <school>15. Frolio Jr Hs</school>
    <school>16. Barbieri Elem</school>
    <school>17. Bishop Stang High</school>
    <school>18. Jackson Mann</school>
    <school>19. Douglas</school>
    <school>20. Brookfield Elementary</school>

    ...

    <school>5996. Hawley Grammar School (was here)</school>
    <school>5997. Shady Hill School</school>
    <school>5998. Cotting School</school>
    <school>5999. Morril 2</school>
    <school>6000. Morril 4 North</school>
    <school>6001. Morril 3</school>
    <school>6002. John W Wynn Middle School</school>
    <school>6003. John W. Wynn Middle</school>
    <school>6004. Boston Architectural College</school>
    <school>6005. High School</school>
    <school>6006. Sargent College</school>
    <school>6007. Stop -n- Go Driving Academy</school>
    <school>6008. Ashland High School</school>
    <school>6009. Center Stage Dance Academy</school>
</Schools>

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
Follow-Ups:
- Re: [xml-dev] I processed a 3GB XML file ... using XSLT streaming
  - From: Liam R E Quin <liam@w3.org>
References:
- I processed a 3GB XML file ... using XSLT streaming
  - From: "Costello, Roger L." <costello@mitre.org>
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]