XML.orgXML.org
FOCUS AREAS |XML-DEV |XML.org DAILY NEWSLINK |REGISTRY |RESOURCES |ABOUT
OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]
Re: [xml-dev] I processed a 3GB XML file ... using XSLT streaming
Nice sample file, but not reall deep [count(/*/*/*/*)=0].

Even xsltproc was able to process "count(/*)" for that file, see (B) below.
It did need 39GB of resident size though (C), and completed in 1:44 min.
dp0-l3 has 12 cores @3GHz and 96GB memory, but xsltproc did only use one.

In (A) xpath++ tool executes non-streaming and includes network transfer
time from  dp0-l3  server to  firestar  DataPower appliance.


(A)
[stammw@dp0-l3 osm]$ time xpath++ "count(/*)" massachusetts.osm
1

real	0m29.488s
user	0m1.112s
sys	0m2.787s
[stammw@dp0-l3 osm]$ time xpath++ "count(//*)" massachusetts.osm
34840742

real	0m31.547s
user	0m1.144s
sys	0m3.305s
[stammw@dp0-l3 osm]$ time xpath++ "count(/*/*)" massachusetts.osm
10926903

real	0m32.047s
user	0m0.996s
sys	0m2.844s
[stammw@dp0-l3 osm]$ time xpath++ "count(/*/*/*)" massachusetts.osm
23913838

real	0m31.377s
user	0m1.005s
sys	0m2.828s
[stammw@dp0-l3 osm]$ time xpath++ "count(/*/*/*/*)" massachusetts.osm
0

real	0m32.280s
user	0m0.966s
sys	0m2.784s
[stammw@dp0-l3 osm]$



(B)
(xpath++ was configured to use xsltproc here)

[stammw@dp0-l3 osm]$ time xpath++ "count(/*)" massachusetts.osm
1

real	1m44.435s
user	1m28.347s
sys	0m15.707s
[stammw@dp0-l3 osm]$


(C)
top - 19:51:09 up 85 days,  9:56,  6 users,  load average: 0.95, 0.68, 0.41
Tasks: 483 total,   3 running, 468 sleeping,   0 stopped,  12 zombie
Cpu(s):  9.1%us,  0.5%sy,  0.0%ni, 90.4%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:  99054652k total, 52322948k used, 46731704k free,  2064480k buffers
Swap: 100663288k total,        0k used, 100663288k free,  3192556k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31459 stammw    20   0 39.6g  39g 1204 R 99.6 41.9   1:41.08 xsltproc


Mit besten Gruessen / Best wishes,

Hermann Stamm-Wilbrandt
Level 3 support for XML Compiler team and Fixpack team lead
WebSphere DataPower SOA Appliances
https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/
https://twitter.com/HermannSW/     http://www.stamm-wilbrandt.de/ce/
----------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"Costello, Roger L." <costello@mitre.org>                                                                                                         |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"xml-dev@lists.xml.org" <xml-dev@lists.xml.org>,                                                                                                  |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |09/13/2013 05:39 PM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |[xml-dev] I processed a 3GB XML file ... using XSLT streaming                                                                                     |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





Hi Folks,

I processed this 3GB Open Street Map (OSM) XML file for the state of
Massachusetts:

http://downloads.cloudmade.com/americas/northern_america/united_states/massachusetts/massachusetts.osm.bz2


An OSM file consists of <node> elements (a node represents a point), <way>
elements (a way represents a line or area), and <relation> elements (a
relationship represents a relationship between other elements). This page
describes the elements:

http://wiki.openstreetmap.org/wiki/Tags

Here is a snippet of the XML file; it shows the data for one of the schools
in Massachusetts (a school is identified by @k="amenity" @v="school"):

<osm version="0.6" generator="osm-extract.pl">
    ...
    <node id="358264143" version="1" timestamp="2009-03-10T04:54:34Z"
		   uid="4732" user="iandees" changeset="774950"
lat="42.2017681"
		   lon="-70.7561527">
        <tag k="gnis:created" v="08/27/2002"/>
        <tag k="gnis:county_id" v="023"/>
        <tag k="name" v="Scituate Center Central School"/>
        <tag k="amenity" v="school"/>
        <tag k="gnis:feature_id" v="602607"/>
        <tag k="gnis:state_id" v="25"/>
        <tag k="ele" v="34"/>
    </node>
    ...
</osm>

Problem: What are all the schools in the state of Massachusetts?

I was able to answer that problem using the new streaming capability in
XSLT 3.0.

Below are two versions of an XSLT program. The first version (non-streaming
version) is how one might try to solve the problem prior to XSLT streaming.
I ran it on the XML file and it quickly halted with an "Out of memory"
error message. The second version uses XSLT streaming and it solved the
problem. There are over 6000 schools in Massachusetts! At the bottom of
this message I show some of the schools.

How long did it take for the streaming XSLT program to process the 3GB XML
file? Answer: 134 seconds.

I ran my streaming XSLT program using oXygen XML, which invoked SAXON. I
ran it on my laptop running Windows 7.

--------------------------------------------------
    Non-streaming approach
--------------------------------------------------
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
                         version="3.0">

    <xsl:output method="xml" />

    <xsl:variable name="MA" select="doc
('../huge-file/massachusetts.xml')" />

    <xsl:template match="/">
        <xsl:apply-templates select="$MA/osm" />
    </xsl:template>

    <xsl:template match="osm">
        <Schools>
            <xsl:for-each select="node">
                <xsl:if test="tag[(@k eq 'amenity') and (@v eq 'school')]">
                    <school>
                        <xsl:value-of select="tag[@k eq 'name']/@v" />
                    </school>
                </xsl:if>
            </xsl:for-each>
        </Schools>
    </xsl:template>

</xsl:stylesheet>

--------------------------------------------------
    Streaming approach
--------------------------------------------------
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
                         xmlns:xs="http://www.w3.org/2001/XMLSchema";
                         exclude-result-prefixes="#all"
                         version="3.0">

    <xsl:output method="xml" />

    <xsl:template match="/">
        <xsl:stream href="../huge-file/massachusetts.xml">
            <count>
                <xsl:for-each select="osm">
                    <xsl:iterate select="node">
                        <xsl:param name="count" select="0"
as="xs:decimal"/>
                        <xsl:next-iteration>
                            <xsl:with-param name="count" select="$count
+1"/>
                        </xsl:next-iteration>
                        <xsl:on-completion>
                            <xsl:value-of select="$count"/>
                        </xsl:on-completion>
                    </xsl:iterate>
                </xsl:for-each>
            </count>
        </xsl:stream>
    </xsl:template>

</xsl:stylesheet>

--------------------------------------------------
    Schools in Massachusetts
--------------------------------------------------
<Schools>
    <school>1. Wayland Middle School</school>
    <school>2. Walnut Hill School for the Arts</school>
    <school>3. Library</school>
    <school>4. Resource Center</school>
    <school>5. Deerfield Academy</school>
    <school>6. Arthur W Coolidge Middle</school>
    <school>7. Lilliput School</school>
    <school>8. L H Coffin</school>
    <school>9. St Joseph Central High</school>
    <school>10. Stoneham High School</school>
    <school>11. St Louis</school>
    <school>12. Franklin</school>
    <school>13. East Falmouth Elem</school>
    <school>14. May Institute (Woburn)</school>
    <school>15. Frolio Jr Hs</school>
    <school>16. Barbieri Elem</school>
    <school>17. Bishop Stang High</school>
    <school>18. Jackson Mann</school>
    <school>19. Douglas</school>
    <school>20. Brookfield Elementary</school>

    ...

    <school>5996. Hawley Grammar School (was here)</school>
    <school>5997. Shady Hill School</school>
    <school>5998. Cotting School</school>
    <school>5999. Morril 2</school>
    <school>6000. Morril 4 North</school>
    <school>6001. Morril 3</school>
    <school>6002. John W Wynn Middle School</school>
    <school>6003. John W. Wynn Middle</school>
    <school>6004. Boston Architectural College</school>
    <school>6005. High School</school>
    <school>6006. Sargent College</school>
    <school>6007. Stop -n- Go Driving Academy</school>
    <school>6008. Ashland High School</school>
    <school>6009. Center Stage Dance Academy</school>
</Schools>

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]

News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 1993-2007 XML.org. This site is hosted by OASIS