OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]
Re: [xml-dev] What is the general direction you are seeing these daysto store and query lots of large complex XML?

I'd go with MarkLogic for that. If the large documents are segmentable. I've run upwards of 100 million XML document files comfortably on a three node cluster.

Kurt Cagle
Principle Evangelist, Semantic Technologies
Avalon Consulting, LLC

On Thu, Mar 5, 2015 at 5:08 PM, liam <liam@w3.org> wrote:
On 2015-03-05 19:05, Gareth Oakes wrote:
Querying 2.4PB (2400TB) of data is going to take a
long time regardless of how you do it.

The best way is probably not doing it, where possible - e.g. project a subset and query that.

MarkLogic is marketed as petabyte-scale,
but XML may not be the most optimal representation of the data, it is
worth considering the alternatives.
Luckily neither MarkLogic nor most other XML-native databases store XML; rather, they store an internal representation that can be mapped to the XPath and XQuery Data Model (the XDM).

A common mistake people make (I'm not saying you're making it, but people reading your post might jump to an incorrect conclusion) is thinking that XML databases store XML (e.g. store pointy brackets), which is a bit like thinking that relational databases operate on CSV files.

Distributed storage and query are probably the way forward,
Depending on the nature of the queries, that does seem likely.

Liam Quin, W3C
XML Activity Lead;
Digital publishing; HTML Accessibility


XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]

News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 1993-2007 XML.org. This site is hosted by OASIS