OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]
Re: [xml-dev] What is the general direction you are seeing these daysto store and query lots of large complex XML?

On Sun, Mar 8, 2015 at 3:00 PM, Ihe Onwuka <ihe.onwuka@gmail.com> wrote:

On Sat, Mar 7, 2015 at 9:05 PM, Liam R. E. Quin <liam@w3.org> wrote:

Sometimes it's about extracting just the bits you need to query and
putting just those in some kind of database - relational, NoSQL,
SlightlySQL, triple store, XML-native, hybrid, whatever, with no hope
of reconstructing the original -- if you need the original, just use
it directly.

Depends. If you are constructing a repository from a denormalized text dump and/or disparate sources you may not want or be able to reconstruct the original.

Actually I can concretize that. It describes most of the databases behind most public movie API's, e.g http://omdbapi.com/. Typically such databases consist of pieces of information cobbled together from various places usually they run on a LAMPish/Java platform. There are research papers detailing the challenges encountered in  such an endeavour 


Matching of MovieLens and ImDb titles.


Extraction and integration of MovieLens and ImDb data

I would expect similar  problems (matching to support correct aggregation, deduplication, disambiguation) to manifest in many Big Data projects simply because  data integration has the potential to add so much value. A "Small Data" project may morph into a Big Data one once you have assembled  information from enough sources/silos. 

I've been allowed to approach this problem with NXD technology. Schemalessness - being able to ingest data from anywhere in any format - and the ability to defer decisions on data management due to a superior extract and transform capability, has been key to dodging many of the problems that the MovieLens group experienced. It has also allowed the focus to shift to more ambitious higher value problems, although challenges remain.


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]

News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 1993-2007 XML.org. This site is hosted by OASIS