Re: [xml-dev] What is the general direction you are seeing these daysto

On Sun, Mar 8, 2015 at 3:00 PM, Ihe Onwuka <ihe.onwuka@gmail.com> wrote:

On Sat, Mar 7, 2015 at 9:05 PM, Liam R. E. Quin <liam@w3.org> wrote:

Sometimes it's about extracting just the bits you need to query and
putting just those in some kind of database - relational, NoSQL,
SlightlySQL, triple store, XML-native, hybrid, whatever, with no hope
of reconstructing the original -- if you need the original, just use
it directly.

Depends. If you are constructing a repository from a denormalized text dump and/or disparate sources you may not want or be able to reconstruct the original.

Actually I can concretize that. It describes most of the databases behind most public movie API's, e.g http://omdbapi.com/. Typically such databases consist of pieces of information cobbled together from various places usually they run on a LAMPish/Java platform. There are research papers detailing the challenges encountered in such an endeavour

http://www.info.univ-tours.fr/~veronika/publications/trapmd2-vp.pdf

Matching of MovieLens and ImDb titles.

http://www.info.univ-tours.fr/~veronika/publications/trapmd1-vp.pdf

Extraction and integration of MovieLens and ImDb data

I would expect similar problems (matching to support correct aggregation, deduplication, disambiguation) to manifest in many Big Data projects simply because data integration has the potential to add so much value. A "Small Data" project may morph into a Big Data one once you have assembled information from enough sources/silos.

I've been allowed to approach this problem with NXD technology. Schemalessness - being able to ingest data from anywhere in any format - and the ability to defer decisions on data management due to a superior extract and transform capability, has been key to dodging many of the problems that the MovieLens group experienced. It has also allowed the focus to shift to more ambitious higher value problems, although challenges remain.