Re: [xml-dev] Converting a variety of data formats, containing variouski

[Thomas Passin <list1@tompassin.net>]

On 10/4/2016 11:53 AM, Costello, Roger L. wrote:
> Scenario: You are tasked to build a system that will receive data from a
> variety of sources. The data arrives in different formats (e.g., CSV,
> binary, tab-delimited, JSON, XML). The data sources provide different
> kinds of data (e.g., one source provides book data, another source
> provides weather data, another source provides gardening data). The data
> will be converted to a common intermediary form and then from the
> intermediary form, placed into a data store.
One of the problems in tackling the question is that we don't know why 
all this data is supposed to end up in a common data store.  The data 
are all about different kinds of things, so presumably there won't be an 
overlap between tables (just as an example, assuming the final data 
store even has tables).  So why?  Is it so that a single query language 
can be used over all the data?  Is it so that there is only one kind of 
data system software to administer or maintain?  Is it to satisfy some 
manager who insists on this scheme?  Is it a research project to see if 
RDF is suitable for some task?

But there are some things I notice that relate.  Some of the example 
formats are essentially the same thing.  e.g., CSV and tab-separated 
data really the same, and they both provide a row-oriented view of the 
data.  They can be viewed as representing the contents of a relational 
table - each row is a relation.  I'm assuming that each row contains the 
same columns throughout the file.  We can't be sure a priory which 
column(s) represent a primary key, or if there even is one, and we can't 
be sure about foreign keys, either.  There's your data conversion problem.

As relations in a relational table, they can be denormalized into groups 
of triples, so basically RDF.

XML, JSON, and binary can potentially contain nested data structures. 
So they have some real similarities, even if there could be specialized 
things that are common to one format, such as processing instructions in 
XML.  That specialized data, and the knowledge about it, have to be put 
into the specific data converter.

RDF can represent nested data structures.  Again, the converter has to 
be designed with specialized knowledge to make this work.

Roger quoted this claim: "RDF is the only data model the [sic] enables 
such conversions to be done in a way that is computationally tractable." 
 I think that is silly on its face.  However, if not RDF, then you 
really want some other existing standard, preferably one that has had a 
lot of development, and has practical software available including a 
query language.  So why not use RDF, which does have all those things?

Well, that might depend a lot on whether you need features that RDF is 
weak on, such as reification so that you can make metastatements about 
data provenance and history, for example.  Since we don't know anything 
about the intended usage of this universal data store, can't go further 
to answer the question.  But in view of the standards issue, and 
depending on the usage pattern, it does seem that RDF *could* be in the 
running.

TomP