Re: [xml-dev] XML to graph

On Wed, Jul 1, 2015 at 2:54 PM, daniela florescu <dflorescu@me.com> wrote:

Ihe,

before you load anything anywhere, you need to do data cleaning on this data
if you do integration from the Web and data has no unique ids…..

The data does have a unique id - title and year. Actually actually it's something that represents a year rather than an actual year such that title and year always give a unique movie (so it might be something like 1998/I etc). The year attribute is normalised to appear on a parent element containing all the other movies of that year.

This particular data is not scraped from the web. I constructed it by integrating 20 or so text dumps each holding a facet of movie data (i.e a directors text dump, an actors text dump etc) where title and year were used to join across data sets.

Unfortunately the unique id is not that useful because it's meaningful and unwieldly and other movie data silos use different identifiers and will not necessarily carry the movie and title here as a foreign key. Thus the id here cannot be relied upon to match other movie data sets. If it could the problem I am trying to solve would not exist. Although this is not data scraped from the web it will be matched against data that was, so in effect there is no real primary key here since what I have can't be used as one.

Similar problem exists in music. You have the same song released under different titles - Billy Ocean Carribean Queen/European Queen or Rap music where you may have a real song title featuring profanity and a sanitised title that can be used for radio airplay.

In particular entity resolution…

Literature is full of data cleaning and entity resolution algorithms.

One that you will find familiar (because it looks very much like XQuery :-) is here:

http://www.inesc-id.pt/ficheiros/publicacoes/1259.pdf

Ok. I'll let you know my thoughts on it and if feasible might even try it out. Although I have a solution it is particular to movies and reliant on the availability of alias information so something less tightly coupled to the movie domain that performs just as well would be of interest.

People have written papers on the challenges of constructing movie databases, mostly centred around the problems I've articulated here, I hadn't read them before I attacked this myself but their approach entailed trying to construct a normalised relational database - something I avoided. Here is one such paper detailing the entity resolution heuristics they used

http://apmd.prism.uvsq.fr/Publications/Rapports/Matching%20of%20MovieLens%20and%20IMDb%20Movie%20Titles_Veronika%20Peralta.pdf

When I embarked on this I didn't realise it had been the object of research so I did something different and I think better because it took advantage of XML technology.