[
Lists Home |
Date Index |
Thread Index
]
Joshua Allen wrote:
> Well, clearly I misunderestimated the desire for
> differentiation between new and updated.
> Of course, *no* scheme will work without being able
> to identify the article uniquely, ...
Right. Unique identifiers are required. Fortunately, Atom
requires this. Once we have unique identifiers and dates, we can start
doing a much better job of determining what items we've already seen.
With RSS, we're stuck having to generate MD5 hashes of each
item we've seen in the past and then using the hashes to determine
"newness." But, there are many cases in which this doesn't work.
We've got some publishers who are rotating ads through their
feeds and thus generating new hashes every time we scan their stuff.
We're also seeing RSS generators that seem to create almost gratuitous
modifications to "old" entries on a regular basis (i.e. white space
appears and disappears, line breaks change, tabs turn into spaces,
etc. "Old" entries get converted to summaries while new entries are
provided in full form... It begins to feel almost like humans editing
the stuff..) These "dynamic" feeds tend to make the MD5 stuff useless.
The result is that we feed way more duplicates to people than
we should have to and folk don't like that. But, I think we're doing
close to the best that is possible given the junk that we have to
parse. What we need is something different to parse.
bob wyman
|
