xml-dev - Re: A processing instruction for robots

[Walter Underwood <wunder@infoseek.com>]

At 04:54 PM 12/8/99 -0800, tbray@textuality.com wrote:
>
>But my big problem is with the idea that individual resources ought 
>to embed robot-steering information. It just feels like the wrong 
>level of granularity.

For really picky indexing and searching, it is wrong. 
Structural markup opens up some really nice possibilities.
An indexer might weight the bibliography less and the
abstract more, for example.

But that sort of tweakiness changes for each search engine.
So I'd implement that as a DTD-specific configuration in
each engine, rather than trying to add processor-specific
markup to each document. In fact, I already implemented it
that way. Use the structure, Luke.

On the plus side, XML tends to be content-rich, without
navbars and decoration. This means that you get better quality
results without resorting to tweaks. For example, you can 
actually search for "Home", "Copyright", or "Help" and get 
relevant results.

>...  The PI has the characteristic that it *has* to be in
>the document and can modify *only* the whole document.  Also I 
>question the ability of authors to do the right thing with this 
>kind of a macro-level control.  Also I question the ability of robot 
>authors to do the right thing at the individual document level.

I'm willing to trust the authors and webmasters. There are a
lot of professionals out there. As for robot authors, if the
robots PI semantics are the same as the HTML robots meta tag
semantics, it should be pretty easy to get right. If they are
different, all bets are off.

>Anyhow, is there enough XML on the web to make this interesting?  
>Serious question, I don't know the answer. -T.

We're seeing XML-backed websites where they want to index
the XML, but serve URLs pointing to the formatted HTML.

wunder
--
Walter R. Underwood
wunder@infoseek.com
wunder@best.com (home)
http://software.infoseek.com/
http://www.best.com/~wunder/
1-408-543-6946

xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev@ic.ac.uk
Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/ and on CD-ROM/ISBN 981-02-3594-1
To unsubscribe, mailto:majordomo@ic.ac.uk the following message;
unsubscribe xml-dev
To subscribe to the digests, mailto:majordomo@ic.ac.uk the following message;
subscribe xml-dev-digest
List coordinator, Henry Rzepa (mailto:rzepa@ic.ac.uk)