[
Lists Home |
Date Index |
Thread Index
]
----- Original Message -----
From: "Paul T" <pault12@pacbell.net>
To: "K. Ari Krupnikov" <ari@cogsci.ed.ac.uk>
Cc: <xml-dev@lists.xml.org>
Sent: Tuesday, January 08, 2002 9:13 PM
Subject: [xml-dev] Re: web crawling (was: Re: [xml-dev] HGRAB. Syndication.
Google. Grey area.)
> <aside>
> I've looked at many robots.txt files and nobody
> disallows the /. Maybe there are some especial websites
> that *do* that, but http://www.metasystema.org/terms.mhtml
> looks like a *very* rare example to me. But that's
> irrelevant, because you make a stronger point.
> </aside>
http://www.kpmg.com/robots.txt
--
THINGS TO DO IF I BECOME AN EVIL OVERLORD #71
If I decide to test a lieutenant's loyalty and see if he/she should be made
a trusted lieutenant, I will have a crack squad of marksmen standing by
in case the answer is no.
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
|
