Lists Home |
Date Index |
----- Original Message -----
From: "Paul T" <email@example.com>
To: "K. Ari Krupnikov" <firstname.lastname@example.org>
Sent: Tuesday, January 08, 2002 9:13 PM
Subject: [xml-dev] Re: web crawling (was: Re: [xml-dev] HGRAB. Syndication.
Google. Grey area.)
> I've looked at many robots.txt files and nobody
> disallows the /. Maybe there are some especial websites
> that *do* that, but http://www.metasystema.org/terms.mhtml
> looks like a *very* rare example to me. But that's
> irrelevant, because you make a stronger point.
THINGS TO DO IF I BECOME AN EVIL OVERLORD #71
If I decide to test a lieutenant's loyalty and see if he/she should be made
a trusted lieutenant, I will have a crack squad of marksmen standing by
in case the answer is no.
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com