Re: [UUG/MLUG] MLUG Server & Robots Policy

[Mike Miller <EMAIL:PROTECTED>]

On Wed, 1 Dec 1999 EMAIL:PROTECTED wrote:

> I would think it would be a benefit to have the MLUG server accessed
> by robots and indexed on search engines.  I do exclude robots from
> machines at work since they aren't really for outside viewing.


It doesn't have to be all-or-nothing.  You can exclude robots from certain
directories.

To read a site's robots.txt file, all you have to do is type in the URL,
for example:

http://www.yahoo.com/robots.txt

This is essentially how the robots read them.  I became aware of the
existence of these files when I saw that my web site was getting a large
number of requests for a file called robots.txt.  So I looked it up and
then I made one and started using it.

Unfortunately, not all robots follow the rules.  I think it was Inktomi
that seemed to ignore my robots.txt file.

Regards,

Mike

-- 
Michael B. Miller
University of Missouri--Columbia
http://taxa.psyc.missouri.edu/~mbmiller/