[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [cobalt-security] What is robots.txt?
- Subject: Re: [cobalt-security] What is robots.txt?
- From: "Colin J. Raven" <cjraven@xxxxxxxxxxx>
- Date: Sun, 11 Mar 2001 06:43:34 -0500 (EST)
- List-id: Mailing list for users to address network security on Cobalt products. <cobalt-security.list.cobalt.com>
On Thu, 8 Mar 2001, Siao Yuan Tan wrote:
> Inside my Cobalt RaQ4r, i found this file robots.txt in /usr/admserv/html/
> folder with the following content:
>
> # Prevent all robots from visiting this site:
>
> User-agent: *
> Disallow: /
>
> I come to know this file from the webalizer report because this file seem to
> have a number of hits to it. Anyone know what is this file doing in my
> server?
It's there to prevent spiders from roving through (and reporting on) your
admin pages. Do you want to see your "users" admin page on a serach
engine (for example)???
IMO that's a highly desireable file to have in your admserv setup, whether
put there by webalizer or not. I'm gonna check my boxen now to see if its
there, and create it if not!!!
Regards,
-Colin
--
Colin J. Raven
Linux Registered User #82296
Sun Mar 11 06:39:00 EST 2001
6:39am up 11 days, 11:11, 1 user, load average: 0.00, 0.01, 0.00