Re: [cobalt-security] What is robots.txt?

[Kul <WebMaster@xxxxxxx>]

"Colin J. Raven" wrote:

> On Thu, 8 Mar 2001, Siao Yuan Tan wrote:
> > Inside my Cobalt RaQ4r, i found this file robots.txt in /usr/admserv/html/
> > folder with the following content:
> >
> > # Prevent all robots from visiting this site:
> >
> > User-agent: *
> > Disallow: /
> >
> > I come to know this file from the webalizer report because this file seem to
> > have a number of hits to it.  Anyone know what is this file doing in my
> > server?

>
> It's there to prevent spiders from roving through (and reporting on) your
> admin pages. .....<snip>

Can I suggest it this is not in fact the case !  - meant nicely :)
It is really there to ***ASK*** robots / spiders not to perform the search (or used in a normal context, to request limits as to what gets searched  i.e. "Disallow /images" is a common example).
"Rude" robots will not even bother looking at the file <g>

I know its only symatics, but it is important for all to realise that this method will "NOT" stop nosey robots !
--
Regards,
Kul