RE: [cobalt-security] What is robots.txt?

["Colin J. Raven" <cjraven@xxxxxxxxxxx>]

>"Colin J. Raven" wrote:
>
>> On Thu, 8 Mar 2001, Siao Yuan Tan wrote:
>> > Inside my Cobalt RaQ4r, i found this file robots.txt in
>/usr/admserv/html/
>> > folder with the following content:
>> >
>> > # Prevent all robots from visiting this site:
>> >
>> > User-agent: *
>> > Disallow: /
>> >
>> > I come to know this file from the webalizer report because
>this file seem to
>> > have a number of hits to it.  Anyone know what is this
>file doing in my
>> > server?
>
>>
>> It's there to prevent spiders from roving through (and
>reporting on) your
>> admin pages. .....<snip>
>
>Can I suggest it this is not in fact the case !  - meant nicely :)
>It is really there to ***ASK*** robots / spiders not to
>perform the search (or used in a normal context, to request
>limits as to what gets searched  i.e. "Disallow /images" is a
>common example).
>"Rude" robots will not even bother looking at the file <g>
>
>I know its only symatics, but it is important for all to
>realise that this method will "NOT" stop nosey robots !
>--
Ah yes indeed, you are *so* correct Kul. I was semantically incorrect,
and the correct explanation is the one you tendered above. Thank you for
the clarification, it *is* a most important distinction.
Regards,
-Colin
--
Colin J. Raven