Re: [cobalt-security] What is robots.txt?

["Robbert Hamburg" <rhamburg@xxxxxx>]

---- Original Message -----
From: Siao Yuan Tan <sy.tan@xxxxxxxxxx>
To: <cobalt-security@xxxxxxxxxxxxxxx>
Sent: Thursday, March 08, 2001 7:18 AM
Subject: [cobalt-security] What is robots.txt?


> Hello,
>
> Inside my Cobalt RaQ4r, i found this file robots.txt in /usr/admserv/html/
> folder with the following content:
>
> # Prevent all robots from visiting this site:
>
> User-agent: *
> Disallow: /
>


It is used to "talk" with search engines. if this file is not present your
sites will not be indexed,
you can also use this to prevent your sites of being indexed.

http://info.webcrawler.com/mak/projects/robots/meta-user.html
is a good start. Or search for robots.txt on the internet.

regards,

Robbert Hamburg

______________________________________
DISCLAIMER:
This emailmessage and any attachment may contain confidential and privileged
material intended for the addressee only.
If you are not the addressee, you are hereby notified that no part of the
emailmessage or any attachment may be
disclosed, copied or distributed, and that any other action related to this
emailmessage or attachment is strictly
prohibited, and will be prosecuted as a criminal offense to the full extent
of the law. If you have received this
emailmessage by error, please notify the sender immediately by return
emailmessage, and delete this message.
HAVA web- and processdesign (HAVA.nl), its subsidiaries and/or its employees
shall not be liable for the incorrect or
incomplete transmission of this emailmessage or any attachments, nor
responsible for any delay in receipt.