Re: [cobalt-security] What is robots.txt?

[shimi <shimi@xxxxxxxxxxxxxxxx>]

will NOT be indexed?
i don't think so.

  Best regards,
     shimi [mailto:shimi@xxxxxxxxxxxxxxxx]


----

There are two major products that come out of Berkeley: LSD and BSD.
We don't believe this to be a coincidence.
   -- Jeremy S. Anderson

 Windows: "Where do you want to go today?"
   Linux: "Where do you want to go tomorrow?"
     BSD: "Are you guys coming or what?"

On Sun, 11 Mar 2001, Robbert Hamburg wrote:

> ---- Original Message -----
> From: Siao Yuan Tan <sy.tan@xxxxxxxxxx>
> To: <cobalt-security@xxxxxxxxxxxxxxx>
> Sent: Thursday, March 08, 2001 7:18 AM
> Subject: [cobalt-security] What is robots.txt?
> 
> 
> > Hello,
> >
> > Inside my Cobalt RaQ4r, i found this file robots.txt in /usr/admserv/html/
> > folder with the following content:
> >
> > # Prevent all robots from visiting this site:
> >
> > User-agent: *
> > Disallow: /
> >
> 
> 
> It is used to "talk" with search engines. if this file is not present your
> sites will not be indexed,
> you can also use this to prevent your sites of being indexed.
> 
> http://info.webcrawler.com/mak/projects/robots/meta-user.html
> is a good start. Or search for robots.txt on the internet.
> 
> regards,
> 
> Robbert Hamburg
> 
> ______________________________________
> DISCLAIMER:
> This emailmessage and any attachment may contain confidential and privileged
> material intended for the addressee only.
> If you are not the addressee, you are hereby notified that no part of the
> emailmessage or any attachment may be
> disclosed, copied or distributed, and that any other action related to this
> emailmessage or attachment is strictly
> prohibited, and will be prosecuted as a criminal offense to the full extent
> of the law. If you have received this
> emailmessage by error, please notify the sender immediately by return
> emailmessage, and delete this message.
> HAVA web- and processdesign (HAVA.nl), its subsidiaries and/or its employees
> shall not be liable for the incorrect or
> incomplete transmission of this emailmessage or any attachments, nor
> responsible for any delay in receipt.
> 
> _______________________________________________
> cobalt-security mailing list
> cobalt-security@xxxxxxxxxxxxxxx
> http://list.cobalt.com/mailman/listinfo/cobalt-security
>