[cobalt-security] What is robots.txt?

[Siao Yuan Tan <sy.tan@xxxxxxxxxx>]

Hello,

Inside my Cobalt RaQ4r, i found this file robots.txt in /usr/admserv/html/
folder with the following content:

# Prevent all robots from visiting this site:

User-agent: *
Disallow: /

I come to know this file from the webalizer report because this file seem to
have a number of hits to it.  Anyone know what is this file doing in my
server?

Thanks.

Siao Tan