Robots.txt
Robots.txt Exclusion Standard Information
Search engines will look in your root domain for a special file named “robots.txt”
The file tells the robot (spider) which files it may spider (download).
Robots.txt Tutorial with examples.
Robots.txt Validator to validate your robots.txt for syntax errors.
Robots Exclusion Standard corrent working draft proposal.
Robots.txt Examples that you can use and modify (rename to robots.txt before using):
