Tutorials & Training from experienced world class trainers
Join 5700+ of your peers
Get enriched. Access 1270+ quality tutorials right away on:
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door. That is why we say that if you have really sensitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
The format and semantics of the "/robots.txt" file is as follows: The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>:<optionalspace><value><optionalspace>". The field name is case insensitive.
The robots.txt file is a good way to prevent the page from getting indexed. However, not every site can use it. The only robots.txt file that the spiders will read is the one at the top html directory of your server. This means you can only use it if you run your own domain.
The file itself is a simple text file, which can be created in Notepad or any other text editor. It needs to be saved to the root directory of your site that is the directory where your home page or index page is. Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.
RSSWhat keeps you awake at night?
Do let us know.
We will try our best to deliver tutorials that will help you sleep much better.
The tutorials at have been written by by passionate teachers who have been using open source tools and technologies for years.
Real time images of their development screens are generously used throughout each tutorial.
Download the tutorials to your desktop as PDFs for ready reference whenever you need
Give us your comments or feedback on any of our tutorials.
Your comments help us deliver the best possible training modules.
Is there a specific tutorial you want written? Just Ask.
