HANDBOOK

SEARCH ENGINES

Robots.txt - Hiding files from Search Engines

Having a search engine index your site is great, but not if they are indexing pages that you don't want people to find. There are lots of reasons you might not want a particular file, file folder, or type of files to be searched. You may have sensitive date, like price lists, that you don't want competitors to find. You do have the ability to tell the search engines want pages, files, or folders to skip.

Robots.txt

When the search engine reviews your site it looks for a particular file to see if there are any restrictions for your site. The file it seeks out is the "Robots.txt" file. It's a text file that you place in the root directory of your site. Every search engine will look for it, regardless if you have it or not. You may have even noticed in your site log an error for the file "robots.txt." That's because the search engines will look always look for it.

In the robots.txt file you can indicate what files or folders are off-limits to indexing. If the file is properly written, then the search engine will skip those off-limit files.

The robots.txt file will not improve your search ranking or make the search engines visit your site more often. It's just a basic courtesy the search engines extend to you, to let them know what you don't want on listed on the search engines.

What does a robots.txt file look like?

There are basic guidelines for writing a proper robots.txt file. Be sure you pay close attention to these guidelines, as a wrong space or character here or there might make it confusing for the search engine...they may read it incorrectly.

Example
User-agent: *
Disallow: /cgi-bin/
Disallow: fruit.html

This example will tell all search engines that they should not index the "/cgi-bin/" directory and not to index the "fruit.html" file.

What do these commands mean?

Here are basic guidelines for using these commands:

"User-agent" is the line where you could tell it what specific search engine you wish to notify.
- You can also use an asterisk "*" which means it applies to all search engines.
"Disallow" is the line where you tell the search engines which files or folders you don't want it to list.
- Each line should have one entry.
- You can add as many disallow lines as you want, there is no limit.
NOTE: This file should be edited in a plain text editor.
- Notepad is a favorite choice.
- Word or other Word Processing programs should not be used.