Home

Building

Hosting

Promotion

Make Money

Advanced Tools

Homebusiness

Resources

ONLINE BUSINESS RESOURCES

Affiliated Buisness Blog

Affiliate Program Directory

Affiliate Marketing Web2.0 Directory

Offshore Incorporation

Web Marketing Guides & Tools

Web Promotion Strategies

Webmaster Tools

Keyword Marketing

How To Feed Up Search Engine Spiders?

Creating A robots.txt File

Some people believe that they should create different pages for different search engines, each page optimized for one keyword and for one search engine. Now, while I don't recommend that people create different pages for different search engines, if you do decide to create such pages, there is one issue that you need to be aware of.

These pages, although optimized for different search engines, often turn out to be pretty similar to each other. The search engines now have the ability to detect when a site has created such similar looking pages and are penalizing or even banning such sites. In order to prevent your site from being penalized for spamming, you need to prevent the search engine spiders from indexing pages which are not meant for it, i.e. you need to prevent AltaVista from indexing pages meant for Excite and vice-versa. The best way to do that is to use a robots.txt file.

You should create a robots.txt file using a text editor like Windows Notepad. Don't use your word processor to create such a file.

Here is the basic syntax of the robots.txt file:

User-Agent: [Spider Name]
Disallow: [File Name]

For instance, to tell AltaVista's spider, Scooter, not to spider the file named myfile1.html residing in the root directory of the server, you would write

User-Agent: Scooter
Disallow: /myfile1.html

To tell Excite's spider, called ArchitextSpider, not to spider the files myfile2.html and myfile3.html, you would write

User-Agent: ArchitextSpider
Disallow: /myfile2.html
Disallow: /myfile3.html

You can, of course, put multiple User-Agent statements in the same robots.txt file. Hence, to tell AltaVista not to spider the file named myfile1.html, and to tell Excite not to spider the files myfile2.html and myfile3.html, you would write

User-Agent: Scooter
Disallow: /myfile1.html

User-Agent: ArchitextSpider
Disallow: /myfile2.html
Disallow: /myfile3.html

If you want to prevent all robots from spidering the file named myfile4.html, you can use the * wildcard character in the User-Agent line, i.e. you would write

User-Agent: *
Disallow: /myfile4.html

However, you cannot use the wildcard character in the Disallow line.

Once you have created the robots.txt file, you should upload it to the root directory of your domain. Uploading it to any sub-directory won't work - the robots.txt file needs to be in the root directory.

I won't discuss the syntax and structure of the robots.txt file any further - you can get the complete specifications from http://www.robotstxt.org/wc/norobots.html

Now we come to how the robots.txt file can be used to prevent your site from being penalized for spamming in case you are creating different pages for different search engines. What you need to do is to prevent each search engine from spidering pages which are not meant for it.

For simplicity, let's assume that you are targeting only two keywords: "tourism in Australia" and "travel to Australia". Also, let's assume that you are targeting only four of the major search engines: AltaVista, Excite, HotBot and Northern Light.

Now, suppose you have followed the following convention for naming the files: Each page is named by separating the individual words of the keyword for which the page is being optimized by hyphens. To this is added the first two letters of the name of the search engine for which the page is being optimized.

Hence, the files for AltaVista are

tourism-in-australia-al.html
travel-to-australia-al.html

The files for Excite are

tourism-in-australia-ex.html
travel-to-australia-ex.html

The files for HotBot are

tourism-in-australia-ho.html
travel-to-australia-ho.html

The files for Northern Light are

tourism-in-australia-no.html
travel-to-australia-no.html

As I noted earlier, AltaVista's spider is called Scooter and Excite's spider is called ArchitextSpider.

A list of spiders for the major search engines can be found at http://www.searchenginewatch.com/webmasters/spiderchart.html

From this list, we find that the spider for Northern Light is called Gulliver. HotBot uses Inktomi [http://www.inktomi.com/] and Inktomi's spider is called Slurp. Using this knowledge, here's what the robots.txt file should contain:

User-Agent: Scooter
Disallow: /tourism-in-australia-ex.html
Disallow: /travel-to-australia-ex.html
Disallow: /tourism-in-australia-ho.html
Disallow: /travel-to-australia-ho.html
Disallow: /tourism-in-australia-no.html
Disallow: /travel-to-australia-no.html

User-Agent: ArchitextSpider
Disallow: /tourism-in-australia-al.html
Disallow: /travel-to-australia-al.html
Disallow: /tourism-in-australia-ho.html
Disallow: /travel-to-australia-ho.html
Disallow: /tourism-in-australia-no.html
Disallow: /travel-to-australia-no.html

User-Agent: Slurp
Disallow: /tourism-in-australia-al.html
Disallow: /travel-to-australia-al.html
Disallow: /tourism-in-australia-ex.html
Disallow: /travel-to-australia-ex.html
Disallow: /tourism-in-australia-no.html
Disallow: /travel-to-australia-no.html

User-Agent: Gulliver
Disallow: /tourism-in-australia-al.html
Disallow: /travel-to-australia-al.html
Disallow: /tourism-in-australia-ex.html
Disallow: /travel-to-australia-ex.html
Disallow: /tourism-in-australia-ho.html
Disallow: /travel-to-australia-ho.html

When you put the above lines in the robots.txt file, you instruct each search engine not to spider the files meant for the other search engines.

When you have finished creating the robots.txt file, double-check to ensure that you have not made any errors anywhere in it. A small error can have disastrous consequences - a search engine may spider files which are not meant for it, in which case it can penalize your site for spamming, or, it may not spider any files at all, in which case you won't get top rankings in that search engine.

An useful tool to check the syntax of your robots.txt file can be found at http://www.tardis.ed.ac.uk/~sxw/robots/check/ . While it will help you correct syntactical errors in the robots.txt file, it won't help you correct any logical errors, for which you will still need to go through the robots.txt thoroughly, as mentioned above.

Software to help you create a robots.txt file

Robogen is a software for Windows 95, Windows 98 and Windows NT based computers which can help you create robots.txt files without having to worry about the correct syntax of a robots.txt file. It comes in two editions: the free Limited edition and the Standard edition which costs $12.95. You can get Robogen from http://www.rietta.com/robogen/ .

In this article, I'll tell how you can use Robogen to simplify the process of creating a robots.txt file. This article assumes that you have read my article on what a robots.txt file is and how you can manually create it. If you missed the article, it is available at http://www.1stSearchRanking.com/t.cgi?1060&robots.htm

I would recommend that you download the trial version of the Standard Edition. The Limited edition does not support as many search engines as the Standard Edition.

Once you have downloaded Robogen and installed it in your hard drive, start the program. Here are the steps that you should follow to create a robots.txt file:

Step 1: Click on the Logon FTP Server button. In the dialog box that appears, type in the address of the FTP server for your web site, make sure that the "This server requires me to log in" option is checked, and then type in the username and the password which you use to access the server in the appropriate boxes.

Now, depending on how your web host has set up the server, the files for your web site may be present in the root directory of the server itself. Or, it may be present in a sub-directory of the root directory. In the latter case, click on the "Use specified root" option and type the relevant sub-directory in the text box besides the option. If you don't know in which sub-directory the files for your web site are present, ask your web host to give it to you.

Done? Now click on OK. After a while, Robogen will display the directories and files which are present in your site in the right pane of the main Robogen program window. If you find that Robogen has not displayed the files in your site, it means that you did not enter the correct sub-directory in which the files of your web site are present. In this case, ask you web host to give the correct sub-directory to you.

Step 2: In the left pane of the main Robogen window, click on the small button present to the right of the drop-down combo box at the top. In the menu that appears, select "Popular Search Engines".

Step 3: Now, click on the drop-down combo box and choose the search engine that you are targeting (say, Northern Light ). In the right pane, select a page which is NOT meant for Northern Light. Now click on the "<<<" button. You will find that the file which you had selected in the right pane is now present in the left pane. In this way, click on each of the pages which are not meant for Northern Light and press the "<<<" button.

Repeat step 3 for each of the search engines that you are targeting. Just select the search engine that you are targeting from the drop-down combo box, select the pages which are not meant for the search engine one by one from the right pane, and click on the "<<<" button after clicking on each age. Repeat this process for all the search engines that you are targeting.

Step 4: Upload the file to your server using the "Upload Robots.Txt" button. If you want, you can also use the Save menu item from the File menu to save a copy of the robots.txt file in your hard drive.

Article by Sumantra Roy. Sumantra is one of the most respected search engine positioning specialists on the Internet. To have Sumantra's company place your site at the top of the search engines, go to http://www.1stSearchRanking.com/t.cgi?1060 For more advice on how you can take your web site to the top of the search engines, subscribe to his FREE newsletter by going to http://www.1stSearchRanking.com/t.cgi?1060&newsletter.htm

INTERNET MARKETING TOOLS

Affiliated-Business.com: The art of making money online...

| Home |

| Building |

| Hosting |

| Promotion |

| Make Money |

| Tools |

| Homebusiness |

| Resources |

| Tell A Friend |

| Add To Favorities |

| Link To Us |

| Partner With Us |

| Site Map |