Tutorial

How to create a robots.txt file for your website.

Whether you have a Boonex Dolphin site or any other type of site, and regardless of the company you use for web hosting you should consider creating one.

A robots.txt file helps crawlers and spiders know which part of your domain to find and read and which parts not to read. There are hundreds of agents out there, so it's important to give some direction.

There are a number of crawlers that claim to follow and abide by the rules of your robots.txt file, but there are plenty that simply ignore it.

If there is a particular robot that you don't want visiting your site and pages then you should use .htaccess which can block them completely.

There are many discussions on whether they are that useful or not. It only takes a few moments to create one so it's something to think about and work on in your spare time.

The general concept is that you can tell crawlers or bots such as the Googlebot that you don't want them looking in certain folders/directories, or files on your site. Basically telling them not to crawl them so they won't be indexed. By default the majority will look to see if your site has this file.

Say you wanted to prevent bots from crawling your Boonex Dolphin admin folder / directory (http://www.yoursite.com/admin).

You can create this rule like:
Disallow: /admin/

Or perhaps:
User-agent: *
Disallow: /admin/


Continue adding folder/directories and files you do not want crawled:
User-agent: *
Disallow: /admin/
Disallow: /inc/
Disallow: /langs/
Disallow: /xml/

Or add specific files to it:
User-agent: *
Disallow: /admin/
Disallow: /inc/
Disallow: /langs/
Disallow: /xml/
Disallow: /greet.php

The path is relative to root and must contain a trailing "/"

Simply use any text editor such as notepad and save the file as robots.txt, then upload the file to your root (http://www.yoursite.com/robots.txt).

Continue adding folders/directories and files to suite your sites needs. This is not Dolphin specific it can and should be used for any website.


One of the helpful things you can do is add your sitemap xml location to it. This way when a crawler or bot comes to your site and finds your robots.txt file you can tell it where to find your sitemap.xml containing all your links and web pages. Sort of a little plus or bonus.

Simply add this to the bottom of the file:
Sitemap: http://www.yoursite.com/sitemap.xml

*Obviously you should have created a sitemap.xml first and have it uploaded to your root too (http://www.yoursite.com/sitemap.xml).

So it would look something like:
User-agent: *
Disallow: /admin/
Disallow: /inc/
Disallow: /langs/
Disallow: /xml/
Disallow: /greet.php
Sitemap: http://www.yoursite.com/sitemap.xml


After you have uploaded it there is nothing more you need to do. The bots will find the file all by themselves once they visit your site.

If you haven't created a sitemap for your site you can see this article for more information: Create and Generate Sitemap Files for your Site.

 
Written By
Tutorial by: Jeremy LeSarge (AKA Ray)

I am the owner and administrator of DialMe.com. I write Tutorials for Boonex Dolphin as well as tips and resources surrounding website programming and development. I enjoy working with WordPress, SEO, and Web Hosting / Servers. I also maintain a WordPress Blog here on this site where you will find a variety of technology and webmaster resources.

Actions
Recommend
Sponsored Links