Скачать или смотреть Robots.txt Formatting | Affilorama

Robots.txt Formatting | Affilorama | In this lesson we will learn robots.txt formatting that you can use on your website to get more traffic. We will also talk about what the robots.txt file is, why you need one, and how to make your own.

View lesson notes: http://affiloramablog.com/seo/robots-...

When your site is indexed by the search engines, it's crawled by the search engine spiders Google Bot, Yahoo Slurp, and Bing Bot. The spiders crawl your site looking for all the content you have on your site so they can help other people find it.

The spiders will automatically index everything they find on your site. The spiders don't know that you have a section of your site that's only for personal pictures, or that you don't want certain pages on your website popping up in the search engine results (for example, past sale offers you have posted about).

What is robots.txt?
Robots.txt is a very small document that exists in the root directory of your website. The robots.txt formatting tells the so-called “robots” that are crawling your site which pages they are allowed to access, and which pages they are not allowed to access.

The first thing a robot web crawler spider thing will do when it visits your site is it will look for your robots.txt file. Usually the spiders will honor your requests and stay off of pages that you’ve marked not to be indexed.

How to make a robots.txt file
Robots.txt formatting starts with you deciding which parts of your website you want the spiders to index, and which pages you don’t want them to index. Also decide if there are any specific bots that you don’t want crawling around your site at all.

Create a new text document and save it as “robots.txt” on your hard drive. Then write the following robots.txt formatting code into your new text document:

To block all spiders from indexing your entire website:

User-agent: *
Disallow: /
To allow all spiders to see all of the content on your site:

User-agent: *
Disallow:
To block all spiders from certain directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /personal/
Disallow: /photos/staffchristmasparty/
To block a certain spider from your entire website:

User-agent: Googlebot
Disallow: /
To allow only a certain spider, and block all others:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /
Helpful hints
Start a new line for each instruction you include in your robots.txt file. A blank line signals the spiders that a new set of instructions is about to be presented.

If you are familiar with other forms of coding, you might be thinking that the asterisk used in the User-agent field is being used as a wildcard value. This is not the case, and the asterisk cannot be used as a wildcard. You cannot use “* GIF” to tell the spiders to disallow all GIF images, for example.

Make sure to save your changes after you’ve selected which commands to include in your robots.txt formatting.

You must name the file robots.txt in all lower-case.

Make sure to upload your robots.txt file to the root directory of your website, ie, yoursite.com/robots.txt

The spiders won’t find your robots.txt file if it is not in your root directory!

Robots.txt Formatting
http://vimeo.com/affiloramablog/robot...

Affilorama
   • Affilorama | Affilorama Free Affiliate Mar...  

Affilo Blueprint
   • AffiloBlueprint | An Inside Look at the Ne...  

Robots.txt Formatting | Affilorama