Robots.txt, Your Wish Is My Commend

Well, robots.txt is the way for webmaster or blogger to tell the search engine which things that can be indexed and which things is best leave it unindexed.

This is getting a bit technical here, if you have a wordpress blog, just skip it and download our robots.txt will do. But wait! before you do that you must know where to put the robots.txt.

Where to put robots.txt?

URL: www.yourblog.com/robots.txt

upload your robots.txt using ftp program to your top level folder of your blog just like showing in the URL link above. You can also create the robots.txt using your web hosting CPanel file manager to create a new file name: robots.txt. And copy the configuration below to your blog.

Download our robots.txt

Robots.txt Configuration Explain

Here we explain a bit about what robots.txt configiration will do

User-agent: *
Disallow: /cgi-bin/

first line “user-agent: *” mean that you allow all the search engine crawler or robot to index. The second line “Disallow: /cgi-bin/” mean that you disallow the search engine to your /cgi-bin/ folder.

Warning! So never do that

User-agent: *
Disallow: /

This will block all your content or blog from the search engine. Remove the “Disallow: /” in there if you have such a configuration.

MORE to come

To allow all robots complete access

User-agent: *
Disallow:

Or create an empty “/robots.txt” file.

To exclude all robots from part of the server

User-agent: *

Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/

To exclude a single robot

User-agent: BadBot
Disallow: /

To allow a single robot

User-agent: WebCrawler
Disallow:
User-agent: *
Disallow: /

Our wordpress robots.txt configuration:
Copy and paste in your robots.txt below, remember
This only work for wordpress!

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$

User-agent: *
# disallow files in /cgi-bin
Disallow: /cgi-bin/

#disallow all files in /wp- directorys
Disallow: /wp-*/

# disallow all files ending in .php
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$

# disallow any files that are stats related
Disallow: /stats*

# disallow wordpress specific folder
Disallow: /category/uncategorized*
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/

————————————————————–

Well we hope now you understand robots.txt better, but if you need more explaination, visit the URL link below to find more information about robots.txt

http://www.robotstxt.org/

Share This

If you enjoyed this post, consider subscribing to our hoobin full feed RSS

or add us into your technorati favor

[Go Back]