Configure robots.txt for WordPress blog to avoid duplicate content penalty

Posted By: KP On June 26, 2009 Under Blogging, Google, Internet, SEO Optimize Tips, Wordpress Categories

robots.txt file defines how robots should access folders of your server.The file should stay in root folder.

WordPress can list a single post many ways.On home page,permalink,category wise,author wise.For search engines they are different URLs.So they will give you duplicate content penalty.

So you should write your robots.txt file as follows:

User-agent:  *

# Disallow all directories and files within important directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
# Disallow all files ending with these unnecessary  extensions
Disallow: /*.php$
Disallow: /*.inc$

Disallow: /*.css$

Disallow: /*.js$

# Disallow single post feeds, categories and trackbacks
Disallow: */trackback/
Disallow: /category/*

Disallow: */feed/

My current robots.txt file : http://blog.kpsofts.com/robots.txt

Don’t give 777 permission to robots.txt file.Always monitor robots.txt from Google webmaster tools.

Add your upload directory if you don’t want to list your image and other files that you have uploaded.It will block it for image search too.

If you want to add sitemap URL in robots.txt as it as below

Sitemap: your site map file link

For example:

Sitemap: http://blog.kpsofts.com/sitemap.xml.gz

Share with Friends ( Sharing is Caring! )


Subscribe For E-mail Notification Of Article Updates

Enter your email address:

NOTICE:
DO NOT FORGET TO CLICK ON THE VERIFICATION LINK AFTER SUBSCRIBING,THE VERIFICATION MAIL WILL BE SENT TO YOU FROM "FeedBurner Email Subscriptions".
( What is this? Why I need to do this?)

Related Posts

« Unexpected PR update- Google Updated PageRank within a month
Send notes that will self-destruct after being read »

Post a Comment

OR