Comments on: Getting to Know Your Bots: Robots.txt 101

By: ashok

ashok — Tue, 17 Jul 2012 12:55:34 +0000

Hello Rhea,

Thanks for great post, Can you please help me regarding the following i have some dynamic pages how should i make my robots.txt file so that search engine does not read these pages:

http://www.abc.com/index.php?option=com_virtuemart&page=shop.pdf_output&showpage=shop.product_details&pop=1&output=pdf&product_id=34&category_id=1&pop=1&vmcchk=1&Itemid=2

Is this fine:
User-agent: *
Noindex:/index.php/*? option=*

Thanks for your help in advance…

Regards
Ashok

By: Lawrence

Lawrence — Fri, 18 May 2012 14:21:06 +0000

While some search providers actually read and respect the robots.txt file, _not all do_.

A recent example I found was Bing.
We found it was explicitly indexing items that were excluded. Not only that, but it was taking substantial bandwidth (over a period of time) sucking up disallowed content using multiple simultaneous ip connections.

A check of the IP ranges used verified that they were indeed owned by Microsoft.
We eventually blocked the ranges in use by Bing for abuse.

More here –
http://www.computersolutions.cn/blog/2012/05/msn-bing-crawler-spider-madness/

By: Jason

Jason — Fri, 04 May 2012 10:25:17 +0000

I was unaware of robots.txt until around 2 weeks ago when I found Google was indexing my test site! Thankfully I’m now a robots due to various research ad this post I’m now a robots guru!

By: Halvorsen

Halvorsen — Mon, 23 Apr 2012 20:19:36 +0000

My favorite :

User-agent: *
Crawl-delay: 10

By: Wissam Dandan

Wissam Dandan — Mon, 23 Apr 2012 18:21:35 +0000

its been there for a while and kinda funny ( last three disallows)

http://www.last.fm/robots.txt

By: Rhea Drysdale

Rhea Drysdale — Mon, 23 Apr 2012 18:18:08 +0000

Amen. (not to go remove my credit card numbers from my robots.txt…)

By: Alex Czarto

Alex Czarto — Mon, 23 Apr 2012 18:12:52 +0000

Important note to add: Be aware that your robot.txt file is public for anyone to view. So don’t rely on robots.txt to hide sensitive information from view, because if you have a malicious user visiting your site one of the first places he will look is your robot.txt file to see if there are any sensitive or vulnerable files listed there.

ie: If he sees this

Disallow: /secret/important-hidden-secret-stuff.html
Disallow: /secret/credit-card-numbers.txt

…and you haven’t taken precautions in securing those files, then you may be in trouble.

By: Rhea Drysdale

Rhea Drysdale — Mon, 23 Apr 2012 18:08:40 +0000

Mark, great point about snoopers. Most 101 types wouldn’t be too worried about that, but absolutely when it comes to competitive fields and sites. LOVE SEER’s and Rishi’s robots, thanks for including them. And, it’s always hilarious to see a robots.txt file indexed. Like ours! Let’s fix that. :D

By: Mark Ginsberg

Mark Ginsberg — Mon, 23 Apr 2012 17:48:20 +0000

Thanks for the post. As netmeg pointed out, to prevent pages and folders being indexed, I’m much more a fan of the noindex meta tag. It’s also less of a tipoff to some pesky SEO looking for secret internal info. I love visiting a robots.txt file and seeing stuff the site wanted to hide – they’re pointing you right to it!

I was actually doing some research and putting together some fun robots.txt files with cute comments in them, an idea I had after the whole SEER rankings fiasco. After checking their robots file, they’ve got a finally line in there – it’s worth checking out – seerinteractive.com/robots.txt. Probably the best one I’ve seen is Rishi’s at explicitly.me/robots.txt. I love how these files themselves are indexed, and you need to find a workaround to prevent them from being indexed themselves.

Anyway, thanks for the post – will definitely reference it when I write about fun robots.txt files.

UPDATE FROM RHEA: Mark also shared this on Twitter: http://www.seomoz.org/robots.txt

By: Rhea Drysdale

Rhea Drysdale — Mon, 23 Apr 2012 16:35:43 +0000

YES! Robots.txt has so many caveats. There should be a giant checklist of “warning!”