Welcome to the Q&A Forum

Thanks Cyrus,

No, for some reason the editor double-spaced the file when I pasted. Other than that, it's the same though.

Yes, I actually tried ordering the exclusions both ways. Neither works.

The robots.txt checkers report no errors. I had actually checked them before posting.

Before I posted this, I was pretty convinced the problem wasn't in our robots.txt but the Seomoz support staff says essentially, "We don't think the problem is with Rogerbot, so it must be in your robots.txt file, but we can't look at that, so if by some chance your robots.txt file is fine, then there's nothing we can do for you because we're just going to assume the problem is on your side."

I figured, with everything I've already tried, and if the fabulous SEOMoz community can't come up with a solution, that'll be the best I can do.

Thanks so much for the tip. Unfortunately still unsuccessful. (shrug)

Hi guys,

We're trying to block Rogerbot from spending 8000-9000 of our 10000 pages per week for our site crawl on our zillions of PhotoGallery.asp pages. Unfortunately our e-commerce CMS isn't tremendously flexible so the only way we believe we can block rogerbot is in our robots.txt file.

Rogerbot keeps crawling all these PhotoGallery.asp pages so it's making our crawl diagnostics really useless.

I've contacted the SEOMoz support staff and they claim the problem is on our side. This is the robots.txt we are using:

User-agent: rogerbot

Disallow:/PhotoGallery.asp

Disallow:/pindex.asp

Disallow:/help.asp

Disallow:/kb.asp

Disallow:/ReviewNew.asp

User-agent: *

Disallow:/cgi-bin/

Disallow:/myaccount.asp

Disallow:/WishList.asp

Disallow:/CFreeDiamondSearch.asp

Disallow:/DiamondDetails.asp

Disallow:/ShoppingCart.asp

Disallow:/one-page-checkout.asp

Sitemap: http://store.jrdunn.com/sitemap.xml

For some reason the Wysiwyg edit is entering extra spaces but those are all single spaced.

Any suggestions? The only other thing I thought of to try is to something like "Disallow:/PhotoGallery.asp*" with a wildcard.