Latest posts made by superseopl
-
RE: Moz crawling bot
Thank you Keri, problem solved.
posted in Link Explorer
superseopl
Mar 19, 2014, 2:00 PM
-
Moz crawling bot
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
posted in Link Explorer
superseopl
Mar 19, 2014, 11:20 AM
Hi guys, my name is Piotr and I'm the founder of SuperSEO, leading SEO agency in Poland.
I'm here to talk some digital marketing. Drop me a message and we'll take it from there.
Cheers!