Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking Keywords are not showing on Moz
i just check my Apk Finds on moz site explorer and check its ranking keywords but it does not show it please help me my site age is 3 months
Link Explorer | | malik12378670 -
Why moz and ahrefs reported a remove link that is exist?
Hi in last few days I checked my website in moz link explorer and ahrefs and it showed lots of my backlink as removed! But when I checked this on the browser its OK is there any problem with these tools? Anyone has this problem? Thanks
Link Explorer | | namibiagonzo0 -
How is Moz DA affected by spam links? Disavow file?
So it does not appear that moz let's you upload your disavow file. So when moz calculates your DA how do spammy links factor in? After digging through our GA it appears our site was hit with the 2016 penguin update and never recovered. Our weekly visitors were 2k, then dropped to 500 and have stayed close to that level for a while. We've used the disavow tool, without success over the past 3 years. During that time we have done link out reach and built around 10 legit good quality DA links since. But we have not recovered. At this point i'm thinking I should just remove the disavow file. Moz says our spam score for our domain is 5%.
Link Explorer | | jessicapremier0 -
If I have a MOZ PRO account, do I still need Screaming Frog?
Hi, I have a MOZ PRO account which is brilliant for me. I am about to move my site from HTTP to HTTPS and was looking for a smooth way to list every page and its details so I can make sure I do the correct redirects etc.. before the switch to SSL. Do I need to get Screaming Frog, or can the tools with MOZ Pro ease my path just as well? Thanks
Link Explorer | | SeoSheikh0 -
Crawling and Links - Showing Links
Hello I have a campaign in Moz that shows a new site as having no domain authority and no links. The site does have links to it, but they don't show up on OSE. Does Moz count links in the comments section of blogs (or is this counted as social, when the commenting system is one that requires facebook login?) I know blog comments are not great for SEO, I just use them when the post/link is 100% genuinely relevant. The site does have actual links on websites not just comment links, so it isn't showing any type of link. What is the best way to see the links the site has? Thanks!
Link Explorer | | wearehappymedia1 -
Does Moz have anything similar to ahrefs batch analysis tool?
I am wanting to do analysis of a whole bunch of URLs at once - i know ahrefs already has a very good tool for this - but I don't really want to have to pay the $79 a month if moz has something similar? I know it has the OSE. Can I do batch analysis in this? Thanks
Link Explorer | | SWD.Advertising1 -
Why doesn't my site show up in Moz Site Explorer
I started this website about 2 months ago, www.guyetteroofing.com. I know it still needs a lot of work, but I can't find any information on it on the site explorer tool.
Link Explorer | | billyguyette1 -
The difference between the statistics MOZ & Ahrefs
Hello dear. I have a website for a year. I made 10K links in one year(from 45 Root Domains).the Ahrefs.com has show full list of link root domains and pages. the ahrefs rank is 49. but MOZ said my Established Links are 5! So I have some questions, please help me: 1- what is difference between MOZ rank and ahrefs rank? 2-When Just-Discovered Link Calculate for my site? 3-why my moz trust is 0.00!?How to improve it? URL: https://esta-register.org
Link Explorer | | allyunit0