Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz link explorer is how much reliable and accurate as compared to others link explorer tools like Ahref
Hi.. I need to check backlinks for one of my client website duplicatebills through Moz Pro link explorer tool then its show that linking domain to this website are 90 but when i cross check these links from other tools like Ahref free backlinking tool then its show a different figure (ahref show that linking domain to this website are 103).
Link Explorer | | dbills
I am confused that which one correct. And MOZ link explorer tools in reliable to which extent?0 -
My Website's Links Are Not Showing In Moz Link Explorer
Hello Team, I was analyzing my domain (https://www.amzonestep.com) in link explored but there are many websites that are not showing there. Questions
Link Explorer | | amzonestep
1. Is this reason website's DA is not increasing 2. Is there wrong steps taken by me for this website. Well, I know there are many factors work in increasing DA but what is should do if these kind on links are not indexing in MOZ. I things these are one of the factors my DA is not Increasing. Please save my job guys. The company has sent me an ultimatum mail of 2 months that they will take back the projects from me. Please save my job. Thanks & Regards Aashirvad Kumar moz.png Screenshot-2020-02-19-at-11.48.21-am.png0 -
Does Moz register backlinks from Twitter/Facebook/LinkedIn?
I've noticed that on sites with decent social presence and regular content posts that Moz isn't showing/recording any inbound links from Twitter, Facebook or LinkedIn. Posts are regularly shared via these three channels but no inbound links ever seem to show up. Does Moz not record such inbound links, or are they all disguised using JavaScript or some other trick?
Link Explorer | | JCN-SBWD0 -
Why moz and ahrefs reported a remove link that is exist?
Hi in last few days I checked my website in moz link explorer and ahrefs and it showed lots of my backlink as removed! But when I checked this on the browser its OK is there any problem with these tools? Anyone has this problem? Thanks
Link Explorer | | namibiagonzo0 -
Fastest way to get new backlinks recognised by Moz?
Hi all, I have seen a few discussions / advice pieces around this, but wondered what the current view is on the fastest way to get new backlinks recognised by Moz? I have seen people say that the content needs to be shared (e.g. on Twitter), and advice around creating Bitly links. I am not convinced that these techniques are working. On a related note, my website is very large. I have around 2.5m pages. Most of these pages are 3-4 clicks from the home page, but I have a small amount of important content much higher up in the site (1-2 clicks from the home page). Does Moz crawl pages nearer the home page more frequently, or does it make a pass of all pages on the site before it starts at the beginning again? If it makes a complete pass of all pages, then presumably making my site smaller will help to get Moz to see new content more quickly? Is there any other value to be gained in making a large site smaller? Many thanks in advance, Neil.
Link Explorer | | NHARRIS10 -
Does using Sucuri block Moz?
I am using Opensite explorer on my site which uses Sucuri and it shows 0 for everything. Does Sucuri stop MOZ from reading the link? I also suspect that using Sucuri has made my SEO suffer because the first page is always saying "redirecting" . Anyone with experience to this? Thanks
Link Explorer | | seoprojecter0 -
Does Moz have anything similar to ahrefs batch analysis tool?
I am wanting to do analysis of a whole bunch of URLs at once - i know ahrefs already has a very good tool for this - but I don't really want to have to pay the $79 a month if moz has something similar? I know it has the OSE. Can I do batch analysis in this? Thanks
Link Explorer | | SWD.Advertising1
