Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawler was not able to access the robots.txt
I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?
Getting Started | | andrewrench0 -
Unsolved Is there anyway to cancel moz local?
Is there anyway to cancel moz local? I can not figure it out. I've tried for hours now.
Getting Started | | SouthBayWebs0 -
How I can increase DA of my site?
Hi, I have my 4 month old blog, PA of site is 17 but DA is still 5. I don't know how to increase DA of site. Please suggest me how to increase DA of Site https://myeasygrader.com/ . Thanks
Getting Started | | markwillson0 -
How to rank website on google with help of Moz ?
I am new here and my question is how rank a website with moz because my website was not ranking in any keywords so plz guide me
Getting Started | | RoyDigitalWorld0 -
Can Moz Monitor a JS Site?
We are building a new site that, on the blog landing page, uses JS to populate the individual blog article links on the page. The links are not viewable in the page source, but do appear once the page fully renders. After running an on-demand crawl of the site (in QA...not indexed yet), it appears that Moz isn't indexing these pages, and it also isn't reading other page elements that load later (like an H1 that is rendered in JS but not in page source). Are we going to be able to use Moz to track this site? Is there some setting to help?
Getting Started | | Rodrigo-DC0 -
Has anyone purchased the MOZ SEO courses? Are they good?
I'm looking to learn more about SEO. Has anyone purchased the courses available through MOZ? If so, are they useful?
Getting Started | | Stevepair0 -
MOZ point
I don't know why, but since a week ago I'm not receiving moz point for my activity on moz forum. Example
Getting Started | | Roman-Delcarmen
Today I posted 3 answer in the Question section but in my moz profile does not show the 3 moz point that normally I receive for that. I week ago suddenly I received 20 moz point, why I dont have any idea, maybe someone mark one of my asnwer as good answer. So my point is where I cant found the exact tracking record of my activity1 -
What is the IP address of the MOZ crawlbot?
Our preproduction site is restricted to a specific IP range. Prelaunch I'd like to use MOZ to check the site but this obviously isn't currently possible. Is there an IP range I can whitelist ?
Getting Started | | mrphome0