Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to SEO? Check out Moz Academy!
Easy-to-access SEO education has always been dear to our mission. Simply log in with your Moz Profile and start learning. Whether you're new to SEO or looking to take your skills to the next level, Moz Academy has fun, affordable courses for the entire team! Check it out now! moz-academy-training.png
Getting Started | | HayleyBowyer2 -
How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?
I have a new site that has all the bots turned off so google can't index my site until I'm finished it. I've been working on this site for a couple months now optimizing and I was wondering if there was anyway I can run a preliminary scan on the site for my titles, URLs, Headers, Alt Tags and pretty much anything else that will grade my work and tell me if i did anything wrong? Can MOZ do this with the Bots turned off? Thanks
Getting Started | | Inframan0 -
Why is analytics.moz.com showing up a traffic source in Google Analytics?
Under Traffic Sources, analytics.moz.com is listed as a source for 37 sessions this past month. What might this be? Is it legitimate traffic or should I be filtering this?
Getting Started | | mphdavidson0 -
Site traffic vs other sites
I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.
Getting Started | | trumpfinc0 -
How do get Moz to spider a Development site PRE LAUNCH?
Hi, Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it? Thanks
Getting Started | | bjs20100 -
Getting Redirect Loops in MOZ using Chrome
Been getting bizarre Redirect Loops from Chrome after I log-in to MOZ. Has anyone had something like this happen? I've tried clearing cache, rebooting, etc. but no luck. Thanks in advance!
Getting Started | | danny.wood1 -
Google Analytics data not loading into Moz
I use several analytics apps on my iphone which require you to connect to your GA account. Each one gets immediate access to the GA data. With Moz, I connected to my GA account 3 days ago after many failed attempts due to some bug, but still have no data showing. Is there some reason why Moz would take several days to access the data where iphone apps take seconds? I am now a week into my 30 day free trial, and have spent lots of time trying to get it to work, but have not been able to do much. I'm afraid of how much of the 30 days will be taken up just getting Moz set up and working, rather than actually using and evaluating what it does. Any possibility of re-starting the 30 days from when it's actually working? I don't mind waiting patiently for some bugs to be fixed, just would like to know that I will still get a 30 day trial. Thanks.
Getting Started | | RLodato1