Unsolved Moz crawler not working
-
Hi
Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt
can the support team help ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved /%25s
Hi Community, has anyone else had a 404 error reported by Moz, where the end of the domain is /%25s? The error comes from my blog home page https://kaydee.net/blog/ But when I look at the source code, I can't see anything that has a space at the end of the URL. I wonder if it is to do with the WordPress search? Thanks in advance for any insight.
Moz Pro | | kaydeeweb0 -
Unsolved Performance Metrics crawl error
I am getting an error:
Product Support | | bhsiao 0
Crawl Error for mobile & desktop page crawl - The page returned a 4xx; Lighthouse could not analyze this page.
I have Lighthouse whitelisted, is there any other site I need to whitelist? Anything else I need to do in Cloudflare or Datadome to allow this tool to work?1 -
Screaming frog, Xenu, Moz giving wrong results
Hello guys and gals, This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results. For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent). The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data. I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it? Many thanks in advanced for any help,
Moz Pro | | GPainter0 -
My moz campaing not update completely
I have create new account and campaign last week, till now my moz campaign not update like: keywords ranking, competitor websites links etc. Why it is so please update me.
Moz Pro | | Surabhi_Dewra0 -
Moz metrics
This discussion is strictly theoretical... I won't hold anyone to their answer. If I have 2 websites that are identical in every way and let's say the domain authority for both is 40, and I 301 redirect one site to the other, what would the DA become? Same question for single pages, both with a PA of 40. If I 301 redirect one page to the other, what does the PA become for the remaining page?
Moz Pro | | AMHC0 -
Is it normal for Moz to report on nofollow pages in crawl diagnostics?
I have a dev version of my website, for example, devwww.website.com. The htaccess page has a noindex and nofollow request, but I got crawl issues reported from these pages in my Moz report. Does this mean that I don't have the development site hidden from search like I thought I did?
Moz Pro | | houstonbrooke0 -
My home page has an "A" rank in Moz and not ranking in Google.
My home page recently dropped from page one in Google to not being ranked for my top keyword. The page has an "A" ranking in MOZ for the keyword. Is there a way to find out the cause. I did have what looked like a duplicate page for a while 3 months ago when a domain was forwarding to my home page incorrectly. Appeared with second domain name instead of primary. Our business has been 95% through internet leads so quite an issue. Is there anyway to find out what is going on.
Moz Pro | | FredRoven0 -
Seo Moz for Clients
One thing I recently pondered following the introduction of logos on reports bringing that level of personalisation for the customers is ... would it be somehow possible to take it a step further and have a login on your own personal site that could somehow enable clients to download and look at the information as and when they wanted. An almost mini sep moz for clients where customers login through your own site ?
Moz Pro | | onlinemediadirect0