How to get moz to crawl a staging domain that is blocked by robots.txt
-
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
-
Hi there!
I'm going to go ahead and marks this as answered since we took care of this in the support queue. For any others that are curious though, You can always add allow/disallow directives for Roger in your robots.txt. I recommend taking a look at our Help Hub for more info
Thanks!
Sam
Moz Helpster
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happened to Moz perks?
Not sure if it completely disappeared or just moved somewhere else. Would be curious to hear!
Getting Started | | Joe.Robison2 -
High total links, but very few root domains?
Hi Moz community!I've just joined and am getting to grips with SEO basics. Right now, I'm looking at the Competitive Link Metrics in Moz Pro, and I'm curious about the following- Of the three competitors that we're following, I'm trying to figure out some differences between two of them - we'll call them A and B. 'A' has 3.6k external followed and total links, with 5 total linking root domains. 'B' (a more prestigious and established company with a much higher DA) has 2.2k total external links, with 180 root domains. So my question is, how can A have nearly 1,000 more links, but only from 5 domains? Any feedback much appreciated! Thanks!
Getting Started | | thegildedteapot0 -
Moz scraper
How often do you Moz do whatever it is they do for me to get up-to-date data?
Getting Started | | infinety0 -
What Moz tool is best to find reasons google has not spidered by site
I just joined Moz and am trying to use the tools however, when I attempt to do so every link comes to a that only allows me access to post questions here. If anyone can tell me what tool is best to find reasons google has not indexed my site, I would greatly appreciate the help. Also if anyone knows why I am keep getting routed to this forum when I try to use any of the tools, I would also appreciate help with this. So far Moz is very frustrating.
Getting Started | | Johndeeray19640 -
I'm setting up a new campaign and getting the error "This does not appear to be a valid URL" What's wrong?
I've tried multiple times (over 2 days) with every variation of the URL with no luck. Any ideas for why the URL does not seem to be working?
Getting Started | | jgrammer0 -
New to MOZ and working with Web Mentions. Can I use operators?
Our name is HostDime but often put as Host Dime (2 words) by news sources and other sites. How do I set up my brand mention so I only get a notice when both words appear, in order, together. I don't want "That host is a dime" and such. Can I use a +Host +Dime?"Host Dime"? Do these operators work in MOZ?
Getting Started | | hostdime0 -
Is there a difference between Moz Pro and Moz Analytics?
I thought that Moz Analytics was now included with Moz Pro. Is this true? I have Moz Pro and I was using the beta version of Moz Analytics back in October. When I log into Moz Analytics, the latest data stops at October 27th and there is no data after that. Is that an indication of when my trial period ended? I have the basic Moz Pro, does that include Analytics portion? Thanks!
Getting Started | | Symmetri0 -
Moz Analytics - Reports
In the guide video (http://moz.com/help/guides/getting-started) a report button is show. However, I cannot see this button in Moz Analytics. I'm assuming this is due to the tool being in beta?
Getting Started | | David_ODonnell0