How to get moz to crawl a staging domain that is blocked by robots.txt
-
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
-
Hi there!
I'm going to go ahead and marks this as answered since we took care of this in the support queue. For any others that are curious though, You can always add allow/disallow directives for Roger in your robots.txt. I recommend taking a look at our Help Hub for more info
Thanks!
Sam
Moz Helpster
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Getting Started: Multiple Campaigns VS Single Campaign
I'm new to Moz and am trying to determine whether to create multiple campaigns broken out into product categories or a single campaign and label keywords by product category. I figure the majority of the folks in the Moz Community are in agency space, so most probably prefer a single campaign per client. We're an in-house marketing team so we have the ability to track up to 10 campaigns but only need to track a single website. I like the idea of having the different campaigns for different product categories, but I don't want to make things overly complex either. Just because you can do something doesn't necessarily mean you should.
Getting Started | | DFrey1 -
Moz campaign for a subdomain with no home page
Hello all, I'm trying to setup a campaign in Moz for one of our subdomains that we use for marketing landing pages. Unfortunately there really is no home/landing page for the subdomain so when I enter it into the campaign settings it says the site isn't accessible. Is there anyway for me to add this? I tried entering the subdomain with a star, e.g. pages.mydomain.com but it would not accept it. Any ideas? Thanks!
Getting Started | | Brando160 -
Why is Moz.com saying that none are linking to www.oneworldcetner.eu
It still says 0 after 6 weeks if i use other tooks like http://openlinkprofiler.org/r/oneworldcenter.eu#.VhbF7ivzJ8E I do see the backlinks that i was counting on was there
Getting Started | | onewordcenter0 -
Error Selecting Search Engine During Moz Campaign Set-Up
Hi, I'm setting up a Moz campaign for a new client, and when I get to the third step, "Choose the search engines for which you would like to track rankings," there's nothing to select. No drop down, or anything, just a blank field under the heading. If I try to advance, I get an error pop-up that says I must select a search engine. Has anyone else run into this problem? What should I do? GlWPAsc zAmzqFf
Getting Started | | tommyKPseo0 -
What is the best use for Moz tools specially keyword difficulty for startup ?
Hello, I'm so new in Moz and SEO world and i just started my website, a WordPress blog, I'm in a content creation period and i want to make it right from the beginning but I'm confused about how to use Moz tools in this period because i don't have content or traffic so no analytic as i think, so What is the best use of Moz tools in this period? About keyword difficulty tool i think this is the most tool i will use in the beginning, how i choose which keywords to use from my keywords list, in this time I'm depending on the on page SEO only, no backlinks no social engagements, which keywords to use to appear fast in search engines for a startup? less than "% difficulty " or between, I"m new in this word Please Moz and SEO experts give me a hand here. Note: I'm using Medium Moz pro plan.
Getting Started | | Romekio1 -
Google + Business Page and Moz Pro
Is there a way to connect my Google + Business page with Moz? The only option I have available to connect when I press the button is my personal Google + Page. Thanks!
Getting Started | | BenhardtConstruction0 -
Fastest way to set up Moz to produce a list of easy-to-rank for keyword opps
I've used Hubspot's keyword tool previous to MOZ, and I know that they import data generated from MOZ into their tool... but I find it a much more convoluted process using Moz's more robust system. ALL I want is to dump in a long list of keyword possibilities, then see where we can easily rank for these terms, especially against our local competitors. How do I set this up???? It's not as self-explanitory as Hubspot! Thanks!!!
Getting Started | | uptongirl340 -
MOZ Starter Crawl not happeneing
Hi I added a new site 48hours + ago and the starter crawler has not even begun collecting data. Any help would be appreciated. cheers Isaac
Getting Started | | sodafizz0