How to get moz to crawl a staging domain that is blocked by robots.txt
-
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
-
Hi there!
I'm going to go ahead and marks this as answered since we took care of this in the support queue. For any others that are curious though, You can always add allow/disallow directives for Roger in your robots.txt. I recommend taking a look at our Help Hub for more info
Thanks!
Sam
Moz Helpster
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to SEO? Check out Moz Academy!
Easy-to-access SEO education has always been dear to our mission. Simply log in with your Moz Profile and start learning. Whether you're new to SEO or looking to take your skills to the next level, Moz Academy has fun, affordable courses for the entire team! Check it out now! moz-academy-training.png
Getting Started | | HayleyBowyer2 -
Moz Pro for Local Catering Company?
Is Moz Pro or Moz Content useful for a local catering company? I've started a trial to try to determine if it will benefit us. Our new site will be up next week so we don't currently have traffic. We will be using some Adsense and direct mail campaigns to get started, but not sure if Moz local will be sufficient. Please help connecting the dots. I know it would be good to rank higher for example "Catering Tahoe, Ca." But how would Moz help me see what kind of content to create for a local catering company? I'm not sure a blog article makes sense.
Getting Started | | Nasoj0 -
How can i start in moz?
I want to know what to do first, how do I start the branding, etc. Thanks!!!
Getting Started | | Gridiron2361 -
How do i stop Moz from indexing my dev site?
Hey, I am new to MOZ, and I am seeing in my page reports that I have duplicate content, but this duplicate content is mostly my development site. dev.domain.com. I have this blocked in robots.txt for google. How do i stop MOZ including it in its reports? Cheers everyone.
Getting Started | | Tholomew1
Bart0 -
Can you help me with setting up a campaign? I get stuck after I put in the domain name and the other information doesn't go through.
Can you help me with setting up a campaign? I get stuck after I put in the domain name and the other information doesn't go through. Regards Roshni M
Getting Started | | harmanseo0 -
Moz Pro Trial...."Oops! Try refreshing the page"
I am in my MOZ PRO trial. I want to add a campaign to see how I like it, but something keeps coming up saying "Oops! Try refreshing the page, if that doesn't work click here to contact our help team" And I can't find an answer to that question here. Please help ASAP Ld3rHLi
Getting Started | | WFC-Sales0 -
Moz Analytics - Custom Report Question
Hello Moz! I haven't been here since the days of SEOMoz, but it's great to be back and i'm really pleased to see lot of shiny new tools at my disposal, but I do have a question in regards to functionality. If I have, for example, 15 clients in my dashboard and would like to run the same report for all of them. Assuming that I just finished creating a custom report for my first client, how can I duplicate this report without having to re-create this report 15 times? Additionally, while I am here. I have noticed that when I add a campaign, the last screen never finishes loading and the window seems to freeze. A quick refresh fixes this, but I thought i'd flag it 🙂
Getting Started | | Paul_Tovey0 -
Google Analytics data not loading into Moz
I use several analytics apps on my iphone which require you to connect to your GA account. Each one gets immediate access to the GA data. With Moz, I connected to my GA account 3 days ago after many failed attempts due to some bug, but still have no data showing. Is there some reason why Moz would take several days to access the data where iphone apps take seconds? I am now a week into my 30 day free trial, and have spent lots of time trying to get it to work, but have not been able to do much. I'm afraid of how much of the 30 days will be taken up just getting Moz set up and working, rather than actually using and evaluating what it does. Any possibility of re-starting the 30 days from when it's actually working? I don't mind waiting patiently for some bugs to be fixed, just would like to know that I will still get a 30 day trial. Thanks.
Getting Started | | RLodato1