Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler
-
I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site
At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot.
What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines
I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says
a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content- But this way Moz will also not be able to crawl the site
b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.
c) Use a
noindex
meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tagPassword Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz bot has trouble crawling Angular JS - I believe it's seeing the SPA (Single Page Application) before Universal. Anyone else have this issue? What is the fix?
The Moz bot user agent detection settings are able to read Universal, but the Single Page Application (SPA) version partially loads on the website before Universal. Because of this, Moz (and possibly search engines) think we have massive duplicate content issues. For example, the crawl report said a particular product page (which has about 1,000 words) has 33,000 words and has duplicate content with over 300 other pages. This makes me believe it's only picking up the SPA version. Has anyone come across this, and what would be the fix?
Moz Bar | | laurengdicenso1 -
Moz DA is not increasing
Hey my website best offset smokers under $1000 is not increasing DA. is there anyway to increase the DA currently its DA is 8
Moz Bar | | 0hjgh60 -
Is there anyway to track on-page grader grades for a site?
As opposed to going through the process of inputting the urls and keywords every couple of weeks, is there a way of Moz to send the scores for those keywords and the domain to which the url belongs?
Moz Bar | | DigitalGuru0 -
All in favor of an "Announcements" category section say MOZ!
So this is just a thought. I am not Rand, therefor I am not the Moz man himself, and therefor I am not telling anybody how to run this website or forum. But, I've been thinking... and usually that a dangerous thing. So many times when I answer a question I end it with: "Keep me posted" "Keep us updated" "Good Luck" In the same light there have been plenty of times when I have had my questions answered by people like EGOL, RobertFIsher, KeriMorget and they have said that they have wanted me to keep them posted. Now sure, some of you are sitting there being Melancholy Mozzers thinking that all those wonderful people are just being polite by adding some nice parting words... but what if I they are being Sincere Seoers? What if I WANT to fill them in and don't want to ask a question? What if... THEY REALLY LIKE ME?!?! I could also send a DM to those 3 people, I mean let's get honest I'm not THAT popular.... but I can dream can't I? Again, this isn't really an SEO question, more of a thought on where I'm supposed to give updates on my project. In all honesty I do want feedback on what I'm doing, and sometimes I don't even know what questions I'm supposed to ask!
Moz Bar | | HashtagHustler1 -
Moz Local | Empty page "Categories"
Dear Moz, Another error, the following url loads an empty page https://moz.com/local/categories Please review Thanks!
Moz Bar | | Bio-RadAbs0 -
Does Moz Analytics Allow for Goal Tracking?
We're currently tracking goals in GA, but would be interested to learn if that can be accomplished in Moz Analytics.
Moz Bar | | CGalownia0 -
Is there a problem with New Campaign crawls being slow?
Hi I've created 3 new campaigns but results not showing as usual. I know the full results take up to 7 days but I usually see some basic info in a few minutes of creating the campaign. This problem happened a while ago and the campaign was never quite right in SEOMoz. Thanks for your help Steve
Moz Bar | | stevecounsell0 -
Moz showing warnings for each dynamic link despite canonicalization?
As you can see in the attached image, Moz is showing a warning for each dynamic URL despite a rel=canonical tag. Is this by design? If so, it is frustrating seeing as it is really just the one page with many links . . . 9h5oDmr.png
Moz Bar | | BlueLinkERP0