Crawling password protected sites such as dev or staging areas to look at sites b4 going live ?
-
Hi
Ive instructed clients to password protect dev areas so dont get crawled and indexed but how do we set up Moz crawl software so we can crawl theses sites for final check of any issues before going live ?
Is there an option i havnt seen to add logins/passwords for crawl software to access ?
cheers
dan
-
ok thanks Chiaryn
is that the actual name of the moz crawler (to allow in Robots) simply rogerbot ? or any other characters etc ?
Also is it not the case that even when blocked by robots.txt G can still crawl/index it once password removed, think i read few comments somewhere on Moz that can still happen somehow ?
Please advise asap ?
Many Thanks
Dan
-
Hey Dan,
Unfortunately, our crawler is not able to access password protected content on your site. If you create a staging subdomain that is not password protected, you could use the robots.txt file to allow rogerbot and block other crawlers, but I'm afraid our crawler will not crawl anything that a normal search engine crawl would not be able to crawl so we cannot crawl password protected pages.
I hope this helps.
Chiaryn
-
i dont suppose either of you are able to help at all with this related question:
http://moz.com/community/q/site-crawl-errors-download-list-of-all-urls
-
i dont suppose either of you are able to help at all with this related question:
http://moz.com/community/q/site-crawl-errors-download-list-of-all-urls
-
Hi Andy
Screaming Frog does have password access feature for your info i have just tried it
All Best
Dan
-
Thanks Matt
I have got screaming frog and can confirm that it has password access feature, but i really want Moz to be able to access too, i would have thought they should have this option somewhere. Are you saying Moz crawls have more info than SF (re 'moz level' analysis) ?
Dev site better password prtected than robots arnt they i think ?
Cheers
Dan
-
Hi Dan
I was about to ask the exact same question, so will keep an eye out for an answer.
I hope it is possible, but I couldn't work it out.
-
I don't know if there's a way to do this in Moz but you could always get Screaming Frog & tell it to ignore robots.txt - that will definitely crawl it. You can check titles, descriptions, canonicals, H1s, etc. that way. It doesn't give the Moz level analysis but it's a start that def works. You can also see if you have parameter issues that way.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 1-page 301 status error but httpstatus.io says its 403
I am trying to run a site crawl for my website and MOZ is only resulting in 1 page crawled with the home page URL Status Code of 301. However when I run it in httpstatus.io it is giving me a 403 status error. Im curious as to why MOZ is saying its a 301 and httpstatus.io is saying 403. Is there anything I can do in MOZ first to get the site crawled before asking my developers to look into the 403 error?
Moz Bar | | JohnConover0 -
Moz can't crawl my new website?
We had a new website go live at the end of April - I keep requesting crawl tests but I get this in the excel copy... URL Title Tag
Moz Bar | | RayflexGroup
http://www.pvc-strip.co.uk 602 : Page redirects to a URL outside the scope of this campaign. I always list the website as https://... but the crawl always returns the http:// version. Not sure what I can do to make sure the website can be crawled?0 -
I can't seem to get Moz Crawl to run? Re-bootyourbody.com. Told its a subdomain...What do I do?
I can't seem to get Moz Crawl to run? Re-bootyourbody.com. Told its a subdomain...What do I do?
Moz Bar | | Joseph.Lusso0 -
Crawl test csv has lost its formatting??
All the columns/heading merged into column A. Anyone else noticed this over the past few days?
Moz Bar | | Moving-Web-SEO-Auckland0 -
URLS appearing twice in Moz crawl
I have asked this question before and got a Moz response to which i replied but no reply after that. Hi, We have noticed in our moz crawl that urls are appearing twice so urls like this - http://www.recyclingbins.co.uk/about/ www.recyclingbins.co.uk/about/ Thought it may be possible rel=canonical issue as can find URL's but no linking URL's to the pages. Does anyone have any ideas? Thank you Jon I did the crawl test and they were not there
Moz Bar | | imrubbish0 -
Signed up for moz reports - have received Moz error report - need someone who is capable to take report and perform cleanup edits within Joomla site?
Looking for someone in the US - please contact me at Mary@workingwebsolutions.com If available and interested in task. Thanks Mary
Moz Bar | | PortlandWebDesign0 -
Moz Crawl Test Tool - SEO Web Crawler showing up with no details
So basically I have ran the Moz Crawl Test tool twice for this url "bubblingwithenergy.info" and both times the report has listed 1 URL when there is obviously a lot more if you check the site. My question is, why is the Moz Crawl only reporting 1 URL when there are heaps? Is there a possibility it is being blocked and if so what would be blocking it? This website is using a CMS called Infusion and it is based off CMSMS (CMS Made Simple). Any answers would be greatly appreciated. Cheers
Moz Bar | | KBB_Digital0 -
Open Site Explorer Broke Still?
Its been a couple days now, and a bunch of the useful open site explorer features are still down... Is this something that you are working, were the features taken down or is it just being neglected? Prior to posting this, I tested on multiple browsers... Windows 7,8 & a Mac Specifically, the criteria below prompts "oops something went wrong" Inbound Links Tab : Drop downs for, "Show Links", "From Pages", "To This Page" Top Pages Tab : Broken Please Advise
Moz Bar | | Southbay_Carnivorous_Plants0