Crawling password protected sites such as dev or staging areas to look at sites b4 going live ?
-
Hi
Ive instructed clients to password protect dev areas so dont get crawled and indexed but how do we set up Moz crawl software so we can crawl theses sites for final check of any issues before going live ?
Is there an option i havnt seen to add logins/passwords for crawl software to access ?
cheers
dan
-
ok thanks Chiaryn
is that the actual name of the moz crawler (to allow in Robots) simply rogerbot ? or any other characters etc ?
Also is it not the case that even when blocked by robots.txt G can still crawl/index it once password removed, think i read few comments somewhere on Moz that can still happen somehow ?
Please advise asap ?
Many Thanks
Dan
-
Hey Dan,
Unfortunately, our crawler is not able to access password protected content on your site. If you create a staging subdomain that is not password protected, you could use the robots.txt file to allow rogerbot and block other crawlers, but I'm afraid our crawler will not crawl anything that a normal search engine crawl would not be able to crawl so we cannot crawl password protected pages.
I hope this helps.
Chiaryn
-
i dont suppose either of you are able to help at all with this related question:
http://moz.com/community/q/site-crawl-errors-download-list-of-all-urls
-
i dont suppose either of you are able to help at all with this related question:
http://moz.com/community/q/site-crawl-errors-download-list-of-all-urls
-
Hi Andy
Screaming Frog does have password access feature for your info i have just tried it
All Best
Dan
-
Thanks Matt
I have got screaming frog and can confirm that it has password access feature, but i really want Moz to be able to access too, i would have thought they should have this option somewhere. Are you saying Moz crawls have more info than SF (re 'moz level' analysis) ?
Dev site better password prtected than robots arnt they i think ?
Cheers
Dan
-
Hi Dan
I was about to ask the exact same question, so will keep an eye out for an answer.
I hope it is possible, but I couldn't work it out.
-
I don't know if there's a way to do this in Moz but you could always get Screaming Frog & tell it to ignore robots.txt - that will definitely crawl it. You can check titles, descriptions, canonicals, H1s, etc. that way. It doesn't give the Moz level analysis but it's a start that def works. You can also see if you have parameter issues that way.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is on-demand crawl reporting numbers that are very different than the regualr site crawl by Moz?
When I look at the site crawl available through my dashboard, it reports 22.9K pages crawled. I just ran a on-demand crawl and it is only reporting 6K pages crawled. Why is the number so much lower?
Moz Bar | | ctripp10100 -
Crawl tests stuck in queue
I have tried to run a number of crawl tests recently for our client's sites outside the US and they have been stuck in the queue for over a week. 3 of them completed, but then 5 are stuck. Anyone experience this? I haven't seen anything about crawl tests having issues right now.
Moz Bar | | rmcgrath810 -
Why doesn't the Keyword Explorer "Explore By Site" work?
Whatever page/domain url I put in here, I get a message coming up saying ""Getting rankings counts failed" Why does his happen? I can't find anything in the Help about this.
Moz Bar | | mfrgolfgti0 -
Problem Downloading Crawl Error Report PDF's
I am trying to download the PDF reports for the various 'crawl errors' - now some of them are quite large but would that justify why I am unable to download - the error is a straightforward one, see attached. Any ideas? Andy aDlViIN
Moz Bar | | TomKing0 -
How do I go about fixing my High Priority issues that SEO moz says I have on a PHP site?
I am been trying to deal with this problem for some time now. I have talked to several IT people and SEO moz. None seem to know how to fix these issues on the type of site our company is. Our biggest issue with is Duplicate Page Content. We also have some title issues. Our site is built with PHP coding and variable, meaning the site is not a typical static website. We have a handful of pages that are dynamic depending on what the users chooses to see and do. So, my problem is I can't just go to a specific page and put the canonical or the redirect. It isn't multiple pages for our category pages, for example, it is just one that builds the page depending on the search. Please help!
Moz Bar | | JoshMaxAmps0 -
Why does the moz crawl test lists page twice?
Hi, I'm running into an issue where some crawlers list my pages twice, once with a trailing slash, once without. I first saw it on a few pages with screaming frog, then saw it happen on all my pages with the moz crawler. The site is www.kidsandart.org and its on Squarespace. I grepped the sitemap.xml I submitted to google webmaster and got 167 distinct pages, all of them without a trailing slash. Any insights on why this is happening, and how to regard moz crawler results would be appreciated. thanks Tom
Moz Bar | | tpushpathadam0 -
Moz crawl sees meta description but there are none
I have a new site I have run a starter crawl on. The crawl came back saying some of the pages do indeed have meta descriptions. When I go to the same page and use the Moz Chrome toolbar it says they do not have meta descriptions. I also know they do not have meta descriptions. Are there any instances in which the Moz crawler would see them when they are not there?
Moz Bar | | SBXMedia0 -
Moz not crawling opencart product pages
hi, i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products. any idea of what can be wrong? im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1). thanks
Moz Bar | | marlvass10