Can I crawl a password protected domain with SEOmoz?
-
Hi everyone,
Just wondered if anybody has been able to use the SEOmoz site crawler for password protected domains? On Screaming Frog you are prompted for the username and password when you set the crawler running, however SEOmoz doesn't. It seems you can only crawl sites that are live and publicly available - can anyone confirm if this is the case?
Cheers,
M
-
Hi,
I don't think it is possible to crawl any password protectec pages. Password protected pages are usually blocked and bots are unable access those pages.
And yes it is true that only sites that are publicly and live can be crawled.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We switched the domain from www.blog.domain.com to domain.com/blog.
We switched the domain from www.blog.domain.com to domain.com/blog. This was done with the purpose of gaining backlinks to our main website as well along with to our blog. This set us very low in organic traffic and not to mention, lost the backlinks. For anything, they are being redirected to 301 code. Kindly suggest changes to bring back all the traffic.
Technical SEO | | arun.negi0 -
Which domain we should continue with?
Hello All, We are working with a client who had manual penalty from Google. We worked on that and now penalty has been removed. Client had already started working on the new domain and now the big dilemma is- Which domain should we continue with? Old or New? We are suggesting them to continue with the old one as that domain had good PR, good backlinks, better visibility on their social profiles etc. What do you suggest? any inputs are highly appreciated. Thanks
Technical SEO | | sachin-sv0 -
Can increase in crawl errors in GWT) be caused by input fields and jquery?
Dear Mozzerz We took over www.urgiganten.dk not long ago and last week we opened up for indexation, after having taken the old website down for a couple of months. One week after opening for indexation we saw a huge increase in crawl errors.Google is discovering some weird links to e.g http://www.urgiganten.dk/30-garmin-urremme/ which returns a 404. In GWT we are told that we are linking to this url from http://www.urgiganten.dk/garmin-urremme. But nowhere on http://www.urgiganten.dk/garmin-urremme will you find this link. However you will find the following script in the source code, which is the only code part that contains "/30-garmin-urremme/":Can it be true that google take the id and adds it to our tld to form a url? We have seen quite a lot of these errors not only on Urgiganten.dk but also some of our other websites!
Technical SEO | | urgiganten0 -
How long before I can use a redirected domain without taking back link juice?
We recently moved our website to a new domain that better matched our brand. I want to use the old domain at some point for another aspect of our business. How long after we do the domain redirect will it be safe to use the old domain again--without affecting the seo of the new domain? Thanks! Harriet
Technical SEO | | zharriet0 -
Umbrella company and multiple domains
I'm really sorry for asking this question yet again. I have searched through previous answers but couldn't see something exactly like this I think. There is a website called example .com. It is a sort of umbrella company for 4 other separate domains within it - 4 separate companies. The Home page of the "umbrella" company website is example.com. It is just an image with no content except navigation on it to direct to the 4 company websites. The other pages of website example.com are the 4 separate companies domains. So on the navigation bar there is : Home page = example.com company1page = company1domain.com company2page= company2domain.com etc. etc. Clicking "home" will take you back to example.com (which is just an image). How bad or good is this structure for SEO? Would you recommend any changes to help them rank better? The "home" page has no authority or links, and neither do 3 out of the 4 other domains. The 4 companies websites are independent in content (although theme is the same). What's bringing them altogether is under this umbrella website - example.com. Thank you
Technical SEO | | AL123al0 -
Buying Expired Domains
Recently i was looking to buy some good quality Expired Domains. But while performing site and links query on such domains in google, none of the domains are showing any links or pages indexed in Google but the same domains are showing hundreds of links in opensiteexplorer for that domains. So does Google has started devaluing expired domains or will expired domains recover all their rankings after re registration by us.
Technical SEO | | amit910 -
Selecting a new domain name
If a two word domain is already taken (e.g. onetwo.com), which of the following is a better alternative? 1. one-two.com 2. onetwo.org Best,
Technical SEO | | ChristopherGlaeser
Christopher0 -
SEOMoz Bug?
Hi-- I was using the SEOMoz toolbar, and I went to this page - http://www.hark.com/clips/dxkdbdggyh-hes-totally-lucid-100-percent and it says "zero" links from anywhere. However, it's clearly linked to from the home page - http://www.hark.com/ Why would this be?
Technical SEO | | TheIronYuppie1