Can I crawl a password protected domain with SEOmoz?
-
Hi everyone,
Just wondered if anybody has been able to use the SEOmoz site crawler for password protected domains? On Screaming Frog you are prompted for the username and password when you set the crawler running, however SEOmoz doesn't. It seems you can only crawl sites that are live and publicly available - can anyone confirm if this is the case?
Cheers,
M
-
Hi,
I don't think it is possible to crawl any password protectec pages. Password protected pages are usually blocked and bots are unable access those pages.
And yes it is true that only sites that are publicly and live can be crawled.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Old domain vs. New keyword domain - Thoughts?
Okay. I'd like to get opinions as to what everyone thinks about domains lately. Here is any example: The current domain is general in nature, in fact, it's a persons name because they are a real estate agent. So the domain is something like JohnDoe.com. Current stats: Has approx. 130 linking domains pointing to it. Has over 300 incoming links from these linking domains. The link profile is clean and not spammy (not to say there are not a few that might be here and there) Was bough in 1994 The new domain would have very little value except it would be keyword rich such as PortlandHomesForSale.com (just an example). What are your thoughts. Thank you.
Technical SEO | | JordanRussell0 -
To 301 redirect or not to 301 redirect? duplicate content problem www.domain.com and www.domain.com/en/
Hello, If your website is getting flagged for duplicate content from your main domain www.domain.com and your multilingual english domain www.domain.com/en/ is it wise to 301 redirect the english multilingual website to the main site? Please advise. We've recently installed the joomish component to one of our joomla websites in an effort to streamline a spanish translation of the website. The translation was a success and the new spanish webpages were indexed but unfortunately one of the web developers enabled the english part of the component and some english webpages were also indexed under the multilingual english domain www.domain.com/en/ and that flagged us for duplicate content. I added a 301 redirect to redirect all visitors from the www.domain/en/ webpages to the main www.domain.com/ webpages. But is that the proper way of handling this problem? Please advise.
Technical SEO | | Chris-CA0 -
Is any know if seomoz update for site crawl.
i belive my site www.breeze-air.com hit by penguin; i found that i had un-natural anchors text and able to remove around 1200 from the 1900 seomoz found. seomoz still shows those anchors - but when i check the link its not there. i removed them 3-4 weeks ago any idea?
Technical SEO | | eoberlender0 -
Keyword in Domain or not?
My on page optimization grade is an "A" with the following factors; Factor Overview <dl class="scoreboard clearfix"> <dt>Critical Factors</dt> <dd>4 / 4</dd> <dt>High Importance Factors</dt> <dd>7 / 7</dd> <dt>Moderate Importance Factors</dt> <dd>8 / 9</dd> <dt>Low Importance Factors</dt> <dd>11 / 11</dd> <dt>Optional Factors</dt> <dd>5 / 5</dd> </dl> The main thing I appear to be missing is keywords in my URL. How truly important is that in today's SEO world and how much time or ranking would be lost if I do not have control to change the external links to my website if I decided to migrate to a keyword relevant url?
Technical SEO | | classa0 -
Exact match domain
Will buying an exact match domain and redirecting it to our main site a good idea, if such a domain is available ? What are the pros and cons ? Are exact match domains still powerful for ranking purpose ?
Technical SEO | | seoug_20050 -
Webmaster tools lists a large number (hundreds)of different domains linking to my website, but only a few are reported on SEOMoz. Please explain what's going on?
Google's webmaster tools lists hundreds of links to my site, but SEOMoz only reports a few of them. I don't understand why that would be. Can anybody explain it to me? Is there someplace to I can go to alert SEOMoz to this issue?
Technical SEO | | dnfealkoff0 -
Referrals from widget domains
Let's say you are doing seo for a company that sells 5000 widgets. Let's call their site www.mainsite.com ... They have already set up ( purchased ) many domains such as www.widget1.com which is one of the widgets they have on their main site. so they might have several hundred of these domains. What is the best use of these domains? Would it be : Have those pages be landing pages with content and pictures and optimized with a link to their main site widget page? Or have that domain have a 301 redirect to go their main site, to the particular page showing that widget for sale off their main site? Miscellaneous sidebar question - Since both the main site widget page and the widget domain might be in competition and might have duplicate content, what are the ramifications of not using the canonical tag, and what other considerations might there be? How much must be duplicate content before considering using the canonical tag?
Technical SEO | | highersourcesites0 -
Sub-Domain Choice Dilemma
We have successfully rolled out 5 sub-domains using very industry specific KWs as the sub, e.g. familylawyers. We're rolling our an employment focused sub.and ideally would use employmentlawyers.XXX.com. However I'm tempted to use a long established (5+ years) sub-domain with a topic related KW that now hosts a non-active blog with PR3 - employment-law.XXXX.com. It has 49 indexed pages, some with PR2. So there's potential for getting a kickstart on traffic and trust with some redirects. Should I go for instant gratification or build for the long haul with the slightly more beneficial URL KWing? I should add that this sub-domain will have thousands of pages that are geo and sub-category focused - a typical URL would be. sub-domain.XXX.com/TX/Austin/wrongful-termination-lawyers.html THX for any opinions.
Technical SEO | | legalseo0