Access denied errors in webmaster tools
-
I notice today Ihave 2 access denied errors. I checked the help which says:
Googlebot couldn’t access a URL on your site because your site requires users to log in to view all or some of your content. (Tip: You can get around this by removing this requirement for user-agent Googlebot.)
Therefore I think it may be because I have added a login page for users and googlebot can't access it. I'm using wordpress and presume I need to amend the robots.txt to remove the requirement for google to log in but how do I do that?
Unless I'm misunderstanding the problem altogether!
-
Samuel
Mat's right - you need to speratae the "access denied" pages into two buckets;
-
those which you want indexed
-
those which you don't want indexed.
For those you want indexed, take them on a page by page basis to figure out why access is being denied. They might not all be because of password protection.
-Dan
-
-
Personally I wouldn't remove the requirement for googlebot if you want to protect that content. Once you do that content is public for two reasons:
1. Any nefarious user can instantly change their user agent and view your content
2. It will appear in the public search results
Much better would be to consider what is and isn't protected by a login. If you want it indexed then it should be seen as public and not behind a login. If it should be private, then protect it.
If you want it to appear in the search to attract people to the site in order to encourage them to register consider having cut down versions that are public and have a strong call-to-action on those to get people to register.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain switch planned - new domain accessible - until the switch: redirect from new to old domain with 307?
Hi there, We are going to switch our local domain oldsite.at to newsite.com in November. As our IT department wants to use the newsite.com already for email traffic till then, the domain newsite.com has to be accessible for public and currently shows the default Apache page without useful content. The old domain has quite some trust, the new domain is a first time registered domain (not known by search engines yet and no published anyhow). The domain was parked till now. I am aware of the steps to take for the switch itself, but: **what to do with the newsite.com domain until everything is prepared for the switch? **I suppose users or search engines find the domain and as there is no useful information available it harms us already. My idea was to 307 redirect newsite.com to the oldsite.at but the concern is that this causes problems as soon as we switch the domain and redirecting with 301 from oldsite.at to newsite.com? Do you have any objections or other recommendations? Thank you a lot in advance.
Intermediate & Advanced SEO | | comicron0 -
Need help with huge spike in duplicate content and page title errors.
Hi Mozzers, I come asking for help. I've had a client who's reported a staggering increase in errors of over 18,000! The errors include duplicate content and page titles. I think I've found the culprit and it's the News & Events calender on the following page: http://www.newmanshs.wa.edu.au/news-events/events/07-2013 Essentially each day of the week is an individual link, and events stretching over a few days get reported as duplicate content. Do you have any ideas how to fix this issue? Any help is much appreciated. Cheers
Intermediate & Advanced SEO | | bamcreative0 -
Link Research Tools - Detox Links
Hi, I was doing a little research on my link profile and came across a tool called "LinkRessearchTools.com". I bought a subscription and tried them out. Doing the report they advised a low risk but identified 78 Very High Risk to Deadly (are they venomous?) links, around 5% of total and advised removing them. They also advised of many suspicious and low risk links but these seem to be because they have no knowledge of them so default to a negative it seems. So before I do anything rash and start removing my Deadly links, I was wondering if anyone had a). used them and recommend them b). recommend detoxing removing the deadly links c). would there be any cases in which so called Deadly links being removed cause more problems than solve. Such as maintaining a normal looking profile as everyone would be likely to have bad links etc... (although my thinking may be out on that one...). What do you think? Adam
Intermediate & Advanced SEO | | NaescentAdam0 -
Rankings dropped off a cliff. Webmaster tools message: No manual spam actions found. Now what?
A week ago my rankings for http://www.top-10-dating-reviews.com (some adult content) dropped and I'm now getting now impressions. I submitted a reconsideration request as I was sure I hadn't violated any rules and today the reply was that no manual spam actions were found. Te email goes onto say there are a variety of other things that could affect rankings such as site architecture, not being able to crawl and algo changes. As far as I'm aware all these issues are fine. I'm not aware of any algo updates last weekend. my question is what can I do now? I need to get my rankings back but there's nothing wrong with my site or practices.
Intermediate & Advanced SEO | | SamCUK0 -
Robots.txt error message in Google Webmaster from a later date than the page was cached, how is that?
I have error messages in Google Webmaster that state that Googlebot encountered errors while attempting to access the robots.txt. The last date that this was reported was on December 25, 2012 (Merry Christmas), but the last cache date was November 16, 2012 (http://webcache.googleusercontent.com/search?q=cache%3Awww.etundra.com/robots.txt&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a). How could I get this error if the page hasn't been cached since November 16, 2012?
Intermediate & Advanced SEO | | eTundra0 -
Old deleted sitemap still shown in webmaster tools
Hello I have redisgned a website inl new url structure in cms. Old sitemap was not set to 404 but changed with new sitemap files,also new sitemap was named different to old one.All redirections done properly Still 3 month after google still shows me duplicate titile and metas by comparing old and new urls I am lost in what to do now to eliminate the shown error. How can google show urls that are not shown in sitemap any more? Looking forward to any help Michelles
Intermediate & Advanced SEO | | Tit0 -
Moving wordpress to main website - errors galore
Hi there A while back I asked whether I should move my established blog on wordpress over to my main website http://www.gardenbeet.com. The overwhelming response was to move it. I still have not seen any benefit other than problems (2mths later). Maybe something is Not Quite Right? Here is a list of issues? Any insights would be welcome. I have installed Yeost SEO plugins 1. A blog category (garden art) ended up outranking my main website for the term Garden Art - so I did a 301 on the garden art category - main website has now regained its ranking in the top 3 position (was number 2 before the move). 2. I have removed the categories from the blog as a 301 on the Garden Art catgegory would not make sense to a user. I decided to use tags as a navigational tool instead. I figured that because I have over 500 tags I could not have had the tags out rank my main website for any key term. So far correct. But now i have wanings from SEO moz 500 missing meta tags,(for the tags) - do i really have to write over 500 meta? over 160 long urls and titles - when i commenced the blog I had no idea that Urls and headings were linked in wordpress ( my developer does not think i should rename the urls and use a 301 as I already have a tonne due to a site rebuild) - is it OK to leave the long urls and fix the title only? On wordpress I had bewteen 400-900 users on my blog a day (using wordpress analytics) - now only 200 (using GA) - Yes I now have increased links to my website but have seen no imrpovement on my SERPS - will i see an improvement in my rankings? my wordpress site use to have page rank of 3 -
Intermediate & Advanced SEO | | GardenBeet0 -
Strange Linking Data in Webmaster Tools
I run a site that was a Wordpress blog with Edirectory software for a directory on the back end. I've scrapped the Edirectory and built the entire site on Wordpress. After the site change I'm seeing about 700 404 Not Found crawling errors, which appear to be old Edirectory pages that no longer exist. My understanding is that they'll cycle out eventually. What troubles me is the linking data I'm seeing. In the "Links to My Site" area of Webmaster tools, I'm seeing 4,430 links to the "About" page, another 2,900 to an obscure deleted directory listing page and only 2,050 to the home page. I show 1,700 links to a terms and conditions pdf and other strange data. To summarize, I'm showing huge numbers of links to obscure pages. Any help would be greatly appreciated.
Intermediate & Advanced SEO | | JSOC0