100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of no-follow links
I'm curious to understand roughly how much % of value a no-follow link has in building authority relative to a do-follow link? I understand that Google seems consistently and growingly focused on value - ie. is the link valuable in growing the business, irregardless of SEO - and perhaps therefore the no-follow / do-follow distinction is becoming a more unnecessary dichotomy. How does Google look at do-follow vs no-follow links? And how much weight now is really given to one compared to the other?
Intermediate & Advanced SEO | | Gavo0 -
Link Audit - Sponsor/Partners Images Links
Hi everyone, 1. I'm conducting a link audit and read that if you are a sponsor or partner of a company, links should be nofollowed. I always no follow them if they are money keywords, but branded I leave alone. is that a good strategy? Or do i nofollow my brand name as well? 2. What if I'm a sponsor and have my company logo on their website that links to my website? How would i know if that link should be nofollowed? a. Does it depend on the "alt" of the image? b. Does it depend on the landing page of the link of the image? c. Do i just no follow image links from sponsor pages and partner pages as a whole? Please keep in mind that I'm sponsoring websites that are relevant to my niche. PLEASE HELP!
Intermediate & Advanced SEO | | Shawn1240 -
Link Juice + Site Structure
Hi All, I have attached a simple website model.
Intermediate & Advanced SEO | | Mark_Ch
Page A is the home page attracting 1000 visitors per month.
One click away is Page B with 400 visitors per month, so on and so forth. You get an idea of the flow and clicks required to get to various pages. I have purposely placed Pages E-G to be 3 clicks away as they yield very little traffic. 1] Is this the best way to distribute link juice?
2] Should I point Pages C + D back to page A to influence its Page Rank (PA) Any other useful advice would be appreciated. Thanks Mark vafnchI0 -
Ecommerce Internal Linking Questions
I am a bit confused at internal linking for ecommerce site. Is it wise to link say all "boots" term in the review section to the boots page? Zappos is doing this. Wouldn't this incur penguin penalty? Since all internal anchor to that page is "boots" ? Scroll down to the bottom and checkout their reviews: http://www.zappos.com/tony-lama-6071l Is this the wise way to go about doing internal linking? Thanks
Intermediate & Advanced SEO | | WayneRooney0 -
Do nofollow links affect link profile?
I've read that it's good to keep a natural link profile. Some naked links, some links going to our company name, some with anchor text, etc. Do nofollow links affect this link profile, or is it only followed links that are taken into account?
Intermediate & Advanced SEO | | lighttable0 -
How to promote some links on google
Hi our site is http://www.mycarhelpline.com If people search on our site in Google by typing - Mycarhelpline they see links - why mycarhelpline, contact us and about us how can we put some other key pages by replacing above pages
Intermediate & Advanced SEO | | Modi0 -
Is Google Webmaster tools Accurate?
Is Google webmaster Tools data completely inaccurate, or am I just missing something? I noticed a recent surge in 404 errors detected 3 days ago (3/6/11) from pages that have not existed since November 2011. They are links to tag and author archives from pages initially indexed in August 2011. We switched to a new site in December 2011 and created 301 redirects from categories that no longer exist, to new categories. I am a little perplexed since the Google sitemap test shows no 404 errors, neither does SEO MOZ Crawl test, yet under GWT site diagnostics, these errors, all 125 of them, just showed up. Any thought/insights? We've worked hard to ensure a smooth site migration and now we are concerned. -Jason
Intermediate & Advanced SEO | | jimmyjohnson0 -
Seasonal links, seasonal ranks
As the garden season begins to wane I notice yet again how my ranking for some garden specific terms - eg ' garden tealight holders' start to rise again.Since I am doing nothing much I can only assume that my competitors have moved their focus to more winter based merchandise. Does anyone have a good understanding of how some websites are able to acheive high rankings during peak season only? I am assuming they are buying advertising (with the follow) for say 3 months before the season peak and manipulating internal linking to direct link juice from one section of the website to the other. Is this strategy risky. Has Google ever made mention of this issue?
Intermediate & Advanced SEO | | GardenBeet0