100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disavowing 100k Affiliate Links
Hi all, hope you're all good. I am updating our disavow file, we've noticed a couple more spammy links which are pointing at or site. While I was at it, affiliate links came to my mind. At the moment we have over 100k+ affiliate links pointing to the root of our site and other categories/products, most of them are do-follow. However, taking a look at WMT, it's one of our 'Who links the most' and the affiliate network is pointing a total of 115,065 links to us. My question; bearing it mind this site generates over 2million hits a month, is it really worth disavowing the entire affiliate link network. This would result is all of those 100,000 links being disavowed over time. Do you think this would result in a positive? Let me know your thoughts.
Intermediate & Advanced SEO | | Brett-S0 -
If I nofollow outbound external links to minimize link juice loss > is it a good/bad thing?
OK, imagine you have a blog, and you want to make each blog post authoritative so you link out to authority relevant websites for reference. In this case it is two external links per blog post, one to an authority website for reference and one to flickr for photo credit. And one internal link to another part of the website like the buy-now page or a related internal blog post. Now tell me if this is a good or bad idea. What if you nofollow the external links and leave the internal link untouched so all internal links are dofollow. The thinking is this minimizes loss of link juice from external links and keeps it flowing through internal links to pages within the website. Would it be a good idea to lay off the nofollow tag and leave all as do follow? or would this be a good way to link out to authority sites but keep the link juice internal? Your thoughts are welcome. Thanks.
Intermediate & Advanced SEO | | Rich_Coffman0 -
Hammered by Spam links
When we moved from one host to another in Wordpress engine, we had this insertion weird redirect thing happen. We 410'd the page cgi-sys/movingpage.cgi, but it hit us hard in the anchors. If you go to ahrefs, we are literally all Asian in anchors text. Anybody have any suggestions, thank goodness it looks like it finally stopped. I am looking for creative ways to repopulate our back end with the right stuff. Any thoughts would be great! Heres a example: allartalocaltours.com/tumi-tote-401.html ↳customerbloom.com/cgi-sys/movingpage.cgi ↳www.customerbloom.com/cgi-sys/movingpage.cgi ↳lockwww.customerbloom.com/cgi-sys/movingpage.cgi
Intermediate & Advanced SEO | | mattguitar990 -
Webmaster tools 404
Hey, I'm getting a soft 404 error on a webpage that has content and is deferentially not a 404. We've redirect a load of urls to the web page. The url has parameters which was used before the redirect but are no longer used on by the new url, these parameters have been carried over in the redirect. Is this whats causing the soft 404 error or is there another problem that may need addressing? Also a canonical has been set on the webpage. Thanks, Luke.
Intermediate & Advanced SEO | | NoisyLittleMonkey1 -
PR links
Its seems that at lot of or competitors are using PR site to place articles with links. They are using the same article across many sites with the same anchor text link - But they seem to be doing very well in the rankings.... I have steered away from this type of linking as I assumed Google wouldn't be keen on this type of activity but I seem to be wrong.... Any views on this?
Intermediate & Advanced SEO | | jj34340 -
Is the Tool Forcing Sites to Link Out?
Hi I have a tool that I wish to give to sites, it allows the user to get an accurate idea of their credit score with out giving away any personal data and with out having a credit search done on their file. Due to the way the tool works and to make the implementation on other peoples sites as simple as possible the tool remains hosted by me and a one line piece of Javascript code just needs to be added to the code of the site wishing to use the tool. This code includes a link to my site to call the information from my server to allow the tool to show and work on the other site. My questions are: Could this cause a problem with Google as far as their link quality goes? - Are we forcing people to give us a backlink to use the tool? (in the eyes of Google) or will Google not be able to read the Javascript / will ignore the link for SEO purposes? Should I make the link in the code Nofollow? If I should make the link a Nofollow any tips on how to make the most of the opportunity from a link building or SEO point of view? Thanks for your help
Intermediate & Advanced SEO | | MotoringSEO0 -
Does the number of links on a page metric include repeated links?
Just wondering if the number of links on the page metric includes links that are repeated? So, if I had 100 links to one page would this count as 100 or 1 link?
Intermediate & Advanced SEO | | Cornwall
If it's the former does this mean more links to one page adds weight? Thanks0 -
Link Building Post Penguin?
I really am lost as to what to do these days.. The problem with my industry is the whole idea of link bait isn't very lucrative. There are no bloggers either, so guest blogging also isn't a very good option. Seems to me like the best thing I can do is just publish content! So, publish a lot of quality content? LOL, sounds like that's right up Google's alley. Where do you publish your content, and what would you say has shown the best results for you personally? We called an SEO company, Arteworks, a few days ago (Friday), and they really didn't go into any details about how they build links. We called them because I saw a post that you commented on, here, and it recommended a few companies at the bottom of the post. (Arteworks being one of them) Really, this is where I get so dang confused... The goal is to build links like the old days, except only use unique content, diversify your pages, and anchor text? Sound about right? Or, should I only create content on my site? Thanks in advance for your time and advice!! Sincerely, Tyler Abernethy
Intermediate & Advanced SEO | | TylerAbernethy0