100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links
Hi 64% of our links come from a .com website and only 30% from .co.uk. We only do business in the UK should I continue with the .com links as they are easier to source. Does this hurt my SEO efforts?
Intermediate & Advanced SEO | | Caffeine_Marketing0 -
Back links with instant effect?!
I realise this is incredibly controversial! And I also realise I’ll get a ton of trolls pulling me to shreds but… I’m in need of running a short-term experiment, but to do the experiment I need to get a test site ranking high very quickly (not worried about if it gets penalised, it is only a short-term test).
Intermediate & Advanced SEO | | seoman100 -
Should sitewise links always be branded
Hello, A client has 8 sitewide links with the same anchor text - their main keyword. They have 84 linking root domains total. 3 of the 8 are his own sites. 4 of the sitewide links are in the footer (all 3 of his own sites interlink in the footer) In the last 5 months, rankings for the the top 2 short-tail keywords have dropped even though they should rank higher. In a few days they're going to do some major rearranging with one of the 8, adding nofollows sitewide because of a partnership disagreement. Would there be any negative consequences, do you think, to right away changing all of the footer links of the 3 sites that the site owner owns to branded anchor text (domain.com)? Should we change all the sitewides to branded anchor text? There have been no problems in GWT.
Intermediate & Advanced SEO | | BobGW0 -
SEO from links in frames?
A site was considering linking to us. Their web page is delivered entirely via frames. Humans can see the links on the page, but it's not visible in source. I'm guessing it means Google can't detect the links, and there is no SEO effect, but I wanted to confirm. Here's the site: http://www.uofc-ulsa.tk/ Example links are the Princeton Review and Kaplan on the right sidebar. Here's the source code: view-source:http://www.uofc-ulsa.tk/ Do those links have any SEO impact?
Intermediate & Advanced SEO | | lighttable0 -
How quickly should you aquire links?
Hi Guys, How often should you aquire links without getting into trouble with Goolge? Should you aqure a linka day? Or a link every 2 days? What should it be? Thanks guys Gareth
Intermediate & Advanced SEO | | GAZ090 -
What is your onsite linking strategy?
So there are a few different routes to take when you're SEOing your site. My quest is to determine which is the best way to approach this. Let's use a real life example of a product. It's project management software, online collaboration software, employee scheduling tool, business process streamlining tool, client management tool and task/to do manager. It works for virtually any industry. I've created my keyword document and it's HUGE. I've created my wireframe with related keyphrases in buckets. Each one of the example keyphrases listed above have slight variations then a whole list of long tails. I have a few options as I see it: Create site sections within the main site that focus on each (This can make the site look slightly sloppy and categories would have to be masked so it doesn't appear spammy) Create a page in the blog relevant to each keyphrase and link all subsequent blog posts within that keyphrase family directly to that blog post (This seems like my best option) and have cta's or conversion mechanisms on this page Link all keyphrases to the home page (Seems like a terrible idea) Not sure if I answered my own question here, but I'd love to hear what everyone else thinks. What are your thoughts?
Intermediate & Advanced SEO | | cmdsonline0 -
Canonical Tag and Affiliate Links
Hi! I am not very familiar with the canonical tag. The thing is that we are getting traffic and links from affiliates. The affiliates links add something like this to the code of our URL: www.mydomain.com/category/product-page?afl=XXXXXX At this moment we have almost 2,000 pages indexed with that code at the end of the URL. So they are all duplicated. My other concern is that I don't know if those affilate links are giving us some link juice or not. I mean, if an original product page has 30 links and the affiliates copies have 15 more... are all those links being counted together by Google? Or are we losing all the juice from the affiliates? Can I fix all this with the canonical tag? Thanks!
Intermediate & Advanced SEO | | jorgediaz0 -
Dark Matter Links
From 2007 - 2004 I worked for Sprint in several positions with my last one being a Corporate Account Manager for fortune 1000 customers. In 2004 I left Sprint after the Nextel merger and created an eCommerce site called thesprintstore.net as a Sprint Nextel preferred partner. I used my inner working knowledge of Sprint to my wonderful advantage and began making 3x my original salary. My desire for more business turned to greed and I began leaking information that consumers loved i.e. phone release dates, price points, warehouse stock levels and tricks of the trade. This garnered me thousands of links from big sites (had no idea at the time) and eventually my site was issued a Cease and Desist order from Sprint's Corporate Headquarters. I recently realized one evening that I had a GEM of a domain with powerful backlinks that I could redirect to my current site TECHeGO.com [staff removed hyperlink]. (Some of the back links are from Engaget, Engaget Mobile, Rimmarkable and even one from Sprint.) The redirection has been in place for months now and I have confirmed that all that sweet Link Nectar is flowing through! I have found it interesting, however, that my back link and referral domain count have never increased leading me to believe that in doing a 301 Redirect existing links become what can only be described as 'Dark Matter Links' i.e. the links are there, simply invisible. Dark Matter Definition: dark matter is matter that is inferred to exist from gravitational effects on visible matter and background radiation, but is undetectable by emitted or scatteredelectromagnetic radiation. Dark Matter Links: dark matter links are visible links that have passed through a 301 redirect which are now inferred to exist but are no longer visible by crawlers? Is there a better definition that could be applied to the term 'Dark Matter Links'?
Intermediate & Advanced SEO | | TECHeGO1