100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Link building strategy
Hello Moz Community, For the last couple of months we have been trying to improve our ranking in Google UK for the keyword "church candles" http://www.wattsandco.com/church-supplies/church-candles.html We’ve been contacting relevant interiors/lifestyle blogs to feature our candles including anchor text linking back to our page. Our anchor text has been predominately our brand (Watts & Co) but also other key search terms (Watts and Co church candles, Watts and Co pillar candles). We have been tracking our ranking for the keyword “Church candles” using the Moz “ Rank Tracker” and we started on position 15 in Google UK. We went up to 12 briefly before moving down every week to 15, 17, 19 and 22. We checked today and we have moved back up slightly to 19. Our progress seems to be a bit slow and inconsistent. We wanted to reach out for any advice on how we can move up? If there was any way we can improve our strategy? Here’s the links we have built so far: http://nostalgiecat.blogspot.co.uk/2015/10/what-autumn-means-to-me.html http://blog.pollyrowan.com/2015/10/5-small-ways-to-decorate-your-home-that.html http://www.happyhomebird.com/2015/10/watts-co-candles-for-cosy-autumn-home.html http://www.frolic-blog.com/2015/10/beeswax-candles-for-fall/ http://hisforhomeblog.com/lighting/watts-co-church-candles/#axzz3qhqN1wzA http://lorilangille.blogspot.co.uk/2015/11/sponsored-post-watts-and-co.html http://www.californiahomedesign.com/product-finds/waxing-poetic-must-have-candles Thanks so much!
Intermediate & Advanced SEO | | roberthseo0 -
How to do Spam Link Analysis before posting a link?
OSE provides Spam analysis for website link profile, Do Moz have a tool to check the link quality before placing a link? How to do Spam Link Analysis before posting a link?
Intermediate & Advanced SEO | | bondhoward1 -
301 Externally Linked, But Non-Producing Pages, To Productive Pages Needing Links?
I'm working on a site that has some non-productive pages without much of an upside potential, but that are linked-to externally. The site also has some productive pages, light in external links, in a somewhat related topic. What do you think of 301ing the non-productive pages with links to the productive pages without links in order to give them more external link love? Would it make much of a difference? Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Are footer links important?
We currently display a list of links in the footer of our site to help boost SEO. They were put in place years ago and in a recent discuss with our UX team they requested we remove them from the site. Do footer links have any value? Or is this an old dated practice that no longer works? If we remove the footer links should we expect to see if have an impact on our SEO traffic?
Intermediate & Advanced SEO | | Mivito0 -
Link Research Tools - Detox Links
Hi, I was doing a little research on my link profile and came across a tool called "LinkRessearchTools.com". I bought a subscription and tried them out. Doing the report they advised a low risk but identified 78 Very High Risk to Deadly (are they venomous?) links, around 5% of total and advised removing them. They also advised of many suspicious and low risk links but these seem to be because they have no knowledge of them so default to a negative it seems. So before I do anything rash and start removing my Deadly links, I was wondering if anyone had a). used them and recommend them b). recommend detoxing removing the deadly links c). would there be any cases in which so called Deadly links being removed cause more problems than solve. Such as maintaining a normal looking profile as everyone would be likely to have bad links etc... (although my thinking may be out on that one...). What do you think? Adam
Intermediate & Advanced SEO | | NaescentAdam0 -
Are Links from Homepage safe?
I have the option of getting a link from 2 websites home pages. My site is a financial service website. I want to know if it is safe to do so. The first site is a hotel site, so their content has no relation to my site And the second is a Google News website about finances They have offered me a section in the sidebar to have a sentence or 2 about my site with a link. Can you tell me would these be safe to get? I don't want to be hit by any penalty
Intermediate & Advanced SEO | | JohnPeters0 -
Any good link buying companies ( http://www.text-link-ads.com )
Hi guys I have been passed this website: http://www.text-link-ads.com Has anyone ever used text-links ads before?? Can anyone please show me the way and suggest any really good lin buying companies? I am really fiding it hard to find good places to place inbound links into our website.. Thanks Gareth
Intermediate & Advanced SEO | | GAZ090 -
Removing Canonical Links
We implemented rel=canonical as we decided to paginate our pages. We then ran some testing and on the whole pagination did not work out so we removed all on-page pagination. Now, internally when I click for example a link for Widgets I get the /widgets.php but searching through Google I get to /widgets.php?page=all . There are not redirects in place at the moment. The '?page=all' page has been rated 'A' by the SEOmoz tool under On Page Optimization reports and performs much better than the exact same page without the '?page=all' (the score dips to a 'D' grade) so need to tread carefully so we don't lose the link value. Can anyone advise us on the best way forward? Thanks in advance.
Intermediate & Advanced SEO | | jannkuzel0