100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links to external site (hotels link)
Hello, I am currently designing the webpages of my website and I am wondering if I should link externally or if it going to hurt me ? I am in the travel industry and for example in the France in the Loire valley, I want to list hotels that people can stay at in pre and pods trip. Is it ok to link to maybe 10 of those hotels websites or can it hurt me ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Dummy links in posts
Hi, Dummy links in posts. We use 100's of sample/example lnks as below http://<domain name></domain name> http://localhost http://192.168.1.1 http:/some site name as example which is not available/sample.html many more is there any tag we can use to show its a sample and not a link and while we scan pages to find broken links they are skipped and not reported as 404 etc? Thanks
Intermediate & Advanced SEO | | mtthompsons0 -
For those of you that used LINK DETOX.
Did you go ahead and remove all the TOXIC and HIGH RISK links? Just the toxic? Were you successful with the tool?
Intermediate & Advanced SEO | | netviper0 -
To recover from Penguin update, shall i remove the links or disavow links?
Hi, One of our websites hit by Penguin update and I now know where the links are coming from. I have chance to remove the links from those incoming links but I am a little confused whether i should just remove the links from incoming links or disavow the links? Thanks
Intermediate & Advanced SEO | | Rubix0 -
Link from archived article.
A strong news site has an "archived.domainname" folder, where they have older articles listed. I can get a link on a page where there is a 4 year old article, which will be in this archived sub-domain. My questions: Will Google view a link from a 4 year old article as less valuable. Will Google notice the article is 4 years old and find it odd why the page all of a sudden has a link to my site, and thus devalue such link the sub-domain "archived" does that tell Google it is old and a link will be less valuable thank you
Intermediate & Advanced SEO | | knielsen0 -
Where do I start with cleaning up our link profile?
We have just taken the steps to start building links to www.towelsrus.co.uk, I am concerned about the state of external links to the site created by previous companies, i.e are they OK, or doing us harm, could I get more out f whats already in place or should we focus purely on getting new links? We also have about 25 pages that have 302 errors and contain external links to the site. What should i do with these. try and get them re-directed to our site with appropriate anchor text or simply put a 301 re-direct in place? In essence where I start, We want to build and increase traffic in particular for towels, bathrobes, dressing gowns and bolster our position as our positions are fluctuating a little but steadier than they have been. Any help appreciated.
Intermediate & Advanced SEO | | Towelsrus0 -
Outgoing affiliate links and link juice
I have some affiliate websites which have loads of outgoing affiliate links. I've discussed this with a SEO friend and talked about the effect of the link juice going out to the affiliate sites. To minimize this I've put "no follows" on the affiliate links but my friend says that even if you have no follow Google still then diminishes the amount of juice that goes to internal pages, for example if the page has 10 links, 9 are affiliate with no follow - Google will only give 10% of the juice to the 1 internal page. Does anyone know if this is the case? and whether there are any good techniques to keep as much link juice on the site as possible without transferring to affiliate links? Appreciate any thoughts on this! Cheers
Intermediate & Advanced SEO | | Ventura0 -
Should I remove paid links?
I recently added about 20 paid links from directories but have since seen a 10% drop in traffic. I did also delete about 1000 pages of content that had no inbound links and were duplicated on other sites on the web and replaced the content with new content supplied by a client but still duplicated on other sites on the web, old URLs no longer valid or linked to, new content on new URLs. Assuming the drop in traffic had nothing to do with the content change mentioned above, should I remove the paid links in an attempt to recover? I don't think the old content was bringing in much traffic as it appeared elsewhere on more authoritive sites than mine.
Intermediate & Advanced SEO | | Mulith0