100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Footer no follow links
Just interested to know when putting links at the foot of the site some people use no-follow tags. I'm thinking about internal pages and social networks. Is this still necessary or is it an old-fashioned idea?
Intermediate & Advanced SEO | | seoman100 -
Can one back-link fluctuates ranking of website with thousands of back-links?
It happend to our website. We have seen major ranking fluctuations for our website because of one back-link. What kind of links those can be? Why Google is not stopping them even though they claim that such back-links will be taken care of?
Intermediate & Advanced SEO | | vtmoz0 -
Links: Links come from bizzare pages
Hi all, My question is related to links that I saw in Google Search Console. While looking at who is linking to my site, I saw that GSC has some links that are coming from third party websites but these third party webpages are not indexed and not even put up by their owners. It looks like the owner never created these pages, these pages are not indexed (when you do a site: search in Google) but the URL of these pages loads content in the browser. Example - www.samplesite1.com/fakefolder/fakeurl what exactly is this thing? To mention more details, the third party website in question is a Wordpress website and I guess is probably hijacked. But how does one even get these types pages/URLs up and running on someone else's website and then link out to other websites. I am concerned as the content that I am getting link from is adult content and I will have to do some link cleansing soon.
Intermediate & Advanced SEO | | Malika10 -
Which links to disavow?
I've got a new client that just fired their former SEO company, which was building spammy links like crazy! Using GSC and Majestic, I've identified 341 linking domains. I'm only a quarter of the way through the list, but it is clear that the overwhelming majority are from directories, article directories and comment spam. So far less than 20% are definitely links I want to keep. At what point do I keep directory links? I see one with a DA of 61 and a Moz spam score of 0. I realize this is a judgement call that will vary, but I'd love to hear some folks give DA and spam numbers. FWIW, the client's DA is 37.
Intermediate & Advanced SEO | | rich.owings0 -
Should I remove footer links?
I added footer links to my site some months ago as I figured that any authority my home page had would be distributed to several of my other most important pages on my site helping them to rank. Would I be better to remove them and would that improve the authority of my home page as less 'link juice' is being distributed. I did originally set up a page per keyword on my site and start building links to each one but as my home page has a good authority I am going to target several keywords on my home page instead as I have some way to go to improve the authority of my other important pages and think this would be a better solution. It would reduce the number of links I have per page however I did see Matt Cutts say that the no more than 100 links per page rule doesn't apply any more. Do footer links add any SEo value?
Intermediate & Advanced SEO | | SamCUK0 -
Links to Facebook pages
I would like to ask if anyone has any knowledge regarding linking to a company's facebook page. I have built a few links to a client's facebook page in an effort to have it rank better in SERPs. I just learned that unlike twitter and linkedin, it is apparently not possibly to directly link to facebook pages. At least it is not possible from a search engine's perspective. If you follow any facebook page link while you are not logged into facebook, you are redirected to the facebook home page. I can't think of any way around this obstacle. I'd love some clever solution such as providing a URL which includes a basic dummy facebook login but there is nothing I am aware of to achieve this result. Does anyone have any ideas on this topic?
Intermediate & Advanced SEO | | RyanKent0 -
Competitors and Directory Links
Hi guys, wanted to get some input and thoughts here. I'm analyzing many competitor links for a specific client (even other clients actually as well) and come across a pretty heavy directory backlink profiles. has anyone here had success with directory listings? Seem many of the competitors backlinks are coming from directories. What say you?
Intermediate & Advanced SEO | | PaulDylan1 -
Value of Newspaper Comment Links
Although most newspaper comment sections are a no-follow zone, I have noticed that some comments I have posted with links end up being followed. The comments are participatory and the links relevant and even add to the conversation. My theory is that some comments are monitored and if the editors are looking to encourage discussion and don't feel like your spamming, why not take the no follow off. I do plan on doing some testing with poor, spammy comments on the same papers but am encouraged and would like to know what other people have found.
Intermediate & Advanced SEO | | phogan0