100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Benefit of internal link in content
Hi, Is there a real benefit to having internal links in content other than at the bottom of a page for example and not surrounded by content. Would the benefit be 1 to 10 or 1 to 1.5 ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Link building
ok mozers i have a few questions. I am starting a new seo campaign and i want to target traffic for "how to make money on autopilot" Question 1. when it comes to link building i have seen some articles saying that i should not send all of my links to my landing page at once but to send links to my backlinks then index then using tiered link building. Is this a must or not? will i get penalized if i build 20 targeted links to my landing page in 1 day, lets say 20, pr7-9 domains? or should i tier it out and link maybe 5 pr9 domains to my landing page, then link 10 pr5 domains to each of those 5 pr9 domains and maybe link 20-pr1 domains to each of those tiered 2 pr5 domains? eq: Tier 1 = 5 PR-9 Tier 2 = 50 PR-5 Tier 3 = 1,000 PR-1 Question 2. Is their a certain amount of backlinks i need to use in order to out do my competitor? or does it just matter on the metrics of my backlinks? and when it comes to indexing these links do i need to index just the 5 pr9 links? or do i need to index all of them? or should i just index the landing page through google webmasters tools and hope it indexes all connecting pages? will doing any of these get my landing page indexed faster in order to rank faster? Question 3. Types of link building. Ok i am targeting guest blogs, wordpress sites, etc to put a link on. Should i focus on smm 'social media marketing' as well? or can i just focus on the traditional seo tactics first? Question 4. Keyword research. ok so my blog post is 'how to make money on autpilot' and from my keyword suggestion tools it picked up a list of keywords suggestions to target. Competition ranges from low to high, search volume ranges from 10 to 1900 visitors per month, after organizing the most relevant keywords to add to my campaign should i target each of the these keywords by creating a link building campaign for each one and target it to my landing page or use it as my 2nd or 3rd tier? those are the questions i really have for now. Here is my blog post http://www.vemomedia.com/how-to-make-money-on-autopilot/ Please feel me in on what i am needing to do in order to get some ranking and on how to run a link building campaign the correct way. Thanx in advance!
Intermediate & Advanced SEO | | djgbshows1 -
Links: Links come from bizzare pages
Hi all, My question is related to links that I saw in Google Search Console. While looking at who is linking to my site, I saw that GSC has some links that are coming from third party websites but these third party webpages are not indexed and not even put up by their owners. It looks like the owner never created these pages, these pages are not indexed (when you do a site: search in Google) but the URL of these pages loads content in the browser. Example - www.samplesite1.com/fakefolder/fakeurl what exactly is this thing? To mention more details, the third party website in question is a Wordpress website and I guess is probably hijacked. But how does one even get these types pages/URLs up and running on someone else's website and then link out to other websites. I am concerned as the content that I am getting link from is adult content and I will have to do some link cleansing soon.
Intermediate & Advanced SEO | | Malika10 -
URL Value: Menu Links vs Body Content Links
Hi All, I'm a little confused. I have read a number of articles from authority sites that give mixed signals over the importance of menu links vs body content links. It is suggested that whilst all menu links spread link juice equally, Google does not see them as favourably. Inserting a link within the body will add more link juice value to the desired page. Any thoughts would be appreciated. Thanks Mark
Intermediate & Advanced SEO | | Mark_Ch0 -
Do links from twitter count in SEOMoz's Toolbar link count?
I am using the Chrome extension and looking at a SERP, when a page is said to have 2000 incoming links, does that include tweets with a link back to this page? What about retweets. Are those counted separately or as one? And what about independent tweets that have exactly the same content (tweet text + link)
Intermediate & Advanced SEO | | davhad0 -
Because Goolge chose this link to my site?
I'm better ranked in Google for that link (http://www.vipgoldrj.com/paginas/ensaios.html) and not in (http://www.vipgoldrj.com/), you know you explain why? In all keywords, except that (luxury escorts in Rio de Janeiro) Sorry my english, I'm from Brazil and I'm using Google translator.
Intermediate & Advanced SEO | | WebMaster0210 -
Penguin or paid link penalty, or both?
Hello, I have a site, macpokeronline.com, that has seen dramatic decrease in visitors in the last few months, it has went down from 800 per day to 200 per day. It is a pretty complex situation. The site owner purchased paid links from reputable mac sites for years (they were more of followed advertisements, but were only there for SEO Purposes), now that i'm going through the link profligate ins OSE, I can see that a majority of their links come from these sites. There is also a branding issue, there are almost 15,000 links with the anchor text of "macpokeronline.com" These are obviously branded links, I don't know the best way to deal with them (though the majority are coming from the paid link sites) We have just sent the request in to remove the paid links from the sites, and i'm guessing since he is paying over $1000 a month for the links, they will be removed quickly. The site has been receiving significantly less traffic since penguin (apr 24-25) We received a message on July 19th which was the generic unnatural link warning, saying that once we remove links make a reconsideration request. Then on July 23rd, we received another message that says they are taking a "very targeted action on the unnatural links instead of your site as a whole" which I have never seen before. This damage was done before I was hired by this client, I just want to get his traffic back up so I can help him even further, I want to know more about the steps I should take. 1. I will definitely remove the paid ads What else should I do, thanks Zach
Intermediate & Advanced SEO | | BestOdds0 -
Link Age as SEO factor?
Hi Guys
Intermediate & Advanced SEO | | VividLime
I have a client who ranks well within a competitive sector of the travel industry. They are planning CMS move which will involve changing from .cfm to .aspx We will be doing the standard redirects etc However Matt's statement here on 301 redirects got me thinking
http://www.youtube.com/watch?v=zW5UL3lzBOA&t=0m24s He says that basically you loose a bit of page rank when you do a 301 redirect. Now, we will be potentially redirecting 1000s of links and my thinking is 'a lot of a little, adds up to a lot' In other words, 1000s of redirects may have a big enough impact to loose some rankings in a very competitive and aggressive space. So recommended that we contact the sites who has the link highest value and ask them to manually change the links from cfm to aspx. This will then mean that there are no loss value as with a 301 redirect. -But now I have another dilemma which I'm unsure about. So the main question:
Is link age factor in rankings ? If I update any links, this will make said link new to Google, so if link age is a factor, would this also lessen the value passed initially?0