100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Link externally from destination pages
Hello, From my destinations pages, is it beneficial to have a link going externally to the hotels my clients are staying in ? In other words is it beneficial to my website or will it hurt me to link external from all my destinations pages to all the hotels my clients are staying in ?
Intermediate & Advanced SEO | | seoanalytics0 -
What link would be better?
Hi Guys, Just wondering what would be better in this instance: finding an old post (with good authority) and getting a link from that old article or creating a brand new article and adding the link to that. Finding an old post (with good authority) and getting a link from that old article Creating a brand new article and adding the link to that. Both naturally link out to the page you want a link too. To me, number 1 as the page already has authority but then again number 2 since Google might place some weight to recency. Any thoughts? Cheers.
Intermediate & Advanced SEO | | spyaccounts140 -
Link conundrum - losing nav/footer links in mobile view
Hi Moz folks! I'm currently moving a site from being hosted on www. and m. separately to a responsive single URL. The problem is, the desktop version currently has links to important landing pages in the footer (about 60) and that's not something we want to replicate on mobile (mainly because it will look pretty awful.) There is no navigation menu because the key to the homepage is to convert users to subscription so any distraction reduces conversion rate. The footer links will continue to exist on the desktop view but, since Google's mobile-first index, presumably we lose these important homepage links to our most important pages. So, my questions: Do you think there is any SEO value in the desktop footer links? Do you have any suggestions about how best to include these 60-odd links in a way that works for mobile? Thanks!
Intermediate & Advanced SEO | | d_foley0 -
Found a cache of old domain names, should I link or 301 redirect
We have found a cache of about 10 URLs, some are ranking above our main URL in Google SERPS. What is the best course of action here? a. Redirect all to the homepage?
Intermediate & Advanced SEO | | moconn
b. Link all domains to the homepage?
c. Link all domains to select pages on on main site, being careful not to anchor text spam
d. 301 redirect all to the main site. Is there any disadvantage to your recommendation? Is there likely to be a penalty incurred? I feel like we'll get the strongest increase in rankings by following option c but it feels like option d may be safer. Thanks in advance for your help!0 -
Hammered by Spam links
When we moved from one host to another in Wordpress engine, we had this insertion weird redirect thing happen. We 410'd the page cgi-sys/movingpage.cgi, but it hit us hard in the anchors. If you go to ahrefs, we are literally all Asian in anchors text. Anybody have any suggestions, thank goodness it looks like it finally stopped. I am looking for creative ways to repopulate our back end with the right stuff. Any thoughts would be great! Heres a example: allartalocaltours.com/tumi-tote-401.html ↳customerbloom.com/cgi-sys/movingpage.cgi ↳www.customerbloom.com/cgi-sys/movingpage.cgi ↳lockwww.customerbloom.com/cgi-sys/movingpage.cgi
Intermediate & Advanced SEO | | mattguitar990 -
Research on building links to a website
Hi building a brand new site with no domain authority. I have created all the content and now want to start building links to the website. Mostly through guest posting, niche directories, broken link building and other whitehat methods. Anyway i was wondering if anyone has seen any good research on the way you should link to a brand new website or any site for that matter. Like in terms of % you should focus at the homepage, inner pages, anchor distribution, internal link structure, etc. A good start would be looking at successful competitors, but i wanted to see if anyone knows any studies on this. My goal is to build a link profile which meets the standards of Google and that lasts! Thanks, Mark
Intermediate & Advanced SEO | | Mikey0080 -
Sitewide blog link and Article links
Hi Guys I just wanted to give you all a heads up on something I adjusted recently that worked really well and wanted to ask for your own experiences on this. 1. We have a blog that adds regular content and within the blog we link from the keyword we are targeting. Standard stuff right ! We were struggling for movement on a keyword so I removed the links from the articles and added a link on the site wide blogroll. The link on the blogroll included the keyword but was a longer descriptive link. Low and behold we got a first page listing when the changed it.The change in ranking was made a few days later. I have always been given the impression that site wide isn't that great ? So explain this one . Of course there are many other factors etc 🙂 What are your experiences and thoughts on what happened here ?
Intermediate & Advanced SEO | | onlinemediadirect0 -
Switching to masked affiliate links
Hi there, I run a content affiliate website where I introduce products in articles and then link to merchants where the user can buy the respective product. Currently I am using regular affiliate links here with the "nofollow" attribute. With growing size of the site, I would like to switch to masked affiliate links, so instead of a link like "jdoqocy.com/click-123" I want to use "mydomain.com/recommend/123". My question here is: When switching to masked affiliate links, does it makes sense to also convert all the older unmasked affiliate links? If yes, what would be the best way to do that - Convert all old links at once or convert them over time (e.g. over a few month)? Currently about 2/3 of my site's outbound links are unmasked, external affiliate links. So I am afraid that changing this relatively large share of links from unmasked external affiliate links to masked links doenst look natural at all... Thank you for your advice!
Intermediate & Advanced SEO | | FabRag0