How to identify 404 that get links from external sites (but not search engines)?
-
one of our site had a poor site architecture causing now about 10.000s of 404 being currently reported in google webmaster tools.
-
Any idea about easily detecting among these thousands of 404, which ones are coming from links from external websites (so filtering out 404 caused by links from our own domain and 404 from search engines)?
-
crawl bandwidth seems to be an issue on this domain. Anything that can be done to accelerate google removing these 404 pages from their index? Due to number of 404 manual submission in google wbt one by one is not an option.
Or do you believe that google automatically will stop crawling these 404 pages within a month or so and no action needs to be taken?
thanks
-
-
Hi Robert,
thanks a lot. So I will not take action to get 404s out of google index.
Regarding your first point, I am not sure I understand how screaming frog would help. I did not use screaming frog yet but link sleuth for status code checks. The status check of 404 in google webmaster tools will probably generally also give 404 status in screaming frog. My objective is to identify among these thousands of 404, the few which are caused by inaccurate or outdated links on external websites so that I can create a 301 for these.
Best,
Daniel
-
icourse
I would suggest downloading the free version of screaming frog for an easy way to get status codes on any or all links.
As to fixing and "crawl bandwidth" being a problem, I disagree. If you are not being crawled it is because of all the 404's. I do not know the timeline for inaction on this, but I do believe "manual submission is not an option" is a recipe for disaster. Because fully analyzing your issues is outside the scope of Q&A, I would suggest you start manually fixing the issues and if on a CMS, start looking at plugins, etc. as a root cause.
Hope that helps
Robert
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site migration/ CMS/domain site structure change-no access to search console
Hi everyone, We are migrating an old site under a bigger umbrella (our main domain). As mentioned in the title, We'll perform CMS migration, domain change, and site structure change. Now, the major problem is that we can't get into google search console for the old site. The site still has old GA code, so google search console verification using this method is not possible, also there is no way developers will be able to add GTM or edit DNS setting (not to bother you with the reason why). Now, my dilemma is : 1. Do we need access to old search console to notify Google about the domain name change or this could be done from our main site (old site will become a part of) search console 2. We are setting up 301 redirects from old to the new domain (not perfect 1:1 redirect ). Once migration is done does anything else needs to be done with the old domain (it will become obsolete)? 3.The main site, Site-map... Should I create a new sitemap with newly added pages or update the current one. 4. if you have anything else please add:) Thank you!
Intermediate & Advanced SEO | | bgvsiteadmin0 -
Getting too many links on Google search results, how do I fix?
I'm a total newbie so I apologize for what I am sure is a dumb question — I recently followed Moz suggestions for increasing visibility on my site for a specific keyword by including that keyword in more verbose page descriptions for multiple pages. This worked TOO well as now that keyword is bringing up too many results in Google for these different pages on my site . . . is there a way to compile them into one result with the subpages like for instance, the attached image for a search on Apple? Do I need to change something in my robots.txt file to direct these to my main page? Basically, I am a photographer and a search for my name now brings up each of my different photo gallery pages in multiple results, it's a little over the top. Thanks for any and all help! CNPJZgb
Intermediate & Advanced SEO | | jason54540 -
Should I noindex the site search page? It is generating 4% of my organic traffic.
I read about some recommendations to noindex the URL of the site search.
Intermediate & Advanced SEO | | lcourse
Checked in analytics that site search URL generated about 4% of my total organic search traffic (<2% of sales). My reasoning is that site search may generate duplicated content issues and may prevent the more relevant product or category pages from showing up instead. Would you noindex this page or not? Any thoughts?0 -
External links from banned websites
Currently working with a client that has seen his rankings diminish after the penguin update. I've manually analyzed all his 600 backlinks and identified approximately 85 external links from websites that have been banned by Google. How do these sites affect his current rankings? Should i just disavow all these links using the Google disavow tool? Any comments would be highly appreciated!
Intermediate & Advanced SEO | | Nick_Johansson0 -
When crawls occur - when will my links show up in Open Site Explorer
Hello everyone, I've been building links for a while now and none of them show up in Explorer. My domain authority hasn't changed for about a month or so. When does Google do crawls and when does SEOMoz do crawls? Thanks
Intermediate & Advanced SEO | | Harbor_Compliance0 -
Getting Started with Link-Building (Advice, Please)
Hi forum, my company has done almost ZERO link building, and most of the traffic we receive to our site is for our branded search terms (people searching for our company name). Our content and on-page SEO is pretty solid, but how would you suggest getting started in link building? We've dabbled in comment marketing, but almost all of these were unfollowed links. We already do PR and submit it to prweb.com. We have submitted to quite a few online directories and are currently working our way through relevant directories provided by SEOMoz. We do not want to pay for links, and we want to do all of this in house. We are committed to putting the time in to get high quality links by hand. Does anyone have any advice? A "Beginner's Guide to Link-Building" would be excellent. The specific site we are working on is http://www.consumerbase.com/index.html if that is of any use. Thanks!
Intermediate & Advanced SEO | | Travis-W0 -
Does Google crawl the pages which are generated via the site's search box queries?
For example, if I search for an 'x' item in a site's search box and if the site displays a list of results based on the query, would that page be crawled? I am asking this question because this would be a URL that is non existent on the site and hence am confused as to whether Google bots would be able to find it.
Intermediate & Advanced SEO | | pulseseo0 -
Search Engine Blocked by robots.txt for Dynamic URLs
Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt I have added following syntax to robots.txt file for all dynamic URLs. Disallow: /*?osCsid Disallow: /*?q= Disallow: /*?dir= Disallow: /*?p= Disallow: /*?limit= Disallow: /*review-form Dynamic URLs are as follow. http://www.vistastores.com/bar-stools?dir=desc&order=position http://www.vistastores.com/bathroom-lighting?p=2 and many more... So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.
Intermediate & Advanced SEO | | CommercePundit0