Handling a Huge Amount of Crawl Errors
-
HI all,
I am faced with a crawl errors issue of a huge site (>1MiO pages) for which I am doing On-page Audit.
-
404 Erorrs: >80'000
-
Soft 404 Errors: 300
-
500 Errors: 1600
All of the above reported in GWT.
Many of the error links are simply not present on the pages "linked from". I investigated a sample of pages (and their source) looking for the error links footprints and yet nothing.
What would be the right way to address this issue from SEO perspective, anyway? Clearly. I am not able to investigate the reasons since I am seeing what is generated as HTML and NOT seeing what's behind.
So my question is: Generally, what is the appropriate way of handling this?
-
Telling the client that he has to investigate that (I gave my best to at least report the errors)
-
Engaging my firm further and get a developer from my side to investigate?
Thanks in advance!!
-
-
Usually an on page audit lists all of the problems and possible reasons why they are happening, not in depth info on how to fix all the issues. That is usually the next phase, "do you want me to work on the site or do you want your dev team to track down the cause of the issues and fix them"
It also depends what type of contract you have with him of course.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Getting 404 Errors on Fonts After Website Redesign
Hi, I recently redesigned my website and I changed a few URLs so I have been keeping an eye on 404 errors and setting up redirects as needed. However, my 404 error log keeps getting filled font related URLs like this: <colgroup><col width="960"></colgroup>
On-Page Optimization | | DohenyDrones
| /fonts/AvenirLTStd-Heavy.otf), url(/fonts/AvenirLTStd-Heavy.woff), url(/fonts/AvenirLTStd-Heavy.eot), url(/fonts/AvenirLTStd-Heavy.eot? |
| /fonts/FrutigerLTStd-Black.otf), url(/fonts/FrutigerLTStd-Black.woff), url(/fonts/FrutigerLTStd-Black.eot), url(/fonts/FrutigerLTStd-Black.eot? |
| /fonts/AvenirLTStd-Heavy.otf), url(/fonts/AvenirLTStd-Heavy.woff), url(/fonts/AvenirLTStd-Heavy.eot), url(/fonts/AvenirLTStd-Heavy.eot? |
| /fonts/AvenirLTStd-Book.otf), url(/fonts/AvenirLTStd-Book.woff), url(/fonts/AvenirLTStd-Book.eot), url(/fonts/AvenirLTStd-Book.eot? |
| /fonts/FrutigerLTStd-Bold.otf), url(/fonts/FrutigerLTStd-Bold.woff), url(/fonts/FrutigerLTStd-Bold.eot), url(/fonts/FrutigerLTStd-Bold.eot? |
| /fonts/FrutigerLTStd-Bold.otf), url(/fonts/FrutigerLTStd-Bold.woff), url(/fonts/FrutigerLTStd-Bold.eot), url(/fonts/FrutigerLTStd-Bold.eot? |
| /fonts/FrutigerLTStd-Black.otf), url(/fonts/FrutigerLTStd-Black.woff), url(/fonts/FrutigerLTStd-Black.eot), url(/fonts/FrutigerLTStd-Black.eot? |
| /fonts/AvenirLTStd-Book.otf), url(/fonts/AvenirLTStd-Book.woff), url(/fonts/AvenirLTStd-Book.eot), url(/fonts/AvenirLTStd-Book.eot? |
| /fonts/AvenirLTStd-Heavy.otf), url(/fonts/AvenirLTStd-Heavy.woff), url(/fonts/AvenirLTStd-Heavy.eot), url(/fonts/AvenirLTStd-Heavy.eot? | Any idea what is causing these 404 errors? Is this something that might hurt my SERP?0 -
Strange error on my website
Hi Wondering if anyone could help out here please. I tried to crawl my website using screaming frog and got a 500 error warning (odd considering it worked yesterday). I tried google speed test and got the following message - PageSpeed Insights received a 500 response from the server. To analyze pages behind firewalls or that require authentication. However, it also crawled the site and gave me a score of 70/100. It showed the site thumbnail, as normal. This was updated this morning and it showed the latest one so must have crawled. I can access the site, as can my 'people', the site is getting hits from real users in Google Analytics so must be working. Am very puzzled how the site can be working and not be working at the same time! If possible i would rather not share the url. The site is a test of concept and is very messy looking. it is more a test to see what happens if i do x... Many thanks Carl
On-Page Optimization | | daedriccarl0 -
Moz Crawl Shows Duplicate Content Which Doesn't Seem To Appear In Google?
Morning All, First post, be gentle! So I had Moz crawl our website with 2500 high priority issues of duplicate content, not good. However if I just do a simple site:www.myurl.com in Google, I cannot see these duplicate pages....very odd. Here is an example....
On-Page Optimization | | scottiedog
http://goo.gl/GXTE0I
http://goo.gl/dcAqdU So the same page has a different URL, Moz brings this up as an issue, I would agree with that. However if I google both URL's in Google, they will both bring up the same page but with the original URL of http://goo.gl/zDzI7j ...in other words, two different URL's bring up the same indexed page in Google....weird I thought about using a wildcard in the robots.txt to disallow these duplicate pages with poor URL's....something like.... Disallow: /*display.php?product_id However, I read various posts that it might not help our issues? Don't want to make things worse. On another note, my colleague paid for a "SEO service" and they just dumped 1000's of back-links to our website, of course that's come back to bite us in the behind. Anyone have any recommendations for a good service to remove these back-links? Thanks in advance!!0 -
Google Crawl Errors from vbseo change
We have vbseo setup on our site and for some reason a setting was changed unexpectedly and was un-noticed where it changed the URL of all the pages and so none of our pages were getting indexed by google any longer due to 401 errors. Most of our SE traffic fell off. We discovered the issue a couple weeks ago and we changed the setting back so that the URLs are the same as they were originally before but in Google webmasters it's still showing crawl errors and our search engine traffic hasn't recovered at all. We have sitemaps being sent daily.
On-Page Optimization | | RudySF0 -
How do I get rid of crawl errors?
I recently revamped all my category pages to make them Google friendly. I did a lot of keyword research and ended up deleting categories and renaming categories. Now Google Webmasters is showing a ton of 404 errors. How can I get rid of them? Do I have to remove the URL one by one with Google Webmasters? Is it something that won't hurt my rankings and will go away in time?
On-Page Optimization | | dealblogger0 -
How much content does Google Crawl on your site?
Hi, We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us? Thanks,
On-Page Optimization | | mdorville
Matt0 -
What's the best way to handle crawling of photo gallery?
When you have a photo gallery with many search filters and loads and loads of pages, is it best to block all the filters and use google's pagination code? Ex: http://photo.net/gallery/photocritique/filter This site has pages for many different queries. While the page titles are unique, the pages are showing duplicated content.
On-Page Optimization | | cakelady0 -
See screenshot: Is this an example of Canonical issue or am I making an error in judgement?
Hi Mozzers. Please see the constructed screen grab from Open site explorer. Do I have canonical Errors with this client? Thanks! jX7X6.jpg
On-Page Optimization | | Giggy0