Crawl Diagnostics Report Lacks Information
-
When I look at the crawl diagnostics, SEOMoz tells me there are 404 errors.
This is understandable, because some pages were removed.
What this report doesn't tell me is how those pages were discovered.
This is a very important piece of information, because it would tell me there are links pointing to those pages, either internal or external. I believe the internal links have been removed.
If the report told me how if found the link, I would be able to take immediate action. Without that information, I have to go so a lot of investigation. And when you have a million pages, that isn't easy.
Some possibilities:
- The crawler remembered the page from the previous crawl.
- There was a link from an index page - i.e. it is in the database still
- There was an individual link from another story - so now there are broken links
- Ditto, but it in on a static index page
- The link was from an external source - I need to make a redirect
Am I missing something, or is this a feature the SEO Moz crawler doesn't have yet?
What can I do (other than check all my pages) to discover this?
-
OK thank you, Ralph
I can work on that.
-
I think it's the SEOMoz crawler, but what I have found is that the error reports are limited here whereas GWT is much bigger and shows the links leading to the error. My guess is that SEOMoz limit the number of crawl errors they show due to limitations set on their crawler i.e. while their crawl is comprehensive, it's not going to capture what Google does.
-
Thank you Ralph.
Yes, had it for years. So is this a GWT report? I thought it was SEOMoz !
No not IIS, Linux.
-
If you download the csv file for the crawl you can sort it by http status to get all of the 404 errors together. Then there is a specific column that contains the referrer that provides the information you are after.
-
This may be a silly question, but have you got Google Webmaster tools installed? That will show you the source of the errors.
If your site is on IIS then you should also use the awesome IIS SEO toolkit provided by Microsoft for free.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When I did my first crawl, I was given some errors.
Do I then need to re-crawl to make sure the errors were fixed accordingly?
Moz Pro | | immortalgamer0 -
No internal links showing up in OpenSiteExplorer report
Hi there A simple one. I was just looking at our site in opensiteexplorer and notice that it says we have no internal links at all: https://www.evernote.com/shard/s244/sh/dd8cc88f-fee4-4ba2-8ec0-f3f8d5c408db/8d3fc3d2aa6fc9d5406051bc91731402 Rather odd as we do. Is this a bug with OSE that anyone knows about or something else? Any ideas / thoughts gratefully received.
Moz Pro | | ArenaFlowers.com0 -
Report on a whole domain
Hi, I am wondering if I can run a report which gives me the OSE results for every page on a domain, as opposed to just a single page? Can't find this anywhere! thanks, Brian
Moz Pro | | VISISEEKINC0 -
Only one page has been crawled
I am running a campaing for three weeks now and first two crawls was ok but the last one is showing only one page crawled. the subdomain I am tracking is: www.cubaenmiami.com I have everything correct in my site. Regards Alex
Moz Pro | | esencia0 -
Wild fluctuation in number of pages crawled
I am seeing huge fluctuations in the number of pages discovered the crawl each week. Some weeks the crawl discovers > 10,000 pages and other weeks I am seeing 4-500. So, this week for example I was hoping to see some changes reflected for warnings from last weeks report (which discovered > 10,000 pages). However, the entire crawl this week was 448 pages. The number of pages discovered each week seems to go back and forth between these two extremes. The more accurate count would be nearer the 10,000 mark than the 400 range. Thanks. Mark
Moz Pro | | MarkWill0 -
"no urls with duplicate content to report"
Hi there, i am trying to clean up some duplicate content issues on a website. The crawl diagnostics says that one of the pages has 8 other URLS with the same content. When i click on the number "8" to see the pages with duplicate content, i get to a page that says "no urls with duplicate content to report". Why is this happening? How do i fix it?
Moz Pro | | fourthdimensioninc0 -
Is there a problem with Weekly Ranking report this week?
I just received my weekly ranking report. 2 Up, 0 down, 8 the same this week. However, last week my page was #2 for "innovation conference" and this week you report "not in top 50." My first thought was penalty, but when I searched and removed personalization, we were still #2 for that search query. So, something must be wrong with the report - either we're 1 down or "not in the top 50 is wrong."
Moz Pro | | KNect3650 -
Crawl Diagnostics bringing 20k+ errors as duplicate content due to session ids
Signed up to the trial version of Seomoz today just to check it out as I have decided I'm going to do my own SEO rather than outsource it (been let down a few times!). So far I like the look of things and have a feeling I am going to learn a lot and get results. However I have just stumbled on something. After Seomoz dones it's crawl diagnostics run on the site (www.deviltronics.com) it is showing 20,000+ plus errors. From what I can see almost 99% of this is being picked up as erros for duplicate content due to session id's, so i am not sure what to do! I have done a "site:www.deviltronics.com" on google and this certainly doesn't pick up the session id's/duplicate content. So could this just be an issue with the Seomoz bot. If so how can I get Seomoz to ignore these on the crawl? Can I get my developer to add some code somewhere. Help will be much appreciated. Asif
Moz Pro | | blagger0