Crawl depth seems off?
-
I'm reviewing my site crawl data and am seeing some very strange things such as:
- The homepage URL has a listed crawl depth of 2.
- Pages that are featured in the main site navigation (which is present on all pages, including homepage) are ranking at a crawl depth of 3.
What am I missing here? Shouldn't my homepage have a crawl depth of 0 or 1? Why would pages linked directly from my homepage have a crawl depth other than 1? (Single click from homepage to that page)?
Thank you!
-
Hi Samantha,
I set up a new campaign using the https:// version of the site and ran a new crawl, but I'm running into the same issue as before. Perhaps this is a bigger question of how site redirects work? I was under the impression that any large-scale redirects (such as from non-www to www or http to https across all pages) can affect crawl time/load time. Rereading your comment, it sounds like what you're saying is those redirects count as layers of crawl depth, as well. By the same token, I'm assuming any redirects (301's in particular) also add a layer of crawl depth.
So, my larger question then is: how can I maximize crawl depth if my site has been redirected from http to https? Will that "extra layer" of crawling always be there as long as the redirect is in place, or is there a way to compress/expedite how the crawl happens?
Thanks for your input on this!
-
Hi Samantha,
That makes sense, thank you. I'll set up a new campaign tracking with "https://" instead!
-
Hey there,
Sam from Moz's Help Team here!
So the thing to keep in mind when you set up a campaign at the root domain level is that we'll be starting the crawl from the http protocol (non-www). In this case - http://logic2020.com/. If you filter by crawl depth in your Site Crawl you'll see that URL with a crawl depth of 0.
It redirects to http://www.logic2020.com/ which has a crawl depth of 1. That URL then redirects again to https://www.logic2020.com/, which is listed with a crawl depth of 2 - hence why links we found on that page have a crawl depth of 3.
I hope this helps to clarify but let me know if you have any other questions!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
Site: www.kpmg.us Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter". Updated robots.txt to allow rogerbot full access: User-agent: rogerbot
Link Explorer | | KPMG-Search-Social
Disallow: Any ideas how to get roger to crawl my site????1 -
Crawling 4XX errors because of URL accented
Hello guys, I am experiencing crawling errors in Moz because the URLs read by spiders contain accents and special characters, which I know isn't best but yet my client needs to keep it. I know that Moz "uses percent encoding to parse the HTML in the source code, so any line breaks and spaces in your HTML links or sitemap links are converted to %0A and %20, causing a 404 error". Is there any way to avoid these errors happening in the dashboard? Or am I supposed to simply ignore it?
Link Explorer | | fran8751 -
Does the Moz Pro site crawl, crawl password protected sites?
So i asked Moz Pro site crawl to crawl my page, and a lot of issues came up - but for password protected sites. Does the Moz Pro site crawl do this? A lot of the issues, are not relevant for a site that is password protected.
Link Explorer | | Minlaering.dk0 -
804 error preventing website being crawled
Hi For both subdomains https://us.sagepub.com and https://uk.sagepub.com crawling is being prevented by a 804 error. I can't see any reason why this should be so as all content is served through https. Thanks
Link Explorer | | philmoorse0 -
Moz cannot crawl domain. Also OSE does not work properly on this specific domain?
Hi all, Moz cannot crawl the domein http://www.hoesjescases.nl.
Link Explorer | | Guapa_zwolle
When I open the crawl report I only see one line: <colgroup><col width="229"><col width="287"><col width="420"><col width="370"><col width="141"></colgroup>
| URL | Time Crawled | Title Tag | Meta Description | HTTP Status Code |
| http://www.hoesjescases.nl | 2015-10-05T12:20:48Z | 404 : Received 404 (Not Found) error response for page. | Error attempting to request page; see title for details. | 404 | Also when running OSE on this domain, Moz only can find 4 root domains while Majestic can find 91 domains. Google seems not to have any problems. What can be the problem for MOZ? Greetings!0 -
Only two pages crawled
Hi community, I'm having trouble viewing data for all of our pages. Moz is only crawling two of our pages, and we have a lot more than that. Any suggestions? Thanks, Emily
Link Explorer | | SkuidEmily0 -
Local listings aren't being crawled?
Hello, SEO brainiacs. Here is the problem I can't figure out:We have a lot of clients and all of them have local listings, such as Insider Pages, Yahoo local, Bing locals, Yelp, Yellow Pages etc etc etc.
Link Explorer | | seomozinator
And all of those local listing have company's website link. But none of those links are considered (found) in reports, such as opensite explorer or in Link Analysis Section of Moz PRO. I do know that MOZ crawl considers only high DA websites, but all of local listing domains have above 80-90 DA score, so they can't be NOT crawled. I will say that some local listings do appear on very few reports, but far not all of them. So, what can be the problem?0 -
Does OSE crawl our site often? Or do we have an other problem? (internal links)
I am wondering if i can find out when OpenSiteExplorer crawled our entire website for the last time.
Link Explorer | | wilcoXXL
A few months ago we added some internal textlinks to all of our productpages. Those are links to the brandpage of that particular product. Some of our brands do have up to 1500 products, so there should be at least 1500 internal links pointing to that brandpage. But OSE still gives me a wrong count. It says there is only 1 internal link pointing to that brandpage.
Is it because OSE never crawled our site again? Or am i missing something here?(maybe too many internal links to one specific brandpage is not okay?) I'll hope you guys can help me out with this one.0