Crawl started on the 18th of December and still hasn't completed!
-
As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be?
Thanks
-
Thanks I'll give that a go!
-
Hi Laurance,
Thanks for writing in and sorry that your crawl never completed. I'm afraid the issue is that all crawlers are being blocked in the robots.txt file of the site and we aren't able to fully access the site to complete a crawl. If you were to update the robots.txt file to allow our crawler to access the site, we would than be able to get the crawl completed for you. You can use the user-agent rogerbot to specify our crawler or you can remove the disallow: / directive from the robots.txt file to allow us access. Once you've done that, let me know and I will have our engineers complete the crawl for you.
I look forward to hearing back soon.
-Chiaryn
-
Hi Laurence,
What do you mean the crawl still hasn't be completed? Do you mean not all your sites are indexed?
I suggest you checking up on your Google Webmaster Tools and see how your site is doing. Once you log in, click on your website then click on the Health tab on the left hand side. You can see all your crawl stats and errors there. Furthermore, you can use the "Fetch as Google" tool and ask Google to crawl your pages again.
Another thing is that if you have a lot of pages and your pages are not interlinking correctly, it may take weeks for Google to crawl all your pages. There is not set time limit on how long it takes. Maybe submit a sitemap to speed up the process of finding all your pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Couple of Moz's bugs
There are still some bugs in a new Moz: I can still find a lot of mentions of "SEOMoz", for example: in a footer, in a Q/A form (SEOMoz Resources), in a new question form (SEOmoz PRO Application, SEOmoz Tools, etc.) On a main form (http://moz.com/pro/home) sometimes my full name is not visible at all, sometimes my MozPoints are hidden (on a left top corner); There is not direct link from http://devblog.moz.com/ to a main website; Regards.
Moz Pro | | ditoroin0 -
Problems with Crawl Diagnostics/New Campaign
Hi all I added a new domain to my campaigns yesterday and got the usual message saying we will have a small set of results ready in a couple of hours, and the rest will be done in a week or so. 24 hours later, this same message is still showing. Anyone else experiencing this or is it just related to my domain? Many thanks. Carl
Moz Pro | | GrumpyCarl0 -
No Crawl data in dashboard
For the second straight week, I have had no crawl data in my dashboard. It seems like the crawler erased all my results in the pro dashboard. Is there a way to manually recrawl my site, since I will have to wait another week to see if it comes back to earth? Thanks
Moz Pro | | bedwards0 -
How can I see the URL's affected in Seomoz Crawl when Notices increase
Hi, When Seomoz crawled my site, my notices increased by 255. How can I only these affected urls ? thanks Sarah
Moz Pro | | SarahCollins0 -
Was there another issue crawling rankings on 9th October?
Again this month I don't appear to have rankings logged for 9th October. Was there an issue earlier in the month?
Moz Pro | | NathanP0 -
Crawl Test has taken over 5 days and still has yet to complete
I am running some crawls on some sites and I have a number still pending. I have one from 7 days ago, a couple from 6 days ago, and 1 from 5 days ago. The confusing thing is that I have run a few others in that same period that have finished already. Do I need to restart the crawls or cancel them and start over?
Moz Pro | | DRSearchEngOpt0 -
Bing Not Complete in Ranking Report
Curious as to what the "FRI" image means under Bing on my Keyword Ranking reports? I saw this on Friday and assumed it hadn't crawled yet (as it was to set to crawl on Friday Jan 27, 2012) but here we are Monday AM and I'm still seeing the same thing. Glf4F.jpg
Moz Pro | | QueenSt0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0