Crawl started on the 18th of December and still hasn't completed!
-
As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be?
Thanks
-
Thanks I'll give that a go!
-
Hi Laurance,
Thanks for writing in and sorry that your crawl never completed. I'm afraid the issue is that all crawlers are being blocked in the robots.txt file of the site and we aren't able to fully access the site to complete a crawl. If you were to update the robots.txt file to allow our crawler to access the site, we would than be able to get the crawl completed for you. You can use the user-agent rogerbot to specify our crawler or you can remove the disallow: / directive from the robots.txt file to allow us access. Once you've done that, let me know and I will have our engineers complete the crawl for you.
I look forward to hearing back soon.
-Chiaryn
-
Hi Laurence,
What do you mean the crawl still hasn't be completed? Do you mean not all your sites are indexed?
I suggest you checking up on your Google Webmaster Tools and see how your site is doing. Once you log in, click on your website then click on the Health tab on the left hand side. You can see all your crawl stats and errors there. Furthermore, you can use the "Fetch as Google" tool and ask Google to crawl your pages again.
Another thing is that if you have a lot of pages and your pages are not interlinking correctly, it may take weeks for Google to crawl all your pages. There is not set time limit on how long it takes. Maybe submit a sitemap to speed up the process of finding all your pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is pagerank still a ranking factor for Google?
Hello Friends, Just want to confirm that the announcement by Google officials for no more pagerank update (Oct 14) is only for the Google toolbar update but pagerank still exists and still a ranking factor for Google? http://searchengineland.com/google-toolbar-pagerank-finally-officially-dead-205277 Please help. Regards
Moz Pro | | chakraseo0 -
No link data for my domains + still no adwords data
All 3 sites In my campaign have no link data analysis done. 5 to 6 months after you said you were working on it Adwords data is still not available in Keyword analysis Partial service is understandable for a few days but not for months. Beyond the nice screens, what data & services do you exactly provide and what are we paying for?
Moz Pro | | ResourceLab0 -
Why isn't there any link analysis data in my pro dashboard?
Hi! I've had a pro account for almost a month now and there still isn't any data for my site in the competitive domain analysis section of the pro dashboard. Am I doing something wrong? Bvxj4.png
Moz Pro | | rhmio0 -
I can't find any inbound links that I know I have on open site, ahref or majestic. Does anyone know why?
Since the linkscape update before Christmas I have built a couple of links to fairly high quality sites. I can't find them on open site though or the other tools I mentioned. I'm a bit concerned there is an issue with my site. Does anyone have any idea why? I'm stumped. My site: www.emporiumofmanliness.co.uk
Moz Pro | | EmpofMan0 -
Crawl Diagnostics - Canonical Question
On one of my sites I have 61 notices for Rel Canonical. Is it bad to have these or is this just something that's informative?
Moz Pro | | kadesmith0 -
Changing the Timeframe of Historical Crawl Data
Hello, Just read a great post about the implications of duplicate content for sites after the most recent Panda update: http://www.seomoz.org/blog/duplicate-content-in-a-post-panda-world?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+seomoz+(SEOmoz+Daily+Blog) In the post is an image or crawl data history that shows months, not days or weeks, worth of trending data as it relates to duplicate content. So my question is this: How do I change my view/date range on my own campaigns so that I can view the trailing months of data rather than only what seems to be the past 4 weeks or so? This would really help me identify the impact of some on page changes we've recently made for a client. Many Thanks, Jared
Moz Pro | | surjm0 -
Analysing competitors' backlinks?
I am doing some competitor analysis and one thing that I want to know is how many links my competitors have and how many different domains link to their sites. I got some figures on this both from the SEO Moz toolbar and from Majestic SEO. My problem is that the figues are massively different. E.g. SEO Moz says 9657 links from 95 domains, Majestic says 4895 links from 1722 domains. Who do I believe? And where do each of these tools get their data from?
Moz Pro | | mascotmike0 -
Scheduling crawls between certain time periods
Hi, today SEOMoz crawled our site and it interfered with an email campaign that we sent out and pretty much brought our site to a crawl (seoMoz even reported numerous 4XX errors). Is there a way to tell the crawler to only allow indexing between certain time periods?
Moz Pro | | RugsUSA0