Crawl started on the 18th of December and still hasn't completed!
-
As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be?
Thanks
-
Thanks I'll give that a go!
-
Hi Laurance,
Thanks for writing in and sorry that your crawl never completed. I'm afraid the issue is that all crawlers are being blocked in the robots.txt file of the site and we aren't able to fully access the site to complete a crawl. If you were to update the robots.txt file to allow our crawler to access the site, we would than be able to get the crawl completed for you. You can use the user-agent rogerbot to specify our crawler or you can remove the disallow: / directive from the robots.txt file to allow us access. Once you've done that, let me know and I will have our engineers complete the crawl for you.
I look forward to hearing back soon.
-Chiaryn
-
Hi Laurence,
What do you mean the crawl still hasn't be completed? Do you mean not all your sites are indexed?
I suggest you checking up on your Google Webmaster Tools and see how your site is doing. Once you log in, click on your website then click on the Health tab on the left hand side. You can see all your crawl stats and errors there. Furthermore, you can use the "Fetch as Google" tool and ask Google to crawl your pages again.
Another thing is that if you have a lot of pages and your pages are not interlinking correctly, it may take weeks for Google to crawl all your pages. There is not set time limit on how long it takes. Maybe submit a sitemap to speed up the process of finding all your pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz Pro crawl signaling missing canonical which are not?
Hi,
Moz Pro | | rolandvintners
I'm trying MozPro considering using it.
One of the tool which is appealing is the crawl and insights.
After quick use, I really question many of the alerts, for instance, I got a "missing canonical tag" on this url: https://vintners.co/wine/grawu_gto#2020 but when I check my markup, there's clearly a canonical tag: <link rel="canonical" href="https://vintners.co/wine/grawu_gto"> Anybody can explain?
I asked Moz Pro staff when being onboarded but didn't get an answer...
Honestly, I'm questioning the value of these crawls, or may be I miss something?0 -
First Campaign: No email when each step is completed?
I joined SEOMoz 6 days ago and set up my first campaign straight away. On my Campaign Overview, it says: "Here's what you can expect over the coming days (we'll send you an email when each step is completed". But I've not received an email saying that any of the steps are completed, so I haven't really used any of the tools yet as I wanted to wait until I had something to work with 🙂 Since it's almost been 7 days (will be 7 tomorrow), do I assume that most if not all of the stages have now been completed and I can get started? Thanks, Iain.
Moz Pro | | iainmoran0 -
Rogerbot did not crawl my site ! What might be the problem?
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore. Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems. What might be the problem? thanks for any tips!
Moz Pro | | inlinear
Holger rogerbot-did-not-crawl.PNG0 -
1 week has passed: Crawled pages still N/A
Roughly one week ago I went pro, and then I created a campaing for the smallish webshop that I'm employed at, however it doesn't seem to crawl. I've check our visitors log and while we find other bots such as google, bing, yandex and so fourth, seomoz bot hasn't been visible. Perhaps I'm looking for a normal useragent, ohwell, onwards. While I thought it might take time, as a small test I added a domain that I've owned for sometime but don't really use, that target site is only 17 pages, now this site was crawled almost within the hour, and I realised that our ~5000pages on the main campaing would take some time, but wouldn't the initial 250 pages be crawled by now? I should add, that I didn't add http:// to the original Campaing, but the one that got crawled I did. I cannot seem to change this myself inorder to spot if that's the problem or not. Anyone has any ideas, should I just wait or is there something I can activly do to force it to start rolling?
Moz Pro | | Hultin0 -
What's the future of SERP Tracking? And... Is SEOMoz's SERP Rank Tracking in compliance with Google Adwords API Terms of Service?
My question is: Is SEOMoz's SERP Rank Tracking in compliance with Google Adwords API Terms of Service? Background: The reason I ask is because Raven Tools is now removing their SERP Reporting tool because it uses scraped Google position data. So, it looks like SEO's will either have to find a new rank tracking tool or find new ways to traffic the effects that rankings have on a website traffic volumes. For instance, there is a way to get the position a search results was in Google when it was clicked. We could create a secondary profile in Google Analytics for each client and use a custom filter to record the position that the keywords was in when the search result was clicked ( http://www.seomoz.org/blog/show-keyword-position-using-filters-and-advanced-segments ) Or perhaps we'll have to use Google Webmaster Tools' SEO Report to get data somehow ( http://support.google.com/analytics/bin/answer.py?hl=en&answer=1308626 ) What are your thoughts on this? As you know, ranking data is still a great way to show clients if they are gaining or losing visibility in the search engines. It helps SEO's to report how effective their efforts have been. Because other ranking software companies uses Adwords API data to show the keyword search volume and advertiser competition of a keyword, they can not or eventually will not be able to use scraped ranking data any more. But, if another rank tracking tool out there doesn't need to be in compliance with the Adwords API TOS because they don't use that API to show search volume and advertiser competition, they can still technically provide their ranking data and not be violating any TOS, right? I'm just trying to understand the best way to continue reporting impact of organic keyword rankings on a website. Does the SEOMoz SERP Tracker comply with Adwords API? Is there another rank tracking tool out there that already is using Average Position data from the GWT SEO Report tool? Should we all just stop reporting rankings to clients altogether? Scott
Moz Pro | | OrionGroup2 -
'Appropriate Use of Rel Canonical', Critical Factor but appears correct on page
Hi, Trying to get the following page ranked unsuccessfully.... http://www.joules.com/en-GB/2/Collections-Quilted-Jackets/c01c02.r16.1 Instead a product page is being ranked, shown below.... http://www.joules.com/en-GB/Womens-Quilted-Jacket/Navy/M_HAMPTON/ProductDetail.raction When I run the on page report card it advises that the Rel Canonical tag needs to point to that page, but we have checked and it looks to be doing that already. Has anyone else had an issue like this? Thanks, Martin
Moz Pro | | rockethot0 -
Dismiss crawl diagnostics error
Hello everyone, Is there a way to dismiss some errors in the Crawl Diagnostics tool so they don't appear again? It happens so that some of the errors are never going to be fixed because of their nature. For example, 'Title too long' errors that point to some of the threads on my forum - it doesn't make sense to change the title of a thread posted by user just for the sake of the error disappearing from the 'Crawl Diagnostics' tool. 🙂 Otherwise the CD interface gets a little bit cluttered with errors which I will never fix anyway. I wonder how others deal with this problem. Thanks.
Moz Pro | | MaratM0 -
How to crawl the whole domain?
Hi, I have a website an e-commerce website with more than 4.600 products. I expect that Seomoz scan check all url's. I don't know why this doesn't happens. The Campaign name is Artigos para festa and should scan the whole domain festaexpress.com. But it crels only 100 pages I even tried to create a new campaign named Festa Express - Root Domain to check if it scans but had the same problem it crawled only 199 pages. Hope to have a solution. Thanks,
Moz Pro | | EduardoCoen
Eduardo0