Websites First Crawl - Over 2 Hour Suggested Wait
-
Hello SEOMoz!
We recently signed up for a free trial and on the pro dashboard it states the following.
"To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days."
It's been nearly 24 hours and we see no results under Crawl Diagnostics however we do under rankings.
Is this normal?
Thanks
-
Thanks!
Last time I checked the robots.txt was there. Anyway I have rcreated this.
Will try creating a new campaign and seeing how long it takes.
Update: worked fine after around 10 minutes crawl!
-
Hi Louis,
You are right, the first starter crawl should be complete within the first couple of hours. One reason it might take longer than normal is if the SEOmoz crawler, rogerbot, has trouble crawling your site.
I took a look inside our system and it appears the website associated with your campaign doesn't have a robots.txt file. Because the SEOmoz crawler is extra polite about crawling, whenever it doesn't find a robots.txt file, this can prevent it from crawling at all.
To fix this, create a text file called "robots.txt" and with the following information in it, and place it at the root directory of your site.
User-agent: *
Disallow:Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt
Unfortunately, it may be too late to add this to fix your starter crawl. So here are some things you can try:
- After adding the robots.txt file, try creating another (duplicate) campaign and seeing if that works.
- Or wait until your next crawl
- Contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO.
-
Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.
I just started one on a 30 page site, so I expect it will show results tomorrow.
-
So this means I will not get any data under crawl diagnostics for a week?
Why does it not give results for up to 250 pages as stated? The domain I selected for crawling has only 20 or so pages.
-
The full crawl diagnostics and rank reports are only generated once per week. So, I would give it at least 7 days before you see full data reported in your new campaign.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Shopify crawl issues
Hi Moz'ers, I am a total newcomer to this level of seo. Recently I transitioned to Shopify and I'm puzzled by why I'm getting 803 errors - incomplete crawl attempts due to server timing out. Wouldn't this have to do with Shopify? How would I go about fixing it? I'm also getting 804 - SSL issues, but I assume that will go away. Any advice? Thanks! Sharon
Moz Pro | | Sharon2016
www.ZeldasSong.com0 -
Is seomoz rogerbot only crawling the subdomains by links or as well by id?
I´m new at seomoz and just set up a first campaign. After the first crawling i got quite a few 404 errors due to deleted (spammy) forum threads. I was sure there are no links to these deleted threads so my question is weather the seomoz rogerbot is only crawling my subdomains by links or as well by ids (the forum thread ids are serially numbered from 1 to x). If the rogerbot crawls as well serially numbered ids do i have to be concerned by the 404 error on behalf of the googlebot as well?
Moz Pro | | sauspiel0 -
Website is not being indexed in Mozscape
Hi, We launched a website Mid-March. Strangely it's not yet being indexed by the Mozscape crawler (therefore I can't use tools like Open Site Explorer). Open Site Explorer keeps telling me: "No Data Availible for this URL". I don't think it's a crawl error of some sort. Google Webmaster tools tells me everything is fine (no crawl errors). Search Engine indexing is also fine. Any thoughts on this? Thanks.
Moz Pro | | Vendic0 -
Amount of Pages Crawled Dropped Significantly
I am just wondering if something changed with the SEOMoz crawler. I was always getting 10,000 or near 10,000 pages crawled. After the last two crawls I am ending up around 2500 pages. Has anything changed that I would need to look at it see if I am blocking the crawler or something else?
Moz Pro | | jeffmace0 -
404 errors in SEOMoz crawl tool
I currently have several 404 errors in the latest crawls from SEOMoz. Here is an example of the error. http://dealerplatform.com/blog/2011/10/23/videos-for-auto-dealers/www.dealerplatform.com/ In all cases the error is a result of www.dealerplatform.com being added to the real url. Anyone seen this before? The site is a wordpress mutlisite. I don't see where this incorrect link is showing up anywhere on the website. Any advice would be helpful. Thanks
Moz Pro | | Chris_Gregory0 -
Crawl diagnostic Notices for rel Canonical increased
Hello, We just signed up for SEO Moz, and are reviewing the results of our second web crawl. Our Errors and Warnings summary have been reduced, but our Notices for Rel Canonical have skyrocketed from 300 to over 5,500. We are using a WP with the Headway theme and our pages already have the rel=canonical along wiht rel=author. Any ideas why this number would go up so much in one week? Thank you, Michael
Moz Pro | | MKaloud0