Crawl Diagnostic | Starter Crawl taken 14hrs.. so far
-
We started a starter crawl 14hrs ago and it's still going, can anyone help on why this is taking so long, when it says '2 hrs' on the interface..
Thanks,
Rory
-
Hi Rory. Most of our help desk is on holiday today, since it's the Fourth of July in the states. We do have a record of your ticket and one other person who is having a slow starter crawl, and a help desk specialist is looking into this now. Sorry for the delays.
Keri
-
I've asked — now heard yet, think i'll wait to hear.
Thanks for your help, appreciate it.
-
Send an email to help (at) seomoz.org for someone to have a look.
-
It's a fairly big site, but it does say:
'To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days.'
There's no option to do anything else, like cancel, reset etc — it just says 'Starter crawl in progress', it's been 16hrs now + bit frustraing as needed to send this through to a client this morning.. Anyone from SeoMoz around to look into this?
-
And here is how you reset the crawl:
1. On your webserver, edit the robots.txt file.
2. Block the seomoz bot from crawling the site by blocking its access to the root.
You can do so by adding the following lines:
User-agent: rogerbot
Disallow: /
This would end the crawl session.
But, before you do this, it may a good idea to check if your site indeed has a lot of content and outgoing links?
-
Rory,
What is the sub-domain that you are crawling? It may just be that there is a lot of content to crawl.
-
How would I reset the crawl? I don't appear to have an option to?
-
Rory,
I would guess that this crawl session has hung-up; it would be a good idea to start a new session. The session could have been left in the middle due to a server side issue on your website or a temporary drop in connection between the API server and your website's server.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site not crawl?
hi all me allow rogerbot in robots.txt but rogerbot can't crawl my site my site: toska-co.ir
Moz Pro | | jahanidawodi0 -
In Crawl Diagnostics, length of title element is incorrect
Hey all, It appears the Moz crawler is misreading the number of characters in my website's page titles. It shows 72 characters for the following page's title element: http://giavan.com/products/orange-crystal-chain-necklace-with-drop The page title for this web page is: Orange Crystal Chain Necklace with Drop | Giavan which is 48 characters. As it stands, this page title is displayed at 48 characters in Google SERPs. I am getting "This Element is Too Long" issue on 925 pages, which is just about the entire site. These issues appeared after I added additional Shopify (Liquid) code to the page title. If you inspect the code, you will see title element looks a bit odd with extra spacing and line breaks. What I'd like to know is whether or not it's necessary to rewrite the Shopify code, for SEM purposes. My feeling is that it's okay because the page titles look fine in SERPs but those 925 Moz crawl errors are kind of scary. Thanks for your help!
Moz Pro | | RichAlbanese0 -
Moz & Xenu Link Sleuth unable to crawl a website (403 error)
It could be that I am missing something really obvious however we are getting the following error when we try to use the Moz tool on a client website. (I have read through a few posts on 403 errors but none that appear to be the same problem as this) Moz Result Title 403 : Error Meta Description 403 Forbidden Meta Robots_Not present/empty_ Meta Refresh_Not present/empty_ Xenu Link Sleuth Result Broken links, ordered by link: error code: 403 (forbidden request), linked from page(s): Thanks in advance!
Moz Pro | | ZaddleMarketing0 -
How can I cancel a running crawl test?
I put in two urls that were incorrect and now I need to cancel the report generation. Is there a way to do this? And if so, would I get my crawl-credits back? Are they cumulative?
Moz Pro | | krenerr0 -
I've got quite a few "Duplicate Page Title" Errors in my Crawl Diagnostics for my Wordpress Blog
Title says it all, is this an issue? The pages seem to be set up properly with Rel=Canonical so should i just ignore the duplicate page title erros in my Crawl Diagnostics dashboard? Thanks
Moz Pro | | SheffieldMarketing0 -
Why does Crawl Diagnostics report this as duplicate content?
Hi guys, we've been addressing a duplicate content problem on our site over the past few weeks. Lately, we've implemented rel canonical tags in various parts of our ecommerce store, over time, and observing the effects by both tracking changes in SEOMoz and Websmater tools. Although our duplicate content errors are definitely decreasing, I can't help but wonder why some URLs are still being flagged with duplicate content by our SEOmoz crawler. Here's an example, taken directly from our Crawl Diagnostics Report: URL with 4 Duplicate Content errors:
Moz Pro | | yacpro13
/safety-lights.html Duplicate content URLs:
/safety-lights.html ?cat=78&price=-100
/safety-lights.html?cat=78&dir=desc&order=position /safety-lights.html?cat=78 /safety-lights.html?manufacturer=514 What I don't understand, is all of the URLS with URL parameters have a rel canonical tag pointing to the 'real' URL
/safety-lights.html So why is SEOMoz crawler still flagging this as duplicate content?0 -
Can you change crawl day of week?
Can I somehow sync the day of the week for each of my campaigns' crawls, so that all campaigns are updated on the same day?
Moz Pro | | ATShock0 -
Why does my crawl report show just one page result?
I just ran a crawl report on my site: http://dozoco.com The result report shows results for just one page - the home page, but no other pages. The report doesn't indicate any errors or "do not follows" so I'm unclear on the issue, although I suspect user error - mine.
Moz Pro | | b1lyon0