Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Running 2 different domains
Hi guys, Something i'd like appreciate any opinion on... We own a local domain www.goodycard.co.nz but we're expanding overseas. I've purchased www.goodyhq.com as we'd like to drop the 'card' from URL. We can't get www.goodycard.com. The local site has a decent ranking, but obviously, the newly created HQ site doesn't. What's the best way to tie these in (or even ditch the .co.nz) but retain rank? And what's the best way for users to discover the HQ site? Any insight would help, as both sites are kind of different. One has a local member base and the other just sells software so it really depends on the region of search. Cheers!
Technical SEO | | r.moss0 -
Desktop & Mobile XML Sitemap Submitted But Only Desktop Sitemap Indexed On Google Search Console
Hi! The Problem We have submitted to GSC a sitemap index. Within that index there are 4 XML Sitemaps. Including one for the desktop site and one for the mobile site. The desktop sitemap has 3300 URLs, of which Google has indexed (according to GSC) 3,000 (approx). The mobile sitemap has 1,000 URLs of which Google has indexed 74 of them. The pages are crawlable, the site structure is logical. And performing a Landing Page URL search (showing only Google/Organic source/medium) on Google Analytics I can see that hundreds of those mobile URLs are being landed on. A search on mobile for a longtail keyword from a (randomly selected) page shows a result in the SERPs for the mobile page that judging by GSC has not been indexed. Could this be because we have recently added rel=alternate tags on our desktop pages (and of course corresponding canonical ones on mobile). Would Google then 'not index' rel=alternate page versions? Thanks for any input on this one. PmHmG
Technical SEO | | AlisonMills0 -
Fetch as Google Desktop Render Width?
What is Google's minimum desktop responsive webpage width? Fetch as Google for desktop is showing a skinnier version of our responsive page.
Technical SEO | | Desiree-CP0 -
Google Indexing Desktop & Mobile Versions
We have a relatively new site and I have noticed recently that Google seems to be indexing both the mobile and the desktop version of our site. There are some queries where the mobile version will show up and sometimes both mobile and desktop show up. This can't be good. I would imagine that what is supposed to happen is that the desktop version is the one that should be indexed (always) and browser detection will load the mobile version where appropriate once the user is on the site. Do you have any advice on what we should do to solve this problem as we are a bit stuck?
Technical SEO | | simonukss0 -
Do mobile and desktop sites that pull content from the same source count as duplicate content?
We are about to launch a mobile site that pulls content from the same CMS, including metadata. They both have different top-level domains, however (www.abcd.com and www.m.abcd.com). How will this affect us in terms of search engine ranking?
Technical SEO | | ovenbird0 -
Will Google Also Penalize Desktop Rankings If Your Site is Not Mobile Friendly?
Apologies if this question has already been answered. I was unable to find it. For desktop organic rankings: Will Google take into consideration mobile-readiness as a ranking factor? Thanks in advance for any reply, Kind regards,
Technical SEO | | Eric_Lifescript
Eric Darby1 -
Prevent mobile site from appearing in the sitelinks of desktop search
Hi, IWe have this mobile page that keeps on appearing in the google search. I even try to put it in the robots.txt to disallow the crawler but still it keeps on popping on the search results. How can I prevent it from displaying?
Technical SEO | | shebinhassan0 -
I think I'm stuck in a 301 redirect loop
Hi all, I'm trying to correct some of my duplicate content errors. The site is built on Miva Merchant and the storefront page, /SFNT.html, needs to be permanently redirected to www.mydomain.com This is what my .htaccess file looks like: #RedirectPermanent /index.html http://dev.mydomain.com/mm5/merchant.mvc? RewriteEngine On RewriteCond %{HTTP_HOST} !^dev.mydomain.com$ [NC] RewriteRule ^(.*) http://dev.emydomain.com/$1 [L,R=301] DirectoryIndex index.html index.htm index.php /mm5/merchant.mvc redirect 301 /SFNT.html http://dev.mydomain.com/ RewriteCond %{QUERY_STRING} Screen=SFNT&Store_Code=MYSTORECODE [NC] When I use this code and navigate to http://dev.mydomain.com/SFNT.html the URL gets rewritten as http://dev.mydomain.com/?Screen=SFNT So I believe this is what's called a "redirect loop".... Can anyone provide any insight? I'm not a developer, but have been tasked with cleaning up the problems on the website and can use any input anyone is willing to offer. Thanks, jr
Technical SEO | | Technical_Contact0