Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
PWA for Desktop Site (Ecommerce)
Hi Folks, Need guidance about using PWA on desktop site. As I know PWA is basically used for mobile site to engage visitor more and let them surf your site like an app. Would it be good SEO practice to use PWA on desktop site(E-commerce site) by calling everything through Javascript and let google Crawler cache only site logo and Hide everything else?
Technical SEO | | Rajesh.Prajapati1 -
How big of a problem is this? - link cannonicalization I think?
Hello! I'm new here. My moz Pro account is flagging my website for having 282 duplicate pages, which it is saying are critical issues. I looked at this and it seems like the problem is that many of my pages are being indexed like: www.joeborders.com and joeborders.com and oeborders.com/index. I think this is an issue with link cannonicalization...right? I contacted my website builder/host a while ago and they said they don't have a way to to link cannonicalization....is this a huge problem?...Is there some way to do it that I'm missing? Should i get a new website builder/host? 😛 for reference, this is what my builder/host said when I asked them about it: "Good question, at the moment we don't offer it, I will add it to our feature request list, as I think it would be a good idea. In a traditional hosting environment this would be using a htaccess file, since we are in ruby on rails environment we would need come up with a custom solution."
Technical SEO | | joebordersmft0 -
Desktop in http and mobile in https
Any experience or advice you can share of having a mix set of pages/urls in one site/domain https and http e.g. mobile in https and desktop in http , (desktop version) http://mydomain/product1 (mobile version)https://m.mydomain.com/product1 att the same time some mobile pages still in http://m.mydomain.com/sectionA/ thanks
Technical SEO | | arnoldcr0 -
Does mobile usability issues reduce your ranking in desktop search results?
Google has been sending emails out about usability errors. Do they have an effect on the rankings for desktop search results?
Technical SEO | | navidash0 -
Mobile Usability update mobile and desktop versions
Hello, A number of our clients have both m.website.com and website.com versions of their site. Per the latest update coming in April from Google, do I need to make sure all clients with just a desktop version are mobile-friendly (according to WT). Also, do I need to make both versions for those clients that have both m.website.com and website.com mobile-friendly or just the m.website.com? Thank you!!!!
Technical SEO | | lfrazer0 -
Switchboard Tags - Multiple desktop pages pointing to one mobile page
I have recently started to implement switchboard tags to connect our mobile and desktop pages, and to ensure that our mobile pages show up in rankings for mobile users. Because our desktop site is much deeper in content than our mobile site, there are a number of desktop pages we would like to have point to one mobile page. However, with the switchboard tags, this poses a problem because it requires multiple rel=canonical tags to be placed on the one mobile page. I'm assuming this will either confuse the search engines, or they will choose to ignore the rel=canonical tag altogether. Any ideas on how to approach this situation other than creating an equivalent mobile version of every desktop page or implementing a user agent detection redirect?
Technical SEO | | JBlank0 -
Mobile URL parameter (Redirection to desktop)
Hello, We have a parallel mobile website and recently we implemented a link pointing to the desktop website. This redirect is happening via a javascript code and results in a url followed by this paramenter: ?m=off Example:
Technical SEO | | echo1
http://www.m.website.com redirects to:
http://www.website.com/?m=off Questions: Will the "http://www.website.com/?m=off" be considered duplicate content with "http://www.website.com" since they both return the same content? Is there any possibility that Google will take into consideration the url ending in "/?m=off"? How should we treat this new url? The webmaster tools URL parameter configuration at the moment isn't experiencing problems but should we submit the parameter anyway in order not to be indexed or should we wait first and see the error response? In case we should submit this for removal... what's the best way to do it? Like this? Parameter: ?m=off Does this parameter change page content seen by the user? - doesn't affect page content Any help is much appreciated.
Thank you!0 -
Mobile site rank on Google S.E. instead of desktop site.
Hello, all SEOers~ Today, I would like to hear your opinion regarding on Mobile site and duplicate contents issue. I have a mobile version of our website that is hosted on a subdomain (m instead www). Site is targeting UK and Its essentially the same content, formatted differently. So every URL on www exists also at the "m" subdomain and is identical content. (there are some different contents, yet I could say about 90% or more contents are same) Recently I've noticed that search results are showing links to our mobile site instead of the desktop site. (Google UK) I have a sitemap.xml for both sites, the mobile sitemap defined as follows: I didn't block googlebot from mobile site and also didn't block googlebot-mobile from desktop site. I read and watched Google webmaster tool forum and related video from Matt Cutts. I found many opinion that there is possibility which cause duplicate contents issue and I should do one of followings. 1. Block googlebot from mobile site. 2. Use canonical Tag on mobile site which points to desktop site. 3. Create and develop different contents (needless to say...) Do you think duplicate contents issue caused my mobile site rank on S.E. instead of my desktop site? also Do you think those method will help to show my desktop site on S.E.? I was wondering that I have multi-country sites which is same site format as I mentioned above. However, my other country sites are totally doing fine on Google. Only difference that I found is my other country sites have different Title & Meta Tag comparing to desktop site, but my UK mobile site has same Title & Meta Tag comparing to desktop. Do you think this also has something to do with current problem? Please people~! Feel free to make some comments and share your opinion. Thanks for reading my long long explanation.
Technical SEO | | Artience0