Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Desktop in http and mobile in https
Any experience or advice you can share of having a mix set of pages/urls in one site/domain https and http e.g. mobile in https and desktop in http , (desktop version) http://mydomain/product1 (mobile version)https://m.mydomain.com/product1 att the same time some mobile pages still in http://m.mydomain.com/sectionA/ thanks
Technical SEO | | arnoldcr0 -
Fetch as Google Desktop Render Width?
What is Google's minimum desktop responsive webpage width? Fetch as Google for desktop is showing a skinnier version of our responsive page.
Technical SEO | | Desiree-CP0 -
What domain name do you think is better for SEO: sirocco-webdesign.com or sirocco-web-design.com?
Hello, I would appreciate it very much if you share with me your thoughts on what domain name I'd better pick out in terms of productive SEO: sirocco-webdesign.com or sirocco-web-design.com? I know hyphens are not good, but second domain looks better, I think.
Technical SEO | | kirupa0 -
Does mobile usability issues reduce your ranking in desktop search results?
Google has been sending emails out about usability errors. Do they have an effect on the rankings for desktop search results?
Technical SEO | | navidash0 -
Does Google think My main keyword is my Company name due to over use in anchor text?
OK, i have recently noticed google is adding a keyword to my titles... I have attached an image which shows the results from a google site:mywebsite search... 8 out of the 10 results on the screen shot clearly have the keyword ( - CCTV Cameras) added to the end of the title. Now a lot of those pages are support pages i.e "Help & Support", "returns" etc etc.. nothing really to do with cctv cameras. The only thing i can think of, is the most common anchor text used on backlinks to my website (according to siteexplorer) is the keyword cctv cameras... (i am currently working on removing a load of the backlinks to my site due to what i think is some kind of google penalty as the anchor text is all keywords etc...) But it seems strange if google has started to discount these backlinks, that it then uses that keyword to add to my titles - unless of course, google thinks that it is my brand name... Any body have any thoughts on this? FqwooQY.gif
Technical SEO | | isntworkdull1 -
Duplicate content issues, I am running into challenges and am looking for suggestions for solutions. Please help.
So I have a number of pages on my real estate site that display the same listings, even when parsed down by specific features and don't want these to come across as duplicate content pages. Here are a few examples: http://luxuryhomehunt.com/homes-for-sale/lake-mary/hanover-woods.html?feature=waterfront http://luxuryhomehunt.com/homes-for-sale/lake-mary/hanover-woods.html This happens to be a waterfront community so all the homes are located along the waterfront. I can use a canonical tag, but I not every community is like this and I want the parsed down feature pages to get index. Here is another example that is a little different: http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html?feature=without-pool http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html?feature=4-bedrooms http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html?feature=waterfront So all the listings in this community happen to have 4 bedrooms, no pool, and are waterfront. Meaning that they display for each of the parsed down categories. I can possible set something that if the listings = same then use canonical of main page url, but in the next case its not so simple. So in this next neighborhood there are 48 total listings as seen at: http://luxuryhomehunt.com/homes-for-sale/windermere/isleworth.html and being that it is a higher end neighborhood, 47 of the 48 listings are considered "traditional listings" and while it is not exactly all of them it is 99%. Any recommendations is appreciated greatly.
Technical SEO | | Jdubin0 -
Mobile site rank on Google S.E. instead of desktop site.
Hello, all SEOers~ Today, I would like to hear your opinion regarding on Mobile site and duplicate contents issue. I have a mobile version of our website that is hosted on a subdomain (m instead www). Site is targeting UK and Its essentially the same content, formatted differently. So every URL on www exists also at the "m" subdomain and is identical content. (there are some different contents, yet I could say about 90% or more contents are same) Recently I've noticed that search results are showing links to our mobile site instead of the desktop site. (Google UK) I have a sitemap.xml for both sites, the mobile sitemap defined as follows: I didn't block googlebot from mobile site and also didn't block googlebot-mobile from desktop site. I read and watched Google webmaster tool forum and related video from Matt Cutts. I found many opinion that there is possibility which cause duplicate contents issue and I should do one of followings. 1. Block googlebot from mobile site. 2. Use canonical Tag on mobile site which points to desktop site. 3. Create and develop different contents (needless to say...) Do you think duplicate contents issue caused my mobile site rank on S.E. instead of my desktop site? also Do you think those method will help to show my desktop site on S.E.? I was wondering that I have multi-country sites which is same site format as I mentioned above. However, my other country sites are totally doing fine on Google. Only difference that I found is my other country sites have different Title & Meta Tag comparing to desktop site, but my UK mobile site has same Title & Meta Tag comparing to desktop. Do you think this also has something to do with current problem? Please people~! Feel free to make some comments and share your opinion. Thanks for reading my long long explanation.
Technical SEO | | Artience0 -
I think my ranking report is not accurate.
I think my ranking are not accurate... It says this but it think it is fault. Under here i put what it says. Is it possible that i get the wrong results? The site is www.mando-control.es And an other questions is : when people in spain search do they automaticly search on google.es or google.com ? | Mando a distancia
Technical SEO | | seoroyal
| Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| Mando distancia | pending | Moved unknown 0 spaces | 2 | Moved unknown 0 spaces |
| Mandos a distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancias Keyword History Ranking Analysis | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces | | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| Mando distancia | pending | Moved unknown 0 spaces | 2 | Moved unknown 0 spaces |
| Mandos a distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancias Keyword History Ranking Analysis | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |0