Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
-
I'm curious as to how accurately rogerbot replicates google's searchbot
I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing.
I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors.
Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
-
Thanks for your response. I was beginning to think this question had been left to rot.
I'm not getting any errors in WMT. What is concerning is that Roger is returning almost 300 errors of dupe content, which is obviously a problem. Screaming frog is no longer finding the pages (they've been blocked in the robot.txt) I guess what I'm trying to ask here is how can I be sure that my dupe content has been effectively blocked from google's spider.
Is there anyway to check?
Thanks for your help.
-
I've see similar concerns from others, it seems "rogerbot" does ignore certain things that other bots consider.
Don't worry about it, if it's not being flagged in WMT it shouldn't be an issue.
Take Roger as a guide rather than an iron fist bot like googlebot.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are there any free (or paid) tools available online that download Meta Tags for ALL URL's of a website?
Hi, I am looking to run an On-Site audit for a website and I'm wondering if there are any tools available online that take the existing Meta Tags on ALL pages of a website and downloads them to a .CSV or .XLS. Would need Meta Title and Meta Description for all pages at the very least. Any suggestions are appreciated - looking for Free or Paid options. Thanks.
Moz Pro | | SEO5Team0 -
I have double-checked the rel canonical is properly employed on our page but the On Page Grader says it's not working?
I have double-checked the rel canonical is properly employed on our page but the On Page Grader says it's not working Here is the URL - http://www.solidconcepts.com/industries/aerospace-parts-manufacturing/ What is wrong with how we are doing things?
Moz Pro | | StratasysDirectManufacturing0 -
How can I cancel a running crawl test?
I put in two urls that were incorrect and now I need to cancel the report generation. Is there a way to do this? And if so, would I get my crawl-credits back? Are they cumulative?
Moz Pro | | krenerr0 -
No Crawl data in dashboard
For the second straight week, I have had no crawl data in my dashboard. It seems like the crawler erased all my results in the pro dashboard. Is there a way to manually recrawl my site, since I will have to wait another week to see if it comes back to earth? Thanks
Moz Pro | | bedwards0 -
Rankings in Google.be - 3 languages
As the site of my customers is in 3 languages, I also want to monitor the rankings in 3 languages. I do have the possibility to monitor them in seomoz: google.be english google.be dutch google.be french However, in the report (http://pro.seomoz.org/campaigns/227154/rankings) I do see the 3 columns, but the title is only google.be, WITHOUT the language selection. Not really helpfull... Any advice? oNDu9
Moz Pro | | nans0 -
Apparent discrepancy on overseas Google results in Rankings
Probably a very noob question this ... I'm based in the UK and have just set up a campaign which includes Google US. There seems to be a discrepancy between some of the rankings found by SEOMOZ and what I see when I simulate the actual search using Adwords > Ad Preview. For example, a particular term shows "not in the top 50" for Google US on the rankings page of my campaign in SEOMOZ. But my US colleagues say that when they run the search they see us on page 1. And when I simulate the search via Adwords Preview, I also see our site ranking well on page 1. I'd like to understand why the discrepancy 🙂 Thanks for any advice. Gord
Moz Pro | | Gord690 -
How do I get my crawl report?
I received a message that my crawl report is complete with a link - went to the link however - when I click on the icon that has the report name and the complete check mark nothing happens looked around can't find the results. Need to bid on this job so it would be helpful to know where to get it. Thanks for all you do. Wickey
Moz Pro | | Wickey0 -
Why isn't my Domain Authority much higher than my competitors'?
Looking at the Competitive Link Analysis report shows my domain outperforming my competitors on Domain MozRank, Domain MozTrust, total links, linking root domains, followed linking root domains, and linking c-blocks. But Domain Authority scores are very similar across all four sites with one competitor actually scoring higher than my domain. How should I unpick this result? I'm assuming that there must a problem with my link profile. Is that a valid assumption? Thanks, Tom fc7Ez.png
Moz Pro | | trbaldwin0