Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
-
I'm curious as to how accurately rogerbot replicates google's searchbot
I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing.
I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors.
Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
-
Thanks for your response. I was beginning to think this question had been left to rot.
I'm not getting any errors in WMT. What is concerning is that Roger is returning almost 300 errors of dupe content, which is obviously a problem. Screaming frog is no longer finding the pages (they've been blocked in the robot.txt) I guess what I'm trying to ask here is how can I be sure that my dupe content has been effectively blocked from google's spider.
Is there anyway to check?
Thanks for your help.
-
I've see similar concerns from others, it seems "rogerbot" does ignore certain things that other bots consider.
Don't worry about it, if it's not being flagged in WMT it shouldn't be an issue.
Take Roger as a guide rather than an iron fist bot like googlebot.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawling doesn't show all of my Backlinks
Hello, I'm trying to make an SEO backlinks report on my website When using the Link Explorer, I see only a few backlinks while I have much more backlinks on this website. Anyone has an idea about how to fix this issue. How can I check and correct this? My website is www.signsny.com.
Moz Pro | | signsny1 -
Why my site so poorly ranks on Google
Hi 🙂 I hope you can help me for my site http://www.majordroid.com/ I really do not understand why my site so poorly ranks on Google. What can I do about that? Are there any errors or something? What can be done about that? Also when I go to the Moz Tools - Open Site Explorer to check how many backlinks I have... It says: 12 Total Links but according to the Google Webmaster tools I have in total 334 links and 138 links to the homepage. see attachment Why Open Site Explorer does not see all the links to my site, which is a possible reason? And last question, what I need to do to remove pages from Google search like this: <cite class="_Rm">/page/2</cite>. <cite class="_Rm">/page/3 and so on...</cite> Thank you 🙂 aK0x345.jpg
Moz Pro | | ivek19870 -
'Too many on-page links': doesn't add up...
Hi All, I'm fairly new to the site, have had a full site crawl and am somewhat confused by a large number of pages (82) reported as showing 'too many on page links'. Mine is a new e-commerce site selling a project management methodology. I have about 80 pages which describe individual templates and contain internal links to additional, related content on the site. The crawl warnings define too many on page links as roughly 100 "on any given page" yet almost none of the pages flagged contain more than 20-30 on links. Any thoughts and ideas about (a) the degree to which this will aeffect my ranking in practice and (b) how to resolve it gratefully received! Thanks in advance, Felix
Moz Pro | | RomanCat0 -
Rankings in Google.be - 3 languages
As the site of my customers is in 3 languages, I also want to monitor the rankings in 3 languages. I do have the possibility to monitor them in seomoz: google.be english google.be dutch google.be french However, in the report (http://pro.seomoz.org/campaigns/227154/rankings) I do see the 3 columns, but the title is only google.be, WITHOUT the language selection. Not really helpfull... Any advice? oNDu9
Moz Pro | | nans0 -
How do I force a crawl?
In the campaign overview it reads that 0 pages were crawled. Also got an email saying that a comprehensive audit will be done in 7 days. But the 'crawl in progress' wheel disappeared. I think it stopped, and I need to submit that report to substantiate buying the tool! How do I force a crawl?
Moz Pro | | ilhaam0 -
Is having the company's address in the footer (or header) of each webpage important for SEO?
Is having the company/office's address in the footer (or header) of each webpage important for SEO? Your page for the Geotarget tool says that having the address in this element helps search engines find your location. My question is, how important or relevant is this to SEO? How does knowing the address influence SEO? Is it best SEO practice to put the address in the footer of every webpage? http://www.seomoz.org/geotarget
Moz Pro | | richardstrange0 -
Backlink confusion
I've gotten confused when it comes to the Competitive Analysis Report in the Web App. I track in my own spreadsheet how many backlinks I accumulate. However, I had thought that the Competitive Analysis report would do that as well. But each month, the numbers of Total LInks- through Linking c-blocks changes up and down drastically. I clicked HELP to try and understand what each of these means, but could you please explain them to me? I was under the impression that External Followed Links are the ones I was going after, however, shouldn't my records match with the one in the web app? Somehow I went from 786 ext followed links in January, to 867 in Feb, to 804 in March, to 804 in April, then 973 in May, and dropped to 791 2 weeks later...I don't think I'm understanding how the web app calculates these numbers. And how come Domain Authority seems to fluctuate up and down? So what would be the best way to track my progress in accumulating backlinks? Would it be my own spreadsheet or do you have a tool for this? Because I'd like to see how many I've gotten since January, but don't understand how to see this in the webapp. HELP!
Moz Pro | | aircyclemegan0