How to get rogerbot whitelisted for application firewalls.
-
we have recently installed an application firewall that is blocking rogerbot from crawling our site. Our IT department has asked for an IP address or range of IP addresses to add to the acceptable crawlers. If rogerbot has a dynamic IP address how to we get him added to our whitelist? The product IT is using is from F5 called Application Security Manager.
-
Hi Joel,
Just wondering if you guys came up with a fix for this one? I have the same issue myself...
Thanks!
-
We have a similar issue where we block the Amazon cloudspace but want to allow rogerbot. Our firewall finds it much easier to block by IP range rather than user-agent (rogerbot).
Can you please tell us more about the solution you suggested?
-
Hi, our IP is changing all the time. We do have an option that may resolve your issue though. I'm going to create a ticket as I will need some personal information regarding your account.
Thanks,
Joel
-
Hi kgoss -
I'm not an SEOmoz employee - but just fyi, here are the IP addresses rogerbot used when accessing my site since the start of 2012:
50.17.63.219
184.72.211.58
50.19.77.29
174.129.177.94
184.73.93.243
107.20.64.186
23.22.26.208
107.22.19.89
50.19.188.211
50.19.195.34
107.22.107.114
107.22.137.134
216.244.72.3
75.101.205.127
67.202.42.32
216.176.191.232
23.20.247.207
216.244.72.9
23.22.48.203
216.176.191.234
184.73.13.203
107.20.11.69
50.17.96.67
184.73.137.220
184.73.22.42
184.72.175.148
67.202.13.192
23.21.33.19
107.22.0.173
107.22.157.76
50.16.164.143
50.16.43.91
107.21.143.222
216.176.191.201
107.20.91.74
50.16.88.111
204.236.213.120
184.73.118.141
50.17.38.180
107.21.150.218
216.244.72.10
216.244.72.11
216.244.72.12
23.22.53.56Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Getting keywords to rank on new landing pages
I've built new landing pages for a website, and have loaded them with researched keywords in the content, alt image attributes and metas etc - but, after a number of crawls, the keywords are currently being matched to other existing web pages on the website. Does anyone have any advice on 'unlatching' these keywords from these pages, and instead getting them to match with pages that have been optimised for them? Many thanks!
Moz Pro | | Darkstarr6660 -
Best Way To Write H1 to Get Max CTR?
Can anyone please share some guidelines regarding how to write the H1 subject of the post which can generate the maximum CTR.?. I noticed even when i able to rank on the featured snippet with good enough search volume the CTR is too low. Anyone share some expert advice please?
Moz Pro | | ndhsne450 -
Rogerbot did not crawl my site ! What might be the problem?
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore. Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems. What might be the problem? thanks for any tips!
Moz Pro | | inlinear
Holger rogerbot-did-not-crawl.PNG0 -
How to get the total external links to a page and total external links to the domain using Mozscape API? I could not see an option in the bit flags
Hi, I was trying to get the data for total external links to a page and total external links to the domain using Mozscape API but I can't see a bit flag which can do that. There are bit flags for external followed links to a page and external followed links to a domain but I wanted the total external links data, is there a way to do that using Mozscape API else I would end up copying the data manually from OSE which would be cumbersome and time consuming. Your help is highly appreciated.
Moz Pro | | HQP0 -
Is the Linkscape Index getting updated?
I know it only gets a big refresh once a month...The last being in the early part of January. But I released a map that went viral (front page of CNET, Wired, Scientific American) that has gotten me hundreds of links from different domains: https://www.google.com/search?q=perceptionbuilder.com+map I got these links in early to mid December and OSE is showing no data at all. Similarly, I have been doing some linkbuilding for clients and links that I got in October and November and those links aren't showing up either. Anyone else experiencing this? Many thanks, Matt
Moz Pro | | coppersix0 -
Does anyone know of a crawler similar to SEOmoz's RogerBot?
As you probably know SEOmoz had some hosting and server issues recently, and this came at a terrible time for me... We are in the middle of battling some duplicate content and crawl errors and need to get a fresh crawl of some sites to test things out before we are hit with the big one? Before I get a million thumbs downs- I love and will continue to use SEOmoz, just need something to get me through this week ( or until Roger is back! )!
Moz Pro | | AaronSchinke1 -
Still Getting Keyword Self-Cannibalization
I used the onsite optimization and made some changes to my post, but am still getting the Self-Cannibalization mention in my results. It lists this: "Real Estate Careers | Keller Williams | Career In Real Estate" as the link that is not supposed to be there. This is not a link, but the title of the post. What am I missing?
Moz Pro | | brentmitchell0
