Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What is the IP address of the MOZ crawlbot?
-
Our preproduction site is restricted to a specific IP range.
Prelaunch I'd like to use MOZ to check the site but this obviously isn't currently possible. Is there an IP range I can whitelist ?
-
Hi,
Moz is hosted on Amazon. See this sample log entry from today:
54.146.235.41 - - [17/Sep/2017:09:48:13 +0200] "GET /best-deals.html/page/12/ HTTP/1.0" 200 15803 "-" "rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-97@moz.com)"[XF 54.146.235.41]
That IP is part of Amazon's ASN.
whois -h whois.radb.net -- '-i origin AS16509' | grep ^route
You can either use something like mod_authz to allow the domain amazonaws.com to connect or you can get each IP from AS16509 and allow that.
Hope it helps.
Regards
Chris
- topic:timeago_earlier,about a year
-
Just to expand on this, I found the following quote from a Moz member of staff:
"I wish I could give you IP addresses, but they change all the time since we host Roger in the cloud. There's not even a reliable range of IPs to give you. You can totally whitelist the useragent rogerbot, but that's the only reliable information about the crawler you can go off of. I hope that helps but let me know if there's any other solution you can think of. Thank you! "
From a previous question: https://moz.com/community/q/is-there-a-whitelist-of-the-rogerbot-ip-addresses
-
Unfortunately this is all I can find: https://moz.com/help/guides/moz-procedures/what-is-rogerbot with no IP range shown.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz trust
Where do I find my sites ranking for a MOZ TRUST score?
Getting Started | Dec 29, 2020, 1:27 AM | Kuhliff1 -
How to rank website on google with help of Moz ?
I am new here and my question is how rank a website with moz because my website was not ranking in any keywords so plz guide me
Getting Started | Apr 30, 2021, 8:27 PM | RoyDigitalWorld0 -
Can Moz Monitor a JS Site?
We are building a new site that, on the blog landing page, uses JS to populate the individual blog article links on the page. The links are not viewable in the page source, but do appear once the page fully renders. After running an on-demand crawl of the site (in QA...not indexed yet), it appears that Moz isn't indexing these pages, and it also isn't reading other page elements that load later (like an H1 that is rendered in JS but not in page source). Are we going to be able to use Moz to track this site? Is there some setting to help?
Getting Started | Aug 25, 2019, 1:02 AM | Rodrigo-DC0 -
Moz not able to crawl our site - any advice?
When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?
Getting Started | Jun 27, 2024, 8:47 PM | MyFamilClubLtd1 -
Moz can't crawl my site.
Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up.
Getting Started | Jan 16, 2018, 10:30 AM | bluerinsevintage
Adam
0 -
MOZ for initial keyword research?
Can I use MOZ for initial keyword research? Can't create an AdWords campaign (no website yet to register) to get use of Google's Keyword Planner & my subscription to Wordtracker is showing me too many unrelated keywords... Peter
Getting Started | Mar 12, 2014, 11:46 AM | nodiffrei0 -
Moz's official stance on Subdomain vs Subfolder - does it need updating?
Hi, I am drawing your attention to Moz's Domain basics here: http://moz.com/learn/seo/domain It reads: "Since search engines keep different metrics for domains than they do subdomains, it is recommended that webmasters place link-worthy content like blogs in subfolders rather than subdomains. (i.e. www.example.com/blog/ rather than blog.example.com) The notable exceptions to this are language-specific websites. (i.e., en.example.com for the English version of the website)." I am wondering if this is still Moz's current recommendation on the subfolders vs subdomains debate, given that the above (sort of) implies that SE's may not combine ranking factors to the domain as a whole if subdomains are used - which (sort of) contradicts Matt Cutts last video on the matter ( http://www.youtube.com/watch?v=_MswMYk05tk ) which implies that this is not the case and there is so little difference that their recommendation is to use whatever is easiest. It would also seem to me that if you were looking through the eyes of Google, it would be silly to treat them differently if there were no difference at all other than subdomain vs subfolder as one of the main reasons a user would use a sud-domain is a technical on for which it would not make sense for Google to treat differently in terms of its algorithm. I notice that in terms of Moz, while most of the site uses subfolders, you do have http://devblog.moz.com/ - and I was wondering if this is due to a technical reason or conscious decision, as it would seem to me that the content within this section is indeed linkworthy (as it has external links pointing to it from external sources), therefore it would seem to not be following the initial advice that is posted in Moz's basics on domains. Therefore I am assuming it is due to a technical reason - or that Moz's adive is out of date with current Moz thinking, and is indeed in line with Matt C in that it doesn't matter. Cheers
Getting Started | Aug 8, 2022, 6:32 PM | James773 -
How long does it usually take Moz to populate information for a new Web site?
We recently launched (9/13/2013) an e-commerce Website and added the campaign to SEO MOZ. Week after week the Domain Rank is 1 and none of our keyword stats or link stats are populated. We have another Moz campaign that posts weekly updates and is doing extremely well. I'm just wondering how long it usually takes Moz to start populating all the analysis stats? I'm also wondering if there might be a campaign setting buried somewhere that I need to enable or maybe it just takes more than 5 weeks? Any insights would be much appreciated. Here's the new URL we need to track with MOZ: http://www.imsportshq.com
Getting Started | Oct 22, 2013, 8:25 PM | Tripper0