Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
IT is blocking AWS via a proxy in front of our server. We've tried allowing the "roberbot" user-agent but crawling functionality still isn't working in my Moz Pro account. Is there a more specific user-agent we can allow in our proxy software? Thank you.
Thank you for your response but this doesn't answer my question. We aren't blocking rogerbot using robots.txt. We need to allow it through the proxy in front of our web server by using the exact user-agent (case sensitive) that is being sent by rogerbot. We've tried "rogerbot" but that isn't working. Based on the 3rd party documentation you linked to there seem to be a variety of possibilities:
roberbot/1.0
roberBot/1.0
RogerBot/1.0
rogerBot
RogerBot
It would be great if Moz provided clear documentation on this.