SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New URL, new physical address, New Name. 30 point drop in Domain Authority. Yikes.
I have a client who is asking for SEO help after renaming their business, getting a new URL, and somehow having an address change (without moving to a new location...weird...I know). This has set them back big time in terms of their domain authority (they went from a 46 to a 15 in DA). The web developers they work with put a 302 redirect in place from their old URL (home page), which had 10,477 links from 52 root domains, to their new URL's home page. Open site explorer shows that they now have 5 links! We can improve some of the local search set backs from the name and address change with a citation audit and clean up, but the domain name change is a killer. So here's my question or questions, really: Do we need to manually rebuild links with partner websites? I know there is debate around the actual link juice passed along from a 302 vs a 301 redirect (despite what has been publicly stated by Google). Or is this just a waiting game while old links get recrawled?
Moz Pro | | TheKatzMeow1 -
Hiring for SEO
so sorry if i offend anyone here ...but dang SEO Experts are the used car sale guy of 1999! we have a small eCommerce company in Jacksonville, FL - and we are always looking for improve our SEO - we have been members of moz for almost 10 years now! We have learned alot with all the tools they offer, We have tried at least 5 local and national SEO consultants, agencies and consultants - NONE HAVE WORKED. The time has come for us to hire an in house SEO do you have any advise on how a small business that wants to grow our SEO can weed thru the thousands of SEO "GURU's" out there today - Everyone is an expert. On a side not if you want to move to Florida and help us grow - www.HolmesStamp.com/team
Moz Pro | | BryanCroft1 -
The pages that add robots as noindex will Crawl and marked as duplicate page content on seo moz ?
When we marked a page as noindex with robots like {<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noindex</a>" />} will crawl and marked as duplicate page content(Its already a duplicate page content within the site. ie, Two links pointing to the same page).So we are mentioning both the links no need to index on SE.But after we made this and crawl reports have no change like it tooks the duplicate with noindex marked pages too. Please help to solve this problem.
Moz Pro | | trixmediainc0 -
SEO moz Report Card
I just ran some on page report cards. As I was playing around with the tool I noticed that I would get different results if I used my primary domain vs a 2nd domain. The main difference was in how the tool was counting keywords on the page. The keyword used was 'vehicle inventory' Primary domain: www.brand-state.com/inventory.htm Title = 1, URL = 0, Meta = 1, H1 = 1, H2-4 = 1 Body =1, Strong = 1, IMG Alt = 1 Total = 7 2nd domain: www.company-name-brand.com/inventory.htm Title = 1, URL = 0, Meta = 1, H1 = 1, H2-4 = 2 Body =5, Strong = 4, IMG Alt = 2 Total = 13 I can understand if the keyword was in the domain, but it's not. So I'm wondering what is going on here - any help or suggestions on what to research would be a great help. Thank you!
Moz Pro | | gormaniavt0 -
How to make a best SEO strategy
Hi I want to make a seo strategy for website. what things I need and how to make is there any tool?
Moz Pro | | Kabooter0 -
Why does SEO Moz say we are ranked lower than appears on Google Searches?
We are currently ranked 18th for the Sydney Vet keyword in SEO Moz ranking tools, however in organic Google search we are ranked third. This search was conducted without Googles personalised results feature. Is this just an error? Or does it have something to do with Google Places not being counted in SEO Moz ranking tools? Any help would be much appreciated.
Moz Pro | | Peter.Huxley590 -
Where is the best SEO Glossary of Terms?
Where do you feel the best existing Glossary of SEO / SEM Terms explained currently resides. (and is there a better way to ask this question?) 🙂
Moz Pro | | iansears0 -
Is having the company's address in the footer (or header) of each webpage important for SEO?
Is having the company/office's address in the footer (or header) of each webpage important for SEO? Your page for the Geotarget tool says that having the address in this element helps search engines find your location. My question is, how important or relevant is this to SEO? How does knowing the address influence SEO? Is it best SEO practice to put the address in the footer of every webpage? http://www.seomoz.org/geotarget
Moz Pro | | richardstrange0