Still Cant Crawl My Site
-
I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us.
I did a fetch as google in our WM tools on our robots txt with success.
SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there.
What is going on here?
-
Hey Joel,
Happy Friday!
Sha
-
Hi Dana,
No problem. Glad you have sorted the problem now.
Have an awesome weekend
Sha
-
Hey Dana,
We've been corresponding in email, but I just wanted to update your thread here as well.
We don't use Amazon's bot, we use Amazon Web Service to host our crawler. If you are no longer blocking AWS you should be able to crawl OK moving forward.
Thanks!
Joel. -
Wish someone would've pointed that out days ago.
Thank you soooooo much for your great answer.
I don't understand though how or why seomoz is using amazons bot...
What if I don't want amazon accessing our site ( i dont). That means we can't use seomoz then??
-
we'll see how this goes. I've removed the blocks for amazonaws...
Thanks .
-
Hi Dana,
I believe SEOmoz utilizes Amazonaws services for crawling, (or at least they did a few months ago) so that may well be your problem.
The best (and quickest) way to confirm this is to go to the SEOmoz Help Hub and click the button at the top of the page to contact the Help Team directly.
Hope that helps,
Sha
-
Whats the web address?
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why did Moz crawl our development site?
In our Moz Pro account we have one campaign set up to track our main domain. This week Moz threw up around 400 new crawl errors, 99% of which were meta noindex issues. What happened was that somehow Moz found the development/staging site and decided to crawl that. I have no idea how it was able to do this - the robots.txt is set to disallow all and there is password protection on the site. It looks like Moz ignored the robots.txt, but I still don't have any idea how it was able to do a crawl - it should have received a 401 Forbidden and not gone any further. How do I a) clean this up without going through and manually ignoring each issue, and b) stop this from happening again? Thanks!
Moz Pro | | MultiTimeMachine0 -
Why is my crawl STILL in progress?
I'm a bit new here, but we've had a few crawls done already. They are always finished by Wednesday night. Our website is not large (by any means), but the crawl still says it's in progress now 3 days later. What's the deal here?!?
Moz Pro | | Kibin0 -
Is my Site Spam?
Recently google dropped our site a big time. Can some body tell me if my site is spammy. Our visibility was 67% and one of our top competitor had the visibility of 72%. www.aa-rental.com
Moz Pro | | tanveer10 -
Why did SEOMoz only crawl 1 page?
I have multiple campaigns and on a few of them SEOMoz has only crawled one page. I think this may have to do with how I set up the campaign. How do I get SEOMoz to crawl more than one page on these campaigns.
Moz Pro | | HermanAdvertising0 -
Moz crawling
Hi Everyone! I'm new to the SEOMoz and wanted to find out if there is a way to decrease the waiting time for the campaign crawl. I have made a lot of changes based on the first crawl and would like to see how these are reflected on the reports, but can't until the next crawl is performed. Any help would be greatly appreciated.
Moz Pro | | coremediadesign0 -
What tools should i use to research the following site
Hi i am researching the following site http://www.telegraph.co.uk/travel/ and i would like to know what keywords they are using to gain traffic and i would like to know what tools i need to use to research how they are achieving great success with google. i am new to the semoz tools so i would like help in using them to study this site and other sites, can anyone please let me know what i need to do and what tools i need to use to research this site and find the links to the site and how they are achieving great success many thanks
Moz Pro | | ClaireH-1848861 -
Crawl Report Technical Issue
I'm having a problem with our campaign especially the crawl report. Because the last update is Nov 4 and isn't supposed to be updated weekly? I already submit a helpdesk support ticket and even send a seperate e-mail regarding this issue but until now the report is still not updated. Anybody here can help me raise this issue/ Thanks.
Moz Pro | | shebinhassan0 -
SEOMoz only crawling 5 pages of my website
Hello, I've added a new website to my SEOmoz campaign tool. It only crawls 5 pages of the site. I know the site has way more pages then this and also has a blog. Google shows at least 1000 results indexed. Am I doing something wrong? Could it be that the site is preventing a proper crawl? Thanks Bill
Moz Pro | | wparlaman0