Still Cant Crawl My Site
-
I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us.
I did a fetch as google in our WM tools on our robots txt with success.
SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there.
What is going on here?
-
Hey Joel,
Happy Friday!
Sha
-
Hi Dana,
No problem. Glad you have sorted the problem now.
Have an awesome weekend
Sha
-
Hey Dana,
We've been corresponding in email, but I just wanted to update your thread here as well.
We don't use Amazon's bot, we use Amazon Web Service to host our crawler. If you are no longer blocking AWS you should be able to crawl OK moving forward.
Thanks!
Joel. -
Wish someone would've pointed that out days ago.
Thank you soooooo much for your great answer.
I don't understand though how or why seomoz is using amazons bot...
What if I don't want amazon accessing our site ( i dont). That means we can't use seomoz then??
-
we'll see how this goes. I've removed the blocks for amazonaws...
Thanks .
-
Hi Dana,
I believe SEOmoz utilizes Amazonaws services for crawling, (or at least they did a few months ago) so that may well be your problem.
The best (and quickest) way to confirm this is to go to the SEOmoz Help Hub and click the button at the top of the page to contact the Help Team directly.
Hope that helps,
Sha
-
Whats the web address?
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why isnt my site in top 10 SERPs?
Our website, https://www.tradewindfinance.com, hasnt improved ranking for keyword " international trade finance companies." we average around the 10th page of Google SERPs. Our DA, PA, and links are higher thank the top 10 competitors. What are we doing wrong or what can I do to increase rank?
Moz Pro | | solution.advisor0 -
Tips on speeding up disavow process and still doing it right
Hello, I'm doing a 10-step disavow check using this article, this word document, and this corresponding spreadsheet template. Doing a disavow for a non-manual penguin penalty (paid links, doorway sites) It's taking a long time. Can you give me some pointers of how to speed things up? I want to do it right but I need to know how I can do it faster. Like tricks on domains to automatically disavow, what not to check, whatever tips you have. I'm using Cognitive SEO and manual checking. I'd like to avoid pricy tools. I'm on a mac. Thanks!
Moz Pro | | BobGW0 -
Why is my MOZ report only crawling 1 page?
just got this weeks MOZ report and it states that it have only crawled: Pages Crawled: 1 | Limit: 10,000 it was over 1000 a couple of weeks ago, we have moved servers recently but is there anything i have done wrong here? indigocarhire.co.uk thanks
Moz Pro | | RGOnline0 -
Crawl Diagnostics: Next crawl date is in the past
Hi - I have quite a few crawl diagnostic errors and warnings. I have attempted to fix many of them but noticed this note at the bottom of the crawl diagnostics chart: "Last Crawl Completed: Mar. 22nd, 2013 Next Crawl Starts: Mar. 29th, 2013" It looks like SEOMoz thinks the next crawl date is Mar 29th, 2013, which is two weeks ago. Is there any way to "force" the crawl and get it back on regular schedule? This may have happened when my account was disabled because my credit card expired...Thoughts?
Moz Pro | | 6thirty0 -
Open Site Explorer - Image Anchor Text
Hey there. I'm researching internal links and In my report I get alot of image links with the following text in the Link Anchor Text column: (img alt)(img)[No Anchor Text]. Is it because my images don't have alt-text or something else... How to optimize correctly? See attached screenshot. Best regards K87xLN8
Moz Pro | | nosuchagency0 -
Crawl Test taking 10+ days and still "In Progress" - normal or glitch?
I started a crawl test for my large site - WallStreetOasis.com - on June 20 and still have not received my results. It still says "Crawl in Progress" 10 days later. Does this seem odd or problematic, or is this normal?
Moz Pro | | WallStreetOasis.com0 -
Wild fluctuation in number of pages crawled
I am seeing huge fluctuations in the number of pages discovered the crawl each week. Some weeks the crawl discovers > 10,000 pages and other weeks I am seeing 4-500. So, this week for example I was hoping to see some changes reflected for warnings from last weeks report (which discovered > 10,000 pages). However, the entire crawl this week was 448 pages. The number of pages discovered each week seems to go back and forth between these two extremes. The more accurate count would be nearer the 10,000 mark than the 400 range. Thanks. Mark
Moz Pro | | MarkWill0 -
Excluding parameters from seomoz crawl?
I'm getting a ton of duplicate content errors because almost all of my pages feature a "print this page" link that adds the parameter "printable=Y" to the URL and displays a plain text version of the same page. Is there any way to exclude these pages from the crawl results?
Moz Pro | | AmericanOutlets0