Website can't be crawled
-
Hi there,
One of our website can't be crawled. We did get the error emails from you (Moz) but we can't find the solution. Can you please help me?
Thanks,
Tamara
-
We had this problem before with the website, where the blog posts couldn't be crawled.
We would highly recommend, speak to an seo agency for advice, as this can negatively affect the organic SEO if the page can't be crawled and indexed.
We had this problem with a company that sells summerhouses; the blog posts couldn't get crawled and negatively affected the organic seo.
-
Hi Jo,
Thanks for checking! I will ask our website provider (HubSpot) if they block AWS or rogerbot.
Thanks,
Tamara -
Hi there,
Jo here from the Moz Help Team.
The best way to investigate issues like this, is to start with our troubleshooting guide here https://moz.com/help/moz-pro/site-crawl/crawl-troubleshooting
I have tried this myself and I've gotten to step 3 where I can see that there is an error when I try you site in this third party status checker.
This is an indication that crawlers like ours and this tools are being blocked from accessing your site at a server level.
Please check with your website admin to see if they are not blocking AWS, or rogerbot. Once you've fixed this with your website admin you can trigger a recrawl of your site if you have a Medium subscription or higher https://moz.com/help/moz-pro/site-crawl/overview
If you're still stuck please reach out to help@moz.com.
Best
Jo
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved How can I create a custom gift box using MOZ's products, and what are some creative ideas for designing the packaging?
Creating a custom gift box using MOZ's products can be a fun and thoughtful way to personalize a gift. Here's a step-by-step guide on how to do it, along with some creative packaging ideas: Step 1: Select the Products
Product Support | | MeltonMark8888
Browse MOZ's product catalog and select the items you want to include in the gift box. Consider the recipient's preferences, interests, and the occasion for which you're creating the gift box. Step 2: Choose the Box
Select a Galaxy Gift Box that will fit the products comfortably. It can be a plain cardboard box or a decorative one, depending on the overall aesthetic you're aiming for. Make sure the box is sturdy enough to protect the contents. Step 3: Gather Materials
Besides the products, gather materials for packaging, such as tissue paper, bubble wrap, or shredded paper for cushioning. You'll also need ribbon, twine, or other decorative elements for the finishing touches. Step 4: Arrange the Products
Arrange the chosen products inside the anniversary gifts for men . Place larger items at the back and smaller items at the front for visibility. Ensure that the arrangement looks aesthetically pleasing and well-balanced. Step 5: Add Filler
Add your chosen filler material to protect the products during shipping and to enhance the presentation. You can use colored tissue paper, shredded paper, or bubble wrap.
Gift Box Max0 -
I have been not using MOZ, my log in can give details,and i was charged for plan of 149$.please refund my money.never did used any tool,it s all inactive .
i have been not using MOZ, my log in can give details,and i was charged for plan of 149$.please refund my money.never did used any tool,it s all inactive .
Product Support | | rush70720 -
Can't reactivate account - duplicate transaction. Anyone experienced this?
as Q explains, my card expired, tried to renew subscription and get a message saying declined, duplicate transaction?
Product Support | | paulazoid0 -
Link to Moz Pro home 404s, and then I stumbled upon Moz's staging site...
I'm wary of giving more information as it's probably best that little to no one be able to access Moz staging for one reason or another. I should also mention that this was done very unintentionally on my part. That said, from the 404 from Moz Pro home, I tried to access "My QA", and that's when I realized that everything I did thereafter was within staging.moz.com (and posted a question similar to this one). Does Moz permit access to their staging site, or did I stumble upon a mistake - or Moz-stake, if you will?
Product Support | | Lumina0 -
Why can I not crawl this site
I wanted to add this site as new campaign: new.kbc.be But it won't accept it. Why?
Product Support | | KBC0 -
Number of pages crawled = 1; Why?
Since November, we've been trying to figure out why, when I select Crawl Diagnostics, my number of pages crawled is only 1. In mid-november, we changed our URL. That is, we went from www.example.com/home-page/ to www.example.com/new-home-page/. My first assumption was that I needed to re-create my Moz profile. That didn't fix it. The only crawl error we get is the no rel="cannonical" found -- but it's there. We find it on every page, including the home page. Our content shows up in search. Moz bar shows us info for every page. I just don't know what else to check. Everything else in my dashboard seems to look as expected. Specifically, I've turned to Crawl Diagnostics to find 4XX errors on our site. Typically we find one or two per week. Sometimes 0. Sometimes 4 or more. But it's been 0 since November. I highly doubt we've arrived at perfection. Any thoughts?
Product Support | | seo-nicole0 -
Can we get direct live chat or contact information? Response in the community is not helping and making the process really slow
Its been almost half of the month and our account status is still showing not upgraded. Again and again trying to contact moz via forum and waiting for the response is hampering our work. Kindly let us know how to contact via live chat?
Product Support | | ShaunPhilips0