Website can't be crawled
-
Hi there,
One of our website can't be crawled. We did get the error emails from you (Moz) but we can't find the solution. Can you please help me?
Thanks,
Tamara
-
@Yenlo same issue. I am facing in my https://jeem.pk/ website. After the Google Core Update. This website was totally deindex from google. and the spam score were increase up to 70 percent.
-
Hello,
I was facing issue in my client website. after the last google update this website https://jeem.pk/ was drop the ranking. I can't fetch the issue. Now, mostly pages were deindexed.
Spam Score is increase after Google update. upto 70.
Let me guide I have 2 questions.
-
-
sometimes web designers when they first design a website they might put the no index on every page, obviously this needs to be removed at a later date if you want to get the website indexed
-
We had this problem before with the website, where the blog posts couldn't be crawled.
We would highly recommend, speak to an seo agency for advice, as this can negatively affect the organic SEO if the page can't be crawled and indexed.
We had this problem with a company that sells summerhouses; the blog posts couldn't get crawled and negatively affected the organic seo.
-
Hi Jo,
Thanks for checking! I will ask our website provider (HubSpot) if they block AWS or rogerbot.
Thanks,
Tamara -
Hi there,
Jo here from the Moz Help Team.
The best way to investigate issues like this, is to start with our troubleshooting guide here https://moz.com/help/moz-pro/site-crawl/crawl-troubleshooting
I have tried this myself and I've gotten to step 3 where I can see that there is an error when I try you site in this third party status checker.
This is an indication that crawlers like ours and this tools are being blocked from accessing your site at a server level.
Please check with your website admin to see if they are not blocking AWS, or rogerbot. Once you've fixed this with your website admin you can trigger a recrawl of your site if you have a Medium subscription or higher https://moz.com/help/moz-pro/site-crawl/overview
If you're still stuck please reach out to help@moz.com.
Best
Jo
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved How can I create a custom gift box using MOZ's products, and what are some creative ideas for designing the packaging?
Creating a custom gift box using MOZ's products can be a fun and thoughtful way to personalize a gift. Here's a step-by-step guide on how to do it, along with some creative packaging ideas: Step 1: Select the Products
Product Support | | MeltonMark8888
Browse MOZ's product catalog and select the items you want to include in the gift box. Consider the recipient's preferences, interests, and the occasion for which you're creating the gift box. Step 2: Choose the Box
Select a Galaxy Gift Box that will fit the products comfortably. It can be a plain cardboard box or a decorative one, depending on the overall aesthetic you're aiming for. Make sure the box is sturdy enough to protect the contents. Step 3: Gather Materials
Besides the products, gather materials for packaging, such as tissue paper, bubble wrap, or shredded paper for cushioning. You'll also need ribbon, twine, or other decorative elements for the finishing touches. Step 4: Arrange the Products
Arrange the chosen products inside the anniversary gifts for men . Place larger items at the back and smaller items at the front for visibility. Ensure that the arrangement looks aesthetically pleasing and well-balanced. Step 5: Add Filler
Add your chosen filler material to protect the products during shipping and to enhance the presentation. You can use colored tissue paper, shredded paper, or bubble wrap.
Gift Box Max0 -
Unsolved 403 crawl error
Hi, Moz( Also reported by GSC)have reported 403 crawl error on some of my pages. The pages are actually working fine when loaded and no visible issue at all. My web developer told me that some times error issues are reported on a working pages and there is nothing to worry about.
Product Support | | ghrisa65
My question is, will the 403 error have bad consequences on my SEO/Page ranking etc. These are some of the pages that have been reported with 403 error but loading fine: https://www.medistaff24.co.uk/hourly-home-care-in-evesham/ https://www.medistaff24.co.uk/contact-us/0 -
Crawl test
I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?
Product Support | | Karen_Dauncey0 -
Why didn't the Moz rankings change yesterday?
As of this morning, the Domain Authority section on our campaign still reads: "Next scheduled update: Jan 25, 2017 PST" and nothing has changed - it is still showing our old DA and how much it went up since last time. Is this a problem on our end, or is Moz experiencing difficulties with DA updates? Thanks in advance!
Product Support | | whiteonlySEO0 -
Why moz do not show website competitive metrics
why moz do not show website competitive metrics? meko.lv and why site meko.lv has high spam score? I dont see any reason for that??? metrics website is meko.lv NKpqhns
Product Support | | Mekounko0 -
Why aren't my keyword rankings updating
My dashboard or keyword rankings haven't updated in 2 months, is this a bug or am I doing something wrong? The account page indicates the campaigns are active. Derp.....
Product Support | | linhha0 -
Campaign lost connection to Google analytic's
Hi, I have a website that I has two Campaigns, one is its performance in the UK and the other is in America. Both of these project use to work and gather information from Google analytic's but recently the UK campaign has started saying it can not connect to Google analytic's and stopped providing information. The problem is both the UK and US campaigns use the same analytic's tracking code and the US campaign can still gather Google analytic's information so I know the tracking information is correct still.
Product Support | | AMA-DataSet
why can the UK campaign no longer connect properly? Thank you in advanced.0 -
Is it possible to split a moz account. We have 5 sites and one has been sold to another company. Can I split that site onto its own new account?
Is it possible to split a moz account. We have 5 sites and one has been sold to another company. Can I split that site onto its own new account?
Product Support | | BuyandSell0