undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Product Support
  3. Website can't be crawled

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Website can't be crawled

Product Support
5
8
1.1k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Yenlo
    Yenlo last edited by Sep 2, 2019, 7:30 AM

    Hi there,

    One of our website can't be crawled. We did get the error emails from you (Moz) but we can't find the solution. Can you please help me?

    Thanks,

    Tamara

    Jeem0912 amsan 2 Replies Last reply Jun 12, 2024, 9:19 AM Reply Quote 0
    • Jeem0912
      Jeem0912 @Yenlo last edited by Jun 12, 2024, 11:27 AM

      @Yenlo same issue. I am facing in my https://jeem.pk/ website. After the Google Core Update. This website was totally deindex from google. and the spam score were increase up to 70 percent.

      1 Reply Last reply Reply Quote 0
      • Jeem0912
        Jeem0912 last edited by Jun 12, 2024, 10:58 AM

        Hello,

        I was facing issue in my client website. after the last google update this website https://jeem.pk/ was drop the ranking. I can't fetch the issue. Now, mostly pages were deindexed.

        Spam Score is increase after Google update. upto 70.

        Let me guide I have 2 questions.

        1 Reply Last reply Reply Quote 0
        • amsan
          amsan @Yenlo last edited by Jun 12, 2024, 9:19 AM

          @Yenlo Same issue I have Been Facing while getting Data About Chewy com i tried Alot to Fix But didnt Get the Solutions so Finally Here I See Same Issue can ayone Guide me too.

          1 Reply Last reply Reply Quote 0
          • sarahwalsh
            sarahwalsh last edited by Jun 11, 2024, 6:49 PM

            sometimes web designers when they first design a website they might put the no index on every page, obviously this needs to be removed at a later date if you want to get the website indexed

            1 Reply Last reply Reply Quote 0
            • topic:timeago_earlier,2 months
            • sarahwalsh
              sarahwalsh last edited by Apr 9, 2024, 2:21 PM

              We had this problem before with the website, where the blog posts couldn't be crawled.

              We would highly recommend, speak to an seo agency for advice, as this can negatively affect the organic SEO if the page can't be crawled and indexed.

              We had this problem with a company that sells summerhouses; the blog posts couldn't get crawled and negatively affected the organic seo.

              1 Reply Last reply Reply Quote 0
              • topic:timeago_earlier,5 years
              • Yenlo
                Yenlo @jocameron last edited by Sep 6, 2019, 5:53 AM Sep 6, 2019, 5:53 AM

                Hi Jo,

                Thanks for checking! I will ask our website provider (HubSpot) if they block AWS or rogerbot.

                Thanks,
                Tamara

                1 Reply Last reply Reply Quote 0
                • jocameron
                  jocameron Staff last edited by Sep 2, 2019, 8:55 AM Sep 2, 2019, 8:55 AM

                  Hi there,

                  Jo here from the Moz Help Team.

                  The best way to investigate issues like this, is to start with our troubleshooting guide here https://moz.com/help/moz-pro/site-crawl/crawl-troubleshooting

                  I have tried this myself and I've gotten to step 3 where I can see that there is an error when I try you site in this third party status checker.

                  This is an indication that crawlers like ours and this tools are being blocked from accessing your site at a server level.

                  Please check with your website admin to see if they are not blocking AWS, or rogerbot. Once you've fixed this with your website admin you can trigger a recrawl of your site if you have a Medium subscription or higher https://moz.com/help/moz-pro/site-crawl/overview

                  If you're still stuck please reach out to help@moz.com.

                  Best

                  Jo

                  Yenlo 1 Reply Last reply Sep 6, 2019, 5:53 AM Reply Quote 0
                  • 1 / 1
                  1 out of 8
                  • First post
                    1/8
                    Last post

                  Got a burning SEO question?

                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                  Start my free trial


                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Azurius

                    Unsolved Crawling only the Home of my website

                    crawl error crawling crawl

                    Hello,
                    I don't understand why MOZ crawl only the homepage of our webiste https://www.modelos-de-curriculum.com We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why? We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.

                    Product Support | Nov 7, 2023, 11:43 AM | Azurius
                    0
                  • sbfhjsdfsdbfh

                    Moz Does Not Show Any Data Of My Website

                    Four month ago I launched my website stylistfontgenerator.com , and i write about latest fonts which we used in our blogs,artices,letters. Well, when i check stats of my website in moz, it doest not show ranking keyword in moz.
                    What is this issue, and how i resolved this issue. Thanks
                    Alex!//

                    Product Support | Nov 28, 2023, 3:38 AM | sbfhjsdfsdbfh
                    0
                  • Elchanan

                    False 5xx Errors for ColdFusion website

                    For several years month after month MOZ crawl reports 5xx errors on many pages. Almost every time all the pages work fine as fa as i could see. Google webmaster tools does not notice any errors. Could anyone explain how to fix this situation? Should i get a refund from MOZ?

                    Product Support | Jan 3, 2019, 6:03 AM | Elchanan
                    0
                  • LavM

                    Can I delete a question that I previously posted on Moz?

                    I posted a question with links to a website. That question now appears in google search when I search for that website, which I don't want. Can I delete the question I posted on Moz?

                    Product Support | Oct 16, 2018, 6:16 AM | LavM
                    1
                  • Karen_Dauncey

                    Crawl test

                    I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?

                    Product Support | Sep 24, 2018, 10:57 PM | Karen_Dauncey
                    0
                  • Mandiram

                    Crawl error robots.txt

                    Hello, when trying to access the site crawl to be able to analyze our page, the following error appears: **Moz was unable to crawl your site on Nov 15, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. Can help us? Thanks!

                    Product Support | Jul 15, 2023, 8:26 AM | Mandiram
                    0
                  • Firestarter-SEO

                    How to block Rogerbot From Crawling UTM URLs

                    I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like: User-agent: rogerbot Disallow: /?utm_source* This does not seem to be working. Any ideas?

                    Product Support | Mar 30, 2017, 2:22 PM | Firestarter-SEO
                    0
                  • potterharry

                    I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.

                    Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.

                    Product Support | May 27, 2015, 9:53 AM | potterharry
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.