undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    What is your Brand Authority?
    Moz

    What is your Brand Authority?

    Check yours now
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • SEO Q&A

      Insights & discussions from an SEO community of 500,000+.

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Product Support
  3. Crawling only the Home of my website

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Unsolved Crawling only the Home of my website

Product Support
crawl error crawling crawl
7
7
1.5k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Azurius
    Azurius Subscriber last edited by Oct 11, 2023, 7:53 AM

    Hello,
    I don't understand why MOZ crawl only the homepage of our webiste https://www.modelos-de-curriculum.com

    We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why?

    We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.

    Theoputra bnbderma 2 Replies Last reply Nov 7, 2023, 11:43 AM Reply Quote 0
    • bnbderma
      bnbderma @Azurius last edited by Nov 7, 2023, 11:43 AM

      @Azurius
      Certainly, I understand your concern about MOZ only crawling the homepage of your website despite adding it correctly and requesting a full crawl of all pages. It's frustrating when testing a tool before subscribing, and not getting the expected results can raise doubts about its functionality. To address this issue, I recommend double-checking your website's settings within the MOZ platform. Ensure that the website URL provided matches the actual structure of your site and that there are no typos or errors in the input. Additionally, review MOZ's documentation or contact their customer support to confirm if there are specific settings or configurations needed for a comprehensive crawl. Sometimes, minor adjustments in the tool's settings or the website's structure can make a significant difference in the crawling process. I hope this helps, and I wish you success in resolving this matter and making an informed decision about your subscription.

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,15 days
      • clippingnext
        clippingnext Banned last edited by Oct 23, 2023, 7:16 AM

        Crawling only the home page of your website is a common practice in web indexing and search engine optimization. This approach allows search engine bots to focus on your site's main landing page, ensuring that it's properly indexed and ranked. Here's how you can specify that only the home page should be crawled:

        Robots.txt File,
        Canonical Tag
        Sitemap.xml,
        Noindex Tags,
        Meta Robots Tag

        By implementing these methods, you can direct search engine crawlers to focus primarily on your website's home page, ensuring that it receives the most attention in terms of indexing and ranking. This can be particularly useful if your home page is the most important and relevant page for your website's SEO strategy.
        https://www.clippingnext.com/

        1 Reply Last reply Reply Quote 0
        • Theoputra
          Theoputra @Azurius last edited by Oct 20, 2023, 4:54 AM

          @Azurius i have the same issue,. I think the answer here is quite helpful

          1 Reply Last reply Reply Quote 0
          • educationedge012
            educationedge012 last edited by Oct 19, 2023, 11:56 AM

            Hello,

            There could be several reasons why MOZ is only crawling the homepage of your website, https://www.modelos-de-curriculum.com. Here are a few possibilities:

            Robots.txt file: Check your website's robots.txt file to ensure that it's not blocking MOZ's web crawlers from accessing other pages. Make sure there are no disallow rules that could restrict access to certain areas of your site.

            Nofollow tags: Ensure that your website doesn't have "nofollow" tags on internal links that may be preventing MOZ from following and crawling those links.

            JavaScript: If your website heavily relies on JavaScript for content rendering, MOZ may face difficulty crawling the content. Ensure that important content is accessible without JavaScript.

            Canonical tags: Check for canonical tags in your website's HTML. If you have specified the homepage as the canonical page for all other pages, this could limit MOZ's ability to crawl additional pages.

            Site structure: MOZ may have trouble crawling pages with complex or unconventional site structures. Ensure that your website follows standard navigation and linking practices.

            Crawl settings: Double-check the settings in your MOZ account to confirm that you have requested a full site crawl, not just the homepage.

            If you've verified all these aspects and MOZ is still not crawling your website correctly, you may want to reach out to their support team for further assistance. They can provide more specific guidance based on your account and settings.

            Testing the tool before subscribing is a wise approach to ensure it meets your needs. I recommend contacting MOZ's support for personalized assistance in resolving the crawling issue with your website. (PMP Exam Prep) (Canada PR) (Study abroad) (Project Management Life Cycle)

            1 Reply Last reply Reply Quote 0
            • talentwide
              talentwide last edited by Oct 19, 2023, 9:42 AM

              Hello,

              The reason Moz is only crawling the homepage of your website could be due to various factors. Here are a few possibilities:

              Robots.txt File: Check your website's robots.txt file to ensure that it doesn't block search engine crawlers from accessing specific pages.

              Meta Robots Tags: Make sure there are no "noindex" meta tags on your internal pages that might prevent them from being indexed.

              Crawl Restrictions: Moz may not have had enough time to crawl all the pages on your website. It might take some time for the tool to explore and index your entire site.

              Sitemap: Ensure that your website's sitemap is correctly submitted to Moz. A sitemap helps search engines find and index all the pages on your site.

              Internal Linking: Check if there are internal links from your homepage to other pages. A lack of internal links can make it harder for search engines to discover and crawl your site.

              Access Permissions: Make sure there are no access restrictions or password protection on certain pages.

              To resolve this issue and ensure Moz crawls all the pages of your website, consider checking and addressing these factors. If the problem persists, it might be helpful to reach out to Moz's customer support for specific assistance with your subscription trial.
              (Canada PR) (Study abroad) (PMP Exam Prep) (Reference Letter for Canada PR)

              1 Reply Last reply Reply Quote 0
              • scentorita
                scentorita last edited by Oct 17, 2023, 2:09 PM

                check your .htaccess if there is something that is blocking crawlers.

                1 Reply Last reply Reply Quote 0
                • 1 / 1
                1 out of 7
                • First post
                  1/7
                  Last post

                Got a burning SEO question?

                Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                Start my free trial


                Browse Questions

                Explore more categories

                • Moz Tools

                  Chat with the community about the Moz tools.

                • SEO Tactics

                  Discuss the SEO process with fellow marketers

                • Community

                  Discuss industry events, jobs, and news!

                • Digital Marketing

                  Chat about tactics outside of SEO

                • Research & Trends

                  Dive into research and trends in the search industry.

                • Support

                  Connect on product support and feature requests.

                • See all categories

                Related Questions

                • andrewrench

                  Unsolved Crawler was not able to access the robots.txt

                  robots.txt crawl error

                  I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?

                  Getting Started | Sep 2, 2024, 4:25 PM | andrewrench
                  0
                • seogod123234

                  Why MOZ just index some of the links?

                  crawl indexing

                  hello everyone i've been using moz pro for a while and found a lot of backlink oppertunites as checking my competitor's backlink profile.
                  i'm doing the same way as my competitors but moz does not see and index lots of them, maybe just index 10% of them. though my backlinks are commenly from sites with +80 and +90 DA like Github, Pinterest, Tripadvisor and .... and the strange point is that 10% are almost from EDU sites with high DA. i go to EDU sites and place a comment and in lots of case, MOZ index them in just 2-3 days!! with maybe just 10 links like this, my DA is incresead from 15 to 19 in less than one month! so, how does this "SEO TOOL" work?? is there anyway to force it to crawl a page?

                  Link Building | May 20, 2024, 3:33 PM | seogod123234
                  0
                • AKCAC

                  GoogleBot still crawling HTTP/1.1 years after website moved to HTTP/2

                  google http https indexation crawl

                  Whole website moved to https://www. HTTP/2 version 3 years ago. When we review log files, it is clear that - for the home page - GoogleBot continues to only access via HTTP/1.1 protocol Robots file is correct (simply allowing all and referring to https://www. sitemap Sitemap is referencing https://www. pages including homepage Hosting provider has confirmed server is correctly configured to support HTTP/2 and provided evidence of accessing via HTTP/2 working 301 redirects set up for non-secure and non-www versions of website all to https://www. version Not using a CDN or proxy GSC reports home page as correctly indexed (with https://www. version canonicalised) but does still have the non-secure version of website as the referring page in the Discovery section. GSC also reports homepage as being crawled every day or so. Totally understand it can take time to update index, but we are at a complete loss to understand why GoogleBot continues to only go through HTTP/1.1 version not 2 Possibly related issue - and of course what is causing concern - is that new pages of site seem to index and perform well in SERP ... except home page. This never makes it to page 1 (other than for brand name) despite rating multiples higher in terms of content, speed etc than other pages which still get indexed in preference to home page. Any thoughts, further tests, ideas, direction or anything will be much appreciated!

                  Technical SEO | May 13, 2024, 8:42 AM | AKCAC
                  1
                • icogems

                  Unsolved how to add my known backlinks manually to moz

                  backlinks crawl

                  hello
                  i have cryptocurrency website and i found backlinks listed in my google webmasters dashboard, but those backlinks dont show in my moz dashboard even after 45 days. so my question is can i add those backlinks to moz, just to check my website real da score thanks,

                  Moz Local | May 26, 2023, 10:21 AM | icogems
                  0
                • fbcosta

                  How to index e-commerce marketplace product pages

                  technical seo ecommerce indexation crawl product page

                  Hello! We are an online marketplace that submitted our sitemap through Google Search Console 2 weeks ago. Although the sitemap has been submitted successfully, out of ~10000 links (we have ~10000 product pages), we only have 25 that have been indexed. I've attached images of the reasons given for not indexing the platform. gsc-dashboard-1 gsc-dashboard-2 How would we go about fixing this?

                  Technical SEO | Mar 4, 2024, 10:20 AM | fbcosta
                  0
                • Yenlo

                  Website can't be crawled

                  Hi there, One of our website can't be crawled. We did get the error emails from you (Moz) but we can't find the solution. Can you please help me? Thanks, Tamara

                  Product Support | Jun 12, 2024, 9:19 AM | Yenlo
                  0
                • Karen_Dauncey

                  Crawl test

                  I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?

                  Product Support | Sep 24, 2018, 10:57 PM | Karen_Dauncey
                  0
                • KBC

                  No crawl data anymore

                  Using moz quite some time, but I don't have any crawl data anymore. What happened? (www.kbc.be)
                  http://analytics.moz.com/settings/campaign/517920.1128516

                  Product Support | Jan 4, 2016, 4:29 AM | KBC
                  0

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy

                Looks like your connection to Moz was lost, please wait while we try to reconnect.