undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Bar
  4. How do you stop Moz crawling a page?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

How do you stop Moz crawling a page?

Moz Bar
6
6
3.6k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Seaward-Group
    Seaward-Group Subscriber last edited by Aug 23, 2013, 6:23 AM

    Hello,

    I have a contact form which generates thousands of duplicate crawl errors. I'm going to use  to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error?

    Thanks!

    1 Reply Last reply Reply Quote 0
    • signcreator
      signcreator Subscriber @AndreVanKets last edited by Nov 28, 2016, 7:23 PM Nov 28, 2016, 7:23 PM

      this is all well and good and I am able to do these, but how do I keep Moz from crawling an index.php file. our site is http://4signs.com  no index file there at all so I'm not sure why it would be crawled.

      thoughts?

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,3 years
      • Peterli
        Peterli last edited by Aug 23, 2013, 1:52 PM Aug 23, 2013, 1:52 PM

        Hi guys,

        Awesome discussion so far 🙂 Yes, Chris is correct in that using noindex as a way to block Moz is not a effective way to do it.  Since our tool is not a typical indexer (such as Google), we don't have some of the behavior of a normal spider.  Instead, Roger is very good at rooting out issues that other crawlers might not notice.   One thing Roger is also good at is obeying robots.txt.... you know him being a robot and all 😛

        You can find more information about our friend here:

        http://moz.com/learn/seo/robotstxt http://moz.com/help/pro/rogerbot-crawler

        So if you are looking to block it from looking at a page without making content changes to your code, I would definitely look into using robots.txt.  You can even use a user-agent specific directive to make sure you don't end up telling other robots/spiders to do the same thing.

        I hope that helps!  Please let us know if you have questions 🙂

        Peter
        Moz Help Team.

        1 Reply Last reply Reply Quote 0
        • Chris.Menke
          Chris.Menke last edited by Aug 23, 2013, 7:19 AM Aug 23, 2013, 7:18 AM

          On http://moz.com/help/pro/rogerbot-crawler Moz gives an answer to the question "We are still seeing duplicate content on Moz even though we have marked those pages as "noindex, follow. Any idea why?

          Moz is not a search engine index, it uses a crawler. If those pages are not blocked by the robots.txt file, then Moz will crawl them. They ignore the noindex tag because they don't index anything. Search engines will honor the noindex tag and not index a page if you specify with the robots meta tag. However, to remove pages from the crawl, disallow them in the robots.txt or metarobots.

          Their answer is not exactly clear, but according to it, no, a meta noindex  will not block rogerbot from crawling your page.

          1 Reply Last reply Reply Quote 0
          • GPainter
            GPainter last edited by Aug 23, 2013, 1:48 PM Aug 23, 2013, 6:30 AM

            Hi Gary,

            You may find the following link helpful - http://moz.com/learn/seo/robotstxt on top of this you can read how to stop the moz bot here - http://moz.com/help/pro/rogerbot-crawler

            If you have blocked bots from your page this will include the Mozbot. Hope this helps.

            1 Reply Last reply Reply Quote 2
            • AndreVanKets
              AndreVanKets last edited by Aug 23, 2013, 6:27 AM Aug 23, 2013, 6:27 AM

              Yes it will.

              signcreator 1 Reply Last reply Nov 28, 2016, 7:23 PM Reply Quote 0
              • 1 / 1
              1 out of 6
              • First post
                1/6
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • joinpakarmy234

                Is Moz more accurate than Ahrefs ?

                When I check my this website https://joinpakarmy.com.pk/  on Moz it gives me DA 3
                But when I check it out on Ahrefs, it tells me there is no authority for this site.
                So who is more accurate in the result?
                thanks

                Moz Bar | Aug 6, 2020, 12:12 PM | joinpakarmy234
                0
              • AllAboutShapewear

                Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt

                Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks

                Moz Bar | Oct 3, 2018, 1:21 PM | AllAboutShapewear
                2
              • WeAreDigital_BE

                Site crawl only shows homepage

                Hi everyone, A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage. Does somebody have an idea why the other pages of the website are not showing up? Thanks,
                Jens

                Moz Bar | Jan 2, 2018, 7:14 AM | WeAreDigital_BE
                0
              • joshuaboyes

                How can I get a list of all my sites landing pages with their page authority?

                How can I get a list of all the pages on my site with their page authority?

                Moz Bar | May 18, 2017, 12:33 PM | joshuaboyes
                0
              • rpaiva

                Moz reporting for C-Blocking

                Hey Mozers, I see Moz has a reporting tool for C-blocking and for november I had 330.  Does this mean 330 Ip addresses came from the same location in the month of november?

                Moz Bar | Jan 2, 2015, 1:44 PM | rpaiva
                1
              • Gavo

                Checking Do-follow using Moz Bar

                Does anyone know how to check if a link is a do-follow or no-follow using the Moz bar? I believe there's a function that highlights and color codes the link so as to tell? Can't seem to turn this function on in my Moz Bar settings. Am I missing something completely obvious?

                Moz Bar | May 27, 2019, 4:17 AM | Gavo
                0
              • Rebeca1

                Weird back link showed in moz crawl

                Some time ago somebody from this site: http://dianibeach.com created a weird link to our site which had on the end db. Later we have realized that the link was coming from every footer on each page. I believe that the back links from footer does not have realy value and even the more of them the less value. We have asked the guy to remove that links as I thought it might harm our site more then help. Now I I was very surprised to find this link in moz crawl error as second top page on our site in current index??? Can somebody explain how is this possible?? The most ridiculous thing is that when I click on that link it realy opens our site! How is that possible, what is it? This is the link: http://villasdiani.com/?db Thank you very much for any help with this

                Moz Bar | Oct 16, 2013, 1:36 PM | Rebeca1
                0
              • AmberHanson

                Is Moz Analytics still available to all MozCon attendees?

                I thought all Mozcon attendees were going to get access to Moz Analytics even after the convention was over. I can't seem to find were to access it from my campaigns manager dashboard and the Analytics landing page says im on queue to get access.

                Moz Bar | Jul 12, 2013, 4:37 PM | AmberHanson
                0

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.