undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    What is your Brand Authority?
    Moz

    What is your Brand Authority?

    Check yours now
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • SEO Q&A

      Insights & discussions from an SEO community of 500,000+.

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Google Search console says 'sitemap is blocked by robots?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Google Search console says 'sitemap is blocked by robots?

Technical SEO
4
10
3.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Extima-Christian
    Extima-Christian last edited by Jan 29, 2018, 7:11 PM

    Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt."

    I don't understand why my sitemap is being blocked? My robots.txt look like this:

    User-Agent: *
    Disallow:

    Sitemap: http://www.website.com/sitemap_index.xml

    It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?

    1 Reply Last reply Reply Quote 1
    • BlueprintMarketing
      BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 5:25 PM Jan 31, 2018, 5:25 PM

      Nice happy to hear that do you work with Greg Reindel? He is a good friend I looked at your IP that is why I ask?

      Tom

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing @davebuts last edited by Jan 31, 2018, 5:23 PM Jan 31, 2018, 5:23 PM

        I agree with David

        Hey is your dev Greg Reindel? If so you can call me for help PM me here for my info.

        Thomas Zickell

        1 Reply Last reply Reply Quote 0
        • Extima-Christian
          Extima-Christian last edited by Jan 31, 2018, 4:08 PM Jan 31, 2018, 4:08 PM

          Hey guys, I ended up disabling the sitemap option from YoastSEO, then installed the 'Google (XML) sitemap' plug-in. I re-submitted the sitemap to Google last night, and it came back with no issues. I'm glad to finally have this sorted out.

          Thanks for all the help!

          BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:25 PM Reply Quote 1
          • davebuts
            davebuts last edited by Jan 31, 2018, 5:28 PM Jan 31, 2018, 12:24 AM

            Hi Christian,

            The current robots.txt shouldn't be blocking those URLs.

            Did you or someone else recently change the robots.txt file? If so, give Google a few days to re-crawl your site.

            Also, can you check what happens when you do a fetch and render on one of the blocked posts in Search Console? Do you have issues there?

            Cheers,

            David

            BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:23 PM Reply Quote 1
            • BlueprintMarketing
              BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 12:02 AM Jan 31, 2018, 12:02 AM

              I think you need to make an https robots.txt file if you are running https if running https

              https://moz.com/blog/xml-sitemaps

              `User-agent: *
              Disallow: /wp-admin/
              Allow: /wp-admin/admin-ajax.php`
              
              Sitemap: https://domain.com/index-sitemap.xml
              
              (that is a https site map)
              

              can you send the sitemap URL or run it though deepcrawl

              Hope this helps?

              Did you make a new robots.txt file?

              ShhJY55.png

              1 Reply Last reply Reply Quote 0
              • Extima-Christian
                Extima-Christian @Alick300 last edited by Jan 30, 2018, 6:02 PM Jan 30, 2018, 6:02 PM

                Thanks for the response. Do you think this is a robots.txt issue? Or could this be caused by the YoastSEO plugin?

                Do you know if this plug-in works with YoastSEO together? Or will it cause issues?

                1 Reply Last reply Reply Quote 0
                • Extima-Christian
                  Extima-Christian @BlueprintMarketing last edited by Jan 30, 2018, 5:59 PM Jan 30, 2018, 5:59 PM

                  Thank you for the response.

                  I just scanned the site using 'Screaming frog'. Under Internal>Directives there were zero 'no index' links. I also check for '404 errors', server 505 errors, or anything 'blocked by robots.txt'.

                  Google search console is still showing me that there are URL's being blocked by my sitemap. (I added a screenshot of this). When I click through, it tells me that the 'post sitemap' has over +300 warnings.

                  I have just deleted the YoastSEO plugin, and I am now re-installing it. hopefully, this fixes the issue.

                  B7KE6nc ShhJY55

                  BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 12:02 AM Reply Quote 0
                  • BlueprintMarketing
                    BlueprintMarketing last edited by Jan 31, 2018, 4:19 PM Jan 30, 2018, 1:12 PM

                    No, you do not need to change or plug-in what is happening is Webmaster tools is telling you that you have no index or no follow were robots xTag somewhere on your URLs inside your sitemap.

                    Run your site through Moz, screaming frog Seo spider or deepcrawl and look for no indexed URLs.

                    webmaster tools/search console is telling you that you have no index URLs inside of your XML sitemap not that you robots.txt is blocking it. This would be set in the Yoast plugin. one way to correct it is to look for noindex URLs &  filter them inside Yoast so they are not being presented to the crawlers.

                    If you would like you can turn off the sitemap on Yoast and turn it back on if that does not work I recommend completely removing the plug-in and reinstalling it

                    • https://kb.yoast.com/kb/how-can-i-uninstall-my-plugin/
                    • https://kinsta.com/blog/uninstall-wordpress-plugin/

                    Can you send a screenshot of what you're seeing?

                    When you see it in Google Webmaster tools are you talking about the XML sitemap itself mean no indexed because all XML sitemaps are no indexed.

                    Please add this to your robots.txt

                    `User-agent:*
                    Disallow:/wp-admin/
                    Allow:/wp-admin/admin-ajax.php` Sitemap: http://www.website.com/sitemap_index.xml
                    

                    I hope this is of help,

                    Tom

                    Extima-Christian 1 Reply Last reply Jan 30, 2018, 5:59 PM Reply Quote 1
                    • Alick300
                      Alick300 last edited by Jan 30, 2018, 4:12 AM Jan 30, 2018, 4:12 AM

                      Hi,

                      Use this plugin

                      https://wordpress.org/plugins/wp-robots-txt/

                      it will remove previous robots.txt and set simple wordpress robots.txt and wait for a day

                      problem can be solved.

                      Also watch this video on the same @ https://www.youtube.com/watch?v=DZiyN07bbBM

                      Thanks

                      Extima-Christian 1 Reply Last reply Jan 30, 2018, 6:02 PM Reply Quote 0
                      • 1 / 1
                      1 out of 10
                      • First post
                        1/10
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • woshea

                        Google Search Console Showing 404 errors for product pages not in sitemap?

                        search console 404 error 404s

                        We have some products with url changes over the past several months. Google is showing these as having 404 errors even though they are not in sitemap (sitemap shows the correct NEW url). Is this expected? Will these errors eventually go away/stop being monitored by Google?

                        Technical SEO | Sep 29, 2023, 7:54 PM | woshea
                        0
                      • HeroDesignStudio

                        301 Redirects, Sitemaps and Indexing - How to hide redirected urls from search engines?

                        We have several pages in our site like this one, http://www.spectralink.com/solutions, which redirect to deeper page, http://www.spectralink.com/solutions/work-smarter-not-harder. Both urls are listed in the sitemap and both pages are being indexed. Should we remove those redirecting pages from the site map? Should we prevent the redirecting url from being indexed? If so, what's the best way to do that?

                        Technical SEO | Aug 11, 2016, 8:57 AM | HeroDesignStudio
                        0
                      • KatherineWatierOng

                        How do I "undo" or remove a Google Search Console change of address?

                        I have a client that set a change of address in Google Search Console where they informed Google that their preferred domain was a subdomain, and now they want Google to also consider their base domain (without the change of address). How do I get the change of address in Google search console removed?

                        Technical SEO | Dec 9, 2015, 1:04 AM | KatherineWatierOng
                        0
                      • docbeans

                        My video sitemap is not being index by Google

                        Dear friends, I have a videos portal. I created a video sitemap.xml and submit in to GWT but after 20 days it has not been indexed. I have verified in  bing webmaster as well. All videos are dynamically being fetched from server. My all static pages have been indexed but not videos. Please help me where am I doing the mistake. There are no separate pages for single videos. All the content is dynamically coming from server. Please help me. your answers will be more appreciated................. Thanks

                        Technical SEO | Aug 12, 2015, 5:26 AM | docbeans
                        0
                      • EcommerceSite

                        Do I need to block my cart page in robots.txt?

                        I have a site with woocommerce. Do I need to block the cart page?

                        Technical SEO | Mar 6, 2015, 5:36 PM | EcommerceSite
                        0
                      • Pete4

                        Why is Google Webmaster Tools showing 404 Page Not Found Errors for web pages that don't have anything to do with my site?

                        I am currently working on a small site with approx 50 web pages.  In the crawl error section in WMT Google has highlighted over 10,000 page not found errors for pages that have nothing to do with my site.  Anyone come across this before?

                        Technical SEO | Feb 10, 2015, 1:44 PM | Pete4
                        0
                      • zeepartner

                        Google indexing despite robots.txt block

                        Hi This subdomain has about 4'000 URLs indexed in Google, although it's blocked via robots.txt: https://www.google.com/search?safe=off&q=site%3Awww1.swisscom.ch&oq=site%3Awww1.swisscom.ch This has been the case for almost a year now, and it does not look like Google tends to respect the blocking in http://www1.swisscom.ch/robots.txt Any clues why this is or what I could do to resolve it? Thanks!

                        Technical SEO | May 7, 2014, 2:14 PM | zeepartner
                        0
                      • reidsteven75

                        How Does Google's "index" find the location of pages in the "page directory" to return?

                        This is my understanding of how Google's search works, and I am unsure about one thing in specific: Google continuously crawls websites and stores each page it finds (let's call it "page directory") Google's "page directory" is a cache so it isn't the "live" version of the page Google has separate storage called "the index" which contains all the keywords searched.  These keywords in "the index" point to the pages in the "page directory" that contain the same keywords. When someone searches a keyword, that keyword is accessed in the "index" and returns all relevant pages in the "page directory" These returned pages are given ranks based on the algorithm The one part I'm unsure of is how Google's "index" knows the location of relevant pages in the "page directory".  The keyword entries in the "index" point to the "page directory" somehow. I'm thinking each page has a url in the "page directory", and the entries in the "index" contain these urls.   Since Google's "page directory" is a cache, would the urls be the same as the live website (and would the keywords in the "index" point to these urls)? For example if webpage is found at wwww.website.com/page1, would the "page directory" store this page under that url in Google's cache? The reason I want to discuss this is to know the effects of changing a pages url by understanding how the search process works better.

                        Technical SEO | Jun 2, 2013, 12:00 PM | reidsteven75
                        0

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.