undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Digital Marketers
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Digital Marketers

      Simplify SEO tasks to save time and grow your traffic.

    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Google Search console says 'sitemap is blocked by robots?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Google Search console says 'sitemap is blocked by robots?

Technical SEO
4
10
3.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Extima-Christian
    Extima-Christian last edited by Jan 29, 2018, 7:11 PM

    Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt."

    I don't understand why my sitemap is being blocked? My robots.txt look like this:

    User-Agent: *
    Disallow:

    Sitemap: http://www.website.com/sitemap_index.xml

    It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?

    1 Reply Last reply Reply Quote 1
    • BlueprintMarketing
      BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 5:25 PM Jan 31, 2018, 5:25 PM

      Nice happy to hear that do you work with Greg Reindel? He is a good friend I looked at your IP that is why I ask?

      Tom

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing @davebuts last edited by Jan 31, 2018, 5:23 PM Jan 31, 2018, 5:23 PM

        I agree with David

        Hey is your dev Greg Reindel? If so you can call me for help PM me here for my info.

        Thomas Zickell

        1 Reply Last reply Reply Quote 0
        • Extima-Christian
          Extima-Christian last edited by Jan 31, 2018, 4:08 PM Jan 31, 2018, 4:08 PM

          Hey guys, I ended up disabling the sitemap option from YoastSEO, then installed the 'Google (XML) sitemap' plug-in. I re-submitted the sitemap to Google last night, and it came back with no issues. I'm glad to finally have this sorted out.

          Thanks for all the help!

          BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:25 PM Reply Quote 1
          • davebuts
            davebuts last edited by Jan 31, 2018, 5:28 PM Jan 31, 2018, 12:24 AM

            Hi Christian,

            The current robots.txt shouldn't be blocking those URLs.

            Did you or someone else recently change the robots.txt file? If so, give Google a few days to re-crawl your site.

            Also, can you check what happens when you do a fetch and render on one of the blocked posts in Search Console? Do you have issues there?

            Cheers,

            David

            BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:23 PM Reply Quote 1
            • BlueprintMarketing
              BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 12:02 AM Jan 31, 2018, 12:02 AM

              I think you need to make an https robots.txt file if you are running https if running https

              https://moz.com/blog/xml-sitemaps

              `User-agent: *
              Disallow: /wp-admin/
              Allow: /wp-admin/admin-ajax.php`
              
              Sitemap: https://domain.com/index-sitemap.xml
              
              (that is a https site map)
              

              can you send the sitemap URL or run it though deepcrawl

              Hope this helps?

              Did you make a new robots.txt file?

              ShhJY55.png

              1 Reply Last reply Reply Quote 0
              • Extima-Christian
                Extima-Christian @Alick300 last edited by Jan 30, 2018, 6:02 PM Jan 30, 2018, 6:02 PM

                Thanks for the response. Do you think this is a robots.txt issue? Or could this be caused by the YoastSEO plugin?

                Do you know if this plug-in works with YoastSEO together? Or will it cause issues?

                1 Reply Last reply Reply Quote 0
                • Extima-Christian
                  Extima-Christian @BlueprintMarketing last edited by Jan 30, 2018, 5:59 PM Jan 30, 2018, 5:59 PM

                  Thank you for the response.

                  I just scanned the site using 'Screaming frog'. Under Internal>Directives there were zero 'no index' links. I also check for '404 errors', server 505 errors, or anything 'blocked by robots.txt'.

                  Google search console is still showing me that there are URL's being blocked by my sitemap. (I added a screenshot of this). When I click through, it tells me that the 'post sitemap' has over +300 warnings.

                  I have just deleted the YoastSEO plugin, and I am now re-installing it. hopefully, this fixes the issue.

                  B7KE6nc ShhJY55

                  BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 12:02 AM Reply Quote 0
                  • BlueprintMarketing
                    BlueprintMarketing last edited by Jan 31, 2018, 4:19 PM Jan 30, 2018, 1:12 PM

                    No, you do not need to change or plug-in what is happening is Webmaster tools is telling you that you have no index or no follow were robots xTag somewhere on your URLs inside your sitemap.

                    Run your site through Moz, screaming frog Seo spider or deepcrawl and look for no indexed URLs.

                    webmaster tools/search console is telling you that you have no index URLs inside of your XML sitemap not that you robots.txt is blocking it. This would be set in the Yoast plugin. one way to correct it is to look for noindex URLs &  filter them inside Yoast so they are not being presented to the crawlers.

                    If you would like you can turn off the sitemap on Yoast and turn it back on if that does not work I recommend completely removing the plug-in and reinstalling it

                    • https://kb.yoast.com/kb/how-can-i-uninstall-my-plugin/
                    • https://kinsta.com/blog/uninstall-wordpress-plugin/

                    Can you send a screenshot of what you're seeing?

                    When you see it in Google Webmaster tools are you talking about the XML sitemap itself mean no indexed because all XML sitemaps are no indexed.

                    Please add this to your robots.txt

                    `User-agent:*
                    Disallow:/wp-admin/
                    Allow:/wp-admin/admin-ajax.php` Sitemap: http://www.website.com/sitemap_index.xml
                    

                    I hope this is of help,

                    Tom

                    Extima-Christian 1 Reply Last reply Jan 30, 2018, 5:59 PM Reply Quote 1
                    • Alick300
                      Alick300 last edited by Jan 30, 2018, 4:12 AM Jan 30, 2018, 4:12 AM

                      Hi,

                      Use this plugin

                      https://wordpress.org/plugins/wp-robots-txt/

                      it will remove previous robots.txt and set simple wordpress robots.txt and wait for a day

                      problem can be solved.

                      Also watch this video on the same @ https://www.youtube.com/watch?v=DZiyN07bbBM

                      Thanks

                      Extima-Christian 1 Reply Last reply Jan 30, 2018, 6:02 PM Reply Quote 0
                      • 1 / 1
                      1 out of 10
                      • First post
                        1/10
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • Digital_Reach

                        Google Search Console "Text too small to read" Errors

                        What are the guidelines / best practices for clearing these errors? Google has some pretty vague documentation on how to handle this sort of error. User behavior metrics in GA are pretty much in line with desktop usage and don't show anything concerning Any input is appreciated! Thanks m3F3uOI

                        Technical SEO | Apr 25, 2019, 1:18 PM | Digital_Reach
                        2
                      • Nomader

                        Crawl solutions for landing pages that don't contain a robots.txt file?

                        My site (www.nomader.com) is currently built on Instapage, which does not offer the ability to add a robots.txt file. I plan to migrate to a Shopify site in the coming months, but for now the Instapage site is my primary website. In the interim, would you suggest that I manually request a Google crawl through the search console tool? If so, how often? Any other suggestions for countering this Meta Noindex issue?

                        Technical SEO | May 2, 2019, 3:41 AM | Nomader
                        1
                      • SO_UK

                        301 Re-directing 'empty' domains

                        Hello, My client had purchased a few domains and 301 re-directed them, pointing to our main website. As far as I am aware the 'empty domains' are brand related but no content has ever been displayed on them, and I doubt they have much authority. The issue here is that we took a dive in ranking for our main keyword, I had a look on ahrefs and found the below: | www.empty-domain/our-keyword | 30 | 19 | 1 | fb 0
                        G+ 0
                        in 4 | REDIRECT 301 TO www.main-domain/our-keyword | 8 Feb '175 d | The ranking dip happened at the same time as the re-direct was re-discovered / re-crawled. Could the 'empty' URL in question been causing us any issues? I understand that this is terrible practice for 301 redirects, I was hoping someone in the community could shed light on any possible solution for this.

                        Technical SEO | Apr 21, 2017, 10:36 AM | SO_UK
                        0
                      • QuantumWeb62

                        Removing site subdomains from Google search

                        Hi everyone, I hope you are having a good week? My website has several subdomains that I had shut down some time back and pages on these subdomains are still appearing in the Google search result pages. I want all the URLs from these subdomains to stop appearing in the Google search result pages and I was hoping to see if anyone can help me with this. The subdomains are no longer under my control as I don't have web hosting for these sites (so these subdomain sites just show a default hosting server page). Because of this, I cannot verify these in search console and submit a url/site removal request to Google. In total, there are about 70 pages from these subdomains showing up in Google at the moment and I'm concerned in case these pages have any negative impacts on my SEO. Thanks for taking the time to read my post.

                        Technical SEO | Jul 27, 2016, 4:24 AM | QuantumWeb62
                        0
                      • Webicultors

                        2 sitemaps on my robots.txt?

                        Hi, I thought that I just could link one sitemap from my site's robots.txt but... I may be wrong. So, I need to confirm if this kind of implementation is right or wrong: robots.txt for Magento Community and Enterprise ...
                        Sitemap: http://www.mysite.es/media/sitemap/es.xml
                        Sitemap: http://www.mysite.pt/media/sitemap/pt.xml Thanks in advance,

                        Technical SEO | Nov 30, 2016, 6:17 PM | Webicultors
                        0
                      • imaginex

                        Should I block Map pages with robots.txt?

                        Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!

                        Technical SEO | Oct 29, 2014, 3:37 PM | imaginex
                        0
                      • allstatetransmission

                        Robots.txt and Multiple Sitemaps

                        Hello, I have a hopefully simple question but I wanted to ask to get a "second opinion" on what to do in this situation. I am working on a clients robots.txt and we have multiple sitemaps. Using yoast I have my sitemap_index.xml and I also have a sitemap-image.xml I do put them in google and bing by hand but wanted to have it added into the robots.txt for insurance. So my question is, when having multiple sitemaps called out on a robots.txt file does it matter if one is before the other? From my reading it looks like you can have multiple sitemaps called out, but I wasn't sure the best practice when writing it up in the file. Example: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Sitemap: http://sitename.com/sitemap_index.xml Sitemap: http://sitename.com/sitemap-image.xml Thanks a ton for the feedback, I really appreciate it! :) J

                        Technical SEO | Feb 5, 2014, 7:40 PM | allstatetransmission
                        0
                      • sparts

                        Why is my site jumping around in google search ?

                        Hi I've been trying to get my page up in google results and I was wondering why the constant fluctuation. For example, on one day the pages is nr. 26, the next day it's nr. 65 then jumps back on say 30 and then in a few more days it's going back to 50. What's the logic behind that ? Thanks Cezar

                        Technical SEO | Oct 16, 2012, 1:43 PM | sparts
                        1

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.