undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Google Search console says 'sitemap is blocked by robots?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Google Search console says 'sitemap is blocked by robots?

Technical SEO
4
10
3.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Extima-Christian
    Extima-Christian last edited by Jan 29, 2018, 7:11 PM

    Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt."

    I don't understand why my sitemap is being blocked? My robots.txt look like this:

    User-Agent: *
    Disallow:

    Sitemap: http://www.website.com/sitemap_index.xml

    It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?

    1 Reply Last reply Reply Quote 1
    • BlueprintMarketing
      BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 5:25 PM Jan 31, 2018, 5:25 PM

      Nice happy to hear that do you work with Greg Reindel? He is a good friend I looked at your IP that is why I ask?

      Tom

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing @davebuts last edited by Jan 31, 2018, 5:23 PM Jan 31, 2018, 5:23 PM

        I agree with David

        Hey is your dev Greg Reindel? If so you can call me for help PM me here for my info.

        Thomas Zickell

        1 Reply Last reply Reply Quote 0
        • Extima-Christian
          Extima-Christian last edited by Jan 31, 2018, 4:08 PM Jan 31, 2018, 4:08 PM

          Hey guys, I ended up disabling the sitemap option from YoastSEO, then installed the 'Google (XML) sitemap' plug-in. I re-submitted the sitemap to Google last night, and it came back with no issues. I'm glad to finally have this sorted out.

          Thanks for all the help!

          BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:25 PM Reply Quote 1
          • davebuts
            davebuts last edited by Jan 31, 2018, 5:28 PM Jan 31, 2018, 12:24 AM

            Hi Christian,

            The current robots.txt shouldn't be blocking those URLs.

            Did you or someone else recently change the robots.txt file? If so, give Google a few days to re-crawl your site.

            Also, can you check what happens when you do a fetch and render on one of the blocked posts in Search Console? Do you have issues there?

            Cheers,

            David

            BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:23 PM Reply Quote 1
            • BlueprintMarketing
              BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 12:02 AM Jan 31, 2018, 12:02 AM

              I think you need to make an https robots.txt file if you are running https if running https

              https://moz.com/blog/xml-sitemaps

              `User-agent: *
              Disallow: /wp-admin/
              Allow: /wp-admin/admin-ajax.php`
              
              Sitemap: https://domain.com/index-sitemap.xml
              
              (that is a https site map)
              

              can you send the sitemap URL or run it though deepcrawl

              Hope this helps?

              Did you make a new robots.txt file?

              ShhJY55.png

              1 Reply Last reply Reply Quote 0
              • Extima-Christian
                Extima-Christian @Alick300 last edited by Jan 30, 2018, 6:02 PM Jan 30, 2018, 6:02 PM

                Thanks for the response. Do you think this is a robots.txt issue? Or could this be caused by the YoastSEO plugin?

                Do you know if this plug-in works with YoastSEO together? Or will it cause issues?

                1 Reply Last reply Reply Quote 0
                • Extima-Christian
                  Extima-Christian @BlueprintMarketing last edited by Jan 30, 2018, 5:59 PM Jan 30, 2018, 5:59 PM

                  Thank you for the response.

                  I just scanned the site using 'Screaming frog'. Under Internal>Directives there were zero 'no index' links. I also check for '404 errors', server 505 errors, or anything 'blocked by robots.txt'.

                  Google search console is still showing me that there are URL's being blocked by my sitemap. (I added a screenshot of this). When I click through, it tells me that the 'post sitemap' has over +300 warnings.

                  I have just deleted the YoastSEO plugin, and I am now re-installing it. hopefully, this fixes the issue.

                  B7KE6nc ShhJY55

                  BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 12:02 AM Reply Quote 0
                  • BlueprintMarketing
                    BlueprintMarketing last edited by Jan 31, 2018, 4:19 PM Jan 30, 2018, 1:12 PM

                    No, you do not need to change or plug-in what is happening is Webmaster tools is telling you that you have no index or no follow were robots xTag somewhere on your URLs inside your sitemap.

                    Run your site through Moz, screaming frog Seo spider or deepcrawl and look for no indexed URLs.

                    webmaster tools/search console is telling you that you have no index URLs inside of your XML sitemap not that you robots.txt is blocking it. This would be set in the Yoast plugin. one way to correct it is to look for noindex URLs &  filter them inside Yoast so they are not being presented to the crawlers.

                    If you would like you can turn off the sitemap on Yoast and turn it back on if that does not work I recommend completely removing the plug-in and reinstalling it

                    • https://kb.yoast.com/kb/how-can-i-uninstall-my-plugin/
                    • https://kinsta.com/blog/uninstall-wordpress-plugin/

                    Can you send a screenshot of what you're seeing?

                    When you see it in Google Webmaster tools are you talking about the XML sitemap itself mean no indexed because all XML sitemaps are no indexed.

                    Please add this to your robots.txt

                    `User-agent:*
                    Disallow:/wp-admin/
                    Allow:/wp-admin/admin-ajax.php` Sitemap: http://www.website.com/sitemap_index.xml
                    

                    I hope this is of help,

                    Tom

                    Extima-Christian 1 Reply Last reply Jan 30, 2018, 5:59 PM Reply Quote 1
                    • Alick300
                      Alick300 last edited by Jan 30, 2018, 4:12 AM Jan 30, 2018, 4:12 AM

                      Hi,

                      Use this plugin

                      https://wordpress.org/plugins/wp-robots-txt/

                      it will remove previous robots.txt and set simple wordpress robots.txt and wait for a day

                      problem can be solved.

                      Also watch this video on the same @ https://www.youtube.com/watch?v=DZiyN07bbBM

                      Thanks

                      Extima-Christian 1 Reply Last reply Jan 30, 2018, 6:02 PM Reply Quote 0
                      • 1 / 1
                      1 out of 10
                      • First post
                        1/10
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • tif-swedensky

                        Unsolved Google Search Console Still Reporting Errors After Fixes

                        google search console sitemaps 400 errors

                        Hello, I'm working on a website that was too bloated with content. We deleted many pages and set up redirects to newer pages. We also resolved an unreasonable amount of 400 errors on the site. I also removed several ancient sitemaps that listed content deleted years ago that Google was crawling. According to Moz and Screaming Frog, these errors have been resolved. We've submitted the fixes for validation in GSC, but the validation repeatedly fails. What could be going on here? How can we resolve these error in GSC.

                        Technical SEO | May 10, 2023, 2:40 PM | tif-swedensky
                        0
                      • JoelssonMedia

                        Page disappears from Google search results

                        Hi, I recently encountered a very strange problem.
                        One of the pages I published in my website ranked very well for a couple of days on top 5, then after a couple of days, the page completely vanished, no matter how direct I search for it, does not appear on the results, I check GSC, everything seems to be normal, but when checking Google analytics, I find it strange that there is no data on the page since it disappeared and it also does not show up on the 'active pages' section no matter how many different computers i keep it open. I have checked to page 9, and used a couple of keyword tools and it appears nowhere! It didn't have any back links, but it was unique and high quality. I have checked on the page does still exist and it is still readable. Has this ´happened to anyone before? Any thoughts would be gratefully received.

                        Technical SEO | Jun 13, 2021, 7:57 PM | JoelssonMedia
                        0
                      • DaleZon

                        Errors In Search Console

                        Hi All, I am hoping someone might be able to help with this. Last week one of my sites dropped from mid first day to bottom of page 1. We had not been link building as such and it only seems to of affected a single search term and the ranking page (which happens to be the home page). When I was going through everything I went to search console and in crawl errors there are 2 errors that showed up as detected 3 days before the drop. These are: wp-admin/admin-ajax.php showing as response code 400 and also xmlrpc.php showing as response code 405 robots.txt is as follows: user-agent: * disallow: /wp-admin/ allow: /wp-admin/admin-ajax.php Any help with what is wrong here and how to fix it would be greatly appreciated. Many Thanks

                        Technical SEO | Jul 9, 2018, 1:28 PM | DaleZon
                        0
                      • EcommerceSite

                        Do I need to block my cart page in robots.txt?

                        I have a site with woocommerce. Do I need to block the cart page?

                        Technical SEO | Mar 6, 2015, 5:36 PM | EcommerceSite
                        0
                      • imaginex

                        Should I block Map pages with robots.txt?

                        Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!

                        Technical SEO | Oct 29, 2014, 3:37 PM | imaginex
                        0
                      • KCBackofen

                        Will an XML sitemap override a robots.txt

                        I have a client that has a robots.txt file that is blocking an entire subdomain, entirely by accident. Their original solution, not realizing the robots.txt error, was to submit an xml sitemap to get their pages indexed. I did not think this tactic would work, as the robots.txt would take precedent over the xmls sitemap. But it worked... I have no explanation as to how or why. Does anyone have an answer to this? or any experience with a website that has had a clear Disallow: / for months , that somehow has pages in the index?

                        Technical SEO | Apr 10, 2013, 8:50 PM | KCBackofen
                        0
                      • BenRWoodard

                        Javascript to manipulate Google's bounce rate and time on site?

                        I was referred to this "awesome" solution to high bounce rates. It is suppose to "fix" bounce rates and lower them through this simple script.  When the bounce rate goes way down then rankings dramatically increase (interesting study but not my question). I don't know javascript but simply adding a script to the footer and watch everything fall into place seems a bit iffy to me. Can someone with experience in JS help me by explaining what this script does? I think it manipulates the reporting it does to GA but I'm not sure.  It was supposed to be placed in the footer of the page and then sit back and watch the dollars fly in. 🙂

                        Technical SEO | Jun 27, 2012, 4:55 PM | BenRWoodard
                        1
                      • A_Q

                        Blank pages in Google's webcache

                        Hello all, Is anybody experiencing blanck page's in Google's 'Cached' view? I'm seeing just the page background and none of the content for a couple of my pages but when I click 'View Text Only' all of teh content is there. Strange! I'd love to hear if anyone else is experiencing the same. Perhaps this is something to do with the roll out of Google's updates last week?! Thanks,
                        Elias

                        Technical SEO | Mar 5, 2012, 10:33 AM | A_Q
                        0

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.