undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Google Search console says 'sitemap is blocked by robots?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Google Search console says 'sitemap is blocked by robots?

Technical SEO
4
10
3.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Extima-Christian
    Extima-Christian last edited by Jan 29, 2018, 7:11 PM

    Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt."

    I don't understand why my sitemap is being blocked? My robots.txt look like this:

    User-Agent: *
    Disallow:

    Sitemap: http://www.website.com/sitemap_index.xml

    It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?

    1 Reply Last reply Reply Quote 1
    • BlueprintMarketing
      BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 5:25 PM Jan 31, 2018, 5:25 PM

      Nice happy to hear that do you work with Greg Reindel? He is a good friend I looked at your IP that is why I ask?

      Tom

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing @davebuts last edited by Jan 31, 2018, 5:23 PM Jan 31, 2018, 5:23 PM

        I agree with David

        Hey is your dev Greg Reindel? If so you can call me for help PM me here for my info.

        Thomas Zickell

        1 Reply Last reply Reply Quote 0
        • Extima-Christian
          Extima-Christian last edited by Jan 31, 2018, 4:08 PM Jan 31, 2018, 4:08 PM

          Hey guys, I ended up disabling the sitemap option from YoastSEO, then installed the 'Google (XML) sitemap' plug-in. I re-submitted the sitemap to Google last night, and it came back with no issues. I'm glad to finally have this sorted out.

          Thanks for all the help!

          BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:25 PM Reply Quote 1
          • davebuts
            davebuts last edited by Jan 31, 2018, 5:28 PM Jan 31, 2018, 12:24 AM

            Hi Christian,

            The current robots.txt shouldn't be blocking those URLs.

            Did you or someone else recently change the robots.txt file? If so, give Google a few days to re-crawl your site.

            Also, can you check what happens when you do a fetch and render on one of the blocked posts in Search Console? Do you have issues there?

            Cheers,

            David

            BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 5:23 PM Reply Quote 1
            • BlueprintMarketing
              BlueprintMarketing @Extima-Christian last edited by Jan 31, 2018, 12:02 AM Jan 31, 2018, 12:02 AM

              I think you need to make an https robots.txt file if you are running https if running https

              https://moz.com/blog/xml-sitemaps

              `User-agent: *
              Disallow: /wp-admin/
              Allow: /wp-admin/admin-ajax.php`
              
              Sitemap: https://domain.com/index-sitemap.xml
              
              (that is a https site map)
              

              can you send the sitemap URL or run it though deepcrawl

              Hope this helps?

              Did you make a new robots.txt file?

              ShhJY55.png

              1 Reply Last reply Reply Quote 0
              • Extima-Christian
                Extima-Christian @Alick300 last edited by Jan 30, 2018, 6:02 PM Jan 30, 2018, 6:02 PM

                Thanks for the response. Do you think this is a robots.txt issue? Or could this be caused by the YoastSEO plugin?

                Do you know if this plug-in works with YoastSEO together? Or will it cause issues?

                1 Reply Last reply Reply Quote 0
                • Extima-Christian
                  Extima-Christian @BlueprintMarketing last edited by Jan 30, 2018, 5:59 PM Jan 30, 2018, 5:59 PM

                  Thank you for the response.

                  I just scanned the site using 'Screaming frog'. Under Internal>Directives there were zero 'no index' links. I also check for '404 errors', server 505 errors, or anything 'blocked by robots.txt'.

                  Google search console is still showing me that there are URL's being blocked by my sitemap. (I added a screenshot of this). When I click through, it tells me that the 'post sitemap' has over +300 warnings.

                  I have just deleted the YoastSEO plugin, and I am now re-installing it. hopefully, this fixes the issue.

                  B7KE6nc ShhJY55

                  BlueprintMarketing 1 Reply Last reply Jan 31, 2018, 12:02 AM Reply Quote 0
                  • BlueprintMarketing
                    BlueprintMarketing last edited by Jan 31, 2018, 4:19 PM Jan 30, 2018, 1:12 PM

                    No, you do not need to change or plug-in what is happening is Webmaster tools is telling you that you have no index or no follow were robots xTag somewhere on your URLs inside your sitemap.

                    Run your site through Moz, screaming frog Seo spider or deepcrawl and look for no indexed URLs.

                    webmaster tools/search console is telling you that you have no index URLs inside of your XML sitemap not that you robots.txt is blocking it. This would be set in the Yoast plugin. one way to correct it is to look for noindex URLs &  filter them inside Yoast so they are not being presented to the crawlers.

                    If you would like you can turn off the sitemap on Yoast and turn it back on if that does not work I recommend completely removing the plug-in and reinstalling it

                    • https://kb.yoast.com/kb/how-can-i-uninstall-my-plugin/
                    • https://kinsta.com/blog/uninstall-wordpress-plugin/

                    Can you send a screenshot of what you're seeing?

                    When you see it in Google Webmaster tools are you talking about the XML sitemap itself mean no indexed because all XML sitemaps are no indexed.

                    Please add this to your robots.txt

                    `User-agent:*
                    Disallow:/wp-admin/
                    Allow:/wp-admin/admin-ajax.php` Sitemap: http://www.website.com/sitemap_index.xml
                    

                    I hope this is of help,

                    Tom

                    Extima-Christian 1 Reply Last reply Jan 30, 2018, 5:59 PM Reply Quote 1
                    • Alick300
                      Alick300 last edited by Jan 30, 2018, 4:12 AM Jan 30, 2018, 4:12 AM

                      Hi,

                      Use this plugin

                      https://wordpress.org/plugins/wp-robots-txt/

                      it will remove previous robots.txt and set simple wordpress robots.txt and wait for a day

                      problem can be solved.

                      Also watch this video on the same @ https://www.youtube.com/watch?v=DZiyN07bbBM

                      Thanks

                      Extima-Christian 1 Reply Last reply Jan 30, 2018, 6:02 PM Reply Quote 0
                      • 1 / 1
                      1 out of 10
                      • First post
                        1/10
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • tif-swedensky

                        Unsolved Google Search Console Still Reporting Errors After Fixes

                        google search console sitemaps 400 errors

                        Hello, I'm working on a website that was too bloated with content. We deleted many pages and set up redirects to newer pages. We also resolved an unreasonable amount of 400 errors on the site. I also removed several ancient sitemaps that listed content deleted years ago that Google was crawling. According to Moz and Screaming Frog, these errors have been resolved. We've submitted the fixes for validation in GSC, but the validation repeatedly fails. What could be going on here? How can we resolve these error in GSC.

                        Technical SEO | May 10, 2023, 2:40 PM | tif-swedensky
                        0
                      • Nadav_W

                        Why can't google mobile friendly test access my website?

                        getting the following error when trying to use google mobile friendly tool: "page cannot be reached. This could be because the page is unavailable or blocked by robots.txt" I don't have anything blocked by robots.txt or robots tag. i also manage to render my pages on google search console's fetch and render....so what can be the reason that the tool can't access my website? Also...the mobile usability report on the search console works but reports very little, and the google speed test also doesnt work... Any ideas to what is the reason and how to fix this? LEARN MOREDetailsUser agentGooglebot smartphone

                        Technical SEO | Feb 6, 2018, 5:08 AM | Nadav_W
                        0
                      • KatherineWatierOng

                        How do I "undo" or remove a Google Search Console change of address?

                        I have a client that set a change of address in Google Search Console where they informed Google that their preferred domain was a subdomain, and now they want Google to also consider their base domain (without the change of address). How do I get the change of address in Google search console removed?

                        Technical SEO | Dec 9, 2015, 1:04 AM | KatherineWatierOng
                        0
                      • fslocal

                        Why is Google's cache preview showing different version of webpage (i.e. not displaying content)

                        My URL is: http://www.fslocal.comRecently, we discovered Google's cached snapshots of our business listings look different from what's displayed to users. The main issue? Our content isn't displayed in cached results (although while the content isn't visible on the front-end of cached pages, the text can be found when you view the page source of that cached result).These listings are structured so everything is coded and contained within 1 page (e.g. http://www.fslocal.com/toronto/auto-vault-canada/). But even though the URL stays the same, we've created separate "pages" of content (e.g. "About," "Additional Info," "Contact," etc.) for each listing, and only 1 "page" of content will ever be displayed to the user at a time. This is controlled by JavaScript and using display:none in CSS. Why do our cached results look different? Why would our content not show up in Google's cache preview, even though the text can be found in the page source? Does it have to do with the way we're using display:none? Are there negative SEO effects with regards to how we're using it (i.e. we're employing it strictly for aesthetics, but is it possible Google thinks we're trying to hide text)? Google's Technical Guidelines recommends against using "fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash." If we were to separate those business listing "pages" into actual separate URLs (e.g. http://www.fslocal.com/toronto/auto-vault-canada/contact/ would be the "Contact" page), and employ static HTML code instead of complicated JavaScript, would that solve the problem? Any insight would be greatly appreciated.Thanks!

                        Technical SEO | Nov 4, 2013, 9:33 PM | fslocal
                        0
                      • DeptAgency

                        Is there a way for me to automatically download a website's sitemap.xml every month?

                        From now on we want to store all our sitemap.xml over the next years. Its a nice archive to have that allows us to analyse how many pages we have on our website and which ones were removed/redirected. Any suggestions? Thanks

                        Technical SEO | Aug 29, 2013, 7:44 PM | DeptAgency
                        0
                      • M_D_Golden_Peak

                        How to Remove /feed URLs from Google's Index

                        Hey everyone, I have an issue with RSS /feed URLs being indexed by Google for some of our Wordpress sites. Have a look at this Google query, and click to show omitted search results. You'll see we have 500+ /feed URLs indexed by Google, for our many category pages/etc. Here is one of the example URLs: http://www.howdesign.com/design-creativity/fonts-typography/letterforms/attachment/gilhelveticatrade/feed/. Based on this content/code of the XML page, it looks like Wordpress is generating these: <generator>http://wordpress.org/?v=3.5.2</generator> Any idea how to get them out of Google's index without 301 redirecting them? We need the Wordpress-generated RSS feeds to work for various uses. My first two thoughts are trying to work with our Development team to see if we can get a "noindex" meta robots tag on the pages, by they are dynamically-generated pages...so I'm not sure if that will be possible. Or, perhaps we can add a "feed" paramater to GWT "URL Parameters" section...but I don't want to limit Google from crawling these again...I figure I need Google to crawl them and see some code that says to get the pages out of their index...and THEN not crawl the pages anymore. I don't think the "Remove URL" feature in GWT will work, since that tool only removes URLs from the search results, not the actual Google index. FWIW, this site is using the Yoast plugin. We set every page type to "noindex" except for the homepage, Posts, Pages and Categories. We have other sites on Yoast that do not have any /feed URLs indexed by Google at all. Side note, the /robots.txt file was previously blocking crawling of the /feed URLs on this site, which is why you'll see that note in the Google SERPs when you click on the query link given in the first paragraph.

                        Technical SEO | Aug 13, 2013, 2:56 PM | M_D_Golden_Peak
                        0
                      • SharewarePros

                        Notice of DMCA removal from Google Search

                        Dear Mozer's Today I get from Google Webmaster tools a "Notice of DMCA removal" I'll paste here the note to get your opinions "Hello, Google has been notified, according to the terms of the Digital Millennium Copyright Act (DMCA), that some of your materials allegedly infringe upon the copyrights of others. The URLs of the allegedly infringing materials may be found at the end of this message. The affected URLs are listed below: http://www.freesharewaredepot.com/productpages/Ultimate_Spelling__038119.asp" So I perform these steps: 1. Remove the page from the site (now it gives 404). 2. Remove it from database (no listed on directory, sitemap.xml and RSS) 3. I fill the "Google Content Removed Notification form" detailing the removal of the page. My question is now I have to do any other task, such as fill a site reconsideration, or only I have to wait. Thank you for your help. Claudio

                        Technical SEO | Sep 3, 2012, 12:08 AM | SharewarePros
                        0
                      • ShearingsGroup

                        Should I block robots from URLs containing query strings?

                        I'm about to block off all URLs that have a query string using robots.txt. They're mostly URLs with coremetrics tags and other referrer info. I figured that search engines don't need to see these as they're always better off with the original URL. Might there be any downside to this that I need to consider? Appreciate your help / experiences on this one. Thanks Jenni

                        Technical SEO | Aug 9, 2012, 1:05 PM | ShearingsGroup
                        0

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.