Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Track AI Overviews in Keyword Research
      Moz Pro

      Track AI Overviews in Keyword Research

      Try it free!
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Robots.txt blocked internal resources Wordpress

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots.txt blocked internal resources Wordpress

    Intermediate & Advanced SEO
    2
    5
    3496
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • Mat_C
      Mat_C Subscriber last edited by

      Hi all,

      We've recently migrated a Wordpress website from staging to live, but the robots.txt was deleted.  I've created the following new one:

      User-agent: *
      Allow: /
      Disallow: /wp-admin/
      Disallow: /wp-includes/
      Disallow: /wp-content/plugins/
      Disallow: /wp-content/cache/
      Disallow: /wp-content/themes/
      Allow: /wp-admin/admin-ajax.php

      However, in the site audit on SemRush,  I now get the mention that a lot of pages have issues with blocked internal resources in robots.txt file. These blocked internal resources are all cached and minified css elements: links, images and scripts.

      Does this mean that Google won't crawl some parts of these pages with blocked resources correctly and thus won't be able to follow these links and index the images? In other words, is this any cause for concern regarding SEO?

      Of course I can change the robots.txt again, but will urls like https://example.com/wp-content/cache/minify/df983.js end up in the index?

      Thanks for your thoughts!

      1 Reply Last reply Reply Quote 2
      • Mat_C
        Mat_C Subscriber @JordanLowry last edited by

        Thanks for the answer!

        Last question: is /wp-admin/admin-ajax.php an important part that has to be crawled? I found this explanation: https://wordpress.stackexchange.com/questions/190993/why-use-admin-ajax-php-and-how-does-it-work/191073#191073

        However, on this specific website there is no html at all when I check the source code, only one line with 0 on it.

        1 Reply Last reply Reply Quote 0
        • JordanLowry
          JordanLowry last edited by

          I would leave all the disallows out except for the /wp-admin/ section. For example, I'd rewrite the robots.txt file to read:

          User-agent: *
          Disallow: /wp-admin/

          Also, you kind of want Google to index your cached content. In the event your servers go down it will still be able to make your content available.

          I hope that helps. Let me know how that works out for you!

          Mat_C 1 Reply Last reply Reply Quote 1
          • Mat_C
            Mat_C Subscriber last edited by

            Thanks for the clear answer.

            I've changed the robots.txt to:

            User-agent: *
            Allow: /
            Disallow: /wp-admin/
            Disallow: /wp-includes/
            Disallow: /wp-content/themes/
            Allow: /wp-admin/admin-ajax.php

            This should avoid problems with not indexing (parts of) cached content.

            Or should I leave all the Disallows out?

            1 Reply Last reply Reply Quote 0
            • JordanLowry
              JordanLowry last edited by

              Hey there --

              Blocking resources with the robots.txt file prevents search engines from crawling content the no-index tag would be better suited for preventing content from being indexed.

              However, previous best practice would dictate blocking access to /wp-includes/ and /wp-content/ directories, etc but that's no longer necessary.

              Today, Google will fetch all your styling and JavaScript files so they can render your pages completely. Search engines now try to understand your page's layout and presentation as a key part of how they evaluate quality.

              So, yeah this might have some impact on your SEO.

              Also, if you're using a plugin to cache content you should want Google to crawl your cache content. And in my experience, Googlebot does a good job of not indexing /wp-content/ sections.

              So, for your example page, https://example.com/wp-content/cache/minify/df983.js it shouldn't end up in their index.

              Hope this helps some.

              1 Reply Last reply Reply Quote 3
              • 1 / 1
              • First post
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • HimalayanInstitute

                WordPress Sub-directory for SEO

                wordpress subdirectory domain authority

                Hi There, I'm working on a WordPress site that includes a premium content blog with approx 900 posts. As part of the project, those 900 posts and other membership functionality will be moved from the main site to another site built specifically for content/membership. Ideally, we want the existing posts to remain on the root domain to avoid a loss in link juice/domain authority. We initially began setting up a WordPress Multisite using the sub-directory option. This allows for the main site to be at www.website.com and the secondary site to be at www.website.com/secondary. Unfortunately, the themes and plugins we need for the platform do not play nicely with WordPress Multisite, so we started seeking a new solution, and, discovered that a second instance of WordPress can be installed in a subdirectory on the server. This would give us the same subdirectory structure while bypassing WordPress Multisite and instead, having two separate single-site installs. Do you foresee any issues with this WordPress subdirectory install? Does Google care/know these are two separate WordPress installs and do we risk losing any link juice/domain authority?

                Intermediate & Advanced SEO | | HimalayanInstitute
                0
              • jamiegriz

                SEO Best Practices regarding Robots.txt disallow

                I cannot find hard and fast direction about the following issue: It looks like the Robots.txt file on my server has been set up to disallow "account" and "search" pages within my site, so I am receiving warnings from the Google Search console that URLs are being blocked by Robots.txt. (Disallow: /Account/ and Disallow: /?search=). Do you recommend unblocking these URLs? I'm getting a warning that over 18,000 Urls are blocked by robots.txt. ("Sitemap contains urls which are blocked by robots.txt"). Seems that I wouldn't want that many urls blocked. ? Thank you!!

                Intermediate & Advanced SEO | | jamiegriz
                0
              • seoanalytics

                Breadcrumbs and internal links

                Hello, I use to move up my site structure with links in content. I have now installed breadcrumbs, is it is useful to still keep the links in content or isn't there a need to duplicate those links ? and are the breadcrumbs links enough. Thank you,

                Intermediate & Advanced SEO | | seoanalytics
                1
              • ThomasHarvey

                Large robots.txt file

                We're looking at potentially creating a robots.txt with 1450 lines in it. This will remove 100k+ pages from the crawl that are all old pages (I know, the ideal would be to delete/noindex but not viable unfortunately) Now the issue i'm thinking is that a large robots.txt will either stop the robots.txt from being followed or will slow our crawl rate down. Does anybody have any experience with a robots.txt of that size?

                Intermediate & Advanced SEO | | ThomasHarvey
                0
              • ThomasHarvey

                What do you add to your robots.txt on your ecommerce sites?

                We're looking at expanding our robots.txt, we currently don't have the ability to noindex/nofollow. We're thinking about adding the following: Checkout Basket Then possibly: Price Theme Sortby other misc filters. What do you include?

                Intermediate & Advanced SEO | | ThomasHarvey
                0
              • jeremycabral

                Wordpress Comments Pagination

                Hi Mozzers What is your view on the following. Should you Paginate comments to increase page speed? If yes, at what # of comments would you begin pagination? (with the objective being decreasing page load times) Apply rel="canonical" back to the main article URL? eg: url/comment-page-1 => url noindex the comment pages? create a "View all" comments page? Thanks in advance for your help! 🙂
                J

                Intermediate & Advanced SEO | | jeremycabral
                0
              • Milian

                Robots.txt, does it need preceding directory structure?

                Do you need the entire preceding path in robots.txt for it to match? e.g: I know if i add Disallow: /fish to robots.txt it will block /fish
                /fish.html
                /fish/salmon.html
                /fishheads
                /fishheads/yummy.html
                /fish.php?id=anything But would it block?: en/fish
                en/fish.html
                en/fish/salmon.html
                en/fishheads
                en/fishheads/yummy.html
                **en/fish.php?id=anything (taken from Robots.txt Specifications)** I'm hoping it actually wont match, that way writing this particular robots.txt will be much easier! As basically I'm wanting to block many URL that have BTS- in such as: http://www.example.com/BTS-something
                http://www.example.com/BTS-somethingelse
                http://www.example.com/BTS-thingybob But have other pages that I do not want blocked, in subfolders that also have BTS- in, such as: http://www.example.com/somesubfolder/BTS-thingy
                http://www.example.com/anothersubfolder/BTS-otherthingy Thanks for listening

                Intermediate & Advanced SEO | | Milian
                0
              • Tone_Agency

                Wordpress blog in a subdirectory not being indexed by Google

                HI MozzersIn my websites sitemap.xml, pages are listed, such as /blog/ and /blog/textile-fact-or-fiction-egyptian-cotton-explained/These pages are visible when you visit them in a browser and when you use the Google Webmaster tool - Fetch as Google to view them (see attachment), however they aren't being indexed in Google, not even the root directory for the blog (/blog/) is being indexed, and when we query:site: www.hilden.co.uk/blog/ It returns 0 results in Google.Also note that:The Wordpress installation is located at /blog/ which is a subdirectory of the main root directory which is managed by Magento. I'm wondering if this causing the problem.Any help on this would be greatly appreciated!AnthonyToTOHuj.png?1

                Intermediate & Advanced SEO | | Tone_Agency
                0

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.