undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Disallow: /jobs/? is this stopping the SERPs from indexing job posts

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Disallow: /jobs/? is this stopping the SERPs from indexing job posts

Intermediate & Advanced SEO
4
9
2.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • JamesHancocks1
    JamesHancocks1 last edited by Sep 3, 2018, 7:10 AM

    Hi,
    I was wondering what this would be used for as it's in the Robots.exe of a recruitment agency website that posts jobs. Should it be removed?

    Disallow: /jobs/?
    Disallow: /jobs/page/*/

    Thanks in advance.
    James

    1 Reply Last reply Reply Quote 0
    • Keszi
      Keszi @JamesHancocks1 last edited by Oct 9, 2018, 5:56 AM Sep 14, 2018, 4:57 AM

      Hi James,

      So far as I can see you have the following architecture:

      • job posting: https://www.pkeducation.co.uk/job/post-name/
      • jobs listing page: https://www.pkeducation.co.uk/jobs/

      Since from the robots.txt the listing page pagination is blocked, the crawler can access only the first 15 job postings are available to crawl via a normal crawl.

      I would say, you should remove the blocking from the robots.txt and focus on implementing a correct pagination. *which method you choose is your decision, but allow the crawler to access all of your job posts. Check https://yoast.com/pagination-seo-best-practices/

      Another thing I would change is to make the job post title an anchor text for the job posting. (every single job is linked with "Find out more").

      Also if possible, create a separate sitemap.xml for your job posts and submit it in Search Console, this way you can keep track of any anomaly with indexation.

      Last, and not least, focus on the quality of your content (just as Matt proposed in the first answer).

      Good luck!

      1 Reply Last reply Reply Quote 1
      • JamesHancocks1
        JamesHancocks1 @Keszi last edited by Sep 13, 2018, 5:34 AM Sep 13, 2018, 5:34 AM

        Hi Istvan,

        Sorry I've been away for a while. Thanks for all of your advice guys.

        Here is the url if that helps?

        https://www.pkeducation.co.uk/jobs/

        Cheers,

        James

        Keszi 1 Reply Last reply Sep 14, 2018, 4:57 AM Reply Quote 0
        • Keszi
          Keszi @Dezzign last edited by Sep 6, 2018, 8:15 AM Sep 6, 2018, 8:15 AM

          The idea is (which we both highlighted), that blocking your listing page from robots.txt is wrong, for pagination you have several methods to deal with (how you deal with it, it really depends on the technical possibilities that you have on the project).

          Regarding James' original question, my feeling is, that he is somehow blocking their posting pages. Cutting the access to these pages makes it really hard for Google, or any other search engine to index it. But without a URL in front of us, we cannot really answer his question, we can only create theories that he can test 🙂

          JamesHancocks1 1 Reply Last reply Sep 13, 2018, 5:34 AM Reply Quote 1
          • Dezzign
            Dezzign @Keszi last edited by Sep 6, 2018, 7:24 AM Sep 6, 2018, 7:22 AM

            Ah yes when it's pointed out like that, it's a conflicting signal isn't It. Makes sense in theory, but if you're setting it to noindex and then passing that on via a canonical it's probably not the best is it.

            They're was link out in that thread to a discussion of people who still do that with success, but after reading that I would just use noindex only as you said. (Still prefer the no index on the robots block though)

            Keszi 1 Reply Last reply Sep 6, 2018, 8:15 AM Reply Quote 1
            • Keszi
              Keszi @Dezzign last edited by Sep 5, 2018, 10:11 AM Sep 5, 2018, 10:09 AM

              Sorry Richard, but using noindex with canonical link is not quite a good practice.

              It's an old entry, but still true: https://www.seroundtable.com/noindex-canonical-google-18274.html

              Dezzign 1 Reply Last reply Sep 6, 2018, 7:22 AM Reply Quote 1
              • Dezzign
                Dezzign last edited by Sep 5, 2018, 7:58 AM Sep 5, 2018, 7:58 AM

                I don't think it should be blocked by robots.txt at all. It's stopping Google from crawling the site fully. And they may even treat it negatively as they've been really clamping down on blocking folders with robots.txt lately. I've seen sites with warning in search console for: Disallow: /wp-admin

                You may want to consider just using a noindex tag on those pages instead. And then also use a canonical tag that points back to the main job category page. That way Google can crawl the pages and perhaps pass all the juice back to the main job category page via the canonical. Then just make sure those junk job pages aren't in the sitemap either.

                Keszi 1 Reply Last reply Sep 5, 2018, 10:09 AM Reply Quote 1
                • Keszi
                  Keszi last edited by Sep 4, 2018, 10:12 AM Sep 4, 2018, 10:12 AM

                  Hi James,

                  Regarding the robots.txt syntax:

                  Disallow: /jobs/? which basically blocks every single URL that contains /jobs/**? **

                  For example: domain.com**/jobs/?**sort-by=... will be blocked

                  If you want to disallow query parameters from URL, the correct implementation would be Disallow: /jobs/*? or even specify which query parameter you want to block. For example Disallow: /jobs/*?page=

                  My question to you, if these jobs are linked from any other page and/or sitemap? Or only from the listing page, which has it's pagination, sorting, etc. is blocked by robots.txt? If they are not linked, it could be a simple case of orphan pages, where basically the crawler cannot access the job posting pages, because there is no actual link to it. I know it is an old rule, but it is still true: Crawl > Index > Rank.

                  BTW. I don't know why you would block your pagination. There are other optimal implementations.

                  And there is always the scenario, that was already described by Matt. But I believe in that case you would have at least some of the pages indexed even if they are not going to get ranked well.

                  Also, make sure other technical implementations are not stopping your job posting pages from being indexed.

                  1 Reply Last reply Reply Quote 1
                  • MattJanaway
                    MattJanaway last edited by Sep 3, 2018, 5:57 PM Sep 3, 2018, 5:57 PM

                    I'd guess that the jobs get pulled from a job board. If this is the case, then the content ( job description, title etc.) will just be a duplication of the content that can be found in many other locations. If a plugin is used, they sometimes automatically add a disallow into the robots.txt file as to not hurt the parent version of the job page by creating thousands of duplicate content issues.

                    I'd recommend creating some really high-quality hub pages based on job type, or location and pulling the relevant jobs into that page, instead of trying to index and rank the actual job pages.

                    1 Reply Last reply Reply Quote 2
                    • 1 / 1
                    1 out of 9
                    • First post
                      1/9
                      Last post

                    Got a burning SEO question?

                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                    Start my free trial


                    Browse Questions

                    Explore more categories

                    • Moz Tools

                      Chat with the community about the Moz tools.

                    • SEO Tactics

                      Discuss the SEO process with fellow marketers

                    • Community

                      Discuss industry events, jobs, and news!

                    • Digital Marketing

                      Chat about tactics outside of SEO

                    • Research & Trends

                      Dive into research and trends in the search industry.

                    • Support

                      Connect on product support and feature requests.

                    • See all categories

                    Related Questions

                    • neverenoughmusic.com

                      My last site crawl shows over 700 404 errors all with void(0 added to the ends of my posts/pages.

                      Hello, My last site crawl shows over 700 404 errors all with void(0 added to the ends of my posts/pages.  I have contacted my theme company but not sure what could have done this.  Any ideas? The original posts/pages are still correct and working it just looks like it did duplicates and added void(0 to the end of each post/page. Questions: There is no way to undo this correct? Do I have to do a redirect on each of these? Will this hurt my rankings and domain authority? Any suggestions would be appreciated. Thanks, Wade

                      Intermediate & Advanced SEO | Apr 2, 2020, 10:19 PM | neverenoughmusic.com
                      0
                    • kbates

                      Trying to get Google to stop indexing an old site!

                      Howdy, I have a small dilemma. We built a new site for a client, but the old site is still ranking/indexed and we can't seem to get rid of it. We setup a 301 from the old site to the new one, as we have done many times before, but even though the old site is no longer live and the hosting package has been cancelled, the old site is still indexed. (The new site is at a completely different host.) We never had access to the old site, so we weren't able to request URL removal through GSC. Any guidance on how to get rid of the old site would be very appreciated. BTW, it's been about 60 days since we took these steps. Thanks, Kirk

                      Intermediate & Advanced SEO | Jul 29, 2019, 12:34 PM | kbates
                      0
                    • RobAus

                      Question about Indexing of /?limit=all

                      Hi, i've got your SEO Suite Ultimate installed on my site (www.customlogocases.com). I've got a relatively new magento site (around 1 year). We have recently been doing some pr/seo for the category pages, for example /custom-ipad-cases/ But when I search on google, it seems that google has indexed the /custom-ipad-cases/?limit=all This /?limit=all page is one without any links, and only has a PA of 1. Whereas the standard /custom-ipad-cases/ without the /? query has a much higher pa of 20, and a couple of links pointing towards it. So therefore I would want this particular page to be the one that google indexes. And along the same logic, this page really should be able to achieve higher rankings than the /?limit=all page. Is my thinking here correct? Should I disallow all the /? now, even though these are the ones that are indexed, and the others currently are not. I'd be happy to take the hit while it figures it out, because the higher PA pages are what I ultimately am getting links to... Thoughts?

                      Intermediate & Advanced SEO | Nov 4, 2015, 10:43 PM | RobAus
                      0
                    • jmorehouse

                      Should I disallow all URL query strings/parameters in Robots.txt?

                      Webmaster Tools correctly identifies the query strings/parameters used in my URLs, but still reports duplicate title tags and meta descriptions for the original URL and the versions with parameters. For example, Webmaster Tools would report duplicates for the following URLs, despite it correctly identifying the "cat_id" and "kw" parameters: /Mulligan-Practitioner-CD-ROM
                      /Mulligan-Practitioner-CD-ROM?cat_id=87
                      /Mulligan-Practitioner-CD-ROM?kw=CROM Additionally, theses pages have self-referential canonical tags, so I would think I'd be covered, but I recently read that another Mozzer saw a great improvement after disallowing all query/parameter URLs, despite Webmaster Tools not reporting any errors. As I see it, I have two options: Manually tell Google that these parameters have no effect on page content via the URL Parameters section in Webmaster Tools (in case Google is unable to automatically detect this, and I am being penalized as a result). Add "Disallow: *?" to hide all query/parameter URLs from Google. My concern here is that most backlinks include the parameters, and in some cases these parameter URLs outrank the original. Any thoughts?

                      Intermediate & Advanced SEO | Jul 7, 2015, 9:05 AM | jmorehouse
                      0
                    • romanbond

                      Redirecting index.html to the root

                      Hi, I was wondering if there is a safe way to consolidate link juice on a single version of a home page. I find incoming links to my site that link to both mysite.com/ and mysite.com/index.html. I've decided to go with mysite.com/ as my main and only URL for the site and now I'd like to transfer all link juice from mysite.com/index.html to mysite.com/
                      When i tried 301 redirect from index.html to the root it created an indefinite loop, of course. I know I can use a RewriteRule.., but will it transfer the juice?? Please help!

                      Intermediate & Advanced SEO | Sep 27, 2021, 9:27 PM | romanbond
                      7
                    • Leverage_Marketing

                      Reducing Booking Engine Indexation

                      Hi Mozzers, I am working on a site with a very useful room booking engine.  Helpful as it may be, all the variations (2 bedrooms, 3 bedrooms, room with a view, etc, etc,) are indexed by Google.  Section 13 on Search Pagination in Dr. Pete's great post on Panda http://www.seomoz.org/blog/duplicate-content-in-a-post-panda-world speaks to our issue, but I was wondering since 2 (!) years have gone by, if there are any additional solutions y'all might recommend.  We want to cut down on the duplicate titles and content and get the useful but not useful for SERPs online booking pages out of the index.  Any thoughts? Thanks for your help.

                      Intermediate & Advanced SEO | Apr 15, 2013, 4:10 PM | Leverage_Marketing
                      0
                    • ScottBaxterWW

                      De-indexed Link Directory

                      Howdy Guys, I'm currently working through our 4th reconsideration request and just have a couple of questions. Using Link Detox (www.linkresearchtools.com) new tool they have flagged up a 64 links that are Toxic and should be removed. After analysing them further alot / most of them are link directories that have now been de-indexed by Google. Do you think we should still ask for them to be removed or is this a pointless exercise as the links has already been removed because its been de-indexed. Would like your views on this guys.

                      Intermediate & Advanced SEO | Sep 1, 2012, 9:32 AM | ScottBaxterWW
                      0
                    • SEODinosaur

                      How to clean up a SERP?

                      I have a new customer and he wants me to clear up the SERP for his branded keyword, the SERP currently has his site and two other sites related to him under his result... Under that is bad reviews and old reports. My client does own the top spot (#1) for his branded name. My client has a: linkedin facebook twitter myspace I was thinking to push all these to the first page, this will clear up some of those bad reviews. What are your thoughts? Have any of you ever had this type of case? I need to get 6 different sites to all rank for the same exact key term, however I have the top spot to link from...

                      Intermediate & Advanced SEO | Mar 25, 2012, 8:13 PM | SEODinosaur
                      0

                    Get started with Moz Pro!

                    Unlock the power of advanced SEO tools and data-driven insights.

                    Start my free trial
                    Products
                    • Moz Pro
                    • Moz Local
                    • Moz API
                    • Moz Data
                    • STAT
                    • Product Updates
                    Moz Solutions
                    • SMB Solutions
                    • Agency Solutions
                    • Enterprise Solutions
                    Free SEO Tools
                    • Domain Authority Checker
                    • Link Explorer
                    • Keyword Explorer
                    • Competitive Research
                    • Brand Authority Checker
                    • Local Citation Checker
                    • MozBar Extension
                    • MozCast
                    Resources
                    • Blog
                    • SEO Learning Center
                    • Help Hub
                    • Beginner's Guide to SEO
                    • How-to Guides
                    • Moz Academy
                    • API Docs
                    About Moz
                    • About
                    • Team
                    • Careers
                    • Contact
                    Why Moz
                    • Case Studies
                    • Testimonials
                    Get Involved
                    • Become an Affiliate
                    • MozCon
                    • Webinars
                    • Practical Marketer Series
                    • MozPod
                    Connect with us

                    Contact the Help team

                    Join our newsletter
                    Moz logo
                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                    • Accessibility
                    • Terms of Use
                    • Privacy

                    Looks like your connection to Moz was lost, please wait while we try to reconnect.