undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Disallow: /jobs/? is this stopping the SERPs from indexing job posts

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Disallow: /jobs/? is this stopping the SERPs from indexing job posts

Intermediate & Advanced SEO
4
9
2.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • JamesHancocks1
    JamesHancocks1 last edited by Sep 3, 2018, 7:10 AM

    Hi,
    I was wondering what this would be used for as it's in the Robots.exe of a recruitment agency website that posts jobs. Should it be removed?

    Disallow: /jobs/?
    Disallow: /jobs/page/*/

    Thanks in advance.
    James

    1 Reply Last reply Reply Quote 0
    • Keszi
      Keszi @JamesHancocks1 last edited by Oct 9, 2018, 5:56 AM Sep 14, 2018, 4:57 AM

      Hi James,

      So far as I can see you have the following architecture:

      • job posting: https://www.pkeducation.co.uk/job/post-name/
      • jobs listing page: https://www.pkeducation.co.uk/jobs/

      Since from the robots.txt the listing page pagination is blocked, the crawler can access only the first 15 job postings are available to crawl via a normal crawl.

      I would say, you should remove the blocking from the robots.txt and focus on implementing a correct pagination. *which method you choose is your decision, but allow the crawler to access all of your job posts. Check https://yoast.com/pagination-seo-best-practices/

      Another thing I would change is to make the job post title an anchor text for the job posting. (every single job is linked with "Find out more").

      Also if possible, create a separate sitemap.xml for your job posts and submit it in Search Console, this way you can keep track of any anomaly with indexation.

      Last, and not least, focus on the quality of your content (just as Matt proposed in the first answer).

      Good luck!

      1 Reply Last reply Reply Quote 1
      • JamesHancocks1
        JamesHancocks1 @Keszi last edited by Sep 13, 2018, 5:34 AM Sep 13, 2018, 5:34 AM

        Hi Istvan,

        Sorry I've been away for a while. Thanks for all of your advice guys.

        Here is the url if that helps?

        https://www.pkeducation.co.uk/jobs/

        Cheers,

        James

        Keszi 1 Reply Last reply Sep 14, 2018, 4:57 AM Reply Quote 0
        • Keszi
          Keszi @Dezzign last edited by Sep 6, 2018, 8:15 AM Sep 6, 2018, 8:15 AM

          The idea is (which we both highlighted), that blocking your listing page from robots.txt is wrong, for pagination you have several methods to deal with (how you deal with it, it really depends on the technical possibilities that you have on the project).

          Regarding James' original question, my feeling is, that he is somehow blocking their posting pages. Cutting the access to these pages makes it really hard for Google, or any other search engine to index it. But without a URL in front of us, we cannot really answer his question, we can only create theories that he can test 🙂

          JamesHancocks1 1 Reply Last reply Sep 13, 2018, 5:34 AM Reply Quote 1
          • Dezzign
            Dezzign @Keszi last edited by Sep 6, 2018, 7:24 AM Sep 6, 2018, 7:22 AM

            Ah yes when it's pointed out like that, it's a conflicting signal isn't It. Makes sense in theory, but if you're setting it to noindex and then passing that on via a canonical it's probably not the best is it.

            They're was link out in that thread to a discussion of people who still do that with success, but after reading that I would just use noindex only as you said. (Still prefer the no index on the robots block though)

            Keszi 1 Reply Last reply Sep 6, 2018, 8:15 AM Reply Quote 1
            • Keszi
              Keszi @Dezzign last edited by Sep 5, 2018, 10:11 AM Sep 5, 2018, 10:09 AM

              Sorry Richard, but using noindex with canonical link is not quite a good practice.

              It's an old entry, but still true: https://www.seroundtable.com/noindex-canonical-google-18274.html

              Dezzign 1 Reply Last reply Sep 6, 2018, 7:22 AM Reply Quote 1
              • Dezzign
                Dezzign last edited by Sep 5, 2018, 7:58 AM Sep 5, 2018, 7:58 AM

                I don't think it should be blocked by robots.txt at all. It's stopping Google from crawling the site fully. And they may even treat it negatively as they've been really clamping down on blocking folders with robots.txt lately. I've seen sites with warning in search console for: Disallow: /wp-admin

                You may want to consider just using a noindex tag on those pages instead. And then also use a canonical tag that points back to the main job category page. That way Google can crawl the pages and perhaps pass all the juice back to the main job category page via the canonical. Then just make sure those junk job pages aren't in the sitemap either.

                Keszi 1 Reply Last reply Sep 5, 2018, 10:09 AM Reply Quote 1
                • Keszi
                  Keszi last edited by Sep 4, 2018, 10:12 AM Sep 4, 2018, 10:12 AM

                  Hi James,

                  Regarding the robots.txt syntax:

                  Disallow: /jobs/? which basically blocks every single URL that contains /jobs/**? **

                  For example: domain.com**/jobs/?**sort-by=... will be blocked

                  If you want to disallow query parameters from URL, the correct implementation would be Disallow: /jobs/*? or even specify which query parameter you want to block. For example Disallow: /jobs/*?page=

                  My question to you, if these jobs are linked from any other page and/or sitemap? Or only from the listing page, which has it's pagination, sorting, etc. is blocked by robots.txt? If they are not linked, it could be a simple case of orphan pages, where basically the crawler cannot access the job posting pages, because there is no actual link to it. I know it is an old rule, but it is still true: Crawl > Index > Rank.

                  BTW. I don't know why you would block your pagination. There are other optimal implementations.

                  And there is always the scenario, that was already described by Matt. But I believe in that case you would have at least some of the pages indexed even if they are not going to get ranked well.

                  Also, make sure other technical implementations are not stopping your job posting pages from being indexed.

                  1 Reply Last reply Reply Quote 1
                  • MattJanaway
                    MattJanaway last edited by Sep 3, 2018, 5:57 PM Sep 3, 2018, 5:57 PM

                    I'd guess that the jobs get pulled from a job board. If this is the case, then the content ( job description, title etc.) will just be a duplication of the content that can be found in many other locations. If a plugin is used, they sometimes automatically add a disallow into the robots.txt file as to not hurt the parent version of the job page by creating thousands of duplicate content issues.

                    I'd recommend creating some really high-quality hub pages based on job type, or location and pulling the relevant jobs into that page, instead of trying to index and rank the actual job pages.

                    1 Reply Last reply Reply Quote 2
                    • 1 / 1
                    1 out of 9
                    • First post
                      1/9
                      Last post

                    Got a burning SEO question?

                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                    Start my free trial


                    Browse Questions

                    Explore more categories

                    • Moz Tools

                      Chat with the community about the Moz tools.

                    • SEO Tactics

                      Discuss the SEO process with fellow marketers

                    • Community

                      Discuss industry events, jobs, and news!

                    • Digital Marketing

                      Chat about tactics outside of SEO

                    • Research & Trends

                      Dive into research and trends in the search industry.

                    • Support

                      Connect on product support and feature requests.

                    • See all categories

                    Related Questions

                    • kbates

                      Trying to get Google to stop indexing an old site!

                      Howdy, I have a small dilemma. We built a new site for a client, but the old site is still ranking/indexed and we can't seem to get rid of it. We setup a 301 from the old site to the new one, as we have done many times before, but even though the old site is no longer live and the hosting package has been cancelled, the old site is still indexed. (The new site is at a completely different host.) We never had access to the old site, so we weren't able to request URL removal through GSC. Any guidance on how to get rid of the old site would be very appreciated. BTW, it's been about 60 days since we took these steps. Thanks, Kirk

                      Intermediate & Advanced SEO | Jul 29, 2019, 12:34 PM | kbates
                      0
                    • gozmoz

                      Moving html site to wordpress and 301 redirect from index.htm to index.php or just www.example.com

                      I found page duplicate content when using Moz crawl tool, see below. http://www.example.com
                      Page Authority 40
                      Linking Root Domains 31
                      External Link Count 138
                      Internal Link Count 18
                      Status Code 200
                      1 duplicate http://www.example.com/index.htm
                      Page Authority 19
                      Linking Root Domains 1
                      External Link Count 0
                      Internal Link Count 15
                      Status Code 200
                      1 duplicate I have recently transfered my old html site to wordpress.
                      To keep the urls the same I am using a plugin which appends .htm at the end of each page. My old site home page was index.htm. I have created index.htm in wordpress as well but now there is a conflict of duplicate content. I am using latest post as my home page which is index.php Question 1.
                      Should I also use redirect 301 im htaccess file to transfer index.htm page authority (19) to www.example.com If yes, do I use
                      Redirect 301 /index.htm http://www.example.com/index.php
                      or
                      Redirect 301 /index.htm http://www.example.com Question 2
                      Should I change my "Home" menu link to http://www.example.com instead of http://www.example.com/index.htm that would fix the duplicate content, as indx.htm does not exist anymore. Is there a better option? Thanks

                      Intermediate & Advanced SEO | Feb 23, 2017, 12:19 PM | gozmoz
                      0
                    • alphonseha

                      Google not Indexing images on CDN.

                      My URL is: http://bit.ly/1H2TArH We have set up a CDN on our own domain: http://bit.ly/292GkZC  We have an image sitemap: http://bit.ly/29ca5s3 The image sitemap uses the CDN URLs. We verified the CDN subdomain in GWT. The robots.txt does not restrict any of the photos: http://bit.ly/29eNSXv. We used to have a disallow to /thumb/ which had a 301 redirect to our CDN but we removed both the disallow in the robots.txt as well as the 301. Yet, GWT still reports none of our images on the CDN are indexed. The above screenshot is from the GWT of our main domain.The GWT from the CDN subdomain just shows 0. We did not submit a sitemap to the verified subdomain property because we already have a sitemap submitted to the property on the main domain name. While making a search of images indexed from our CDN, nothing comes up: http://bit.ly/293ZbC1While checking the GWT of the CDN subdomain, I have been getting crawling errors, mainly 500 level errors. Not that many in comparison to the number of images and traffic that we get on our website. Google is crawling, but it seems like it just doesn't index the pictures!? Can anyone help? I have followed all the information that I was able to find on the web but yet, our images on the CDN still can't seem to get indexed.

                      Intermediate & Advanced SEO | Jul 8, 2016, 10:35 AM | alphonseha
                      0
                    • boostaman

                      How to check if the page is indexable for SEs?

                      Hi, I'm building the extension for Chrome, which should show me the status of the indexability of the page I'm on. So, I need to know all the methods to check if the page has the potential to be crawled and indexed by a Search Engines. I've come up with a few methods: Check the URL in robots.txt file (if it's not disallowed) Check page metas (if there are not noindex meta) Check if page is the same for unregistered users (for those pages only available for registered users of the site) Are there any more methods to check if a particular page is indexable (or not closed for indexation) by Search Engines? Thanks in advance!

                      Intermediate & Advanced SEO | Jan 20, 2016, 10:30 PM | boostaman
                      0
                    • McTaggart

                      Membership/subscriber (/customer) only content and SEO best practice

                      Hello Mozzers, I was wondering whether there's any best practice guidance out there re: how to deal with membership/subscriber (existing customer) only content on a website, from an SEO perspective - what is best practice? A few SEOs have told me to make some of the content visible to Google, for SEO purposes, yet I'm really not sure whether this is acceptable / manipulative, and I don't want to upset Google (or users for that matter!) Thanks in advance, Luke

                      Intermediate & Advanced SEO | Mar 19, 2015, 7:10 PM | McTaggart
                      0
                    • friendoffood

                      Removing UpperCase URLs from Indexing

                      This search   -  site:www.qjamba.com/online-savings/automotix gives me this result from Google: Automotix online coupons and shopping - Qjamba
                      https://www.qjamba.com/online-savings/automotix
                      Online Coupons and Shopping Savings for Automotix. Coupon codes for online discounts on Vehicles & Parts products. and Google tells me there is another one, which is 'very simliar'.  When I click to see it I get: Automotix online coupons and shopping - Qjamba
                      https://www.qjamba.com/online-savings/Automotix
                      Online Coupons and Shopping Savings for Automotix. Coupon codes for online discounts on Vehicles & Parts products. This is because I recently changed my program to redirect all urls with uppercase in them to lower case, as it appears that all lowercase is strongly recommended. I assume that having 2 indexed urls for the same content dilutes link juice.  Can I safely remove all of my UpperCase indexed pages from Google without it affecting the indexing of the lower case urls?  And if, so what is the best way -- there are thousands.

                      Intermediate & Advanced SEO | Dec 12, 2014, 6:34 PM | friendoffood
                      0
                    • wtfi

                      Keyphrase / Keyword arrangement

                      Hi all, What are your thoughts on the arrangement of keyphrases / words? For example, does it make a difference if the words are arranged in the following way: "Keyword 1 Keyword 2" or "Keyword 2 Keyword 1" Both ways make a phrases which is favourable in the search engines. Can I stick with 1 way or should I be going with both arrangements. Hope that is clear 🙂

                      Intermediate & Advanced SEO | Oct 23, 2012, 7:42 PM | wtfi
                      0
                    • udemy

                      Disallowed Pages Still Showing Up in Google Index. What do we do?

                      We recently disallowed a wide variety of pages for www.udemy.com which we do not want google indexing (e.g., /tags or /lectures). Basically we don't want to spread our link juice around to all these pages that are never going to rank. We want to keep it focused on our core pages which are for our courses. We've added them as disallows in robots.txt, but after 2-3 weeks google is still showing them in it's index. When we lookup "site: udemy.com", for example, Google currently shows ~650,000 pages indexed... when really it should only be showing ~5,000 pages indexed. As another example, if you search for "site:udemy.com/tag", google shows 129,000 results. We've definitely added "/tag" into our robots.txt properly, so this should not be happening... Google showed be showing 0 results. Any ideas re: how we get Google to pay attention and re-index our site properly?

                      Intermediate & Advanced SEO | Feb 25, 2012, 3:48 PM | udemy
                      0

                    Get started with Moz Pro!

                    Unlock the power of advanced SEO tools and data-driven insights.

                    Start my free trial
                    Products
                    • Moz Pro
                    • Moz Local
                    • Moz API
                    • Moz Data
                    • STAT
                    • Product Updates
                    Moz Solutions
                    • SMB Solutions
                    • Agency Solutions
                    • Enterprise Solutions
                    Free SEO Tools
                    • Domain Authority Checker
                    • Link Explorer
                    • Keyword Explorer
                    • Competitive Research
                    • Brand Authority Checker
                    • Local Citation Checker
                    • MozBar Extension
                    • MozCast
                    Resources
                    • Blog
                    • SEO Learning Center
                    • Help Hub
                    • Beginner's Guide to SEO
                    • How-to Guides
                    • Moz Academy
                    • API Docs
                    About Moz
                    • About
                    • Team
                    • Careers
                    • Contact
                    Why Moz
                    • Case Studies
                    • Testimonials
                    Get Involved
                    • Become an Affiliate
                    • MozCon
                    • Webinars
                    • Practical Marketer Series
                    • MozPod
                    Connect with us

                    Contact the Help team

                    Join our newsletter
                    Moz logo
                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                    • Accessibility
                    • Terms of Use
                    • Privacy

                    Looks like your connection to Moz was lost, please wait while we try to reconnect.