undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • SEO Q&A

      Insights & discussions from an SEO community of 500,000+.

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Robots txt. in page with 301 redirect

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Robots txt. in page with 301 redirect

Technical SEO
2
6
4.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Kilgray
    Kilgray last edited by Mar 7, 2016, 9:42 AM

    We currently have a a series of help pages that we would like to disallow from our robots txt.

    The thing is that these help pages are located in our old website, which now has a 301 redirect to current site.

    Which is the proper way to go around?

    1- Add the pages we want to disallow to the robots.txt of the new website?

    2- Break the redirect momentarily and add the pages to the robots.txt of the old one?

    Thanks

    1 Reply Last reply Reply Quote 0
    • LauraSultan
      LauraSultan @Kilgray last edited by Mar 7, 2016, 2:25 PM Mar 7, 2016, 2:13 PM

      In that case, you'd need to add the robots meta tag at the page level before the tag.

      or

      1 Reply Last reply Reply Quote 1
      • Kilgray
        Kilgray @LauraSultan last edited by Mar 7, 2016, 1:21 PM Mar 7, 2016, 1:21 PM

        Hey, for some time we will keep the files in the old domain. Should we break the redirect and insert the disallows to the robot.txt of the old site?

        LauraSultan 1 Reply Last reply Mar 7, 2016, 2:13 PM Reply Quote 0
        • LauraSultan
          LauraSultan @Kilgray last edited by Mar 7, 2016, 12:06 PM Mar 7, 2016, 12:06 PM

          So, the problem is that the robots.txt file can't be accessed because of the 301 redirect to the new domain?

          Do you plan to keep the help files on the old domain, or will they be removed completely?

          Kilgray 1 Reply Last reply Mar 7, 2016, 1:21 PM Reply Quote 0
          • Kilgray
            Kilgray @LauraSultan last edited by Mar 7, 2016, 11:25 AM Mar 7, 2016, 11:25 AM

            Hi Laura,

            Thanks for your reply. I don't want to disallow the URLs these pages are being redirected to. Actually these URLs are in the old version but still can be accessed. So to put it simply, this is my case:

            1- This was our current website: www.kilgray.com (With a 301 redirect)

            2- This is our new website: www.memoq.com

            3- I would like to disallow the following links on the old website that are still visible (haven't been redirected):

            http://kilgray.com/memoq/2015-100/help-en/index.html

            http://kilgray.com/memoq/2014/help-en/

            http://kilgray.com/memoq/2013R2/help-full-en/

            http://kilgray.com/memoq/60/help-en/

            LauraSultan 1 Reply Last reply Mar 7, 2016, 12:06 PM Reply Quote 0
            • LauraSultan
              LauraSultan last edited by Mar 7, 2016, 11:11 AM Mar 7, 2016, 11:11 AM

              Do you want to disallow the URLs that these pages are being redirected to? If not, there's no need to add anything to the robots.txt file.

              If you do want to disallow the URLs that these pages are being redirected to, use relative URLs in your robots.txt file. For example, let's say olddomain.com/old-help-page/ is being redirected to newdomain.com/new-help-page/. If that's the case, add the following to your robots.txt file.

              Disallow: /new-help-page/

              There's no need to disallow the specific URLs that are being redirected to something else.  Are you trying to get them removed from Google's index or something?  If so, Google will update their index eventually based on your 301 redirects.

              Kilgray 1 Reply Last reply Mar 7, 2016, 11:25 AM Reply Quote 0
              • 1 / 1
              1 out of 6
              • First post
                1/6
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • AmandaBridge

                Disallow wildcard match in Robots.txt

                This is in my robots.txt file, does anyone know what this is supposed to accomplish, it doesn't appear to be blocking URLs with question marks Disallow: /?crawler=1
                Disallow: /?mobile=1 Thank you

                Technical SEO | Aug 28, 2018, 9:50 AM | AmandaBridge
                0
              • shawnbeaird

                301 Redirect for multiple links

                I just relaunched my website and changed a permalink structure for several pages where only a subdirectory name changed. What 301 Redirect code do I use to redirect the following? I have dozens of these where I need to change just the directory name from "urban-living" to "urban", and want it to catch the following all in one redirect command. Here is an example of the structure that needs to change. Old 
                domain.com/urban-living (single page w/ content)
                domain.com/urban-living/tempe (single page w/ content)
                domain.com/urban-living/tempe/the-vale (single page w/ content) New 
                domain.com/urban 
                domain.com/urban/tempe 
                domain.com/urban/tempe/the-vale

                Technical SEO | Jan 7, 2017, 3:01 PM | shawnbeaird
                0
              • Webicultors

                2 sitemaps on my robots.txt?

                Hi, I thought that I just could link one sitemap from my site's robots.txt but... I may be wrong. So, I need to confirm if this kind of implementation is right or wrong: robots.txt for Magento Community and Enterprise ...
                Sitemap: http://www.mysite.es/media/sitemap/es.xml
                Sitemap: http://www.mysite.pt/media/sitemap/pt.xml Thanks in advance,

                Technical SEO | Nov 30, 2016, 6:17 PM | Webicultors
                0
              • Emory_Peterson

                301 Redirects Relating to Your XML Sitemap

                Lets say you've got a website and it had quite a few pages that for lack of a better term were like an infomercial, 6-8 pages of slightly different topics all essentially saying the same thing.  You could all but call it spam. www.site.com/page-1 www.site.com/page-2 www.site.com/page-3 www.site.com/page-4 www.site.com/page-5 www.site.com/page-6 Now you decided to consolidate all of that information into one well written page, and while the previous pages may have been a bit spammy they did indeed have SOME juice to pass through. Your new page is: www.site.com/not-spammy-page You then 301 redirect the previous 'spammy' pages to the new page.  Now the question, do I immediately re-submit an updated xml sitemap to Google, which would NOT contain all of the old URL's, thus making me assume Google would miss the 301 redirect/seo juice.  Or do I wait a week or two, allow Google to re-crawl the site and see the existing 301's and once they've taken notice of the changes submit an updated sitemap? Probably a stupid question I understand, but I want to ensure I'm following the best practices given the situation, thanks guys and girls!

                Technical SEO | Mar 23, 2015, 2:10 PM | Emory_Peterson
                0
              • Whittie

                Robots.txt on subdomains

                Hi guys! I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?

                Technical SEO | Feb 19, 2015, 2:59 PM | Whittie
                0
              • imaginex

                Should I block Map pages with robots.txt?

                Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!

                Technical SEO | Oct 29, 2014, 3:37 PM | imaginex
                0
              • Artience

                301 Redirect & Cloaking

                HEllo~~~~ People. I have a question regarding on cloaking. I will be really greatful if you can help me with question. I have a site www.example.com and it is targeting for multi countries. So I use sub directories for targeting multi countries. e.g. www.example.com/us/ www.example.com/de/ www.example.com/hk/ ....... so on and on. Therefore, when people type www.example.com, I use IP delivery to send users to each coutries. Here is my question. I use 301 redirect for IP delivery, which means when user enter www.example.com, my site read user's IP and send them to right country site by 301 redirect. In this case, is there any possibility that Google considers it as cloaking? Please people.... share me some ideas and thoughs.

                Technical SEO | Jul 15, 2011, 11:56 AM | Artience
                0
              • craigycraig

                301 Redirect "wildcard" question

                I have been looking at the SEOmoz redirect guide for some advice but I can't seem to find the answer : http://www.seomoz.org/learn-seo/redirection I have lots of URLs from a previous version of a site that look like the following: sitename.com/-c-25.html?sort=2d&page=1 sitename.com/-c-25.html?sort=3a&page=1 etc etc. I want to write a redirect so whenever a URL with the terms "-c-25.html" is requested it redirects to a specified page, regardless of what comes after the question mark. These URLs were created by our previous ecommerce software. The 'c' is for category, and each page of the cateogry created a different URL. I want to do these so I can rediect all of these URLs to the appropraite new cateogry page in a single redirect. Thanks for any help.

                Technical SEO | Dec 23, 2012, 6:28 PM | craigycraig
                0

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.