The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Forum
    2. Categories
    3. SEO Tactics
    4. Technical SEO
    5. Is there any value in having a blank robots.txt file?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Is there any value in having a blank robots.txt file?

    Technical SEO
    6 4 4.9k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • NicDale
      NicDale last edited by

      I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place.  Is there any merit in having a blank robots.txt file?

      What is the minimum you would include in a basic robots.txt file?

      1 Reply Last reply Reply Quote 0
      • LiamVictor
        LiamVictor last edited by

        I know this is four years old, but there's value in having a blank robots.txt as some tools (including the latest version of the Moz crawler) will baulk at sites without a robots.txt file.

        1 Reply Last reply Reply Quote 0
        • NicDale
          NicDale last edited by

          Thanks for both of your replies.  As per my question it was around whether there is any value having a blank robots.txt file.  Philipp's answer was right on the money.

          1 Reply Last reply Reply Quote 0
          • Modi
            Modi @zeepartner last edited by

            i mentioned same only, The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site."

            n has added - More and more people use robots,txt to disallow access to some administration or private folders of the site

            1 Reply Last reply Reply Quote 0
            • zeepartner
              zeepartner last edited by

              No use in having a blank robots.txt. Minimum requirement if you want to have your site crawled is this:

              User-agent: *
              Allow: /
              

              Note that Gagans example above will block the entire site.

              Modi 1 Reply Last reply Reply Quote 1
              • Modi
                Modi last edited by

                Hi, This is what i got

                " Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called_The Robots Exclusion Protocol_. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

                User-agent: *
                Disallow: /
                
                

                The "<tt>User-agent: *</tt>" means this section applies to all robots. The "<tt>Disallow: /</tt>" tells the robot that it should not visit any pages on the site."

                More and more people use robots,txt to disallow access to some administration or private folders of the site . If you dont want to hide anything then may be you can leave it blank

                1 Reply Last reply Reply Quote 0
                • 1 / 1
                • First post
                  Last post

                Got a burning SEO question?

                Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                Start my free trial


                Explore more categories

                • Moz Tools

                  Chat with the community about the Moz tools.

                  Getting Started
                  Moz Pro
                  Moz Local
                  Moz Bar
                  API
                  What's New

                • SEO Tactics

                  Discuss the SEO process with fellow marketers

                  Content Development
                  Competitive Research
                  Keyword Research
                  Link Building
                  On-Page Optimization
                  Technical SEO
                  Reporting & Analytics
                  Intermediate & Advanced SEO
                  Image & Video Optimization
                  International SEO
                  Local SEO

                • Community

                  Discuss industry events, jobs, and news!

                  Moz Blog
                  Moz News
                  Industry News
                  Jobs and Opportunities
                  SEO Learn Center
                  Whiteboard Friday

                • Digital Marketing

                  Chat about tactics outside of SEO

                  Affiliate Marketing
                  Branding
                  Conversion Rate Optimization
                  Web Design
                  Paid Search Marketing
                  Social Media

                • Research & Trends

                  Dive into research and trends in the search industry.

                  SERP Trends
                  Search Behavior
                  Algorithm Updates
                  White Hat / Black Hat SEO
                  Other SEO Tools

                • Support

                  Connect on product support and feature requests.

                  Product Support
                  Feature Requests
                  Participate in User Research

                • See all categories

                • One robots.txt file for multiple sites?
                  renalynd27
                  renalynd27
                  0
                  4
                  2.2k

                • Oh no googlebot can not access my robots.txt file
                  BistosAmerica
                  BistosAmerica
                  0
                  9
                  7.8k

                • Does Bing ignore robots txt files?
                  Nightwing
                  Nightwing
                  0
                  3
                  4.2k

                • Robots.txt file getting a 500 error - is this a problem?
                  themegroup
                  themegroup
                  0
                  3
                  4.0k

                • Robots.txt File Redirects to Home Page
                  kchandler
                  kchandler
                  0
                  6
                  6.5k

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy

                Looks like your connection to Moz was lost, please wait while we try to reconnect.