Robots.txt | any SEO advantage to having one vs not having one?
-
Neither of my sites has a robots.txt file. I guess I have never been bothered by any particular bot enough to exclude it.
Is there any SEO advantage to having one anyways?
-
It's good practice, especially if you are operating a CMS that can create accessible URLs that cause duplicate content problems, create "junk" pages, etc. For example: http://www.asos.com/robots.txt
Google dislikes search results pages being indexed, so you can block those off, e.g. http://moz.com/robots.txt
You can disallow the archive.org bot if you don't want old versions of your site appearing in its search engine, and as others have said you can point to your xml sitemap.
It's not a bad resource to have at your disposal for site hygiene / maintenance reasons, but it's not an absolute necessity either.
-
There are actually a couple good reasons but in short, it's "best practice" so it won't hurt by adding it in. It wont take more than a couple minutes.
-
Just good practice. One SEO advantage would be to include a reference to your sitemap within the robots.txt file.
Aside from that, if you want all of your pages crawled and don't have a sitemap (although you should), no need for a robots.txt file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 vs 302
Hello everyone! I'm working with a site right now that is currently formatted as subdomain.domain.net. The old version of the site was formatted as domain.net, with domain.com and several other variants redirecting to the current format, subdomain.domain.net. All of these redirects are 302, and I'm wondering if I should have all these changed to 301. Many of our old backlinks go to the old format of domain.net and i know the juice isn't being passed through, but i was wondering if there is any reason why you may want a 302 over a 301 in this case? Any insight would be appreciated. Thanks!
Technical SEO | | KathleenDC0 -
Do I have a robots.txt problem?
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO This version shows no errors or warnings- http://imgur.com/uqbmbug Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
Technical SEO | | EcommerceSite0 -
Blocked URL's by robots.txt
In Google Webmaster Tools shows me 10,936 Blocked URL's by robots.txt and it is very strange when you go to the "Index Status" section where shows that since April 2012 robots.txt blocked many URL's. You can see more precise on the image attached (chart WMT) I can not explain why I have blocked URL's ? because I have nothing in robots.txt.
Technical SEO | | meralucian37
My robots.txt is like this: User-agent: * I thought I was penalized by Penguin in April 2012 because constantly i'am losing visitors now reaching over 40%. It may be a different penalty? Any help is welcome because i'm already so saturated. Mera robotstxt.jpg0 -
Robots.txt file
How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line? Do i need to install a Robots.txt file on each page?
Technical SEO | | gimes0 -
Country domain: Seo for other languages
Hi, I have an italian domain (.it) for an italian hotel, it is an old authoritative domain (1997) and it is well optimized for the keywords that include the city the hotel is in, now the page is decently positioned in Google Italy. There are many problems to have the same rank for German version (in google.de, google.at). The German version is in the /de folder. The hotel has another .com domain, much less authoritative (2007), in a German server, but it was and is only a simple redirect 301 (by code) to the German version in the .it domain. (obviously the rank for this domain is almost nonexistent). Do you have any suggestion? Thank you.
Technical SEO | | depi0 -
Importance of SEO Friendly URLs
Hey SEOZ! How important do you all think SEO friendly URLs are to SEO. Here is an example: Non: http://www.domain.com/cart.php?m=product_detail&p=2964 Friendly: http://www.domain.com/product_name_here.html I have always heard mixed reviews but did an experiment comparing results on the same domain and actually noticed quite a difference with the friendly ones. Thanks!
Technical SEO | | 6thirty0 -
Wordpress plugins for SEO
Hello I am new to wordpress I just have started using it. Can anyone suggest me some useful tools / plugins / setting for SEO? I am further intrested in sepeeding up Wp. Any good advive on wp an seo would be appriciated.
Technical SEO | | sesertin0 -
Global SEO Targeting
Hi, I have a website currently on the domain example.co.uk (.com is not available) I'm looking to enter other markets such as Brazil, Russia - obviously content will need to change to suit the desired market / language. I'm looking for some information on the best practice to enter foreign markets. I was thinking maybe to create individual sites for each location eg: example.br example.ru This way I could localise each site in terms of business directories, content, language etc. Or have my example.co.uk with various languages on it Experience, suggestions are welcomed - thanks.
Technical SEO | | Socialdude0