Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta Robots index & noindex Both Implemented on Website
I don't want few of the pages of website to get indexed by Google, thus I have implemented meta robots noindex code on those specific pages. Due to some complications I am not able to remove meta robots index from header of every page Now, on specific pages I have both codes 'index & noindex' implemented. Question is: Will Google crawl/index pages which have noindex code along with index code? Thanks!
On-Page Optimization | | Exa0 -
Meta Geotag - two locations on one website
I have a client that I would like to do a Meta Geotag for. They have two locations. Am I able to do two meta geotags on their website? Thanks!
On-Page Optimization | | OOMDODigital0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
Two keywords in one page
Hi guys, I have a question...is it possible to posicionate two keywords in one only page? If yes, how would it be the process so that Google take note of that action/s. How many criteria/keywords are recommended to positionate in one site? Thanks all
On-Page Optimization | | juanmiguelcr0 -
What is the best way to have two online stores, each focusing on a specific country?
The dilemma here of course is that the stores will be very similar in content, with the domains only being different in that the store targeting the US will be a .com domain and the Canadian store is a .ca domain.
On-Page Optimization | | absoauto0 -
How to optimize for a product by two names
So let us assume I am selling an item on my website and it comes in a large array of varieties. Let us also assume that this item is commonly referred to by two different names. (i.e. Cover & Case, Car & Automobile, Notepad & Notebook) Both of these names that are used, in regards to this product, have, for the sake of argument here, the exact same search volume. I want to make sure that I rank for both terms. In my Title Tags I am currently thinking about the following methodology to help that cause. "GE Motors Super Fast and Awesome Car / Automobile" "Ghostwriter Kids Notebook / Notepad" "Super Soft Pillow Cover / Case" Notice I have the space in between the words and the / but my question is if this is necessary or not? What is Google's policy on how they view that / ? Can I do this and still have Google see it as two different words? "GE Motors Super Fast and Awesome Car/Automobile" "Ghostwriter Kids Notebook/Notepad" "Super Soft Pillow Cover/Case" Apologies if this is a fairly basic question but cannot seem to find this information.
On-Page Optimization | | DRSearchEngOpt0 -
Rate of similarity between two pages
hello, what is the maximum rate of similarity between two pages to avoid duplicate content? We often talk (in France) at a rate of 70% Thanks for your answers Denis
On-Page Optimization | | android_lyon0 -
I have a direct question about file structure.
This question is about a new file structure and SEO friendly URL's. Does a file name make a difference? I have a direct question about file structure. Our old site was formated with a URL of http://rousechamberlin.com/about_us.aspx our new site is structured http://rousechamberlin.com/AboutUs/ no file no extension. As the SEO guy of the company and not the programmer my feeling is this is killing us. Does anybody have any thoughts on this?
On-Page Optimization | | HeadWebChef0