Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Redirects - Large .htaccess file question
We are moving about 5000 pages from root into different folders. We need to individually 301 each page because the are sitting at root level now: mysite.com/page.com We want to move them to: mysite.com/folder/page.html etc I dont think redirect match can works because of the different files names and folders they are being moved in to. Will 5000 entries in .htacess slow site loading? Any other suggestions how to handle?
On-Page Optimization | | leadforms0 -
Two sites, one with a ccTLD domain, the other with TLD domain, same content
Hi there! I have a site which can be accessed with two different domains: one ccTLD for Spain: www.piensapiensa.es one TLD www.piensapiensa.com Should I take care of something regarding SEO? I have also a redirection from www.piensapiensa.com to piensapiensa.com. I have set up them in webmasters tools individually, with the same sitemap obviously. Thanks in advanced.
On-Page Optimization | | juanmiguelcr0 -
Positioning the keyword in two pages
Hi there! I've decided to use four criteria (keywords) for my website. The "problem" is that I have to use the same keyword (criteria) in two different pages. Is there a problem If I do this? On the other side, there are two sections of the web that (I assume) must have title and description tag as well as a keyword/criteria (Contact and Registration)....any advice?¿should they have a ttile and a description?¿Should they have a keyword associated? Thanks in advance for the answer.
On-Page Optimization | | juanmiguelcr0 -
Site Maps / Robots.txt etc
Hi everyone I have setup a site map using a Wordpress pluggin: http://lockcity.co.uk/site-map/ Can you please tell me if this is sufficient for the search engines? I am trying to understand the difference between this and having a robots.txt - or do I need both? Many thanks, Abi
On-Page Optimization | | LockCity0 -
How to optimize for a product by two names
So let us assume I am selling an item on my website and it comes in a large array of varieties. Let us also assume that this item is commonly referred to by two different names. (i.e. Cover & Case, Car & Automobile, Notepad & Notebook) Both of these names that are used, in regards to this product, have, for the sake of argument here, the exact same search volume. I want to make sure that I rank for both terms. In my Title Tags I am currently thinking about the following methodology to help that cause. "GE Motors Super Fast and Awesome Car / Automobile" "Ghostwriter Kids Notebook / Notepad" "Super Soft Pillow Cover / Case" Notice I have the space in between the words and the / but my question is if this is necessary or not? What is Google's policy on how they view that / ? Can I do this and still have Google see it as two different words? "GE Motors Super Fast and Awesome Car/Automobile" "Ghostwriter Kids Notebook/Notepad" "Super Soft Pillow Cover/Case" Apologies if this is a fairly basic question but cannot seem to find this information.
On-Page Optimization | | DRSearchEngOpt0 -
Title Tag Targeting Two Geographic Modifiers
How would you suggest writing a homepage title tag which needs to target both state and city? We are targeting these two phrases: Lincoln personal injury lawyer
On-Page Optimization | | blueroom
Nebraska personal injury lawyer Any thoughts on how you'd write the title tag? I know targeting Nebraska will be challenging but it's what the higher ups want to do. Would you suggest: Lincoln Nebraska Personal Injury Lawyer ?0 -
Two word keyords
Term Extractor gives me "realestate" as a result and I'd like to have "realestate CityName" instead. How ca I tell search engine to "skip" single word keyword and that two word keyword is more important on that page?
On-Page Optimization | | vdjacic0 -
What reasons exist to use noindex / robots.txt?
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
On-Page Optimization | | digitalstream0