Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal Linking : File Name or URI/filename.html
Hi, For crawlability, what would be the best way to do internal linking. /aboutus.html or www.xyz.com/aboutus.com Would this make any difference to the website in terms of SEO or bot crawling the website? Regards, Sree
On-Page Optimization | | jungleegames0 -
Robots file include sitemap
Hello, I see that google, facebook and moz... have robots.txt include sitemap at the footer.
On-Page Optimization | | JohnHuynh
Eg: http://www.google.com.vn/robots.txt Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this? Thanks,0 -
Does concatenating two words make them lose their SEO value
I'm working with a client whose product titles are things like "InternetSecurity" and for branding purposes has thus far required any reference to the products be concatenated in this fashion (on page, links, URL, title tags). It seems to me that would greatly reduce their SEO relevance. Any thoughts? Thanks, Mark
On-Page Optimization | | DenverKelly0 -
In my report of my website it was indicated that I had 19 links/locations blocked by meta-robots. What does this mean and how do I fix it. My website is a Wordpress website.
In my report of my website it was indicated that I had 19 links/locations blocked by meta-robots. What does this mean and how do I fix it. My website is a Wordpress website.
On-Page Optimization | | cyaindc0 -
Yahoo small business host has two web sites and confuses the spiders
Both another consultant and my first seomoz test stated that I have two websites and that confuses the spiders. . One is www.mystore.com ( example) and the other is mystore.com. Yahoo states that's how they role and I can not delete one. I can do a 301 redirect and redirect one site to the other but they do not have a recommendaation as to which one to redicrect. Which one should I redirrect?
On-Page Optimization | | Wales0 -
Image Optimization - File Name Important?
I am currently working on a site with 100+ recipes that all have image file names that are relevant, but not optimized for keyword purposes. I'm wondering - from an SEO perspective - would it be worth my time to go back through all of the images and rename them with keywords in mind? On my own site I have always done this as a "best practice" but I'm curious - does it make a difference to search engines? Does anyone have any recent research/experiences that they would like to share? Thanks!
On-Page Optimization | | EssEEmily0 -
URL with two forward slashes //
We have a potential client with a URL structure in this fashion: http://www.site-url.com//cpage/page.html pretty strange, right? my question is: How bad are the 2 forward slashes // for SEO? How bad is it to have that extra layer of /cpage in the URL? this doesn't appear to serve any other purpose than making the URL longer than necessary.
On-Page Optimization | | Motava0 -
Image files names - should each be different ?
Hi SEOs I have created a page on my website with a certain tutorial - let's say "How to change a tyre in your car". I have Grade A from On-page report using SEOmoz tool. I have put a relevant alt description for each photo I have put in that tutorial and I have used relevant keyword as a name for each image. So for example this would look like that: changing-tyre-1.jpg changing-tyre-2.jpg My question is should I name these file differently according to their content ? For example if on my photo there is some tool I have to use to change the tyre should the file have the name relevant to the subject of the article/tutorial on the page or to the content of the image ? Is that a stupid question ? Or am I getting too fussy ? Thanks
On-Page Optimization | | lolskizz0