Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should you have two separate pages for synonym keywords?
Suppose that you want to rank for two keyword phrases that mean the same thing but are slightly different in spelling. When should you put both keyword phrases on one page versus two pages? What are the pros and cons?
On-Page Optimization | | ProjectLabs0 -
Robots file include sitemap
Hello, I see that google, facebook and moz... have robots.txt include sitemap at the footer.
On-Page Optimization | | JohnHuynh
Eg: http://www.google.com.vn/robots.txt Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this? Thanks,0 -
Positioning the keyword in two pages
Hi there! I've decided to use four criteria (keywords) for my website. The "problem" is that I have to use the same keyword (criteria) in two different pages. Is there a problem If I do this? On the other side, there are two sections of the web that (I assume) must have title and description tag as well as a keyword/criteria (Contact and Registration)....any advice?¿should they have a ttile and a description?¿Should they have a keyword associated? Thanks in advance for the answer.
On-Page Optimization | | juanmiguelcr0 -
How to optimize for a product by two names
So let us assume I am selling an item on my website and it comes in a large array of varieties. Let us also assume that this item is commonly referred to by two different names. (i.e. Cover & Case, Car & Automobile, Notepad & Notebook) Both of these names that are used, in regards to this product, have, for the sake of argument here, the exact same search volume. I want to make sure that I rank for both terms. In my Title Tags I am currently thinking about the following methodology to help that cause. "GE Motors Super Fast and Awesome Car / Automobile" "Ghostwriter Kids Notebook / Notepad" "Super Soft Pillow Cover / Case" Notice I have the space in between the words and the / but my question is if this is necessary or not? What is Google's policy on how they view that / ? Can I do this and still have Google see it as two different words? "GE Motors Super Fast and Awesome Car/Automobile" "Ghostwriter Kids Notebook/Notepad" "Super Soft Pillow Cover/Case" Apologies if this is a fairly basic question but cannot seem to find this information.
On-Page Optimization | | DRSearchEngOpt0 -
What is the importance of image file names and H1, H2 and H3 tags
Hello, My client thinks they will get better rankings if we change the product image names to reflect the keywords for that page, is this true? I believe I just have to add "alt tags" but not sure now. Also, they think its very important to have H1, H2 and H3 tags on each page. I thought this wasn't a big factor anymore. Is it? Thanks
On-Page Optimization | | maximumrank0 -
What does this mean on first step up setting up a campaign? "Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here."
I am BRAND new to this, and setting up my first campaign. I choose subdomain, and entered www.pdsaz.com. This is the message I receive: We have detected that the domain www.pdsaz.com and the domain pdsaz.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
On-Page Optimization | | cschwartzel0 -
I have two pages ranking for the same keyword.
The index page and the targeted landing page for that keyword. They have different content, title, meta but I am competing with myself for the main keyword in the industry. What is the best way to fix this? 301 the keyword page to the index page?
On-Page Optimization | | Aftermath_SEO0 -
Does Google respect User-agent rules in robots.txt?
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site. LinkSmart uses a bot to establish the linking. The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking. LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent. I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore? Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
On-Page Optimization | | lzhao0