What are your thoughts on security of placing CMS-related folders in a robots.txt file?
-
So I was just about to add a whole heap of CMS-related folders to my robots.txt file to exclude them from search, and thought "hey, I'm publicly telling people where my admin folders are"...surely that's not right?!
Should I leave them out of the robots.txt file, and hope for the best that they never get indexed? Should I use noindex meta data on every page?
What are people's thoughts?
Thanks,
James
PS. I know this is similar to lots of other discussions around meta noindex vs. robots.txt, but I'm after specific thoughts around the security aspect of listing your admin folders in a robots.txt file...
-
surly your admin folders are secured?, it would not matter if someone knows where they are.
-
As a rule, you want to avoid using robots.txt files whenever possible. It does not consistently protect you from crawlers and when it does block crawlers it kills any PR on those pages.
If you can block those pages with a noindex tag, it would be a preferable solution.
With respect to security for a CMS site, it really needs to be a comprehensive effort. Many site owners take a couple steps and then have a false-sense of security. Here are a few thoughts:
-
try the site address with /administrator after it to access Joomla and other sites
-
try the site address or blog with /wp-admin/ after it to access Joomla sites
-
make up a webpage and try accessing it to view the site's 404 page
-
right-click on a page and choose View Page Source. Often you will see the name of the CMS clearly listed. Other times you will see clear clues such as /wp/ in folder names. Other times you will find unique extensions such as Yoast SEO which will give you an idea of the CMS
Once a bad guy knows which CMS is in use, they know the default folder structure and more. The point is it requires a lot more effort then most people realize to hide the CMS in use. I applaud your effort, but be very thorough about it. There is a lot more involved then simply covering your robots.txt file.
-
-
I found three options for you: http://www.techiecorner.com/106/how-to-disable-directory-browsing-using-htaccess-apache-web-server/
I think if you do it with.htacces that is a folder specific file than nobody will be able to detect where admin contet is located.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Thoughts on Botify?
Has anyone used Botify? Is this type of software necessary for a site with under 5K pages?
Technical SEO | | SoulSurfer80 -
Little confused regarding robots.txt
Hi there Mozzers! As a newbie, I have a question that what could happen if I write my robots.txt file like this... User-agent: * Allow: / Disallow: /abc-1/ Disallow: /bcd/ Disallow: /agd1/ User-agent: * Disallow: / Hope to hear from you...
Technical SEO | | DenorL0 -
Best tool to find Related keywords with a Keyword provided.
Hi, Best tool to find Related keywords with a Keyword provided. Basically i want to give a keyword and find all related keywords we can use to write articles. Also any way we can find what keyword a page is getting traffic based on? Thanks
Technical SEO | | skandlikp90 -
Sub-domain or sub-folder for a blog?
Traditional thinking suggests sub-domains are treated as separate sites and so don't pass on link juice, but I've heard mixed opinions. I'm very much a believer in sub-folders but I'm interested to hear some other opinions. Thoughts?
Technical SEO | | underscorelive0 -
How do i show my link xls file to google?
i have removed lots of links and contacted lots of webmaster to clean up my link profile. I have a large xls file to send to google for them to see that we have done a lot to clean up the bad links. How would i show this file to google? is there a place where I can post it? or email ? thank you nick
Technical SEO | | orion680 -
What would happen if 301 redirects were not in place
Good Morning from 14 degrees C sunny Wetherby UK 🙂 My question is please.... "When a new site is given a total makover ie old urls are re written to radically different ones I know if you dont set up 301 redirects the infamous 404 error page will rear its head. But i wonder if 301 redirects were not configured how long on average does it take google to index the new site and serp links finally point to the new site". Thanks in advance 🙂
Technical SEO | | Nightwing0 -
Robots.txt Syntax
Does the order of the robots.txt syntax matter in SEO? For example (are there potential problems with this format): User-agent: * Sitemap: Disallow: /form.htm Allow: / Disallow: /cgnet_directory
Technical SEO | | RodrigoStockebrand0 -
URL length - Moving from 6 folder deep to 3 folder deep
I have a website with friendly URL, My product page are generated from a database. When I click on the product, I go 6 folders deep. (ex. nameofthesite/courses-and-seminard/blablabla-catalog/information-technology/blablabla-window/blabla-server/active-directory-with-windows-server-2008.html I'm thinking about moving the product page to the thrid folder, so it would become nameofthesite.com/courses-and-seminars/course-catalog/active-directory-with-windows-server-2008.html I want to know, would it be a major element in my SEO. Is URL length is a really important factor. Because I need to move 450 pages. Second, if I move the page, do I only need to add redirect 301 or I need to do something else. I suppose I also need to change my breadcrumb navigation also.
Technical SEO | | Adviso0