What are your thoughts on security of placing CMS-related folders in a robots.txt file?
-
So I was just about to add a whole heap of CMS-related folders to my robots.txt file to exclude them from search, and thought "hey, I'm publicly telling people where my admin folders are"...surely that's not right?!
Should I leave them out of the robots.txt file, and hope for the best that they never get indexed? Should I use noindex meta data on every page?
What are people's thoughts?
Thanks,
James
PS. I know this is similar to lots of other discussions around meta noindex vs. robots.txt, but I'm after specific thoughts around the security aspect of listing your admin folders in a robots.txt file...
-
surly your admin folders are secured?, it would not matter if someone knows where they are.
-
As a rule, you want to avoid using robots.txt files whenever possible. It does not consistently protect you from crawlers and when it does block crawlers it kills any PR on those pages.
If you can block those pages with a noindex tag, it would be a preferable solution.
With respect to security for a CMS site, it really needs to be a comprehensive effort. Many site owners take a couple steps and then have a false-sense of security. Here are a few thoughts:
-
try the site address with /administrator after it to access Joomla and other sites
-
try the site address or blog with /wp-admin/ after it to access Joomla sites
-
make up a webpage and try accessing it to view the site's 404 page
-
right-click on a page and choose View Page Source. Often you will see the name of the CMS clearly listed. Other times you will see clear clues such as /wp/ in folder names. Other times you will find unique extensions such as Yoast SEO which will give you an idea of the CMS
Once a bad guy knows which CMS is in use, they know the default folder structure and more. The point is it requires a lot more effort then most people realize to hide the CMS in use. I applaud your effort, but be very thorough about it. There is a lot more involved then simply covering your robots.txt file.
-
-
I found three options for you: http://www.techiecorner.com/106/how-to-disable-directory-browsing-using-htaccess-apache-web-server/
I think if you do it with.htacces that is a folder specific file than nobody will be able to detect where admin contet is located.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will a robots.txt disallow apply to a 301ed URL?
Hi there, I have a robots.txt query which I haven't tried before and as we're nearing a big time for sales, I'm hesitant to just roll out to live! Say for example, in my robots.txt I disallow the URL 'example1.html'. In reality, 'example1.html' 301s/302s to 'example2.html'. Would the robots.txt directive also apply to 'example2.html' (disallow) or as it's a separate URL, would the directive be ignored as it's not valid? I have a feeling that as it's a separate URL, the robots disallow directive won't apply. However, just thought I'd sense-check with the community.
Technical SEO | | ecommercebc0 -
How to rank in Google Places
Normally, I don't have a problem with local SEO (more of a multi-channel sort of online marketing guy) but this one has got me scratching my head. Look at https://www.google.co.uk/search?q=wedding+venues+in+essex Theres two websites there (fennes and quendon park) that both have a much more powerful DA but don't appear in the Google Places (Google + Business or whatever it's labeled as). Why are websites such as Boreham house ranking top in the map listings? Quendon Park has a Google places listing, it's full of content, the NAP all matches up. Its a stronger website. Boreham House isn't any closer to the centroid than Quendon Park Just got me struggling this one
Technical SEO | | jasonwdexter0 -
The use of robots.txt
Could someone please confirm that if I do not want to block any pages from my URL, then I do not need a robots.txt file on my site? Thanks
Technical SEO | | ICON_Malta0 -
Does anyone have any thoughts about this site: web-archive-uk.com?
They have "archived" a lot of our content. We have high-quality content that has been copied by others. We are in the process of sending DMCA notices to the relevant search engines that have indexed these sites. As well as to the actual sites. Are we right to believe that copied content on 3rd party websites damages/diminishes the seo of our own site? (Leaving aside the straight forward theft issue.) BTW actually impressed by the swift responses from Google to DMCA requests.
Technical SEO | | dexm100 -
How should I properly setup my .htaccess file?
I have searched google for 'how to setup .htaccess file' and it seems that every website has some variation. For example: RewriteCond %{HTTP_HOST} ^yoursite.com RewriteRule ^(.*)$ http://www.yoursite.com/$1 [R=permanent,L] On SEOMOZ someone posted this: RewriteCond %{HTTP_HOST} !^www.yoursite.com [NC] RewriteRule (.*) http://www.yoursite.com/$1 [L,R=301] On yet another website, I found this: RewriteEngine On RewriteCond %{HTTP_HOST} !^your-site.com$ [NC] RewriteRule ^(.*)$ http://your-site.com/$1 [L,R=301] As you can see there are slight differences. Which one do I use? I'm on Apache CentOS and I have HTML5 websites and several Joomla! wesites. Would the HTACCESS File be different for both?
Technical SEO | | maxduveen0 -
Best way to change from one CMS to wordpress??
I have a client that was working with another SEO consultant and they pretty much dropped the ball big time! The SEO company took his site off of wordpress and put it on another CMS (http://www.wsinetsuccess.com/WSI-E-Fusion). My client would like to take hes site off of the WSI E Fusion platform and back onto wordpress. My question is how is the best way to going about doing this with out loosing all of the PR to the site? Should I find all of the URLs that are on the site and just 301 them to the right page? Thank you very much for your help.
Technical SEO | | pakevin0 -
Google Places - need to update
I've got a client who verified their Google Places listing years ago, and noone knows who so I can't access it. The business is now moving and I need to update the address. What should I do? thanks
Technical SEO | | garymeld0 -
Robots.txt file getting a 500 error - is this a problem?
Hello all! While doing some routine health checks on a few of our client sites, I spotted that a new client of ours - who's website was not designed built by us - is returning a 500 internal server error when I try to look at the robots.txt file. As we don't host / maintain their site, I would have to go through their head office to get this changed, which isn't a problem but I just wanted to check whether this error will actually be having a negative effect on their site / whether there's a benefit to getting this changed? Thanks in advance!
Technical SEO | | themegroup0