How to prevent directory from being accessed by search engines?
-
Pretty much as the question says, is there any way to stop search engines from crawling a directory? I am working on a Wordpress installation for my site but don't want it to be listed in search engines until it's ready to be shown to the world. I know the simplest way is to password-protect the directory but I had some issues when I tried to implement that so I'd like to see if there's a way to do it without passwords. Thanks in advance.
-
But don't forget to remove that Disallow out of Robots.txt when you go live - if you want those pages to be indexed (and also the Meta-robots noindex nofollow).
Otherwise you might be pulling your hair out trying to figure out why none of your pages are getting indexed in the SERPs.
-
You're absolutely right! I left that part out. Thanks
-
The robots.txt file does not guarantee that your pages will not show up in search results! Your best bet after password protection is adding a NoIndex meta tag to you page headers.
Google have openly said that they obey this tag (Matt Cutts).
-
Xee,
It always help, and it is very easy to implement. This function to show the path to the sitemap ir very good.
-
It's not required to have the ending slash. At least, it works for us without it.
-
As it is, my site is just phpBB3 forums (www.bearsfansonline.com); would a sitemap really help that much?
-
If you don't have an robot.txt file, you need to include some important stuff first.
First, do you have a sitemap.xlm for your website? If not, its very important and you should creat it at: http://www.xml-sitemaps.com/
Create a robot.txt file and include the follow:
User-agent: * allow: / disallow: /directoryname
Sitemap: http://www.yousite.com/sitemap.xmlWith this you will inform all robots where is your sitemap. You should read more about robots.txt in this great post: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
shouldn't you put a slash at the end of the directory in the robots file?
you can create the robots file through the Google Webmaster Tools
-
I don't have a robots.txt file in my root. Do I just create a text file, put the above lines into it, and upload it to my root after changing the name?
-
I'm assuming you want all search engines blocked from this directory. If so, edit your robots.txt file to state the following. This will block all bots from accessing a folder/directory on your site
User-agent: *
Disallow: /directoryname
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Console Indexed Page Count vs Site:Search Operator page count
We launched a new site and Google Search Console is showing 39 pages have been indexed. When I perform a Site:myurl.com search I see over 100 pages that appear to be indexed. Which is correct and why is there a discrepancy? Also, Search Console Page Index count started at 39 pages on 5/21 and has not increased even though we have hundreds of pages to index. But I do see more results each week from Site:psglearning.com My site is https://wwww.psglearning.com
Technical SEO | | pdowling0 -
Google webmaster and analytics access to others
Hi, Google webmaster and analytics access to others with restricted and user access is this ok and secure? With this access can anyone tamper anything? thanks
Technical SEO | | mtthompsons0 -
How to get found on local google search?
Hey When look for particular local businesses on Google like "Manchester Hotels" for example; at the top of Google page come all business up that are marked on the city map (with A,B,C..), then all the others follow. So my question is: "How can I get my business on that map?". Thank you! Ve
Technical SEO | | MissVe0 -
Increase Search Ranking for CEO
Hi guys My company CEO is concerned that when her name is googled pictures of a glamour model appear in the image results area. The glamour model shares a second name with our CEO and this is why the model's images are appearing. I have been asked to rectify this situation. My CEO has a linked in page and twitter account which are underused but no personal page on our company website. I was thinking of buying the url for the CEO's name and optimizing a small site for her name with bio etc and links to twitter, lined in etc. Would this be the best strategy? Thanks Gavin
Technical SEO | | gavinr0 -
Why isn't Google pushing my Schema data to the search results page
I believe we have it set up right. I'm noticing all my competitors schema data is showing up which is really giving them a leg up on us. We have a high ranking website so I'm just not sure why it's now showing up. Here is an example URL http://www.airgundepot.com/3576w.html I've used the Google webmaster tools tester and it all looks fine. Any ideas? Thanks in advance.
Technical SEO | | AirgunDepot0 -
Remove a directory using htaccess
Hi, Can someone tell me if there's a way using htaccess to say that everything in a particular directory, let's call it "A", is gone (http 410 code)? i.e. all the links should be de-indexed? Right now, I'm using the robots file to deny access. I'm not sure if it's the right thing to do since Google webmaster tools is showing me the link as indexed still and a 403 error code. Thanks.
Technical SEO | | webtarget0 -
Optimising multiple pages for the same search term
We were having a discussion on title tags and optimising multiple pages for the same term. We rank well for the phrase 'chanel glasses' which points to our Chanel brand page. The Chanel brand page is optimised for this term, and has the phrase 'Chanel glasses' at the front of its title tag. Previously, the title tag on our home page had the words 'Chanel glasses' at the start in an attempt to rank twice for the term (as one of our competitors has managed). This never worked (though at the time, our DA/PA was lower than it is now). For this reason I switched the title tag on the homepage to try and rank for 'designer glasses'. My belief is, given we already rank highly for the term on a more relevant landing page, trying to rank for it again on the home page is not the best use of a title tag on our highest PA page. We may as well use it for something more generic like 'designer glasses' (though this term does not convert nearly as well, nor does it currently rank as well for us as we've not been attempting to get 'designer glasses' as anchor text. Plus it's more competitive. Another generic term maybe be preferable). My colleague's view is we should attempt to do what our competitor has done and try and rank twice on page one for this term. I like the idea of dominating the top results, but I feel that since attempting to get double-listed hasn't worked for us so far, we should use the homepage for optimising for a different term ( ideally something that we don't already rank for elsewhere on the site). I see his point of view - if we were ranking nowhere for the search term then, yes we should concentrate on getting one page to rank, not two. But since we already rank well for the term, perhaps his strategy is preferable? Just for clarity, the title tags are not duplicate, but the idea was to share many of the same keywords between the two title tags. What are your thoughts SEOmoz?
Technical SEO | | seanmccauley0 -
Blocking other engines in robots.txt
If your primary target of business is not in China is their any benefit to blocking Chinese search robots in robots.txt?
Technical SEO | | Romancing0