How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it a good idea to optimize for keywords that have no search volume if they're ranked?
Hello Moz Community, I have some questions I hope some of you can help with. We’re doing SEO work for a client that provides outsourced IT and managed IT services in Phoenix, AZ and cities in the Phoenix metro area (i.e. Glendale, Tempe, Scottsdale, etc.) They’re currently ranked for or targeting the following keywords: • consulting phoenix az (1)
Intermediate & Advanced SEO | | marnipatterson
• outsourced it phoenix (2)
• phoenix it support (3)
• it services Scottsdale (5)
• it consulting firm phoenix (targeting)
• it solutions phoenix (targeting) We have recommended the following keywords based on monthly search totals, competitive level and difficulty ratings in Moz. • IT consulting phoenix
• it consultant company
• outsourced it
• it support services
• it consulting services
• outsourcing it
• outsourced tech support Questions
1. While I know it’s a good idea to optimize for keywords that you're currently ranked for, there’s no search volume for any of these. So, I recommended non-geo versions since Google provides search results based on the user’s location. Will this preserve the company's current rankings?
2. If not optimizing for their current keywords will hurt their rankings, will using the current keywords as secondary keywords suffice? If so, do we need to include them in the content for keyword density?
3. Since search engine algorithms now focus so heavily on user intent, I assume we’re covered for all variations of a keyword (i.e. outsource it, outsourced it, outsourcing it, etc.) Is this correct?
4. They want to rank for “cloud services” and “cloud solutions.” Both are very competitive with high difficulty rankings. So, I recommended “cloud migration” and “cloud strategy” as alternatives since these are the main services they provide. Will including “cloud services” and “cloud solutions” as secondary keywords help them increase their rankings for both? If you’ve dealt with a similar situation, I'd appreciate your insight and advice. Thanks!0 -
How-to: Geography Based Search Results for Job Board
Hi, I’m working with a client that is building a job board website. We want to have the ability to show search results in Google that show job results for a specific area. For example, “Office Administrator Jobs in Chicago” or “Marketing Assistant Positions in Cleveland”. How can we go about setting this up? Do I need to create a separate landing page for each of these exact searches? Or can this be done with a taxonomy or the job search function? This is a WordPress built website and we're using the Yoast SEO plugin. Any help would be appreciated!
Intermediate & Advanced SEO | | cslingerland0 -
Zero search count and still they are earning good. How??????
Here is one website - listdose.com Alexa rank - 28,665 They have around 1000 pages, But 80% keywords used by them have 0 search count. They target only one keyword per page. So how are they earning good money and how are they ranking well in alexa without having any good search count kewyords ? Is this good idea to target 0 search count keywords to create a blog.
Intermediate & Advanced SEO | | ross254sidney0 -
Sitemap Folders on Search Results
Hello! We are managing SEO campaign of a video website. We have an issue about sitemap folders. I have sitemaps like ** /xml/sitemap-name.xml .** But Google is indexing my /xml/ folder and also sitemaps and they appear in search results. If i will add Disallow: /xml/ to my robots.txt and remove /xml/ folder from webmaster tools, Google could see my sitemaps? or it ignores them? Will my site effect negatively after remove /xml/ folder completely from search results? What should i do?
Intermediate & Advanced SEO | | roipublic0 -
Handling Similar page content on directory site
Hi All, SEOMOZ is telling me I have a lot of duplicate content on my site. The pages are not duplicate, but very similar, because the site is a directory website with a page for cities in multiple states in the US. I do not want these pages being indexed and was wanting to know the best way to go about this. I was thinking I could do a rel ="nofollow" on all the links to those pages, but not sure if that is the correct way to do this. Since the folders are deep within the site and not under one main folder, it would mean I would have to do a disallow for many folders if I did this through Robots.txt. The other thing I am thinking of is doing a meta noindex, follow, but I would have to get my programmer to add a meta tag just for this section of the site. Any thoughts on the best way to achieve this so I can eliminate these dup pages from my SEO report and from the search engine index? Thanks!
Intermediate & Advanced SEO | | cchhita0 -
Could Temporarily Linking New Directory Pages to my Homepage Help SEO?
Within my website we maintain a nationwide directory of auto repair shops. When we add or significantly update / modify a particular listing, would it help improve the individual search engine rankings, Google PageRank, and / or Page Authority of the new auto shop page if we linked these pages to an area on the home page for "Our Newest Featured Shops" or "Latest Member Additions" or something of the nature? Each new shop profile would then be linked directly from the homepage for a period of time. I assume that it might be crawled and added to the indexes quicker, but would there be other benefits? If so, would those benefits only be temporary if eventually the new shop no longer linked to the homepage? Would keeping all featured shops in rotational display on the homepage make any difference? Any input is appreciated. Thanks. Kelly Vaught
Intermediate & Advanced SEO | | kelly_vaught0 -
Blog not showing up when searching for exact post/META titles
I am working on a blog http://www.possessionista.com which is a very popular fashion blog. It is very well established with a 100% natural link profile and zero spammy stuff. The blog ranks #1 for random fashion terms like "kourtney kardashian cat eye sunglasses" and "emily maynard boots". The problem I am experiencing is that none of the actual titles of her posts or any of the content in the post results in her blog showing up if searched. EX: http://www.possessionista.com/2011/10/pippa-middletons-zip-jacket.html When you search "Pippa Middleton's Zip Jacket" on google her blog is nowhere to be found. Try searching allintitle:"Pippa Middleton's Zip Jacket" and she's nowhere to be found either. Even search "The other day, I met with my friend Kiran for our monthly mutual admiration society" on google and she's nowhere to be found even thoguh this is a unique snippet from her post. This post is already indexed and cached with the above mentioned details. i've also tested dozens of older posts as well. Same issue. You can actually do this to see a more clear picture: Do a google search for: allintitle:Bachelorette Fashion: Episode 2 - Ashley Hebert Brown site:possessionista.com That will bring up her blog which means google recognizes that the phrase is in her META title. Now do a google search for: allintitle:Bachelorette Fashion: Episode 2 - Ashley Hebert Brown without the site: included. She does not pop up but other people do. I did find that she had a duplicate title tag for a few weeks, but I've fixed that. Her posts used to pop up #1 when you search the title, but now obviously not. I am kind of at a loss and have tried a bunch of options with no success. Oh, one other thing is that some people do scrape her content, but only a few like maybe 10 and they've always been doing it even when she used to rank for her own post titles. Have you guys experienced this issue? Do you have any ideas of how to fix it?
Intermediate & Advanced SEO | | modparent0 -
SEOMoz Link Directory - As Silly as I think it is?
Don't get me wrong, I LOVE LOVE LOVE SEOmoz, but their "Link Directory" (www.seomoz.org/directories) is a bit deceiving. I was looking for a list of DIRECTORIES that Moz recommends, not a bunch of places where you can pay for advertising. On top of that, it also lists dmoz as one of the spots to get links from, but have you ever actually ever been able to get a link from dmoz? I know I haven't, and we've been trying to get a link for years. Anyone else disappointed in this list? Does anyone have a good list of directories? -Andy P.S. I love you SEOmoz! Don't hate me for this critique!
Intermediate & Advanced SEO | | alhallinan1