How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google related searches
Hello, Are the related searches, the words that I should use when writing my content. For ex : when I type online spreadsheet in google, in the related searches it list online spreadsheet open source and spreasheet download. Does it means that when writing content I should included those terms in order to be relevant on the keyword online spreadsheet ? because they are considered closely related by google ?
Intermediate & Advanced SEO | | seoanalytics0 -
Why our page not ranking even searching for exact h1 tag?
Even I search for exact h1 tag heading from our homepage, it's (homepage) not been showing up on TOP of the results. Other websites with partial match of search query are ranking above us; why this is happening? And other website with same text as normal paragraph is ranking on top. But not out h1 tag from homepage? How come normal text of unrelated website is ranking above h1 heading from homepage of own website?
Intermediate & Advanced SEO | | vtmoz0 -
When you can't see the cache in search, is it about to be deindexed?
Here is my issue and I've asked a related question on this one. Here is the back story. Site owner had a web designer build a duplicate copy of their site on their own domain in a sub folder without noindexing. The original site tanked, the webdesigner site started outranking for the branded keywords. Then the site owner moved to a new designer who rebuilt the site. That web designer decided to build a dev site using the dotted quad version of the site. It was isolated but then he accidentally requested one image file from the dotted quad to the official site. So Google again indexed a mirror duplicate site (the second time in 7 months). Between that and the site having a number of low word count pages it has suffered and looked like it got hit again with Panda. So the developer 301 the version to the correct version. I was rechecking it this morning and the dotted quad version is still indexed, but it no longer lets me look at the cache version. Out of experience, is this just Google getting ready to drop it from the index?
Intermediate & Advanced SEO | | BCutrer0 -
Customer Experience vs Search Result Optimisation
Yes, I know customer experience is king, however, I have a dilema, my site has been live since June 2013 & we get good feedback on site design & easy to follow navigation, however, our rankings arent as good as they could be? For example, the following 2 pages share v similar URLs, but the pages do 2 different jobs & when you get to the site that is easy to see, but my largest Keyword "Over 50 Life Insurance" becomes difficult to target as google sees both pages and splits the results, so I think i must be losing ranking positions? http://www.over50choices.co.uk/Funeral-Planning/Over-50-Life-Insurance.aspx http://www.over50choices.co.uk/Funeral-Planning/Over-50-Life-Insurance/Compare-Over-50s-Life-Insurance.aspx The first page explains the product(s) and the 2nd is the Quote & Compare page, which generates the income. I am currently playing with meta tags, but as yet havent found the right combination! Originally the 2nd page meta tags were focussing on "compare over 50s life insurance" but google still sees "over 50 life insurance" in this phrase, so the results get split. I also had internal anchor text supporting this. What do you think is the best strategy for optimising both pages? Thanks Ash
Intermediate & Advanced SEO | | AshShep10 -
Local Showroom Tel Showing Up in SERP for Branded Term Search?
We encountered an issue: when user search for a branded term in UK on Google UK, one of the showroom got listed in SERP under site links as a location with Location icon and takes over a few lines - their address and phone number are displayed. This caused confusion for user as it's only one of our showrooms and we don't want to direct calls to the showroom this way. I'm suspecting somehow the localization factor came into play but not sure how to remove the showroom phone number from being displayed. Anyone has any insight/experience with this? Thanks
Intermediate & Advanced SEO | | b.digi0 -
Do you think that google places have influence on organic search?
Hi everyone, Do you think that google places have influence on organic search? i mean if you are on google places with your companies, your ranking must be better for that query? isn't it ? if you agree with me i'll bring you this stange case: my clients has a company profile on google places but it is only fourth on the organic results for that query and his competitor is ranked first with: no google places profile no authority and no company in the city ( in fact it is not in the city but 30 miles far ) the query is "poltrona frau brescia" poltrona frau= name of the company brescia= city in wich the company is located thanks poltrona-frau-brescia
Intermediate & Advanced SEO | | guidoboem0 -
Directory and Classified Submissions
Are directory submissions and Classified Submissions still a good way to create backlinks? Or they are obsolete methods and should be discontinued?
Intermediate & Advanced SEO | | KS__0 -
SEOMoz Link Directory - As Silly as I think it is?
Don't get me wrong, I LOVE LOVE LOVE SEOmoz, but their "Link Directory" (www.seomoz.org/directories) is a bit deceiving. I was looking for a list of DIRECTORIES that Moz recommends, not a bunch of places where you can pay for advertising. On top of that, it also lists dmoz as one of the spots to get links from, but have you ever actually ever been able to get a link from dmoz? I know I haven't, and we've been trying to get a link for years. Anyone else disappointed in this list? Does anyone have a good list of directories? -Andy P.S. I love you SEOmoz! Don't hate me for this critique!
Intermediate & Advanced SEO | | alhallinan1