How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Key word with city or town or local search for a nation wide website
Hi We need some expert advise here, we are dealing with batteries, tyres, online at nation level in india, and we have dealer registered with us at local places... and also we dont show the dealer details on the product page, once the order is recieved will be pass on the deal locally... we want rank well in the local key words search as many of times user search for "buy car battery in bangalore" here is bangalore is one location.. We want to develop one url like this www.abc.com/car-battery/bangalore/productname.html and also i will add breadcrumb like this car-battery>Bangalore > Productname which will be canonical to www.abc.com/productname.html (where product details are there). Pls suggest is this way i should go.. or??
Intermediate & Advanced SEO | | Rahim1190 -
An improved search box within the search results - Results?
Hello~ Does anyone have any positive traffic results to share since implementing this? Thanks! MS
Intermediate & Advanced SEO | | MargaritaS0 -
Searching for SEO/SEM with eCommerce Experience
I've debated on whether I should ask this on the Q&A forum. Many times I have started to write it, then changed my mind. So today, after sending another round of emails out to potential SEM and SEO consultants and getting zero response, I said what the heck and decided to post it. I will point the finger at myself and say I must be doing something wrong in my approach and the way I am seeking out consultants or maybe what I am not saying. However, I cannot seem to get SEO companies to return my phone calls or emails. Is my company too small? Are most of the companies recommended here too busy with other work to worry about following up? Is my company and brand not sexy enough to deal with? Am I coming across wrong in my emails and phone calls? With that being said, I decided to write a post here and clearly state some basic and general requirements in hope someone interested will respond. I would be more than happy to answer and specific questions about my company, budget, etc. in a PM. I am looking for an SEO firm or consultants who: Will deal with small businesses with revenue under $2M. I have budgeted for this and am ready to start a project immediately. Have experience and a successful track record working with eCommerce sites who sell a large number of products. Interested in a long term relationship that will produce long lasting, durable results. Try to understand my business and what it will take to make grow my ONLINE profitability. I look forward to speaking to anyone interested. Jake
Intermediate & Advanced SEO | | jake3720 -
Hiding Text in an SEO friendly way - is it possible?
Hello, I have a client who has very little (practically no) text content on his ecommerce website, on the home page and category / sub cat pages. We have drafted some text for him - but the designer has fought back against this as he feels it will break the design. Our proposed solution is to have some text visible - and the rest will be text that is hidden but can be revealed by clicking Read More. We are planning to follow these recommendations : http://www.shimonsandler.com/collapsible-div-seo-friendly/ We are not hiding text for the sake of it - but more to improve the UX. We of course want the text to be accessible - i.e. readable by screen readers. Does anyone have any experience or opinions in respect to taking this course of action, and is there anything we should make sure we either do or not do to stay on the side of the BIG G? Kind Regs, Rich
Intermediate & Advanced SEO | | RichBestSEO1 -
Keyword search filter in Google Adwords: broad? exact? phrase?
Hello all I am working in my website and analysing the potential best keywords for the SEO (post/page name and url path name). 1. I am using Google Adwords. Any other tool you would recommend? 2. Which selection should I make in the Google Adwords Keyword Tool in order to know the monthly global searches of the keywords I should target? Exact? Phrase? Broad? For instance, KEYWORD SEARCH:"Information about Madrid" BROAD MATCH: 300,000 EXACT MATCH: 1,500 Te potential of the keyword is 300,000? 300,000 searches are undertaken on a month that contains that sentence and its variations? Or the relevant keyword potential is the exacta match traffic? Thank you very much! Antonio
Intermediate & Advanced SEO | | aalcocer20030 -
De-indexed Link Directory
Howdy Guys, I'm currently working through our 4th reconsideration request and just have a couple of questions. Using Link Detox (www.linkresearchtools.com) new tool they have flagged up a 64 links that are Toxic and should be removed. After analysing them further alot / most of them are link directories that have now been de-indexed by Google. Do you think we should still ask for them to be removed or is this a pointless exercise as the links has already been removed because its been de-indexed. Would like your views on this guys.
Intermediate & Advanced SEO | | ScottBaxterWW0 -
Subdomain or directory path?
Hi Mozzers, Client: Important carpet cleaner player in the carpet cleaning industry Main Goal: Creating good content to Get more organic traffic to our main site Structure of the extra content: It will act like a blog but will be differentiated from the regular site by not selling anything but just creating good content. The look and design will be different from the client's site. SEO Question: Which option is more beneficial, creating a subdomain or adding a regular page within the website following a directory path URL? If possible, please state what are the advantages and disadvantages of these 2 options in terms of SEO. Thank you and have a great weekend everyone,
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
Canonical Tags & Search Bots
Does anyone know for sure if search engine bots still crawl links on a page whose canonical tags are set to a different page? So in short, would it be similar to a no-index follow? Thanks! -Margarita
Intermediate & Advanced SEO | | MargaritaS0