Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any alternative techniques to display tabbed content without using Javascript / JSON and be SEO Friendly?
John Mueller's input in the EGWMH hangout suggests that Google MAY ignore expandable content served by Javascript. Are there any alternative techniques to display tabbed content without using Javascript / JSON and be SEO Friendly? I do however view these as good for website interactivity and UX - and see many examples of websites performing well and ranking highly whilst using these techniques - are there any Google friendly ways to serve content on a page so that search bots can recognise and choose to crawl / consume the content as legitimate fodder?
Web Design | | Fergclaw0 -
Web Designer Needed
So I feel like I've almost convinced my boss to hire out a re-design for our website (which is SORELY needed.) I'm at the step where I need to start finding quotes and site builders. I want a responsive design site built modern and professional... Love the moz site but not responsive enough. My current favorite example is at www.hasoffers.com Does anybody have any suggestions of where to go? And as a bonus, if anybody is brave enough to ballpark what a site like this would cost that would be awesome. Doesn't have to be exact or even that close... Thanks for any advice given.
Web Design | | jesse-landry0 -
New more "helpful" internal linking causing SERP & traffic drop?
Still dealing with the weird traffic drop on my website. I have removed a bunch of old links from a defunct blog, 301 thin pages, added text to remaining pages. I'm still stumped. So awhile ago I freshened up my website and thought I was "helping people" by making sure they could CONTACT the studio more easily... I added more links to the "contact page" I thought this would help conversions...This changed the number of links to my entire site....Would this be the problem with my ranking/traffic drop? http://bayareaboudoir.com/babinternal1.pdf
Web Design | | Squee1 -
Yes or No for Ampersand "&" in SEO URLs
Hi Mozzers I would like to know how crawlers see the ampersand (& or &) in your URLs and if Google frown upon this or not? As far as I know they purely recognise this as "and" is this correct and is there any best practice for implementing this, as I know a lot of people complained before about & in links and that it is better to use it as &, but this is not on links, this is on URLs. Reason for this is that we looking to move onto an ASP.Net MVC framework (any suggestions for a different framework are welcome, we still just planning out future development) and in order to make use of the filter options we have on our site we need a parameter to indicate the difference on a routing level (routing sends to controller, controller sends to model, model sends to controller and controller sends to view < this is pattern of a request that comes in on the framework we will be using). I already have -'s and /'s in the URLs (which is for my SEO structuring) so these syntax can't be used for identifying filters the user clicks or uses to define their search as it will create a complete mess in the system. Now we looking at & to say; OK, when a user lands on /accommodation and they selects De Kelders (which is a destination in our area) the page will be /accommodation/de-kelders on this page they can define their search further to say they are looking for 5 star accommodation and it should be close to the beach, this is where the routing needs some guidance and we looking to have it as follow: /accommodation/de-kelders/5-star&close-to-the-beach. Now, does the "&" get identified by search engines on a URL level as "and" and does this cause any issues with crawling or indexation or would it be best to look at another solution? Thanks, Chris Captivate
Web Design | | DROIDSTERS0 -
Looking for a Static Blog Roll (Directory) plug in / tool
I am looking to add a static page to my site to allow people to add their site to a 'master directory". I'd like to make this as SEO friendly as possible. Here's it what I would really like to be able to do. 1. Someone visits the page and can either browse the blog directory on that page, or 2. Fill out a short from that has their blog name, url, short description of their blog, their state, country, and maybe the option to upload an image as well. This way people can search by the state / country to find people near by. 3. I'd like the links to be no-follow. 4. I'm not sure if there is an easy way to do this (I think I've seen some plug in's that may be able to do it.) 5. I'm also not sure if there is anything I'm missing here that could make this better. What I don't want is just a long list of links like everyone has on their side bar. I'd like to make it better, more user friendly, more seo friendly, and more helpful than that. Any creative input on how I might do this is appreciated. Thanks. 🙂
Web Design | | NoahsDad0 -
What is the current thought how/if cufon affects keywords?
What is the final opinion about how (and if) cufon affects keywords in H1 tags?
Web Design | | Court_LOQUA0 -
Using "#" anchors to display different content
If I have a page that has an area on the page that acts like a widget and has three different tabs. These tabs provide 3 different types of information relevant to the page subject matter. By default when someone goes to the page one of the tabs is showing but you have to click on the others to see the info on them. Is it OK to use domain.com/topic#TAB1, domain.com/topic#TAB2, domain.com/topic#TAB3 to create shortcut links so that people can land on the page and have that predetermined tab showing. I'm wondering what search engines might think. Essentially all the content of all three tabs is there for people to see but they'd have to click to see the other tabs. I don't consider the content to be hidden. But I'd like to hear people's thoughts.
Web Design | | Business.com0 -
How WP Themes work with Navigation Structure for SEO and JQuery Headers?
I am trying to find the best WP Theme for our company. I noticed most of them do not offer a left hand side navigation on the home pages, and usually are on the right side on the inside pages. I always thought that Home Page links were very important for SEO. Currenly we have a left drop down navigation with all of our product catagories, keyword optimized. The structure follows for all the pages. Is this not as important to Search Engines anymore? Is it better to have a products link, to all the products and then the inside pages, have just a navigation bar, for that particular catagory? This seems to be very common on all the templates i am seeing. I also noticed, and really like the JQueary Tabs. I would use this for displaying, PDFs and Specifications Charts. Also, some home page images are using a jquery slider with some text, linking to a page. Is Jquery the new javascript and do search engines see what is in the code? I also noticed they all have footers that have links and some other information. Is this a SEO must have?
Web Design | | hfranz0