Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way to host my website.com/BLOG URL PATH from a different host than my main website.com host?
Is there a way to host my website.com/BLOG URL PATH from a different host than my main website.com host? Is it accomplish-able with DNS settings or are there other considerations that might lead to complications doing this? Specifically, we are investigating install WordPress on a dedicated host, JUST to power the blog for our main website, but our main website is on an internal proprietary hosting and CMS. So basically we're trying to host: website.com --> OFF OF CURRENT INTERNAL HOSTING website.com/blog/ --> OFF OF THIRD PARTY HOSTING (USING WORDPRESS) I know this is a technical question beyond the scope of SEO, but I'm figuring there are members of the community that may have tried this already so I'm floating it here. Many thanks! Cheers.
Web Design | | AlexVelazquez0 -
Web Designer Needed
So I feel like I've almost convinced my boss to hire out a re-design for our website (which is SORELY needed.) I'm at the step where I need to start finding quotes and site builders. I want a responsive design site built modern and professional... Love the moz site but not responsive enough. My current favorite example is at www.hasoffers.com Does anybody have any suggestions of where to go? And as a bonus, if anybody is brave enough to ballpark what a site like this would cost that would be awesome. Doesn't have to be exact or even that close... Thanks for any advice given.
Web Design | | jesse-landry0 -
Blog vs. News/Editorial Layout?
We're in the coupon blogging space & saw that one of the larger coupon sites move away from the more traditional blog layout: http://thekrazycouponlady.com they now have more of an editorial type layout. Here is another site which is more similar to our layout: http://hip2save.com. So here are my questions: Which layout type do you feel better serves their visitors & why? How does the affect the SEO of the site? How does it affect the advertising revenues? Which layout do you prefer? Is there strategy in this move for the coupon blog, or is this just a preference on how they now display their content? We're making some updates to our design soon & I wanted to get some feedback on the overall direction we take.
Web Design | | seointern0 -
How keywords per page to keep from being "spammy"?
Hi all, I am currently doing a marketing internship for a B2B company that does all sorts of out-sourced recruiting work. I have some experience with SEO, but not completely confident. My first question is, I know Google sees websites that load up on keywords as "spammy", so what is the appropriate number of keywords per page? Currently, I was thinking about this setup: 1 keyword for the URL 1 keyword per alt tag (1 per page, at most) 2 keywords per each title tag (approximately 4 pages that I am going to follow internally, not following the "about us" page). After that, I was thinking of adding 2-3 more keywords in each meta description and 2-3 in the body copy. That would equate to 6-8 keywords on each page, is this too many and should keywords be repeated (on the same page or across multiple pages)? Since this website is brand new (zero links), would it make sense to nofollow all of the internal links so that they homepage can gain ranking as quickly as possible within Google?
Web Design | | wlw20090 -
Duplicate content and blog/twitter feeds
Hi Mozzers, I have a question... I'm planning to add a blog summary/twitter feed throughout my website (onto every main content page) and then started worrying about duplicate content. What is best practice here? Let me know - thanks, Luke PS. I sat down and re: blog feed... thought that perhaps it would help if I fed different blog posts through to different pages (which I could then edit so I could add<a></a> text different from that in blog). Not sure about twitter.
Web Design | | McTaggart1 -
Looking for a Static Blog Roll (Directory) plug in / tool
I am looking to add a static page to my site to allow people to add their site to a 'master directory". I'd like to make this as SEO friendly as possible. Here's it what I would really like to be able to do. 1. Someone visits the page and can either browse the blog directory on that page, or 2. Fill out a short from that has their blog name, url, short description of their blog, their state, country, and maybe the option to upload an image as well. This way people can search by the state / country to find people near by. 3. I'd like the links to be no-follow. 4. I'm not sure if there is an easy way to do this (I think I've seen some plug in's that may be able to do it.) 5. I'm also not sure if there is anything I'm missing here that could make this better. What I don't want is just a long list of links like everyone has on their side bar. I'd like to make it better, more user friendly, more seo friendly, and more helpful than that. Any creative input on how I might do this is appreciated. Thanks. 🙂
Web Design | | NoahsDad0 -
Using "#" anchors to display different content
If I have a page that has an area on the page that acts like a widget and has three different tabs. These tabs provide 3 different types of information relevant to the page subject matter. By default when someone goes to the page one of the tabs is showing but you have to click on the others to see the info on them. Is it OK to use domain.com/topic#TAB1, domain.com/topic#TAB2, domain.com/topic#TAB3 to create shortcut links so that people can land on the page and have that predetermined tab showing. I'm wondering what search engines might think. Essentially all the content of all three tabs is there for people to see but they'd have to click to see the other tabs. I don't consider the content to be hidden. But I'd like to hear people's thoughts.
Web Design | | Business.com0