Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way to host my website.com/BLOG URL PATH from a different host than my main website.com host?
Is there a way to host my website.com/BLOG URL PATH from a different host than my main website.com host? Is it accomplish-able with DNS settings or are there other considerations that might lead to complications doing this? Specifically, we are investigating install WordPress on a dedicated host, JUST to power the blog for our main website, but our main website is on an internal proprietary hosting and CMS. So basically we're trying to host: website.com --> OFF OF CURRENT INTERNAL HOSTING website.com/blog/ --> OFF OF THIRD PARTY HOSTING (USING WORDPRESS) I know this is a technical question beyond the scope of SEO, but I'm figuring there are members of the community that may have tried this already so I'm floating it here. Many thanks! Cheers.
Web Design | | AlexVelazquez0 -
Incorporating Spanish Page/Site
We bought an exact match domain (in Spanish) to incorporate with regular website for a particular keyword. This is our first attempt at this, and while we do have Spanish speaking staff that will translate/create a nice, quality page, we're not going to redo everything in Spanish page. Any advice on how to implement this? Do I need to create a whole other website in Spanish? Will that be duplicate content if I do? Can I just set it up to show the first page in Spanish, but if they click on anything else it redirects to our site? I'm pretty clueless on this, so if anything I've suggested is off-the-wall or a violation, I'm really just spit-balling, trying to figure out how to implement this. Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
Are pages not included in navigation given less "weight"
Hi, we recently updated our website and our main navigation was dramatically slimmed down to just three pages and no drop down under those. Yet we have many more important pages, which are linked to once on one of those main three pages. However, will this hurt those other pages because they are not included in navigation (some of which were starting to get good traction in rankings)?
Web Design | | LuaMarketing2
Thanks!0 -
HTTPS redirecting to subdomain; do I need to 301 to HTTP first?
Hello there, quick question on HTTPS downgrade to HTTP. Originally, full site was HTTPS. Redesign eliminated need for secure layer. Site currently has https 301'd to sub.domain.com. Ideally, site would live at domain.com, but with the old site having been indexed https, the security warning triggers when you visit, hence the 301 to a subdomain. Thinking possible solutions here are to a) add back in secure layer or b) 301 all old https pages to new http pages. Have I overlooked something? With (a), I will see no SEO impact, correct? With (b), can I expect impact similar to a relaunch? Have seen a lot of chatter that there is some impact in switching http to https, but not much the other way. Any help much appreciated!
Web Design | | alimo0 -
Best way to handle Spanish/English WEBSITE
Hey guys, How are you doing? I have a website (www.aceromart.com ) in which the primary language is Spanish. The company is oriented toward the Mexican Audience. However, recently we are dealing with many U.S companies. Also, we want to be included in the Yahoo Directory and several other directories which demand a U.S version of the website. So i want to have a U.S version of the website. My ecommerce is based in the NETWORK Solutions platform and has around 1,000 products and pages. What is the best way to include an English version of the website. Ive seen some pages using the Google traslate tool, which only traslates the text. Hope to hear some of your ideas, Regards,
Web Design | | JesusD0 -
Suggestions for content slider/image slider copy/paste application.
Hey Moz Community, I am looking for a content slider that can be easily changed by non-technicals for posting different styles of content/calls to action and this seems to be best: http://www.slidedeck.com/ I have installed a nivo slider on a Seattle Painting site, and flash slider on a commercial painting site. But I want my blog clients to be able to format..then copy/paste code..linke embedding a video. opinions? Thanks John
Web Design | | johnshearer0 -
What is the current thought how/if cufon affects keywords?
What is the final opinion about how (and if) cufon affects keywords in H1 tags?
Web Design | | Court_LOQUA0 -
Old SEO keyword "articles", are they hurting rankings?
Hello, About two years ago, the company I work for hired an SEO firm to improve organic rankings on our site. The SEO company's primary method for doing this was producing "articles" that are not really articles but keyword stuffed pages with lots of hidden, internal links to other legitimate pages on our site. Examples: http://www.creamright.com/Isi-Chargers-articles.html http://www.creamright.com/How-To-Make-Whipped-Cream-article.html http://www.creamright.com/Cream-Whipper-articles.html Obviously, this strategy wasn't greatly successful and we cancelled our work with the firm. However, we still have all of the "articles" on the site (about 50-60 pages total) and each page is navigable from the html and XML sitemaps. Additionally, the SEO firm we used built a lot of useless links to these pages from BS directory sites which are all still active. The question I have is whether we should remove these "article" pages or should leave them alone? Although I'm sure they aren't helping any of our SEO efforts, could deleting the pages after two years negatively impact our search rankings? Thanks in advance for any help on this, Doug M.
Web Design | | Loganshark1