Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do we need both an .XML Sitemap and a .aspx sitemap?
Hi Mozers, We recently switched servers and it came to my attention that we have two sitemaps a XML version of the sitemap and a .aspx version of the sitemap. This came to light as the .aspx version of the sitemap is causing the site to come to a screeching halt as it has some complex code and lists over 80,000 products. My question is do we need both versions of the sitemap? My understanding is that the XML version is for Search Engine bots and the .aspx version is for customers. I can't imagine that anyone is using our .aspx version as it is basically a page with 80,000 links and it's buried away on the site, so we were hoping to kill off the .aspx version of the sitemap and keep the .xml version for Search Engine Bots. I wanted to check here first to make sure we did not any negative search engine implications. Any help would be most appreciated. Thanks so much! Patrick
Web Design | | gatorpool0 -
Is The HREF Link "Title" Tag Needed on Mobile Websites?
Hello To Those Who Are Wiser Than I, I am wondering if the href link "title" tag is needed, or serves any purpose, on mobile websites? Also, does it effect SEO in any way? I ask because generally the href link title tag provides more information to the user when they scroll their mouse over the link - but this action does not happen on mobile! Users have no mouse and thus no extra information would be displayed. I'm really wondering if it still matters for SEO purposes on mobile though. -The UnEnlightened
Web Design | | Stew2220 -
Pageless/Single Page Design and Migration Questions
Hello, We are starting a content audit and migration to a new CMS. We would like to take content and present more on a pageless/Single Page type design instead of having visitors drill down so many levels to find the content. What should we be aware of from an SEO perspective. Here is Example of current pages and structure: http://www.saintpetershcs.com/GraduateMedicalEducation/PediatricResidency/
Web Design | | sphcs
Subpages include: Overview, Curriculum, Faculty, Residents, Benefits, How to APply Here is example of what we would like to do:
http://themeforest.net/item/medicalpress-health-and-medical-wordpress-theme/full_screen_preview/7789703 As you scroll information is populated: Duke Medicine also has something similar. https://www.dukemedicine.org/treatments/cancer What are your thoughts?0 -
Google Translate with "no-follow" just for users' use, no ranking
Hello, I tried to search but here is a little unique situation. I would like to translate my website in 2-3 different languages not for ranking purpose, but only for some minority users within Italy to understand the content in their native language. Using "no-follow" with a Google Translate link would damage SEO? if not I would like to use it. Here are few points: Give users the ability to switch the content to their language with a link Tell Google not to follow the translated pages, because I don't want them to be used for ranking or searches I would start simply with Google Translate to see if people actually are interested, then later translate by human but still don't want google to follow I could also start with human translation instead of Google Translate if really needed, I know it is a no no. What I'm very interested is to make sure that those pages under "no-follow" won't affect my SEO in good or bad right now, because we would like to keep as it is. Thanks a lot
Web Design | | angelowei0 -
Can multiple domains compete with one another if they have the same/similar content?
Could an ecommerce site with a .co.nz nd .com.au domain compete with one another and hard organic rankings if the content of the pages are the same? All the links would be identical (apart from .co.nz or .com.au) the product descriptions, pages titles etc... would all be identical or similar (our page titles are ever so slightly different). Could this be hurting us? Thanks in advance ^ Paul
Web Design | | kevinliao0 -
Link Pages/Directory
Hello, What is best practise for dealing with alot of links. I was thinking of breaking them download to alphabet pages i.e. all A on one page etc... BUT should I then make the links clickable on this list OR that they load to a sub company page which has a clickable link to there website.
Web Design | | JohnW-UK0 -
I need to buy a website template - anyone knows a good place to buy it? where you should buy one template?
I used the www.templatemonster.com or http://themeforest.net/, but is difficult to find a template with a focus on SEO. Also with the necessity to include text in the beginning of the website, is difficulty to found something really good.
Web Design | | Naghirniac0 -
Site-wide footer links or single "website credits" page?
I see that you have already answered this question before back in 2007 (http://www.seomoz.org/qa/view/2163), but wanted to ask your current opinion on the same question: Should I add a site-wide footer link to my client websites pointing to my website, or should I create a "website credits" page on my clients site, add this to the footer and then link from within this page out to my website?
Web Design | | eseyo0