Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do we need to interlink homepage to the highest?
Hi, So our homepage is not even at top 5 pages interlinked most. I could see a ranking drop even when other pages overtaken homepage in internal linking. Homepage is the highest priority page we are expected to rank for our primary keyword. So, not linking homepage internally high but giving importance to other pages confuses Google and outrank homepage in rankings? Thanks
Web Design | | vtmoz0 -
I want to create a 301 redirect on a WordPress site, nothing's working...
Hello all, I'm hoping someone out there can give me a hand with this. I'm trying to modify my .htaccess file so that the site will go from maxcarnagemusic.com to www.maxcarnagemusic.com and also, so viewers will be redirected to www.maxcarnagemusic.com/home when they try to access the site. I've tried a few different things, including adding the 301 redirect plugin for Wordpress, but nothing seems to work. Can someone out there show/tell me how to create an htaccess file that will execute as much. I apologize in advance, my Apache experience is very, very limited. Thank you all in advance!
Web Design | | maxcarnage0 -
"Design by" footerlinks of webdevelopment firms
Hi Guys, Since our company does a lot of web development jobs next to our SEO work, we came across the topic of the “Design by” or “Webdesign by” links that our competitors are placing in the footer of clients websites. When I did some backlink analysis of our competitors these links make up the biggest part of their backlink profile. That way, I think they are important for the promotion and SEO of web development companies. I would love to have your opinion about how to use these links for SEO 1. Side wide vs homepage I think there are two ways here, going for a link at the bottom of every page or just one link at the bottom of the homepage. When you look at exposure the most ideal situation would be a side wide link (more eyeballs). From a SEO standpoint this might cause trouble. 2. Follow vs nofollow I think this is a very interesting one since there seems to be a lot of different opinions about if this is advertisement or not. A normal banner in a sidebar would get the nofollow tag to match Google’s guidelines but what about this “designed by” link? I know few designers and “putting a stamp” on their work is pretty common and more likely viewed as something that they deserve then an act of commerce. Besides this discussion adding the nofollow tag could prevent any issues with side wide links but reduces any SEO benefits to zero. Another option is adding the nofollow tag to all the links expect the one on the homepage. 3. Anchor text and descriptive text Since anchor texts play a big role in SEO (and in detecting spam) I’m interested in your opinion about the use of anchor texts with such links. Just the company name or the web address sounds like a safe choice (especially with side wide links) but it’s pretty attempting to link words like “Webdesign” instead of the company name. Do you think it’s safe to use this, or any other keywords in 0,5,10,15,20,25% of the times? And do you think the descriptive text “Webdesign and SEO by” got any value (or causes potential risks) when added before the link? 4. Alternatives or extra link opportunities Here are some other options I found and thought about considering getting links from client websites. Asking, or earning a link in their first blogpost. Yjpiieee our website is online, thanks guys of company X. Links in the disclaimer or legal parts of the website. Link in the main content of the about us. Most likely in the bottom a little section about the credits of the design etc. Links on partner / friend / links pages of the website. I saw all of these links in the backlink profiles of our competitors. Last but not least, my current standpoint: At this moment we ask our clients if we may place a small text and link in the footer of their website next to the copyright claim. We use texts like “Webdesign and SEO by Web Whales”. With only our brand name as anchor text (to avoid penalties). It’s a side wide follow link. Side wide because we want the exposure and follow because we want the SEO benefit. We considered just making the homepage link follow, but this kind of feels like manipulation so we just went for all follow since this is pretty normal on the web. Beside that we celebrate new websites with our clients to increase the change getting mentioned on the first blog or partner page. Occasionally we ask them for links on these pages. What’s your opinion about this topic?
Web Design | | Bob_van_Biezen0 -
Multiple websites for different service areas/business functions?
I'm wondering what the implications are for having multiple domains for different service areas of a company? I realize having multiple domains for one company can be troublesome because of the possibility of duplicate content, keyword cannibalization, and linkbuilding to multiple domains. But when the domains are for very different service offerings/unique business functions that each serve their own purpose (and have different positionings), is there a downside to having more than one domain? Any thoughts would be appreciated!
Web Design | | KevinBloom0 -
New more "helpful" internal linking causing SERP & traffic drop?
Still dealing with the weird traffic drop on my website. I have removed a bunch of old links from a defunct blog, 301 thin pages, added text to remaining pages. I'm still stumped. So awhile ago I freshened up my website and thought I was "helping people" by making sure they could CONTACT the studio more easily... I added more links to the "contact page" I thought this would help conversions...This changed the number of links to my entire site....Would this be the problem with my ranking/traffic drop? http://bayareaboudoir.com/babinternal1.pdf
Web Design | | Squee1 -
I need help with international SEO for two sites?
I'll try to keep this clear... I am working with an company based in Germany, they own company.com/de and company.com/en, and that's how they are currently structuring their domains. They also own companyusa.com that they really want to show up in USA only. They want to keep company.com/en for England/english speaking Europe and company.com/de for their German audience in Germany. They are wanting us to optimize/SEO for companyusa.com, and they want that URL to show up as the top google search in the USA for their "company" keyword. What is showing up now is www.company.com/en 1st in Google because it's been around longer and it has more domain authority. What is the best practice for us optimize companyusa.com so that it is the top dog in the USA while not messing up the other domains? Should we merge? Subfolders all around? Thanks for all the input.
Web Design | | Rocket.Fuel0 -
Duplicate home page /index.asp /index.php etc
We recently moved www.devoted2vntage.co.uk to shopify but seem to have multiple home page variants still in google index. I am concerned that these will be causing duplicate content. I have redirected the offending URLs below to www.devoted2vintage.co.uk/ and have set up a canonical URL but need an expect to tell me if I have taken the current steps and if not, exactly what I need to do. www.devoted2vintage.co.uk/index.php www.devoted2vintage.co.uk/index.htm www.devoted2vintage.co.uk/index.html www.devoted2vintage.co.uk/index.shtml www.devoted2vintage.co.uk/index.aspx www.devoted2vintage.co.uk/index.cfm www.devoted2vintage.co.uk/index.pl www.devoted2vintage.co.uk/index.asp
Web Design | | devoted2vintage0 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0