Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tips on finding the right Senior Designer / Design Director
Hello Everyone, I manage a fairly large educational website that we are looking to completely redesign to improve the overall site user experience and usability. In the past I, a non-designer, business person, would just roughly draw how I thought the site should look and our developer (who is good, but not a web designer) would just try his best to make everything look profession. Over the years, it has become painfully obvious that we need to invest in more design expertise and move towards a modern, smartly designed website. So my question: Where are the best places to find good freelance designers? I have of course conducted web searches, browsed elance, and asked my network for referrals. However, I am finding that most of the really good ones, ones who have to ability to take charge and lead us through this entire process, and who have at least a basic understanding of SEO principles, work for larger integrated development shops who also expect their people to develop the new site as well. We already have a developer and are primarily looking for the design expertise. Does anyone in the Moz community have any suggestions or even referrals? Thanks! Eric
Web Design | | Eric_R0 -
Rich snippets tool/ schemer.org update?
We have a domain in the netherlands that shows stars (rich snippets) in adwords and in the organic search results. As we do the the in rich snippets test tool we see some errors as shown in the attachments, the errors are shown in review aggregate - but stars are stil shown in paid and organic search. Do we need to follow-up these errors? Do they influence ranking - or the stars in the near future? Z0fCKRW
Web Design | | remkoallertz1 -
Redirects Not Working / Issue with Duplicate Page Titles
Hi all We are being penalised on Webmaster Tools and Crawl Diagnostics for duplicate page titles and I'm not sure how to fix it.We recently switched from HTTP to HTTPS, but when we first switched over, we accidentally set a permanent redirect from HTTPS to HTTP for a week or so(!).We now have a permanent redirect going the other way, HTTP to HTTPS, and we also have canonical tags in place to redirect to HTTPS.Unfortunately, it seems that because of this short time with the permanent redirect the wrong way round, Google is confused as sees our http and https sites as duplicate content.Is there any way to get Google to recognise this new (correct) permanent redirect and completely forget the old (incorrect) one?Any ideas welcome!
Web Design | | HireSpace0 -
New To SEO Management, I just want to double check that my idea will work.
I am new to SEO management. I had a 3 month SEO copy writing internship and a 5 month SEO temp job. In both I mostly wrote copy, but I've been teaching myself SEO on the side, I became Google certified. I ended up getting a telemarketing job and somehow the conversation of SEO came up and I winded up managing their SEO for 12 dollars an hour. They say that every lead generated from the website that turns into a sale will be worth 10 dollars and if and when the sales exceed my paycheck I will starting making commission so long as it stays above my hourly. SEO is very fun and this is like my dream job. They are leaving the planning 100% up to me and I want to make sure that what I am doing will work. My plan is as follows: Part 1: Page Authority via backlinks and social media We are health care brokers and my boss, the owner has a lot of contact. He is talking with large unions like, "The Teamsters," and large company retirment groups like, "Blue flame," which is apparently in some way connected to DTE or GE. Long story short, I am trying to get him to convince them to give us a back link to our main page. He also has a ton of clients that own companies. This is good because they may be persuaded to give us backlinks too. In addition, the tech guy thinks he can implement something where we can get a google +1, facebooks likes/shares, twitter likes and shares and pintrest pin it's that would be a part of an email that we send to people within the list of 12,000 clients. From what I can see, from the client base and the people we are working with we should be able to raise the page authority substantially despite the fact that the site is only a few months old and is not yet out of the sand box. I have been slowly picking off each error with SEO MOZ's website crawling. Part 2: Making a Insurance Jargon Dictionary Guide For The Tri-purpose of gathering traffic, proving our professionalism and helping people understand semi-complex insurance jargon. I could build these 2-3 keywords would be addressed per page and they would be defined in a way to help people looking for terms understand them, while simultaneously netting a strong keyword density and a strong page. I think as far as I can tell there are no issues. Part 3: The dictionary pages will pull in new traffic and the home page will receive links and distribute link juice to the sub-pages. This subpages will guide traffic back to the main page with no-follow links to direct people from the unique termed landing pages to the home page for insurance processing. As far as I can tell my logic is solid and on paper this should work. Am I missing anything (like key details, flaws in my plan)?
Web Design | | Tediscool0 -
Does anyone know if the YOAST SEO plugin works 100% on a responsive wordpress template?
Hi, Does anyone know if the YOAST SEO plugin works 100% on responsive wordpress template? In other words, when you configure the custom title and metadescription is it also configured for the tablet and cellphone views as well? Thanks Carla
Web Design | | Carla_Dawson0 -
Blog.yoursite.com or yoursite.com/blog/?
Rand made the following post in 2009. In July of 2013 I question whether this is still best practice. Obviously, much has changed in our industry in four-and-a-half years! "...if you're seeking to maximize your ranking ability for a given piece of content, it's my personal belief that you should, most of the time, keep it on 1 subdomain under 1 root domain (but feel free to use subfolders as it makes sense). Starting a blog? I almost always recommend yoursite.com/blog over blog.yoursite.com. Want to launch a new section of content? Use yoursite.com/newstuff rather than newstuff.yoursite.com." Source: http://moz.com/blog/understanding-root-domains-subdomains-vs-subfolders-microsites
Web Design | | JorgeUmana0 -
Does anyone think the <figcaption>attribute from HTML5 will have any influence for image search?</figcaption>
There is a <figure>element that is supposed to provide better descriptions of image on the web in HTML5 - do you think that will replace the importance of the "Alt" tag? Link to figcaption description </figure>
Web Design | | RankSurge2 -
Finally have a budget for a great seo ecommerce site but need help choosing wordpress, joomla, modx, magneto or? Thank you in advance for your generosity of time
We finally have a budget and want to dump our intuit/homestead site www.originalartbroker.com Our budget is 5k-10k but could do more if needed. I am slowly catching my competition with this homestead site that I built. But I do realize it is time to step back, figure out what is best, and hire a pro to get the job done. I am green in the seo and web development arena so please go easy on me and please help to point me in the right direction. Just went out on a limb a couple years ago playing with homestead.com site software and built what we have today. Didn't know anything about website development...it sort of just happened. I feel and know that homestead.com solution is hindering what we could be doing due to the bloated nature of the site and inability to perform such task as 301 redirects etc.... I have been able to slowly attain first page seo rankings on keywords based of the artists we carry using this po-dunk homestead platform to build my site after a ton of work education thanks to seomoz and a lot of you. But, have never asked for help and could really use some generosity of time in explaining a solution that would work best for our business. Do we just go with a wordpress site that is similar to our current setup and use their plugins? Do we use a cms software solution like magneto or joomla? We will only have 200-300 pieces at any given time. We are constantly selling and buying new pieces providing us content. We are need of a site that can perform well in terms of seo. I have heard of a lot of people talking about joomla, wordpress, and magneto. Would like to be able to have a product catalogue that ultimately sends whatever inventory we are uploading to our social sites and blogs so I don’t have to pump the product out to all of these sites. We offer free custom framing with our pieces and it would be nice to have a program that could wrap the photos of the pieces with the different frames for our customers. When I add a new piece I would like this software to have a predesigned product page that it plugs the information into. I would like it to create the url extension based of the artists name, medium used, and piece name to create unique and individual urls. I would like it to also create its own H tags throughout that product page according to the artist name description, and medium used. I would like to be able to sink this up to google merchant and other sites to carry our product. Bottom line is we sell art. We sell pieces by specific artists. We are constantly buying and selling. I need something powerful that keeps up with our content
Web Design | | forecastedinvestments1