Block parent folder in robot.txt, but not children
-
Example:
I want to block this URL (which shows up in Webmaster Tools as an error):
http://www.siteurl.com/news/events-calendar/usa
But not this:
-
The idea from Andrew is nice, but my guess would be that you're targeting multiple events so that might run into issues. What you could do is add some more regular expression and make it like this:
Disallow: ^/news/events-calendar/usa$
-
You could use "allow" in your robots.txt file for just this problem.
allow: news/events-calendar/usa/event-name
disallow: /news/events-calendar/usa
See the allow directive section of this page: https://en.wikipedia.org/wiki/Robots_exclusion_standard
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Location for Copy Block
We are having discussions around the appropriate location to place the SEO copy block on an eCommerce category page. Would like to get the communities opinion to share with the creative team.
Web Design | | TukTown0 -
Fixing Render Blocking Javascript and CSS in the Above-the-fold content
We don't have a responsive design site yet, and our mobile site is built through Dudamobile. I know it's not the best, but I'm trying to do whatever we can until we get around to redesigning it. Is there anything I can do about the following Page Speed Insight errors or are they just a function of using Dudamobile? Eliminate render-blocking JavaScript and CSS in above-the-fold content Your page has 3 blocking script resources and 5 blocking CSS resources. This causes a delay in rendering your page.None of the above-the-fold content on your page could be rendered without waiting for the following resources to load. Try to defer or asynchronously load blocking resources, or inline the critical portions of those resources directly in the HTML.Remove render-blocking JavaScript: http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js http://mobile.dudamobile.com/…ckage.min.js?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…pts/blogs.js?version=2015-04-02T13:36:04 Optimize CSS Delivery of the following: http://fonts.googleapis.com/…:400|Great+Vibes|Signika:400,300,600,700 http://mobile.dudamobile.com/…ont-pack.css?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…kage.min.css?version=2015-04-02T13:36:04 http://irp-cdn.multiscreensite.com/kempruge/files/kempruge_0.min.css?v=6 http://irp-cdn.multiscreensite.com/…mpruge/files/kempruge_home_0.min.css?v=6 Thanks for any tips, Ruben
Web Design | | KempRugeLawGroup0 -
Wordpress Theme is blocking alt tags. Does anybody know of any special plugins?
We have a special wordpress theme for nataliecass.com. Unfortunately the theme is blocking all the alt tags (this is a photography website...alt tags are very important). Does anybody know of any special WP plugins for alt tags? Thanks
Web Design | | VanguardCommunications0 -
Best way to move blog from subdomain to folder?
Hey all, Our company has 4 product websites, and each has its own separate blog. They are currently set up as subdomain blogs (blog.company.com) hosted on wordpress.com, but I would like to transition them over to root folders (company.com/blog) in order to improve accessibility and SEO. What is the best way to go about doing this? Should I continue to host the blogs on wordpress or are there better options? Would I migrate the blog posts over or just redirect? I'd like to get a general framework/plan of action going in order to know what to expect. Thanks!
Web Design | | kslusarski0 -
For A Corporation With 3 Distinct Business Divisions, Is It Better To Go With 1 Domain & 3 Sub-Domains, 1 Domain & 3 Folders, or 3 Domains for SEO Purposes?
Hi, I am working on a project right now for an existing client, we have one domain up and running well, they want to create an 'umbrella' site to cover three current business divisions and roll everything up under that main site, including the existing site on a totally different domain (would migrate over and 301 redirect from current domain). From what I've researched, I am inclined towards one main domain with three sub-domains due to the amount of content for each business division being significantly different enough that it seems to deserve separation from each other. However, in terms of SEO and maintaining consistent domain authority, would anyone recommend it be better to structure this as just folders/categories falling under the main domain instead of separate sub-domains for each division, and focus keyword targeting on pages tailored to that end within the main domain structure rather than spreading out link-juice to different sub-domains? Thanks!
Web Design | | Dan_InboundHorizons0 -
How to fix and issue with robot.txt ?
I am receiving the following error message through webmaster tools http://www.sourcemarketingdirect.com/: Googlebot can't access your site Oct 26, 2012
Web Design | | skehoe
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. The site has dropped out of Google search.0 -
Subdomain vs Sub Folder
Hi. This is my first question here guys and gals so please be gentle. We are creating a town based website that will hold news, events, articles and relevant local content. We are also creating a business directory that will be part of this. My question relates to an issue we have regarding how the directory will be treated as part of the "network". Although we aren't 100% sure on the domain name yet, we want the directory to be found as easily as the domain itself. Even better, if the two could be treated seperately but strengthen the overall theme for the domain we'd be in keyword heaven. Therefore in your opinion (hopefully with pro's and con's) which do you think will rank better in Google, Yahoo and Bing (or all three);
Web Design | | rufo
The directory seperate as a subdomain or as a folder within the main domain? So for example - providing we are including links between the two sections and the site is www.sitetown.com Which is better for the directory itself? http://www.sitetown.com/directory http://directory.sitetown.com or you're mad Steve, use a seperate domain altogether www.sitetowndirectory.com I hope you can help, obi-wan, you're my only hope..... PS - seomoz still rocks. Thanks
Steve0 -
Correct use for Robots.txt
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
Web Design | | EricVallee340