Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Block parent folder in robot.txt, but not children
-
Example:
I want to block this URL (which shows up in Webmaster Tools as an error):
http://www.siteurl.com/news/events-calendar/usa
But not this:
-
The idea from Andrew is nice, but my guess would be that you're targeting multiple events so that might run into issues. What you could do is add some more regular expression and make it like this:
Disallow: ^/news/events-calendar/usa$
-
You could use "allow" in your robots.txt file for just this problem.
allow: news/events-calendar/usa/event-name
disallow: /news/events-calendar/usa
See the allow directive section of this page: https://en.wikipedia.org/wiki/Robots_exclusion_standard
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What type of website is best for seo.
I need a new website for my health insurance business. What type is best for SEO? Many thanks
Web Design | | laurentjb1 -
Reducing cumulative layout shift for responsive images - core web vitals
In preparation for Core Web Vitals becoming a ranking factor in May 2021, we are making efforts to reduce our Cumulative Layout Shift (CLS) on pages where the shift is being caused by images loading. The general recommendation is to specify both height and width attributes in the html, in addition to the CSS formatting which is applied when the images load. However, this is problematic in situations where responsive images are being used with different aspect ratios for mobile vs desktop. And where a CMS is being used to manage the pages with images, where width and height may change each time new images are used, as well as aspect ratios for the mobile and desktop versions of those. So, I'm posting this inquiry here to see what kinds of approaches others are taking to reduce CLS in these situations (where responsive images are used, with differing aspect ratios for desktop and mobile, and where a CMS allows the business users to utilize any dimension of images they desire).
Web Design | | seoelevated3 -
Does changing content and design of the website gonna affect my all the backlinks i have made till now
i have been working on my link profile for a month now, after learning about 5 step moz methodology i have decided that i would like to change all of the content of my site and taylor it to what my customers need, am i gonna loose all the domain authority if make changes? if it gonna affect, hows that gonna come out
Web Design | | calvinkj0 -
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
Payment Options in Footer
Hi Moz Community. I'm currently working on cleaning up a website footer on an e-commerce site. I was curious to find out if including payment options is important? Either text or image icons. Including Visa, AMEX etc. Are there benefits of displaying them? Image attached. Thanks vcdoL
Web Design | | Kdruckenbrod0 -
Bing Indexation and handling of X-ROBOTS tag or AngularJS
Hi MozCommunity, I have been tearing my hair out trying to figure out why BING wont index a test site we're running. We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
Web Design | | AU-SEO
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing. We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site. In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header. With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag. However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive. I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing. I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages. I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content. Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages. Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl. Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests? Thanks in advance for your assistance....0 -
Should the parent directory of the main site-navigation be clickable or not?!?
Highly discussed in our team is the question: Should all parent navigation items be clickable, or only the ones that have no child menu appearing on mouse over? At Starwood Germany, we would like to adjust the main navigation for all our websites in order to improve consistency and user friendliness. At the moment, most of our websites feature both clickable non-clickable parent items, depending on whether the items have a corresponding child menu (appearing on mouse over) or not. See example here: http://www.imperialvienna.com/en Some of our team members believe it might be irritating and/or confusing for the user if some items are clickable while others are not. What do you think? Any thoughts and insights would be truly appreciated!
Web Design | | DFM_GSA0 -
IP block in Google
Our office has a number of people performing analysis and research on keyword positions, volume, competition etc. We have 1 external static IP address. We installed the static IP so we can filter out our visits in Google Analytics. However by 10 AM we get impssible CAPTCHA's or even get blocked in Google. Do you have any experience with such an issue? Any solutions you can recommend? Any help would be appreciated! SXI5A.png
Web Design | | Partouter0