Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Practices for Homepage Title Tag
Hi, I would like to know if there is any update about the best practices for the homepage title tag. I mean, a couple of years ago, it was still working placing main keywords in the homepage title tag. But since the last google SERP update, the number of characters that are being shown were reduced, and now we try to work with 55 and 56 characters. That has reduced our capacity of including many keywords on the title tag. Besides, search engines are smarter now to choose the correct inner page to show in SERP. But I am wondering if the Homepage Title should have a branded orientation or should include main keywords, cause it is still working that strategy. I would appreciatte any update in this issue. Thank you!
Intermediate & Advanced SEO | | teconsite0 -
Google Not Pulling The Right Title Tag & Meta Description
Hi guys. We've found Google is pulling the wrong information for our title tag and meta description. Instead of pulling the actual title tag, Google is pulling the menu name you click on to get to the page: "Bike Barcelona" instead of "Barcelona Bike Tours | ...." Also, we've found that, instead of pulling the meta description we wrote, Google is using text from the pages copy. Any tips?
Intermediate & Advanced SEO | | BarcelonaExperience0 -
H2 Tags- Can you have more than 1 H2 tag
Hi All, Screaming frog has identified that we have a few H2 tags on our pages , although we only have 1 H1 tag. We have numerous H3,H4's etc. I am wondering, is it good SEO to have only 1 H2 tag like with H1 tag or can you have more ? thanks Peter
Intermediate & Advanced SEO | | PeteC120 -
Questions Regarding Wordpress Blog Format, Categories and Tag pages...
I'm looking to make some optimizations to a website I'm working on but wanted more input before I get started: Currently, when blogs are posted to the website, the URL for each post looks like this:www.mywebsite.com/blogpost I've heard that for whatever reason, the best practice is to make sure that the blog posts get posted to a blog sub-directory like so: www.mywebsite.com/blog/blogpost If I were to make this change, I'm assuming I would have to 301 redirect all of the existing blogs to their new locations. Is this change worth doing and are there any other considerations I should be taking into account? Also, I'm aware that there are certain schools of thought that category and tag pages should be no-indexed to avoid duplicate content issues. Can anyone shed some light on this from first hand experience? Thanks in advance!
Intermediate & Advanced SEO | | goldbergweismancairo0 -
Should I block wordpress archive and tag?
I use Wodpress and Wordpress SEO by Yoast. I've set ip up to add noindex meta tag on all archive and tag pages. I don't think its useful to include thoses pages in search results because there's quite a few. Especialy the tag archive. Should I consider anything else or change my mind? What do you think? Thanks
Intermediate & Advanced SEO | | Akeif0 -
Does link building through content syndication still actually work?
I stumbled across this old SEOmoz whilteboard http://www.seomoz.org/blog/whiteboard-friday-leveraging-syndicated-content-effectively and was wondering if this is still a valid technique given the Panda & Penguin updates. Is anyone here still doing this (and seeing results)?
Intermediate & Advanced SEO | | nicole.healthline0 -
Canonical Tag Uses Source Title and Meta Data?
When optimising a regional same language micro site within a sub folder of a .com it dawned on me that our use of the hreflang and canonical meta elements will render individual elements such as H1 and title obsolete. As a canonical tag takes the canonical source title and meta right? It would still have value in optimising localised headings though? Appreciate any thoughts, suggestions (o:
Intermediate & Advanced SEO | | 3wh0 -
HTML5 and using multiple H1 tags
Hi All, Our dev team have just asked me a very interesting question........ Within the context of an HTML5 page, where it is supported and encouraged to use multiple H1 tags, will the use of multiple H1 tags be detrimental to SEO? or does Google fully understand how HTML5 works and therefore not penalise a website for using multiple H1 tags? I have an opinion on this that if it helps usability and user experience then it is likely that it will be good for SEO. It would be really good to hear views of people who have tried this or have decided against it! Thanks
Intermediate & Advanced SEO | | A_Q0