Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Near Duplicate Title Tag Checker
Hi Everyone, I know there are a lot of tools like Siteliner, which can check the uniqueness of body copy, but are there any that can restrict the check to the title tags alone? Alternatively, is there an Excel or Google Sheets function that would allow me to do the same thing? Thanks, Andy
Intermediate & Advanced SEO | | AndyRSB0 -
Still Need to Write Title & Description Tag?
My SEO has advised me that Google has stopped using title and description tags for search results and as such, it is no longer necessary to write specific title and description tags. I see that Yoast seems to pull text to create these tags and sometimes it looks like it reflects the best elements of the content, sometimes it does not. Should I be asking our SEO team to write dedicated title and description tags or is it best practices to leave it to the Yoast plugin? My SEO is of the opinion that writing these tags is not a productive use of time as Google will serve results based on the user inquiry rather than the content on his tags. It sounds logical but it would be reassuring to receive further confirmation of this. Thoughts?"
Intermediate & Advanced SEO | | Kingalan1
Thanks, Alan0 -
Recommendations for the length of h1 tags and how much does it matter. What is the major disadvantage if the h1 tags are slightly longer.
recommendations for the length of h1 tags and how much does it matter. What is the major disadvantage if the h1 tags are slightly longer.
Intermediate & Advanced SEO | | MariaMcGrath0 -
Noindex / Nofollow multiple reviews pages?
I have well over a hundred pages of reviews (10 per page). I know this is solid content and I'd hate to not be able to leverage it, but I'm running into the issue of having duplicate title tags and H1s on all of the pages. What's the best way to make use of the review content without have those types of issues? Is a noindex / nofollow strategy something I should be considering here for Page 2 and beyond? Thanks! Edit: I did additional digging into pagination strategies and found this terrific article on Moz. I'm thinking it should address my questions regarding review pages as well.
Intermediate & Advanced SEO | | Andrew_Mac0 -
Using cononical tag instead of 301
I've got a bit of an odd situation... My business partner and I split up, and he's going to keep the company name. The website that I built for the company has some links to it, and I've managed to build up some DA and PA. I want to get the link juice over to my new website. My former partner doesn't care about the link juice, he just wants a website that he can show people. SO, I can't do a 301 or 302, because that would take down the existing site. Can I just use a canonical tag that refers link power to my new website? Would this be harmful in any way? What should I do to accomplish getting the link power without a redirect, and without contacting each person who has given us a backlink?
Intermediate & Advanced SEO | | Zing-Marketing0 -
Why does the site I am working on have so few visits from organic search results?
Hello! I am not very experienced with SEO, but I am trying to help out on a site that has been around since 2010 and has well over a thousand pages of high-quality, original content, with more being added all the time. Only around 65 of the site's daily visits come from organic search results; this seems very low. There has already been significant SEO work done on the site. Is there something about the site that strikes anyone as obviously getting in the way of organic traffic? The URL is ellenjovin.com. I would appreciate any thoughts you may have. Thank you very much!
Intermediate & Advanced SEO | | nyc-seo0 -
Panda Updates - robots.txt or noindex?
Hi, I have a site that I believe has been impacted by the recent Panda updates. Assuming that Google has crawled and indexed several thousand pages that are essentially the same and the site has now passed the threshold to be picked out by the Panda update, what is the best way to proceed? Is it enough to block the pages from being crawled in the future using robots.txt, or would I need to remove the pages from the index using the meta noindex tag? Of course if I block the URLs with robots.txt then Googlebot won't be able to access the page in order to see the noindex tag. Anyone have and previous experiences of doing something similar? Thanks very much.
Intermediate & Advanced SEO | | ianmcintosh0 -
Are tags an issue in SEO
SEOMoz saw that my tags were duplicate pages. Are tags a serious issue in SEO? Should I remove it entirely to prevent the duplicate pages?
Intermediate & Advanced SEO | | visualartistics0