Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Optimizing for Two Keywords - H Tag Best Practices?
Hey Everyone, I have to do a local SEO campaign. My landing pages need to target two keywords. I was wondering if you could look over this proposed H tags I've written and give me your thoughts. Houses for Sale and Commercial Real Estate in Houston, TX Houses for Sale in Houston, TX Commercial Real Estate in Houston, TX Am I heading in the right or wrong direction?
Intermediate & Advanced SEO | | Charles_Murdock0 -
Is a Rel Canonical Sufficient or Should I 'NoIndex'
Hey everyone, I know there is literature about this, but I'm always frustrated by technical questions and prefer a direct answer or opinion. Right now, we've got recanonicals set up to deal with parameters caused by filters on our ticketing site. An example is that this: http://www.charged.fm/billy-joel-tickets?location=il&time=day relcanonicals to... http://www.charged.fm/billy-joel-tickets My question is if this is good enough to deal with the duplicate content, or if it should be de-indexed. Assuming so, is the best way to do this by using the Robots.txt? Or do you have to individually 'noindex' these pages? This site has 650k indexed pages and I'm thinking that the majority of these are caused by url parameters, and while they're all canonicaled to the proper place, I am thinking that it would be best to have these de-indexed to clean things up a bit. Thanks for any input.
Intermediate & Advanced SEO | | keL.A.xT.o0 -
Will authorship for a reviewer, not an author work?
I'm working with a client that owns a medical site. All content is reviewed by someone from their medical board (doctors or nurses), but the content is written by a variety of authors. I'm wondering if we could create authorship profiles for the doctors and nurses. Would there be any problem with that? (even though they didn't write the content, they just reviewed it for medical accuracy). The name of the reviewer is included on every article. Any thoughts / feedback / similar experiences would be helpful.
Intermediate & Advanced SEO | | nicole.healthline0 -
Is this structure valid for a canonical tag?
Working on a site, and noticed their canonical tags follow the structure: //www.domain.com/article They cited their reason for this as http://www.ietf.org/rfc/rfc3986.txt. Does anyone know if Google will recognize this as a valid canonical? Are there any issues with using this as a the canonical?
Intermediate & Advanced SEO | | nicole.healthline0 -
NOINDEX content still showing in SERPS after 2 months
I have a website that was likely hit by Panda or some other algorithm change. The hit finally occurred in September of 2011. In December my developer set the following meta tag on all pages that do not have unique content: name="robots" content="NOINDEX" /> It's been 2 months now and I feel I've been patient, but Google is still showing 10,000+ pages when I do a search for site:http://www.mydomain.com I am looking for a quicker solution. Adding this many pages to the robots.txt does not seem like a sound option. The pages have been removed from the sitemap (for about a month now). I am trying to determine the best of the following options or find better options. 301 all the pages I want out of the index to a single URL based on the page type (location and product). The 301 worries me a bit because I'd have about 10,000 or so pages all 301ing to one or two URLs. However, I'd get some link juice to that page, right? Issue a HTTP 404 code on all the pages I want out of the index. The 404 code seems like the safest bet, but I am wondering if that will have a negative impact on my site with Google seeing 10,000+ 404 errors all of the sudden. Issue a HTTP 410 code on all pages I want out of the index. I've never used the 410 code and while most of those pages are never coming back, eventually I will bring a small percentage back online as I add fresh new content. This one scares me the most, but am interested if anyone has ever used a 410 code. Please advise and thanks for reading.
Intermediate & Advanced SEO | | NormanNewsome0 -
How can I check if the FOLLOW,NOINDEX tag is working?
Hi everyone! After reading about pagination practices, a few days ago we introduced the <meta name="robots" content="FOLLOW,NOINDEX" /> tag, to prevent duplicate content. You can find an example below: http://www.inmonova.com/en/properties?page=2 I have been checking yahoo site explorer and result pages still get indexed. My question is: Am I doing something wrong? Is the code incorrect (follow,noindex - noindex,follow)? Or does it just take some time to have effect? Thanks in advance.
Intermediate & Advanced SEO | | inmonova0 -
What is the downside with having too long of a title tag?
With Google placing so much relevance on title tags, it seems to help if you mention local cities within your title tag. I'm wondering if the positive of having more keywords in a title tag outweighs the negative of having too long of one?
Intermediate & Advanced SEO | | TLSNET0 -
H1 Tags
Quick and easy most likely - Just need to clear a few point. I understand each page within the site should only have one H1 tag which should be the most important one. I also believe these only effect google ranking very slightly? right? Currently my CMS is system is pulling the H1 tag in from the page and automatically using the page heading that is on the page IE) the heading used for the content. Should this be a keyword / key phrase instead? and will it be duplicate if i used the same one on various pages in my site? Cheers guys look forward to hearing your feedback
Intermediate & Advanced SEO | | wazza19850