Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Translating meta tags using WPML and AIO SEO
Having a heck of a time finding info on this one... We're working on a multilingual website which uses WPML. I've used the All in One SEO plugin to customize meta data (title, description, etc). These strings do not appear in the list of translations in WPML. Does anyone have any experience with this setup? How do you enable WPML to translate meta data set via the AIO plugin? Thanks!
Intermediate & Advanced SEO | | jonmc0 -
H Tags in Menu
Hi I am checking the H2 tags on this page https://www.key.co.uk/en/key/dollies-load-movers-door-skates I have noticed my dev team have implemented H2's on the categories in the menu. Will this completely confuse Google as to what that page is about? In my opinion those links shouldn't be heading tags at all
Intermediate & Advanced SEO | | BeckyKey0 -
Changing title tags - any potential issues?
Hello all, I am planning to change the title tags throughout a site and am vaguely aware (perhaps wrongly!) that changing title tags across a site is a risk factor - can be a spam flag if changes (to a specific title tag) are implemented too regularly, for example. Would you change title tags across a site in one go, or implement changes gradually - to avoid any risk of upsetting Google. Do you have any insights/tips on the implementation of title tag changes?
Intermediate & Advanced SEO | | McTaggart1 -
H3 Tags - Should I Link to my content Articles- ? And do I have to many H3 tags/ Links as it is ?
Hello All, On my ecommerce landing pages, I currently have links to my products as H3 Tags. I also have useful guides displayed on the page with links useful articles we have written (they currently go to my news section). I am wondering if I should put those article links as additional H3 tags as well for added seo benefit or do I have to many tags as it is ?. A link to my Landing Page I am talking about is - http://goo.gl/h838RW Screenshot of my h1-h6 tags - http://imgur.com/hLtX0n7 I enclose screenshot my guides and also of my H1-H6 tags. Any advice would be greatly appreciated. thanks Peter
Intermediate & Advanced SEO | | PeteC120 -
Does Link Detox Boost Work?
That is a question I am sure many of your have been asking since they launched the product several weeks ago. Cemper claims they helped get a penalty removed in 3 days by using this product. Sounds great doesn't it? Maybe even sounds too good to be true. Well, here is my experience with it. We have been working to get a site's rankings back up for several months now. While it has no penalty, it clearly got hit by the algo change. So we have been very busy creating new content and attempting to remove as much "keyword rich" links as possible. This really hasn't been working very well at all, so when I heard about link detox boost I thought this was the answer to our prayers. The basic idea is link detox boost forces google to crawl your bad links so it know you no longer have links from those sites or have disavowed them. So we ran it and it was NOT cheap. Roughly $300. Now, 3 weeks after running it, the report only shows it has actually crawled 25% of our links, but they assure us it is a reporting issue and the full process has ran its course. The results. No change at all. Some of our rankings are worse, some are better, but nothing worth mentioning. Many products from Link Research Tools are very good, but i'm afraid this isn't one of them. Anyone else use this product? What were your results?
Intermediate & Advanced SEO | | netviper2 -
Canonical tag - but Title and Description are slightly different
I am building a new SEO site with a "Silo" / Themed architecture. I have a travel website selling hotel reservations. I list a hotel page under a city page - example, www.abc.com/Dallas/Hilton.html Then I use that same property under a segment within the city - example www.abc.com/Dallas/Downtown/Hilton.html, so there are two URLs with the same content Both pages are identical, except I want to customize the Title and Description. I want to customize the title and description to build a consistent theme - for example the /Downtown/Hilton page will have the words "Near Downtown" in the Title and Description, while the primary city Hilton page will not. So I have two questions about this. First, is it okay to use a canonical tag if the Title and Description are slightly different? Everything else is identical. If so, will Google crawl and comprehend the unique Title and Description on the "Downtown" silo? I want Google to see that I have several "supporting" pages to my main landing page(s). I want to present to Google 5 supporting pages in each silo that each has a supporting keyword theme. But I'm not sure if Google will consider content of pages that point to a different page using the canonical tag. Please see this supporting example: http://d.pr/i/aQPv Thanks for your insights. Rob
Intermediate & Advanced SEO | | partnerf0 -
Should you use a canonical tag on translated content in a multi-language country?
A customer of ours has a website in Belgium. There two main languages in Belgium: Dutch and French.
Intermediate & Advanced SEO | | Zanox
At first there was only a Dutch version with a .be extension. Right now they are implementing the French Belgium version on the URL website.be/fr. All of the content and comments will be translated. Also the URL’s will change from Dutch to French, so you've got two URL’s with the same content but in another language. Question: Should you use a canonical tag on translated content in a multi-language country? I think Google will understand this is just for the usability for a Multilanguage country. What do you guys think???0 -
Meta NoIndex tag and Robots Disallow
Hi all, I hope you can spend some time to answer my first of a few questions 🙂 We are running a Magento site - layered/faceted navigation nightmare has created thousands of duplicate URLS! Anyway, during my process to tackle the issue, I disallowed in Robots.txt anything in the querystring that was not a p (allowed this for pagination). After checking some pages in Google, I did a site:www.mydomain.com/specificpage.html and a few duplicates came up along with the original with
Intermediate & Advanced SEO | | bjs2010
"There is no information about this page because it is blocked by robots.txt" So I had added in Meta Noindex, follow on all these duplicates also but I guess it wasnt being read because of Robots.txt. So coming to my question. Did robots.txt block access to these pages? If so, were these already in the index and after disallowing it with robots, Googlebot could not read Meta No index? Does Meta Noindex Follow on pages actually help Googlebot decide to remove these pages from index? I thought Robots would stop and prevent indexation? But I've read this:
"Noindex is a funny thing, it actually doesn’t mean “You can’t index this”, it means “You can’t show this in search results”. Robots.txt disallow means “You can’t index this” but it doesn’t mean “You can’t show it in the search results”. I'm a bit confused about how to use these in both preventing duplicate content in the first place and then helping to address dupe content once it's already in the index. Thanks! B0