Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How (or if) to apply re canonical tags to Shopify?
Anyone familiar with Shopify will understand the problems of their directory structure. Every time you add a product to a 'collection' it essentially creates a duplicate. For example... https://www.domain.com/products/product-slim-regular-bikini may also appear as: https://www.domain.com/collections/all/products/product-slim-regular-bikini https://www.domain.com/collections/new-arrivals/products/product-slim-regular-bikini https://www.domain.com/collections/bikinis/products/product-slim-regular-bikini etc, etc It's not uncommon to have up to six duplicates of each product. So my question is twofold: Firstly, should I worry about this from an SEO point of view? I understand the desire to minimise potential duplicate content issues and also in focussing the 'juice' on just one page per product. But I also planned on trying to build the authority of the collection pages. If I request Google not to index the product pages which link off the collections, does this not devalue these collections pages? Secondly, I understand the correct way to fix these is using 'rel canonical' tags, but I'm not clear about HOW to actually do this. Shopify support has not been very helpful. They have provided two different instructions, so just added to the confusion (see below). Shopify instruction #1: Add the following to the theme.liquid file... <title><br />{{ page_title }}{% if current_tags %} – tagged "{{ current_tags | join: ', ' }}"{% endif %}{% if current_page != 1 %} – Page {{ current_page }}{% endif %}{% unless page_title contains shop.name %} – {{ shop.name }}{% endunless %}<br /></title>
Intermediate & Advanced SEO | | muzzmoz
{% if page_description %} {% endif %} Shopify instruction #2: Add the following to each individual product page... So, can anyone help clarify: The best strategic approach to this inherent SEO issue with Shopify (besides moving to another platform!)? and If 'rel canonical' tags is the way to go, exactly where and how to apply them? Regards, Murray1 -
Google Ignoring Canonical Tag for Hundreds of Sites
Bazaar Voice provides a pretty easy-to-use product review solution for websites (especially sites on Magento): https://www.magentocommerce.com/magento-connect/bazaarvoice-conversations-1.html If your product has over a certain number of reviews/questions, the plugin cuts off the number of reviews/questions that appear on the page. To see the reviews/questions that are cut off, you have to click the plugin's next or back function. The next/back buttons' URLs have a parameter of "bvstate....." I have noticed Google is indexing this "bvstate..." URL for hundreds of sites, even with the proper rel canonical tag in place. Here is an example with Microsoft: http://webcache.googleusercontent.com/search?q=cache:zcxT7MRHHREJ:www.microsoftstore.com/store/msusa/en_US/pdp/Surface-Book/productID.325716000%3Fbvstate%3Dpg:8/ct:r+&cd=2&hl=en&ct=clnk&gl=us My website is seeing hundreds of these "bvstate" urls being indexed even though we have a proper rel canonical tag in place. It seems that Google is ignoring the canonical tag. In Webmaster Console, the main source of my duplicate titles/metas in the HTML improvements section is the "bvstate" URLs. I don't necessarily want to block "bvstate" in the robots.txt as it will prohibit Google from seeing the reviews that were cutoff. Same response for prohibiting Google from crawling "bvstate" in Paramters section of Webmaster Console. Should I just keep my fingers crossed that Google honors the rel canonical tag? Home Depot is another site that has this same issue: http://webcache.googleusercontent.com/search?q=cache:k0MBLFcu2PoJ:www.homedepot.com/p/DUROCK-Next-Gen-1-2-in-x-3-ft-x-5-ft-Cement-Board-172965/202263276%23!bvstate%3Dct:r/pg:2/st:p/id:202263276+&cd=1&hl=en&ct=clnk&gl=us
Intermediate & Advanced SEO | | redgatst1 -
H2 vs. H3 Tags for Category Navigation
Hey, all. I have client that uses tags in the navigation for its blog. For example, tags might appear around "Library," "Recent Posts," etc. This is handled through their WordPress theme. This seems fairly standard, but I wonder whether tags are semantically appropriate. Since each blog post is fairly lengthy (about 500-1000 words) with multiple tags, would it be more appropriate to use tags for this menu navigation? Are we cutting into the effectiveness of our tags by using them for menu navigation? The navigation is certainly an important page element, and it structures content, so it seems that it should use some header tag. Anyways, your thoughts are greatly appreciated. I'm a content creator, not an SEO, so this is a bit out of my skillset.
Intermediate & Advanced SEO | | Ask44435230 -
Duplicate Title tags even with rel=canonical
Hello, We were having duplicate content in our blog (a replica of each post automatically was done by the CMS), until we recently implemented a rel=canonical tag to all the duplicate posts (some 5 weeks ago). So far, no duplicate content were been found, but we are still getting duplicate title tags, though the rel=canonical is present. Any idea why is this the case and what can we do to solve it? Thanks in advance for your help. Tej Luchmun
Intermediate & Advanced SEO | | luxresorts0 -
Canonical tag - but Title and Description are slightly different
I am building a new SEO site with a "Silo" / Themed architecture. I have a travel website selling hotel reservations. I list a hotel page under a city page - example, www.abc.com/Dallas/Hilton.html Then I use that same property under a segment within the city - example www.abc.com/Dallas/Downtown/Hilton.html, so there are two URLs with the same content Both pages are identical, except I want to customize the Title and Description. I want to customize the title and description to build a consistent theme - for example the /Downtown/Hilton page will have the words "Near Downtown" in the Title and Description, while the primary city Hilton page will not. So I have two questions about this. First, is it okay to use a canonical tag if the Title and Description are slightly different? Everything else is identical. If so, will Google crawl and comprehend the unique Title and Description on the "Downtown" silo? I want Google to see that I have several "supporting" pages to my main landing page(s). I want to present to Google 5 supporting pages in each silo that each has a supporting keyword theme. But I'm not sure if Google will consider content of pages that point to a different page using the canonical tag. Please see this supporting example: http://d.pr/i/aQPv Thanks for your insights. Rob
Intermediate & Advanced SEO | | partnerf0 -
How does the use of Dynamic meta tags effect SEO?
I'm evaluating a new client site which was built buy another design firm. My question is they are dynamically creating meta tags and I'm concerned that it is hurting their SEO. When I view the page source this is what I see. <meta name="<a class="attribute-value">keywords</a>" id="<a class="attribute-value">keywordsGoHere</a>" content="" /> <meta name="<a class="attribute-value">description</a>" id="<a class="attribute-value">descriptionGoesHere</a>" content="" /> <title id="<a class="attribute-value">titleGoesHere</a>">title> To me it looks like the tags are not being added to the page, however the title is showing when you view it in a browser and if use a spider view tool, it sees the title. I'm guess it is being called from a DB. So I'm a little concerned though that the search engines are not really seeing the title and description. I'm not worried about the keywords tag. Can anyone shed some light on how this might work? Why it might not being showing the text for the description in the page code and if that will hurt SEO? Thanks for the help!
Intermediate & Advanced SEO | | BbeS0 -
Why Put an H1 Tag On A Product?
Why would you put an H1 tag on a product name? I came across this in another forum and thought I'd float it here.
Intermediate & Advanced SEO | | AWCthreads0 -
Why should your title and H1 tag be different?
Is it dangerous to have your H1 tag and your title the exact same thing? My thought was that it's not be the best use of space, but that it couldn't cause harm. What do you think?
Intermediate & Advanced SEO | | MarieHaynes7