Sanity Check: NoIndexing a Boatload of URLs
-
Hi,
I'm working with a Shopify site that has about 10x more URLs in Google's index than it really ought to. This equals thousands of urls bloating the index. Shopify makes it super easy to make endless new collections of products, where none of the new collections has any new content... just a new mix of products. Over time, this makes for a ton of duplicate content.
My response, aside from making other new/unique content, is to select some choice collections with KW/topic opportunities in organic and add unique content to those pages. At the same time, noindexing the other 90% of excess collections pages.
The thing is there's evidently no method that I could find of just uploading a list of urls to Shopify to tag noindex. And, it's too time consuming to do this one url at a time, so I wrote a little script to add a noindex tag (not nofollow) to pages that share various identical title tags, since many of them do. This saves some time, but I have to be careful to not inadvertently noindex a page I want to keep.
Here are my questions:
-
Is this what you would do? To me it seems a little crazy that I have to do this by title tag, although faster than one at a time.
-
Would you follow it up with a deindex request (one url at a time) with Google or just let Google figure it out over time?
-
Are there any potential negative side effects from noindexing 90% of what Google is already aware of?
-
Any additional ideas?
Thanks! Best... Mike
-
-
Hi Michael
The problem you have is the very low value content that exists on all of those pages and the complete impossibility of writing any unique Titles, Descriptions and content. There are just too many of them.
With a footwear client of mine I no indexed a huge slug of tags taking the page count down by about 25% - we saw an immediate 22% increase in organic traffic in the first month. (March 18th 2017 - April 17th 2017) the duplicates were all size and colour related. Since canonicalising (I'm English lol) more content and taking the site from 25,000 pages to around 15,000 the site is now 76% ahead of last year for organics. This is real measurable change.
Now the arguments:
Canonicalisation
How are you going to canonicalise 10,000+ pages ? unless you have some kind of magic bullet you are not going to be able to but lets look at the logic.
Say we have a page of Widgets (brand) and they come in 7 sizes. When the range is fully in stock all of the brand/size pages will be identical to the brand page, apart from the title & description. So it would make sense to canonicalise back to the brand. Even when sizes started to run out, all of the sizes will be on the brand page. So size is a subset of the brand page.
Similar but not the same for colour. If colour is a tag then every colour sorted page will be on the brand page. So really they are the same page - just a slimmer selection. Now I accept that the brand page will contain all colours as it did all sizes but the similarity is so great - 95 % of the content being the same apart from the colour, that it makes sense to call them the same.
So for me Canonicalisation would be the way to go but it's just not possible as there are too many of them.
Noindex
The upside of noindex is that it is generally easier to put the noindex tag on the page as there is no URL to tag. The downside is that the page is then not indexed in Google so you lose a little juice - I would argue by the way that the chances of being found in Google for a size page is extremely slim, less than 2% of visits came from size pages before we junked them and most of those were from a newsletter so reality is <1% not worth bothering about You could leave off the nofollow so that Google crawls through all of the links on the pages - the better option.
Considering your problem and having experience of a number of sites with the same problem Noindex is your solution.
I hope that helps
Kind Regards
Nigel - Carousel Projects.
-
Hi Chris & Nigel,
Thank you for the considered responses. Good points about canonicalizing. A part I find frustrating is that the shared title tag across dozens or hundreds of pages will be across many different products/groups of products. So, the title tag is not a solid way to group canonicals.
Since the url patterns vary, I don't see how I could group these by which dozens or hundreds canonicalize to which one page, let alone make the change in Shopify other than one page at a time. My understanding is that this title tag manipulation is the only handle Shopify gives for making these bulk changes.
Gah!
So, here are my follow up questions:
-
How big of a negative is this in it's as-is state and how much better will noindexing most of the 90% make it Google Organic-wise? I ask because even the BS title tag to noindex project is a huge time suck.
-
If more is ever revealed about how to more efficiently group and canonicalize in Shopify, would adding the canonical after noindexing capture that lost authority later or would the previous noindex have irretrievably lost that?
-
Given all that, would you continue as I am?
Thanks! Best... Mike
-
-
Hi Mike
I see this a lot with sites that have a ton of tag groups. One site I am working on has 50,000 pages in Google caused by tags appending themselves to every version of a URL, the site only has 400 products. Example
Site/size-4
Site/womens/size-4
Site/womens/boots/size-4
Site/womens/boots/ankle/size-4
Site/womens/clarks/boots/size-4Etc etc - If there are other tags like colour and features, this can cause a huge 3 dimensional matrix of additional pages that can slow down the crawl of the site - Google may not crawl all of the site as a result.
If it's possible to canonicalse then that is the best option as juice and follows are retained - very often it would be the page with the tag lopped off that the tag should cite.
In extreme circumstances I would consider noindexing the pages as they offer very skinny content and rubbish Meta because it's impossible to handle them individually. I have seen significant improvement in organics as a result.
Personally I don't think it's enough to simply leave Google to figure it out although I have seen some sites with very high DA get away with it.
To be honest I am pretty shocked that Shopify doesn't have a feature to cope with this
Regards
Nigel
Carousel Projects.
-
Hello Michael Johnson and Mozzers,
I have seen Shopify do this a few times, though I do not have clients on that particular platform at the moment. It is frustrating. You're right to want to resolve this issue. Between duplicate content, authority conflicts, and an inflated crawl budget, one issue or another is bound to hold back site performance.
Is this what you would do? Not immediately, no. I want to see those pages canonicalized. That way, your preferred pages get all the juice back from their respective canonical link. Is this an option for you?
**Deindex request... and s_ide effects?**_ Canonical tags would make these part irrelevant (yay less work!). To be thorough though: I'd let Google figure it out unless you have strong evidence your crawl budget is maxed. And I don't see any negative side effects from noindexing duplicate content. If worse comes to worse, you have a good plan.
Shape that content,
CopyChrisSEO and the Vizergy Team
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will URLS With Existing 301 Redirects Be as Powerful As New URLS In Serps?
Most products on our site have redirects to them from years of switching platform and merely trying to get a great and optimised URL for SEO purposes. My question is this: If a product URL has alot of redirects (301's), would it be more beneficial to me to create a duplicated version of the product and start fresh with a new URL? I am not on here trying to gain backlinks but my site is tn nursery dot net (proof:)
Intermediate & Advanced SEO | | tammysons
I need some quality help figuring out what to do.
Tammy0 -
Pages with URL Too Long
I manage a number of Shopify stores for ecommerce clients. MOZ keeps kindly telling me the URLs are too long. However, this is largely due to the structure of Shopify, which has to include 'collections' and 'products'. For example: https://domain.com.au/collections/collection-name/products/colour-plus-six-to-seven-word-product-name MOZ recommends no more than 75 characters. This means we have 25-30 characters for both the collection name and product name. VERY challenging! Questions: Anyone know how big an issue URLs are as a ranking factor? I thought pretty low. If it's not an issue, how can we turn off this alert from MOZ? If it is an issue, anyone got any ideas how to fix it on Shopify sites?
Intermediate & Advanced SEO | | muzzmoz0 -
Rankings Tanked since new Site redesign land new url Structure ? Anything Glaringly Obvious I need to check ?
Hi All, I've just checked my rankings and everything on my eCommerce Site has pretty much tanked really badly since my new URL structure and site redesign was put in a place 2 weeks ago. My url structure was originally long and had underscores but we have now made it clean, shorter and use hyphens. We also have location specific pages and we have incorporated these into the new url structure.Basically it now pretty much follows the breadcrumb trail on our website. We were originally a general online hire site but now we have become niche and only concentrating on one types of products, so we got rid of all the other categories/products and pages we do not deal with anymore. Our Rankings issue , was only bought to light in the most recent MOZ Ranking report so it's looking site google hates our new store. Someone mentioned the other day, that Google may have been doing a Panda/Penguin refresh last weekend, but I am surprised to have dropped like 20 to 50 places for most of my keywords. We have set up the 301 redirects, We have also made the site alot smaller and set up a few thousand 404's to get rid of a lot of redundant pages . We have cut down massively on the thin/duplicate content and have lots of good new content on there. We did new sitemaps , set up schema.org. , increase text to code ratio . Setup our H1-H5 tags on all our pages. made site mobile responsive.. Basically , we are trying to do everything right. Is there anything glaringly obvious , I should be checking ?. I attach a Short url link if anyone wants to have a quick glance- http://goo.gl/7mmEx i.e Could it be a problem with the new urls or anything else that I should be looking at ?.. I.e how can I check to make sure the link juice is being passed on to the new url ? Or is all this expected when doing such changes ? Any advice greatly appreciated .. Pete
Intermediate & Advanced SEO | | PeteC120 -
How careful do you need to be about changes to readable URLs?
We are moving to Sitecore where the standard out the box is that if you change page title it amends the URL as well. I am worried that this will lead to SEO issues and am considering whether we need to get it locked down so that if the page title is amended (only in a minor way) it does not also change the URL. I have never worked with readable URLs before - what are the implications of the URL not exactly matching the wording of the page title?
Intermediate & Advanced SEO | | alzheimerssoc0 -
Google tagged URL an overly-dynamic URL?
I'm reviewing my campaign, and spotted the overly-dynamic URL box showing a few links. Reviewing it, they are my Google Tagged URLs (utm_source, utm_medium_utm_campaign etc) I've turned some internal links to Google Tagged URLs but should these cause concern?
Intermediate & Advanced SEO | | Bio-RadAbs0 -
Noindex, rel=cannonical, or no worries?
Hello, SEO pros, We need your help with a case ↓ Introduction: Our website allows individual contractors to create a webpage where they can show what services they offer, write something about themselves and show their previous projects in pictures. All the professions and services assigned accordingly are already in our system, so users need to pick a profession and mark all services they provide or suggest those which we missed to add. We have created unique URLs for all the professions and services. We have internal search field and use a autocomplete to direct users to the right page. **Example: ** PROFESSION Carpenter (URL: /carpenters ) SERVICES Decking (URL: /carpenters/decking) Kitchens (URL: /carpenters/kitchens) Flooring and staircases (URL: /carpenters/flooring-and-staircases) Door trimming (URL: /carpenters/door-trimming) Lock fitting (URL: /carpenters/lock-fitting) Problem We want to be found by Google search on all the services and give a searchers a list of all carpenters in our database who can provide a service they want to find. We give 15 contractors per page and rank them by recommendations provided by their clients. Our concern is that our results pages may be marked as duplicate since some of them give the same list of carpenters. All the best 15 carpenters offer door-trimming and lock-fitting. So, all the same 15 are shown in /carpenters, /carpenters/lock-fitting, /carpenters/door-trimming. We don't want to be marked as spammers and loose points on domain trust, however we believe we give quality content since we gave what the searchers want to find - contractors, who offer what they need. **Solution? ** Noindex all service pages to avoid duplicate content indexed by Google OR rel=canonical tag on service pages to redirect to profession page. e.g. on /carpenters/lock-fitting page make a tag rel=canonical to /carpenters. OR no worries, allow Google index all the professions and services pages. Benefits of indexing it all (around 2500 additional pages with different keywords) is greater than ttagging service pages with no index or rel=canonical and loosing the opportunities to get more traffic by service titles. We need a solution which would be the best for our organic traffic 🙂 Many thanks for your precious time.
Intermediate & Advanced SEO | | osvaldas0 -
URL for New Product
Hi, We are creating a section on our established existing website to display our new marketplace product & associated category pages. This marketplace will be a section of the site where our users can sell online training courses that they've created. It will be branded on our site as the Marketplace. Is it important to include 'marketplace' in the URL? Or would it be better to include a relevant keyword such as 'training-courses' instead? Or both? I've assumed I shouldn't use both as that would increase the length of the URLs and number of subfolders.
Intermediate & Advanced SEO | | mindflash0 -
Canonical URL Question
Hi Everyone I like to run this question by the community and get a second opinion on best practices for an issue that I ran into. I got two pages, Page A is the original page and Page B is the page with duplicate content. We already added** ="Page A**" />** to the duplicate content (Page B).** **Here is my question, since Page B is duplicate content and there is a link rel="canonical" added to it, would you put in the time to add meta tags and optimize the title of the page? Thanks in advance for all your help.**
Intermediate & Advanced SEO | | DRTBA0