Interest in optimise Google Crawl
-
Hello,
I have an ecommerce site with all pages crawled and indexed by Google.
But I have some pages with multiple urls like : www.sitename.com/product-name.html and www.sitename.com/category/product-name.html
There is a canonical on all these pages linking to the simplest url (so Google index only one page). So the multiple pages are not indexed, but Google still comes crawling them.
My question is : Did I have any interest in avoiding Google to crawl these pages or not ?
My point is that Google crawl around 1500 pages a day on my site, but there are only 800 real pages and they are all indexed on Google. There is no particular issue, so is it interesting to make it change ?
Thanks
-
Hi!
Have you no indexed the pages too? That may help to make sure that they aren't being crawled if that's concerning you. May at least give Google another signal not to crawl those pages.
Obviously it's not a catch all as there's only so much you can do to tell Google not to crawl a page. Sometimes if the alternative page is linked to internally (which it sounds like it is), then it will automatically crawl it even though you've said it has a canonical on it as you're showing that the page is important to your site.
May be worth testing a few pages to see if it has an impact.
-
Hi there!
From my experience, the best results I was ever able to achieve for a Client is when we consolidated all URLs to a single URL solution. Canonicals are amazing, no doubt. But I've experienced a canonical structure being ignored if there are instances where the canonical structure isn't 100% 'correct.'
If there is a way that you can have your website navigation & internal/XML sitemap reinforce your preferred URL, that would certainly reduce the number of URLs Google would crawl. Then, if you permanently (301) redirect all the now non-navigable URLs to the single preferred URL, you should see a significant boost in traffic (from consolidating all of the authority into a single page, now reinforced throughout your entire website).
If that's not possible, and you have to have multiple URLs within your site for budget/platform constraints, then yes, let Google crawl them. Otherwise the algo won't be able to see your canonical tag across them.
So in short: If you have a means to reduce the number of duplicates and redirect them - awesome. If you don't have a means to reduce duplicates, opening them up to Google is good, too.
For more information on making sure your canonical structure is set up properly, check out this Moz blog post: https://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website dropped out from Google index
Howdy, fellow mozzers. I got approached by my friend - their website is https://www.hauteheadquarters.com She is saying that they dropped from google index over night - and, as you can see if you google their name, website url or even site: , most of the pages are not indexed. Home page is nowhere to be found - that's for sure. I know that they were indexed before. Google webmaster tools don't have any manual actions (at least yet). No sudden changes in content or backlink profile. robots.txt has some weird rule - disallow everything for EtaoSpider. I don't know if google would listen to that - robots checker in GWT says it's all good. Any ideas why that happen? Any ideas what I should check? P.S. Just noticed in GWT there was a huge drop in indexed pages within first week of August. Still no idea why though. P.P.S. Just noticed that there is noindex x-robots-tag in headers... Anyone knows where this can be set?
Intermediate & Advanced SEO | | DmitriiK0 -
April Google Update?
Since April 16 (when Jews ate Matzah) Google hurt one of our clients badly. They are well-known and beloved brand with hundreds of employees and locations across USA.
Intermediate & Advanced SEO | | Elchanan
I can’t see any signal of organic update, or penalty (neither Google Places). No message on GWT Nothing has been changed on and off site. All keywords' ranking are looking like this All tools showing good analysis: MOZ, Barracuda, MajesticSeo Content is good and not duplicated, etc. Do one of you is aware of significant Google update?
What do you think/suggest?0 -
Penalty from Google due to spam that was not our doing
Hi, My company has enjoyed pretty good rankings for our main keywords in Google for the past 13+ years we have been in business. We have always been very white-hat about our SEO -- always erring on the side of not doing anything rather than risking a penalty. Well, last Thursday, we received the dreaded Google penalty due to a pattern of unnatural links. The hit is devastating - we are not even in the top 50 for our own company name anymore. Through research, we believe we have found the culprit -- and it has nothing to do with any of our own actions. We operate a discussion forum, and there was a link to one of the threads that was being used as the target for a lot of link spam -- Chinese blog comments, etc. We had nothing to do with this, but obviously someone had an agenda and was working on spamming to this page. Who knows - it may have even been the company that was being discussed negatively in the thread, attempting to have us blocked. We discovered it the same day we received the penalty notice, and issued a reconsideration request, detailing what we believe we found. So far, we haven't disavowed any of the links, but I am thinking we should. We have asked Google if they're able to just turn off any link juice for that one page, especially since we don't know who is doing this spamming, and whether they will continue. Has anyone experienced something similar? How does one prevent themselves from receiving a penalty that they had nothing to do with? What is there to keep any competitor from launching a spammy link-building campaign to get their competitor removed from Google? Is there anything we can do to resolve this? Thanks for any and all thoughts...
Intermediate & Advanced SEO | | kylie_rw0 -
Website No Longer Ranking In Google:
My website was on first page google couple of months ago, now nothing. Shows up in Bing page one. Some queries/pages still showing OK, but some not at all. Example "residential elevators illinois" found nowhere. http://www.accesselevator.net is the website. Have found 900 poor quality links and used disavow tool. Any further suggestions? Their Page Rank also went from a 3 to a 2. Implemented nofollow on all outgoing links. Need advice.
Intermediate & Advanced SEO | | trailblazerzz90 -
Google Fetch Issue
I'm having some problems with what google is fetching and what it isn't, and I'd like to know why. For example, google IS fetching a non-existent page but listing it as an error: http://www.gaport.com/carports but the actual url is http://www.gaport.com/carports.htm. Google is NOT able to fetch http://www.gaport.com/aluminum/storage-buildings-10x12.htm. It says the page doesn't exist (even though it does) and when I click on the not found link in Google fetch it adds %E@%80%8E to the url causing the problem. One theory we have is that this may be some sort of server/hosting problem, but that's only really because we can't figure out what we could have done to cause it. Any insights would be greatly appreciated. Thanks and Happy Holidays! Ruben
Intermediate & Advanced SEO | | KempRugeLawGroup0 -
My own brand name disappeared from google?
Hi, about 20-30 hours ago my own brand name disappeared from google results (We redirected old domain to new one about a month ago) My website is: www.websiteplanet.com If you search for Website Planet in google you will not find our homepage any longer.
Intermediate & Advanced SEO | | Ouzan
Not only that the brand name disappeared but we also dropped in rankings and lost about %50 of the organic traffic we had. It's important for me to say that we have never done any sort of blackhat or even greyhat SEO, at all. I could probably come up with many ideas of why it happened but maybe one of you mozzers already experienced this and could enlighten me. Will really appreciate any kind of response/help. Thanks.0 -
Clickable links in Google SERP
Hi, I came across clickable links appearing below the meta description in Google SERP. The links are not necessarily the top pages for that particular domain and sometimes tend to reflect the newly added stock. Is there a way to determine what factors account for this? Thanks, mryxN.png
Intermediate & Advanced SEO | | RaksG0 -
Google WMT Change of address.
Hey all. I've got two domain names. -rt112media.com -route112media.com http://rt112media.com is my main address. The preferred domain. My Google WMT account has 4 sites listed that I am a verified owner on. 1.)rt112media.com 2.)www.rt112media.com 3.)route112media.com 4.)www.route112media.com I recent changed the address in WMT for route112media.com to have the preferred domain rt112media.com. I did this on Nov 23. It is currently saying their is a request and I can withdrawl. Now I'm needing to set the preferred for www.rt112media.com to be rt112media.com but it will not let me because of the pending route112media.com. The problem is I'm getting a ton of errors under my www.rt112media.com diagnostics. I'm not getting any under rt112media.com but I am thinking it is still effecting my page rank. I have the 301's for all set up as a 301 wildcard to rt112media.com. They all redirect to the preferred domain I wan (rt112media.com.) Suggestions?
Intermediate & Advanced SEO | | Route112Media0