Sitemap url's not being indexed
-
There is an issue on one of our sites regarding many of the sitemap url's not being indexed. (at least 70% is not being indexed)
The url's in the sitemap are normal url's without any strange characters attached to them, but after looking into it, it seems a lot of the url's get a #. + a number sequence attached to them once you actually go to that url. We are not sure if the "addthis" bookmark could cause this, or if it's another script doing it.
For example
Url in the sitemap: http://example.com/example-category/0246
Url once you actually go to that link: http://example.com/example-category/0246#.VR5a
Just for further information, the XML file does not have any style information associated with it and is in it's most basic form.
Has anyone had similar issues with their sitemap not being indexed properly ?...Could this be the cause of many of these url's not being indexed ?
Thanks all for your help.
-
Anders,
Thanks for the reply. I definitely agree a self referring canonical might just be a good extra addition on these product pages, so I'm definitely adding that to our list of to do's if it does not improve.
In terms of indexing pages - We have not restricted crawl frequency, we have it set to "allow google to determine the optimal crawl rate". No other warnings found within the search console either.
Thanks for your help.
-
I agree - i probably would ignore everything after the "#".
But have you tried added a <link rel="canonical" href="http://example.com/page-url" /> to your pages and see if this will update it? Also: Add the sitemap to your robots.txt if not allready done.
Regarding indexing pages - have you restricted crawl frequency in Google Search Console, or is it set to be determined by GoogleBot? Any other warnings or messages in Search Console?
Best regards,
Anders -
Lesley,
Thanks for the confirmation on that one and the article. Since it doesn't seem like a lot of people on the site are using that address share function, I do not think it would do any harm to remove it.
At least we know the root cause of why it's doing it to the url's. Now the real question is...could it be getting in the way of indexing those url's ?...one would think not, as from what I've read, google would simply ignore what comes after the #.
Thoughts ?
Appreciate the help.
-
Patrick,
We'd prefer to keep the actual url's private, however I can provide further information to help hopefully allow the community to dissect this further:
- It's an E-commerce website, meaning many facets, filters, and possible duplicate content angles
- It seems many of the static pages (/products main page, /contact,etc) are indexed, however it seems the individual products are mostly not being indexed through the sitemap
- While the url's found in webmaster tools under "index" has also steadily been going down, it definitely doesn't correspond with the lack of pages indexed vs submitted within the sitemap
- We have checked robots.txt, and it is not blocking any important pages. (I also had them allow robots to crawl css and js so google could have full access)
- The individual product pages all have the "addthis" feature, meaning they all have a #. + number sequence added to the url's. However one would think this wouldn't be the cause of this lack of indexation ?
Thanks for your help.
-
Yes, add this is doing this to your url. I hate it, that is one reason why I do not use them.
Here is an article on how to remove them, http://support.addthis.com/customer/portal/articles/1013558-removing-all-hashtags-anchors-weird-codes-from-your-urls
-
Hi there
Could you provide you website's URL? It would help the community take a deeper look - thanks!
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sub domain? Micro site? What's the best solution?
My client currently has two websites to promote their art galleries in different parts of the country. They have bought a new domain (let's call it buyart.com) which they would eventually like to use as an e-commerce platform. They are wondering whether they keep their existing two gallery websites (non e-commerce) separate as they always have been, or somehow combine these into the new domain and have one overarching brand (buyart.com). I've read a bit on subdomains and microsites but am unsure at this stage what the best option would be, and what the pros and cons are. My feeling is to bring it all together under buyart.com so everything is in one place and creates a better user journey for anyone who would like to visit. Thoughts?
Technical SEO | | WhitewallGlasgow0 -
Help! How to Remove Error Code 901: DNS Errors (But to a URL that doesn't exist!)
I have 2 urgent errors saying there are 2 x error code 909's detected. These don't link to any page - but I can tell there is a mistake somewhere - I just don't know what needs changing. http://www.justkeyrings.co.ukhttp/www.justkeyrings.co.uk/printed-promotional-keyrings http://www.justkeyrings.co.ukhttp/www.justkeyrings.co.uk/blank-unassembled-keyrings Could someone help please? screen-shot-2015-08-11-at-13.18.17.png?t=1439292942
Technical SEO | | FullSteamBusiness0 -
Need advice for new site's structure
Hi everyone, I need to update the structure of my site www.chedonna.it Basicly I've two main problems: 1. I've 61.000 index tag (more with no post)2. The category of my site are noindex I thought to fix my problem making the category index and the tag noindex, but I'm not sure if this is the best solution because I've a great number of tag idexed by Google for a long time. Mybe it is correct just to make the category index and linking it from the post and leave the tag index. Could you please let me know what's your opinion? Regards.
Technical SEO | | salvyy0 -
Why am I not showing up in the SERP's or Google Local?
I have been trying to optimise the following site for both Google SERP's and Google Local - Pixel Primate The URL has been around for around 3 years now but they just updated the website and launched it in December 2012. I did the on-page optimisation early in January 2013 and Google seems to have indexed the changes, for the home page at least. One major keyword I am targeting for the home page is 'Web Design Leicester'. I understand that the DA is fairly low (24) so this is something I need to improve. However, I've experienced positive results fairly quickly from just on-page optimisation for other sites I have worked on. The site just doesn't seem to be ranking at all for any keywords. Maybe the industry type is just extremely competitve but I find it very strange to not be visible anywhere in the SERPs. The site does not seem to have any penalties as it ranks for 'Pixel Primate' and all pages appear when doing a site: search. Also what's strange is that I set up the Google Local listing years ago but it doesn't appear anywhere in the local listing, not even when I search for it manually. Any suggestions would be appreciated.
Technical SEO | | CWseo0 -
How do I properly use the canonical tag to avoid negative effect from having identical content on 2 url’s?
To illustrate… I have same website uploaded at 2 locations (url’s). Only the domain extensions are different. www.myexample.com
Technical SEO | | swiftseo
www.myexample.org The benefit is that I may run some promos on one location and not the other to help in product surveys/testing. The website content is 98% identical and I understand this content duplication may cause SEO problems. The domain I wish to use for rankings etc is www.myexample.com 1) How do I go about avoiding seo problem? Do I need to place the canonical tag at www.myexample.org ie 2) Do I also place the exact same tag at the .com location or not necessary there? Is there an alternative or more effective option to resolving the problem?0 -
Is it worth changing our blog post URL's?
We're considering changing the URL's for our blog posts and dropping the date information. Ex. http://spreecommerce.com/blog/2012/07/27/spree-1-1-3-released/ changes to http://spreecommerce.com/blog/spree-1-1-3-released/ Based on what I've learned here the new URL is better for SEO but since these pages already exist do we risk a minor loss of Google juice with 301 redirects? We have a sitemap for the blog posts so I imagine this wouldn't be too hard for Google to learn the new ones.
Technical SEO | | schof0 -
How do i properly combine these two schema's from schema.org
So we're redoing our reviews/testimonials page on our website right now and moving over to the schema.org format as described here: http://schema.org/Review But we would like to combine each of our reviews with a location for which it was reviewed using this: http://schema.org/LocalBusiness What i can't wrap my head around would be the correct syntax? is it just the first block and then the next block? or is there a way of putting the actual physical address within the review page itself? So is this the correct way to do a page full of reviews that are reviewing various physical locations? * <div< span="">itemprop="reviews" itemscope itemtype="http://schema.org/Review"></div<>* <span< span="">itemprop="name">Value purchase</span<> -* by <span< span="">itemprop="author">Lucas</span<>,* <meta< span="">itemprop="datePublished" content="2011-03-25">March 25, 2011</meta<>* <div< span="">itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating"></div<>* <meta< span="">itemprop="worstRating" content = "1"/></meta<>* <span< span="">itemprop="ratingValue">4</span<>/* <span< span="">itemprop="bestRating">5</span<>stars* <span< span="">itemprop="description">Great microwave for the price. It is small and</span<>* fits in my apartment. 1. <div< span="">itemscope itemtype="http://schema.org/LocalBusiness"></div<> 2. # <span< span="">itemprop="name">Beachwalk Beachwear & Giftware</span<> 3. <span< span="">itemprop="description"> A superb collection of fine gifts and clothing</span<> 4. to accent your stay in Mexico Beach. 5. <div< span="">itemprop="address" itemscope itemtype="http://schema.org/PostalAddress"></div<> 6. <span< span="">itemprop="streetAddress">3102 Highway 98</span<> 7. <span< span="">itemprop="addressLocality">Mexico Beach</span<>, 8. <span< span="">itemprop="addressRegion">FL</span<> 10. Phone: <span< span="">itemprop="telephone">850-648-4200</span<> <div< span="">itemprop="reviews" itemscope itemtype="http://schema.org/Review"></div<>* <span< span="">itemprop="name">Value purchase</span<> -* by <span< span="">itemprop="author">Lucas</span<>,* <meta< span="">itemprop="datePublished" content="2011-03-25">March 25, 2011</meta<>* <div< span="">itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating"></div<>* <meta< span="">itemprop="worstRating" content = "1"/></meta<>* <span< span="">itemprop="ratingValue">4</span<>/* <span< span="">itemprop="bestRating">5</span<>stars* <span< span="">itemprop="description">Great microwave for the price. It is small and</span<>* fits in my apartment. <div< span="">itemscope itemtype="http://schema.org/LocalBusiness"></div<> <span< span="">itemprop="name">Beachwalk Beachwear & Giftware</span<> <span< span="">itemprop="description"> A superb collection of fine gifts and clothing</span<> to accent your stay in Mexico Beach. <div< span="">itemprop="address" itemscope itemtype="http://schema.org/PostalAddress"></div<> <span< span="">itemprop="streetAddress">3102 Highway 98</span<> <span< span="">itemprop="addressLocality">Mexico Beach</span<>, <span< span="">itemprop="addressRegion">FL</span<> Phone: <span< span="">itemprop="telephone">850-648-4200</span<>
Technical SEO | | adriandg0 -
Duplicate content and URL's
Hi Guys, Hope you are all well. Just a quick question which you will find nice and easy 🙂 I am just about to work through duplicate content pages and URL changes. Firstly, With the duplicate content issue i am finding the seo friendly URL i would normally direct to in some cases has less links, authority and root domain to it than some of the unseo friendly URL's. will this harm me if i still 301 redirect them to the seo friendly URL. Also, With the url changed it is going to be a huge job to change all the url so they are friendly and the CMS system is poor. Is there a better way of doing this? It has been suggested that we create a new webpage with a friendly URL and redirect all the pages to that. Will this lose all the weight as it will be a brand new page? Thank you for your help guys your legends!! Cheers Wayne
Technical SEO | | wazza19850