Google is not indexing all URLs
-
My website have company and events profile from 200 countries. So it does have lots of URL. Earlier in August 2014, Google used to crawl 90% of URLs we submit. Thing goes wrong when we shifted from http to https. We lost traffic. But we are gaining it slowly. Main concern is that, It still does not indexed all submitted URLs. It have crawled merely 8% of all URLs submitted. site address is businessvibes.com
Any help would be appreciated.
-
Hi,
It sounds like something may be getting blocked or not crawled correctly, possibly because of the move from http --> https. I assume this is handled by the .htaccess file?
I would also check to see if there are https page issues on the site. If it's bit, you may have to do some digging around, but you can see this by using Google Chrome --> right click on the page --> inspect element --> console. This will show you the errors on the page.
-Andy
-
Hi Mossa,
Thanks for advice. We have already 2 millions of pages which are crawled. and good number of page views and session/day.
I will go with your point
-
You can see, almost all company and event profile have internal linking for other profile and events
-
We are doing that but we have pages in billion which is literally time taking as it have countries and events from 200 countries
-
It is very difficult to get backlining for billions of URLs, so branding and other activities are there to take care of that.
-
http to https have 301 redirection and all sitemap have been updated.
-
-
Before I start, why your website is using “_blank” even on internal links? This does not make any sense to me and I was literally pissed after three clicks.
Anyways, if you are interested in increasing the indexed pages here are the few things you could do:
- Improve internal linking of the website. Especially when possible link pages that are not indexed to the pages that are indexed.
- Get some social media shares (tweets) to pages that are not indexed.
- Build some quality links to the pages that are not indexed in Google.
Also, make sure that the 301 redirections from http tp https versions are correct and working and xml sitemap is updated.
If you do that I am sure that Google will not only index your pages but will also rank better from the desired key phrases.
Hope this helps!
-
Hi Martijn,
We have various sitemap
www.businessvibes.com/sitemap.companyprofile5.xml
www.businessvibes.com/sitemap.companyprofile6.xml
www.businessvibes.com/sitemap.companyprofile7.xml
And many More
-
Hi,
Do you have sitemaps that could provide some insights into your indexation rate for the data that you've submitted?
M.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How often does the indexing take place?
Hi there, I am currently looking at one of my campaigns and have noticed that since i last checked on the 31st Jan there have been no change in analytics despite me having done work on the campaign itself. I saw in the external links section that it said "38 down since last index" and I was just wondering how often the index takes place. Any help with this would be greatly appreciated! Many thanks.
Reporting & Analytics | | VoodooCreativeLtd0 -
Drop in indexation but increase in organic traffic
We've had a puzzling drop in indexed pages on our ecommerce website. My crawl returns just over 25k items. Until 19/6 we had about 23-24k indexed. Then we experienced a sudden drop from 19/6 to 26/6: from 23,400 to 18,999, losing 4.4k pages from one week to the next. At the same time, our organic traffic has not decreased, it actually increased, however, it's only been a couple of weeks so that may be coincidence. A few things that have happened during the past few weeks: 31/5: we implemented pagination on category pages to avoid issues with duplicate content - could it be that this led to a decrease in indexed pages 3 weeks later? However, I can only find about 1.5k pages in my crawl that are page 2+ 18-19/6: we had some website outages over the weekend; as a B2B business, we don't get much traffic over the weekend, so I can't see an impact to traffic. However, the following week, indexation dropped by another 250 (then stayed the same this past week), so I don't think this was a factor. 21/6: we retired another website and migrated it to our main website. However, all pages were redirected to existing pages so no new pages were created for the migration. This doesn't really explain a decrease in indexation, but may account for some of the increase in organic traffic; however not all as the retired website hardly got any organic traffic. So, should we be worried? As our website is quite large, it would probably be quite difficult to pin point exactly which pages dropped off the index, but a loss of 19% of pages is quite significant. Then again, it doesn't appear to have negatively impacted organic traffic... Have you got any suggestions for what I should be looking at to find out what happened? Should I be worried at this point? I will definitely continue to have an eye on how our organic traffic (and indexation) develops but I am not sure if there is anything I can do at this point. I'd appreciate your advice on this, to make sure I am not missing something blindingly obvious. Thanks! RmWaNib JJm4tC3
Reporting & Analytics | | ViviCa10 -
Google Analytics View Filters
Using the same GA property, I would like to set up three filtered views: 1. Tracking across one subdomain and one primary domain (example: shop.example.com & example.com) 2. Track only primary domain (example.com) 3. Track only subdomain (shop.example.com) Can this be achieved by using view filters? If so, how do they need to be set? Also, according to this article: https://moz.com/blog/cross-domain-subdomain-tracking-in-google-analytics, with cross domain tracking, I need to ignore self-referrals, which can only be done at the property level. If set up to ignore example.com referrals, will this cause problems with filter 2 and 3?
Reporting & Analytics | | Evan340 -
No-indexed pages are still showing up as landing pages in Google Analytics
Hello, My website is a local job board. I de-indexed all of the job listing pages on my site (anything that starts with http://www.localwisejobs.com/job/). When I search site:localwisejobs.com/job/, nothing shows up. So I think that means the pages are not being indexed. When I look in Google Analytics at Acquisition > Search Engine Optimization > Landing Pages, none of the job listing pages show up. But when I look at Acquisition > Channels > Organic and then click Landing Page as the primary dimension, the /job pages show up in there. Why am I seeing this discrepency in Organic Landing pages? And why would the /job pages be showing up as landing pages even though they aren't indexed?
Reporting & Analytics | | mztobias0 -
Testing IP Exclusion Filters in Google Analytics
(I hope I haven't missed something simple 😉 I have add IP exclusion filters in Google Analytics. What I haven't been able to determine is how to test to see if it is working. I'd imagine there would be a dip in traffic but if a site is low volume it might not even register. Thanks!
Reporting & Analytics | | AUDigitalMarketing0 -
PDFs and indexing
Hello and good morning. I work for a paint manufacturing company in the UK on their seo campaigns across a couple of websites, this is my question. as paint and chemicals require data and tech sheets by law, available to be downloadable for said product, should these be included in the sitemap, we auto generate our sitemaps which they include these files, with low priorities and never change in terms of name etc. they basically have a name of say 092847.pdf for example which cannot be changed, but from an seo view this doesn't mean a thing? so theres my question should they be included and would they carry any value?
Reporting & Analytics | | TeamacPaints0 -
What to Index?
We are using wordpress and seo plugin from yoast. We have set indexed all the posts, but not categories or tags in order to avoid a duplicate as those categories contain the posts. My question is, is it possible to set for index rather the categories and then set posts non-index? Would be then posts in categories still index?
Reporting & Analytics | | VillasDiani0 -
Google Analytics Goals are not recording
Hello! I've set up Goals within Google Analytics and they aren't registering. I'm wondering if someone can review what I've set up to see if I've entered something wrong. I'm trying to track enrollments. Here's the site info: Enroll URL: http://www.careplusdentalplans.com/individuals/enroll-careplus/ Upon completion, the user goes to a Thank You page and our CMS generates a trailing URL parameter. For example: http://www.careplusdentalplans.com/index.php/individuals/enroll-careplus-thank-you/#123456789 My Goal info in GA: Goal URL: /individuals/enroll-careplus-thank-you/
Reporting & Analytics | | SmileMoreSEO
Match Type: Head Match (to disregard trailing URL parameter)
Goal Funnel: Step 1: /individuals/enroll-careplus/ I'm curious if the problem lies in either: The URL: Notice how the URL adds /index.php/ on the thank you page. In that case, should I enter the full URL in the Goal instead of the truncated URL? The Funnel: Is it necessary to show the initial Enroll page or if I'll see that in the Flow Visualization regardless? Thanks in advance for helping me out. Erik0