Google indexing is slowing down?
-
I have up to 20 million unique pages, and so far I've only submitted about 30k of them on my sitemap.
We had a few load related errors during googles initial visits, and it thought some were duplicates, but we fixed all that. We haven't gotten a crawl related error for 2 weeks now.
Google appears to be indexing fewer and fewer urls every time it visits. Any ideas why? I am not sure how to get all our pages indexed if its going to operate like this... love some help thanks!
-
Try getting links for important pages. As for google crawling and indexing: i have websites that contain 10k of URL's on which i know, half of that is not relevant for search at all. Google does it completely automaticly and cuts out the most duplicate or non-relevant pages in comparison to what i offer in my sitemap and brings 4k of pages or so in actual search.
It's really no issue. You cant tell me you got 20 million of unique (user) generated content.
-
Share your URL for better responses.
Have you submitted a complete XML sitemap through Google Search Console? And are all the pages free of a no-index meta tag?
20 million unique pages can only be created in some automated fashion based on a much smaller amount of original content, so Google may perceive these pages to be spammy and not worth indexing.
-
Ryan,
G doesn't crawl/index every url it encounters. Its decision to do so is based on it perceived value of the the page/the page the link is one. You can increase that value/PR/PA by sculpting your internal links, revised silo-ing of your content and/or acquiring backlinks to pages deeper within your site.
If you're a brand new site with 20M pages, what reason have you give google to crawl all those pages. Just submitting them or listing them on a sitemap isn't enough of a reason. I mean, 20M pages! google's like damn dude, what's up with all that?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No Longer Indexed in Google (Says Redirected)
Just recently my page, http:/www./waikoloavacationrentals.com/mauna-lani-terrace, was no longer indexed by google. The sub pages from it still are. I have not done anything sketchy with the page. When I went into the google fetch it says that it is redirected. Any ideas what is this all about? Here is what it says for the fetch: Http/1.1 301 moved permanently
Technical SEO | | RobDalton
Server: nginx
Date: Tue, 07 Mar 2017 00:43:26GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Keep-Alive: timeout=20
Location: http://waikoloavacationrentals.com/mauna-lani-terrace <title>301 moved permanently</title> <center> 301 moved permanently </center> <center>nginx</center>0 -
How preproduction website is getting indexed in Google.
Hi team, Can anybody please help me to find how my preproduction website and urls are getting indexed in Google.
Technical SEO | | nlogix0 -
Some of my website urls are not getting indexed while checking (site: domain) in google
Some of my website urls are not getting indexed while checking (site: domain) in google
Technical SEO | | nlogix0 -
Will blocking the Wayback Machine (archive.org) have any impact on Google crawl and indexing/SEO?
Will blocking the Wayback Machine (archive.org) by adding the code they give have any impact on Google crawl and indexing/SEO? Anyone know? Thanks! ~Brett
Technical SEO | | BBuck0 -
Why Google not picking My META Description? Google itself populate the description.. How to control this Search Snippets??
Why Google not picking My META Description? Google itself populate the description.. How to control this Search Snippets??
Technical SEO | | greyniumseo0 -
Penalized by google. How to find out?
Our webpage performs very bad on some keywords relating to one product. At the SeoMoz-ranking page i can se we are number 9 but we have the highest (higher than our competitors) rating in almost every category (at least 25 of 30) on the keyword difficulty report. How do i find out why this is so, or if we have been penalized by google?On other search-engines (yahoo, bing etc) we are number one! And we have the highest pagerank among the competitors...
Technical SEO | | alsvik0 -
Is google all over the place tonight?
Is it me or is google all over the place tonight? Whilst checking my rankings I came across a site with a page authority of 29 and 23 links from 5 domains ranking at number 6 for a competitive keyword! This site came from nowhere and I'm getting different results every time I search! Weird....
Technical SEO | | SamCUK0