Google has discovered a URL but won't index it?
-
Hey all, have a really strange situation I've never encountered before. I launched a new website about 2 months ago. It took an awfully long time to get index, probably 3 weeks. When it did, only the homepage was indexed.
I completed the site, all it's pages, made and submitted a sitemap...all about a month ago. The coverage report shows that Google has discovered the URL's but not indexed them. Weirdly, 3 of the pages ARE indexed, but the rest are not.
So I have 42 URL's in the coverage report listed as "Excluded" and 39 say "Discovered- currently not indexed." When I inspect any of these URL's, it says "this page is not in the index, but not because of an error." They are listed as crawled - currently not indexed or discovered - currently not indexed.
But 3 of them are, and I updated those pages, and now those changes are reflected in Google's index. I have no idea how those 3 made it in while others didn't, or why the crawler came back and indexed the changes but continues to leave the others out.
Has anyone seen this before and know what to do?
-
Good luck!
-
Thanks Will, appreciate the insight. I'm going to get the Bing and Google wordpress plugins on there to see if that helps, build up a few more links and give it some time to wait and see. Thanks!
-
You're not the only person reporting odd indexation happenings here on Q&A (see for example this question). And, just like I found for that question, your site appears to have more pages indexed in Bing than in Google - which at least seems to point to us not having missed something obvious like meta noindex or similar.
I did also read Google saying that they had issues with the site: command (link) but I don't think that can have anything to do with your situation as they say they have now fixed that issue, and I couldn't find any other pages on your site even with non-site: searches (i.e. it does genuinely appear as though those pages are missing from the index).
While I am loathe to point just at links these days, I do wonder if in this case it is just a case of needing some more authority for the whole site before it is seen as big enough and important enough to justify more pages in the index.
-
Thanks, I've actually submitted request to be indexed multiple times over the last 3 weeks to no avail.
-
Hey Daniel. I agree with Chris. I have also noticed slow indexation recently. Might be a pain in the arse, but maybe you should request each page to be indexed individually in Search Console to add them to the high priority queue.
-
Hi Will, thanks for reaching out! No, not yet resolved. Still struggling to figure this out. I sent you a message on Facebook and Linkedin- would love to connect and try to get this figured out!
-
We keep adding blog posts almost every day, still not getting in the index for some reason. Discovered, yes. Crawled, yes. But not indexed, and no errors or anything.
-
Hi Daniel. Did you get this resolved / did it resolve itself? I'd happily take a look if you'd like if not - just let me know the URL.
-
My advice is, start listing more reviews! It will be picked up by google automaticly. You gotta be a bit more patient. New websites take awefully alot of time to be indexed.
I had a domain of 10+ years of age, replaced it's website, within one day completely reindexed. I have new domains, they can take up to weeks or even a month to be indexed. It's normal.
-
I actually got a quality link 2 weeks ago, but the blog post the link was published in still isn't indexed by Google either. The rest of his site is, just not his newest article for some reason, and it's 2 weeks old now. Another mystery...
-
It's a review website, and only 3 of my 24 reviews are indexed. All are discovered, most even crawled, but only those three in the index. And when I updated them, the search listing in Google results was updated within a few days. So they came back, are aware of the changes, but just not adding the others to the index.
And there are no affiliate links on this site at all. No spam, no links to spam, and I've attached a blog with 500+ word well written articles (about 20 so far) and none of the blog posts are indexed either.
I've never seen anything like this. The content is good, but almost none of it is getting indexed for some reason, despite being discovered and crawled.
-
Get quality links.
-
It's a new domain, no previous ownership, and no issues detected in search console for manual actions or security. There's no robots, noindex or any of that going on. They just won't index a bunch of the pages for some reason and it's very odd.
-
Perhaps the content on those 42 pages or so is alot copy content based? Or pages that really dont matter to be up in search?
-
I'd hold off worrying about it for now. I've heard many people talk about slow indexation lately. In the mean time, aside from the obvious check-for- nofollow- noindex-robots.txt suggestions, have you looked into the history of this domain? By chance was it penalized before you bought it?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google's Knowledge Panel
Hi Moz Community. Has anyone noticed a pattern in the websites that Google pulls in to populate knowledge Panels? For example, for a lot of queries Google keeps pulling data from a specific source over and over again, and the data shown in the Knowledge Panel isn't on the target page. Is it possible that Google simply favors some sites over others and no matter what you do, you'll never make it into the Knowledge box? Thanks.
Intermediate & Advanced SEO | | yaelslater0 -
404 broken URLs coming up in Google
When we do a search for our brand, we are get the following results in google.com.au (see image attachment). As outlined in red, there are listings in Google that result in 404 Page Not Found URLs. What can we do to enable google to do a recrawl or to ensure that these broken URLs are no longer listed in Google? Thanks for your help here! sBqpvtj
Intermediate & Advanced SEO | | Gavo0 -
Google Indexing of Images
Our site is experiencing an issue with indexation of images. The site is real estate oriented. It has 238 listings with about 1190 images. The site submits two version (different sizes) of each image to Google, so there are about 2,400 images. Only several hundred are indexed. Can adding Microdata improve the indexation of the images? Our site map is submitting images that are on no-index listing pages to Google. As a result more than 2000 images have been submitted but only a few hundred have been indexed. How should the site map deal with images that reside on no-index pages? Do images that are part of pages that are set up as "no-index" need a special "no-index" label or special treatment? My concern is that so many images that not indexed could be a red flag showing poor quality content to Google. Is it worth investing in correcting this issue, or will correcting it result in little to no improvement in SEO? Thanks, Alan
Intermediate & Advanced SEO | | Kingalan10 -
Google Frequently Indexing - Good or Bad?
Hi, My website is only 4 months old and receives about 40 to 50 organic visits every day. It currently has about 100 pages out of which only 3-4 rank in the top 10 for the target KWs. I usually try to publish, at least 1 article a day but sometimes certain articles are more than 2000 words long with a few of infographics and hence takes way more time (maybe even 3 days to publish one) Only over the last week, I am observing that every time i am publishing a page (usually daily) google is indexing them the same day. This I have heard happens for moderately big sites but my site is really small at this stage. Note: For the first 80 pages, I used to "fetch as googlebot" in webmasters as otherwise my site would be crawled once in 2 weeks but over the last 3-4 weeks, i rely on googles scheduled visits. Is this a good or bad sign? I would like to assume its good because of my engagement. Though for only organic visits, my Gogle Analytics bounce rate is 65% in analytics out of the remaining 35%, the avg time on site >7 mins. That means if someone sticks to my site, they consume a lot of my content. Also, since analytics' bounce rate is not same as the search bounce (back button) I would like to consider that the bounce is actually lesser than that.
Intermediate & Advanced SEO | | dwautism0 -
Google showing high volume of URLs blocked by robots.txt in in index-should we be concerned?
if we search site:domain.com vs www.domain.com, We see: 130,000 vs 15,000 results. When reviewing the site:domain.com results, we're finding that the majority of the URLs showing are blocked by robots.txt. They are subdomains that we use as production environments (and contain similar content as the rest of our site). And, we also find the message "In order to show you the most relevant results, we have omitted some entries very similar to the 541 already displayed." SEER Interactive mentions that this is one way to gauge a Panda penalty: http://www.seerinteractive.com/blog/100-panda-recovery-what-we-learned-to-identify-issues-get-your-traffic-back We were hit by Panda some time back--is this an issue we should address? Should we unblock the subdomains and add noindex, follow?
Intermediate & Advanced SEO | | nicole.healthline0 -
What if you can't navigate naturally to your canonicalized URL?
Assume this situation for a second... Let's say you place a rel= canonical tag on a page and point to the original/authentic URL. Now, let's say that that original/authentic URL is also populated into your XML sitemap... So, here's my question... Since you can't actually navigate to that original/authentic URL (it still loads with a 200, it's just not actually linkded to from within the site itself), does that create an issue for search engines? Last consideration... The bots can still access those pages via the canonical tag and the XML sitemap, it's just that the user wouldn't be able to access those original/authentic pages in their natural site navigation. Thanks, Rodrigo
Intermediate & Advanced SEO | | AlgoFreaks0 -
Can you see the 'indexing rules' that are in place for your own site?
By 'index rules' I mean the stipulations that constitute whether or not a given page will be indexed. If you can see them - how?
Intermediate & Advanced SEO | | Visually0 -
Google indexing flash content
Hi Would googles indexing of flash content count towards page content? for example I have over 7000 flash files, with 1 unique flash file per page followed by a short 2 paragraph snippet, would google count the flash as content towards the overall page? Because at the moment I've x-tagged the roberts with noindex, nofollow and no archive to prevent them from appearing in the search engines. I'm just wondering if the google bot visits and accesses the flash file it'll get the x-tag noindex, nofollow and then stop processing. I think this may be why the panda update also had an effect. thanks
Intermediate & Advanced SEO | | Flapjack0