Google has discovered a URL but won't index it?
-
Hey all, have a really strange situation I've never encountered before. I launched a new website about 2 months ago. It took an awfully long time to get index, probably 3 weeks. When it did, only the homepage was indexed.
I completed the site, all it's pages, made and submitted a sitemap...all about a month ago. The coverage report shows that Google has discovered the URL's but not indexed them. Weirdly, 3 of the pages ARE indexed, but the rest are not.
So I have 42 URL's in the coverage report listed as "Excluded" and 39 say "Discovered- currently not indexed." When I inspect any of these URL's, it says "this page is not in the index, but not because of an error." They are listed as crawled - currently not indexed or discovered - currently not indexed.
But 3 of them are, and I updated those pages, and now those changes are reflected in Google's index. I have no idea how those 3 made it in while others didn't, or why the crawler came back and indexed the changes but continues to leave the others out.
Has anyone seen this before and know what to do?
-
Good luck!
-
Thanks Will, appreciate the insight. I'm going to get the Bing and Google wordpress plugins on there to see if that helps, build up a few more links and give it some time to wait and see. Thanks!
-
You're not the only person reporting odd indexation happenings here on Q&A (see for example this question). And, just like I found for that question, your site appears to have more pages indexed in Bing than in Google - which at least seems to point to us not having missed something obvious like meta noindex or similar.
I did also read Google saying that they had issues with the site: command (link) but I don't think that can have anything to do with your situation as they say they have now fixed that issue, and I couldn't find any other pages on your site even with non-site: searches (i.e. it does genuinely appear as though those pages are missing from the index).
While I am loathe to point just at links these days, I do wonder if in this case it is just a case of needing some more authority for the whole site before it is seen as big enough and important enough to justify more pages in the index.
-
Thanks, I've actually submitted request to be indexed multiple times over the last 3 weeks to no avail.
-
Hey Daniel. I agree with Chris. I have also noticed slow indexation recently. Might be a pain in the arse, but maybe you should request each page to be indexed individually in Search Console to add them to the high priority queue.
-
Hi Will, thanks for reaching out! No, not yet resolved. Still struggling to figure this out. I sent you a message on Facebook and Linkedin- would love to connect and try to get this figured out!
-
We keep adding blog posts almost every day, still not getting in the index for some reason. Discovered, yes. Crawled, yes. But not indexed, and no errors or anything.
-
Hi Daniel. Did you get this resolved / did it resolve itself? I'd happily take a look if you'd like if not - just let me know the URL.
-
My advice is, start listing more reviews! It will be picked up by google automaticly. You gotta be a bit more patient. New websites take awefully alot of time to be indexed.
I had a domain of 10+ years of age, replaced it's website, within one day completely reindexed. I have new domains, they can take up to weeks or even a month to be indexed. It's normal.
-
I actually got a quality link 2 weeks ago, but the blog post the link was published in still isn't indexed by Google either. The rest of his site is, just not his newest article for some reason, and it's 2 weeks old now. Another mystery...
-
It's a review website, and only 3 of my 24 reviews are indexed. All are discovered, most even crawled, but only those three in the index. And when I updated them, the search listing in Google results was updated within a few days. So they came back, are aware of the changes, but just not adding the others to the index.
And there are no affiliate links on this site at all. No spam, no links to spam, and I've attached a blog with 500+ word well written articles (about 20 so far) and none of the blog posts are indexed either.
I've never seen anything like this. The content is good, but almost none of it is getting indexed for some reason, despite being discovered and crawled.
-
Get quality links.
-
It's a new domain, no previous ownership, and no issues detected in search console for manual actions or security. There's no robots, noindex or any of that going on. They just won't index a bunch of the pages for some reason and it's very odd.
-
Perhaps the content on those 42 pages or so is alot copy content based? Or pages that really dont matter to be up in search?
-
I'd hold off worrying about it for now. I've heard many people talk about slow indexation lately. In the mean time, aside from the obvious check-for- nofollow- noindex-robots.txt suggestions, have you looked into the history of this domain? By chance was it penalized before you bought it?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
All urls seem to exist (no 404 errors) but they don't.
Hello I am doing a SEO auditing for a website which only has a few pages. I have no cPanel credentials, no FTP no Wordpress admin account, just watching it from the outside. The site works, the Moz crawler didn't report any problem, I can reach every page from the menu. The problem is that - except for the few actual pages - no matter what you type after the domain name, you always reach the home page and don't get any 404 error. I.E. Http://domain.com/oiuxyxyzbpoyob/ (there is no such a page, but i don't get 404 error, the home is displayed and the url in the browser remains Http://domain.com/oiubpoyob/, so it's not a 301 redirect). Http://domain.com/WhatEverYouType/ (same) Could this be an important SEO issue (i.e. resulting in infinite amount of duplicate content pages )? Do you think I should require the owner to prevent this from happening? Should I look into the .htaccess file to fix it ? Thank you Mozers!
Intermediate & Advanced SEO | | DoMiSoL0 -
What is Google supposed to return when you submit an image URL into Fetch as Google? Is a few lines of readable text followed by lots of unreadable text normal?
I am seeing something like this (Is this normal?): HTTP/1.1 200 OK
Intermediate & Advanced SEO | | Autoboof
Server: nginx
Content-Type: image/jpeg
X-Content-Type-Options: nosniff
Last-Modified: Fri, 13 Nov 2015 15:23:04 GMT
Cache-Control: max-age=1209600
Expires: Fri, 27 Nov 2015 15:23:55 GMT
X-Request-ID: v-8dd8519e-8a1a-11e5-a595-12313d18b975
X-AH-Environment: prod
Content-Length: 25505
Accept-Ranges: bytes
Date: Fri, 13 Nov 2015 15:24:11 GMT
X-Varnish: 863978362 863966195
Age: 16
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT
X-Cache-Hits: 1 ����•JFIF••••��;CREATOR: gd-jpeg v1.0 (using IJG JPEG v80), quality = 75
��C•••••••••• •
••
••••••••• $.' ",#(7),01444'9=82<.342��C• ••••
•2!!22222222222222222222222222222222222222222222222222��•••••v••"••••••��••••••••••••••••
•���•••••••••••••}•••••••!1A••Qa•"q•2���•#B��•R��$3br�
••••%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz���������������������������������������������������������������������������•••••••••••••••••••
•���••••••••••••••w••••••!1••AQ•aq•"2�••B���� #3R�•br�0 -
How to switch from URL based navigation to Ajax, 1000's of URLs gone
Hi everyone, We have thousands of urls generated by numerous products filters on our ecommerce site, eg./category1/category11/brand/color-red/size-xl+xxl/price-cheap/in-stock/. We are thinking of moving these filters to ajax in order to offer a better user experience and get rid of these useless urls. In your opinion, what is the best way to deal with this huge move ? leave the existing URLs respond as before : as they will disappear from our sitemap (they won't be linked anymore), I imagine robots will someday consider them as obsolete ? redirect permanent (301) to the closest existing url mark them as gone (4xx) I'd vote for option 2. Bots will suddenly see thousands of 301, but this is reflecting what is really happening, right ? Do you think this could result in some penalty ? Thank you very much for your help. Jeremy
Intermediate & Advanced SEO | | JeremyICC0 -
Google Indexed Old Backups Help!
I have the bad habit of renaming a html page sitting on my server, before uploading a new version. I usually do this after a major change. So after the upload, on my server would be "product.html" as well as "product050714".html. I just stumbled on the fact G has been indexing these backups. Can I just delete them and produce a 404?
Intermediate & Advanced SEO | | alrockn0 -
Web developer won't 301 redirect to my new website....what can I do?
I want to come away from our third party web developer and use a new website with different web developers, however the web developer wont 301 redirect the old url's to the new ones. Is he required to do this by law? Is there away of doing the 301 redirects without him? Any advice would be much appreciated. Many thanks,Tom
Intermediate & Advanced SEO | | CoGri0 -
Why the archive sub pages are still indexed by Google?
Why the archive sub pages are still indexed by Google? I am using the WordPress SEO by Yoast, and selected the needed option to get these pages no-index in order to avoid the duplicate content.
Intermediate & Advanced SEO | | MichaelNewman1 -
What NAP format do I use if the USPS can't even find my client's address?
My client has a site already listed on Google+Local under "5208 N 1st St". He has some other NAPs, e.g., YellowPages, under "5208 N First Street". The USPS finds neither of these, nor any variation that I can possibly think of! Which is better? Do I just take the one that Google has accepted and make all the others like it as best I can? And doesn't it matter that the USPS doesn't even recognize the thing? Or no? Local SEO wizards, thanks in advance for your guidance!
Intermediate & Advanced SEO | | rayvensoft0 -
Google suddenly indexing and displaying URLs that haven't existed for years?
We recently noticed google is showing approx 23,000 indexed .jsp urls for our site. These are ancient pages that haven't existed in years and have long been 301 redirected to valid urls. I'm talking 6 years. Checking the serps the other day (and our current SEOMoz pro campaign), I see that a few of these urls are now replacing our correct ones in the serps for important, competitive phrases. What the heck is going on here? Is Google suddenly ignoring rewrite rules and redirects? Here's an example of the rewrite rules that we've used for 6+ years: RewriteRule ^(.*)/xref_interlux_antifoulingoutboards&keels.jsp$ $1/userportal/search_subCategory.do?categoryName=Bottom%20Paint&categoryId=35&refine=1&page=GRID [R=301] Now, this 'bottom paint' url has been incredibly stable in the serps for over a half decade. All of a sudden, a google search for 'bottom paint' (no quotes) brings up the jsp page at position 2-3. This is just one example of something very bizarre happening. Has anyone else had something similar happen lately? Thank You <colgroup><col width="64"></colgroup>
Intermediate & Advanced SEO | | jamestown
| RewriteRule ^(.*)/xref_interlux_antifoulingoutboards&keels.jsp$ $1/userportal/search_subCategory.do?categoryName=Bottom%20Paint&categoryId=35&refine=1&page=GRID [R=301] |0