Home Pages of Several Websites are disappearing / reappearing in Google Index
-
Hi,
I periodically use the Google site command to confirm that our client's websites are fully indexed.
Over the past few months I have noticed a very strange phenomenon which is happening for a small subset of our client's websites... basically the home page keeps disappearing and reappearing in the Google index every few days. This is isolated to a few of our client's websites and I have also noticed that it is happening for some of our client's competitor's websites (over which we have absolutely no control).
In the past I have been led to believe that the absence of the home page in the index could imply a penalty of some sort. This does not seem to be the case since these sites continue to rank the same in various Google searches regardless of whether or not the home page is listed in the index.
Below are some examples of sites of our clients where the home page is currently not indexed - although they may be indexed by the time you read this and try it yourself. Note that most of our clients are in Canada.
My questions are:
1. has anyone else experienced/noticed this?
2. any thoughts on whether this could imply some sort of penalty? or could it just be a bug in Google?
3. does Google offer a way to report stuff like this?
Note that we have been building websites for over 10 years so we have long been aware of issues like www vs. non-www, canonicalization, and meta content="noindex" (been there done that in 2005). I could be wrong but I do not believe that the site would keep disappearing and reappearing if something like this was the issue. Please feel free to scrutinize the home pages to see if I have overlooked something obvious - I AM getting old.
site:dietrichlaw.ca - this site has continually ranked in the top 3 for [kitchener personal injury lawyers] for many years.
site:burntucker.com - since we took over this site last year it has moved up to page 1 for [ottawa personal injury lawyers]
site:bolandhowe.com - #1 for [aurora personal injury lawyers]
site:imranlaw.ca - continually ranked in the top 3 for [mississauga immigration lawyers].
site:canadaenergy.ca - ranks #3 for [ontario hydro plans]
Thanks in advance!
Jim Donovan, President
-
I just took the first domain you gave me I tested them on two tools you lack canonical's on all but the homepage for all three and all three failed the https://varvy.com test
- imranlaw.ca
- dietrichlaw.ca
- canadaenergy.ca
- burntucker.com past the Varvy test but has only one canonical https://cl.ly/hPdN https://cl.ly/hPoe
- bolandhowe.com is the probably the most affected it has way too many 200 code URLs canonical's pointing to the HTTPS however they should be using a 301 redirect See search engine land post below & these photos https://cl.ly/hPyM & https://cl.ly/hPUj
Preform a search and replace see: https://cl.ly/hPe6
- https://searchenginewatch.com/sew/how-to/2291162/seo-audit-findings-4-hidden-technical-problems-that-can-send-dangerous-signals-to-search-engines
- https://searchenginewatch.com/sew/how-to/2300520/technical-seo-for-nontechnical-people
I took the domainIn number three above and ran it through screaming frog I found no canonical's for all but one URL. Take a look at what most of the URLs appear like.
In addition found that you have a redirect chain photos below they should go straight to HTTPS://www.canadaenergy.ca
I would utilize HSTS as well this will help considerably. And adding canonical's
https://cl.ly/hPJd to https://cl.ly/hPr1 to https://cl.ly/hPyj
Domain number two
the same situation you have one canonical URL homepage nothing else has a canonical
domain number one imranlaw.ca same situation see below no canonical except for the homepage
| Address | http://www.imranlaw.ca/ |
| URL Encoded Address | http://www.imranlaw.ca/ |
| Status Code | 200 |
| Status | OK |
| Content | text/html; charset=ISO-8859-1 |
| Size | 13160 |
| Title 1 | Mississauga Immigration Lawyer & Canadian Citizenship Attorney |
| Meta Description 1 | Imran Khan - Canada Immigration lawyer and Canadian Citizenship attorney Mississauga Immigration Lawyer |
| H1-1 | Canadian Immigration & Naturalization Lawyer |
| H2-1 | Imran Khan Law Office offers Legal Services in Immigration Law and Real Estate Law Matters. |
| Meta Robots 1 | index,follow |
| Canonical Link Element 1 | http://www.imranlaw.ca/ |
| Word Count | 275 |
| Level | 1 |
| Inlinks | 28 |
| Outlinks |19
|
| Address | http://www.imranlaw.ca/contact |
| URL Encoded Address | http://www.imranlaw.ca/contact |
| Status Code | 200 |
| Status | OK |
| Content | text/html; charset=ISO-8859-1 |
| Size | 14503 |
| Title 1 | Mississauga Immigration Lawyer - Contact |
| Meta Description 1 | Imran Khan - Canada Immigration lawyer and Canadian Citizenship attorney Contact Imran Khan |
| H1-1 | Contact Imran Khan Law Office |
| Meta Robots 1 | index,follow |
| Word Count | 276 |
| Level | 2 |
| Inlinks | 28 |
| Outlinks | 17 |A few domains the ones above which are listed below as well fail to be able to be seen by a synthetic Googlebot. Are you running them all on the same server?
You have some domains and in .com and others that end in .ca if you are looking in Google.ca and have geo-targeted the .com domains to Canada you should see them there. However if you're looking in Google.com obviously you cannot geo-target .CA domains to the United States therefore they would not show up in .com unless very rarely.
Deep crawl and screaming frog are going to be a best friends on this one. Please let me know if I can be of more help
here are my findings using a basic tool
and put it into https://varvy.com
The results were
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
HTTP headers
Page headers when accessed as Googlebot.
Headers:
pages could not be found
https://varvy.com/hierarchyandlinks.html
Same thing for imranlaw.ca
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
For canadaenergy.ca
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
Amount of links
Amount of links not excessive.
0 links found on page.
Guideline states: 'Limit the number of links on a page to a reasonable number (a few thousand at most).'
Considering the amount of links on a page
**I wouldUse a tool like deepcrawl.com or screamingfrog.co.uk/seospider **
two determined exactly what is wrong with all three Domains which failed a very basic test of being able to be detected by Googlebot.
Hope this helps,
Tom
-
Hi Jim,
If analytics confirms that traffic is still landing on the homepage, then I think this is just Google reporting different pages when you perform a site: - It certainly doesn't sound like a penalty of any sort.
It is worth noting that Google did confirm some time back that site: doesn't bring back every page every time and is best used as a guide. Does the sitemap in Search Console show a healthy number of indexed links?
If you want a discussion on this, then it would be worthwhile also posting over at the Websearch Help Forums at Google and see what others have to say about it.
I hope this helps a little.
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Get List Of All Indexed Google Pages
I know how to run site:domain.com but I am looking for software that will put these results into a list and return server status (200, 404, etc). Anyone have any tips?
Technical SEO | | InfinityTechnologySolutions0 -
Google no longer is promoting my website!
My traffic just fell off the cliff and I can not figure out what has happened! Help!
Technical SEO | | PatriciaBeth0 -
Skip indexing the search pages
Hi, I want all such search pages skipped from indexing www.somesite.com/search/node/ So i have this in robots.txt (Disallow: /search/) Now any posts that start with search are being blocked and in Google i see this message A description for this result is not available because of this site's robots.txt – learn more. How can i handle this and also how can i find all URL's that Google is blocking from showing Thanks
Technical SEO | | mtthompsons0 -
Google Indexed Only 1 Page
Hi, I'm new and hope this forum can help me. I have recently resubmit my sitemap and Google only Indexed 1 Page. I can still see many of my old indexed pages in the SERP's? I have upgraded my template and graded all my pages to A's on SEOmoz, I have solid backlinks and have been building them over time. I have redirected all my 404 errors in .htaccess and removed /index.php from my url's. I have never done this before but my website runs perfect and all my pages redirect as I hoped. My site: www.FunerallCoverFinder.co.za How do I figure out what the problem is? Thanks in Advance!
Technical SEO | | Klement690 -
Do Collections in Shopify create Duplicate Pages according to Google/Bing/Yahoo?
I'm using the e-commerce platform Shopify to host an e-store. We've put our products into different collections. Shopify automatically creates different URL paths to a product in multiple collections. I'm worried that the same product listed in different collections is soon as different pages, and therefore duplicate content by Google/Bing/Yahoo. Would love to get your opinion on this concern! Thanks! Matthew
Technical SEO | | HappinessDigital0 -
Google is indexing proxy (mirror) site.
We moved the site to a new hosting. Previously the site used Godaddy Windows Hosting with white domain masking. After moving the site we just mirrored the site. We have to use mirrored domain for PPC campaigns because it mirrored site contains true BRAND name and there is better conversion with that domain plus all trade marked keywords are approved for mirrored domain. Robots.txt User-agent: * Host: www.hermitagejewelers.com Disallow: /Bin Disallow: /css www.hermitagejewelers.com is the main domain. Mirror site is www.ermitagejewelers.com (Without the "H" at the beginning) Most of the keywords are now picked up by mirror site. I have not noticed any major changes in ranking except that it ranks for mirror site. We updated the sitemap. Website is designed very poorly (not by us). Also, we submitted the change address request for ermitagejewelers to hermitagejewelers in webmasters. Please let me know any advice to fix that problem. Thank you.
Technical SEO | | MaxRuso1 -
Redesign existing websites / worried about urls / mapping
Hi Guys, While redesigning existing websites that will have page name changes such as: example.com/products to be called example.com/solutions example.com/about-us to be called example.com/about should I 301 the old url to the new url. In the past I have not done this & I'm just wondering from an SEO point of view how bad is this? (On a scale of 1 to 10 how bad is this not 301ing urls, 10 being really bad & 1 being fine), Thanks.
Technical SEO | | Socialdude0 -
Home page URL disappears in Google after switching to WordPress
It was a 10 page static HTML page website. 3 year old, PR2. Monday night, copied a WordPress from somewhere to this website's public_html folder and activate it. The home page was "index.html" before switching to WordPress. Now this html file (index.html) has been deleted, so WordPress' Home page can work. All other 9 static html pages are still there in Google index. Just notice it today that the home page URL disappears in Google completely. Why? All other 9 static html pages' URL are still in Google. robots.txt is Allow: / What may have gone wrong to remove the home domain URL from Google index? Thank you for your help!
Technical SEO | | johnzhel0