Home Pages of Several Websites are disappearing / reappearing in Google Index
-
Hi,
I periodically use the Google site command to confirm that our client's websites are fully indexed.
Over the past few months I have noticed a very strange phenomenon which is happening for a small subset of our client's websites... basically the home page keeps disappearing and reappearing in the Google index every few days. This is isolated to a few of our client's websites and I have also noticed that it is happening for some of our client's competitor's websites (over which we have absolutely no control).
In the past I have been led to believe that the absence of the home page in the index could imply a penalty of some sort. This does not seem to be the case since these sites continue to rank the same in various Google searches regardless of whether or not the home page is listed in the index.
Below are some examples of sites of our clients where the home page is currently not indexed - although they may be indexed by the time you read this and try it yourself. Note that most of our clients are in Canada.
My questions are:
1. has anyone else experienced/noticed this?
2. any thoughts on whether this could imply some sort of penalty? or could it just be a bug in Google?
3. does Google offer a way to report stuff like this?
Note that we have been building websites for over 10 years so we have long been aware of issues like www vs. non-www, canonicalization, and meta content="noindex" (been there done that in 2005). I could be wrong but I do not believe that the site would keep disappearing and reappearing if something like this was the issue. Please feel free to scrutinize the home pages to see if I have overlooked something obvious - I AM getting old.
site:dietrichlaw.ca - this site has continually ranked in the top 3 for [kitchener personal injury lawyers] for many years.
site:burntucker.com - since we took over this site last year it has moved up to page 1 for [ottawa personal injury lawyers]
site:bolandhowe.com - #1 for [aurora personal injury lawyers]
site:imranlaw.ca - continually ranked in the top 3 for [mississauga immigration lawyers].
site:canadaenergy.ca - ranks #3 for [ontario hydro plans]
Thanks in advance!
Jim Donovan, President
-
I just took the first domain you gave me I tested them on two tools you lack canonical's on all but the homepage for all three and all three failed the https://varvy.com test
- imranlaw.ca
- dietrichlaw.ca
- canadaenergy.ca
- burntucker.com past the Varvy test but has only one canonical https://cl.ly/hPdN https://cl.ly/hPoe
- bolandhowe.com is the probably the most affected it has way too many 200 code URLs canonical's pointing to the HTTPS however they should be using a 301 redirect See search engine land post below & these photos https://cl.ly/hPyM & https://cl.ly/hPUj
Preform a search and replace see: https://cl.ly/hPe6
- https://searchenginewatch.com/sew/how-to/2291162/seo-audit-findings-4-hidden-technical-problems-that-can-send-dangerous-signals-to-search-engines
- https://searchenginewatch.com/sew/how-to/2300520/technical-seo-for-nontechnical-people
I took the domainIn number three above and ran it through screaming frog I found no canonical's for all but one URL. Take a look at what most of the URLs appear like.
In addition found that you have a redirect chain photos below they should go straight to HTTPS://www.canadaenergy.ca
I would utilize HSTS as well this will help considerably. And adding canonical's
https://cl.ly/hPJd to https://cl.ly/hPr1 to https://cl.ly/hPyj
Domain number two
the same situation you have one canonical URL homepage nothing else has a canonical
domain number one imranlaw.ca same situation see below no canonical except for the homepage
| Address | http://www.imranlaw.ca/ |
| URL Encoded Address | http://www.imranlaw.ca/ |
| Status Code | 200 |
| Status | OK |
| Content | text/html; charset=ISO-8859-1 |
| Size | 13160 |
| Title 1 | Mississauga Immigration Lawyer & Canadian Citizenship Attorney |
| Meta Description 1 | Imran Khan - Canada Immigration lawyer and Canadian Citizenship attorney Mississauga Immigration Lawyer |
| H1-1 | Canadian Immigration & Naturalization Lawyer |
| H2-1 | Imran Khan Law Office offers Legal Services in Immigration Law and Real Estate Law Matters. |
| Meta Robots 1 | index,follow |
| Canonical Link Element 1 | http://www.imranlaw.ca/ |
| Word Count | 275 |
| Level | 1 |
| Inlinks | 28 |
| Outlinks |19
|
| Address | http://www.imranlaw.ca/contact |
| URL Encoded Address | http://www.imranlaw.ca/contact |
| Status Code | 200 |
| Status | OK |
| Content | text/html; charset=ISO-8859-1 |
| Size | 14503 |
| Title 1 | Mississauga Immigration Lawyer - Contact |
| Meta Description 1 | Imran Khan - Canada Immigration lawyer and Canadian Citizenship attorney Contact Imran Khan |
| H1-1 | Contact Imran Khan Law Office |
| Meta Robots 1 | index,follow |
| Word Count | 276 |
| Level | 2 |
| Inlinks | 28 |
| Outlinks | 17 |A few domains the ones above which are listed below as well fail to be able to be seen by a synthetic Googlebot. Are you running them all on the same server?
You have some domains and in .com and others that end in .ca if you are looking in Google.ca and have geo-targeted the .com domains to Canada you should see them there. However if you're looking in Google.com obviously you cannot geo-target .CA domains to the United States therefore they would not show up in .com unless very rarely.
Deep crawl and screaming frog are going to be a best friends on this one. Please let me know if I can be of more help
here are my findings using a basic tool
and put it into https://varvy.com
The results were
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
HTTP headers
Page headers when accessed as Googlebot.
Headers:
pages could not be found
https://varvy.com/hierarchyandlinks.html
Same thing for imranlaw.ca
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
For canadaenergy.ca
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
Amount of links
Amount of links not excessive.
0 links found on page.
Guideline states: 'Limit the number of links on a page to a reasonable number (a few thousand at most).'
Considering the amount of links on a page
**I wouldUse a tool like deepcrawl.com or screamingfrog.co.uk/seospider **
two determined exactly what is wrong with all three Domains which failed a very basic test of being able to be detected by Googlebot.
Hope this helps,
Tom
-
Hi Jim,
If analytics confirms that traffic is still landing on the homepage, then I think this is just Google reporting different pages when you perform a site: - It certainly doesn't sound like a penalty of any sort.
It is worth noting that Google did confirm some time back that site: doesn't bring back every page every time and is best used as a guide. Does the sitemap in Search Console show a healthy number of indexed links?
If you want a discussion on this, then it would be worthwhile also posting over at the Websearch Help Forums at Google and see what others have to say about it.
I hope this helps a little.
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing a site from Google index with no index met tags
Hi there! I wanted to remove a duplicated site from the google index. I've read that you can do this by removing the URL from Google Search console and, although I can't find it in Google Search console, Google keeps on showing the site on SERPs. So I wanted to add a "no index" meta tag to the code of the site however I've only found out how to do this for individual pages, can you do the same for a entire site? How can I do it? Thank you for your help in advance! L
Technical SEO | | Chris_Wright1 -
My Website's Home Page is Missing on Google SERP
Hi All, I have a WordPress website which has about 10-12 pages in total. When I search for the brand name on Google Search, the home page URL isn't appearing on the result pages while the rest of the pages are appearing. There're no issues with the canonicalization or meta titles/descriptions as such. What could possibly the reason behind this aberration? Looking forward to your advice! Cheers
Technical SEO | | ugorayan0 -
How to check if an individual page is indexed by Google?
So my understanding is that you can use site: [page url without http] to check if a page is indexed by Google, is this 100% reliable though? Just recently Ive worked on a few pages that have not shown up when Ive checked them using site: but they do show up when using info: and also show their cached versions, also the rest of the site and pages above it (the url I was checking was quite deep) are indexed just fine. What does this mean? thank you p.s I do not have WMT or GA access for these sites
Technical SEO | | linklander0 -
Do I use /es/, /mx/ or /es-mx/ for my Spanish site for Mexico only
I currently have the Spanish version of my site under myurl.com/es/ When I was at Pubcon in Vegas last year a panel reviewed my site and said the Spanish version should be in /mx/ rather than /es/ since es is for Spain only and my site is for Mexico only. Today while trying to find information on the web I found /es-mx/ as a possibility. I am changing my site and was planning to change to /mx/ but want confirmation on the correct way to do this. Does anyone have a link to Google documentation that will tell me for sure what to use here? The documentation I read led me to the /es/ but I cannot find that now.
Technical SEO | | RoxBrock0 -
Investigating a huge spike in indexed pages
I've noticed an enormous spike in pages indexed through WMT in the last week. Now I know WMT can be a bit (OK, a lot) off base in its reporting but this was pretty hard to explain. See, we're in the middle of a huge campaign against dupe content and we've put a number of measures in place to fight it. For example: Implemented a strong canonicalization effort NOINDEX'd content we know to be duplicate programatically Are currently fixing true duplicate content issues through rewriting titles, desc etc. So I was pretty surprised to see the blow-up. Any ideas as to what else might cause such a counter intuitive trend? Has anyone else see Google do something that suddenly gloms onto a bunch of phantom pages?
Technical SEO | | farbeseo0 -
Can Google show the hReview-Aggregate microformat in the SERPs on a product page if the reviews themselves are on a separate page?
Hi, We recently changed our eCommerce site structure a bit and separated our product reviews onto a a different page. There were a couple of reasons we did this : We used pagination on the product page which meant we got duplicate content warnings. We didn't want to show all the reviews on the product page because this was bad for UX (and diluted our keywords). We thought having a single page was better than paginated content, or at least safer for indexing. We found that Googlebot quite often got stuck in loops and we didn't want to bury the reviews way down in the site structure. We wanted to reduce our bounce rate a little, so having a different reviews page could help with this. In the process of doing this we tidied up our microformats a bit too. The product page used to have to three main microformats; hProduct hReview-Aggregate hReview The product page now only has hProduct and hReview-Aggregate (which is now nested inside the hProduct). This means the reviews page has hReview-Aggregate and hReviews for each review itself. We've taken care to make sure that we're specifying that it's a product review and the URL of that product. However, we've noticed over the past few weeks that Google has stopped feeding the reviews into the SERPs for product pages, and is instead only feeding them in for the reviews pages. Is there any way to separate the reviews out and get Google to use the Microformats for both pages? Would using microdata be a better way to implement this? Thanks,
Technical SEO | | OptiBacUK
James0 -
Pages not Indexed after a successful Google Fetch
I am trying to understand why google isn't indexing key content on my site. www.BeyondTransition.com is indexed and new pages show up in a couple of hours. My key content is 6 pages of information for each of 3000 events (driven by mySQL on a wordpress platform). These pages are reached via a search page, but no direct navigation from the home page. When I link to an event page from an indexed page it doesn't show up in search results. When I use fetch on webmaster tools the fetch is successful but is then not indexed - or if it does appear in results it's directed to the internal search page e.g. http://www.beyondtransition.com/site/races/course/race110003/ has been fetched and submitted with links but when I search for BeyondTransition Ironman Cozumel I get these results.... So what have I done wrong and how do I go about fixing it? All thoughts and advice appreciated Thanks Denis
Technical SEO | | beyondtransition0 -
Are Google now indexing iFrames?
A client is pulling content through an iFrame, and when searching for a snippet of that exact content the page that is pulling the data is being indexed and not the iFrame page. Seen this before?
Technical SEO | | White.net0