Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Adwords Offered to Redesign My Website & A/B Test... Thoughts?
I received a call from the Google Adwords team today offering to redesign major pages of my website & A/B test their design against mine. The rep said that this is not a standard service they offer and that we had been "selected". Despite it's overt marketing overtones, I would hate to pass by the opportunity to apply Google's conversion insight to my site. Has anyone experienced this before? Good experience? Bad experience?
Conversion Rate Optimization | | Glathrop1 -
Any General feedback or Experience on Page Titles and Click Through Rates ?
Moz, I have managed to climb onto page one for a couple of keywords i am really happy with, Just looking for any feedback on page titles and different effects on CTR The page i am working with is product pages I tried a simple headline which was a branded product keyword, this way when the keyword matches the user search it will stand out as the the majority of the title is bold because of this match,, Example below <title>Jeffs<strong> Lollipops</strong></title> and my other option is to look at testing a more descriptive title to fit the limit <title>A selection of sweet Strawberry and Cherry<strong> Lollipops</strong> from Jeff<//Title></p> <p>has anyone looked into this and tested for results in CTR ? i can see there being a youmoz about it somewhere?</p> <p>Alot of people i assume will just say test test test, but any feedback on how other moz users tackle title tags for CTR would be great,</p> <p>Thanks</p> <p>James</p> <p> </p> <p> </p></title>
Conversion Rate Optimization | | Antony_Towle0 -
Adwords shows clicks but no conversions in GA
Hi Folks, I was wondering if anyone has come across a similar situation, I have setup adwords campaigns and included these in my Google Tag Manager setup and deployed the GTM snippet to a clients site. In adwords I can see clicks (2,000+) however in Google Analytics I can see only a handful ot conversions. Is there anyway I can test/track a conversion to ensure that it is being processed and report correctly? Also within AdWords Conversion Tracking I have noticed that under the "Webpages" tab that pages other than the pages that should have the conversion code on them are listed. Have I setup my Adwords camapign or GTM tags incorrectly? Any and all feedback or insight would be greatly welcomed and appreciated.
Conversion Rate Optimization | | icanseeu
Regards,
Dave0 -
Is it better to get a new site instead of maintaining a category? I'm stuck.
If you had a website "Food for Dogs" and: Food for German shepherds was improving in search traffic & conversions Snack Food for German shepherds had no change and never resulted in conversions but resulted in many leads for #1 Food for Poodles was unchanged but improving conversions Here's the kicker... Food Container for Dogs was super high traffic but with no conversions or any activity to other categories Would you redirect "food container for dogs" to a different site and delete that tab from the original site completely? It's still related but does not offer any $ value or leads to any of the other categories. Just note that we have a different interest in this category that will support the other categories at one time, but not any time soon. We also have a major group on LinkedIN that's grown to thousands of members for the "food container for dogs" category. On top of it all, the "food container for dogs" has been increasing in rank on Google for several keyphrases to positions 2-5 (nationally) Other important question: Would keeping "food container for dogs" influence the ranking of the other keywords? (by dilution of the site's PR or any other reason?)
Conversion Rate Optimization | | HMCOE0 -
SEO For Personalized Content Pages
Hi, We are building a site that has content that personalizes based each visitor. This is being done to serve the most relevant content for conversion purposes and NOT for SEO purposes. So, there are many diferent versions of each page based on the visitor's location, device, browser, keyword-used, traffic source, etc. Obviously, we don't want this to appear like cloaking (it is genuine content personalization) and wondered how Google would view this? What is the safeguard for protecting the website from potential penalties? Any help would be greatly appreciated! Thanks in advance.
Conversion Rate Optimization | | Clickthroo0 -
Can anyone recommend software for recording phone calls through Adwords?
Hey Mozzers, I want to start recording phone calls as a result of PPC campaigns. Can anyone recommend an easy to implement system for setting this in place which will allow me to track calls placed as a result of PPC advertising. The ability to record the calls for training purposes would be a bonus. Thanks in advance Geoff
Conversion Rate Optimization | | SEM-Freak0 -
Can fun, slightly unprofessional text be good for conversion rates?
I operate on a freemium model. The three plans are currently called "Free", "Gold" and "Platinum". Besides being incredibly square and boring, these titles are also meaningless. I'm doing a slight redesign of my site, and the new (unreleased) site has a very fun, playful feel. I'm also trying to make everything more streamlined and obvious. I'm considering renaming the three plans "Boring Amateur", "Kick-Ass Amateur" and "Bona Fide Professional" on the registration page and simply "Boring", "Kick-Ass" and "Professional" elsewhere. There are lots of places on the site where I have to refer to "amateurs" vs "professionals" - these tags would make that easier to explain. Easier = more sales. What do you think about using unprofessional text like this? Is it too risky, possibly offending too many people and losing sales? Or is it a good eye-catcher? I know the best answer is "test it." But this is one of those changes that I don't want to have to put my users through too many times. Thanks for any thoughts!
Conversion Rate Optimization | | PatrickGriffith0 -
Good content but high bounce rate
Hi all, We run a website offering specialist turkey tours and gulet cruises. Our website holds a few information and recipes pages which are doing well in the engines and bring in alot of traffic. However these pages are crippling our bounce rate with bounces of about 80-90%. The time on site for these pages are above the websites average, but it seems that once the reader has finished with the information/ recipe, they bounce back to the Search engines. An example of one of our recipe pages can be found here: http://www.petersommer.com/turkey/turkish-food/pilav-rice/ We have thought about moving and redirecting these pages from our main site to a new website, but we are worried that it may effect our position in the SERP's. Can anybody offer any advice on this? Thanks
Conversion Rate Optimization | | petersommertravels0