Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Experiences Tracking Phone Call Conversions?
Tracking conversions via phone call can be challenging. The Adwords phone ads only register a conversion if someone clicks on the phone number from their mobile phone. In many cases a landing page on a website has a lead/contact form AND a phone number displayed. The user calling that number would not be registered as a conversion unless you setup multiple virtual forwarding phone numbers for prominent keywords, ad groups and/or campaigns. This can be costly for smaller advertisers. Has anyone done or know of any empirical data on the average percentage of leads generated via phone call versus online form on a landing page. I know it can vary from business type to business type (and the form layout itself will have a modest effect.) More specifically I'm interested in the professional business services industry such as Engineering, Surveying, Commercial Real Estate, Accounting, etc. So for example if you conversion rate for online form submission on a landing page is 3%, and your phone number is prominently displayed, what would be a realistic ballpark estimate for the actual conversion rate including phone calls? 25% more? 50% more? Double? "Ballpark" is the keyword here. Just seeing if anyone has measured this and what their results were.
Conversion Rate Optimization | | JCCMoz1 -
Which eCommerce site you consider using best practices? Site we can learn from
Hi, I'm looking to hear thoughts and suggestions as per sites that you consider to have great practices in the eCommerce world. Almost none of the sites do everything good so you can split your suggestions by any criteria -
Conversion Rate Optimization | | BeytzNet
the site structure
conversion funnel
Converting product pages (good design)
content creation and blog management / structure
content marketing
SEO guidelines / practices
... Thanks0 -
How to test unconditional free shipping?
Hi everyone, I am considering a switch to offering unconditional free shipping. By this I mean no minimums or contingencies. Before I do this I would like to be able to test the concept for a period of time, or for a section of my audience before fully committing. The issue is it will take a lot of time and $$$ to convert the website to market the free shipping properly. I have thought about using a promo code but I am not sure that will give a proper sample due to people getting confused. Has anyone seen how to test this concept? Thanks!
Conversion Rate Optimization | | Mjstout0 -
Can someone give me some good articles about conversion?
I finally bought the seo moz pro version. I consider myself pretty knowledgeable in seo, content/social media marketing, but I am a rookie at conversion optimization. HELP PLEASE! I am trying to convert traffic into phone calls and/or contact forms.
Conversion Rate Optimization | | SwanJob0 -
Can trackingcodes in your url be seen as duplicate content or break other rules?
I recently gave ReachLocal control to manage my Adwords campagne. I didnt want "Reverse Proxy" on so I decided to go with the Remote Conversion Tracking. So they track my website in every stap of the way. Therefor I had to place a Javascript on each page. But what worries me are these kind of URL's I see appearing in my Analytics account: /webpaginaname.html?scid+29903&cid=12343&tc+13012316371028544&rl_key=32850e0a2d4499e02a711d0b969044d2&kw+8307565&pub_cr_id=16701148613&dynamic_proxy=1&primary_serv=www.website.com&rl_track_landing_pages=1 Could this be seen as duplicate content by Google? Or could it break any other rule you know of? Should I be worried? Thanks a lot!
Conversion Rate Optimization | | wellnesswooz0 -
SEO For Personalized Content Pages
Hi, We are building a site that has content that personalizes based each visitor. This is being done to serve the most relevant content for conversion purposes and NOT for SEO purposes. So, there are many diferent versions of each page based on the visitor's location, device, browser, keyword-used, traffic source, etc. Obviously, we don't want this to appear like cloaking (it is genuine content personalization) and wondered how Google would view this? What is the safeguard for protecting the website from potential penalties? Any help would be greatly appreciated! Thanks in advance.
Conversion Rate Optimization | | Clickthroo0 -
I am trying to make my site a better user experience please help
Hi, i want someone to review my site and let me know what steps i need to do to make it a better experience for the user and to lower my bounce rate which is about 67 at the moment. I am trying to get the advertising and the user friendly to match, i need the adverts to make money and need to make sure i have them in the right place so they get clicked more. I want to reduce my bounce rate and want people to not just read one page but to go to different pages and would like advice on what i need to do. here is the site http://www.in2town.co.uk and here are the example pages that i would like to focus on to increase the user friendly and keep people on my site while making sure i have the adverts in the right place http://www.in2town.co.uk/news/gossip-celebrity/alex-reid-says-he-is-fed-up-with-chantelle-houghton-lies http://in2town.co.uk/news/travel-holiday-news/sainsburys-customers-10-points-back-on-easyjet-flights-with-nectar http://in2town.co.uk/news/small-business-news/how-emerging-technology-can-help-reduce-small-business-costs I am looking for ideas on how to increase the experience and how to keep people longer on my site while at the same time have people clicking the ads by increasing the conversion rate.
Conversion Rate Optimization | | ClaireH-1848860 -
Can Adwords campaigns affect other campaigns performance?
I understand that keywords and adgroups within a campaign can affect the performance of other keywords within that parent campaign. I was interested to understand whether campaigns in my account can affect one another. I have an adwords account with around 30 campaigns. Some are heavy hitters and others get minimal to 0 traffic. Will these low quality campaigns drive up the costs of my high end campaigns?
Conversion Rate Optimization | | peigenesis0