A/B Tests: How To Verify Difference In Average Order Value?
-
Hi there! When the data from an A/B test shows a difference in AOV between the variants, how do you determine if this difference is statistically probable or not? Best Regards, Martin
-
Thank you, David.
-
Perhaps this will get your on the right track, from http://flintanalytics.com/how-to-determine-if-you-should-advertise-higher-priced-goods-to-mac-users/
Average Order Value Statistical Significance
To do this you need to download the transaction data for both variants, because you need to be able to determine the variance in order value. You can download this data by going to conversions > ecommerce > transactions in Google Analytics. Once there, Google only lets you export 500 rows of transactions at one time, so to keep it simple I ensured my date ranges were under 500 transactions.
Once you get the order values for each set of customers in Excel you need to perform a t-test so that you can compare the means of the two groups. This t-test will give you the p-value that you will use to determine if the difference in average order values between the groups is significant. Yout formula in excel should look like the following:
=t.test(array of group 1, array of group 2,two-tailed distribution, two-sample equal variance homoscedastic)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Direct / (none) Spam Traffic Help
In July 2015, we experienced an over 1,000% increase in traffic and it has remained like that ever since. It's all spam traffic and I have no clue how to get rid of it. I added in your typical .htaccess blocks from known culprits with little to no effect. Read up on Ghost traffic and applied filters to no effect. The spam is completely distributed as far as I can tell both geographically as well as by network providers. Where once we had pretty decent bounce rates of around 50%, now, since all my Analytics data is meaningless - it's around 90%. I could apply a filter but beyond my GA account providing no insights, I'm also concerned about the increased use of server resources. I'd ideally like to stop the traffic completely. The only distinguishing feature of the traffic that I have been able to determine is browser size. Comparing June 2015 to July 2015 we saw the following: Browser size visits: 620 x 460 = 6,828 vs 0, 610 x 450 = 175 vs 0, 1330 x 630 = 71 vs 1, 1890 x 940 = 67 vs 0, 780 x 580 = 58 v 5. Other than that, I can find no unifying theme to the traffic beyond being traffic hitting our homepage and having no medium. Nothing special that I am aware of happened in July. We didn't do any sort of...really anything. We did have our network compromised by ransomware in the beginning of June, which we promptly ignored and restored backups - at no point did we try to contact the criminals, but I am doubtful there is any connection considering that our website is remotely hosted. If anyone has any suggestions or has seen anything like this before, please let me know. spam-traffic.jpg
Reporting & Analytics | | Nivik230 -
Differences in site search revenue in GA
I just put in a piece of software to replace a really bad built in site search engine on my 3dcart website. Now I am trying to measure the change, but I am having some issues. When I check the ecom data in the conversions section of GA with the built in segment Performed Site Search, I get promising results. Approximately 5% revenue increase over LY. But if we jump to behavior, site search, usage, and then check the visits with site search, I get a decrease by 4%. And the actual revenue is off, by like double (150k compared to 80k) Anyone have any idea why I am getting these results? The site search function is set up. Tracking is enabled, query parameter is keyword and search url is /search.asp?keyword=
Reporting & Analytics | | ShockoeCommerce0 -
Main Website Redirects to Mobile Website, Mobile Website counts this as direct traffic, is there a way to tell what the source/medium is?
Hello, The situation is that someone is arriving on my main website https://www.example.com and being redirected to http://m.example.com. When this happens my analytics says that the traffic is all direct coming to my mobile site. However, I know people clicking on my google cpc, and some google organic users are hitting the main website and being redirected. Before we didn't have as good of a redirect on our main website so I could tell organic and cpc traffic coming in, now my main website has a huge drop in these categories because they are redirecting to mobile but I can't tell on my mobile how much traffic from each is going to the mobile site. Is there a way to fix this? Is it because my main website is https:// and mobile is a http:// (as I know that sometimes makes traffic direct) or is it a bigger problem that can't be resolved? Thanks
Reporting & Analytics | | oxfordseminars0 -
Why do I have differences in my User Flow Report?
On the user flow report I am showing a first interaction of /category as my second highest starting page. If I click group details it shows some of my different categories. However, at the bottom of the user flow report it has some of the category pages out as individual categories. Why is this? And how do I force it to show all category pages together like it does with my /item pages?
Reporting & Analytics | | EcommerceSite0 -
Are Panda/Penguin Penalties not Global but only fired for specific Google CCTLDs?
An international portfolio of web sites has been suffering from Panda and Penguin over the last months. Essentially many of the international sites completely disappeared from the top 50 when before we had many Top 5 positions for competitive key words. Today we noticed that it appears that for all this time rankings for local keywords have not been affected for domains outside of the most relevant Google CCTLD for the page. To give an example: Domain.it / Keyword 1 (IT) = Italian Competitive Keyword Google.it: Keyword 1 (IT) April 2012 Position 4
Reporting & Analytics | | sp80
Google.it: Keyword 1 (IT) December 2012 Position --- Google.com: Keyword 1 (IT) April 2012 Position 1
Google.com: Keyword 1 (IT) December 2012 Position 1 Google.de: Keyword 1 (IT) April 2012 Position 3
Google.de: Keyword 1 (IT) December 2012 Position 4 Have other people observed such behavior? Does this give any pointers towards how the recovery strategies should be drafted? We have experienced that the more search volume a keywords has received the harder the Panda/Penguin impact. So one hypothesis could be that because the international Google domains do not receive overly significant traffic for the localized keywords Panda/Penguin protection algorithms are not being applied with the same force. Any thoughts are welcome. /Thomas1 -
Core audience, daily / weekly uniques
In Google analytics or site catalyst, is there anyway to target IP addresses to see how many of my daily uniques are daily, weekly, monthly and what % are just 'tourists'/'one-offs'? My site is a local news site... Or is there another way to find this info. I am trying to work out what my core audience is (that keeps coming back every day or every week). Kind regards
Reporting & Analytics | | MirandaP0 -
Google Search Results inconsistent from different computers
Recently after some optimization activities - I do not see much movement in search rankings - my client is seeing the results on page 1 position 3 and I see page 2 for the same keyword. How does Google change ranking based on past searches and how can I get an accurate picture of what the actual rank is?
Reporting & Analytics | | devonkrusich0 -
What's the best way to track moble traffic / sales from a subdomain in Google Analytics?
Hello, I've been trying to figure out the best way to track traffic and sales for a subdomain mobile site and would appreciate any feedback or guidance. I've set up an advanced segment but have found that I'm not able to source traffic since the segments get disabled when trying to view top content pages. I've also setup an additional sub-profile under the main domain profile in google analytics and have set up an advanced filter using the following guildelines http://code.google.com/apis/analytics/docs/tracking/gaTrackingSite.html In the Profile Settings page, click the Add Filter link. Choose Add New Filter and provide the filter a name. Choose Custom Filter and select Advanced on the Filter type settings. Under Advanced settings: FieldA should be set to Hostname FieldB should be set to Request URI Set the values for both Field A and Field B to (.*), which is an expression that captures all characters. Set the Output To --> Constructor option to Request URI and provide $A1$B1 as the value for that choice. I'm unsure if I've set up the profile / filter correctly, the traffic stats are different than the segment. The mobile profile is also tracking all ecommerce transactions instead of just mobile. Here's the code I have in place. we're using the Traditional code Thanks, Jamie
Reporting & Analytics | | marketing_zoovy.com0