When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Metadescription (SERP's) with green checkmarks: spammy or not?
HI Will a Metadescription with GREEN checkmarks be perceived as spammy in general? Or is one with BLACK checkmarks better? Depends on the topic/target audience, etc.? --> see screenshots. No doubt that the green will stand up more (increases CTR) in the SERPS but if it is perceived as less trustworthy or even spammy by the target audience its not worth doing it. Any experience with that? Would really like to know what do you think about it. Cheers, Cesare 0G3Fd
Conversion Rate Optimization | | Cesare.Marchetti0 -
How does Shopify attribute conversions?
I'm trying to resolve a number of conversion discrepancies found in Google Analytics. I discovered that Shopify also provides its own 'conversion details'. In some cases they correlate, which is awesome! But in many cases they do not. Anyone know why this is? And which is more reliable (remembering that Shopify is an actual, real order)? Does anybody know how Shopify attributes conversions (ie. last click, first click, or some hybrid of the two)?
Conversion Rate Optimization | | muzzmoz0 -
UX and/or Conversion Help for Ecommerce
Good Morning, We recently revamped and relaunched our ecommerce website and we believe we have built a great site that is easy to navigate, find what you're looking for, get all details, and purchase. We were hoping to get some of the Moz community to take a look at the site and let us know if you have any suggestions on usability or conversion techniques that we could implement to improve user experience and amount of orders placed. You can view our site here Looking forward to your feedback. Thanks in advance
Conversion Rate Optimization | | Prime850 -
Google showing in snippets last individual rating instead of aggregate rating
the product pages of my ecommerce site contain schema markup.
Conversion Rate Optimization | | lcourse
according to rich snippet tool from google all looks fine and properly formatted
however in the SERP google shows the last rating in the snippet instead of the aggregate rating. any idea how to show the aggregate ratings? URL: https://www.humidordiscount.co.uk/adorini-triest-deluxe-rosewood-humidor below extract of schema code as recognized by the rich snippet tool: Product name:adorini Triest Deluxe Rosewood Humidor image:https://www.humidordiscount.co.uk/952-large_atch/adorini-triest-deluxe-rosewood-humidor.jpg description:High-quality multiple ..... brand [Organization]: name:Adorini offers [Offer]: price:141 priceCurrency:GBP availability:http://schema.org/InStock aggregateRating [AggregateRating]: worstRating:1 ratingValue:4.5 bestRating:5 ratingCount:460 review [Review]: description:It 's my first .... reviewRating [Rating]: ratingValue:4 worstRating:1 bestRating:5 author [Thing]: name:Alessandro M review [Review]: description:excellent ... reviewRating [Rating]: ratingValue:5 worstRating:1 bestRating:50 -
Can using the Google Keywords tool and/or SEOmoz cause problems for my company's IP address?
I just started with a company a few weeks back, working on-site doing SEO work. I've been using SEOmoz as well as Google Adwords' Keywords tool. I just got a message from one of our reps saying that people at the office (and this is a huge office) have reported tons of CAPTCHAs being thrown up frequently. I told them I had been running searches with Google Keywords tool and SEOmoz, but would that really cause that much of a problem? Is there anything I can do in order to fix this, some keyword finders that won't get me in trouble, etc.?
Conversion Rate Optimization | | SmokewagonKen0 -
What is the best solution to track conversions on local plumbing site (non-mobile)?
What is the best solution to track conversions on our local plumbing site? 95% of leads coming from phone calls, the rest – from the contact form we have on every page. Is there a reliable solutions to track conversion rate for non-mobile local site? As a side note, we have 6 different phone numbers on our site that users can call in order to reach us.
Conversion Rate Optimization | | dinok_ya0 -
Can Adwords campaigns affect other campaigns performance?
I understand that keywords and adgroups within a campaign can affect the performance of other keywords within that parent campaign. I was interested to understand whether campaigns in my account can affect one another. I have an adwords account with around 30 campaigns. Some are heavy hitters and others get minimal to 0 traffic. Will these low quality campaigns drive up the costs of my high end campaigns?
Conversion Rate Optimization | | peigenesis0 -
Can Changing Meta Descriptions Negatively Impact SERP's?
I have just had a page start ranking well in key SERP's and I would like to change the meta description and add a price as we are extremely competative in that line. Could changing the meta description now a page is ranking negatively impact the SERP placing? Does anyone have any experience with this?
Conversion Rate Optimization | | robertrRSwalters0