When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is being in position 1-3 without a featured snippet better for conversions?
I'm a cosmetic dental practice so we're local and want people to visit our website, look around and make a booking. I've followed very closely the featured snippet debate on WBF and in the blogs here and everywhere else on the web. I've got many 1-3 positions and have been optimising for featured snippets with great success. I have noticed though that once I get a featured snippet the CTR of the page that the snippet is taken from drops significantly. This can often put me from position one into two or three. The same thing happens sometimes on the local listings. You get in the pack and people start visiting your homepage rather than the particular services page in the SERP. At a conference I was at, here https://www.youtube.com/watch?v=1iXlVcbpvd0 (11:14) there is a suggestion that Vodafone de-optimise because a position one or two is better for their sales than a featured snippet. After all my interests are not the same as google. They want to answer questions and I want to sell cosmetic dentistry. So is answering a question without bringing a user to my site a good idea? I've certainly seen the amount of enquiries drop once the category has a number of featured snippets. Are they as good as everyone says they are? I think possibly not from my data and this Vodafone talk is the first time i've heard anyone agree. I've scoured the web for answers and it's all just "GET FEATURED SNIPPETS", "HERES HOW TO GET THEM" etc. But my business is not about getting more traffic it's about selling. Could someone please point me to research on conversion rate and featured snippets. or ROI and featured snippets? Thanks.
Conversion Rate Optimization | | Smileworks_Liverpool0 -
Is my destination url affecting QS?
Hello, I'm running a PPC campaing targeting "Storenvy App" using the ads below
Conversion Rate Optimization | | JustinButlionand the following DESTINATION LINK: http://www.yotpo.com/landing_page?utm_source=google&utm_medium=cpc&utm_campaign=s-storenvy&redirect=http://lp.yotpo.com/storenvy-app.html As you can see there is a 301 redirect from http://www.yotpo.com/landing_page?utm_source=google&utm_medium=cpc&utm_campaign=s-storenvy TO http://lp.yotpo.com/storenvy-app.html How this will affect QS? Thanks in advance for all your answers! AdDzQi0.jpg
0 -
How Would You Market A Photo Of Jay-Z Wearing Your Client's Apparel?
If Jay-Z was spotted wearing your client's sweatshirt, how would you maximize the marketing potential of this photo and opportunity to sell more sweatshirts?
Conversion Rate Optimization | | farmiloe0 -
Please Settle a Bounce Rate Debate
Here's the Question: If a person clicks a PPC ad and hits the landing page, and the landing page has a form to fill out embedded in it without having to click, does that count as a bounce if the person leaves the page immediately after filling out and submitting the form or does the submission negate the bounce tally? Hope this makes sense and thanks in advance.
Conversion Rate Optimization | | jesse-landry0 -
Can someone give me some good articles about conversion?
I finally bought the seo moz pro version. I consider myself pretty knowledgeable in seo, content/social media marketing, but I am a rookie at conversion optimization. HELP PLEASE! I am trying to convert traffic into phone calls and/or contact forms.
Conversion Rate Optimization | | SwanJob0 -
Conversion Rate - site feedback
Hi We gave a website a bit of a facelift last nov with the aim of increasing its position within Search Engines and getting it to begin making sales. The site: www.funkyfootstools.com is now moving in the right direction for its keyphrases, footstools and ottomans although it still needs some work. The issue I have is that from the visitors that come to the site, the stats prove they have a look around, they are just not purchasing - I am not sure if this is because of the layout, ie. usability or if there is something else. Any ideas & feedback would be greatly appreciated. Thanks Carl
Conversion Rate Optimization | | keane-1145170 -
Huge Increase in Search Traffic - Expected Drop in Conversions?
We've recently implemented a successful strategy whereby we've increased non-branded organic search traffic by over 50%. We started with our niche products and secured those rankings and then expanded the program to include local results. Rev is up, trans# is up, but conversion rate dropped. Here is the question...is there an acceptable ratio between the percentage growth in visits and the percentage decrease in conversions?
Conversion Rate Optimization | | NaHoku1