When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is being in position 1-3 without a featured snippet better for conversions?
I'm a cosmetic dental practice so we're local and want people to visit our website, look around and make a booking. I've followed very closely the featured snippet debate on WBF and in the blogs here and everywhere else on the web. I've got many 1-3 positions and have been optimising for featured snippets with great success. I have noticed though that once I get a featured snippet the CTR of the page that the snippet is taken from drops significantly. This can often put me from position one into two or three. The same thing happens sometimes on the local listings. You get in the pack and people start visiting your homepage rather than the particular services page in the SERP. At a conference I was at, here https://www.youtube.com/watch?v=1iXlVcbpvd0 (11:14) there is a suggestion that Vodafone de-optimise because a position one or two is better for their sales than a featured snippet. After all my interests are not the same as google. They want to answer questions and I want to sell cosmetic dentistry. So is answering a question without bringing a user to my site a good idea? I've certainly seen the amount of enquiries drop once the category has a number of featured snippets. Are they as good as everyone says they are? I think possibly not from my data and this Vodafone talk is the first time i've heard anyone agree. I've scoured the web for answers and it's all just "GET FEATURED SNIPPETS", "HERES HOW TO GET THEM" etc. But my business is not about getting more traffic it's about selling. Could someone please point me to research on conversion rate and featured snippets. or ROI and featured snippets? Thanks.
Conversion Rate Optimization | | Smileworks_Liverpool0 -
Track bounce rate per PPC keyword
Hi all Wondering if anyone could give me a nudge in the right direction with this please. When running an adwords campaign is there a way in which one can track the bounce rate on the landing page per keyword sending paid traffic? I've had a play about in adwords and GA and couldn't find the right option. The site I am sending traffic to is not a ecommerce site so cannot set up conversion tracking so we are going to monitor the bounce rates of different keywords and drop the ones which are bouncing much higher than others as it seems better to invest the budget in those words which are keeping people on the page for longer. Many thanks Carl
Conversion Rate Optimization | | carl_daedricdigital0 -
Question about conversion rate in ppc
Hi guys, Im managing the campaigns in the company that im working and so far i think that the results are o.k more or less. The problem is how can i really know if it is o.k ?
Conversion Rate Optimization | | WayneRooney
I really don’t have any data to compare to… For example, this is campaign from the insurance niche from the uk : Clicks – 132
Impr – 6,958
CTR- 1.90%
Conversion – 20 (Not inducing the one that calls the office)
Conversion rate – 16%
Total cost – 704£ How can i know by these number that the campaign is working ? Same here (T-shirts campaign for the Spanish market): Clicks – 1,219
Impr – 26,177
CTR- 4.66%
Conversion – 56
Conversion rate – 4.5%
Total cost – 493€ Probably you will say, check how much you spend and how much you got back ?
This is also a problem for me. I can see the numbers of the conversion but because we are not eCommerce so i cant really know the total income. Do you know if there’s is a way that each for quote that coming to our system
so i will know where it came from ? Like a comment in the url or something ? I think that 90% of our income/quotes comes from the organic as we are ranking very high in Google so its a bit mixed in the system. Any feedback or tips here on this will be very welcome…0 -
How to find average organic traffic growth rate for ecommerce startup?
Happy Friday Everyone, I have a quick question: I am in the process of crunching some numbers to try and figure out an organic traffic forecast for a client. One of the crucial metrics I can't seem to figure out is traffic rate of growth over the next year or so. The reason this is complicated for me is that I have only ever done this sort of thing for an established business before, and had plenty of past organic traffic data to work with. What I am looking for is a source that could give me a reasonable idea of what kind of growth rates I might expect for a startup with practically no data; I would be perfectly happy with a national average for online retail startups. Any help would be greatly appreciated. Thanks!
Conversion Rate Optimization | | G2W0 -
Suggestions for Conversion Rate Optimization low cost companies?
Does anyone have a list of suggestions for lower cost companies to do conversions. I am a new business and just wanted my "Cart" check out with maybe some suggestions for improvement and a little testing to see if it is done correctly. Thank you, Boo
Conversion Rate Optimization | | Boodreaux0 -
What's the Best Method of Content Distribution
This question is both specific and open ended. I have a new web project I've been working on content for. I paid to have a really sweet research article written and I am in the process of turning it into an info-graphic as well. Being that the the website is so new it doesn't have many customers and few to no blog readership. Now that I will soon have great content for the site should I : A. Post on the Blog ? ( Its a Blogger on a blog. domain.com ) B. Post as a static page on the website ? (domain.com/content) C. Something Else ? Whats the easiest to promote and push traffic to?
Conversion Rate Optimization | | CDUBP
Which will rank best?
Which will rank longer?
Which will rank faster? Any other suggestions to get the content in front of the niche market?0 -
Thoughts on meta descriptions - best click through rates
Hi All, I recently heard a tip from a prominent SEO'r that meta tags should be custom written by professional copywriters. Well, I want to head down that path. What are your thoughts on these two options: Buy the top-selling <xyz product="">. 20 year warranty, 30 day money guarantee + free, fast shipping! Includes <1-2 features or included accessories of product></xyz> or Buy our top-rated <xyz product="">today</xyz> & save! Over 20 5-star rated reviews + free, fast shipping! Includes <1-2 features or included accessories of product> Would you focus on the warranty, reviews, product accessories, money back guarantee, free shipping over a certain cutoff, customer service (which we have a toll free number for) or another factor? Are there any good articles out there outlining this? Do you think the way I've structured it is good/readable? What about the use of '+' instead of '&'? Is the cutoff limit still 156 chars? Any thoughts/views/ideas are most appreciated! Thanks
Conversion Rate Optimization | | bradkrussell0 -
How to track PPC conversions?
I use both adwords and Microsoft Adcenter. I am looking to be able to track my conversions from each PPC site. I do not have a "Thank You" or "Check Out" page that I can tag because any sales that I make are through clicks on a link that goes offsite to an affiliate company that I receive commission from. However, the link on my site is a "Buy Now" button, so to get some idea of which source is giving me better conversions I'd like to know whether adcenter or adwords is sending more people that end up clicking this link. Canb anyone suggest a way or a program to set up in this manner? Thanks in advance for your help!
Conversion Rate Optimization | | VictorVC0