When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We've a client that wants to repurpose some of the content on there site but not lose rankings. However its changing the content to an alternative to ....
The Current URL Example would be test.co.uk/prodcut-catogory/product-1 . This side of the business is being sold however they have products elsewhere in the range that can be used as alternatives to the product range that will be moved, so they want to amend these pages with an " alternative " slant. Example idea would be to change to the following : test.co.uk/prodcut-catogory/alternative-to-product-1, trying not to change the URL to much, keeping the majority of the content the same but adding new copy stating the benefits of using the alternative product. So it becomes a comparison between the two. I appreciate I will have to amend the URLs, titles, meta, H1s, add new copy & preform a 301's. Option 2 Simply write a second page which is all brand new content talking about it being an alternative to the products and doing a 301 from the original page but not amending that. Be great to get peoples feedback on this approach and if there is anything else that I could do to minimise the drop in rankings once I complete the changes. Thank you D
Conversion Rate Optimization | | SDOwner0 -
Do the clickable images on pages are bad that they will increase bounce rate and distract users?
Hi Moz community, I'm trying to get some helpful answers on this. We are planning to employ clickable images on our website pages which opens in new tab if users click on them. This is to make sure that they can see the large screenshots of our product. I wonder will this increase the bounce rate and drop the rankings. Please share your experience and guidance on this. Thanks
Conversion Rate Optimization | | vtmoz0 -
How to track conversions from a third party email campaign?
Heyy.. New to the Moz community, first question so here goes! I'm going to be running an email campaign with an e-magazine and I want to know how to track conversions from the email they're sending out so I can monitor success/failure. I've found Google's URL builder:
Conversion Rate Optimization | | 9868john
https://support.google.com/analytics/answer/1033867?hl=en
But this doesn't really explain what i'm trying to do. Can anyone of you in this fine community shed some light on this please?
Johnny1 -
Better Conversions with Java Script Pop Up Form or with Independent Page (URL)?
Greetings MOZ Community? I manage a commercial real estate site (www.nyc-officespace-leader.com). Visitors view listings (for example:http://www.nyc-officespace-leader.com/listings/303-broadway-building-office-sublet-11189sf). If they want more details they click on the "Interested In This Space, Contact a Broker" button. They get directed to a contact us form like: http://www.nyc-officespace-leader.com/contact-us?from_listing=303-broadway-building-office-sublet-11189sf We are redesigning this form. Remove much of the text and allowing the visitor to check questions pertaining to the property, like "Schedule a Tour", "Get Floorplans", "Details on Similar Space'. Currently this conversion form is on its own URL, and as such has menus on top, which I am concerned could distract the visitor. Would we get better conversions if this form was a Java script pop up with no menus. That way there would be no distraction for the visitor and we would have a better chance of converting them. Any thoughts? Also, we are using Google Tag Manager and Google Analytics to track conversions. Would changing to a Java Script pop up form have any implications for tracking? Thanks everyone!!! Alan
Conversion Rate Optimization | | Kingalan10 -
Brand Name: Conversion Rate Optimizing: Eponymous VS Generic
Good day all, We are entertaining the idea of changing the Domain Name & Brand Name for straplessfashion.com. While still in development, the website sells women undergarments. We did some survey research and it seems that women (age 18-40) really liked either "Demi's Dream" or "Almost Skin" as a good name. More specifically, Almost Skin seems to be the more popular choice over "Demi's Dream". I think "Almost Skin" catches attention quicker and may work best in PPC. However, when personalizing a brand name is considered, "Demi's Dream" might be the best choice. Would you please offer your advise on which of these two names might be the best for a new Domain and Brand name for the website mentioned above? Which would convert better? Thank you for your help. Carlos
Conversion Rate Optimization | | 90miLLA0 -
Conversion Suggestions or Ideas
Late last year we did an overhaul of our website. We didn't change we did or what we sell just the format and overall look. What used to work for Adwords and Adcenter conversions doesn't seem to work as well on the new site. Any suggestions on where I should start to turn this around?
Conversion Rate Optimization | | EcommerceSite0 -
Can using the Google Keywords tool and/or SEOmoz cause problems for my company's IP address?
I just started with a company a few weeks back, working on-site doing SEO work. I've been using SEOmoz as well as Google Adwords' Keywords tool. I just got a message from one of our reps saying that people at the office (and this is a huge office) have reported tons of CAPTCHAs being thrown up frequently. I told them I had been running searches with Google Keywords tool and SEOmoz, but would that really cause that much of a problem? Is there anything I can do in order to fix this, some keyword finders that won't get me in trouble, etc.?
Conversion Rate Optimization | | SmokewagonKen0 -
Weeding out irrelevant analytical data to see truer conversion rates
Here is the scenario. We have many brick and mortar store locations as well as an ecommerce website. It's hard to get exact, but my estimates seem to be that approximately 1/3 of the visitors to our website are interested only in obtaining information about the brick and mortar store locations and not interested in ecommerce transactions. Of course this kills the conversion rate. We use google analytics and I'd like to somehow be able to quantify with more accuracy what the "real" conversion rate might be. Is there some method to weed out specific pages/traffic (like brick and mortar landing pages) from being taken into account when conv. rate is calculated? The number that matters for conv. rate of course is "visits" and not unique pageviews, so I'm not sure that really would do anything helpful. Any tips?
Conversion Rate Optimization | | dickslee230