slide 1: 10 A/B Testing Mistakes that you should
avoid
Mistakes are the integral part of your learning process but when making them is costing you
money then it is the time you need to stop making them.
The most reliable way to increase your conversion rate is by A/B testing but you can only do so
if you are conducting your A/B tests the right way.
Here we will show you the 10 ways A/B testing can ruin your site and not only this you will also
get to know that how you can avoid these mistakes.
1 Testing the variant without the control
Not fully testing the variation and putting it live is the most stupid mistake we all make while
A/B testing.
Before even testing the variants and putting them live by deleting the original ones is what you
need to stop doing.
What you should do
To get to know which variants are bringing you better conversion rates you need to test both
simultaneously. So it’s never a good idea to not test the things and push them live.
slide 2: So it is recommended to actually run an A/B test and see which one is better.
slide 4: 2 Running too many tests on a single page
By running too many tests on a page one after the other you fall victim to local maximum and
opportunity cost.
Opportunity cost: you lose the opportunity of optimizing other pages as you keep on running
tests on the same page.
Local Maximum: once you hit the peak of your conversion rate the gains you’ll get will be
miniscule no matter how many tests you run.
When you keep on running tests then the consistent gain in conversion rate is very less like
0.4. So you should not run too many tests on a page.
What you should do
You should set up a time that for how long you should run a test to achieve a particular
conversion rate.
So it is better to set your goals beforehand and work according to them to avoid losing
opportunities. And once you see that tests are bringing miniscule change and then you should
stop doing any more tests on that page.
3 Testing more than one element in an A/B Test
When you start A/ B testing then you want to optimize all of your pages to increase conversion
rates. And this is the most common mistake that most of the marketers make.
It’s never good to test more than one element in an A/B test as you do not get significant
results from it. And you don’t get to know what actually worked in your interest and what did
not.
What you should do
While running an A/B test you need to make sure that you are testing only one element in it.
Yes you can create multiple tests and test other elements in them one by one.
So it is highly recommended to make one change in one A/B test to get to know what actually
works.
slide 5: 4 Running an A/B Test for far too long
If you call off your A/B tests too late then you end up wasting both your precious time and
efforts. In that time you could have done some other tests and would have optimized other
elements.
You not only waste your time but you also miss out revenue when you run a test for far too
long than it is required to get a particular statistical significance.
What you should do
To make sure that you don’t waste your time in running a test for a long duration of time then
you must set your goals and work for them in the backward direction.
5 Setting the test up wrong
If you set up the test wrong then you may not get the right results.
For example if you are already testing one variant and then in between the test if you
introduce another variant then the new variant does not help you get desired results.
Because when a new variant is introduced in the middle then it isn’t treated as the part of the
original test. That’s why you always set it up in the right manner.
What you should do
slide 6: It is suggested that you should always start the test from the scratch that is from the very
beginning because when you introduce a variant in between then it doesn’t work in your
interest.
So if you want to test a new variant then start a completely new test.
6 Not running an A/B test before implementing
For example you run a generalized test and your variant helps you increase the conversion
rate up to 30. And you start applying it to your entire site from design to call-to-
action everywhere.
And if your website conversion rate starts dropping then you must not wonder why because
the reason is clear you didn’t run an actual A/B test before implementing it.
What you should do
Before making changes to your site you should set up a new test on a new page and see
whether the results you got from the significant test are true for other pages also or not.
slide 7: So it is recommended to always run an A/B test before implementing it on your website.
7 Installing the wrong A/B Testing tools
There are so many tools with the help of which you can run A/B tests. But finding the right tool
which can help you A/B test everything present on your site is a bit hard.
But if you don’t install your tools the right way then the results you are getting from the tests
must be wrong.
What you should do
The great solution to this problem is to find a tool that contains in-built A/B testing. You should
be using the tools which have A/B testing as the part of their core functionality.
Because when you use a tool that has built-in A/B testing then you do not need to worry that
whether it is installed correctly or not.
For example Sumo helps you test frequency action of conversion Improvement statistical
significance etc.
8 Doing A/B Testing without keeping a goal in mind
Whenever you go down on an A/B testing road you should have a specific goal in mind.
Because if you don’t know for what you are running your test then how will you get to know
that whether you were able to achieve your goal or not.
slide 8: Running test with no goal is like sailing on a boat somewhere without any map in hands.
What you should do
If you want your tests to be more efficient and effective then you must set a goal before
running a test.
Because when you set up a goal you set yourself up for a more structured test.
9 Ignoring the data while A/B testing
Why are you running A/B tests if you do not want to select the clear winner and want to make
your own assumptions at the end
The silliest mistake marketers make is by ignoring the data they get from the results and doing
what they think is right for their site and then end up getting their site in trouble.
What you should do
slide 9: You should set your A/B test the right way and trust the data you get from the tests and make
the changes to the site accordingly.
Even if the results you get are against your taste and what you expected you should go with
the data because only the numbers can help you take the best decisions.
10 Giving up after one test
If you give up after running a single test then you miss out the chances of succeeding if you
would have conducted more tests.
For example if you A/B tested an image on your landing page and got a 10 gain in your
conversion rate and settled with it. But if you would have run the test again then there are
chances that you may have got a better conversion rate.
What you should do
You should not give up after running a single test rather you should run more number of tests
to get a better lift in the conversion rate. So it is recommended to not to give up after just one
test.
slide 10: Hope that you will become aware of the above mentioned mistakes and avoid making these in
future. And rather run the A/B tests the correct way to get better conversion rates.
Source: 10 A/B Testing Mistakes that you should avoid