Matik development director and Agny managing partner Artyom Ovechkin has written a note about the reason why a great many of A/B-tests fail and why it’s necessary to pay more attention to preparation rather than testing itself.

According to marketingcharts data, А/В-testing is considered to be the most efficient method of managing conversion. 60% out of more than 1000 polled marketing specialists in Europe consider this type of page testing to be more effective. Other quite popular method is analysis of visitors’ path throughout a website. It got 52% of positive reviews. A method of comparing to competitors had the lowest rates — 26%.

According to another investigation of the same source, only one out of seven А/В-tests is recognized as statistically presentable and successful. I can tell it from my own experience that А/В-tests haven’t entered the pool of obligatory marketing tools. It’s a destiny of either the biggest or the “most advanced” companies. As practice shows, testing is a long period which can take more than 1 month.

Majority of companies prefer to work faster rather than focus on quality. But taking into account market reduction and lack of money in business, I can say that the stimulus “to spend money faster” will calm down significantly. There’s an evident tendency to “spend the budget wisely with a prior preparation work”. A/B-testing is a thorough preparatory work which will help you to increase website conversion many times.

But as we have already mentioned, the majority of test show insignificant improvement. Why does it happen? There are several reasons.


Solid data ground

The main reason of failures is absence of preparatory work. Yes, preparation stage itself needs some preparation! Don’t be confused: the more thorough is the preparation, the better is the effect. We need to get used to it.

A/B-test is a final stage of the process. Before carrying out the test it’s necessary to collect the following data:

  • Usability-analysis of the website made by a specialist. The analysis will indicate key mistakes if the page, show you the most important things and will clear what and for that should you test. Usability-analysis is a map with roads from A to B. Periods of a usability-audit depends on the capacity, age of the website and depth of the analysis. So it may be 1 day to 2 weeks.
  • Data on website attendance for quite a long period (from half a year). They will show how users interact with the website, help to find out weaknesses and what you need to improve. It costs you nothing to introduce an analytics service (for example, Google Analytics).
  • And finally, use heatmaps.

The idea preparation steps is to collect maximum data about the current state of the website. In order to compare A and B you need to have not only an enhanced B variant but also a current A variant. Otherwise, you won’t be able to understand what exactly gets better.

60-70% of time should be spent on pre-preparatory stage (before A/B-testing). 30-40% is for creating pages and launching tests.

Now we have solid data of what’s wrong with a website and what “places and zones” with what aim should you test. It’s the first difference of right tests from wrong ones. “Losers” rely on intuition and common sense (“Buy” red button should work better). Winners hold statistics in their hands and auditors’ detailed analysis. That’s why it’s so important to collect statistics from the first days of your website.

According to marketingcharts, only 13% of companies have solid data. Others prefer intuition and common ideas of “what’s good and what’s bad” in website usability.


Solid understanding of your client

Other part of preparation work is understanding why a client chooses your company and your website. Forget about “search results”. You need to answer the question what’s is so special about your website that people make purchases?

Surely, you need to answer a negative question as well: what’s there on your website that people don’t make purchases?

As your understand a well-known “Buy” red button may increase conversion, but if a problem lies deeper, then slight page optimization (cosmetic changes) won’t give you significant results worth your efforts.

You should formulate hypotheses on the basis of your answers to the questions: what can I do to turn “no” into “yes”.

Eric Reyes, a popular marketing specialist, suggest to find answers with a help of series of “Why?” questions. Put your intuition aside and go to people. Ask your clients and visitors why they have or haven’t chosen your company. Ask them “Why?” several times:

  • Why did you think about purchasing?
  • Why did you click our website link?
  • Why did you choose our goods?
  • Why did you fill out the application form?
  • Why will you make another purchase with us?

As you have already understood each question may have a negative form.

I personally don’t recommend to use online-polls as people answer them thoughtlessly. It would be better to ask 10 real people individually by phone rather than cover 100 people with the help of a soulless program. A contact with a living person will give you an insight.


Solid design foundation

Experts recommend to use frameworks as a solid base for creation of design for tested pages. Models are sketches of pages used as hypotheses. It would be better to see how they look like.


They are created faster than ready design-project pages. You work on enhancing hypotheses and a scenario how a visitor reacts to this or that variant on this stage. In some cases, several real users may be involved in the process. But as for me, I don’t think that it will have a significant effect and evaluation of hypotheses and design is a complex topic and not all the clients have a sufficient level of abstract thinking so they could evaluate models.

Once you have solid analytics, report on client polls and several approved models, you’re ready for a real A/B-testing. Order creation of ready pages on the basis of your models and launch them. And then you’ll be one of the 7 companies with significant A/B-test results.

We use cookies to personalize our service and to improve your experience on the website and its subdomains. We also use this information for analytics.

Got it!