KISSmetrics A/B Test Report: Know How Your Tests Impact Your Customer Acquisition Funnel


A/B tests tell you whether changes you want to make will improve your conversions. The most common example is a test that compares two website headlines. Half of the people who visit the website receive one headline (control), while the other visitors see the other headline (variant). The purpose of the test is to determine which headline wins at converting people to sign up.

We’ve written a bunch of posts and produced many webinars on the value of A/B testing. For an overview on how A/B tests work, check out these resources:

Blog Posts:


In this post, we’ll discuss running an A/B test in KISSmetrics. We’ll go through why KISSmetrics is a great tool for tracking your A/B tests. You’ll learn how to run and optimize your A/B testing process using one of the best A/B test reporting tools on the market.

Let’s get started.

Why You Should Use KISSmetrics to Track A/B Tests

With the KISSmetrics A/B Test Report, you can track how your A/B tests impact an entire funnel. You aren’t limited to running open-rate or click-through tests. Instead, you can test against actions that actually matter to your business.

You can run a test at the top of the funnel (i.e., homepage headline test) and see how it impacts actions further down the funnel (sign ups). And, you can go as deep (far down) in the funnel as you’d like. The A/B Test Report doesn’t stop at sign ups. Use the Report to measure variables that actually move the critical levers of your business.

All Your Data is Tied to People

KISSmetrics tracks real people, not visitors or sessions; real people.

Why does this matter?

Because if you want to run A/B tests that go deep in your funnel, you’ll need people-tracking analytics. Running tests for outcomes deep in the funnel typically require multiple visits on multiple devices and take an extended period of time. The only way to accomplish this accurately is to track each person.

And, since you’ve already set up tracking in KISSmetrics, you don’t need to spend additional time with another tool tracking your data. This allows you to spend less time tracking and more time building.

Less Time Tracking, More Time Building

Because you’ll spend less time tracking tests and more time building, you’ll be able to get tests out the door quicker.

This matters because the more tests you run, the higher your conversions will be. If you run high quality tests back to back, you’ll eventually find the variables that increase conversions. Then, you can build off those variables and keep testing to further increase your success. You can pitch out the variables that don’t move the needle and quickly move on to your next test.

Flexible Integration Regardless of What Tool You Use

If all your funnels are already in KISSmetrics, the only thing you need to worry about is tagging each group of test visitors with a KISSmetrics property. Each group of visitors in a test should be assigned whichever variation they were in.

If you’re using one of our integrations, KISSmetrics automatically tags visitors with the appropriate property, depending on which test they were in.

If you’re running the A/B test yourself, you can set up property tracking.

If you’re using a marketing automation, email, or landing page tool to run tests, you just need to tag each group of visitors with a KISSmetrics property.

With KISSmetrics, you have the flexibility to choose what works best for your needs. If you want to run some light A/B tests, you can use Optimizely, Visual Website Optimizer, or any of our other integrations. If you want to run a more major test, you can build the test yourself. Or, if you use a marketing automation tool such as HubSpot or Marketo, you can build your tests there and track the data in KISSmetrics.

Integration Takes Less Than 2 Minutes

If you have both your A/B testing tool and KISSmetrics code installed on your site, integrating is easy. There’s no code to add or back-end modifications to make. With many of our integrations, it’s as easy as flipping a switch or checking a box.

Using one of our integrations, you can use whichever tool you prefer to run tests, and you’ll have the benefits of using KISSmetrics to track how the tests affect your funnel.

Also, you can build your tests the way you normally do. Once you’ve integrated, the data collected from the test will also be sent to KISSmetrics.

After a test has started, you’ll need to know how long it should run. Let’s get into that now.

How to Build and Track Your Tests

To build an A/B test and track it in KISSmetrics, you can either use one of our integrations or build it yourself in KISSmetrics. We integrate with the following tools:

If you’re running the test yourself, you can easily set up tracking. Check out our documentation if you’re interested in running a test using our JavaScript library or your own server side code.

When creating the test, name the property the name of your test. You then split control people by giving them a control value on the property, and you split variant people by giving them a variant value on the same property.

At KISSmetrics, we create the test ourselves for any major test we run (full page redesigns, product redesigns, sign up flows, etc.).

The Two Steps for Integrating Optimizely into KISSmetrics

If you’re using Optimizely to build your tests, you’ll need to do a couple of things to get the data into KISSmetrics:

1. Go to the options icon, and click on “Analytics Integration”:

2. Flip “KISSmetrics Integration” to ON:

You also can go to your Optimizely dashboard, click on Settings>KISSmetrics Integration and check the box for “Integrate this project with KISSmetrics”:

The Three Steps for Integrating Visual Website Optimizer into KISSmetrics

If you’re using Visual Website Optimizer to build your tests, you’ll need to do a few things to get the data into KISSmetrics:

1. Go to the Settings tab:

2. Click on “Installation Code”:

3. Scroll down to “Web Analytics,” click on KISSmetrics, and follow the two-step instructions:

How Long to Run Your Test

Data is very volatile in the first few days of a test. Even if you’re testing on a high traffic site, you’ll still want to wait at least one week. Because the sample size is smaller in the first few days of a test, a small number of people will have a big impact.

The A/B Test Report recommends that you let a test run for at least a week. This will help ensure that sufficient data is received and that false positives have a minimal impact.

There are different rules of thumb that suggest how long you should run a test. The Report uses 95% certainty and 1 week of data before it calls a winner. At KISSmetrics, we let our tests run at least 1 week; we make sure at least 2000 people get in the test; and we wait until we reach 99% certainty.

There is an opportunity cost for letting tests run too long. For example, if you let a test that gives you a 6% boost run for 6 months, you could have found two 20-30% wins in that same time, which would have a much larger impact on your growth. At KISSmetrics, we kill tests if the impact looks like it’s less than 10% when the test has been running for 1 month.

You’ll need to find the balance between getting enough data and making sure you don’t run a test too long. If too many tests run longer than necessary, you’ll have fewer overall tests and it will take you longer to find the variables that move the needle. This is why it’s important to get your workflow set before you run tests. Before you begin a test, you’ll need to know what will determine a winner, and when it will be called.

If you’re looking for more information on choosing winners, check out our blog post where our Data Scientist Will Kurt discusses this topic.

Check Your Results in KISSmetrics

Now, let’s go back to our common example of comparing two website headlines.

We’ll wait one week, and then we’ll use the A/B Test Report to see how things are progressing. When using this Report, we first need to select the Event we’re tracking. In our case, we’re looking at how signup rates compare for our two headlines.

We’ll click the dropdown box and select the event “Signed up”:

Remember that Optimizely (and many of our other integrations) automatically saves the test name as property. If you build your own test, the name of your property will be the name of your test.

For our example test, we’ll select the property “Optimizely Homepage Headline Test GA,” which was a previous test we ran in Optimizely:

We click “Run Report” and get our data:

This is how our A/B Test Report looks. At the top, we select the baseline for our experiment. In our case, we’re choosing the original headline and comparing it against the variant. As you can see, highlighted in bright blue, the Report has declared the variation headline a winner after 4 days of running the test.

Below that declaration, we see various metrics about the number of people in the experiment, the percentage improvement, and how certain it is. “Certainty” in KISSmetrics means statistical significance. When a test is statistically significant, it basically means that the data has become reliable enough to call a winner.

Next, we see a graph. The dotted line represents how the variation headline performed against the original headline.

And, in the last section, we see the number of people who saw each headline, how many people converted, the conversion rate for each headline, the percentage of improvement, and the certainty of improvement. In this test, the variation is the winner with a 36.65% improvement over our control, and the certainty is over 99%.

Using Multiple Variants

The A/B Test Report isn’t limited to one variant. In fact, the report can handle as many variations as you need.

Here’s how the Report appears with multiple variants:

In this test, we tested two variants against the original control. We selected the original as our baseline. The test ran for 2 months, had over 13,000 people in the experiment, and did not produce a clear winner. We’ll stick with the original.

Launch the Variant If It’s a Winner with at Least 95% Certainty

As mentioned, you’ll want to launch a variant only if it has achieved certainty. Data in the first few days of a test is unstable and can change wildly with only a couple of conversions. To ensure that you reach certainty with your test, you’ll need enough people in the test to minimize false positives, or variations that appear to be winners but actually aren’t. The industry standard says it’s safe to call a test when it reaches 95% certainty. If you want to be extra careful to minimize false positives, make changes only when certainty reaches 99%. At KISSmetrics, we always wait for 99% certainty.

The Five-Step A/B Test Process

Let’s recap a few of the main points:

  1. Why use the A/B Test Report — With this Report, you’ll be able to test against actions that matter. You’re not limited to click-through or open-rate tests. You can test for outcomes that actually matter to your business, no matter how far down the funnel. You can do this because all your data gets tied back to people. This makes it a super accurate platform for A/B testing. And because you’ve got tracking set up in KISSmetrics, there is no need to spend any extra time setting up additional tracking. This matters because the more tests you run, the higher your conversions will be. You’ll find the few variables that actually matter and you’ll be able to build off of them.
  2. Easily track your A/B tools using KISSmetrics — The A/B Test Report does not create the test for you. It reports the results of the test, does the math to give you a recommendation, and ties all that back to people. We have five integrations that you can use to tie your data back to KISSmetrics. Or, if you prefer you can build your own A/B testing tool and integrate it into KISSmetrics.
  3. Run your tests until a winner is determined — Most A/B tests will take at least one week to produce a clear winner. And, typically, the data will be volatile in the first few days of a test. A couple of conversions have a bigger impact than they should. You usually have to wait at least a week, or until at least 2000 people are in the test. At KISSmetrics, we like to wait until our tests hit 99% certainty. The industry standard says you can determine a winner at 95%. Whichever percentage of certainty you choose, you’ll want to make sure a clear winner is established.
  4. Only two steps are required to run the test in KISSmetrics — The first thing you’ll need to do is select the Event you are testing. Then, select the property.
  5. Launch the variant if you achieve at least 95% certainty — It is bad A/B testing practice to declare a winner before it achieves certainty. Ensuring you reach certainty will reduce the number of false positives, or variations that appear to be winners but are not. If you don’t wait for enough certainty, you’ll lower your conversion rates accidentally. This is why many companies have difficulty raising conversion rates over time. They launch false positives too often, which eventually kills the effects of any of their real wins.

To get started using the A/B test report, login or sign up for a KISSmetrics account now.

Some of the links in the post above are “affiliate links.” This means if you click on the link and purchase the item, we will receive an affiliate commission. Regardless, we only recommend products or services we believe will add value to our readers.

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Leave a Reply

Your email address will not be published. Required fields are marked *

CommentLuv badge