A/B testing in Klaviyo

Overview

Data-driven decisions are the difference between good and great marketing. When it comes to figuring out what your audience will respond well to, A/B testing is your best ally. With an A/B test, you can showcase different types of copy and creative work to a subset of your audience to test what is more likely to make them complete the call-to-action (CTA) of your form or email. Let’s look at six best practices for your A/B testing to ensure you don’t leave money on the table.

A/B testing is currently only available for emails and signup forms; you cannot A/B test SMS messages.

1. Develop a Hypothesis

You can develop a hypothesis in two different ways: on a micro and macro scale. If you feel that your marketing strategy is performing well and you simply want to refine your communication with your customers, you might consider changing some of the following:

• Colors for CTA
• Text in CTA
• Subject line
• Sender name
• Number of offers
• Imagery using predictive analytics

If you’re considering making a significant change to your marketing, then you can use A/B testing to see if your new idea resonates with your audience on a macro scale. Examples might include:

• Different brand colors
• New layouts
• Change in brand voice
• Participation with a new brand community

2. Test One Variable at a Time

While you may have a handful of ideas you want to test, it’s best to test one variable at a time. For example, if you change the color of the CTA and also modify the text on it, and you see a spike in the signup rates for forms, you won’t know if the spike was due to the color change or the text on the CTA. By testing one variable at a time, you can be sure about what your audience is responding to the most.

3. Ensure Sufficient Audience

Imagine you have a great idea for a form, you A/B test it, and on the first day, the first 10 people who see the form click through and make a purchase. Over the week, the next 2000 people immediately abandon the form. If you had ended the test after the first day, you would think your form was great, even though a larger audience showed otherwise. By waiting until your results are statistically significant or until you have a good sample size of viewers, you ensure you know which form is truly better for your brand.

4. No More Than Four Variations

The same goes for the number of variations you create. The more variations you add, even while testing one variable at a time, the larger the audience you’ll need to obtain reliable results. Use no more than four variations of one variable per test to get timely results, allowing you to run more tests.

5. Do Not Edit a Live Test

Once you’ve done the work to set up your test, you should not make adjustments to your test that could influence the results. Fortunately, you cannot make changes to a live A/B test for campaigns and forms in Klaviyo, so this best practice specifically applies to flows. If you want to modify a template, first end your A/B test or cancel the sending of the email in the case of a campaign. Then restart the test to ensure you get the most accurate data.

6. Ask for Feedback

If you plan to make a significant change to your marketing, A/B testing is a great place to see if your audience responds well to new ideas. Directly asking for feedback from your customers is a useful strategy to use in conjunction with your A/B tests. A simple way to communicate with customers who are part of an A/B test is to tag those who engage with your CTA.

After tagging those who interacted with your form or email, you can send a plain text email asking them for feedback. Since you’re collecting information only from those who interacted with your form or email, you’ll also want to engage customers who are very invested in your brand. Feedback is a great way to involve your community and turn them into advocates for your brand.

Additional Resources

• A/B Testing a Campaign
• A/B Testing an Individual Flow Email
• A/B Testing Flow Branches
• A/B Testing a Signup Form

‍