5 Steps to Creating Better A/B Testing Reports for Executives

“So which won: A or B?”

No matter how long you’ve been running a testing program—or how long your executive has overseen a testing program—it’s easy to be thrown off by this question.

Your executive, and the leadership team with whom he or she collaborates, want an answer (usually a simple, to the point one). So, suppress the urge to deliver the technically right one, which, usually, sounds something like, “Is there ever really a winner? Website testing is a mix of art and science, with nuances to every test campaign alongside an intricate interplay of KPIs. Even a null result that falls short of statistical significance can teach us something.”

The executive answer should sound more like this: “B.”

If you’re deep in the test data day-to-day, you may be finding it hard to collapse everything you know about each test into a simple “Here’s the winner!” reply. But your leadership team will value your ability to do so.

Don’t worry: All it takes is one PowerPoint slide. That’s the language of executives, and it allows you to translate the test data from complex Excel spreadsheets into a concise visual communication for an Executive Test Report. If you do this well, then any executive at a glance will be able to understand the test and determine whether “A” or “B” won.

Here’s how to do it:

1. Capture full screenshots of “A” and “B.”

One picture is worth 1,000 Excel cells. Make sure you get images of the complete context of both the “A” control and the “B” experiment versions for every test, whether on the home page, product catalog or promotional email. This lets your audience quickly see the placement and nature of the test.

2. Highlight what’s being tested.

The full screenshots are there for context. Fade them out in order to highlight the specific change between “A” and “B.” Use the transparency and shadow settings in PowerPoint to contrast the background image from the focus of the test. This helps make the difference pop.

3. Align the hypothesis to the metrics.

The saying “You can’t manage what you don’t measure” comes into play here. If your hypothesis can’t be measured in your testing tool or third-party analytics, you can’t test it. Aligning to the metrics will also help you—and your audience—easily determine the winner.

4. Show only the metrics that matter.

Yes, you may have tracked 10 KPIs, but according to the hypothesis, only one or two metrics matter. And for your executive audience, those are the metrics that matter. If you work for a retailer, for example, engagement metrics may be important to you, but, more than likely, the executive will want to know if average order value or conversion rate went up.

5. Put these pieces on one, and only one, slide.

Laura Arand at Cornerstone Brands was the first person to give me the “one test, one slide” advice. And it’s the most important part of this process. Limiting yourself to one slide ensures you will bubble up only the images and data that are most important. It also makes your presentation easy to share and understand. The heading gives this campaign context while viewed through a larger testing strategy. The “Next Step” footer indicates learnings and pending iterations.

Once you have this slide format set for one test, keep repeating it to make a full deck of your most recent test results, which you can distribute to leadership. Keep the tests with “null” results out; only focus on those A/B tests that showed a significant difference in commerce metrics.

Then, when that executive drops the “Which won?” question again, you’ll be able to show the slide. And you’ll be guaranteed to deliver the right answer.

Test report image courtesy of Shutterstock.

5 Steps to Creating Better A/B Testing Reports for Executives