Ecommerce Strategy
Dec 11, 2023
Building a Consistent Testing Program for your E-Commerce Site
A Guide to Establishing a Reliable AB Testing Framework for Consistent Growth
Drew Marconi
An e-commerce site should never be static. Every day, you are welcoming a different mix of potential customers who have come from different channels and are evaluating different competitors. In order to stay ahead, you need to be consistently testing in order to find a new edge. That applies to testing products, prices, shipping rates, discounts, website content, and UX.
That’s easy enough to say, but we find that a lot of brands struggle to be consistent with their testing. They will have bursts of energy, run a few tests, but then lose momentum after running a couple of experiments. There’s some value here, but the key to long-term value is to set up a consistent, reliable approach to AB testing that ensures you’ll always be testing things. Having a methodical, repeatable approach will ensure you’re learning as much as possible, and maximizing the potential impact of your AB testing.
In this blog post, we'll share a few of the essential steps to build a consistent testing program at an e-commerce brand, based on what we’ve seen with our most successful clients.
Keys to setting up a consistent, repeatable AB testing program
Assign a clear owner of AB testing program
Distributed responsibility kills momentum. Start by designating an owner for the testing program within your team. This individual should take the initiative to ensure that testing efforts are organized and your strategy is working. They are ultimately in charge of driving things forward
You may choose to work with a CRO agency and have them own the program. You should still have a single internal owner to ensure this relationship is generating impact (BTW - if you’re looking for a CRO agency check out our partnerships page).
Set a Goal
Make an estimate about how much revenue or profit lift you want to achieve from your testing program. We’d recommend measuring this in terms of profit-per-visitor, but revenue per visitor or conversion rate could also work. This is going to be a guess, and that’s OK – you can take a couple approaches:
Ask your peers or agencies what lift they think is possible (FWIW – Intelligems sees an average of ~6% profit per visitor lift for our customers after a year).
Sum up how much you’re “spending” on the program between any software tools, agency fees, and time that you’re dedicating to it. How much do you need to improve to cover these costs and make it worthwhile?
This step is really helpful for a couple of reasons:
It sets a goal for your owner – Maybe they won’t hit it, but it will really clarify what kind of tests should be prioritized and light a fire.
It ensures you are measuring impact from day 1, and helps get that infrastructure set up.
Brainstorm as a Team
Even though there’s a single owner, test ideas should come from all over the organization.
Before implementing any tests, gather your team for a brainstorming session. Throw ideas at the wall and build a list. You can try a brainstorming game like Crazy 8s to get people going and thinking.
The owner should compile these into a list, and encourage folks to bring more ideas over time.
Track data sources that can give new test ideas
That brainstorming will help come up with some “top down,” creative ideas about how to improve the site.
But there are also ways to generate ideas “bottom up.” For example, you may look for “weak points” in your funnel where customers drop off, and try to test new ideas in those spots. Or perhaps you can highlight really strong elements of your site and make them more prominent.
The point here is to pick a couple of analyses that you can track over time, and mine for ideas. Some tools that can be useful here are:
Google Analytics funnel views
Heatmap software that can show where customers are (or are not) clicking
Post Purchase Survey Data from Customers
Customer feedback from post-purchase surveys can be a great way to generate new hypotheses for future tests and help you iterate on current tests you’re running.
Post-purchase surveys let you gain valuable insight into your customer satisfaction, lets you understand user preferences, and identify the weak points of your testing strategy.
You can also gain qualitative insight to add to the quantitative results from your strategy, so you can dig deeper into your results and add productive changes to your tests to improve them.
Prioritize Ideas and keep a roadmap
When faced with multiple testing ideas, prioritize them using the ICE framework (Impact, Confidence, Ease). Here’s an article we found that digs deeper into this strategy.
Identify tests with the highest potential impact, those you are confident in implementing, and those that are relatively easy to execute. This prioritization ensures that your testing efforts yield meaningful results efficiently.
Keep your backlog of test ideas groomed to make sure you always know what the next test will be - and have it ready to roll. That will significantly cut down on any “dead time” between tests.
Run Tests to Statistical Significance – But Use Your Best Judgment
While reaching statistical significance is crucial for confidence in your results, avoid the trap of waiting for a 99% significance level.
Getting that last bit of confidence can take a very long time. It may make you more certain of that decision, but there’s an opportunity cost – you could be running a new test instead.
Pick a threshold – we see a lot of brands use 85%, but you can afford a higher threshold if you are large – and stick to it!
Avoid Dead Time and Stay Consistent
Treat your website traffic as your testing budget. You want to “spend” all of that budget on AB tests so that you can learn the maximum amount. Any time that you don’t have a test running is a wasted budget – you need to stay consistent, and the owner needs to be responsible for this.
There are some cases where a test shouldn't be live – e.g. a major sale period – and that’s OK, having good internal communication should make this relatively painless.
Celebrate the Wins and Learn From the Losses
Your first 3 tests may be losers – that’s OK. A single big win could provide the entire lift you need for a year, so don’t get discouraged.
You need to make sure you’re learning from the losses – what didn’t work about the test that lost? What new hypotheses does that generate? How can you use that to come up with better test ideas?
But also celebrate the wins – share with the team when a test wins, and highlight the impact so that everyone stays bought into the testing program.
Next Steps
After you’ve nailed this down, and are feeling comfortable, you can start to get into personalization and segmentation. Perhaps your test group performed great for mobile, but poor on desktop. Instead of just picking one winner, you can roll out a differentiated experience where mobile users see version A, and desktop users version B. Or perhaps you realize that you should offer a different discount to returning visitors vs. new customers.
These kinds of personalizations are possible in most AB testing tools (including Intelligems), and all comes down to understanding what your customers will react to. You can continue to develop this intuition by testing, and treating it as a continuous process instead of a one and done deal. Being in tune with your customers also helps keep them happy and makes sure that your profits are as good as they can be.
Final Thoughts
Building a successful testing program for your e-commerce business requires a strategic approach and a commitment to consistency. By appointing an owner of your testing program while keeping the team in the loop and making sure you’re listening to your customers while sticking to a well-organized plan, you’ll see results that really make an impact on your bottom line. Remember not to be discouraged if all does not go according to plan immediately, as testing is an iterative process, so chances are your first attempt won’t be your best. In the dynamic world of e-commerce, adaptability and continuous improvement are the keys to sustained success.
Ecommerce Strategy
AB Testing
Content Testing
