Optimization is an important thing. Apart of that is A/B testing and it can be a big scary thing if you’re just getting started with optimization of your website. Start with some basics to lay the groundwork, and keep them in mind when things get complicated.
- Nail down your most important goals before you start
Its important to decide what success is before you start an experiment. Sit down with your stakeholders for the site and have a brainstorming session of what you are trying to accomplish. Chances are your list is going to be pretty long, but you need to take from that list 1-3 primary goals before you start doing any experiments. In addition, every goal you set for an experiment needs to be clear and actionable. ‘Make the page better’ doesn’t work or ‘help people find what they are searching for’ are not clear, actionable goals. A good hypothesis has a variation (the B then your A) and clearly defined metrics.
2. Think Big
A big problem can be the complexity of A/B testing. There are so many things on any site to test, and it can be tempting to think up a million test you could do (blue v. orange header! page title variations!) Stop! Stop there. Take a step bac, and think about the big picture for your site. One of the best cases for A/B testing is a site redesign — test the alternative against the original and based off results, hone in the details of your progress. Don’t get stuck with the small stuff right off the bat.
3. Think simple
You should ask yourself a few questions when running your experiments: What are you trying to show with your idea? What key metrics should it drive? Are all of your goals and variations applicable to achieve that? There’s a good statistical reason to keep it simple: error rates are affected by the number of variations running at a time, the more tests you have the longer it will take to determine the winner/loser. For a while PlanPlus Online used a service that allowed us to understand the importance of A/B testing. They showed us how it worked, and how it would help us achieve better conversion goals. We would constantly monitor the pages and see how they were doing with a product called Unbounce. This service allows us to see if our pages are doing well against each other, create separate variations to the “winning” page to make it as good as possible.
4. You’re probably wrong
Repeat this to yourself: “What I think doesn’t matter, and what you think doesn’t matter.” A challenge that can be faced across organizations large and small is establishing a testing culture. A testing culture mean that people buy into the idea of doing test and putting their ego aside. Don’t let the HIPPO (High Paid Person’s Opinion) determine what wins or loses. You’ve got to be able to take your data and make it work so that your company will do well and your customers are hearing what they want.
5. You will fail, a lot
Failure is a part of the game. Just as you get excited for cool stuff to test, there is a reality check: most tests end in a statistical tie (no fun right?). How do you move forward after your big idea is crushed with a null result? Look at the pieces to create new hypotheses. You don’t need to just throw it all away, there may have been a tie, but maybe one segment like mobile or existing customers could have determined the winner/loser. Then you can dig into that a little further and develop new hypotheses.
6. Use qualitative feedback to find hypotheses
It’s best practice for optimization that you use both qualitative and quantitative tools. A qualitative tool is like surveys, user videos that can help you step outside the corporate mindset and hear what your real users think — it can alert you for areas of improvement on your site that you may have not seen. If you hear a problem surfaced a few times during this, create a hypothesis to test a solution and deploy it.
7. Use reliable stats platform: it’s more complicated than you think
Just like PlanPlus Online uses Unbounce, you need to be using a reliable platform. But that’s not all, you’ve got to make sure you’ve got your information good and ready to go. PlanPlus Form Builder is also a great tool to capture leads that come from your landing page platform and distribute to your sales team. Some final ideas to help make your testing better: Don’t peek. If you check all the time how your experiment is performing, you make a decision of whether or not to continue running it, and this increases the likelihood of seeing false positive.
You might be discovering a false positive rate for false discovery rate. To calculate your false positive rate you need to know the total number of A/B test combinations (variations multiplied by goals multiplied by pages) — something that is difficult to account for in testing frameworks.
Make sure you know what you’re doing and where you want to go when it comes to your A/B testing. It will help you out in the long run and increase your conversion rate of potential customers.
