Let\u2019s imagine a scenario together:<\/p>\n\n\n\n
You want to optimize a page on your website. You think running more tests will help you get more winning results. After a few weeks, you analyze the results and can\u2019t decipher which tests led to what outcome because everything was active at the same time.<\/p>\n\n\n\n
Not only have you wasted time, but you\u2019ve also lost a portion of your budget on it. <\/p>\n\n\n\n
Contrary to popular belief, sometimes less testing is better in an optimization program.<\/p>\n\n\n\n
For each test, your goal is to validate your hypothesis with data. Data can get overwhelming, and running too many tests on the same page or audience at once can lead to poor data cleanliness.<\/p>\n\n\n\n
\n\u201cIt\u2019s important to look at behavior goals to assess why your metrics improved after a series of tests. So if you\u2019re running too many similar tests at once, it will be difficult to pinpoint and assess exactly which test led to the positive result.\u201d<\/p>\nNatalie Thomas<\/cite><\/blockquote>\n\n\n\n
<\/p>\n\n\n\n
Considerations for How Many Tests to Run<\/h2>\n\n\n\n
Test ideas should come from research. Testing is typically 80% research and 20% experimentation, so the more you research customer pain points and come up with strong hypotheses to solve them, the more you can determine quality test ideas.<\/p>\n\n\n\n
So, you might be asking what is “too many tests”?<\/p>\n\n\n\n
There\u2019s no one answer to the ideal number of tests you should run. It depends on:<\/p>\n\n\n\n
\n
- Your optimization goals<\/li>\n\n\n\n
- The complexity of your site<\/li>\n\n\n\n
- Your optimization strategy<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
Guidelines in Testing<\/h2>\n\n\n\n
While I can\u2019t tell you exactly how many tests to run, the following guidelines can help you determine if you are running too many or too few tests. As a general rule of thumb:<\/p>\n\n\n\n
If your win rate is low, you need to increase the quality and tone down the quantity.<\/p>\n\n\n\n
If your win rate is high, you\u2019re too cautious, so your testing quality and learnings won\u2019t be very meaningful.<\/p>\n\n\n\n
A \u201cgood\u201d win rate depends on what, where, and how you\u2019re testing:<\/p>\n\n\n\n
What:<\/strong> If you\u2019re testing on a site with an enormous amount of data, you might feel comfortable failing regularly because even a small win here and there has a large dollar value in the end. If you don\u2019t have a lot of data, your testing program takes time, and you\u2019ll be tempted to make sure every test counts and swings for the trees.<\/p>\n\n\n\n
Where:<\/strong> At volume, large companies can start optimizing even the smallest parts of the funnel, like the return customer dashboard and the reordering experience. Smaller organizations may want to focus on only the highest volume landing pages and the most popular products or services. Limited pages mean a limited number of tests, so your cadence or volume of tests won\u2019t be as consistent.<\/p>\n\n\n\n
How: <\/strong>For organizations with many variables contributing to the bottom line (e.g., cancellation rates, return rates), post-test analysis could take months. Long post-test analysis cycles may become a limiting factor in your testing velocity and win rate.<\/p>\n\n\n\n
If you\u2019re wondering how to find the balance between quantity and quality, the short answer is: You should only run the number of tests you can research and manage.<\/p>\n\n\n\n
Once you shift your focus from trying random tests based on your gut feeling to solving specific problems your customers face, your results will increase significantly.<\/p>\n\n\n\n