{"id":101865,"date":"2022-10-20T10:32:30","date_gmt":"2022-10-20T17:32:30","guid":{"rendered":"https:\/\/thegood.com\/?post_type=insights&p=101865"},"modified":"2024-07-29T08:39:06","modified_gmt":"2024-07-29T15:39:06","slug":"cro-testing","status":"publish","type":"insights","link":"https:\/\/thegood.com\/insights\/cro-testing\/","title":{"rendered":"Too Many Tests Can Kill Experimentation Programs: Here\u2019s How"},"content":{"rendered":"\n

Key Takeaways<\/h3>

By the end of this article, you should have the knowledge and resources to \u201ccheck the box\u201d in these areas\u2026<\/p><\/div>

  • Why more testing doesn\u2019t always mean better results<\/li>
  • How you can run more of the RIGHT kind of tests<\/li>
  • Potential problems with your experimentation program<\/li><\/ul><\/div>\n\n\n\n

    Contrary to popular belief, sometimes less testing is actually better in a conversion rate optimization program.<\/p>\n\n\n\n

    Let\u2019s imagine a scenario together. You\u2019re hoping to optimize your bestselling product page for more conversions. You think more testing means more opportunities for winning results<\/a>, so you focus on the velocity of testing rather than the quality of your tests.<\/p>\n\n\n\n

    After a few weeks, you analyze the results and can\u2019t decipher which tests led to what outcome because everything was active at the same time.<\/p>\n\n\n\n

    Not only have you wasted time, you\u2019ve also lost a portion of your testing budget.<\/p>\n\n\n\n

    In order to optimize your site effectively and efficiently, you need to find the delicate balance between your testing cadence, velocity, and analysis.<\/p>\n\n\n\n

    In this article, we\u2019re going to teach you how to do just that.<\/p>\n\n\n\n

    What is the problem with running too many tests?<\/h2>\n\n\n\n

    For each test your goal is to validate your hypothesis with data, right? Ultimately, you want to know if the change will increase conversions for your business.<\/p>\n\n\n\n

    So, unless you\u2019re a giant like Amazon, running too many tests could completely wreck your experimentation program.<\/p>\n\n\n\n

    First of all, test ideas should come from data or research<\/a>. So, if you\u2019re a small team running 50 tests each month, you likely aren\u2019t running the right tests for your site. You\u2019re wasting money on the quantity of testing and you aren\u2019t spending enough time and energy coming up with quality test ideas.<\/p>\n\n\n\n

    Focusing on the quality of ideas helps you run more tests that impact the high-value goals for your brand.<\/p>\n\n\n\n

    Running too many tests on the same page or audience at once can lead to poor data cleanliness.<\/p>\n\n\n\n

    For example, if you are running five tests on your product bundling and subscription features all at the same time, with the same goal to increase the average order value, how will you know which one moves the needle? Your data is messy. The results will be difficult to attribute to a single change and your learnings may not be clear enough to provide you with a next step in your optimization roadmap<\/a>.<\/p>\n\n\n\n

    Our director of UX and Strategy, Natalie Thomas<\/a>, says:<\/p>\n\n\n\n

    \n

    \u201cIt\u2019s important to look at behavior goals to assess why your metrics improved after a series of tests. So if you\u2019re running too many similar tests at once, it will be difficult to pinpoint and assess exactly which test led to the positive result.\u201d<\/p>\n<\/blockquote>\n\n\n\n

    What does it look like when you do it right? You can run a homepage test and product page<\/a> test simultaneously with the primary goal of increasing the conversion rate. But for the secondary goal on the homepage you might be looking at bounce rate, while on the product page you are looking at add to cart.<\/p>\n\n\n\n

    In this situation, both the homepage and product page test have the same primary goal, but secondary goals are validation of which optimization actually produced the result. You can monitor these different goals to know what works.<\/p>\n\n\n\n

    How many tests should I run per month?<\/h2>\n\n\n\n

    There\u2019s no one answer to the ideal number of tests you should run. It depends on your goals, the complexity of your site, and your optimization strategy.<\/p>\n\n\n\n

    For example, The Good works with a major online media organization. Their goal is to register and subscribe new users to their service.<\/p>\n\n\n\n

    Every test we run is based on complex calculations that analyze what site elements they can afford to adjust and which test ideas will be the most profitable. After all of the research, analysis, and data exploration that we do, we end up running one test per quarter.<\/p>\n\n\n\n

    Alternatively, we have clients that have sufficient traffic and are willing to test ideas based on a bit of competitive research and user data. They end up running 3-5 tests per month.<\/p>\n\n\n\n

    While we can\u2019t tell you exactly how many tests to run, the following guidelines can help you determine if you are running too many or too few tests.<\/p>\n\n\n\n

    As a general rule of thumb:<\/p>\n\n\n\n

      \n
    • If your win rate is low, you need to increase the quality and tone down the quantity.<\/li>\n\n\n\n
    • If your win rate is high, you\u2019re too cautious, so your testing quality and learnings won\u2019t be very meaningful.<\/li>\n<\/ul>\n\n\n\n

      A \u201cgood\u201d win rate depends on what, where, and how you\u2019re testing:<\/p>\n\n\n\n

      What:<\/strong> If you\u2019re testing on a site with an insane amount of data, you might feel comfortable failing regularly because even a small win here and there has a large dollar value in the end. If you don\u2019t have a lot of data, your testing program takes time and you\u2019ll be tempted to make sure every test counts and swings for the trees.<\/p>\n\n\n\n

      Where:<\/strong> At volume, large companies can start optimizing even the smallest parts of the funnel, like the return customer dashboard and the reordering experience. Smaller organizations may want to focus on only the highest volume landing pages and most popular products. Limited pages mean a limited number of tests, so your cadence or volume of tests won\u2019t be as consistent.<\/p>\n\n\n\n

      How:<\/strong> Based on the complexity of your product ecosystem and the number of downstream variables you’re analyzing, you may determine that a simple conversion rate is enough to call a winner. For organizations with many variables contributing to the bottom line (e.g., cancellation rates, return rates), post-test analysis could take months. Long post-test analysis cycles may become a limiting factor in your testing velocity and win rate.<\/p>\n\n\n\n

      Quality vs. quantity of testing<\/h2>\n\n\n\n

      Should you run smaller and more frequent tests<\/a>? Or should you focus on bigger and fewer tests?

      The answer is you should only run the number of tests you can research and manage.<\/p>\n\n\n\n

      If you can\u2019t monitor all the tests you run it will be difficult to find out exactly which tests lead to specific results.<\/p>\n\n\n\n

      Testing is typically 80% research<\/a> and 20% experimentation, so the more you research customer pain points and come up with strong hypotheses to solve them, the more you can determine high probability conversion opportunities.<\/p>\n\n\n\n

      Additionally, the bigger the predicted change in conversion rate, the quicker you see statistical significance. Higher converting tests let you quickly identify and implement a winner, so you can move on to other tests.<\/p>\n\n\n\n

      Once you shift your focus from trying random tests based on your gut feeling to solving specific problems your customers face, your quantity AND quality of testing will increase significantly<\/p>\n\n\n\n