\u201cA\/B testing significantly improves startup performance and […] this performance effect compounds with time.\u201d \u2014 Koning, Hasan, Chatterji, Experimentation and Startup Performance: Evidence from A\/B testing<\/a><\/em><\/p>\n<\/blockquote>\n\n\n\n Given these blockbuster outcomes for experiment-led companies, it\u2019s obvious that experimentation correlates with success. But does having an experimentation team guarantee<\/em> better overall performance?<\/p>\n\n\n\n Case studies make the connection between experimentation and KPI growth seem inevitable. So, it\u2019s logical to expect that once your experimentation engine is firing on all cylinders, you should see a noticeable impact on the data. But is it practical? If I run experiments, will my conversion rates go up?<\/p>\n\n\n\n It depends.<\/p>\n\n\n\n After running thousands of experiments, we\u2019ve learned that what happens after a release isn\u2019t a simple one-to-one outcome that mirrors the success of your experiments. But the story of why is not so simple.<\/p>\n\n\n\n In this article, we explore:<\/p>\n\n\n\n If you\u2019ve ever researched an experimentation partner, you\u2019ve probably come across one that guarantees conversion rate increases. It\u2019s an enticing prospect that experimentation = an automatic increase in KPIs. Industry insiders like Shiva Manjunath are generally against this \u201csnake oil\u201d<\/a> tactic, and at The Good, we simply say anyone who guarantees they can increase your conversion rate is either lucky or lying.<\/p>\n\n\n\n There are plenty of reasons experiment results don\u2019t map one-to-one with the real-world outcomes you expect. So, let\u2019s explore three core issues that prevent you from seeing those charts go up and to the right:<\/p>\n\n\n\n Metrics are influenced by much more than a website or app experience. Direct and indirect influences impact metrics like conversion rate, revenue, and customer satisfaction.<\/p>\n\n\n\n In fact, we\u2019ve identified over 55 variables that contribute to swings in KPIs. Factors like traffic quality, seasonality, competitor promotions, and even the economy play a huge role in whether or not website visitors will convert\u2014now or in the future.<\/p>\n\n\n\n As we learned during the COVID-19 pandemic, even the largest experimentation wins may not eclipse outsized influences that dampen conversion rates. (True story: we once saw a single social media intern drive so much new traffic that conversion rates fell by a whole percentage point. Great for the intern\u2019s portfolio, bad for conversion rates.)<\/p>\n\n\n\n The opposite is also true. When outside factors are working in tandem, the results can be outstanding. We worked with Alisha Runckel during her tenure at Laird Superfood, <\/a>running hundreds of experiments together, contributing to their industry-high conversion rates.<\/p>\n\n\n\n But experimentation wasn\u2019t solely responsible for her success. Alisha and her team also made good offline decisions. Their holistic approach included direct mail, bundling incentives, and even a packaging overhaul informed by sentiment testing.<\/p>\n\n\n\n \u201cThere are no silver bullets. Ecommerce success is an accumulation of good decisions made over time.\u201d \u2013 Alisha Runckel, Laird Superfood, Humm, Hannah Andersson<\/em><\/p>\n<\/blockquote>\n\n\n\n If done correctly, A\/B tests are a trustworthy way to estimate the effect of a treatment despite shifting external influences. But once the test period is over, you no longer have a baseline or \u201ccounterfactual\u201d scenario to compare to.<\/p>\n\n\n\n Even if you\u2019re capable of running precise counterfactual scenarios, <\/a>there\u2019s no easy <\/em>way to understand what performance would have looked like if you hadn\u2019t implemented the changes. As a result, once a test period is over, we\u2019re generally ignorant of whether we\u2019ve improved a metric that was already in decline, which makes post-launch attribution fuzzy at best.<\/p>\n\n\n\n 5 + 5 = 10, right? Not in experimentation.<\/p>\n\n\n\n While there are many reasons that experiment results are not summative, let\u2019s focus on one of them today\u2014segmentation.<\/p>\n\n\n\n Many experiments only impact a segment of users, and in those cases, the observed improvement is diluted by its proportion of the larger population. Therefore, a win with a segment, or subset, of users does not predict the same gains for the whole.<\/p>\n\n\n\n Let\u2019s run a scenario to demonstrate:<\/p>\n\n\n\n When we implement the changes in production, we shouldn\u2019t expect desktop traffic to perform in an identical manner. The reasons could be many. Maybe the change isn\u2019t as conventional for desktop devices. Or maybe the experiment hypothesis only had to do with solving for mobile users\u2019 lack of patience and need for speed.<\/p>\n\n\n\n Whatever the reason, practitioners often run experiments that are only meant to impact a subset of their audience, and the rest of their visitors won\u2019t experience the same benefit. That\u2019s true for splits by device type, landing page, and similar segments, and it\u2019s why we generally don\u2019t extrapolate the results of a segment and apply them to the whole.<\/p>\n\n\n\n A 5% gain with one audience and a 5% gain with another does not equal a 10% lift overall. Test results are not summative.<\/p>\n\n\n\n The final reason (at least for today) that your overall KPIs might not match the result of your experiment is the occurrence of false positives.<\/p>\n\n\n\n False positives are what we call it when our experiment data indicates that our hypothesis is true when it actually is not.<\/p>\n\n\n\n \u201c[False positives] appear to generate an uplift but will not actually generate any increase in revenue.\u201d <\/a>\u2014\u00a0Goodson, Most Winning A\/B Tests are Illusory<\/a><\/em><\/p>\n<\/blockquote>\n\n\n\n False positives may sound like an atrocious error on the part of the experimenter, but they are actually par for the course. Even rigorous and experienced experimentation teams expect about a 26% false positive rate,<\/a> meaning about one in four \u201cwinning\u201d experiments is observed as the result of chance and not a true<\/em> winner.<\/p>\n\n\n\n To account for false positives, some practitioners even go so far as to re-test all winning experiments that fall outside of a certain threshold, then split the difference to estimate real-world effects (more on that later).<\/p>\n\n\n\n In my experience, most practitioners aren\u2019t so persnickety about false positives. Their approach is to accept that some \u201cwins\u201d are truer than others and, as a result, anticipate a slightly lower return when comparing test results to real-world outcomes. Still, those blessed with the traffic and time might consider the practice of re-running tests for improved confidence.<\/p>\n\n\n\n Does the pervasiveness of false positives mean experimentation doesn\u2019t work? Of course not. It just means that we should approach \u201cwins\u201d with an informed skepticism and not expect a one-to-one relationship between the results observed during a test period and real-world performance.<\/p>\n\n\n\n![\"A](\"https:\/\/thegood.com\/wp-content\/uploads\/The-Good-SP-experimenters-index-example.png\")
<\/figure>\n\n\n\n\n
Why don\u2019t we always see one-to-one results from our experiments?<\/h2>\n\n\n\n
\n
Problem 1: Post-launch variables make attribution less than clear.<\/h3>\n\n\n\n
![\"A](\"https:\/\/thegood.com\/wp-content\/uploads\/The-Good-Variables.png\")
<\/figure>\n\n\n\n![\"A](\"https:\/\/thegood.com\/wp-content\/uploads\/The-Good-55-Conversion-Rate-Factors-888x1024.png\")
<\/figure>\n\n\n\n\n
![\"A](\"https:\/\/thegood.com\/wp-content\/uploads\/The-Good-Counterfactual-Graph.png\")
<\/figure>\n\n\n\nProblem 2: Test results are not summative<\/h3>\n\n\n\n
\n
Problem 3: False positives<\/h3>\n\n\n\n
\n