{"id":106779,"date":"2024-01-22T06:55:41","date_gmt":"2024-01-22T14:55:41","guid":{"rendered":"https:\/\/thegood.com\/?post_type=insights&p=106779"},"modified":"2025-05-21T09:27:17","modified_gmt":"2025-05-21T16:27:17","slug":"what-is-peeking","status":"publish","type":"insights","link":"https:\/\/thegood.com\/insights\/what-is-peeking\/","title":{"rendered":"What Is Peeking And How Do I Avoid It?"},"content":{"rendered":"\n

Have you ever sacrificed the purity of a test to make a quicker decision?<\/p>\n\n\n\n

Maybe you\u2019ve taken a look at the data before you reached your prescribed sample size and felt the test was already giving obvious results in one direction or the other.<\/p>\n\n\n\n

Almost every product owner running tests faces this dilemma. And trust me, I get it.<\/p>\n\n\n\n

The pursuit of results at an adequate sample time takes patience, and there are plenty of reasons to try to rush to the finish line: As you wait for a test to run, you may be prolonging a negative experience on your site. Or you may be losing out on presenting a positive experience to all your visitors. You also could be struggling to hold off stakeholders who want a quick decision and implementation.<\/p>\n\n\n\n

But it\u2019s crucial to let your tests run for their pre-established amount of time. If you don\u2019t, you risk \u201cpeeking,\u201d which increases error rates, causes False Positives, and invalidates results.<\/p>\n\n\n\n

What is peeking?<\/h2>\n\n\n\n

Peeking is the act of looking at your A\/B test results with the intent to take action before the test is complete.<\/p>\n\n\n\n

Because most experiments have a 70% chance of looking \u201csignificant\u201d before they are truly done collecting sufficient data, peeking at test results too early can introduce bias and potentially alter the course of decision-making based on incomplete information. Waiting until the test is complete allows for a more accurate assessment of statistical significance.<\/p>\n\n\n\n

\"what<\/figure>\n\n\n\n

Think about it like this. If you flip a coin twice and get heads both times, you could incorrectly assume that the coin will land on heads 100% of the time.<\/p>\n\n\n\n

However, if you flipped it a statistically significant amount of times, you\u2019d get closer to the true rate of 50%.<\/p>\n\n\n\n

Testing is the same. If you don\u2019t allow enough time for an experiment to run and are constantly peeking at the results to make faster decisions, you are likely to make incorrect assumptions.<\/p>\n\n\n\n

Plenty of mistakes can be made when running a test, but peeking is trickier to avoid than the rest. It is tempting to even the most experienced experimentation practitioners.<\/p>\n\n\n\n

So, let\u2019s talk about it and how you can avoid getting caught in its crosshairs.<\/p>\n\n\n\n

How To Avoid Peeking: Set Clear Minimum Standards Before You Run A Test & Stick To Them<\/h2>\n\n\n\n

Before running a test, clearly define minimum standards to ensure the results are valid. These are the same standards to use when interpreting the results of a test.<\/p>\n\n\n\n

Pre-determine your Significance Level<\/h3>\n\n\n\n

The general rule is to let your tests reach 90+% statistical significance, but the exact number can vary slightly depending on your team\u2019s risk tolerance.<\/p>\n\n\n\n

NASA scientists likely need a 99.999999% statistical significance before feeling sure of a decision, while an ecommerce site owner might only need 85% statistical significance to feel confident in their decisions.<\/p>\n\n\n\n

Establishing the significance level helps your team come to a consensus about the level of error or False Positives you\u2019re willing to accept in your test.<\/p>\n\n\n\n