Here’s what they’re missing: the most successful product-led companies don’t just follow the playbook. They rewrite it based on what their actual users reveal through experimentation.

While everyone else copies best practices, companies that layer experimentation into their PLG strategy are discovering the specific insights that accelerate their growth. In a world where everyone has access to the same tactics, the ability to learn about your own users (and do it faster) becomes a moat.

Companies like Booking.com, Netflix, and Amazon didn’t achieve their dominance by following conventional wisdom. They made experimentation central to their success, running thousands of experiments annually to optimize their user experience. And you don’t need their resources to adopt their approach.

What is product-led growth?

Product-led growth is a strategy that emphasizes the product itself as the primary driver of customer acquisition, conversion, and retention.

Traditionally, companies have relied on sales and marketing tactics to create leads and drive customer adoption. Ads and websites had to do most of the selling, and the onus was on the potential user to read ads, navigate websites, choose between feature matrices, and, at times, go through a complicated sales process (on or off-site).

In a product-led growth model, companies remove as many obstacles as possible to acquiring free registered users. This approach often involves offering a free or freemium version of the product, allowing users to experience its value before committing to a paid subscription.

If the experience is good enough to keep them using it, and the paid features are valuable enough, then the hope is that users will ultimately convert into paying customers. In this way, the product serves as the main vehicle for customer acquisition and expansion.

Just like test driving a car, they let you test drive their product and discover the value on your own, before making a purchase decision.

Companies that successfully implement a product-led growth strategy often benefit from increased customer loyalty, higher conversion rates, lower customer acquisition costs, and sustainable long-term growth.

The shift from “launch and learn” to “test and learn”

Plenty of companies, between product-market fit and scale, run their growth strategies on a “launch and learn” philosophy. They build features based on hunches, ship them to users, then analyze the results afterward. This approach can work, but when operating on a product-led growth model, product decisions carry outsized impact. The product experience influences pretty much every KPI from acquisition to retention.

When you launch first and learn later, you’re essentially gambling with your users’ experience. Every poorly conceived feature, every friction point, every missed opportunity represents lost revenue and potentially churned customers. More importantly, it represents wasted development resources that could have been deployed more strategically.

This is where experimentation comes in. Instead of “launch and learn,” companies can shift to “test and learn.” This means experimentation and analysis of results happen pre-launch, not after. Changes are validated with real users before full implementation, minimizing risk and maximizing ROI.

Experimentation before implementation gives you an understanding of real customer behavior and clearly indicates how you can repeat results by uncovering the why behind those behaviors.

How experimentation amplifies PLG success

Experimentation is only helpful to a product-led growth strategy when it is done right. So what are some of the ways to implement that will amplify PLG success?

1. Systematic optimization across the customer journey

The most effective approach to PLG experimentation uses frameworks like ROPES (Registration, Onboarding, Product, Evangelize, Save) to systematically optimize each stage of the customer experience. Rather than randomly testing features, successful companies identify specific levers within each stage and experiment systematically.

For example:

• Registration phase: Testing form length, social proof elements, and value propositions
• Onboarding phase: Experimenting with tutorial formats, progress indicators, and time-to-value optimization
• Product phase: Testing feature discoverability, UI changes, and user flow improvements
• Evangelize phase: Optimizing sharing mechanisms, referral programs, and viral loops
• Save phase: Testing retention tactics, upgrade prompts, and churn prevention strategies

This systematic approach ensures that experimentation efforts are strategic rather than scattered, creating compounding improvements across the entire user journey.

2. Accelerated learning through parallel testing

Traditional A/B testing approaches test one hypothesis at a time, which can drastically slow your learning velocity. Advanced PLG companies run multiple experiments simultaneously across different parts of their product experience, dramatically increasing the rate at which they gather insights.

The key to successful parallel testing is ensuring experiments don’t interfere with each other. As Natalie Thomas, our Director of UX and Strategy, explains: “It’s important to look at behavior goals to assess why your metrics improved after a series of tests. So if you’re running too many similar tests at once, it will be difficult to pinpoint and assess exactly which test led to the positive result.”

Successful parallel testing requires:

• Creating testing roadmaps that cover independent product areas
• Building small, cross-functional teams assigned to each area
• Establishing clear metrics and success criteria for each test
• Implementing proper statistical controls to avoid interference

3. Rapid experimentation for faster innovation

Speed matters in PLG. Market opportunities disappear quickly, and user expectations evolve constantly.

So, one of the main objections to implementing an experimentation strategy is that testing cycles often take weeks or months to complete. But high-performing PLG companies have found ways to cut this time in half without losing statistical rigor. Key strategies include:

Supplementing A/B Tests with Rapid Testing: Not every hypothesis requires a full A/B test. Qualitative research, user interviews, and rapid prototyping can validate concepts quickly before investing in development.

Modular Testing Approaches: Instead of starting from scratch each time, successful teams create reusable components like design templates, testing frameworks, and analysis processes to reduce setup time.

AI-Powered Research: Using artificial intelligence as a research assistant to speed up data collection, user recruitment, and insight generation.

Prioritization Frameworks: Implementing systematic prioritization (like the ADVIS’R framework) to ensure high-impact experiments get fast-tracked through the process.

4. Data-driven feature development

Experimentation helps PLG companies avoid the biggest roadmap mistake: prioritizing low-impact features. Instead of building what seems logical, experimentation reveals what actually drives user behavior and business metrics.

This is particularly important as you scale beyond basic PLG practices. When you’re competing with other product-led companies, the quality of your feature decisions becomes a key differentiator. Companies that systematically test and validate features before full development consistently outperform those that rely on intuition.

The most successful approach combines quantitative testing with qualitative insights. This means not just measuring what users do, but understanding why they do it. This deeper understanding enables teams to build features that truly resonate with users rather than features that just check boxes.

5. Building an experimentation-first culture

An outcome of adding experimentation to a product-led growth strategy is that it will help build the practice into your company culture. To do that, you can follow a few key steps.

Start with infrastructure

Before you can effectively use experimentation to support PLG, you need the right infrastructure. This includes:

• Testing platforms that can handle both simple A/B tests and complex multivariate experiments
• Analytics systems that provide real-time insights into user behavior
• Data pipelines that connect user actions to business outcomes
• Collaboration tools that enable cross-functional teams to work together effectively

Establish clear processes

Successful experimentation requires discipline. Teams need clear processes for:

• Hypothesis formation and validation
• Test design and statistical planning
• Resource allocation and project management
• Results analysis and decision-making
• Knowledge sharing and organizational learning

Foster cross-functional collaboration

The most impactful experiments often come from unexpected sources. Engineers closest to the code understand technical constraints and opportunities. Designers see user experience friction points. Customer success teams hear directly from users about pain points.

Creating space for these diverse perspectives to contribute to experimentation efforts often leads to breakthrough insights that no single team would discover independently.

The compound effect of systematic experimentation

What makes experimentation so powerful for PLG companies is its compound effect. Each successful experiment doesn’t just improve one metric. It teaches you something about your users that informs future experiments.

Over time, this creates an accelerating cycle of improvement. Companies that have been systematically experimenting for years possess a deep, nuanced understanding of their users that newcomers can’t easily replicate. This understanding becomes a sustainable competitive advantage.

Moreover, experimentation capabilities themselves improve with practice. Teams get faster at designing tests, more sophisticated in their analysis, and better at translating insights into action. The infrastructure and culture that support experimentation become organizational assets that compound over time.

Experimentation as your PLG multiplier

Product-led growth without experimentation is like driving with your eyes closed. You might reach your destination, but probably not efficiently, and certainly not safely. Experimentation transforms PLG from a collection of best practices into a systematic approach to user-centered product development.

The companies that win in today’s competitive SaaS landscape aren’t just those with the best products; they’re those that can consistently improve their products based on real user insights. They’ve made experimentation not just a tactic, but a core organizational capability.

Ready to transform your PLG strategy with systematic experimentation? The Good specializes in helping product-led companies build experimentation capabilities that drive sustainable growth.

Our Digital Experience Optimization Program combines strategic frameworks like ROPES with hands-on experimentation support to help you uncover the specific insights your business needs to scale. Let’s explore how experimentation can accelerate your growth →

The post How Does Experimentation Support Product-Led Growth? appeared first on The Good.

So, as a leader, it’s important to adapt. Kalah Arsenault knows this well.

Over the course of her career, she’s been tasked with everything from turning data into actionable insights and advocating for data-driven analysis to building experimentation programs.

Currently, she leads the marketing optimization team at Autodesk, the global leader in 3D design, engineering, and entertainment software.

We had the chance to sit down with her and get the inside scoop on:

• Standing up an A/B testing program
• A simple prioritization model making an impact
• Measuring and circulating optimization learnings

Marketing optimization for a leading software company

As the marketing optimization team lead, Kalah digs into all the nooks and crannies of the company’s marketing efforts to make it more effective and efficient.

The marketing optimization team at Autodesk sits on the operations team at the intersection between marketing operations and technology. Partnering with marketing teams to improve campaign effectiveness, Kalah and the marketing optimization team bridge the gap between data, marketing know-how, and testing expertise.

When shakeups a few years ago halted all A/B testing on the Autodesk website, Kalah was eager to partner with the website team to re-enable experimentation. A self-proclaimed marketing, analytics & optimization enthusiast, Kalah brings a consistent data-backed ethos to her work. And her background tee’d her up for success. Kalah jump-started her professional life in advertising and ecommerce. The experience working in stakeholder-facing roles gave her a unique ability to turn data into stories and prove the value of iterating your way to success.

Standing up an A/B testing program

The challenge was clear. Without an experimentation program in place, the team was left without the data needed to fuel good decision-making.

“The data will tell you what is the right choice and it takes decision-making out of the process,” she said when asked how data plays a role in her decision-making. It can even go so far as to be said that they don’t just affect the process, they are the process. “Experimentation and data can be the decision-making process.”

So, it was crucial to get the A/B testing program back on its feet in order to bring that clarity to the work she was doing day-to-day.

To start, Kalah and her team put their experience into practice, creating an A/B testing roadmap. This was a crucial step, requiring them to define goals, align with stakeholders, and assess priorities and risks of optimization. Because of a new organizational structure, on top of the complexity of rebuilding the A/B testing program, there was an added obstacle to working across different marketing teams.

The optimization and web teams worked together to establish clear parameters, agreements, and definitions of what could or could not be tested. There is now a huge, pre-approved sandbox to play in, allowing optimizers the chance to find iterations that improve UX and marketing KPIs.

Whether you’re a researcher, an analyst, a marketer, or an optimization specialist, a well-made roadmap connects you with the clear steps needed to begin experimenting.

For Kalah, this meant:

• Identifying objectives for the testing program
• Establishing marketing and website challenges
• Isolating testing opportunities
• Formulating testing hypotheses
• Prioritizing testing opportunities

With frameworks in place, they were ready to get back to work.

While other optimization leaders can follow a similar strategy of aligning with stakeholders and building a roadmap, standing up an A/B testing program is no small feat. So, if you don’t have the resources or a dedicated team like Autodesk, she has some advice.

“What I primarily suggest is hiring someone who specializes in the practice. I think the expertise to identify optimization opportunities, design the tests, see it through implementation, measure the results, and provide recommendations and next steps is incredibly impactful.”

And while there are some savvy marketers that can do this, she emphasizes that “it's a separate skill set and expertise.” So whether you hire for that as a full-time role or you look to agencies to bring that expertise, Kalah strongly recommends companies consider experts to lead the charge.

For example, “at a high level, a test may show one version outperforming another,” she says. “But digging deeper often reveals different results by segment, whether by job profile, country, or industry. We aim to look beyond primary KPIs to fully understand what’s driving the outcome.” That level of nuance is hard to find in a busy marketer, so it’s best to have dedicated optimizers around who can take the time to know and understand audiences.

A prioritization model to drive velocity

With the A/B testing program back up and running, Kalah and her team had their plates full finding efficiencies and improvements across Autodesk’s marketing efforts.

Not only were there opportunities identified in their research, but teams across the organization were submitting requests and ideas for their consideration.

The list was long. The optimization team wasn’t sure what the “most important first thing” to work on was, and marketing stakeholders didn’t understand why their projects weren’t top of the list for testing. There was an opportunity to clarify and get more done quickly.

The solution? A prioritization model aimed at:

• Increase testing volume
• Aligning teams
• Saving time

While lots of testing folks would hear “prioritization model” and go straight to the mathematical elements, Kalah needed a model that was simple, easy to calculate, and transparent for all parties.

Kalah and her team built out an auto-calculated prioritization model as part of their optimization requests intake process. It involves three elements:

• Business impact: Measured based on whether the request aligns with the marketing plan, which is agreed upon by everyone from CMO to entry-level marketing team member.
• Level of effort: Internal criteria that identify a higher or lower level of effort.
• Urgency: Assess the request with questions like: Does it need to be executed immediately? Does it impact a larger project immediately? Does this effort have backing from a senior leader in marketing?

The intake process asks questions related to the criteria mentioned, and then logic set up in Asana auto-calculates the prioritization of the experiment or optimization. “This is what saves us time and energy,” she says. It eliminates looping conversations and time to manually prioritize things amongst the team.

Kalah emphasizes the power of this setup. “We don't do mathematical calculations to assess the level of business impact or length of time to reach statistical significance. That's too resource-intensive, and we'd be spending all our time assessing and prioritizing. With our automated prioritization model, we can spend our time on launching and analyzing tests and making business impact.”

And it worked.

“We were able to double the amount of tests our team took on within one year. So from this compared to last year we doubled the volume of testing with a new operating and prioritization model.”

Measuring marketing optimization success

The volume of tests is just one of the key metrics Kalah identified for measuring her team’s success.

It can be tough to find just one KPI to prove the value of optimization, given the nature of working across teams, products, and audiences. So, instead of focusing on 1:1 measurement, they look at a variety of metrics, including:

• Volume of tests
• Volume of analyses
• Customer satisfaction score
• How many marketers are seeing/learning from the findings

In the end, her team’s goal is to look at how marketing campaigns are performing and then give advice on how to make them better. So, while each test or optimization has its own KPI related to growth, as a team they are measured more holistically.

With prioritization and test volume locked down, she is ready to move the needle on insights shared.

“I'd really love to put more energy towards amplifying the impact of each test and getting the findings out to as many marketing teams as possible. We've already seen this working through a newsletter that's sharing our testing results and analysis work. We've also been hosting quarterly brown bag style meetings with the most universally applicable test results that marketers could implement themselves.”

This year, Kalah is also hoping to find new ways to turn insights into action. “I am also hoping to dive into data visualizations and figuring out how to make our findings more snackable and basically getting to a place where people want to read them and it's easy and enjoyable.”

These goals directly align with her team’s measurements for success. Other optimization leaders can take a page from Kalah’s playbook here by letting individual tests focus on the marketing metrics and determine departmental success based on insights, experiments, or other relevant measurements.

How can you replicate some of Kalah’s success?

Kalah’s advice to those new to optimization is simple yet impactful: start small, and stay curious. “Get to know your data, experiment with tools, and don’t be afraid to make tweaks,” she says. “You might be surprised at the impact even small changes can have.”

Her overarching message is one of optimism and opportunity. “Optimization is about evolving and improving—for your customers, your organization, and yourself,” she concludes.

Yet, good optimization leaders know that you can’t do it all alone. Internally, Kalah’s team employs a mix of full-time employees, contractors, and agency partners to meet the demands of scaling optimization efforts. “Contractors and agencies can help manage peaks in the workload,” she notes.

“I come from an agency background. I've always been a fan of working with full-time employees, but I realized as we're trying to scale and grow the amount of impact we're making as a team, it's really important to have contractors or agency partners to support higher demand and the peaks and valleys of work.”

By embracing a data-driven mindset, prioritizing strategically, and fostering cross-team collaboration, Kalah exemplifies what it means to lead impactful optimization efforts. If you need an expert partner to help manage a robust roadmap, get to know our Digital Experience Optimization Program.

The post How Kalah Arsenault’s Team Stood Up An A/B Testing Program & Doubled Volume With A New Prioritization Model appeared first on The Good.

About This Episode:

The benefits of experimentation are well-researched and documented. But that doesn’t always equate to an increasing conversion rate.

In this episode, Jon and Ryan explore why experimentation results don’t always map one-to-one with the real-world outcomes you expect.

Check out the full episode to learn:

• How post-launch variables make attribution less than clear.
• Why experiment segmentation means test results aren’t summative.
• The effect of false positives.

Experimentation done right is associated with increased overall performance across a number of factors. It can be a catalyst for better decision-making and can help assure your digital property is performing at its best.

If you have questions, ideas, or feedback to share, connect with us on LinkedIn. We’re Jon MacDonald and Ryan Garrow.

• Apple Podcasts
• Google Podcasts
• Spotify
• Castbox
• Castro
• Overcast
• Pocket Casts
• Stitcher
• Subscribe via RSS

Episode Transcript:

Announcer: [00:00:00] You’re listening to Drive & Convert a podcast about helping online brands to build a better e commerce growth engine with Jon MacDonald and Ryan Garrow.

Ryan Garrow: All right, Jon, we know that website or apps should always benefit from experimentation. It’s been documented, you’ve written books that have mentioned this multiple times.

At this point, if you’re episode 120 in with us and you haven’t realized this, we failed miserably. All right. So we agree that experimentation is great, but we know it doesn’t always equate to increasing conversion rates. And there’s some frustration that we’ve had people come to both of us and say, Hey, I’ve been doing this Experiments.

Why are my conversion rates not going up? And so today we get to talk about some of the why behind that answer some of these questions at scale, because two to many is much more efficient. You won’t always see [00:01:00] direct correlation across the entire website. Conversion rates go up every time you do an experiment.

So question one for you today would be, Jon, does having an experiment experimentation team guarantee better overall performance?

Jon MacDonald: Yeah, that’s a great question. And I’ll say this. Selfishly, I just want to record this so I can send it to everybody who asks me this question because I think I feel like I get this every day.

Lead to come in saying, we’re doing testing. It’s not working. And we want to change providers and, or our clients who are saying, Hey, we have all these other metrics that look great, but conversion rates not increasing. What’s going on. And so I’d love to, selfishly just be able to send them a recording and say, to this, but here’s the reality

Ryan Garrow: can pull these out and we can get some LinkedIn snippets, all these fun things.

Jon MacDonald: Awesome. There we go. Let’s spread the good word. The reality, Ryan, is that all else being equal, once startups begin experimentation, they really see strong gains within the first few years, right? [00:02:00] It’s well documented, they see the gains, and they continue to see that growth for years to come.

I’m not just saying this, I’m not just blowing smoke here, this is actually according to Bloomberg, right? And they’ve done some research, and what they did is they formed a, what they call, experimentation index. So these are firms, large companies that are doing digital experimentation within their companies, generally at a high level A B testing, but lots of other validation techniques as well.

And they compared that to the S&P 500, okay? Now, we’ll put this up on our site at thegood. com along with an article around all this, but I’ll show you the chart. The growth divide is actually pretty large and it’s only getting larger over time. So the reality is, does experimentation guarantee better overall performance and, it’s really logical to expect that once experimentation engine is firing on all cylinders, you’re [00:03:00] doing an internal team, you’re running experiments often that you should really see a noticeable impact on the data. So a common question really is, if I run experiments, will my conversion rates go up? And you’ll hear this all the time, as I mentioned. So the reality is that after running thousands of experiments, we’ve learned that what happens after a release, Of what you tested is not a simple one to one outcome that mirrors the success of the experiment, but the story of why it’s not so simple.

That’s why I wanted to talk about this today.

Ryan Garrow: Fascinating that, it’s also surprising to me that people haven’t gotten to the point where they understand that there are so many moving pieces on a site that one metric doesn’t work in a vacuum yes, you’re converting your, you worked on your category pages.

Great. Okay. If you’re smart, you’re going to move up the funnel and push harder where conversion rates are lower. And so you get more traffic and your business grows, but you won’t necessarily work to expect [00:04:00] conversion rate. Wow. Okay. Sorry, I’m probably jumping ahead, but it’s, I have no, this is the reality of the

Jon MacDonald: conversations, right?

That you and I have every day where folks have become into these expectations. And this is why we’ve moved on from CRO to what we’re calling digital experience optimization, because. We’re not just impacting conversion rate, right? We are impacting the entire digital experience, which. Is we have a chart, in fact, I’ll turn it even in this article, there is this chart that we put together.

It’s a whole wheel of all the areas that experimentation can impact in time in the digital experience. And if you look at that conversion rate is a very small sliver of that by it just is. So it is one thing in a slew of metrics that provide a return on investment here. And if you just focus on conversion rate, you’re probably doing yourself a disservice in terms of the longevity, [00:05:00] it’s, et cetera.

There’s just a laundry list of reasons why it’s a bad idea.

Ryan Garrow: Okay. So why is it that we can’t see that? Cause ideally we’d like to see that. And maybe 10 years ago when somebody was. Promising results, somebody could see it in a one off, but why can’t we see that anymore?

Jon MacDonald: It’s funny you phrased it that way. And it’s funny in a good way, because if you’ve ever researched an experimentation partner, you’ve probably come across them that will guarantee conversion rate increases. And at the good, we have a phrase. Because we often have leads come in and say I talked to so and they guaranteed this lift.

If you can’t guarantee that, then we’re not going to work with you. And the phrase that we have is simply anyone who says they can guarantee an increase in your conversion rates is either really lucky. Or they’re lying. So you can choose which one do you want to be a part of the lucky crowd or do you want somebody lies to you?

I don’t know, but it’s one of those two. Okay. And there are plenty of reasons [00:06:00] why experiment results don’t map one to one with that real world outcome. that you would expect once you launch the variant that won in the testing. So let’s explore these really, I say there’s three key issues that prevent you from seeing the charts go up into the right.

Okay. There is that post launch variables. Okay. There’s experimentation, segmentation, and there’s the effect of false positives. So post launch variables, let’s just call it segmentation and the effect of false positives. Okay. So I want to talk about each of those three today.

Ryan Garrow: Got it. I love that. Jon, he simplifies things wonderfully, which I appreciate.

You’re also way more, probably political in your responses. Be like, would somebody guarantee something on my side of the equation on traffic? I’m like, if they’re guaranteeing your results on your traffic, driving on Google or Microsoft or meta, they’re idiots. I’m sorry. You can’t do that without looking at the account, understanding the nuances of a company and then guarantee [00:07:00] results like no, they’re idiots.

That’s me versus .

Jon MacDonald: And like you said earlier, it’s one metric in a vacuum, it does not work. So yeah, we’ll break these down. But yes, luckier line is as far down that, that non PC path is. Yeah, we’ll stay on the PC path

Ryan Garrow: with Jon and, he being the nicer guy in the, in this equation here.

Problem one with not getting a one to one result in your experiments would be post launch. And so what is it about post launch variables that muck up the results being extremely clear and like obvious?

Jon MacDonald: Yeah. Yeah. So post launch variables, they really make attribution less clear, right? So metrics are influenced by much more than the website or app experience.

And I hate to say that because that’s the only part that we can affect is the website or app experience, right? Like directly direct and indirect influences really impact metrics, like your conversion rate. Obvious one, revenue, maybe [00:08:00] even as broad as customer satisfaction. In fact, we’ve identified over 55 of these variables that contribute to swings in your KPIs.

And this is that pie chart I was talking about. That’s that graph. There’s 55 variables on there. That’s, and then we probably should come up with more, but honestly, after 55, we were like, I think people will get the point.

Ryan Garrow: That’s a lot. That’s a lot of variables.

Jon MacDonald: But, there’s factors like traffic quality.

If you’re just not getting the right traffic, they’re not going to convert or seasonality. Maybe you sell winter coats and it’s a hundred degrees out, God bless you with global warming. You might want to look at t shirts. There’s competitor promotions, maybe your biggest competitor ran a 50 percent off sale and it just took away all of your customers for that time frame.

Even the greater economy, I keep hearing right now from so many e com brands. That are really struggling, and it’s not because of their [00:09:00] product or their marketing or any of these other factors. It’s just the economy has shifted. The COVID years are over, folks. People aren’t going to only buy on your website anymore.

These all play a really huge role in whether or not your website visitors will convert now or in the future. And, I’m here. As we learned during the COVID pandemic, right? Even the largest experimentation wins may not eclipse the outside influences. That can either gain or hurt your conversion rates.

A really good way to emphasize this is this true story from one of our clients. That we once saw a single social media intern drive so much new traffic that conversion rates fell by a whole percentage point. And this is on a very large frame, okay? The intern was like, yes, I did my job. Amazing for their portfolio, their resume to be like, here are the metrics of, what we did, right?

The [00:10:00] problem is, it’s really bad for conversion rates. Because the problem is that they drove all this unqualified traffic. It’s probably a great meme, they got everyone there, but they weren’t going to buy, right? And we couldn’t control that, right? Once a test period is over, we generally are unable to tell whether we’ve improved a metric that was already rising or already in decline.

Which really makes post launch attribution fuzzy at best. And again, this goes back to anyone who told you otherwise is either lucky or a lie.

Ryan Garrow: Dang. Love to be that intern that screws up all your work.

Jon MacDonald: It was good for them, I’m sure.

Ryan Garrow: But, yeah. Oh man, a whole percentage point. That’s massive. That’s really cool, dawg.

Made a great problem to have. You have too much traffic and they’re, the middle, upper funnel is buzzing about your brand. Man, that’d be great.

Jon MacDonald: Yeah, it’s great, but it’s not going to significantly increase. It’s revenue right away, unless it’s really qualified traffic right now. [00:11:00] It’s great that the brand got out there, right?

It’s great that maybe the long tail of this is more revenue, but in the immediate future when we get a call and say, Hey, our conversion rate dropped 1 percent this month, what’s what gives you’re supposed to be in charge of that for us. We start digging into it. We’re like, wow, this one post got you several million visits.

Like good for that. That’s not qualified traffic.

Ryan Garrow: Got it. Okay. Then the next problem you mentioned was the test results weren’t summative. They didn’t, five plus five equals 10, except when you were doing an experiment. So tell us more about that. Like, how does that work?

Jon MacDonald: That’s just not math, right? There are many reasons that experiment results are not summative. And I just, let’s just focus on one of them today, which is segmentation. Okay. So many experiments only impact a segment of users. Can we agree on that? Like you’re not test, you’re not going to test with every single visitor that comes in.

And [00:12:00] even if you do, you need a control set who aren’t going to have it. So at the best, you have your control and your variable, and that’s the easiest possible test. It never works out that way. There’s always a sub, a segment of traffic, of sub subset, excuse me, of traffic that you’re gonna run a test with, right?

So a win with that segment or subset of users does not predict the same gains from the whole, right? It needs to be said. It’s obvious. I think a lot of brands don’t go in with that mindset. So as a practitioner you’re often running experiments that are only meant to impact that subset of an audience.

And the rest of the visitors won’t experience that same benefit. That’s just, that’s testing, that’s experimentation, right? And this is true for splits by device type, by Landing page by, I don’t know, and similar type of segments. There’s hundreds of ways you can segment, right? [00:13:00] And that’s why we generally don’t actually the results of a segment and apply them to the whole, right?

So a 5 percent gain with one audience and a 5 percent gain with another does not equal a 10 percent lift overall. That’s where the five plus five does not equal 10, right? Basically test results are not summative. They don’t add up in that way. Okay. And you need to have that on his view.

Announcer: You’re listening to Drive & Convert, a podcast focused on ecommerce growth.

Your hosts are Jon MacDonald, founder of The Good, a conversion rate optimization agency that works with ecommerce brands to help convert more of their visitors into buyers. And Ryan Garrow of Logical Position, a digital marketing agency offering pay-per-click management, search engine optimization, and website design services to brands of all sizes.

Ryan Garrow: so if I’m going to simplify it from my brain thinking about traffic, if we’re running an experiment on these [00:14:00] five product pages that we’re driving traffic to, but we’re also driving traffic to, let’s say 10, 000 other products on the site.

Just because the experiment does lift conversion rates on these five products, 5 percent each on average, doesn’t mean that the traffic on all 10, 000 product pages is going to increase at the same level.

Just because there’s so many different purchase attempts depending on the product and the price point. Okay.

Jon MacDonald: Yeah. Yeah. It’s the same thing. It applies to users, visitors as it applies to product pages. There’s so many ways to segment the testing. And so that, that’s another way to segment and definitely makes sense.

Ryan Garrow: Okay. Then the third piece of the whole issues you’re simplifying is false positives. Like you think something’s going to work and then it doesn’t.

Jon MacDonald: Yeah. That’s a great way to to think about it. It’s false positives. Everyone in experiment data indicates that. The hypothesis is true, but when it’s actually not right, so false positives may sound like an atrocious error on the part of the person writing the experiments, but [00:15:00] they’re actually just par for the course.

You just got to get used to them and understand. And that’s why even rigorous and experienced experimentation teams really expect at least a quarter. I have a false positive rate, meaning one in four winning experiments is observed as the result of chance and not a true winner. And that goes with all of science, I think the question here is to be thinking about is, does this pervasiveness of false positives mean experimentation does not work?

And I think the answer is, of course, not. I’ll take the 75 percent any day, but it just means we need to approach these wins with an informed skepticism, right? I don’t expect a one to one relationship between the results that are observed during the test period and real world experience of performance.

I think that the best way to look at this is to say, okay, I’m going to run this test. And if it wins, I expect some lift on the site. There will be gains. I don’t know what that [00:16:00] gain will be yet. But we’ve proven out, I feel very comfortable saying there is a gain to be had here and not a loss.

Ryan Garrow: No, it’s, I like that.

And I am surprised that one out of four have false positives. That’s just me not knowing enough about DxO. But that also tells me you’re playing a numbers game like just everything else in Ecom. Yeah. Instead of doing just four experiments and getting frustrated. You got to do a thousand experiments and you’ve got 750 that actually did something right and are taking your brand to a better level.

And so I guess what’s your response then? I guess running experience is not going to fix all your problems. It’s not the one single thing, right? That’s your silver bullet for all of these problems on your site.

Jon MacDonald: Yeah, look, It’s tempting to look at fuzzy post launch attribution and false positives and say experimentation is not really worth it.

I want to be clear, and I’m throwing all the negatives out here today, but I believe that it is very much worth it. And I want to be clear that, there are people much [00:17:00] smarter than myself that swear by a test everything approach, right? Because there’s simply no better or even a more rigorous way to.

Quantify the impact of changes in your bottom line. Now, I don’t think that there’s a test everything approach that is a good fit for any brand, but I do think that having that mentality can be beneficial. Now, problems, they do arise when we tout experimentation as a tool. This omnipotent growth lever, the magic bullet to success, whatever you want to call it, silver bullet, as you mentioned, right?

And that’s because like it can increase your confidence in decision making. Okay, you have great data to back up a decision. That’s really helpful in the boardroom and beyond, right? It can help you measure the discrete impact of good design. So you don’t just have a creative director, or I don’t want to pick on creative directors.

There’s a lot of great ones out there, or just some creative who says, This is beautiful. Everyone loved it, right? You’re able to put numbers to that, right? [00:18:00] And it can also settle internal debates about what’s direction to head, right? You’re able to say I love that your spouse thinks this is a great color, but our users who actually are paying us every day think this is the best color.

And here’s the data behind that, right?

Ryan Garrow: That’s a difficult battle to fight. Let me tell you,

Jon MacDonald: it’s a difficult battle, but one where I’ve had to fight before.

Ryan Garrow: Yes.

Jon MacDonald: Still have the battle scars, if you can’t tell, but I think, what experimentation won’t do is compensate for all of the external forces that are going to hamper your business.

If you’re looking for that confidence, the precision. Experimentation is just an incredible tool and it’s going to add to your toolkit. But if you’re looking for a silver bullet I’m still looking too, Ryan. So hopefully we can find it together.

Ryan Garrow: If it exists, we will eventually find it. I just have my doubts.

Jon MacDonald: Maybe episode 220 and we’ll have found it, but not by one.

Ryan Garrow: then we’ll be on a beach sipping margaritas and we won’t tell everybody because. It would ruin our margarita [00:19:00] life. Love it. I love you. What do you say then when somebody is frustrated that they’ve been running experiments? So what’s your response to your clients?

When at that’s the case Hey, we’re doing all this work. We’re paying you all this money. And my conversion rate is that’s got to be frustration.

Jon MacDonald: I totally understand the frustration if they are looking in it, just conversion and that’s why it’s really important to have this conversation up front.

And I try to make sure I set those expectations appropriately. And with every lead that comes in and every conversation I have. Because they really need to trust the process and use experimentation to its full potential. Which is not just for conversion rate increases. Look, one thing has been proven time and time again.

Experimentation, done right. Is associated with increased overall performance across so many factors. Experimentation can’t combat outsized economic and environmental factors. It can be a [00:20:00] catalyst for better decision making and it can help assure you that your digital property is going to perform at its best, despite whatever is going on the outside and.

That’s where optimization really comes in. And I think where it’s really valuable.

Ryan Garrow: Yeah. And I think that it becomes challenging, especially as we get into this, whatever this new economic time period looks like or how long it lasts. Top line shrinks, which generally shrinks bottom line, which generally shrinks experimentation budgets, which, and then it becomes a possible downward spiral that unfortunately you have to have faith that if you keep pushing forward.

You will, your product, your site will continue to get better and better, and you can better fight a shrinking pie because you can get more aggressive and capture more of it. But what I see happen often is you get, things are bad, cut marketing, cut all these costs. And I’m like, there’s probably some fat to be cut, but you can’t cut the core things that are helping drive your business because your [00:21:00] competitors are going to smoke you if they’re.

Still investing in experimentation allows them to get more aggressive on their marketing. Just

Jon MacDonald: think about the going back to one of the first points I made the experimentation index that Bloomberg put together versus the S& P 500. And that chart will show you very quickly the gains that experimentation teams are able to provide on top of a normal S& P 500.

And that gap is only widening. But if you aren’t doing experimentation, or you cut it because you feel like, hey, I need to cut the fat and you don’t see conversion rate gains immediately, understand that there are a couple things at play here, right? And it’s not the experimentation that’s causing those.

It could be the outside factors. And maybe we’re, you’re, experimentation is helping out with all these other 54 plus possible [00:22:00] metrics that are conversion rate and that those potentially are, Providing value and a good return on your investment.

Ryan Garrow: Yeah, I agree with all of that. And thank you for enlightening me.

Is there a, is there anything you can do to help guide clients to have kind of a vacuum set within their data to say we did get, yes. It’s flat, but if you look at, this data in a vacuum, which is a very small subset, it did get better. Or does that even exist?

Jon MacDonald: This is why we came out with our five factor scorecard.

And I think we’ve done an episode on that in the past, but if you go to the good. com and click through the big blue button in the top right of every page, just go click on that. You can learn about it, but there are five areas that our team has determined that really benefit from experimentation. And should be the scorecard for, are you getting a return on this investment?

And it’s not about looking at just conversion rate or a specific metric. It’s about looking at more [00:23:00] holistic gains. And so I would, we really don’t have time today to go in the entire, into the entire scorecard, but I would say that is what we’ve done. At the good. Now, there are probably other methods out there as well, and I’m sure there are this is the one we found that these five areas really matter to brands that on that experimentation index, they’ve all done these and excelled at these.

And if you aren’t doing these and aren’t excelling in those five areas. Then, an experimentation program or optimizing for your digital experience can really help improve those. And that’s everything from getting buy in from your team on, okay, we’re going to continue to do optimization, or we are going to start doing it, to resourcing it correctly.

You could run, I’ve seen brands run a lot of experimentation, but never implement anything, right? Because they can’t implement or maybe they have some custom platform that if they implement anything it’s just, it’s like a ball of [00:24:00] yarn and they’re afraid they’re going to pull the wrong string and not everything up.

So there’s a lot of these types of things you got to consider and, again, If there’s any takeaway from today’s discussion, it’s purely that measuring your optimization on one metric alone. is unlikely to get you to want to do optimization. And if you don’t do optimization, you are at a severe disadvantage to all of your competitors.

So you really need to be doing optimization and you need to be paying attention to the right way to measure that success. I

Ryan Garrow: love it. Yeah. And if you want to have one metric that improves your site, I can do it within an hour by just cutting off all your non brand traffic and your conversion rate increases, and you did no optimization.

Jon MacDonald: See?

Ryan Garrow: And your business will go in the tank. It’ll be great.

Jon MacDonald: Or, you know what? Offer everybody 99 percent off or a hundred percent off. Just give your stuff away. Your conversion rate will improve. It’ll improve. It’s great. This goes in a vacuum. There are too [00:25:00] many ways. To make yourself lucky. Let’s put it that way.

Ryan Garrow: Got it. Anybody promising you anything is an idiot. I’ll go and say it for Jon, but

Jon MacDonald: lucky you’re like, we’ll work on it, Ryan, we’ll work on it for you.

Ryan Garrow: Jon. I appreciate the education. Thank you.

Announcer: Thanks for listening to Drive & Convert with Jon MacDonald and Ryan Garrow. To keep up to date with new episodes, you can subscribe at driveandconvert.com

The post Drive and Convert (Ep. 120): I’m Running Experiments. Why Hasn’t My Conversion Rate Gone Up? appeared first on The Good.

About This Episode:

This week on Drive & Convert, Jon and Ryan discuss how to quickly validate ideas so you can launch site updates with confidence.

Check out the full episode to learn:

1. The process of smoke testing (and where the name comes from).
2. The benefits of using smoke testing for quick validation.  
3. The 5 steps of the smoke testing process.

If you have questions, ideas, or feedback to share, connect with us on LinkedIn. We’re Jon MacDonald and Ryan Garrow.

• Apple Podcasts
• Google Podcasts
• Spotify
• Castbox
• Castro
• Overcast
• Pocket Casts
• Stitcher
• Subscribe via RSS

Episode Transcript:

Announcer: [00:00:00] You’re listening to Drive and Convert, a podcast about helping online brands to build a better ecommerce growth engine with Jon MacDonald and Ryan Garrow. 

Ryan Garrow: All right, Jon, you know me fairly well. And you know that my default is to come up with really cool ideas and move fast and break things and figure it out as I go.

I build the car as I drive it. And it doesn’t work all the time. And for most of my ideas, my wife shoots them down from the very beginning. And I don’t even get a chance to go work fast and try to break it because she’s it’s a bad idea. But I’ve learned if I can quickly bake something up and be like, I think it’s a good idea.

Let me, I’m not executing the idea, but I’m just going to do this really quick thing to prove that it’s not dumb. I have a much better shot with my wife to get an idea through the system and get approval in the family business ideas. And I thought I was pretty unique in this. And then you are, you’ve told me about this cool idea you have about or this new thing that I’ve never heard of about how companies do this on their websites all the time.[00:01:00] 

And there’s even terms for it. Something about smoking and, but it lets us get, you can do it with less data. You don’t have to go through the full A/B testing to prove an idea. It’s quicker. I, at the end of the day, it appears that we get to move fast on websites with this cool idea of yours with a lot more confidence.

And I, for me, that is I’m all in on this. And so how do we do this? Tell me what I need to smoke to get this thing working. 

Jon MacDonald: Yeah there’s a lot of options for what you could smoke for sure, but we’ll leave that for another day. Especially in Oregon. Yes, exactly. Look, whether you’re building products or marketing campaigns, time and money are just precious commodities.

So I hear you. It makes sense. You want to move fast with confidence, right? There’s so much pressure these days to make the right decision the first time. And that is something that we haven’t really seen a lot over the years. It’s always been something in entrepreneurship and businesses, but lately we’re finding that experimentation is important. Nobody wants to try [00:02:00] things and fail for a year until they find something that actually moves them towards their goals. And I don’t think this is a generational thing. I hear that quite a bit, but I do think it’s because marketing and the technology stacks around that has been moving so quickly that everybody expects to be able to optimize their site quickly now too.

And A/B testing can take some time. It really can to do it right. I think this is where a term I’m calling smoke testing becomes really invaluable. And I say I’m calling, we should talk a little bit about what it is and where it came from. We’ll get into that today. But it’s not a term that I invented or that The Good invented.

It’s a commonly used term in optimization. Actually a lot of different fields, but smoke testing is providing a quick and early feedback on your ideas, the functionality you add to your website, your products. It allows you to validate concepts without really just jumping into full scale development.

So it comes from lean methodologies [00:03:00] to some degree, right? Helps you make informed decisions and reduce the risk of costly mistakes. That’s the best way to think about it. 

Ryan Garrow: I think that sounds amazing. I would, I probably want to use this in a bunch of other areas, but what does it, what does smoke testing mean when you’re speaking about website and DXO?

Jon MacDonald: Yeah, so smoke testing is a process where you would test the basic functionality of an idea or a product before you implement it fully. The goal is to capture valuable feedback. Before you spend time or money or any other resources in developing something for real. Normally we would have done this with A/B testing, right?

But in A/B testing, you often have to develop both options, right? It does require development resources, right? Here, your goal is. To run a smoke test and if it passes, you move forward with confidence. If it fails, you just abandon it without having spent a lot of time and money. Smoke testing also goes by the name of confidence testing or validation testing.

So in short, it’s gonna [00:04:00] help you validate your idea And smoke test can even preempt the MVP, or in lean, what you might call minimum viable product concept. 

Ryan Garrow: Got it. I don’t want to call it confidence testing because that’s not nearly as cool as saying, Hey, we’re going to go through some smoke tests and figure out what works.

Jon MacDonald: And in reality, if we’re getting super technical, which I know we probably don’t need to do, but confidence testing is a completely different thing as well. 

Ryan Garrow: Got it. Okay. So what, so give me an example, so I can put my brain around this. What is, what If you’re going to do smoke tests on a site, what does that walk me through that?

Jon MacDonald: Okay. Let’s use maybe hypothetical online media site. I’ll just throw that out there. And they want to implement a snooze my subscription feature. So I subscribe to the economist who’s one of our clients and I want to snooze my subscription, but the economist doesn’t really know if it’s going to stop people from canceling, right?

They just say, Hey, this might be a downgrade opportunity, or, Hey, I’m going on vacation for a while. I’m not going to use this, [00:05:00] whatever it might be. So what they could do is instead of building out the full functionality on their site, they could add a button that says, as part of the cancellation process, offer the option instead, I want to snooze my subscription.

And they would measure how many clicks they’re getting on that. 

Ryan Garrow: He’s put an event tag on there like a GA event tag. 

Jon MacDonald: Yeah. Whatever tracking you’re using, GA, et cetera. Yes. Now there are two ways to look at this. Maybe the button doesn’t really do anything but take you to a simple page that says this features.

Okay. Coming soon. Thanks for your interest or in this type of situation, what I would probably want to do is have it connect you to a customer service, right? Something that they can help you implement this, but it’s not a fully automated process at this point, right? It might be something like, Oh, great.

We’ll pause your subscription. We’ll be back in touch to confirm. You can just do something that simple, right? Okay. 

Ryan Garrow: Got it. 

Jon MacDonald: Where it’s very manual, because you’re not sure if people want this. Don’t build out the functionality. Just do whatever is the [00:06:00] absolute minimum. 

Ryan Garrow: I can even imagine that you would just even put like a an email button like snooze at and it would just go to customer support and be like, Hey, this isn’t quite ready or to manual process.

And you’d be like, Oh, I can still pause their subscription for three weeks. Exactly. 

Jon MacDonald: The idea here though, is not what happens after they click the button. Really it’s the number of clips on that button. That’s going to help you understand how many people would consider using such a feature. Because there’s a whole optimization you can do of that feature later, right?

You could go out and say, Hey what are the different, you could test the different pause lengths, right? You could smoke test a hundred different things after you test that people even care about this. So the goal here is what’s the smallest thing you can test that’s going to give you the right direction to head in.

Then you’re going to continue to run a ton of these smoke tests at each step. 

Ryan Garrow: Got it. Cause what in theory you could be. testing this. So snooze subscription. And in reality, 100 percent of them are going to cancel anyway. So why delay this way for [00:07:00] whatever reason? 

Jon MacDonald: Yeah. So why even offer it then? You’re just like, okay, instead I’m going to focus on smoke testing ways, other ways to get people to not cancel. 

Ryan Garrow: Okay. So why do we, I guess before I asked for a why, where did smoke testing come from? Cause it doesn’t, he don’t smoke. Seems a weird analogy in my head at least.

Jon MacDonald: I hear you. And I agree. The first time I heard it, I was Oh, that’s interesting. Okay. I don’t know if you’ve ever heard. I went to, have you ever heard of, there’s a website out there called HIGHdeas. It’s a hilarious site if you ever just want to kill some time, it’s people who basically it’s like ideas they came up with while they were high.

Okay. And I was like, so is this just like coming up with random ideas and testing them? And you’re just like, Oh, whatever. And it’s a fun play on words or whatever. And it turns out, no, it’s got much more boring kind of background to it. 

Ryan Garrow: Dang it. I’m going to probably go with that as my original idea though.

Jon MacDonald: Yeah. So we don’t really know where the term came from. If you do, I did research on this. Natalie on my team has written an article about this. It comes out this week at the [00:08:00] good. com. She did a ton of research on it. We don’t know where it came from, but there are two origin stories that popped up more than once.

So one is that it comes from the plumbing industry. This is because they use smoke and to test for leaks and cracks in pipes, so it’s a quick way to identify a problem. So another possible origin that came up multiple times was from hardware testing, like hardware, like electronic devices. where they are initially switched on and tested for signs of smoke in their components to make sure that everything’s wired together.

Ryan Garrow: Yeah. Smoke is bad. Okay. 

Jon MacDonald: Yeah. So it’s a quick and easy way to be like, was everything wired? Okay. Oh yeah. Okay. Nothing smoking. We can move it along in QC. But nothing’s going to blow up. We’ve already tested to make sure it’s not going to blow up right away. So I would hope that now that I think about that, that every single device that plugs in that I’m using has gone through smoke testing.

Probably not the case, but I would hope that would be the case. 

Ryan Garrow: There’s probably some things on Amazon that haven’t gone through [00:09:00] that. 

Jon MacDonald: Yeah. Don’t go to Temu or whatever that is. So it’s hard to say if either of these are true, but I think they are potential smoke testing origins and they are feasible uses of smoke testing is a larger type of context, but it really has evolved into a widely used term in software development, and I think that’s what’s cool about this. It’s a subset of tests used to assess if a software build is stable enough for the next development stage. It identifies bugs that kind of just block the release of a product pretty quickly.

However, In classic marketing fashion, because marketers like to latch onto everything, growth teams have really adopted and redefined smoke testing as a quick tool to validate ideas. So that’s how we’re going to talk about it or I’m talking about it. 

Announcer: You’re listening to Drive and Convert, a podcast focused on ecommerce growth.

Your hosts are Jon MacDonald, founder of The Good, a conversion rate optimization agency that works with ecommerce brands to help convert more of their [00:10:00] visitors into buyers, and Ryan Garrow of Logical Position, a digital marketing agency offering pay-per-click management, search engine optimization, and website design services to brands of all sizes.

If you find this podcast helpful, please help us out by leaving a review on Apple Podcasts and sharing it with a friend or colleague. Thank you. 

Ryan Garrow: Okay, I get the concept. , but in a basic Shopify store. Why even use it? Just find an app, try something. Is there something Yeah, that makes it more applicable than other things.

Jon MacDonald: Oh, for sure. Look, as I mentioned earlier, the beauty of smoke testing is that it provides quick validation without requiring significant resources. And that’s where I think for midsize or smaller e comm sites, it really can make sense, right? So you get to find out if an initiative has genuine interest or demand before you invest anything into that, right?

That means smoke tests, save time and money. But feedback is also another benefit. So if you put out a preliminary [00:11:00] idea, you get to quickly learn what potential users think and feel about it. So that’s also really valuable, right? And third, they’re just great tools for prioritization of functionality and features.

If your resources are limited, it’s important to focus on what’s going to bring you the most value. And that’s where smoke testing can really come in. 

Ryan Garrow: No, I am, I’m a big fan of this. I assume I can’t go smoke test five things at once on a site because that would be a lot of buttons that probably don’t do what you intend for them to do.

Yeah, you don’t want to. 

Jon MacDonald: The idea here is to not. have the consumer know that you’re smoke testing something. That should be your goal. What is the minimum you can do without making it so obvious that you have, are putting up a facade? That’s what I would look at. 

Ryan Garrow: Okay. I think that’s a great goal for this then.

Okay. And I love it because, we work with a ton of small businesses that don’t have traffic. For full A/B tests on a lot of things where the finance is to jump into the deep end with a DXO organization like yourself. So it’s in my mind, this is really [00:12:00] helping somebody say, I got to do something, but I don’t, and I have some ideas, but I don’t know yet what it’s going to be.

So I’m going to, I’ve got these few, like list of smoke tests I want to test rather than try all of them all at once. What are the steps I would take on the first one to say, okay, this is the smoke test I want. How do I know that I’ve gone through the process to either say it’s a good idea or a bad idea?

Jon MacDonald: Yeah fortunately, I think there’s simplicity in smoke testing. So setting one up is really straightforward, right? I think what we could do is break this down into a handful of steps because I like to make a process of everything, as you 

Ryan Garrow: do, yes. 

Jon MacDonald: I think there are five steps that I would do here and look at this.

The first is, of course, start with the end in mind. So establish your acceptance criteria, right? Okay. If you understand what your go, no go decision is, you’re going to have clear terms for how the outcomes of your tests will determine what you do next. Okay. So for example, just have a definition of what success is.

So if we generate x dollars in preorders for the new product within the next month…great [00:13:00] question, right? So I think this is a great way to test if you should even launch a new program, right? Or a new product or anything like that, right? You just basically say, Hey, we’re going to put up a product detail page, not have the product, and then just say it’s pre order. And how many pre orders can we get before we launch the product?

What do we need? You can always say, We won’t charge your card until we ship. And then just email people and be like, Oh, really? Sorry. We canceled that because there just wasn’t enough interest. If you are super interested in something similar, here’s another product. So the first step is.

What’s your acceptance criteria? Then second is design that simple experiment. Keep in mind though, that this should be really simple, right? You might create a landing page for that product and collect pre orders like I just talked about, or build something that’s just quick and dirty, but still presentable like the pause my subscription functionality that I mentioned, right?

And I’ll use functionality [00:14:00] with air quotes because you’re, again, you’re not building the functionality. The third step, I would think, should be to drive traffic to the experiment, okay? If you don’t have traffic on your site, I would find other ways to put it in front of your audience. And how you do this depends on the nature of your test and, of course, who you’re trying to get to interact with it.

But if you’re building an online experiment, excuse me, online service or experience of any type, and you don’t test that with fake AdWords campaigns ahead of time, I think you’re crazy. So you really should be trying to drive traffic and you’re going to have to pay for that. But AdWords is a great tool for that.

You’ll see how many people click on it and go in. Yeah. I feel you. Speaking my language. All right. And he’ll help you set up that fake AdWords campaign and get fake. 

Ryan Garrow: And fake means that you’re. You’re going to buy some keywords and measure interaction rates and see if it has potential to convert and drive revenue, right?

Because this is something that’s going to be important. Yeah. 

Jon MacDonald: It’s not something you’re planning on doing for the longterm right up front. It may be an optimized campaign for instance, [00:15:00] right? You may be spending more than what you would want to spend because your quality score is low or whatever. You’re just trying to get something quick and dirty done, right?

And then you really are going to want to work to optimize that campaign and make sure you’re spending appropriately. Once you say, okay, yeah, I’ve validated this. So far we have established the criteria, design the experiment, drive traffic to it. Once you’re ready to drive traffic, launch it and then track that engagement, right?

Let it run and collect data according to that success criteria we set up in step one. But while you’re tracking, you may come up with new ideas. This is almost always what happens. You’re going to have that idea, be tracking it and get some data and you’re immediately going on, shut it off and say, Oh, I’m going to do this instead.

Resist that temptation. The first thing you should do is document those ideas. You can run additional experiments or adjust it to inform better decisions, but give it a little bit of time. How much time is really up to you. This is a smoke test. After all, you don’t want to do. a ton of invest a ton of time, [00:16:00] but don’t run it for a couple hours.

Don’t run it for a day, right? Give it a, a little bit of time there. Lastly, what you’re going to want to do is make your decision pretty cut and dry. But the reality here is you need to understand whether the results warrant you going forward with the idea. And that’s really it. So hopefully at this point you’ll have as much data as possible to support your decision, but that won’t always be the case, right?

Sometimes you just need to make a judgment call and that’s okay with smoke testing. If that’s the case, you might also decide to do an additional test with a little more effort. But again, don’t spend a lot of time making your decisions. This is really all about being fast and decisive. 

Ryan Garrow: And I think for, if I’m looking at this, there’s a couple of things like it, my decision making would come down to the amount of data I have would be like, would my life, what is that enough data for my wife?

Can she say, we could do this over here in the business world? Like my business partner, the executive team, is this enough data to say, yes, we can invest in this. And I do that a lot, actually, logical position. I’m like, Hey, we got to do something [00:17:00] quick. We can’t, we’re not going to put a bunch of resources behind something that may or may not work.

So Garrow’s going to go do some, thing I’m over here that. Maybe it does something, maybe it doesn’t, but it doesn’t break anything to do the test. And that’s really been one of my overarching goals for a lot of things, but I can say I have really simplified this process into one step, Jon. Ask Jon.

And then it’s much easier. You just, I just have one step. 

Jon MacDonald: Love that. If I’m a user, you’re, the goal here is to get user feedback before implementation, right? So yes, I think before diving into development, it’s always smart to gather feedback, whether that’s a smoke test or a Jon test or whatever you want to call it.

I think smoke testing, what we talked about today, I think it’s practical. I think it’s efficient. Let’s you test the waters before you really commit any resources. And I think that’s just a great ethos for optimization in general, right? Too much stuff with CRO over the years has been flooding the market with things that are [00:18:00] either too simple, like just do this checklist, whether or not you know, it’s going to work for you or too complicated.

Like you got to run all these A/B tests are going to take months. and a lot of resources and development and everything else. This, I think, is a great medium. It’s a great way to get quick feedback with data backing it and move forward. And it doesn’t take building out your ideas to give yourself proof or your wife proof that it will work.

Ryan Garrow: Yeah, this is honestly one of my favorite ideas because it can work across an organization of any size, which tends to be some of the limitations with a lot of the things you and I know is as out there in the market or available to companies, because we work with a lot of large ones that says, man, this is a super cool program, but if you’re doing less than 20 million a year, it’s just not going to work or not.

You’re not gonna be able to afford it. Whereas this is, man, if you can just get a simple thing on a site, which, Upwork for 50 bucks may get you enough development talent to get something done on your site for a smoke test. So I love it. 

Jon MacDonald: And with Shopify, a lot of these things can be done in the editor without you really [00:19:00] needing to do anything, hire a developer at all.

So something to consider. 

Ryan Garrow: Nope. I love it, Jon. Thanks for telling me about smoke tests. It had nothing to do with what I originally thought about when you said smoke. 

Jon MacDonald: Me neither. I was, my hypothesis was quickly shot down by the team as well. Yeah. All right. 

Announcer: Thanks for listening to Drive and Convert with Jon MacDonald and Ryan Garrow.

To keep up to date with new episodes, you can subscribe at driveandconvert.com

The post Drive and Convert (Ep. 118): Launching Site Updates With Confidence appeared first on The Good.

All else being equal, once startups begin experimentation, they see strong gains within the first few years and continued growth in years to come.

“A/B testing significantly improves startup performance and […] this performance effect compounds with time.” — Koning, Hasan, Chatterji, Experimentation and Startup Performance: Evidence from A/B testing

Given these blockbuster outcomes for experiment-led companies, it’s obvious that experimentation correlates with success. But does having an experimentation team guarantee better overall performance?

Case studies make the connection between experimentation and KPI growth seem inevitable. So, it’s logical to expect that once your experimentation engine is firing on all cylinders, you should see a noticeable impact on the data. But is it practical? If I run experiments, will my conversion rates go up?

It depends.

After running thousands of experiments, we’ve learned that what happens after a release isn’t a simple one-to-one outcome that mirrors the success of your experiments. But the story of why is not so simple.

In this article, we explore:

• Why experiment results don’t always net out one-to-one
• Why we still believe that experimentation is a powerful tool to add to your toolkit

Why don’t we always see one-to-one results from our experiments?

If you’ve ever researched an experimentation partner, you’ve probably come across one that guarantees conversion rate increases. It’s an enticing prospect that experimentation = an automatic increase in KPIs. Industry insiders like Shiva Manjunath are generally against this “snake oil” tactic, and at The Good, we simply say anyone who guarantees they can increase your conversion rate is either lucky or lying.

There are plenty of reasons experiment results don’t map one-to-one with the real-world outcomes you expect. So, let’s explore three core issues that prevent you from seeing those charts go up and to the right:

1. Post-launch variables
2. Experiment segmentation
3. The effect of false positives

Problem 1: Post-launch variables make attribution less than clear.

Metrics are influenced by much more than a website or app experience. Direct and indirect influences impact metrics like conversion rate, revenue, and customer satisfaction.

In fact, we’ve identified over 55 variables that contribute to swings in KPIs. Factors like traffic quality, seasonality, competitor promotions, and even the economy play a huge role in whether or not website visitors will convert—now or in the future.

As we learned during the COVID-19 pandemic, even the largest experimentation wins may not eclipse outsized influences that dampen conversion rates. (True story: we once saw a single social media intern drive so much new traffic that conversion rates fell by a whole percentage point. Great for the intern’s portfolio, bad for conversion rates.)

The opposite is also true. When outside factors are working in tandem, the results can be outstanding. We worked with Alisha Runckel during her tenure at Laird Superfood, running hundreds of experiments together, contributing to their industry-high conversion rates.

But experimentation wasn’t solely responsible for her success. Alisha and her team also made good offline decisions. Their holistic approach included direct mail, bundling incentives, and even a packaging overhaul informed by sentiment testing.

“There are no silver bullets. Ecommerce success is an accumulation of good decisions made over time.” – Alisha Runckel, Laird Superfood, Humm, Hannah Andersson

If done correctly, A/B tests are a trustworthy way to estimate the effect of a treatment despite shifting external influences. But once the test period is over, you no longer have a baseline or “counterfactual” scenario to compare to.

Even if you’re capable of running precise counterfactual scenarios, there’s no easy way to understand what performance would have looked like if you hadn’t implemented the changes. As a result, once a test period is over, we’re generally ignorant of whether we’ve improved a metric that was already in decline, which makes post-launch attribution fuzzy at best.

Problem 2: Test results are not summative

5 + 5 = 10, right? Not in experimentation.

While there are many reasons that experiment results are not summative, let’s focus on one of them today—segmentation.

Many experiments only impact a segment of users, and in those cases, the observed improvement is diluted by its proportion of the larger population. Therefore, a win with a segment, or subset, of users does not predict the same gains for the whole.

Let’s run a scenario to demonstrate:

• We experiment with a punchy new landing page and checkout flow on mobile traffic only, which is 50% of our total traffic.
• The experiment shows an increase in revenue among that audience of 5%.

When we implement the changes in production, we shouldn’t expect desktop traffic to perform in an identical manner. The reasons could be many. Maybe the change isn’t as conventional for desktop devices. Or maybe the experiment hypothesis only had to do with solving for mobile users’ lack of patience and need for speed.

Whatever the reason, practitioners often run experiments that are only meant to impact a subset of their audience, and the rest of their visitors won’t experience the same benefit. That’s true for splits by device type, landing page, and similar segments, and it’s why we generally don’t extrapolate the results of a segment and apply them to the whole.

A 5% gain with one audience and a 5% gain with another does not equal a 10% lift overall. Test results are not summative.

Problem 3: False positives

The final reason (at least for today) that your overall KPIs might not match the result of your experiment is the occurrence of false positives.

False positives are what we call it when our experiment data indicates that our hypothesis is true when it actually is not.

“[False positives] appear to generate an uplift but will not actually generate any increase in revenue.” — Goodson, Most Winning A/B Tests are Illusory

False positives may sound like an atrocious error on the part of the experimenter, but they are actually par for the course. Even rigorous and experienced experimentation teams expect about a 26% false positive rate, meaning about one in four “winning” experiments is observed as the result of chance and not a true winner.

To account for false positives, some practitioners even go so far as to re-test all winning experiments that fall outside of a certain threshold, then split the difference to estimate real-world effects (more on that later).

In my experience, most practitioners aren’t so persnickety about false positives. Their approach is to accept that some “wins” are truer than others and, as a result, anticipate a slightly lower return when comparing test results to real-world outcomes. Still, those blessed with the traffic and time might consider the practice of re-running tests for improved confidence.

Does the pervasiveness of false positives mean experimentation doesn’t work? Of course not. It just means that we should approach “wins” with an informed skepticism and not expect a one-to-one relationship between the results observed during a test period and real-world performance.

Experimentation is an incredible tool, but it’s not a silver bullet

If the caveats listed above deteriorate your confidence in experimentation, let me stop you right there.

It’s tempting to look at fuzzy post-launch attribution and false positive rates and say, “Is experimentation really worth it?” We believe it is. And people much smarter than myself swear by a “test everything” approach because there is simply no better or more rigorous way to quantify the impact of changes on your bottom line.

Problems do arise when we tout experimentation as an omnipotent growth lever—a magic bullet to success. But if disappointment is the gap between expectation and reality, harnessing the power of experimentation is simply an expectation-setting exercise.

Similar to OpenDoor’s Brian Tolkin, rather than an all-powerful growth lever, we view experimentation as a confidence-generating mechanism.

“Experimentation is all about increasing your conviction in the problem or the solution.” — Brian Tolkin, Lenny’s Podcast

Experimentation can increase your confidence in your decision-making, help you measure the discreet impact of good design, and settle internal debates about which direction to head. What experimentation won’t do is compensate for all the external forces hampering your business.

If you’re looking for confidence and precision, experimentation is an incredible tool to add to your toolkit. If you’re looking for a silver bullet, we’re still looking, too.

Assure the best possible experimentation outcomes with these tactics

It might be discouraging to hear that the hailed “silver bullet” of experimentation isn’t going to solve all of your problems. But, hopefully, you’re excited to know it is still proven to help your digital property perform at its best.

If you want to push your organization towards digital excellence and get the best outcomes from experimentation efforts, a well-run program is key. Take a measured approach to incorporating it into your growth practice and consider a few tips from veteran practitioners to make sure you are maximizing the effectiveness of experimentation.

Give your A/B tests ample time + traffic—without stopping short

It’s tempting to periodically check the progress of a test to see how things are trending, but there’s a name for this kind of behavior: peeking.

Maggie Paveza of The Good defines peeking as the act of “looking at your A/B test results with the intent to take action before the test is complete.” As Evan Miller describes in his article, How Not to Run an A/B Test, “The more you peek, the more your significance levels will be off.”

Avoid peeking by calculating test traffic requirements during the planning process and not stopping tests earlier than planned.

• Use predetermined calculations to set the acceptance criteria, test duration, and minimum traffic levels
• Analyze a test’s results only after you’ve reached predefined thresholds
• Don’t stop tests earlier than planned

Following these simple steps will increase the trustworthiness of your results and reduce your rate of false positives.

Re-run some winning tests to verify your results

If you and your team decide that precision is more important than time, you may opt to re-run winning tests that have a p value above a certain threshold, say .01-.05, to gain additional assurance that the effects measured were not the result of chance.

Experimentation veterans like Ron Kohavi recommend repeating some winning tests a second time “to check that the effect is real.”

“When you replicate, you can combine the two experiments, and get a combined p value using something called Fisher’s method or Stouffer’s method, and that gives you the joint probability. — Kohavi, The Ultimate Guide to A/B Testing, Lenny’s Podcast

By running the test a second time and analyzing results across the two experiments, the newly calculated effect likely represents the truer difference between the treatment and the control. The result is that you’re less likely to implement a “winner” that was simply observed due to chance.

Get comfortable diagnosing metric decline

While we can’t define every factor impacting conversion rates, there are some factors that are easier to spot in the data than others. Factors like seasonality, fluctuating traffic from various segments, and even traffic bots are fairly easy to track down, but you have to know how to spot them.

Luckily, Elena Verna, Head of Growth at Dropbox, created a decision tree for doing just that. Elena’s if/then flowchart helps growth specialists “quickly diagnose troubling conversion rate trends within 48 hours,” according to Reforge.

Whether you’re looking to evaluate the impact of your experiments or track down wayward KPI signals, getting comfortable with defining the source of KPI changes is a valuable skill that will help you build trust and authority within your organization and help you understand those outside factors that might dampen the impact of your experimentation program.

Trust the process and use experimentation to its full potential

One thing has been proven time and time again: experimentation done right is associated with increased overall performance across a number of factors.

While experimentation can’t combat outsized economic and environmental factors, it can be a catalyst for better decision-making and help assure your digital property is performing at its best—despite what’s going on outside.

By setting proper expectations, calculating test parameters before launching, mitigating false positives, and getting comfortable with attributing metric change to specific fluctuations, you can rest assured that your test data can be trusted and you’re performing at your best.

The post I’m Running Experiments. Why Hasn’t My Conversion Rate Gone Up? appeared first on The Good.

It’s also doing a disservice to your users and your optimization efforts.

To no fault of digital product owners, the industry has put an unproductive emphasis on conversion rates. But based on over 15 years of experience optimizing, I promise that websites and apps won’t reach their full potential under the current industry expectations and implementations of conversion rate optimization.

I am here to propose a better solution: digital experience optimization. This evolved approach brings together all the diverse disciplines and tactics needed to build a better digital journey and get us re-centered on what really matters: the customer.

Let’s talk about digital experience optimization, how it compares to conversion rate optimization, and hopefully answer if it is the right solution for you.

What Is Digital Experience Optimization?

Digital experience optimization (DXO) is the process of making meaningful changes to your website or app to meet the needs of its users and business. This is achieved through data analysis, research, strategy, experimentation, and other disciplines.

DXO is a foundational approach that involves looking at the digital journey as a whole to see not just how customers engage with you, but why they make certain decisions. It analyzes behavior from the moment they enter your app or website through post-conversion and finds/executes on improvement opportunities.

DXO is bigger than tweaking colors, copy, or images on your website or app. Yes, digital experience optimization will improve the look and feel of your digital product, but it can also change how you think about and address your customers, improve the culture of experimentation on your team, and skyrocket buy-in from stakeholders by using data to inform decisions.

Digital experience optimization crafts comprehensive user-centered journeys that address customer needs at every stage of their journey.

What Is Conversion Rate Optimization?

Conversion rate optimization is defined as a process that increases the percentage of users who perform a desired action on a website or app.

The desired action or conversion is typically a sale or high intent signal, for example, purchasing a product, signing up for a trial, or filling out an inquiry form.

When “optimization” is tacked on to the end of “conversion rate,” the process becomes synonymous with A/B testing. You will often hear “conversion rate optimization program” and “testing program” used interchangeably.

Conversion Rate Optimization (CRO) vs Digital Experience Optimization (DXO)

When compared to DXO, CRO is very narrow and limiting. DXO practitioners are still excellent at A/B testing to validate or invalidate changes before implementation, but also:

• Pull back from conversions to understand and improve the whole digital journey
• Incorporate other research methods to supplement what can be learned from A/B testing
• Leverage an array of expertise and disciplines to build ongoing optimization practices

Like the traditional definition of CRO, DXO is a process aimed at iterative improvement. However, DXO creates large-scale, sustainable impact that conversion rate optimization can’t achieve due to inadvisable hyper-focus on conversion rates and A/B testing.

While writing this comparison, I’m tempted to go so far as to advocate that conversion rate optimization (as it is currently defined and understood in the industry) shouldn’t exist anymore. Let’s simply remove it from our vocabulary.

I say this as the founder of one of the industry’s first conversion rate optimization firms. In conjunction with the industry’s narrowing understanding of CRO, we have seen the reality of user needs and what drives engagement evolve. Technology, users, the industry, and the world have changed, making conversion rate optimization a practice of the past.

Don’t get me wrong; we should all be measuring the conversion rates of our digital products and putting in the work to make sure they are consistently improving. But how that happens can’t be boiled down into one research method. One metric can’t signal the health of your digital product.

So, I’d like to invite you to expand your approach to website and app improvements with digital experience optimization.

8 Digital Experience Optimization Use Cases & Benefits

DXO empowers ecommerce leaders and digital product owners to make smarter, more effective decisions that enhance user experience and drive business success.

The goal of the practice is to unlock the full potential of a website, app, or digital product through research, strategy, testing, and implementation. Let’s take a look at its more specific use cases and benefits so you can decide if it’s a good fit for you.

1. Improve the Look and Feel of Your Website

There’s a certain degree of trust instantly imparted whenever we see a well-designed website with attractive visuals. In fact, half of users say that they use a site’s design to form an opinion on the business.

DXO is useful here as it goes beyond superficial changes, incorporating user feedback and behavioral data to drive design decisions that not only look good but also perform well. This is a great way to build those hard-to-measure qualities like trustworthiness, reliability, and safety that customers crave.

2. Prioritize Your Ideas And Improvements Effectively

You probably have hundreds of ideas you’d like to try, but each change comes with a cost of workforce, time, and money. How do you choose the ones worth exploring?

DXO helps by providing frameworks to evaluate and rank these ideas based on their potential impact on user experience and business goals. It zeros you in on the ideas that are more likely to move the needle for your organization.

“We are much more targeted and focused on what we can actually do,” says Justin Albano, Digital Marketing Manager at IDX. “We’re not sitting there wondering what we should be doing or what’s going to make a difference. We know what we need to do now, and we’re getting after it.”

By focusing strictly on research, data, and clearly defined goals, DXO simplifies the decision-making process. You can stay laser-focused on those improvements that actually help over the long term and avoid distractions from less impactful initiatives.

3. Form a Deepened Understanding of Customers

DXO can give you a more profound comprehension of your customers’ behaviors, preferences, and needs. This insight is crucial for developing more effective strategies, creating personalized experiences that truly resonate with users, and improving overall customer engagement.

These insights come through web analytics, user flow analysis, customer surveys, session recordings, and other techniques.

4. Build a Valuable Knowledge Center of Consumer Insights

Similarly, but a more tangible potential benefit and use case, is that DXO builds a rich knowledge center filled with actionable insights on your potential customers. This repository becomes an invaluable asset for the organization as it helps you with informed decision-making and strategic planning across all levels.

This library can include hard data as well as direct customer feedback. This type of knowledge bank helps you improve your digital touchpoints and customer interactions, meet customer expectations, avoid negative experiences, and ultimately improve customer loyalty.

5. Create A Culture Of User-Centered Data-Driven Decision Making

It’s no secret that data-based decisions are better than best practices or “gut feelings.” A strong DXO program uses real-world data from your ideal audience to guide decision-making.

DXO fine-tunes your decision-making by using comprehensive analytics and key conventions to guide decisions from the beginning. This ensures that every change contributes positively to the user experience.

Data isn’t the end-all-be-all, but practicing DXO creates a culture where decisions are not just based on hunches or past experiences but are informed by your customers. It ensures that your work aligns with the actual needs and behaviors of the users.

6. Save Time and Resources by Validating Design Decisions

DXO enables you to validate or invalidate design decisions before implementing them at full scale. This process saves significant time and resources by preventing the rollout of features that may not meet user expectations or business goals.

Strategic focus helps companies achieve more with less, leveraging their existing assets efficiently. For organizations looking to stretch their resources to their fullest potential, DXO ensures that every effort and investment is optimized for maximum return.

7. Grow Your User Base and Increase Conversions

If your goal is to expand your user base or convert more visitors into customers, DXO provides a structured approach. By optimizing the user journey at every step, your digital presence can meet more visitors’ needs and convert more into loyal customers.

Think of it like pleasing more people earlier in your funnel so more potential customers march down the conversion path.

8. Improve Stakeholder Buy-In

If internal politics or too many opinions are hindering progress for you, DXO offers a neutral, data-driven perspective that focuses on what’s best for the user and the company.

Presenting roadmaps and strategies established by DXO can help drive changes that might otherwise be stalled by misaligned internal priorities or opinions.

How To Measure Digital Experience Optimization

Measuring the performance of your digital experience may seem like a tall task compared to just focusing on one number (conversion rate), but it’s sure to set you up for more success.

Luckily, there are tools in place to help you gauge improvement of your program that are proven indicators of digital health, for example, a 5-Factors Scorecard.

The 5-Factors Scorecard is based on a study of hundreds of digital leaders’ optimization challenges to reveal the five factors that the highest-performing companies have in common:

• Data Foundations: Goals, ownership, and good data form the backbone.
• User-Centered Approach: A comprehensive roadmap and a high-context approach.
• Resourcing: Resources support adequate capabilities and pace.
• Toolkit: A variety of tools for planning, measurement, and protocols.
• Impact & Buy-In: Tools and practices increase relevance and perceived efficacy.

Research shows that improvement in these areas leads to measurable business outcomes. Get a 5-Factors Scorecard to highlight the areas of your digital experience that need work and use it as the baseline for your digital experience optimization measurement. To track improvement, re-take the quiz and compare the new results every three months.

Remember, digital experience optimization is a comprehensive solution to complex and diverse digital challenges. Measuring its success should be similarly comprehensive.

Ready To Get Started With DXO?

The web and the way users experience stores, platforms, and media have changed. If you want to be successful, your thinking has to evolve as well. Digital experience optimization is a holistic approach to improving the user experience, and by extension, your goals and revenue.

If you’re ready to get started, check out our Digital Experience Optimization Program.

We start with a full-funnel analysis of your digital experience, using methods like heatmap analysis, session recordings, and usability testing to diagnose your digital challenges and prescribe a solution. The goal is to understand, thematically, the biggest barriers and opportunities.

When the audit is complete, we’ll build your custom Digital Experience Optimization Program including everything you need (and nothing you don’t) to complete an optimization puzzle, create an engaging experience for your users, and build a better digital product.

The post What Is Digital Experience Optimization? appeared first on The Good.

If your media consumption looks anything like mine, you’ve probably read countless articles about all the ways in which tech leaders leverage data to make informed decisions.

But unfortunately, as is the case with industry buzzwords, lots of organizations say they make “data-backed decisions” but simply don’t have the culture or toolkit to support them. Period.

They may run a UX research sprint or mine the data to understand their challenges, but if they fail to validate their ideas, they fall short of actually being data-backed. They are just data-informed.

With the advent of analytics platforms, tracking pixels, and attribution modeling, it seems like everyone is an analyst. Or at least they play one in meetings.

The Data Evolution

As Stanford’s Ramesh Johari notes, “100 years ago, data was expensive and slow and practitioners were trained statisticians. Today, data is cheap and real-time, and everyone is a practitioner.”

With the increased access to data, leaders are antsy to put that data to work. We regularly speak with leaders who want to transform their company’s relationship to their data: they run heat mapping tools on their site, have numerous analytics tools at their disposal, and their teams have dabbled in A/B testing. But many still struggle to make sense of the abundance of data and choose the right path forward. They struggle to move from data-informed to data-backed.

And being data-backed isn’t about just having a good data story to support your argument for why another treatment is better. You have to go beyond that and do the evaluative research to actually PROVE it to create a culture that is data-driven.

Though the language is similar, we make a crucial distinction between data-informed and data-backed.

Generative vs Evaluative Research

Many organizations aspire to be data-backed, but they often fall short by primarily relying on generative research methods. 

Generative methods are great for understanding what’s happening on a website and forming hypotheses about what would work better.

Generative research methods include things like: 

• Heatmap analysis
• Surveys
• Data analysis
• Observational analysis
• Open card sorting
• Reviews theming
• Social listening

Alternatively, evaluative research can be used to substantiate ideas with evidence. Evaluative research methods include things like: 

• A/B testing
• First-click testing
• Comparison testing 
• Tree testing

When we ask so-called “data-driven” companies what data they use, they typically list a number of generative research methods. Only about 25% of them list any evaluative research methods and rarely do they go beyond A/B testing.

Generative research, those methods on the left, are great for understanding what’s happening on a website and forming hypotheses about what would work better.

But in order to move forward with confidence that your solutions will actually work, you need the methods on the right. Evaluative research is how we move from data-informed to data-backed and eventually develop a company culture that is data-driven.

What to continue learning?

• Here’s a great example of a data-driven culture and efficiency-first mindset in action: How Andy Lightman’s Ecommerce Philosophy Created Millions in Revenue. “We could end up using half our development budget on building something that doesn’t perform… We didn’t want to overburden the development pipeline with projects that don’t have a financial impact.” 
• Dive into the evaluative research methods you can use to support your effort to be a data-driven organization: What is Rapid Testing and How is it Different from A/B Testing?

The post What Is The Difference Between Data-informed, Data-backed, And Data-driven? appeared first on The Good.

About This Episode:

Digital experiences are always changing and evolving. 

So it should come as no surprise that the evolution of AI, automations, and online tools has also shifted some perspectives and progressed certain ways of thinking.

In this special episode of Drive and Convert, Jon and Ryan share what lessons they’ve learned in the past four years by revisiting some of the earliest episodes of the show and talking about a few recent ones.

Here’s a brief look at some of the topics they changed their perspectives on:

1. No to external automation -> Yes to platform automation 
2. You shouldn’t benchmark -> Multi-KPI benchmarking can provide great value
3. Pull back on socials -> Platform shops are performing really well  
4. Look at Google Analytics -> Try exploring other tools
5. Test everything -> Validate your decisions

Find out what else they changed their minds on!

If you have questions, ideas, or feedback to share, connect with us on LinkedIn. We’re Jon MacDonald and Ryan Garrow.

• Apple Podcasts
• Google Podcasts
• Spotify
• Castbox
• Castro
• Overcast
• Pocket Casts
• Stitcher
• Subscribe via RSS

Episode Transcript:

Speaker 1:
You’re listening to Drive and Convert, a podcast about helping online brands to build a better e-commerce growth engine, with Jon MacDonald and Ryan Garrow.

Jon:
Ryan, I have some good news and I have some bad news. The good news is that we have hit 100 episodes of Drive and Convert. How’s that for good news?

Ryan:
That is some phenomenal news. Like, hey, if nothing else, we are just stubborn and keep doing things.

Jon:
I love it. Okay, the bad news, if you’re ready for that. Now, perhaps this is not a surprise to anyone, but we were not always right across those 100 episodes. So we’ll get into more of that in a moment, but I first want to just take a second and thank everyone who has listened to us over the past four years. I can’t believe it’s been four years already, and it really is the listeners that have made this a success. So nothing makes us more happy than to have somebody mention that they’re an avid listener and I’m on sales calls and people tell me all the time how they listened to this and had a call with a former client of ours that we’ve shared over the years and going to re-engage with them and they even mentioned how much they appreciate the show. And so it’s always great to hear that.
But I just wanted to ask if you’ve gotten anything out of this podcast, please, please, please do us a quick favor and give us a five-star rating in your favorite podcast app and ideally, leave us a review. That is the number one thing you can do to help us expand the reach of the show. And so for delivering 100 episodes of value, as a gift to you, I hope you’ll give us a gift back and just leave us a review and help us get the word out about the show. All right. With that out of the way, now why did I call out that I have not always been right?

Ryan:
Because you know my wife.

Jon:
It’s because we’re going to take this … There you go. She should be our guest.

Ryan:
She should be, like, “Let me tell you how Ryan is wrong.”

Jon:
There you go. Let’s just get our spouses on the line and then the show writes itself, right? Because today, we’re going to take the opportunity to discuss where we’ve changed our minds, where maybe just looking back, it turns out that we were wrong or maybe the landscape has just shifted a bit and we decided that that would be a great topic to discuss for our hundredth episode over four years. So much has changed. We’ve said a lot of things on this show that maybe we have new perspective on, and so we can talk about that a little bit. And because I’m such a kind co-host, I’m going to let you walk into the fire first round.
So let’s chat about an episode where you’d like to maybe let’s just say correct yourself and then I’ll do one of mine and we can just go back and forth. Does that sound good?

Ryan:
That sounds great. Excited to talk about where I may have been incorrect.

Jon:
This reminds me of that TV commercial that’s been running during football a lot, where they bring the ref with the instant replay when the two people are arguing. I told you to pack the camping gear! And the husband is like, “No, you didn’t tell me that.” He’s like, “Should we ask the replay?”

Ryan:
Yes.

Jon:
So that’s where we are today. So where do you think we should start?

Ryan:
Well, I think on my side, we’re going to go back deep to one of our first episodes probably where we weren’t even recording very well and we had all kinds of fun issues. But beyond that part, I think in episode six, I was talking about automation in ads.
So for most businesses, smart campaigns and full automation in Google is not my recommendation.
The market has come and platforms have come so far in just under four years that I’ve had to change my mind in what I’m recommending to businesses online. Google in particular, I will say has probably become the furthest, the quickest when it comes to automation. So three and a half years ago, almost four years ago, automation was generally outside of Google and Microsoft and there were a lot of platforms that would help with that automation. And really inside the platform, we were starting to see smart shopping raise its head and we had dynamic search ads, which was really where the platforms maxed out as far as official automation. There was some enhanced CPC things which would send bids up or down, but in general, I was fairly against automation because I wanted control and I wanted brands to be able to control things, see what was happening, move the levers appropriately when that happened.
And now, we have this thing called Performance Max. And Google’s Bard automation has really become an integral part of Google Ads. You can’t advocate for external automation systems now that would fight with Performance Max and some of the automations built within Google. So I can’t advocate for anybody using systems like that. Initially, I did not like Performance Max. If you’re from Google listening to this, sorry, I didn’t like it. I like transparency, and Performance Max is a black box and I just don’t like it.
But the data is telling us you have to use it. It’s giving preference in shopping ads, which are still where you’re going to find new to file customers most on Google so you have to lean into Performance Max. So forget what I said about don’t like automation, you have to use it, and whether you like it or not, whether I like it or not, it’s a part of what you should be doing on Google.
Now, if you are going to lean into that campaign type and the automation built within that, you need to understand how to analyze. Because it’s a black box, you have to know how to pull reports on it. You have to know where your spend is going. And it becomes very important to understand feeds because it’s a real good garbage in, garbage out. If you are using the automated plugins that send your website data into Google so if you’re on Shopify, there’s a free app on Google that if, hey, if you’re a small business and you need to use the free thing, you have to do it. It’s not terrible. It makes things function.
But what makes sense on your website from a title or description perspective may not make sense on Google. In fact, I was just talking to another brand yesterday that sells spark plugs and their titles make perfect sense on their website for their brand. But when you take that from a non-brand search for a spark plug on Google, it makes zero sense. It’s going to be confusing, if anything, and they’re going to get no clicks.
So feed optimization is extremely important when you’re looking into automation, and that’s where you actually have a lot of levers you can control. There’s areas within the feed you can still stuff up keywords appropriately that makes sense, that don’t violate any of Google’s rules. So understand feeds.
And control what you can. You’re not going to get a lot of search queries from Performance Max. It’s okay. It is what it is, but there’s a new Performance Max that’s becoming more readily available as of now in the recording early in 2024 called feed-only where you’re really going to be working on shopping feeds. It’s not supposed to create videos for you. It’s not supposed to create a lot of text ads. It’s a feed focused, basically shopping only automated feed Performance Max campaign.

Jon:
Okay.

Ryan:
If it does what it says it’s going to do, I really do like it, but it’s still unknown exactly what Google is going to be doing with that. But based on what they’ve told us and what it looks like as of now, I like it a lot. It’s going to get preference.

Jon:
The hot take here is that you like automation now. And I do think over the past four years, AI has come a long way from when we had this initial conversation.

Ryan:
Oh, we weren’t even using the word AI I think in early 2020. If you were and you were neophyte, early on ChatGPT adopter, if it was even available maybe in a very limited beta, yes, you have to lean into the AI. And if your company is trying to grow right now, you probably have to mention that you have AI just somewhere in your presentation deck. But use AI, use automation, and use it appropriately and have checks and balances in place that you do check and make sure that it is doing what you want it to do and really focus on lifetime value.
So that’s the last point on this one. If you’re not focused on lifetime value and you have no way to get that, you’re not going to be able to set the right goals in Performance Max to scale, and the aggressor is going to win when everybody is using the same campaign. So if you’ve got 20 competitors, you’re all MAP priced, and if you all have the same goal, there’s going to be some problems. Those that have a lower ROAS target within Performance Max and a higher quality feed will win probably 90% of the time. And so quality feed, have that lifetime value built into your goals so you know that, hey, if I sell something today, three to six months later, I can win and get that profit back.
And if you convert better than your competitors, guess what? You’re going to be able to set better goals or you’re going to be able to get more aggressive because you’re going to get more things. So you can’t avoid companies like Jon and The Good when you’re trying to force automation within your accounts because higher conversion rates allow Google to spend more money for you and you get more revenue out of it. Off my soapbox.

Jon:
I’m surprised to hear that Google set up a system where you spend more money to win. That’s what-

Ryan:
And it’s really always been that way. It’s Google’s world that we get to live on.

Jon:
Fair enough.

Ryan:
So they get to make the rules. And so whether you want to fight against it and beat on a drum against Google, it doesn’t matter. That’s where people go to find new products. And if you want to play, you’ve got to pay for it. You just better have a plan for capturing revenue without them going back to Google again.

Jon:
There you go.

Ryan:
All right. So Jon, what’s your first mistake you want to address?

Jon:
Oh yeah, right. Okay. So we did an episode not too long ago, actually. Well, I don’t know, episode 75, so maybe about a year ago. And the title was Benchmarks are Bull S. We’ll do that so we don’t get the explicit tag here. So in that show, I talked a lot about how you really shouldn’t benchmark.
I think industry benchmark is the obvious starting point for anyone looking to set goals and that’s why we end up here. But unfortunately, comparing your conversion rate to the other companies or just even industries as a whole is just meaningless.
I still believe that comparing your conversion rate or any of your metrics to other companies or even in industry as a whole is just meaningless. So I still hold that true, don’t get me wrong, because increasing conversion rate is only a goal, and I think that’s another thing that I talked a lot about in there. There are lots of different metrics and goals you should be thinking about and if you just look at one of them, you’re really going to shoehorn yourself into just comparing yourself on one metric, which is not going to be helpful.
But CRO as a whole, as an industry has evolved to become more and more commoditized, and I think there’s a lot more here to consider now because of that commodity focus that’s come onto the industry. So yes, you still need to get data to add to your perspective, but performance is due to a number of circumstances. You can cut ads. You can have a low season. We’ve worked with a company that sells gear for forest firefighters, and what we found was in a wet season, their sales are way down because guess what? Nobody is out there using fire gear than they normally would.
So all of this is just to say that benchmarks are just too simplistic to be useful. And so in that sense, they’re still BS, but I think a more sophisticated team here would take a more nuanced approach to considering other factors and conventions. And so instead of benchmarking, I really think teams need to analyze with some of these key conventions in mind. And that’s where I would change my mind on this is I just said, “Hey, all benchmark is out the window.” And I still think one metric, comparing yourself to one metric is not good, but if you are able to look at yourself in a more holistic way, then it can work and it will help you to at least understand where you can improve.
Now, do I think you should look at one metric like your conversion rate and say, “Oh, my competitor is at 2%. I’m at 1%. I need to get to 2% as well.” I don’t think that’s helpful because what happens when you hit 2%? You’re not going to stop optimizing. You’re not going to be like, “Okay, I’m done. I hit that number and I’m going to stop.” So I still caution against that again, but I would start with looking at a handful of metrics that should work for every company, things like demographics. So make sure you’re checking assumptions about your audience.
Seasonality and promotions, uncover when you should run optimizations. You don’t want to run them necessarily in your low season. Maybe there are some more riskier ones you want to try during a low season, but you’re going to learn a lot more during your high season.
Device strategy. So decide to optimize for desktop or mobile or both. Or if you’re going to push to an app, that’s pretty common in D2C now, look at top of funnel versus bottom of funnel. So it needs to play with your acquisition strategy and whether or not people are actually ready to purchase at that stage.
Site search, I’ve talked a lot about, I think we even did a whole episode on site search. So you really want to understand what shoppers are looking for.
And product analysis. What’s driving the revenue? Is it one product? Is it a suite of products? Is it a bundle? So all of this really led our team to rethink a lot of this, and what we did was we came up with a standardized assessment that we’re actually going to be releasing for anyone to use. So in that sense, yeah, you’ll be benchmarking yourself to some degree, help you understand where you can improve. We’re calling it the five factors assessment because actually five factors that you’ll be assessing yourself on. So that’s coming out pretty shortly here at Q1 of 2024. So if you want to know as soon as it’s publicly available, you’re going to want to sign up for Good Question, which is our newsletter. So just go to thegood.com/newsletter and sign up.

Ryan:
Dang, I like it. Benchmarks are not always BS.

Jon:
There you go.

Ryan:
All right. So my next one, Jon, is not going back quite as far as I did the previous time but we’re going to go back to the wonderful world of iOS update, and it had a massive impact on all things digital but it really hit the social channels hard.
It’s a painful situation for lots of e-commerce brands and of all sizes. It’s not just picking on small ones or big ones. It’s not great. And to a degree, as an agency that manages Facebook but also a client of Facebook that spends money, it’s frustrating almost that Facebook wasn’t prepared for this.
At the time, I think this was back on episode 37, I didn’t stick a fork in Meta and say they’re done, but I came pretty close to saying it’s not going to be pretty. The Meta platforms had been tracking a lot of things that likely went beyond some privacy things. So the iOS update I think was good, but it had a big impact on Meta specifically. In fact, I talked to a brand guest within the last week that sold kitty litter subscriptions online, and in ’21, they spent over two million that year primarily on social channels. And last year, that dropped to half a million. Just that particular brand hadn’t evolved, but that’s really what I expected for a lot of brands to have happen because there’s just not going to be stuff there. You have to change and pivot.
And that brand has some things they need to work on, but I also, in that episode said, “Meta is going to be doing some stuff. Probably they’re not going to lay down and just let Apple kill them.” And I think that’s happened, but I really want to go back and clarify a little bit of what we are now seeing in the social platforms because it’s Meta’s. I won’t say they’re officially back, but it’s pretty good. You’ve just got to be doing some other things around Meta when you’re looking at data because it has changed. We do have Facebook shop, Instagram shop that’s come back or come into the play that I think is showing some great promise. It’s not fully baked yet, I don’t think, but it’s allowing people to stay inside the platform and perform bottom of funnel transactions.
TikTok Shop, I haven’t seen the numbers from holiday yet, but I’m expecting just disgustingly positive numbers coming out of TikTok because of how they handle the holiday season on TikTok Shop. And I don’t even think-

Jon:
I was going to say they subsidized a lot of that, right?

Ryan:
They did. For the untrained eye, it’s going to be, “Oh my, gosh, they did this, but they did, if you look at their bottom line of their earnings, it’s going to show that they spent a lot of money to get that, but all of those people that bought on TikTok Shop during the holiday season, TikTok now has your quick checkout information, so it’s going to be extremely easy to click a button and get something shipped. So I think social has come through the iOS update extremely well.
And I did say in that episode to look at GA for insights, which at the time made sense, Google Analytics. We were still in Universal Analytics, but since, we’ve been forced out of Universal Analytics into what’s called GA4 that I just can’t advocate for. You want to have it because free and it’s a good backstop and you need to make the data as correct as you can and make sure that the pixel is tracking from the social platforms post iOS and even Google Ads have the right data so you need to probably look at a little data or field to make sure that your Google Analytics is tracking correctly. But GA4 is rough.

Jon:
Yeah, I’ve heard recently somebody, I don’t want to name names, but somebody was saying it’s GA good for nothing because it really is. It’s really rough.

Ryan:
It is.

Jon:
They took a good product and slaughtered it.

Ryan:
Yeah. They took it out behind the barn and it’s not good. So if you liked Universal like I did-

Jon:
It’s back there with Optimize, right?

Ryan:
Yeah. It’s like why would you do that? But they were forced to. In a privacy centric world online, Universal Analytics wasn’t going to work and this was their best shot at something that just kind of functioned. It was a rebranding of some other app thing. So I can’t necessarily fault them because they had to move somewhat quick, but it doesn’t feel like they’re investing in that platform.
And so I think most brands need to find an external tool, and I think right now, Triple Whale at scale for small and big brands is phenomenal, and I think you need to be looking at that and making sure that that is tracking, especially on social because it’s allowing you to see attribution better. No commerce is going to be important to seeing post-purchase survey stuff. Really powerful on TikTok, especially when you’ve got … and when people don’t buy on TikTok shop, you’re going to see three, four-week attribution windows that is just crazy long for me in my mind of a $50 product that they saw a month ago and then go to the website to buy.

Jon:
You’re more of an impulse buyer.

Ryan:
I’m an impulse buyer, but it’s probably largely because my brain can’t think back that far. So if I saw something a month ago and I didn’t buy it, you as a brand have probably lost me forever. You’ve got to buy my eyeballs again because I-

Jon:
Or it’s stuck in your subconscious and then you see it on TikTok again and you’re like, “Oh, yeah.”

Ryan:
Yeah. Social is going to be doing very well moving forward still. But I think you, as a brand, you need to make sure that you’ve got additional insights into that social channel to be able to push it appropriately. And so if you’ve seen your social spend drop because you can’t see the ROAS in the pixel, you’d better be focused more on creative. I think in that one, I said very granular campaign builds on social channels. Now because of the AI, back to my previous one, the automation built into the Meta in particular, you can get more general ad groups targeting larger audiences and then test creative.
And so I think social, instead of focusing really on bid management and getting really nuanced into the data, it’s becoming more general and it’s more of a creative focus, honestly. And so you’ve got to have good creative you’re testing. You can put a good amount of creative in an ad group, find the winners, test more, get rid of the losers in the creative side and keep leaning into an audience to try to find better and better creative and keeping it fresh. If you have stale creative, which I’ve seen a lot of brands like, well, this worked really well in 2021. It should still work for cat lovers in 2024 because people still love cats. Yes, I do love cats, but there’s still going to be a lot of better creative that you need to be looking at for it.
And I think you need to be dark posting. So if you don’t have a plan right now on social, in this post-iOS arena, influencers are key and you need to have ones you can dark post on and leverage the trust that they have to find new users at a good cost. It’s not going to end the game for you if you don’t have influencers, but it’s going to make your life a lot easier. And so have a plan of acquiring influencers steadily. And it’s not fun, by the way, because there’s not a real, solid, inexpensive system that lets you go out and scale influencer acquisition. It is a time intensive thing that your social media manager probably needs to be spending time daily doing, nurturing and finding new influencers.
And that mid to upper funnel post-iOS, Facebook was doing really well mid and driving people into the lower funnel for acquisition. It’s just their creepy data was phenomenal. They’re still in the mid funnel. I don’t envision them getting really deep into the lower funnel anytime soon. But there are other channels you can look at. YouTube from a social perspective, I still am not a fan of. Probably four years ago, I probably mentioned it as an opportunity because I did like it pre-COVID, and then COVID messed up a lot of their data it seems, and it’s just not great.
It’s very top of funnel where now you’ve got some video platforms like MNTN, for example, that is designed for e-com brands to get a ROAS on connected tv, which four years ago I could not have envisioned. It’s like the democratization of television ads now where you might not be able to go linear TV, which is I’m going to go buy a Superbowl ad because the budgets are too big. But you can do connected TV because everybody’s got smart TVs and everybody is streaming everything now that there are ads available with us down to a thousand dollar a month spend. So you can do connected TV inexpensively, as long as you’ve got a reasonable way to get creative and test it.

Jon:
That’s great.

Ryan:
Yeah, excited about post-iOS. We may not have some of that creepy data again, but the market has come back with really good data that we can optimize off of.

Jon:
That’s great to hear because I remember everyone being so freaked out about this iOS update. And look, I like the privacy, but man, that’s going to hurt my business. And so I’m glad to see that we all got through that. Typical Apple fashion, when they kill something, usually it ends up being really painful to start and then in the long term, it works out.

Ryan:
Except GA good for nothing. That may have been an output of that.

Jon:
Well, that’s a Google issue. Yes. Yeah, that’s true. Fair, fair. All right. So my next one is a phrase that I have said a lot on this show and that is test everything. It’s interesting. I just want to set the record straight on this because I’ve said it so much, I feel like I need to, kind of, I don’t know about walk it back, but I will say that this is an area that the team and I at The Good have definitely changed our thinking around. And I think that the word instead of test should be validate. Brands should be validating everything, not necessarily AB testing everything. And that’s really the key shift here. It was episode 47 where I introduced what we call rapid testing as the next evolution of CRO.
That’s really where rapid testing comes from, is how do we augment these heavier lifts to just be able to do something and improve it out. Now, in most cases, you’re right. I’m going to say, “Hey, you really need to be doing full AB testing or you really need to be thinking about doing a more iterative approach. This isn’t just a quick win.” And I think the goal here is going in with your eyes wide open.
So wow, that was two years ago now that I was thinking about that. And I talked a little bit about how it’s changed how we thought about AB testing at the time. And I think we even over that last 18 months or so, we’ve really refined more to call it rapid validation, and specifically using the term validation instead of testing because we’re doing a lot less AB testing these days and a lot more of rapid validation strategies.
So what is this doing? Well, this is really allowing us to move much more quickly, but still be data backed. That’s the benefit of AB testing, is to be data backed, but here, we’re researching a lot of optimizations on that digital journey and then we’re quickly being able to validate those opportunities as opposed to having to do AB testing, which can take a month, two months depending on traffic. Or we could run a small test on a high traffic page and get an answer in a week, but it’s probably not going to move the needle because it’s going to be such a small test.
So really, the idea behind this is that it’s a cornerstone of risk avoidance, and it’s really a way for teams to save money before investing in optimizations or design changes. So that’s really something AB testing can’t do. So also, I would say as CRO has become a commodity, the leader of that has been AB testing. AB testing has become a commodity. Every pop-up tool, every Shopify app, all of those, all have some type of AB testing built in. Heck, I was even approached by an SMS company a couple of weeks ago who wanted to do a call because they were like, “Hey, I really want to know more. We want to build in AB testing in our platform. Can we discuss that a little bit?” And I thought that was really interesting because now everybody, every tool has an option to AB test, whatever you’re doing in that tool. So it really has become a commodity.
And I think that because of that, it’s become watered down, and the only way to really move quickly is to really do what we’re calling validation, rapid validation, which is other research methods that are not AB testing. So that’s where I’ve changed my thinking. I don’t think you should test everything. I think you should validate everything.

Ryan:
Now, what I hear when you say that is, oh, you can now move much quicker and maybe CRO is less expensive. Now, is that true or does that mean I can get to a higher conversion rate much quicker because I don’t have to go through the extensive AB testing and I can go like that, “Hey, that change was great. Go, go, go, go, go.” Because that’s what I would love to do.

Jon:
Yes. Well, that’s part of the goal, is to move quicker with optimization. So that was one of the goals. And one of the things we consistently over 15 years of doing this have heard from our clients is, “Hey, we want to move quicker. We want to move quicker.” And we’re like, “Well, the data can only move as quickly as the data can move. That’s not a limitation we’re putting on it. It’s not a limitation you’re necessarily putting on it, but there’s just not enough traffic to move that much quicker.” And it’s also why we’ve always had a minimum traffic level for working with brands, because otherwise, we couldn’t possibly AB test.
So this solves both those problems and it allows us to move a lot more quickly. Now, does it cost less? I would say the value is the same, if not more. So in terms of a budget, I would say the investment levels needs to still be there, and I think the return on that investment is definitely there, if not more than it was with just AB testing. So in that sense, in terms of the budget, yeah, I bet you could start finding people eventually. Just like AB testing became commoditized, this type of thinking will become commoditized a handful of years down the road, and in which case, the price will come down. Right now, it’s not a cost benefit necessarily but it is a speed benefit.

Ryan:
Somewhat related but I guess a side note, do you envision platforms themselves doing CRO at scale? If you think about Amazon, they’re doing the CRO for the company selling on there, essentially. They know it’s their platform. They’re making money on it. Shopify, in a similar fashion, they make money on every transaction because they’re the merchant processor for most of those brands on there. In theory, it could make sense for Shopify to be like, “We’re just going to help everybody do better by doing the CRO on all of our templates and everything.”

Jon:
Yeah, it’s interesting. Maybe I’ll just answer that by jumping into my last point here that I’m going to change my mind on, which is the future of CRO. We’re going way back to episode 14, which is when you’re making predictions on the future, it makes it pretty easy to look back now and say, “Oh, shoot, we had an episode about this.”
It’s so true today that people are used to Amazon checkout. They’re used to the Shopify checkout. They’re used to these platforms that have grown to be the monsters in the space. And if you really deviate from those best practices, then you are potentially creating a barrier.
I started looking through all these episodes and I was like, “Oh, shoot. I actually did have a whole episode where I made predictions, and I wonder how those have held up.” But one of those was that Amazon is going to impact the way we view checkout and conversion process in general. The prediction was I agreed that most platforms would move to a checkout that resembled Amazon’s just because consumers get used to conventions and makes it easier for them to follow those conventions. And the reality on that is that instead, Amazon has really focused on two initiatives that have had way more impact, and that’s buy with Prime and Amazon Pay. And in fact, the standardized third-party checkouts like Bolt have pretty much gone under. I don’t know anyone who’s using Bolt anymore-

Ryan:
Or if they are-

Jon:
But all of those type of …

Ryan:
We have a shared client that’s like, “They’re expensive and it doesn’t seem to be showing the value anymore.” I’m like, “I would agree with the data on that.”

Jon:
Yeah, yeah, yeah, exactly. So I think that unfortunately, I was wrong on that. I did think that Amazon checkout would really become the standard. I think now Shopify has done, as you mentioned, has done a good job with their checkout, but they even came out with the ability over the past year maybe, the ability to alter the checkout for almost everybody. Nobody is really altering the checkout. Everybody you go to, it still looks like the Shopify checkout. So if folks aren’t doing it, it’s really not having much of an impact right now.

Ryan:
That future of CRO episode you had, let’s just keep going on that. All the Google updates going through that, it just changes so fast. We could make a prediction that’s exactly opposite of what we believe, and it could have actually be right two years from now. That’s how fast we change now. So what were you seeing back then?

Jon:
Yeah. At the time, Google was just rolling out the updates to include site experience into their organic algorithm where they were saying, “Hey, you need to have a good site experience or we’re going to start ranking you lower.” So the prediction at the time was that if you haven’t optimized your site’s consumer experience, it was going to impact your rankings and your organic traffic was just going to go way down.
The reality on that is, it’s true, they did put a lot more emphasis on user experience, but it’s also false in that it didn’t really force brands to make as many changes as I would have liked to have seen. I thought it would have a way bigger impact, but also that’s a good thing because, for The Good, because there still is a lot of needed help from these brands, and I think Google did not do a good job of giving them a checklist. Did not say these are the specific things we’re ranking you on. They instead came out with site speed tools and they have a couple of webmaster tools and things that really give you some idea of, “Hey, your buttons are too small or the text is too small or things are too close together on screen.” Those are really, really weird high-level tips. And how do you know, oh, I need to make that button bigger? Okay. That’s not true optimization.
So I think the list of what they’re ranking you on and those type of user experience aspects is pretty poor, quite honestly. So I don’t think it made the brands do as many changes as I would have predicted.

Speaker 1:
You’re listening to Drive and Convert, a podcast focused on e-commerce growth. Your hosts are Jon MacDonald, founder of The Good, a conversion rate optimization agency that works with e-commerce brands to help convert more of their visitors into buyers, and Ryan Garrow of Logical Position, a digital marketing agency offering pay-per-click management, search engine optimization and website design services to brands of all sizes. If you find this podcast helpful, please help us out by leaving a review on Apple Podcasts and sharing it with a friend or colleague. Thank you.

Ryan:
Google and tying conversion rates to organic results just still seems so foreign to what the reality is because maybe a conversion to one brand is not what they’re tracking because some people, some brands I can look at are tracking the conversion or an event in analytics as a page view. How does Google know, just looking at high-level data what that conversion is and is that really valuable to the user? It could be yes or no, but it would require Google to have almost a second analytics behind the scene that says, “Oh, we are tracking our version of conversions, not letting the brands decide what their value is. We’re going to try to decide what that success on that page looks like.”

Jon:
Makes me wonder if they’re actually doing that. It’s possible, but again, privacy changes, privacy laws as of recent, highly unlikely, but it’s possible.

Ryan:
It’s possible. I think too. They have to focus just on content, I think, and what’s on that page and the basic metrics of a website,

Jon:
Yeah, which dumbs it down enough that is less useful.

Ryan:
This is great. Where else were you wrong, Jon?

Jon:
Let’s see. What else did I say? Oh, I have other places I was wrong. That’s for sure. All right. So I said that CRO would become more accessible to brands of all sizes, especially those smaller brands. So my point was that AB testing tools had gone from $10,000 a month to free in terms of Google Optimize at the time, RIP, and that happened over a span of just a few years. Because it became more popular, everybody wanted to do it, but most brands couldn’t afford the toolset alone to do it.
So I predicted we were going to see a democratization of CRO and that would continue to happen. The reality is that 100% that happened, I know this is one I got right, but the challenge with this is really that it has watered down the effectiveness of CRO as a whole, as an industry. Like SEO had gone through this a while back where if you knew SEO and it was held closely by a handful of people who did it right for the most part, but there were a few bad actors in there who did some black hat stuff and really then they had to combat it, and it just became more democratized over time and more people would try more black hat stuff, and you would end up chasing a Google algorithm at that point.
Well, I think now, what has happened is it’s so easy to go on Twitter or X and find an influencer who, and I’ll put it in air quotes, “does CRO.” And for a thousand dollars a month, they’ll say, “Hey, we’ll run some AB tests on your site.” But again, your site shouldn’t even be running AB tests because you don’t have enough traffic and you’re not going to get the needle move that much. So yes, it is now accessible, but should brands be doing it? I don’t think so. That’s the challenge.
I think there’s the misalignment of how it’s happening and who is using it. So I was right in the sense that it technically is accessible to more brands now, even small ones. You can go get it, pay a lot less. The tools are darn near free. There’s no more free tools out there, unfortunately. But I do think that it’s good to see more awareness out there. I disagree with how it’s being enacted. Yeah. Let’s see.
One other one was, I think the question you posed in that episode for me was, what do we all currently expect on an e-commerce site that will not be part of an e-commerce site in two or three years? Well, here we are three years later, and I said that putting your credit card in a website is going to go away completely. I said, “You know what? There’s too many easy ways to pay.” At the time, Bolt was really coming up. You had all these other one-click checkout things. I was like, “There’s no way you’re even going to have to have a credit card or know your credit card anymore.”
The reality is it has not gone away a hundred percent. Apple Pay, Google Pay, Amazon Pay have all made it super easy and it’s definitely out there and it’s prevalent at almost every site. I think if anything, what has happened is PayPal has gotten slaughtered more than anything else. PayPal is very rarely an option on sites to pay anymore because you have to go to this third-party site, still got to log in, still got to do all these. It’s not one click. And yeah, it used to be the closest thing to that, but now with Apple, Google and Amazon having one-click checkouts and payment options, it really has made it super easy.
But credit card, still an option. And credit cards, I still use it. Yeah, I have everything stored in 1Password, so it’s still close to one click for me. It’s not exactly one click, but it’s close. But I still use that a lot more than I use any of the other branded one-click pay options. So I was wrong on that. It has not gone away completely.

Ryan:
Well, and I think part of the reason is because Google and Apple have made it so easy to just have your credit card stored on Safari or Chrome and it’s like a face ID, it automatically fills it in, or I put my CID for my credit card in on Chrome, and it’s not a big deal. It’s easier for me to do that than go, “Go authorize Amazon Pay.” Just even Shop Pay annoys me with their texting. I’m like, “Just no. Let me just put my credit card in and I don’t need to be in anybody else’s list to keep that going.”
And I think that’s part of the problem too, is I think, at least me, I feel like just the unsubscribe process just becomes just stupid and it annoys me enough that I’m like, “Yeah, I’ll do this, but then I have to go back and unsubscribe, and if I forget, I just keep getting emails and it annoys me that I just prefer not to do it.” And I’m probably in the minority in that I hate texting or brands texting me. I’m not a fan of SMS nurturing, even though I have to advocate for it because some people like it and respond well, but a lot of these pay things …

Jon:
Well, hey, you do read it. It does get to you. Right?

Ryan:
Yeah.

Jon:
And I think that’s their selling point right now, is email, you have no idea if it’s going to actually get there. SMS, very likely someone has seen it. And that’s really the, I don’t know if we can call it arbitrage at this point, is that you know that people are going to see it. I think that will eventually fade.
Our power was out for eight days here in Portland at our house, and I ended up spending it with some family who has two high school boys, and their phones had 200 unread text messages. And I’m like, “How do you do it?” They’re like, “Well, I saw the notification. I just didn’t mark it as read or I decided not to engage.” Or they were like, “Yeah. It’s like every brand I’ve ever ordered from has texted me and I don’t want them.” And I’m like, “You don’t just reply stop?” They’re like, “Oh, I can do that? Just reply stop and it shuts them off?” I’m like, “Yeah.” But they’re, I don’t know, lazy or just don’t care. They just don’t do it.

Ryan:
Once you fall back far enough-

Jon:
It’s not important to them.

Ryan:
There’s not a way yet to mass unsubscribe. I want to select these 50 messages and I want to push stop in all of them. Hopefully, Apple, if you’re listening, do that. That would be great for those types of people. I just want to stop all of these from these 50 ones I just clicked. We’ll get there, I assume, and once there is a simpler mass opt out or something like that, yeah, I’ll give you my text information for that and I’m going to immediately push stop because I got my 25% off or whatever.
All right. So to finish it out, I’ve got my last one that I think back in episode 46, I talked about balancing traffic.
I would say minimize profit. We’re going to eat beans for a while because we have to get big enough to be able to insulate us with our own data and our own customers so that we don’t have to rely on continuing to push against a Google algorithm that is determining who wins and who loses. I don’t want somebody else telling me I’m a winner or a loser. I want to determine that myself. And I think that the writing on the wall right now says that’s where we’re going to be in a couple of years.
As I was talking through this at the time, we were still in the wonderful world of Universal Analytics, and I wanted brands to be focused on balancing your channel traffic so that you’re not dependent on a channel because things can change quickly. So if all of your traffic was coming from Facebook before the iOS update, you were in big trouble. And the whole point really was how do you avoid that as a brand so that you’re diversified? Like an investment portfolio, you want to be diversified, so if bonds go down, stocks go up and vice versa. So you could be fluid.
So I do still think that is relatively decent advice, but my thinking has evolved around channel diversity to more about looking at it from a funnel diversity. You don’t want to be just down at the bottom of the funnel. You want to be mid funnel and upper funnel. And generally speaking, your bottom of the funnel hasn’t changed much in the last few years. There’s demand capture on Google at the bottom as people search for your product be there. You’ve got email and remarketing all at the bottom where it’s more I want to capture the transaction.
I think it’s really where you look at the mid funnel. Where are you moving people from awareness to consideration? That area is more about, “Hey, I need multiple opportunities in the middle of the funnel in case one goes away, and I still need to be at top of funnel.” And so making sure that in those areas, you’ve got multiple areas that are being covered and that you’re still pushing people through the process.
And I think in that one as well, I thought TikTok was more of the impulse buy. We had a brand that was at the time doing really well at a 39.99 price point, and we saw them growing on TikTok. They advertised on it and it was like these people just see it and then they buy it, and that’s where TikTok is going to live. If you can’t get somebody to buy a hundred dollar product where there’s more consideration, very wrong obviously now that we’re in a different world of TikTok and attribution and we can now look back with better tools like, for example, nopCommerce and Triple Whale, which I’ve already mentioned, allow you on TikTok to see that it’s about a four-week consideration process on TikTok. It’s not the impulse buy that I originally thought, but if you don’t have that funnel diversity to keep them in the conversation and through the consideration process, you’re likely not going to be able to see that.
It’s not going to be, yeah, I advertise on TikTok and then four weeks later somewhere, they’re going to come back to my site and they’re going to buy something. No, it actually takes some work and you have to be able to say, “All right. I’m going to hit you with this ad and then you’re going to be on this ad and I’m going to remarket to you through various channels, maybe through Google remarketing, through social channel remarketing on Facebook.” It’s not a one and done, and you have to be very considerate and aware of those different channels and how they’re going to be moving through that.
And then we now have influencers and Spark Ads on TikTok that are able to do some quick things, so they are sometimes down further at the bottom of the funnel, and so you have to be aware that social, while generally it’s going to be on awareness top of funnel, but also mid funnel pushing them through, if you have an influencer with aggressive promotions with you or a special version that’s just for their followers. For example, I’ve got a company that we’re partnered with that’s going to be working with Mr. Beast. At this exact moment, it doesn’t get bigger than Mr. Beast on social if he’s advocating for you. There’s not a brand out here that would say, “Don’t work with Mr. Beast.” I would give Mr. Beast part of my company and a special product for his followers only. It would be dumb not to.
And so being aware of doing that and saying that there is some opportunity at the bottom of the funnel with social, but again, diversifying that, knowing that we’re a month away from a dramatic change that we weren’t expecting. And you have to be able to see that and pivot and move your money appropriately.
And even, look, we had a brand yesterday that somebody hacked into their Meta account and put them on, at least Facebook and Meta are investigating this, but they hacked into their account and put them on an invoice program, brand didn’t know. Now they’re three months late on their invoice program. Meta is like, “Hey, you didn’t pay your invoice. You’re off.” They got cold. “Done. You’re turned off.” And they were investing heavily in Meta and had moved Google budget up to that to capture more awareness than they had in ’23. Had to pivot yesterday. We’re like, “Hey, thankfully we have all these campaigns that were mid funnel on Google that we can now boost back up.” But those things happen. We are in a hacker-centric world online now that something-

Jon:
Got to use that two-factor authentication, right?

Ryan:
Yeah. And still, it’s just started so we have no idea. I’ll probably have to revisit this on this podcast in a couple of months when we find the solution like, “Okay, this can’t happen. Brands, protect yourself against this.” But at this point, we didn’t even know somebody could get into the finance section of Meta, apply for an invoice, put them on it, and then somehow that you got to pay it. We don’t know where. So just be ready to pivot quickly and have that channel diversity, ready to move money as things dictate when you start looking at the data.

Jon:
Well, Ryan, thank you for being a great co-host for a hundred episodes. I look forward to the next hundred. I was shocked after a hundred episodes, we still haven’t ran out of things to talk about, so I think we should keep going.

Ryan:
Yeah. Things keep changing. We’ll keep talking.

Jon:
All right. Well, thanks for your time today, Ryan, and congrats on a hundred and we’ll see you for 101.

Ryan:
Thank you. You as well. Thanks, Jon.

Speaker 1:
Thanks for listening to Drive and Convert with Jon MacDonald and Ryan Garrow. To keep up to date with new episodes, you can subscribe at driveandconvert.com.

The post Drive and Convert (Ep. 100): Our Biggest Lessons From 100 Episodes appeared first on The Good.

Maybe you’ve taken a look at the data before you reached your prescribed sample size and felt the test was already giving obvious results in one direction or the other.

Almost every product owner running tests faces this dilemma. And trust me, I get it.

The pursuit of results at an adequate sample time takes patience, and there are plenty of reasons to try to rush to the finish line: As you wait for a test to run, you may be prolonging a negative experience on your site. Or you may be losing out on presenting a positive experience to all your visitors. You also could be struggling to hold off stakeholders who want a quick decision and implementation.

But it’s crucial to let your tests run for their pre-established amount of time. If you don’t, you risk “peeking,” which increases error rates, causes False Positives, and invalidates results.

What is peeking?

Peeking is the act of looking at your A/B test results with the intent to take action before the test is complete.

Because most experiments have a 70% chance of looking “significant” before they are truly done collecting sufficient data, peeking at test results too early can introduce bias and potentially alter the course of decision-making based on incomplete information. Waiting until the test is complete allows for a more accurate assessment of statistical significance.

Think about it like this. If you flip a coin twice and get heads both times, you could incorrectly assume that the coin will land on heads 100% of the time.

However, if you flipped it a statistically significant amount of times, you’d get closer to the true rate of 50%.

Testing is the same. If you don’t allow enough time for an experiment to run and are constantly peeking at the results to make faster decisions, you are likely to make incorrect assumptions.

Plenty of mistakes can be made when running a test, but peeking is trickier to avoid than the rest. It is tempting to even the most experienced experimentation practitioners.

So, let’s talk about it and how you can avoid getting caught in its crosshairs.

How To Avoid Peeking: Set Clear Minimum Standards Before You Run A Test & Stick To Them

Before running a test, clearly define minimum standards to ensure the results are valid. These are the same standards to use when interpreting the results of a test.

Pre-determine your Significance Level

The general rule is to let your tests reach 90+% statistical significance, but the exact number can vary slightly depending on your team’s risk tolerance.

NASA scientists likely need a 99.999999% statistical significance before feeling sure of a decision, while an ecommerce site owner might only need 85% statistical significance to feel confident in their decisions.

Establishing the significance level helps your team come to a consensus about the level of error or False Positives you’re willing to accept in your test.

Achieve Appropriate Sample Size

Set a goal sample size that is representative of your audience and large enough to account for variability. It’s necessary to calculate your sample size before the test to determine how long to run a test to achieve rapid but reliable results.

If the test is stopped before it reaches a significant number of visitors, the results may not be valid. That’s because when the number of sessions or conversions is low, there is a high likelihood that changes will be observed by chance.

As the test collects more data, the conversion rates converge toward their true long-term values. This is known as “regression to the mean.” We often see a false positive on the first day of running an A/B test, and we expect those changes to regress to the mean or “normalize” over time.

For example, we might see a novelty effect of existing users who are already familiar with your site who are reacting positively to the changes made in your experiment. That would result in a false positive that would normalize as users get used to the change. That’s why seeing 90+% statistical significance isn’t a stopping rule alone.

Set a Minimum Test Duration

Letting your test run for a pre-allotted amount of time is key to avoiding the pitfalls of peeking.

We suggest a minimum of two weeks to account for two full business cycles. This leaves room for any unexpected variables (maybe your competitor is running a sale that week, which lowers your traffic volume, or there is a federal holiday, so fewer people are online shopping).

One important factor is looking for a good understanding of the performance range on any test, and that range gets smaller with more data. Testing tools may show statistical significance with even a small sample size, but even those tools will recommend running tests for at least two weeks.

Like I said, before that, you’re peeking, which can cause you to have false positives.

For a test duration cap, everyone is different. As our Director of CRO and UX Strategy, Natalie Thomas, says:

“Every team has a different tolerance for test duration. I know teams that will let a test run for six months and others that only want to prioritize initiatives that will see significance in two weeks. Having this litmus just assures folks are talking about their tolerance up-front.”

– Natalie Thomas

Set a Minimum Number of Conversions

Set a goal for the number of conversions or actions taken that will be a large enough sample size for your audience to know if the test was a winner.

This will vary based on the primary goal you’re focused on (e.g., ecommerce transactions versus inquiries, for example), so you’ll need to know an average number of conversions you get in a week.

Look for Alignment with Secondary Goals 

Part of the test setup process is defining a primary goal (for us, typically transactions or increasing conversion rate) that will determine if a test is ‘successful,’ but secondary goals will provide more insight into behavior, for example, adds to cart, visits to the next stage in the funnel.

If you’re at a point where you’re trying to analyze results, and these are not aligned (e.g., conversion rate is up, adds to cart are down, visits to product pages are unaffected), it could mean you don’t have enough data yet to tell the whole story.

A/B Testing Like The Experts

In the same way opening the oven before a cake is fully baked can impact the cake’s final consistency, prematurely analyzing test results can lead to skewed outcomes.

When testing on a website or app, the goal is to gather user-centered evidence that helps you make a decision.

You’re looking for a signal, not the final answer.

But to be confident in that signal, you need to set your tests up with pre-established standards. It helps your team align on when to end the test and makes sure you avoid the trickiest pitfall of experimentation: peeking.

The post What Is Peeking And How Do I Avoid It? appeared first on The Good.

You want to optimize a page on your website. You think running more tests will help you get more winning results. After a few weeks, you analyze the results and can’t decipher which tests led to what outcome because everything was active at the same time.

Not only have you wasted time, but you’ve also lost a portion of your budget on it. 

Contrary to popular belief, sometimes less testing is better in an optimization program.

For each test, your goal is to validate your hypothesis with data. Data can get overwhelming, and running too many tests on the same page or audience at once can lead to poor data cleanliness.

“It’s important to look at behavior goals to assess why your metrics improved after a series of tests. So if you’re running too many similar tests at once, it will be difficult to pinpoint and assess exactly which test led to the positive result.”

Natalie Thomas

Considerations for How Many Tests to Run

Test ideas should come from research. Testing is typically 80% research and 20% experimentation, so the more you research customer pain points and come up with strong hypotheses to solve them, the more you can determine quality test ideas.

So, you might be asking what is “too many tests”?

There’s no one answer to the ideal number of tests you should run. It depends on:

• Your optimization goals
• The complexity of your site
• Your optimization strategy

Guidelines in Testing

While I can’t tell you exactly how many tests to run, the following guidelines can help you determine if you are running too many or too few tests. As a general rule of thumb:

If your win rate is low, you need to increase the quality and tone down the quantity.

If your win rate is high, you’re too cautious, so your testing quality and learnings won’t be very meaningful.

A “good” win rate depends on what, where, and how you’re testing:

What: If you’re testing on a site with an enormous amount of data, you might feel comfortable failing regularly because even a small win here and there has a large dollar value in the end. If you don’t have a lot of data, your testing program takes time, and you’ll be tempted to make sure every test counts and swings for the trees.

Where: At volume, large companies can start optimizing even the smallest parts of the funnel, like the return customer dashboard and the reordering experience. Smaller organizations may want to focus on only the highest volume landing pages and the most popular products or services. Limited pages mean a limited number of tests, so your cadence or volume of tests won’t be as consistent.

How: For organizations with many variables contributing to the bottom line (e.g., cancellation rates, return rates), post-test analysis could take months. Long post-test analysis cycles may become a limiting factor in your testing velocity and win rate.

If you’re wondering how to find the balance between quantity and quality, the short answer is: You should only run the number of tests you can research and manage.

Once you shift your focus from trying random tests based on your gut feeling to solving specific problems your customers face, your results will increase significantly.

The post What Is The Problem With Running Too Many Tests? appeared first on The Good.