Sample Chapter: Design for Impact

This is a sample chapter from Erin Weigel‘s book Design for Impact: Your Guide to Designing Effective Product Experiments. 2024, Rosenfeld Media.

Chapter 1: Conversion Design Drives Impact

THE POINT
Many product design and development teams use opinions to make decisions. A better approach is to gather reliable evidence, measure the impact of their work, and make optimal decisions by learning from the Conversion Design process.

Have you ever sat around a table at work and discussed endlessly which version of a design is best? I know I have. Early in my first real tech job at a meal delivery service, I was asked to redesign part of a home page. The goal was to sell more of our secondary product line, which was meal replacement shakes. With the task in hand, I got started.

I designed one option with photos of people. Product photos dominated a second design, and a third one used only icons and words. I even had a fourth version with icons, people, and products, which, admittedly, was a bit much. Then our team had to decide which version to use (Figure 1.1).

Figure 1.1 Four different versions of the design execution to choose from.

We each shared our opinions:

• The marketer wanted pictures of people to“create an emotional connection.”
• The business manager wanted product photos because it“was clear what we were selling.”
• The developer wanted only text because it would “be quick to do.”
• And, I, the designer, wanted icons and text because it“looked less cluttered.”

These were all reasonable options, but which one would have the impact we were aiming for? We needed to break free from our opinions to focus on user needs and business goals. To do that, we went outside the office and did some quick preference tests with passersby.

“Which one do you like best?” we asked. After some feedback from strangers, we made a decision and rolled out the new design.

My question at the end of all this was: “Did the new design have the impact we wanted? Was it going to sell more shakes?” At the time, there was no way to know. The results were as “shaky” as the product we were trying to sell.

Even if we had looked at the sales data before and after the change, we wouldn’t have been confident that the change we made caused any of the impact that we saw. That’s because the change wasn’t isolated. The data could have been influenced by something else, such as the time of year, a special offer, an ad, etc. In this scenario, the data was iffy at best. It could have given us a clue, but it couldn’t give us confidence. In the end, we made an opinionated best guess. We released the “nicest looking” option with just logos and some text.

Conversion Design Is Evidence Collection

Conversion Design is a way to work that moves teams beyond opinion. It uses research, A/B testing, and data analysis to create reliable evidence, which enables consistently better decision-making. With this process, you can know with confidence that changes are resulting in the intended impact. In other words, did the change you made actually help you achieve your goal?

Evidence is at the heart of Conversion Design. It draws on the experiences of three disciplines (Figure 1.2):

• Design with its focus on solving customer problems
• Science by way of unbiased, evidence-based decisions through the use
  of the scientific method
• Business from a value-creation and transfer perspective

This type of experiment is called an interrupted time series. It’s unfortunately not very reliable unless it’s repeated multiple times.

Figure 1.2 Conversion Design combines design, business, and science.

Accurate Measurement Beats Opinion

A core model from science that’s used in Conversion Design is the hierarchy of evidence. This is a way to assess the reliability of different kinds of data or information gathered from studying something (Figure 1.3).

Figure 1.3 The hierarchy of evidence as used in evidence-based medicine.

The hierarchy of evidence’s broad base represents expert opinion. Although it’s the least reliable form of evidence—because of the likelihood that it’s biased—it’s nonetheless a decent starting point. True experts are people who have learned from a lot of evidence further up in the hierarchy.

TIP: TRUE EXPERTS OFTEN DON’T CALL THEMSELVES EXPERTS
A relevant quote attributed to Albert Einstein is: “The more I learn, the more I realize how much I don’t know.” Beware of people who claim unwavering expertise in a topic but who don’t have strong evidence to back up their claims. Seek out experts who want their opinions tested. True experts often doubt their own knowledge and embrace the fact they don’t know everything. Self-proclaimed “experts” are likely to state their opinions as facts and just push their opinions through.

One step up from expert opinions are observational studies. These include usability tests, shadowing sessions, and surveys. A drawback of observational studies, as with expert opinion, is the possible bias of the observer.

The next step up is randomized controlled experiments. A randomized controlled experiment is a scientific study in which participants are randomly placed into different groups. Random means that each visitor has an equal chance of being assigned to different groups. Splitting participants into randomized groups is a simple way to collect reliable evidence because it allows you to “isolate” a variable you want to learn about. Isolation ensures that the variable you’re examining is the only thing that’s different among the groups. This helps you know that if the results are different, it’s likely that your isolated variable is what made the difference.

Well-designed randomized controlled experiments provide very reliable evidence because they enable relatively accurate measurements. As the pioneering American computer scientist Admiral Grace Hopper once said, “One accurate measurement is worth a thousand expert opinions.”

Finally, at the top you have a systematic review. This occurs when scientists look critically and without bias at all the evidence collected about a given topic to form a well-rounded conclusion. This review provides a holistic summary of the findings from all available knowledge.

A/B Testing Your Way to Reliable Data

A/B testing is a common form of randomized controlled experiment. To run an A/B test, you make a change to one part of a user’s experience. Then you expose a random half of the users to the original experience and the other random half to the changed experience. To understand how it works, think back to the story of my team making a change to a home page at the beginning of this chapter. Figure 1.4 illustrates the original and the change.

Figure 1.4 The parts of the home pages that were tested are highlighted.

The original is called the control. It’s the “A” in an “A/B test.” You might also hear some people call it base, which is short for baseline. The change is called the treatment or variant. It’s the “B” in an “A/B test.” Instead of showing the change to every home page visitor, it’s only shown to a random segment of the visitors. The rest of the visitors see the original version. After the two versions (Figure 1.5) are shown to enough visitors, you can stop the experiment and learn if there’s an impact on sales from the result.

Figure 1.5 Here, 50% of visitors see the original (base) and the other 50% see the new version (variant).

What do you think the results will be?

• Will the change be too small to detect—and have an “inconclusive” result?
• Will it do better than the original—and have a “positive” result?
• Or, will the original do better than the change—and have a “negative” result?

Changes often make no measurable difference. But sometimes you get surprising results. In this example in Figure 1.6, the change performed 3% worse on overall sales than the control.

This change unintentionally made the customer experience and business outcome worse. Because the sample was randomized, we knew that the decrease in sales was more likely due to the change in the variant rather than chance. This effect was something we couldn’t have known confidently without an experiment. You need to know the impact of your work because your job is to create value for customers and the businesses that serve them—not destroy it.

Figure 1.6 In this case, the variant performed worse on the success metric. Sales of the secondary product line, meal replacement shakes, were less with the change.

Another Way to Think

Unlike other design processes, which are based on linear thinking, Conversion Design is based on systems thinking. Systems thinking, as illustrated in Figure 1.7, is a practice and set of tools that helps you understand and capture things in a dynamic and interconnected way.

Figure 1.7 Linear thinking (left) leans on simple cause-and-effect relationships. Systems thinking (right) identifies many interacting variables and how they relate to one another.

Systems thinking lets you understand how and where multiple things interact, and ultimately—their impact on one another. With it, you can represent and understand the world around you more accurately to make optimal decisions in the face of complexity.

Conversion Design works much like a machine (Figure 1.8). Teams work together and lean on each other’s strengths at different points in the Conversion Design process by representing a wide range of disciplines. Everyone plays an important role to get through each step.

Figure 1.8 The Conversion Design process enables teams to learn from reliable evidence to help them make better decisions.

As the gears in a machine turn and flow into and out of one another, they create an “output.” This output creates a loop (a “flywheel”) that amplifies the whole system by reinfusing energy back into the process (Figure 1.9).

Figure 1.9 The “flywheel” throws off energy in the form of collective knowledge, which gets reinfused back into the system to make it stronger.

In the case of Conversion Design, the reinfused energy comes in the form
of collective knowledge—the combined brainpower of a group of people. As you gain knowledge, you take it with you as you begin the process again. The main learning happens during the testing phase of the Conversion Design process in which a cause-and-effect relationship is examined. This cause/ effect relationship is a hallmark of linear thinking. In this way, Conversion Design benefits from both ways of thinking: systems thinking to embrace the complexities of the real world and linear thinking to isolate the cause/ effect relationships that teams can learn from.

Conversion Design Creates Value

Value creation—and the eventual transfer of that value to the next-in-line stakeholder—is the heart of all work. Conversion Design helps teams identify and deliver value, as well as finding and avoiding damage to value that already exists.

For customers, value comes in many forms. Common ones are time saved, money saved, convenience, and social connection. Value for business is often measured by growth in company profits through an increase in customer purchases, as well as increased customer attention and action. For employees, value might be compensation, benefits, and (hopefully) a sense of purpose.

Just as Conversion Design creates knowledge as an output, it creates customer value as well. It’s represented as a dashed line to show it’s not guaranteed, it’s intermittent (Figure 1.10).
Unfortunately, many people conflate the idea of “conversion” with just being the sales part of business value creation. In Conversion Design, conversion stems from the Latin word, convertere, which means to change. The root of conversion has nothing to do with money, value, or sales. It’s about transformation.

Figure 1.10 Conversion Design sometimes creates customer value as an output, too. But that output gets infused into another system.

The word design is also often misunderstood. It’s typically mistaken for “making something look nice.” But at its core, design gives shape to ideas and makes them real. In other words, “design is the rendering of intent,” as Jared Spool, prominent designer and founder of the design school Center Centre, calls it.

“Conversion,” when combined with “design,” means “to create intentional change.” But not just any change: improvement. True product improvement creates value for customers. And though Conversion Design might sound fancy, it simply means good design that fulfills its purpose of creating value for customers and the businesses that serve them.

The Value Cycle

Eventually, value created by the Conversion Design process for customers gets transferred back into the business. This concept is captured by the “value cycle,” which is a key concept within Conversion Design, as shown in Figure 1.11.

Figure 1.11 The value cycle transfers value from one stakeholder to the next.

The value cycle is a loop that represents how a company, such as a product or service, creates and delivers value to customers. Because it’s a loop, companies eventually get some form of that value back. The company uses this returned value to continuously improve or create new products through its employees and the tasks they do, which starts the cycle again.

The goal of Conversion Design is to feed this value cycle by understanding what changes (if any) are needed, who those changes would benefit (or not), and most importantly, ensure that any changes made lead to the intended effect. In a business’s value cycle, Conversion Design begins when the business invests in the Conversion Design process as the employee’s default way of working. When you put it all together, Figure 1.12 shows how the Conversion Design process “shakes out” graphically within a business value cycle:

Figure 1.12 The Conversion Design process drives the value cycle forward.

EXPERT ADVICE FROM THE FIELD

DIVERGE AND STAY OPEN WITH KEVIN ANDERSON

Many designers use the linear design thinking process called the Double Diamond to guide their work.8 It’s a two-phase process, as shown in Figure 1.13.

FIGURE 1.13 The Double Diamond design process diverges and converges to end up with a single solution.

In the strategy phase, you “diverge”—go conceptually broad—to learn about the problem and generate many ideas to solve it. Then you “converge”—narrow down your options—to decide what solution to execute. In the execution phase, you “diverge” by exploring the different shapes the solution could take. Then you “converge” to decide which specific execution of the solution you’ll deliver.

In my experience, designers often converge quickly and in isolation. When they do this, lots of great ideas never see the light of day. And in experimentation, the more ideas, the better. What I’d love to see is designers be more open with their ideas—share them raw, early, and often to get input. I believe that designers don’t do this because teammates and stakeholders don’t know the best ways to give feedback. Asking for and giving feedback are skills that teams should work to improve. My advice is that if you use the Double Diamond as a guide: diverge and stay open. Then share—and test—all the possibilities.

Kevin Anderson

Product Manager of Experimentation at Vista

Kevin is an active member of the experimentation community and founded www.experimentationjobs.com in 2020 to help practitioners find great places to work.

Conversion Design Leverages the Compound Effect

Many companies follow outdated product development processes, which don’t include systems thinking, the scientific method, and experimentation. They often release products without understanding the most likely impact of their changes. This approach results in releasing a mix of positive, negative, and inconsequential decisions. The positive and negative decisions cancel one another out, so the results reflect the mixed success rate, shown in Figure 1.14.

Figure 1.14 Typical product development results in both good and bad decisions, which leads to varied impact.

With Conversion Design, teams learn and improve. As a result, the products that teams build improve, too. The Compound Effect is when many small, good decisions combine over time to create an exponential effect.9 Conversion Design creates this effect by lowering the rate at which teams make bad decisions. To get the most benefit from the Compound Effect, you should only keep the positive changes. Toss the unimpactful and negative changes in the trash. In time, good builds upon good, upon good.

The results of those consistently good decisions executed over the course of a long period of time could look something like Figure 1.15, the coveted “hockey stick” growth line that most businesses aim for.

Figure 1.15 Many good decisions compound over time create an exponentially positive effect.

Initially, the line stays relatively flat and grows slowly. This gradual incline is normal at first. Eventually, when the Compound Effect takes hold, it spikes quickly. Working this way helps you create and receive value faster than your competitors, which gives you a big advantage. When you stay agile and learn fast, you can adapt to the changing competitive landscape and focus on delivering the most impactful work.

Good Process Is Important

At this point you may think, “Another process?! That’s exactly what we don’t need.” And that’s understandable. Processes get a bad rap because businesses pile one on top of another, which can slow things down. But the right type and amount of process can ensure quality, speed, and alignment. When you don’t have a clear process, it leaves room for stakeholders to disagree on the approach. This can cause different—sometimes conflicting—decisions. And when people don’t have the same goal and the same plan to get there, team confidence and employee engagement drops.

Tried-and-true processes overcome this. When people know what needs to be done and how, they can understand where they add value within the process. Better yet, they understand where their teammates bring value, too. This understanding of one another’s strengths cultivates healthy team dynamics, which (in addition to helping you not hate your job) increases the team’s collective intelligence. Collective intelligence is the high-quality thinking and creativity that emerges within a group. It’s powerful because when diverse minds work together, two benefits emerge:

• Lots of unique perspectives, which build empathy. And expanded empathy leads to more effective ideas to serve your ideal customers.
• A broader skill set, which increases the pool of skills available, and leads to better execution of those ideas.

When you combine these benefits, your team can outperform any narrow-minded competition. But you need a solid and inclusive process to get everyone on the same page.

The Seven Steps of Conversion Design

At the end of the Conversion Design process, you should know with confidence that the change you tested was good, bad, or made no difference. But exactly how do you get to that point? Conversion Design involves these seven interconnected steps or phases:

1. Understand. In this phase, you look at customer problems through the lens of business goals. The purpose is to develop actionable insights that will enable you to make measurable product improvements that positively impact both customers and the business.
2. Hypothesize. Next is turning those insights into testable ideas. These are thoughts or questions that experimentation can help you answer. This is done by way of the scientific method, which is a structured thought process that allows you to collect and learn from reliable evidence.
3. Prioritize. From your list of testable ideas, you strategically guess which ones are likely to solve customer problems in a way that positively impacts the business. Then you weigh short-term gains against long-term business goal alignment and put them in an order that makes sense.
4. Create. Here is when your idea begins to take shape. You design and develop a testable product change (i.e., a new feature, content refinement, different process flow, etc.) to expose to your customers.
5. Test. This phase introduces your testable idea to the real world via an A/B test. This part of the process is where you can discover cause-and-effect relationships that you can learn from.
6. Analyze. Once the test is complete, you look at the data to understand the impact of the change. At this phase, you learn if the overall effect of the change made the impact you were aiming for—or not.
7. Decide. Finally, you use what you learned during the process to make an optimal decision for all stakeholders.

Now let’s step into the Conversion Design process and cycle through the stages. First up: The Understand phase in Chapter 2.

The Important Bits

• Teams no longer have to rely on opinion to make design decisions. They can use the Conversion Design process to collect reliable evidence to make optimal decisions.
• Conversion Design is a product development and design process, which is based on systems thinking. It captures the dynamic, interconnected nature of reality.
• Conversion Design combines three disciplines: science, business, and design, in order to drive measurable, positive impact on both the customer experience and business results by making optimal, evidenced-based decisions.
• Teams methodically work their way up the hierarchy of evidence to get reliable information to make optimal decisions.
• The most reliable form of evidence is a randomized controlled experiment. A/B tests are the most basic form of a randomized controlled experiment, and it’s an important part of the Conversion Design process.
• Teams should work together and lean on one another’s strengths to harness the collective intelligence of the group.
• The outputs of the Conversion Design process are collective knowledge and customer value. Value creation within a business’s value cycle is what moves the business forward.
• Eventually, through experimentation and optimal decision-making, the Compound Effect kicks in. This occurs when many small, good decisions come together over time (while throwing away the bad) to make a big positive impact.
• The seven steps in the Conversion Design process are: Understand, Hypothesize, Prioritize, Create, Test, Analyze, and Decide.

back to Design for Impact »