SUMMER SALE! Get up to 50% off our books

Shop now

Podcasts

Sample Chapter: Managing Priorities

This is a sample chapter from Harry Max’s book Managing Priorities: How to Create Better Plans and Make Smarter Decisions. 2024, Rosenfeld Media.

Chapter 1: The Missing Ingredient

It was autumn of 2014 in the Northwest, and the first time I’d been to Tacoma, Washington. This wasn’t an obvious hotel for a corporate workshop—not boutique-y at all—more like the “ranch-style, you-could-be-any where” type. The breakfast room had community tables draped with synthetic white tablecloths, so writing a to-do list on them with my Space Pen was out of the question.

Staring back at me through the glass sneeze guard were scrambled eggs, floppy bacon, oily sausages, and partially cooked hash browns. I set the halves of my bagel on the toaster conveyor belt, round-side down. Another day, another hotel breakfast.

I was hopeful that the 21st Century Leadership workshop I was here to attend would be a good change of pace. I’d been sent to Tacoma by Rackspace, the San Antonio–based cloud-computing company where I’d been working as VP of Product & Experience Design for several years. The senior leadership team (SLT) had suggested that this conference, and perhaps some time away from corporate headquarters, could do me some good.

By this point, Rackspace had outgrown its scrappy startup origins, but it was still a far cry from the behemoth it is today as I write these words: a company that posts over $3 billion in annual revenue and has over 6,500 employees working in over a dozen locations around the globe.

My mind wandered back four years prior to this conference, to when I was lured to Texas from California, the only place I’d lived and worked since high school, by my friend and former colleague, Mark, who would soon turn out to be my boss.

“Take a long weekend,” Mark had insisted as we sat upstairs at Red Rock Coffee in Mountain View in 2010, about a mile from Google headquarters. “Check out the Castle. Meet Lanham, the CEO, and Lew, the president. See for yourself. It’ll be fun.”

“The Castle?” I scratched my head. “Let me think about it.”

Two weeks later, Mark picked me up at the San Antonio airport. We got off 35, made a right, and drove up to a massive building surrounded by a sea of blacktop. Mark explained that Rackspace was housed in a converted shopping mall, but I had imagined it was a lot smaller. Outside the main entrance, there was a galvanized water tower brandishing the words “Home of Fanatical Support.” Intriguing.

Even with Mark’s earlier download, I was quite unprepared for what was to come when he flashed his badge and the receptionist waved us in. I stepped out of the cool San Antonio air and into the bustle of a corporate amusement park: various flags hung from the 25-foot ceiling as phones rang, computers whirred, and people hustled this way and that. A spiral slide connected the second floor to the ground floor. The people in front of me looked to be hard at work, and yet their smiles and laughter and sense of spirited collaboration made it clear that everybody was having a hell of a lot of fun.

“Inconceivable!” I thought to myself, channeling my inner Vizzini from The Princess Bride (as one does). I knew right then and there I’d be moving to Texas. That is, if they’d have me.

A vibration in my pocket jolted me back to Tacoma. It was Gigi, Rackspace’s VP of Engineering.

“Everything okay?” I asked.

“Not really. Mark is out,” she said, her voice uncharacteristically shaky.

“What do you mean he’s ‘out’?”

“They announced it this morning. They replaced our Mark. Can you friggin’ believe it?”

She interrupted me before I had time to fumble out a response. “Can you come back? We need you here. Now.”

“Sh*t,” I said, not entirely under my breath. My heart sank.

I probably should have seen this coming. Six months earlier, when Lew (the president and Mark’s boss) stepped down, I wondered how long the Rackspace ride would last. Then, a few short months after that, our beloved CEO, Lanham, the living embodiment of Rackspace culture, was unceremoniously ousted by the Board of Directors. 1 It stunk, but leadership changes like this happen sometimes when an organization changes strategic direction or top priorities.

One half of my bagel slid off the conveyer belt in all its mostly toasted glory, taking me back to the matter at hand. “What good will it do?” I asked.

A newfound clarity presented itself in Gigi’s voice. “I don’t know exactly. You keep everybody calm.”

It was an interesting observation, if not a touch ironic. Funny how consuming copious amounts of espresso can help one keep other people calm. “Let me see when I can get a flight.”

In those four minutes, I experienced what Elon Musk, love him or hate him, euphemistically refers to as an “unplanned disassembly. 2 ” Now all our best-laid plans, the decisions they were based on, and the priorities we had stacked on top of one another were all up in the air, spinning wildly like those numbered balls in a novelty bingo machine.

What should we do next? What could we do? How could I help to right the ship? Was it even possible to right it? If Mark couldn’t make it work, how could I, someone not on the SLT, possibly help?

Figuring all of this out was going to take a lot more than a one-person crisis response in a hotel breakfast room. It was going to take a systematic approach, a rigorous process that would make it crystal clear what mattered and what didn’t. It was all about priorities. Fortunately for me, everything I had accomplished up to this point in my career—on purpose and sometimes not—revolved around developing such a system.

I hung up and called the office, informing them that, God willing, I was going to be back at the Castle by the end of the day. I had a job to do.

• • •For companies and organizations to survive, they need to prioritize. But to thrive, their priorities must mesh like gears to synchronize the work that teams are planning and doing, so they can make progress consistently and predictably. And those teams, in turn, must be composed of individual contributors and front-line managers who are tightly aligned, laser focused, and exceptionally productive.

None of these statements are controversial; chances are you agree with them yourself. However, as soon as you step into the real world, you see how much easier it is to make these pronouncements than to implement them. In too many companies (perhaps yours?), workers are overwhelmed, stressed out, and have a vague sense they are not working on the right things or making enough progress. Teams seem to be working at cross-purposes and juggling too many priority ones (an irony worth thinking about).

The results are sadly predictable. Supposedly, must-do, no-fail organizational goals don’t get started, completed, or resourced adequately. Executives’ pet projects suck up precious resources. And morale—to say nothing of revenue—is well below what it could be. And all throughout the organization, there’s a nagging sense that things could be better. A lot better.

In hindsight, many of these tragic organizational failures are painfully obvious and often perfectly preventable. But what to do? Who has the luxury of hindsight or a huge interest in waiting around until things implode before doing what you can to chart a better course?

How Did We Get Here?

On the surface, there may appear to be many different reasons a company finds itself in utter misalignment. But, in my experience, almost all organizational troubles originate from a single cause: senior executives or administrators and frontline personnel seem to be living in alternate dimensions. Leadership is primarily concerned with figuring out what to do and why. Frontline management and personnel focus on how to pull it off and when.

This dissonance has steep costs: declining performance, eroding market share, crumbling brand reputation, and dwindling value for customers, shareholders, and other critical stakeholders. But the hidden costs may be even more insidious.

Doubling down on the wrong priorities is all but guaranteed to create an endless stream of burnt-out and overwhelmed individuals, no one more so than the new or newly promoted manager. The newly minted manager’s passion and commitment to quality—the very talents that got them noticed in the first place—quickly become casualties of excessive competing interests. Before long, constant demands disconnected from explicit strategy and direction, coupled with an underlying drumbeat to do even more, faster, now, reduce them to an exhausted, reactive shell just looking to get through the day.

Having the right priorities, and the self-discipline to limit those priorities, can offer a way out of this mess. Prioritization is the process of identifying items of any type and arranging them in order of importance, superiority in rank, or privilege. It’s the antidote to accelerating fragmentation, a defense against the infinite distractions that keep individuals, teams, and organizations from operating at peak potential. It’s the missing ingredient in a recipe that’s not quite right. Prioritization was, as you will discover over the course of this book, what helped me contribute to stabilizing the situation at Rackspace and turning things around at other companies since then.

Often when people hear the term prioritization, their mind points to personal productivity gurus and time management hacks. They think of tasks, to-do lists, and other techniques for getting more out of their day. These tips certainly have their place, but they aren’t the focus of this book. My main focus here is on deciding what to consider rather than concrete strategies for getting things done during your workweek (though we will touch on this some).

Prioritization is a lot harder than it might sound. For one, finding good resources on the topic is a challenge. Actionable, how-to information is too often buried deep in the guts of books, blogs, videos, and the like. And when useful information on prioritization is available, it’s usually not identified as such, which makes it hard to find.

In addition, in much of the business world, prioritization is typically the purview of project management, product development, and annual planning. The outcome is that management tends to view prioritization as no more than choosing which projects get funded or which products, services, features, and widgets should be built. In this view, prioritization isn’t a process—it’s a light switch.

Although they may not realize it, many organizations do have access to proven approaches for defining high-quality priorities. There are companies and consultants that specialize in it and powerful methodologies and tools available if you know where to look.

But in this regard, companies are like people: they don’t know what they don’t know. They simply rely on what they’ve done previously, reusing improvised approaches that spit out unrefined “priorities.” The problem isn’t just that they are prioritizing poorly, but that they aren’t even aware it’s the keystone problem to everything else, so they don’t and can’t fix it.

These slapdash priorities, in turn, become the inputs to their financial and operational planning processes. The results, again, are predictable: the lack of rigor and clearly defined objectives invariably leads to resource conflicts that ripple across the organization over time. Perhaps there’s a better way.

My approach, honed over my thirty years in the trenches of Silicon Valley and articulated in this book, is to make the process of prioritizing a first-class citizen. I offer a repeatable process and simple taxonomy for prioritizing anything, as well as including strategies, tactics, techniques, and tools you can use depending on your individual needs. I separate episodic, one-off efforts from periodic and continuous prioritization. Lastly, I look at the important differences between prioritizing for yourself and setting group priorities as a member of a team or a larger organization.

Proper prioritization grants a deep competitive advantage to the people, teams, and companies that master it. Yet, if you’re like most people, you don’t have a great model for doing it. I’m here to help.

Who Is This Book For?

I like to think of this book as a short self-help course for businesses.

Managing Priorities: How to Create Better Plans and Make Smarter Decisions combines real stories, practical tools, and timeless insights to equip anyone interested in becoming a more effective manager or leader with the skills, knowledge, and mindset to prioritize anything, anytime, anywhere. Being able to identify what true priorities are and are not, understanding the process of prioritizing, and learning to recognize the inputs and outputs of successful prioritization are all deeply valuable skills. Use this book to create more impactful plans and make better informed decisions. Use it to do a better job. Use it to lower your stress.

More specifically, though, this book is geared toward new or recently promoted managers and entrepreneurs involved in strategy activation, planning, or resource allocation in companies or organizations, for-profit or not. It’s for people who have recently increased, or want to increase, their scope of responsibility and are not yet overly committed to the status quo and default approaches. It’s also for students of business looking for a helpful primer on management.

The overarching theme of this book is business, regardless of industry. It does not offer domain-specific solutions, methodologies, or protocols for fields such as clinical healthcare, education, emergency response, or the military. Moreover, this book assumes a certain degree of knowledge. It is not meant for those readers seeking an introduction to managing people, projects, money, or data. What it does provide are strategies, tactics, tools, and techniques that are applicable across entire ranges of disciplines—including, most likely, yours. This book, and the processes it describes, can benefit leaders in almost every industry.

back to Managing Priorities »

Frequently Asked Questions

These common questions and their short answers are taken from Leah Buley and Joe Natoli’s book The User Experience Team of One: A Research and Design Survival Guide (2nd edition). You can find longer answers to each in your copy of the book, either printed or digital version.

What is a user experience team of one?
A UX team of one is someone who works in a situation where they’re the key person advocating for a user-centered product design philosophy. If you’re the only person in your company practicing (or aspiring to practice) user-centered design with little to no support, you are most definitely a UX team of one. However, even in organizations with multiple UX professionals, you may find yourself part of a team where you are the only UX person. ..which makes you an honorary member of the UX team of one club. Chapter 3 explains the kinds of challenges that UX teams of one commonly face—and explains what to do about them.
I’m a freelancer. Is this book for me?
The User Experience Team of One, 2nd Edition focuses primarily on people working in or with organizations. But while it’s not explicitly geared toward freelancers, consultants, or contractors, much of this book is still very relevant for independents, because they, too, must often work with the cross-functional teams of their clients. And if you’re considering going out on your own, be sure to check out the section “Going Independent” in Chapter 9.
What’s different about life as a UX team of one?
Working without the support of colleagues in the same profession, or without a high degree of UX maturity inside your organization, presents several unique challenges:
- You feel like a jack of all trades, but master of none. You do a variety of work: probably some design, some research, some writing, some testing, and some evangelism. You care about your work, and you want to do it well. But being a generalist, you likely feel as if you’re being spread increasingly thin. You may also wonder at times if you’re “doing it right,” or why all these techniques you read about every day on social media don’t seem possible in your environment.
- You need to evangelize—or justify. You probably work with or for an organization that doesn’t yet “get it;” they haven’t fully bought into the value and purpose of UX. Or, even if they do value user experience, they may not be in a position to fully fund and build a robust UX practice. Either way, this means that in addition to your daily workload, you’re constantly tasked with fighting for the UX improvements you propose, as well as trying to influence and increase the UX maturity of the organization as a whole. That’s a pretty tall order.
- You’re learning on the job. You don’t have anyone to turn to for advice, so you need to figure out how to do your work on your own. You may have UX or product design professionals on social media or professional communities that you can turn to for peer-to-peer advice, but in your day-to-day work, you’re on your own. Quite often, the best you can do is make an educated guess—and then trust and defend your hunches as to the best next steps.
- You’re working with constrained resources. The biggest challenge for teams of one is time. There’s enough work on any given day to keep an entire team of UX folks busy—but there’s only one of you.
- You’re charting your own course. No one in your organization has done this before. So, you’re figuring out your own career path without a guide or a manual to follow, trying to build the airplane as you fly it.What makes this role interesting is the dramatic tension between needing to inspire through expertise and trying to build your own expertise at the same time. That dynamic means you have a very unique set of challenges that go well beyond simply trying to do good UX or product design work. It makes skills like facilitation, flexibility, assertiveness, and persuasiveness central to the team of one’s toolkit. This tension requires both practical and philosophical considerations—and that simple fact is the inspiration for this book.Chapters 2 and 3 explain the working conditions that a team of one often experiences, while Chapters 4 through 9 provide specific methods that are optimized for those working conditions.
Is this just an intro to UX book?
Yes and no. While this book is intended to be accessible to people who are just starting out in user experience, we believe the advice and techniques here apply equally to seasoned practitioners. For example, Chapter 1 provides an overview of user experience and can serve as a basic introduction to the field. But the methods in Chapters 4 through 9 aren’t the same typical UX methods you see, read, and hear about every day. We’ve chosen them specifically because of their power to educate and involve others who may not be familiar with or supportive of user-centered design, while requiring less time and fewer resources than their more formal counterparts.

« back to The User Experience Team of One

Frequently Asked Questions

These common questions and their short answers are taken from Tomer Sharon’s book Validating Product Ideas: Through Lean User Research . You can find longer answers to each in your copy of the book, either printed or digital version.

What is lean user research?
Lean user research is a discipline that provides insights into users, their perspectives, and their abilities to use products and then gives this information to the right people at the right time so that the research is invaluable for developing products. Lean user research focuses on answering three big questions about people: What do people need? (See Chapter 1.) What do people want? (See Chapter 5.) Can people use the thing? (See Chapter 6.)
How is lean user research different than “regular” user research?
Lean user research is mostly conducted by non-researchers who have burning questions about their audience (or potential audience). They want to answer these questions quickly, effectively, and on their own without hiring a professional. Lean user research is not perfect and can be at times quick and dirty, meaning some corners are cut. For example, since non-researchers might not have very good control of their body language, lean user research calls for more indirect approaches to learning. It values remote techniques over in-person ones (see Chapters 7 and 8).
Does this book include everything I need to know about user research?
No! This is a book for product developers and managers who are not skilled researchers. Therefore, research techniques are described in a relatively prescribed manner, skipping underlying factors, options, and dilemmas. The goal here is to help non-skilled product developers to do their own far-from-being-perfect-yet-effective research. If you want to learn more about research techniques described in this book, there are multiple excellent resources available. These are listed on the companion website at leanresearch.co.
I prefer to spend three free hours on writing more code rather than reading yet another book. If I only have time to read one chapter, which one should I read?
The short answer is Chapter 3 “How Do People Currently Solve a Problem?”
The long answer is read the introduction first and then the table of contents, and see if any of the chapters discusses a burning question you might have right now. If so, read that chapter first. I recommend reading Chapter 3 regardless, because observation, the research technique described there, is a fundamental tool about learning from users.
Should I read this book cover to cover?
No! This is a “doing,” not a “reading” book. The best way to digest the content of the book is to first scan it to identify a burning question (or questions) you (or your team) currently have. Then access the relevant chapter, read its premise, roll up your sleeves, and start going through the steps while completing the activities described in them. Reading about these activities won’t get you anywhere. As Ric Flair used to say, “To be the man, you have to beat the man.” Or if you are not a pro-wrestling fan, “If you want to shoot, shoot. Don’t talk!”
UX is about designing interfaces. Does this book include guidance on that?
Heck, no! UX is about so many things: user interface, interaction design, user research, information architecture, visual design, content strategy, and more. This book includes tons of advice about user research; however, it will not help you with guidance on how to design screens and wireframes. Some of the chapters will help you get insights about your screen design (for example, Chapter 6) or your product’s information architecture (Chapter 8), but there aren’t any design guidelines in here.

back to Validating Product Ideas

Sample Chapter: Surveys That Work

This is a sample chapter from Caroline Jarrett‘s book Surveys That Work: A Practical Guide for Designing and Running Better Surveys. 2021, Rosenfeld Media.

Chapter 1: Goals: Establish Your Goals for the Survey

In this chapter, you’re going to think about the reason why you’re doing the survey (Figure 1.1).

Figure 1.1
It’s easier to hit a target if you know which one you’re aiming for.

By the end of the chapter, you’ll have turned the list of possible questions into a smaller set of questions that you need answers to.

Write down all your questions

I’m going to talk about two sorts of questions for a moment:

Research questions
Questions that you put into the questionnaire

Research questions are the topics that you want to find out about. At this stage, they may be very precise (“What is the resident
population of the U.S. on 1st April in the years of the U.S. Decennial Census?”) or very vague (“What can we find out about people who purchase yogurt?”).

Questions that go into the questionnaire are different; they are the ones that you’ll write when you get to Chapter 3, “Questions.”

Now that I’ve said that—don’t worry about it. At this point, you ought to have neatly defined research questions, but my experience is that I usually have a mush of draft questions, topic titles, and ideas (good and bad).

Write down all the questions. Variety is good. Duplicates are OK.

Give your subconscious a chance

If you’re working on your own, or you have the primary responsibility for the survey in a team, then try to take a decent break between two sessions of writing down questions. A night’s sleep gives your subconscious a chance to work out what you really want to find out. If that isn’t practical, then maybe try a walk in the fresh air, a break to chat with a friend, or anything else that might provide a pause.

Get plenty of suggestions for questions

If you’re working with a team or you’re in an organization, then often when word gets out that there’s a survey ahead, colleagues will pile in with all sorts of suggestions for their questions. This can feel a little overwhelming at first, but it’s best to encourage everyone to contribute their potential questions as early as possible so that you can carefully evaluate all of them, focus on some goals for this specific survey, and have a good selection of other questions available for follow-up surveys and other research.

If I’m too restrictive at the very beginning, I find that everyone tries to sneak just one little extra essential question into the questionnaire a day—or even an hour—before the fieldwork starts. By then, it is too late to test the little extra questions properly, and they could sink my whole survey.

But while you’re still establishing the goals for the survey? Great! Collect as many questions as possible. Encourage everyone to join in—colleagues, stakeholders, managers, whoever you think might be interested. If you’re running a workshop, give the introverts some space by having a bit of silent writing where everyone captures their individual question ideas by writing them down.

Create a nice big spreadsheet of all the suggestions, a pile of sticky notes, or whatever idea-gathering tool works for you.

Ideally, make it clear that there’s a cutoff: suggestions before a particular date will get considered for this survey; miss the date, and they’ll be deferred until the next opportunity. This helps to encourage the idea of many Light Touch Surveys.

Challenge your question ideas

When you’ve gathered or created question ideas, it’s time to confront them with these four detailed challenges in Figure 1.2:

What do you want to know?
Why do you want to know?
What decision will you make based on the answers?
What number do you need to make the decision?

Figure 1.2
What decisions will you make based on the answers?

Ask: What do you want to know?

Surprisingly, I find that the question suggestions that I create or collect from colleagues often do not relate to what we want to know. Many times, I’ve challenged a question by saying, “OK, so you’re thinking about <xxx question>. What do you want to know?” and it turns out that there’s a gap between the question and the reason for asking it.

Probably the most common example is the question: “Are you satisfied?” The question is OK but very general.

Ask: Why do you want to know?

I’m usually working with someone else when I’m doing a survey. To help narrow down from ”every possible suggestion” to a sensible set of goals for the survey, I ask “Why do you want to know the answers to these questions?” and we then go on to challenge ourselves with the three questions in Figure 1.2.

If I’m on my own, then I find it helps to add “this time” or “right now”—to help me focus on the practical matter of getting my ideas down to something manageable. Come to think about it, that’s not a bad idea for a team, too—it helps all of them realize that they don’t have to ask everyone everything all at once.

Ask: What decision will you make based on the answers?

If you’re not going to make any decision, why are you doing the survey?

Look very hard at each of the suggested questions and think about whether or not the answers to them will help you make a decision.

Don’t worry at this stage about the wording of the questions or whether people will want to answer them. You’ll work on those topics in upcoming chapters.

But if the answers to a question won’t help you make a decision, set that question aside. Be bold! The question might be fascinating. You might be looking forward to reading the answers. But you’re trying to focus really hard on making the smallest possible useful survey. You don’t need to waste the question—it can go into the possible suggestions for next time.

At this point, you’ll have some candidate questions where you know what decisions you’ll make based on the answers.

Ask: What number do you need to make the decision?

In the opening chapter, “Definitions,” I emphasized that a survey is a quantitative method and the result is a number. Sometimes you’ll realize at this point that although you have candidate questions, you do not need numeric answers to them in order to make the decisions. That’s fine, but it also means that a survey is probably not the right method for you. Your work so far will not be wasted because you can use it to prepare for a more appropriate method.

Choose the Most Crucial Question (MCQ)

If you were only allowed answers to one of your candidate questions, which would it be?

That’s your Most Crucial Question (MCQ).

Most Crucial Question

is the one that makes a difference. It’s the one that will provide essential data for decision-making.

You’ll be able to state your question in these terms:

So that we can decide _______.

At this stage, don’t worry if it’s a Research Question (in your language, maybe even full of jargon) or the question that will go into the questionnaire (using words that are familiar to the people who will answer).

Test your goals: Attack your Most Crucial Question

Try attacking every word in your Most Crucial Question to find out what you really mean by it. Really hammer it.

Here’s an example: “Do you like our magazine?”

Who is “you”? Purchaser, subscriber, reader, recommender, vendor, or someone else?
What does “like” mean? Admire? Recommend? Plan to purchase? Actually purchase? Obsessively collect every edition? Give subscriptions as gifts?
What do you mean by “our”? Us as a brand? A department? A team? As a supplier to someone else?
What do you mean by “magazine”? Every aspect of it? The paper edition? The online one? The Facebook page? The article they read most recently? Some parts, but not others? Does it matter if they’ve read it or not?

I found a great attack on a question by Annie Pettit, survey methodologist. She starts with the question:

“When was the last time you bought milk?”

Here’s how Annie attacks “bought” and “milk”:

Do you mean only cow milk? What about milk from goats, sheep, buffalo, camel, reindeer? Or what about milk-substitutes from nuts or plants like soy, almond, rice, and coconut that are labeled as milk? Were you really trying to figure out if we put a liquid on cereal? (Pettit, 2016)

(And she added a whole lot more about topics, like whether or not chocolate milk counts.)

Decide on your defined group of people

When you’ve really attacked your MCQ, look back and think about your “defined group of people”—the ones who you want to answer. Add them to your statement like this: We need to ask (people who you want to answer). The question (MCQ goes here). So that we can decide (decision goes here).

If your defined group of people is still vague—“everyone” or something equally woolly—then try attacking again. A strong definition of the group you want to answer at this point will help tremendously when you get to the next chapter, “Sample.”

But before you proceed to Chapter 2, let’s pause for a moment and think about your plans.

Check that a survey is the right thing to do

Is your research question something that you must explore by asking people, or would it be better to observe them?

Do you want to know “why?”—qualitative—or “how many?”—quantitative?

Let’s look at this definition again:

survey

is a process of asking questions that are answered by a sample of a defined group of people to get numbers that you can use to make decisions.

I’m going to contrast that with this definition:

interview

is a conversation where an interviewer asks questions that are answered by one person to get answers that help to understand that person’s point of view, opinions, and motivations.

Both of them rely on asking: the interview is about “why”— qualitative—and the survey is about “how many”—quantitative, as in Figure 1.3.

Figure 1.3
Contrasting interviews as qualitative and surveys as quantitative.

Must your MCQ be answered by people?

One of my favorite questions was on a printer manufacturer’s survey:

“How many pages do you print in a month?”

I had no idea. I knew the answer was more than one and less than a full box of paper because I hadn’t bought a box of paper that month—but I didn’t feel sufficiently motivated to work out how many pages are in a full box. I guessed, wildly. Very poor data.

The real irony, though, was that my printer was connected to their customer feedback program and was giving them the exact figure all the time: their analytics should have told them.

Here’s another example that arrived in my inbox recently:

We need to ask visitors to our website whether pop-ups make them feel less like buying from us so that we can decide whether to remove pop-ups.

I’m sure that client must have some good business reasons for using pop-ups that make them hesitate about removing them, but asking people whether they “feel like buying” is a notoriously unreliable thing to do. They may feel like buying, but not actually buy, or feel unlike buying, but buy anyway. (We’ll return to this topic in Chapter 3 when we look at the “Curve of Prediction.”)

There’s a much better quantitative method for questions like this: A/B testing, where you publish two versions and use analytics to decide which one contributes better to the desired outcome. A/B tests and the many other different types of analytics silently observe what people do without bothering them with questions. These are contrasted with surveys in Figure 1.4.

Figure 1.4
Analytics and A/B tests are ways of observing how many people do something without asking them.

Do you want to find out “why”?

You may have spotted that we’re sneaking up on the four-way matrix in Figure 1.5. The quadrant we haven’t yet looked at is the top-left corner: observing to find out “why.”

It’s not always obvious why people are doing something. For example, if people tell you they can’t find things on your website, then search log analytics will tell you what they are searching for—but not why they are searching. Did they try searching straight away? Did they try a few clicks without success? Did they see your term for what they’re searching for but not recognize it because they had something different in mind?

Here’s another MCQ that I see quite often:

We need to ask visitors to our website the question: “What do you dislike about our site?” so that we can decide what to improve.

Figure 1.5
A matrix for choosing the right method.

Leaving aside the problem that “What do you dislike” doesn’t have a numeric answer, you’ve got the more fundamental problem that there isn’t a direct connection between “What do you dislike” and “What should we improve?” You need to know why people dislike something in order to get ideas about how to change it.

You might turn to interviews, but it’s unreasonable to expect most people to retain all the little details that made something easy or difficult. Observing them as they use the thing is much easier for them—and much richer data for you.

In a usability test, you can observe a participant who is tackling some tasks—often in a research facility. Or you can go out to observe people in their natural setting, a field study.

Consider “why” alongside “how many”

A four-way matrix always makes it look as if the ideas are separate, doesn’t it? Of course, in reality, the techniques complement each other.

The route in Figure 1.6 is one that I took around the matrix for a client recently.
Analytics showed that sales of one product had dropped.
Usability tests revealed that people thought the website was no longer maintained, so the product must also be out-of-date.
Interviews at the same time revealed that people often left a long gap between deciding to buy the product and actually using it.
A survey told us that the out-of-date problem was affecting more people than the wait-to-use problem.

Figure 1.6
One of many possible routes around the matrix.

I would love to encourage you to try some triangulation.

Triangulation

is when you use a mixture of research methods and compare the results to improve your overall insights.

A draft presentation can help you decide between “why” and “how many”

A couple of years ago, I was chatting about surveys with user experience consultant Natalie Webb. Her tip was:

“Create a draft of your presentation, based on the results you expect to get from your survey.”

It seemed a strange idea to me at first, but the more I’ve tried it, the more I like it as a way of testing whether I’ve really thought enough about what I want to ask and whether the number that I will get as a result of my survey really will help me to make a decision—the “so what” of surveys in Figure 1.7.

Figure 1.7
A draft presentation helps you to think about the “so what?” of your survey.

I worried that by drafting the presentation first, I’d be somehow constraining the direction of the research—preventing my team from thinking freely about what they were doing, closing down what they might learn.

Gradually, I realized that this is part of the power of surveys. Because you’re finding out “how many” of something, you need to understand the “why” before you start. If you don’t yet know enough about “why,” then you should be choosing to start with observation and interviews.

Think about what sort of number you need

Thinking about the “so what” and the number that you’ll need for the decision you’ll make also helps with another point to consider now: what sort of number do you need as your result? It may seem early, but statisticians will tell you that you must work out your statistical strategy before you collect the data, not afterward.

Do you need to know the actual number of people who answer a question in a particular way? For example, when I helped with a survey about planning an office move, I wanted to know how many people said that when the office moved to the new location, their commute would become excessively long.

Is it the proportion who answer one way rather than another? For example, I wanted to compare the proportion of people who claimed they would leave if the office moved to a new location to the proportion who said they would be likely to accept the change.

Are you looking for a mean (the arithmetical average)? For example, I might have considered whether increasing the mean commute by more than an hour would kill the idea.

Are you looking for a median (the value right in the middle when you place them all in order from largest to smallest)? Means can get easily distorted by one or two outlandishly large values. If one person’s commute suddenly became nearly impossible—10 hours or more—that would greatly increase the mean, but the median wouldn’t be affected very much.

And for design, I’m often looking at ranges and modes. The range is the difference between the largest and the smallest values, so with a 10-hour commute and another commute that’s zero because the person lived in an apartment above the possible new location, my range would be 10 hours. The mode is the most frequent value, and something that I find I have to consider very carefully for many design challenges—both to design for the people who answered with the most frequent value and to make sure that I’m not accidentally excluding people who don’t fit “the norm” for any reason.

Or something else? You may be doing a comparative survey so you’ll be considering what you want to compare from this survey to the next, or a modeling survey where you’ll do all sorts of advanced statistical manipulations, or something quite different.

Whatever you’re planning to do with the answers to your survey, some careful thought at this stage about those statistics will be well worth the time you put into it—and may send you back to have another review of your Most Crucial Question and how you plan to use it.

Determine the time you have and the help you need

So, you have a Most Crucial Question, you know the decision you’ll make, and you’ve thought a bit about the type of number you need to make that decision. It’s a good moment to think about timing and who needs to be involved.

First, think about the time available:

When do you need to have a result, and how much time can you put into it?
If you’re lucky enough to have team members to work with, how much time can they spare?
When will you deliver the report from the survey?

Next, think about the tools:

Do you or your organization already have a survey tool?
Do you know how to use it?
Will you need to buy or subscribe to one?

Finally, and perhaps most importantly, who else is involved?

Who needs to be involved in the survey but isn’t part of your team, such as the privacy or legal people?
Who will get the results from the survey?
Who is involved in making the decisions based on the results?

Interview first, survey later

A common mistake is to think that you’ll do a survey first and then do follow-up interviews with some of the people who answer.

The rule is: interview first, survey later. Two especially useful types of interviews are:

Interviews to find out what your defined group of people think about the topic of your survey (covered in Chapter 2)
Cognitive interviews—a special type of interview just for survey questions—to help you discover whether the questions are working (Chapter 3)

And, in fact, to get the best results from your survey, you’ll complement these interviews with two other techniques from the matrix, aa noted in Figure 1.8:

Usability tests of the questionnaire (Chapter 4, “Questionnaire”)
A pilot test between the usability test and the survey itself (Chapter 5, “Fieldwork”)

Figure 1.8
We’ll use techniques from other parts of the matrix on our way to the survey.

If you want a couple of ideas for how to fit all those activities into the time you have available, then skip ahead to Chapter 8, “The Least You Can Do^TM.” A recent survey where I worked hard to get a single Most Crucial Question took me four days—spread out over a month, admittedly, but only because I had a week’s vacation in the middle.

What could possibly go wrong with the goals?

For many years, I was quite a purist about surveys. If you’d asked me “What can go wrong when choosing a goal for your survey?” I’d have answered, “Insisting on doing a survey when it’s the wrong method for the research problem.”

These days, I’ve mellowed. I know that sometimes colleagues or clients will carry on with a survey for all sorts of reasons, good and bad, when it’s not the ideal thing to do. If that’s happening to you, don’t worry. Keep making good choices, aim for a Light Touch Survey, and iterate as much as possible. No matter what the outcome is, you’ll definitely learn a lot about how to do a better survey next time.

Strictly between you and me, I’ve also become more relaxed about some of the other aims of this chapter. Couldn’t get down to exactly one Most Crucial Question? If you still have dozens of MCQs: definitely not. But five or six candidates for MCQ? Not so bad—you can whittle them down when you start working on them in Chapter 3. Not entirely clear about the decision you’ll make? Have a go, and revisit it when you’ve done some more steps. You can iterate, after all.

But I wouldn’t often admit that to the team or the client because I know that when we can agree on one Most Crucial Question with a clear decision to be made, the rest of the survey process is going to be much easier and quicker. So I try pretty hard to persuade them to get there.

To be valid, the goals and questions must match

This brings me to the first of the challenges that you’ll meet through the steps of the survey process. In this chapter, you’ve been looking at the first tentacle of the Survey Octopus: “The reason you’re doing
it,” as shown in Figure 1.9.

Figure 1.9
Lack of validity.

There’s always an error between each tentacle and the next one. In this case, it’s “lack of validity.”

Lack of validity

happens when the questions you ask do not match the reason why you are doing the survey and what you want to ask about.

Or in other words:

A survey is valid when the questions you ask are a good match to the reason why you are doing the survey and what you want to ask about.

So work really hard on the reason why you are doing it, the decision that you’ll make, and that Most Crucial Question.

At this point, you will know

To have an easier ride with the next steps in the survey process, it helps a lot of at this point if you know:

The resources you have for the survey
Who you want to answer your question—your defined group of people
The decision you’ll make based on the results
The Most Crucial Question to help you make the decision
Whether the Most Crucial Question needs to be answered by people or not
Whether a survey is the right thing to do

Back to Surveys That Work

Enterprise UX 2018 Program

Featured Trainings

Membership FAQ

What’s the difference between free and paid memberships?

Paid memberships include access to all conference content, including session recordings, session and sketchnotes, and decks, 30 days after each conference ends. Paid memberships also include special discounts on workshops and books.

What about ebooks that I’ve already purchased?

You will have access to your purchased ebooks regardless of your membership status.

Can the discounts apply retroactively?

No; you can only apply discounts to future purchases after you become a member.

How do I keep up with all these offerings?

We’ll send you occasional emails when we launch new books, workshops, and conferences.

What will you do with my information?

We will never sell or lease your personal information. Learn more in our privacy policy.

What if I want to cancel my membership?

You can cancel your membership at any time; access will expire at the end of the original term of your membership. You can change your account settings here.

Will more content be added?

Yes! We produce four conferences a year, so every three months you will receive access to the new conference content. New community call recordings for each of our communities are added every month, and we continuously add new features. Please let us know what content and functionality you would like to see added.

Membership Beta Test Feedback and Instructions

Below is a step-by-step set of instructions for testing membership with questions to capture your feedback at every stage.

Page 1 will take you through testing Free Membership.
Page 2 will take you through testing a trial of our Gold Membership.
Page 3 will ask general questions about your overall experience.

Don’t worry if you don’t have time to finish in one sitting! As long as you’re logged in, you can save your answers by clicking “Save Draft” at the end of the page. When you’re ready to come back to it, make sure you’re logged in and come back to this page https://rosenfeldmedia.com/membership-beta-test-feedback-and-instructions/). Your answers will still be here, and you can continue working.

If you encounter any issues, please email [email protected]

Membership Beta Test