AR2021- UX Research Excellence Framework (Molly Stevens and George Zhang, Booking.com and CourseHero)

—> We’ve worked together for nearly 14 years, sometimes at different companies, but have always collaborated together
—> We are going to talk through a research excellence framework developed by us through our work together at Uber
  • We’d like to extend our thanks to the extended research leadership team, which grew from 30 to 100, as they developed a lot of the frameworks discussed today

 

—> Our framework involves:
  • Understanding our impact as researchers
  • Making sure we are using the right methods
  • Moving up research earlier in the decision tree through partnering with cross-functional stakeholders

 

—> Our audience for the framework is managers and research leads with growing teams
  • When you start managing, you need to figure out how to lead people, manage performance, and all other tasks
  • Growth of team and structure of work you do, and the work of team over course of years changes—and you typically don’t have a lot of tools to manage this
—> The framework is meant to resolve this

 

—> Framework is meant as a starter structure:
  • Start with something and figure out what works for you on the organizational level, as you seek to balance it all out
—> In some ways, skills learned at Uber, helped Molly make it through the pandemic with a modicum of balance

 

—> So how did we evolve to our framework over time?
—> The work environment at Uber was very fast paced and chaotic,  with many highs, lows, and extremes (such as company footprint, speed, data gathered etc.)
—> Serious changes were happening most of the time. To develop our research practice, we needed to put in place structures

 

—> So let’s reflect on the major changes you’ve had to manage in workplaces (especially in 2020). These include:
  • Design Studies
  • Deliverables (like which populations to study)
  • Organizational Changes (like selling an Uber division to China)
  • Global Impacts (like the pandemic, and everyone went into)
—> Frameworks allowed us to make right decisions, for us, our teams, and our researchers

 

—> We asked folks to give answers about how they’ve had to adjust due to sudden circumstances before the talk, and their stories were about
  • Times when they needed to scale research (outside of North America, such as studies in Asia or South America)
  • Getting buy-in from stakeholders on work important for business
  • Getting headcount and thinking where head-count goes, and setting up their team for success
  • Showcasing impact, and what it means to their team
—> So lets loo at how we can put up a structure around certain elements, and help manage the issues raised, as well as others

 

—> But how did we get here? At Uber we had big changes that required a lot of shifts

 

 

—> We had a framework of active process of understanding the problem (like being asked to do multiple conflicting things), and then executing and adjusting
—> These things didn’t move cleanly, so we would try to define the problem and come up with multiple solutions, and try them out.
  • From there we would commit to changes, and then execute and adjust

 

—> So regarding impact, how do we think about it and talk about it with stakeholders?

 

 

—> For strategy you need to know what to prioritize: what work to do, when, and how to choose that work
—> We would get research requests come in and were not sure how to prioritize, and needed a decision on how to move forward
  • Hard to have reference point for three distinct requests

 

 

—> Topics are even more mixed-bag in terms of priorities that are requested
  • Not ABC request, rather A or 3 or # requests
—> We wanted to think of better way to approach topics, and be more strategic

 

—> So we focused on scoring the project
  • There are many ways to score a project like customer impact, technical feasibility, user group impacted
—> You need to think about what’s important to your team and what’s important to teams you work with, to figure out which projects to pursue
  • Also considered collaboration quality, and business performance

 

 

—> We took guesses of where issue would fall within a rubric
  • Ranged from scale of 1, with little customer impact, to 5, which was a fundamental change for user models
—> Rubric key was key in helping Uber adjust to changes on the fly

 

—> Our impact matrix helped us think of work that should be done, over the course of the year
  • This prevented the team from not just doing one-time work, but also more frequent low-level tasks like satisfaction surveys, and benchmarking repetitive actions
—> The matrices helped explain to project stakeholders where user researchers were looking for impact with the projects they were doing

 

—> So think of things that could be happening for your team, and pick the box with methods that will take you towards that direction

 

—> In an example project where we used prioritization, we wanted to better understand global perception of privacy when riding with Uber

 

—> So we went to three major Uber markets (Jakarta, Berlin, Sao Paulo), and the questions used for each study would vary according to each market

 

 

—> The impacts across the business of this study were as follows:
  • The study led to short-term experiments and long-term roadmaps
  • And it provided an ongoing measure of how privacy needed to be measured

 

—> We realized safety and privacy were also part of team and that the two worked together

 

—> Next, we need to drive for rigorous methods and insights, and focus on results that are reliable/valid as well as simple/scalable

 

 

—> We believe people have seen this chart about validity and reliability.
—> Research is more a science than art, and it shares same the attribute of reliability and validity
  • So we strive for results that are valid and reliable that other UX researchers can use

 

 

 

—> Above is a chart showing the toolkit of measures that can be used to collect evidence and data
  • Each tailored to get insight and data in unique way
—> The problem is that there are too many choices/methods to choose from
  • Being simple is very complex, and difficult to do well
  • Being complex is easy to do
—> For research excellence, follow principle of parsimony, and keep the methods simple
  • Start with one method, and only add other ones when needed

 

 

—> An example of an easy method is an intercept survey sent to Uber drivers after they concluded their riding for the day
  • At this moment, there would be a daily-check in of how satisfied you were with your experience as a driver
—> The survey was very simple, mobile friendly and easy to use, and could work across languages very easily
  • The survey was more reliable than NPS
  • Response rate is high (preventing concerns about non-response)
—> The survey was impactful, connecting with back-end, people were satisfied, and  this generated long-term values over time
  • Survey supported Uber’s 180-days of change campaign in 2017 to mend relationship with its drivers, and is still running

 

—> Finally, work closely with cross functions, and jump into product development quickly
—> What does partnership mean?
  • We need to move from a messy problem to a clear solution

 

—> In 2017, we created cross-functional partnership journey. It included:
  • Five stages for product development
  • Who facilitated each step of development
  • All functions associated with each step
—> Research is marked in purple, and set-up at the very beginning, and research moved upstream to solve problems quickly

 

—> Example of this is when Uber tried to launch an electric moped for people to use
—> There were two options for this: Build everything from scratch or partner with existing operators in cities Paris
  • A researcher brought the team executives into Paris to do a field test, before any action was taken
  • Executives realized that  test that the hurdles for in-house development were too high
  • Research led to Uber saving millions of dollars

 

 

—> We kept the framework, and let it evolve and improve
—> Created research excellence award, and as of 2018  we had a happy performing team and global team

 

—> We believe we can all evolve the base framework, and have used prioritization framework at places like Booking.com and Course Hero
  • The framework will be based around priorities that are important for business.

 

 

—> Next, you are free to use this framework for yourself!

 

 

Q&A
  1. Do you consider ethical excellence to be part of the impact of a user researcher in a product org?
A: We think ethics is an important part of any using research team and whether it can be founding part of how you evaluate people’s work, and choose projects
  • Can be part of ResearchOps
Building out ethics training, and as a leader, being able to back people up when they face ethical concerns
  1. How would this framework evolve/change, when working in a net new space (where no prior research or behavioral knowledge exists)?
A: If you are the first UX researcher in a big org, you have a clean slate, and can build from there.
Thoughts go to impact, and what needs to change there.
  • What is psychological emotional social well-being
  • Once you go through impact landscape,  consider methodology
Many research angles are subjective and hard to see
  • Need good metrics
Need to work with data science, and customer support, to understand what people are doing in the work
Collaboration goes back to impact, and need hearing ears in company, and need to deliver same insight until changes are happening.
  • Journey is difficult but rewarding, as we did this a lot with Uber
  1. Re: impact: In what scenarios would you prioritize work that has high business importance vs customer impact?
A: Will talk about scoring of customer, business, and score for team
  • Goal is to create closest apple to apple comparison as possible
  • Will have info in business already, and whether people are excited about the project
  • Most valuable part was having conversations.
  • Will prioritize certain surveys
  • Goal to surface visibility of choices, which are often invisible
At first many user/business interests appear to be conflict, but upon further study they tend to work well
  • Overall with early cancellation feature, but if cancellations happen early, then system can give choice for both sides, and having more control
  • Could provide more choice and involvement