AR2021- UX Research Excellence Framework (Molly Stevens and George Zhang, Booking.com and CourseHero)
-
We’d like to extend our thanks to the extended research leadership team, which grew from 30 to 100, as they developed a lot of the frameworks discussed today
-
Understanding our impact as researchers
-
Making sure we are using the right methods
-
Moving up research earlier in the decision tree through partnering with cross-functional stakeholders
-
When you start managing, you need to figure out how to lead people, manage performance, and all other tasks
-
Growth of team and structure of work you do, and the work of team over course of years changes—and you typically don’t have a lot of tools to manage this
-
Start with something and figure out what works for you on the organizational level, as you seek to balance it all out
-
Design Studies
-
Deliverables (like which populations to study)
-
Organizational Changes (like selling an Uber division to China)
-
Global Impacts (like the pandemic, and everyone went into)
-
Times when they needed to scale research (outside of North America, such as studies in Asia or South America)
-
Getting buy-in from stakeholders on work important for business
-
Getting headcount and thinking where head-count goes, and setting up their team for success
-
Showcasing impact, and what it means to their team
—> But how did we get here? At Uber we had big changes that required a lot of shifts
-
From there we would commit to changes, and then execute and adjust
—> So regarding impact, how do we think about it and talk about it with stakeholders?
-
Hard to have reference point for three distinct requests
-
Not ABC request, rather A or 3 or # requests
-
There are many ways to score a project like customer impact, technical feasibility, user group impacted
-
Also considered collaboration quality, and business performance
-
Ranged from scale of 1, with little customer impact, to 5, which was a fundamental change for user models
-
This prevented the team from not just doing one-time work, but also more frequent low-level tasks like satisfaction surveys, and benchmarking repetitive actions
—> So think of things that could be happening for your team, and pick the box with methods that will take you towards that direction
—> In an example project where we used prioritization, we wanted to better understand global perception of privacy when riding with Uber
—> So we went to three major Uber markets (Jakarta, Berlin, Sao Paulo), and the questions used for each study would vary according to each market
-
The study led to short-term experiments and long-term roadmaps
-
And it provided an ongoing measure of how privacy needed to be measured
—> Next, we need to drive for rigorous methods and insights, and focus on results that are reliable/valid as well as simple/scalable
-
So we strive for results that are valid and reliable that other UX researchers can use
-
Each tailored to get insight and data in unique way
-
Being simple is very complex, and difficult to do well
-
Being complex is easy to do
-
Start with one method, and only add other ones when needed
-
At this moment, there would be a daily-check in of how satisfied you were with your experience as a driver
-
The survey was more reliable than NPS
-
Response rate is high (preventing concerns about non-response)
-
Survey supported Uber’s 180-days of change campaign in 2017 to mend relationship with its drivers, and is still running
-
We need to move from a messy problem to a clear solution
-
Five stages for product development
-
Who facilitated each step of development
-
All functions associated with each step
-
A researcher brought the team executives into Paris to do a field test, before any action was taken
-
Executives realized that test that the hurdles for in-house development were too high
-
Research led to Uber saving millions of dollars
-
The framework will be based around priorities that are important for business.
—> Next, you are free to use this framework for yourself!
-
Do you consider ethical excellence to be part of the impact of a user researcher in a product org?
-
Can be part of ResearchOps
- How would this framework evolve/change, when working in a net new space (where no prior research or behavioral knowledge exists)?
-
What is psychological emotional social well-being
-
Once you go through impact landscape, consider methodology
-
Need good metrics
-
Journey is difficult but rewarding, as we did this a lot with Uber
-
Re: impact: In what scenarios would you prioritize work that has high business importance vs customer impact?
-
Goal is to create closest apple to apple comparison as possible
-
Will have info in business already, and whether people are excited about the project
-
Most valuable part was having conversations.
-
Will prioritize certain surveys
- Goal to surface visibility of choices, which are often invisible
-
Overall with early cancellation feature, but if cancellations happen early, then system can give choice for both sides, and having more control
-
Could provide more choice and involvement