AR2021-But Do Your Insights Scale? (Katy Mogal, Google Assistant)

—> My mission is to help product teams become better by leading research teams with a strategic point of view and utilizing survey and analytics data

 

 

 

—> My question for today’s talk is how are we as researchers shaping the org we work in?
  • We wanted to change qualitative insights on product strategy, and ended up changing how the org viewed the qualitative work our group did
—> Here are the steps taken along the way

 

—> I began my work with Google in 2019. Team had not been doing much in-person group and ethnography
  • We got an email from product and wanted to investigate how to improve product market fit

 

—> Product team came across an article, that focused on creating surveys that asked users how they would feel if they could no longer use a product
  • Those who were most disappointed would unlock the product market fit
—> So the project focus would be on understanding the needs of those who found value in the Google Assistant and engaged with the product.

 

 

—> UX research was glad to be given seat at strategy table, with teams that informed key business decisions such as Marketing, Data Science, and BizOps Strategy

 

—> Each team brought in team members to help,  but only the three people in UX research had the exposure to deep qualitative insights for product decisions
  • Partial exception applied to marketing

—> Project lead proposed approach of product market fit survey, with UX research  focused on refining the survey
  • BizOPs meanwhile would conduct an existing analysis

 

—> UX research said this was great, but we also needed deep investigate research to figure out the “why”, as other metrics focus on “what”

 

—> We floated a deep dive to understand “why” through the course of diary studies, and one-on-one interviews

—> The team said this would be great, then asked “Can you talk to 100 people?”
  • We couldn’t do this. The project was on a ridiculously tight timeline, with only a six week lead time, and we didn’t have the time to interview that many people

 

—> Our first response was, how could we get seat at table with this seemingly impossible ask?
—> But, as we try to get research further upstream in company decision making, we aren’t always given a seat at the table.
  • So we need to create our own opportunity and look for a  folding chair

 

—> How to do this? A calculated risk, taking advantage of Google’s culture of experimentation
  • This kind of evaluation is based on product decision and strategy
—> We decided to spin up a qualitative study on the side, without alignment up-ahead. The goal was to try to do the work, and let insights speak for themselves
  • Insights could help make the case for the work
—> This decision was a bit like rock climbing, where you need to leap off wall to get to the next hand-hold, and it is scary, even if in harness and held by a partner
  • We decided to take the  leap to show what qualitative insights could do

 

—> The first step in changing hearts and minds involves calculated risks
  • Megan Kierstead said the higher you go, the more qualities like bravery matter compared to methodology

 

—> So we had a digital diary study with 50 people, along with a longitudinal analysis of the data
  • We also created a platform for collaborators to get exposure to the research data

 

—> We invited stakeholders on project to our digital diary platform to see how people uploaded diary entries,
  • Goal of seeing entries and provide material for the team to look at
—> One team member got curious and checked out our data
  • Found a curious partner, and an analyst had rabbit-holed into video diaries, who acted as  our research associate
—> More importantly, his enthusiasm was contagious for his colleagues, and they started to go in and check it out
  • We got stakeholder buy-in as a result
—> After diary studies, we then had one-on-one interviews, with our colleagues joining us
  • Colleagues joined, and everyone had observed a few interviews

 

—> Our second takeaway?  Look for curios partners, and collaborators who will fall into work and evangelize the work for you
  • Once they recognized value digital diary delivered, they told their peers about it
—> At Google’s UX leadership summit, someone asked how UX could be recognized as peer to disciplines like engineering
  • The recommendation was to find engineering leaders who were converted to the benefits of UX, and let them tell case studies to their peers
—> For us, we found an analyst eager to help and let them do advocacy for us

 

—> Now we were up and running, and stakeholders were using our observations
—> There was a ‘war room’ for project, but we realized we could turn it into a design studio, and providing compelling data charts
  • We printed out qualitative quotes and worked with them
  • We saw collaborators take ownership of UXR data and saw how it related to their data

 

 

—> Another inflection point was when a project leader asked how to put the information we had gathered together
  • We proposed an affinity diagram map for all the data, where people presented post-its to group, an clustered groups with meaning
—> Affinity diagrams helped us engage with team, and it was a success
  • We got data out of laptops and onto walls
  • We treated our diagram work as a small experiment that was fit within the existing war room structure
    • It made participation easy and hard to object to
—> Since we had seeded ideas, through engaging early and letting people see data themselves, insights from UX research were percolating in people’s minds
  • We came up with six draft insights, and from there had a path to conclusions

 

—> Lesson 3: Throughout the process, do little experiments
  • It’s an active process of trying solutions, and feeding learnings into new experience
  • It helps when making change at org (introduce it over time, see what sticks, repeat)

 

—> We then put together report for the leadership off-site and our six insights became the structure of the report’s narrative.
  • Four of these insights were integrated into the 2020 product strategy

 

—> The senior VP of product sent his report out to his direct reports, saying he had a great example of how qualitative insights matter

 

—> The biggest confirmation of success was that the team got an email from the VP of Product, who asked them to help with planning process, and focus on more qualitative insights

 

—> BizOps also asked the team to repeat the clustering exercise

 

—> Now insights were key ingredient in UX strategy, we now had a seat at the strategic table

 

—> A final lesson: We should leverage our researcher superpowers, and how they can be used beyond interviewing people
  • We need to know how people are motivated and their insights and fears
  • We can use our power of empathy to drive organizational culture, not just product strategy
—> So understand the people around you, their barriers, like any research project
  • By understanding stakeholders and what they need, you change their minds about the value of UX

 

—> So how will this play out in your own lives?
  • Opportunity like ours don’t come every day
  • But we can use each of these steps in our organization, in smaller scale
—> We can find where we can take risks, what partners can we find, what low stakes things we can try out, and what research powers we can deploy to understand insights from partners

 

—> Thank you! Any questions?

 

Q&A
  1. Which tool did you use for the diary study?
A: The tool used was dscout
  1. Diary studies are amazing, though sometimes the challenge is in the tool, setup and digital collaboration. As well as the time and effort taken to extract it and put it on a wall – how did you manage that? What sort of effort did that take?
A: We had to de-prioritize work streams to free people up to do that
—> Diary study fit the needs of the projects and was accessible way to engage people with project, even when we didn’t have higher level alignment
—> So think of ways to expose people to qualitative work, in a way that will excite them
  1. How much time did it take from research to report, how many participants did you do the research with? And how many people were involved – was it mainly the 3 researchers?You also mentioned “we” in the talk. How many researchers were on the effort and what were their backgrounds/roles?
A: No team to do this, but asked people to allocate time from main jobs to help out—as side hustle
—> Could feel opportunity to show what qualitative insights could do was important
—> Had quanatiative researcher who acted as bridge between them and data scientists
—> Talked to 20 people, interviewed 12, and edited video compilations
—> Most important you need to show what qualitative insight will bring, and that others types of insight can’t
  1. We have a similar story at Microsoft, and we definitely have CVP buy in but the mid level execs and managers are still lagging behind with championing research (I’ll believe it when they show us the dough :money_mouth_face:). Culture change is incredibly hard when they’re entrenched in 3 decades of deprioritizing UX. Interested in your thoughts about how to scale change at every level?
A: It’s very context dependent
—> Use your research superpowers, find one opportunity, and take small steps
  1. How, if at all, has this changed folks from asking for 100+ data points?
A: We showed what output looks like, tied interviews and videos to bring meaning to work
—> It was about building trust that we know how to build samples that are big enough
  • A quant user researcher explained why data could be believed
  1. If you had to describe to an executive why qualitative research is helpful in scaling insights, what would you say?
A:  Surveys/Logs give us what of data, and we can be confident in rigor of data
—> But we are still relying our own assumptions as to what is driving behaviors
  • There is risk of developing products that don’t meet people’s needs, since we don’t know what these needs are.
  1. I love that your stakeholder became so passionate about your diary study! At the same time if one of my stakeholders would go in to review raw data and do their own analysis, I would fear that the data might get misrepresented or cherry picked – was that a problem at all in your case?
—> Yes, we need to carefully manage people while validating their passion for the work. This is hard to do .