AR2021-Measuring Up: Using Product Research for Organizational Impact (Mac Smith, Google Search and Assistant)

 

—>  I’ve been excited to give this talk for a while, as last few years has seen role of research shift
  • Previously focused on how/executing decisions that were already made
—> We are now using data to address product strategy, to how org should align based on information we have on customers
  • We have greater leadership roles within the org
  • Leadership growth is organic and we need to figure out the best way for us to take leadership roles
—> You can change orgs top level goals with the data you collect today
  • But you need to change how you work, who you partner with, what you do with the data you collect
—> I will tell how I moved from product features to shape organizational goals

 

 

—> I partnered with number of businesses as they grew into large orgs
—> Goals will reflect what he learned from two year journey
—> Looking back, three stages to move from focusing on features to shaping org goals
  • Prepare
  • Align
  • Deliver
—> Prep and alignment took the most time, but delivery is the shortest
  • If other two done well, delivery is easy to land
—> Three points occur in stages, and the moment for aligning and delivering, can shift focus and impact for your team

 

 

—> In the prepare stage you assess the situation and know what is going on
  • We ask: How can we leverage our research skills for influence? What’s the nature of org challenge we are not influencing today?
—> We need to know the opportunities and gaps, so we could set ourselves up effectively

 

—> When I moved into search, I turned my research lens on my org to see how the org viewed itself, and how did research team view itself within the org
—> Our research team had deep view of customers, but shallow view of stakeholders
—> So we approached our stakeholders with a series of structured interviews about the same questions.
  • Where are we?
  • How did we get here?
  • What do we do now?
—> Then we asked about how they used customer information, what they wanted to know, and how we could help
—> In talking with the stakeholders, a few things became clear
  • Search was large, heavily networked and success in the organization was through personal relationships
  • Stakeholders were deep domain experts and knew products, and the context of product use
  • The were data-friendly. If you showed right data and right people, you could make change within the company
  • Leadership was not sure how to engage with the research team
    • How could they engage research when not focusing on usability/experimentation problems?

 

—> We also needed to apply a level of rigor to research team. So we had the same structured interview techniques used on our team members
—> We evaluated our researchers on:
  • How was their craft?
  • How was their business acumen?
  • What were their skills and ability to advise?
—> We had great researchers with serious domain expertise, but we had weak business acumen and advising skills

 

—> These issues led to symptoms of disengaged staff from the org, who saw themselves as having lack of power in the org, and didn’t feel they were heard
  • We had the great skill of constant customer contact, but didn’t know who to communicate our knowledge to
—> There was also the challenge of stakeholders trusting the data presented by researchers, where small sample sizes could be generalizable
  • Three things needed to be addressed

 

 

—> So what to do?
  • The assessment became the basis for how the research team would work and to address org goals
—> For disengagement, we found relevant goals to energize the team
—> We had constant customer contact, so we shifted the team’s mindset to leverage strength they already had
—> To gain trust in research data we used partnerships we had throughout the organization

 

 

—> We set an audacious goal to a VP, to show two year plan for possibility of hitting goal
—> Presented goal by end of two year period, to have research address leadership on top issues for annual strategic planning
  • The VP was skeptical, but willing to go ahead
—> We had basic skeleton of where to go, and needed to raise the bar in terms of impact
  • The two-year timeline put our reputation on the line as a leader, but accountability is the cost of getting into the decision making room
  • Our intention made the goal real

 

 

—> We then had a mindset shift that customer contact was not ’strategic’
  • Moving from false dichotomy of tactical studies and strategic studies.
  • Any customer contact is touching on strategic issue
    • Business and customer rely on exchange of value. The better we can understand this exchange, the better strategic decisions can be made by the org
—> When you deal with business/customer you are touching underpinning of any company strategy
  • So any study can be strategic
—> So understand how your questions map up to the organization’s goals
  • Note: Solutions are hypothesis tied to orgs larger goals, and you can work backwards to figure out goals the organization is solving for
—> In talking with customers, you get key information about them
—> So we identified tactical work that led to strategic implications
—> A good example was of our team doing usability studies for purchasing tasks on search
  • Goal to develop features that were more useful, but researchers came back and were frustrated  how customer behavior was not lining up with business expectations
  • The business and the customer were not solving the same problems

 

—> We also took time to actually invest in skills the team needed, like communication and looking at business acumen and advising skills
—> We didn’t have much time to do this. So what did we do?
  • We realized the goal wasn’t to prepare team more, but rather shift stages from preparation to alignment

 

—> We built up energy for long-term goal, and worked on how to communicate our value
—> Now we needed to align with others and other insight functions

 

—> Needed to align with other groups on projects, as landing research successfully is balancing act (with right person, time, need)
  • Can be done at feature level, but don’t have bandwidth to account perspectives

 

—> Find a champion before we identify problems will solve
  • The champion is the person with the highest position to outline business implications, and organizational alignment needed
  • Person should believe customer information has benefit for them
    • In our case, we had a VP who led a Customer Insight team
  • We asked the VP to provide access to see where team could focus
    • In return offered to trade team to answer questions VP had on his team
—> A great champion can help reduce the effort of figuring out your org alignment

 

 

—> Knew annual decsions came at certain times of the year, so questions needed to be set-up 6-12 months ahead of time
—> Our champion helped us get into the room with a Product VP, and we asked him how UX research could answer his questions
  • We got peppered with questions, as VP had questions about customers, with the data unavailable from direct reports
—> VP mentioned whether we support customer journeys as well as we could
  • We flashed back to customer intent not matching up with business intent in usability studies
—> We took 3 questions from VP, to make sure he could have set of compelling answers

 

 

—> Theres’s an organizational problem with presenting insight data
  • There are multiple disciplines with same goal of trying to influence decision making
  • We are not accountable for final decisions, but each have strength and clear trade-off in our approach
—> UX why something happens in humanistic terms
—> Data Science has high view, but can’t only answer what/how questions
—> Market Research, good at defining markets, and can partner on why, but has distance from product teams
—> We went to the different insight groups for alignment and asked them to work together with us so that executives would have the best possible insights
  • 10-15 person working group where they combined methods across areas

 

—> After 18 months of prep, delivery was relatively easy
  • Doing hard work and setting up relationships was our main goal
—> So don’t start delivery and do three things at once
  • The hard work happens before you even do the research

 

 

—> We had up-level descriptive research work, which showed disconnect search delivering fast/quick results and complexity of people demonstrating slow/long-term deliberation in their search results
—> The search process for a long-term decision is a complex journey
  • Doubles-back on itself, and saw multiple break points
  • Product team agreed there was a gap, but wasn’t sure whether gap was important enough to be raised above narrow product area

 

 

—> So we worked with data science and market research to show that noisy process flows were generalizable, and matched with many types of tasks and journey within a product
—> Product was at scale and size for the executives to think how org should respond to this issue, rather than an individual team

 

—> We used language of business to make our findings relevant, i.e. how many people are using search?
  • Pointed out queries were complex, and accounted for key percentage of business revenues
—> We took the data shown in our study and contextualized it on our business bottom line

 

 

—> By setting up argument of: If you address UX challenge we believe you can increase metrics relevant to bottom line by certain percentage
  • This shifted conversation from do this because it’s right, to pointing out that business will benefit in an explicit way

 

 

—> This led to real organizational changes
  • Annual goals changed to reflect wording and topics used in our research
  • Researchers were invited to annual planning meetings, and were seen as experts on the work
  • We were often invited to  executive planning process

 

 

—>  In conclusion, the straightforward process of prepare, align, deliver, can help upgrade your research to focusing on organizational goals

 

—> Thank you for listening! I look forward to questions.

 

Q&A
  1. Do you support individual researchers “turning their lens” on a regular basis? If so, how? Feels easiest to do when one is new?
A;  Support researchers in doing the work. Take on things to shift organizational priorities over a long-period fo time
Actually easier to do when you have established yourself in right relationships. Alignment is easy if you have relationships with people you need to influence
  1. Did what other research teams at Google were doing help, hurt, or feel irrelevant? Why/In what way?
A: Would say in general, other research teams are often reflections of research leaders leading groups
  • If we are aligned it’s very helpful
Talk about idea of prototyping results, by sketching them out
  • Helped with teams who had similar project backgrounds, and could go to other teams in areas
  1. What about other UXR teams?
A: Helps if you have team with specific examples of what you want to do, and evidence to convince others, as well as expertise that can be suggested
  1. Who are these Data Science people in an org? What titles do they have? I don’t know if my company has any?
A: Data science, in that people have data science already
  • Researchers who analyze large scale behavioral data of people using product you worked on
Tends to be predominantly quantitative, and focus inwards
  • Functional title and may have people who do work outside of data science
Engineering team may be doing it themselves at small companies
  • Look for titles involving analytics