AR2021-Measuring Up: Using Product Research for Organizational Impact (Mac Smith, Google Search and Assistant)
—> I’ve been excited to give this talk for a while, as last few years has seen role of research shift
-
Previously focused on how/executing decisions that were already made
—> We are now using data to address product strategy, to how org should align based on information we have on customers
-
We have greater leadership roles within the org
-
Leadership growth is organic and we need to figure out the best way for us to take leadership roles
—> You can change orgs top level goals with the data you collect today
-
But you need to change how you work, who you partner with, what you do with the data you collect
—> I will tell how I moved from product features to shape organizational goals
—> I partnered with number of businesses as they grew into large orgs
—> Goals will reflect what he learned from two year journey
—> Looking back, three stages to move from focusing on features to shaping org goals
-
Prepare
-
Align
-
Deliver
—> Prep and alignment took the most time, but delivery is the shortest
-
If other two done well, delivery is easy to land
—> Three points occur in stages, and the moment for aligning and delivering, can shift focus and impact for your team
—> In the prepare stage you assess the situation and know what is going on
-
We ask: How can we leverage our research skills for influence? What’s the nature of org challenge we are not influencing today?
—> We need to know the opportunities and gaps, so we could set ourselves up effectively
—> When I moved into search, I turned my research lens on my org to see how the org viewed itself, and how did research team view itself within the org
—> Our research team had deep view of customers, but shallow view of stakeholders
—> So we approached our stakeholders with a series of structured interviews about the same questions.
-
Where are we?
-
How did we get here?
-
What do we do now?
—> Then we asked about how they used customer information, what they wanted to know, and how we could help
—> In talking with the stakeholders, a few things became clear
-
Search was large, heavily networked and success in the organization was through personal relationships
-
Stakeholders were deep domain experts and knew products, and the context of product use
-
The were data-friendly. If you showed right data and right people, you could make change within the company
-
Leadership was not sure how to engage with the research team
-
How could they engage research when not focusing on usability/experimentation problems?
-
—> We also needed to apply a level of rigor to research team. So we had the same structured interview techniques used on our team members
—> We evaluated our researchers on:
-
How was their craft?
-
How was their business acumen?
-
What were their skills and ability to advise?
—> We had great researchers with serious domain expertise, but we had weak business acumen and advising skills
—> These issues led to symptoms of disengaged staff from the org, who saw themselves as having lack of power in the org, and didn’t feel they were heard
-
We had the great skill of constant customer contact, but didn’t know who to communicate our knowledge to
—> There was also the challenge of stakeholders trusting the data presented by researchers, where small sample sizes could be generalizable
-
Three things needed to be addressed
—> So what to do?
-
The assessment became the basis for how the research team would work and to address org goals
—> For disengagement, we found relevant goals to energize the team
—> We had constant customer contact, so we shifted the team’s mindset to leverage strength they already had
—> To gain trust in research data we used partnerships we had throughout the organization
—> We set an audacious goal to a VP, to show two year plan for possibility of hitting goal
—> Presented goal by end of two year period, to have research address leadership on top issues for annual strategic planning
-
The VP was skeptical, but willing to go ahead
—> We had basic skeleton of where to go, and needed to raise the bar in terms of impact
-
The two-year timeline put our reputation on the line as a leader, but accountability is the cost of getting into the decision making room
-
Our intention made the goal real
—> We then had a mindset shift that customer contact was not ’strategic’
-
Moving from false dichotomy of tactical studies and strategic studies.
-
Any customer contact is touching on strategic issue
-
-
Business and customer rely on exchange of value. The better we can understand this exchange, the better strategic decisions can be made by the org
-
—> When you deal with business/customer you are touching underpinning of any company strategy
-
So any study can be strategic
—> So understand how your questions map up to the organization’s goals
-
Note: Solutions are hypothesis tied to orgs larger goals, and you can work backwards to figure out goals the organization is solving for
—> In talking with customers, you get key information about them
—> So we identified tactical work that led to strategic implications
—> A good example was of our team doing usability studies for purchasing tasks on search
-
Goal to develop features that were more useful, but researchers came back and were frustrated how customer behavior was not lining up with business expectations
-
The business and the customer were not solving the same problems
—> We also took time to actually invest in skills the team needed, like communication and looking at business acumen and advising skills
—> We didn’t have much time to do this. So what did we do?
-
We realized the goal wasn’t to prepare team more, but rather shift stages from preparation to alignment
—> We built up energy for long-term goal, and worked on how to communicate our value
—> Now we needed to align with others and other insight functions
—> Needed to align with other groups on projects, as landing research successfully is balancing act (with right person, time, need)
-
Can be done at feature level, but don’t have bandwidth to account perspectives
—> Find a champion before we identify problems will solve
-
The champion is the person with the highest position to outline business implications, and organizational alignment needed
-
Person should believe customer information has benefit for them
-
In our case, we had a VP who led a Customer Insight team
-
-
We asked the VP to provide access to see where team could focus
-
In return offered to trade team to answer questions VP had on his team
-
—> A great champion can help reduce the effort of figuring out your org alignment
—> Knew annual decsions came at certain times of the year, so questions needed to be set-up 6-12 months ahead of time
—> Our champion helped us get into the room with a Product VP, and we asked him how UX research could answer his questions
-
We got peppered with questions, as VP had questions about customers, with the data unavailable from direct reports
—> VP mentioned whether we support customer journeys as well as we could
-
We flashed back to customer intent not matching up with business intent in usability studies
—> We took 3 questions from VP, to make sure he could have set of compelling answers
—> Theres’s an organizational problem with presenting insight data
-
There are multiple disciplines with same goal of trying to influence decision making
-
We are not accountable for final decisions, but each have strength and clear trade-off in our approach
—> UX why something happens in humanistic terms
—> Data Science has high view, but can’t only answer what/how questions
—> Market Research, good at defining markets, and can partner on why, but has distance from product teams
—> We went to the different insight groups for alignment and asked them to work together with us so that executives would have the best possible insights
-
10-15 person working group where they combined methods across areas
—> After 18 months of prep, delivery was relatively easy
-
Doing hard work and setting up relationships was our main goal
—> So don’t start delivery and do three things at once
-
The hard work happens before you even do the research
—> We had up-level descriptive research work, which showed disconnect search delivering fast/quick results and complexity of people demonstrating slow/long-term deliberation in their search results
—> The search process for a long-term decision is a complex journey
-
Doubles-back on itself, and saw multiple break points
-
Product team agreed there was a gap, but wasn’t sure whether gap was important enough to be raised above narrow product area
—> So we worked with data science and market research to show that noisy process flows were generalizable, and matched with many types of tasks and journey within a product
—> Product was at scale and size for the executives to think how org should respond to this issue, rather than an individual team
—> We used language of business to make our findings relevant, i.e. how many people are using search?
-
Pointed out queries were complex, and accounted for key percentage of business revenues
—> We took the data shown in our study and contextualized it on our business bottom line
—> By setting up argument of: If you address UX challenge we believe you can increase metrics relevant to bottom line by certain percentage
-
This shifted conversation from do this because it’s right, to pointing out that business will benefit in an explicit way
—> This led to real organizational changes
-
Annual goals changed to reflect wording and topics used in our research
-
Researchers were invited to annual planning meetings, and were seen as experts on the work
-
We were often invited to executive planning process
—> In conclusion, the straightforward process of prepare, align, deliver, can help upgrade your research to focusing on organizational goals
—> Thank you for listening! I look forward to questions.
Q&A
-
Do you support individual researchers “turning their lens” on a regular basis? If so, how? Feels easiest to do when one is new?
A; Support researchers in doing the work. Take on things to shift organizational priorities over a long-period fo time
Actually easier to do when you have established yourself in right relationships. Alignment is easy if you have relationships with people you need to influence
-
Did what other research teams at Google were doing help, hurt, or feel irrelevant? Why/In what way?
A: Would say in general, other research teams are often reflections of research leaders leading groups
-
If we are aligned it’s very helpful
Talk about idea of prototyping results, by sketching them out
-
Helped with teams who had similar project backgrounds, and could go to other teams in areas
-
What about other UXR teams?
A: Helps if you have team with specific examples of what you want to do, and evidence to convince others, as well as expertise that can be suggested
-
Who are these Data Science people in an org? What titles do they have? I don’t know if my company has any?
A: Data science, in that people have data science already
-
Researchers who analyze large scale behavioral data of people using product you worked on
Tends to be predominantly quantitative, and focus inwards
-
Functional title and may have people who do work outside of data science
Engineering team may be doing it themselves at small companies
-
Look for titles involving analytics