Day 2–Finding the Unknown in the Known: Harnessing meta-analysis and literature review

— Excited to share insights into topic that will transform how you can transfrom research

  • Meta-analysis and literature reviews to impact of every insight we gather

 

— Manage UXR at Thumbtack, and social psychologist by training, and throughout career transformative insights come from what’s overlooked and underutilized

— How often do we overlook knowledge we already have?

  • Easy to tackle next big challenge, but how often do we ask what can we build on?

— Each study represents investment of time, resources, creativity, and expertise

  • What if answers are hidden in what we already know

— Explore how to revisit research for full potential

— So let’s talk through what we are talking about versus not

  • Amplifying act of work already done, to ignite new questions or priorities

— Wells and support tickets and no replacement for new insights when needed

— Complement, not substitute

  • Uncovering unknown in known and using that to move forward smarter and faster

 

— Some key terms to capture are meta-analysis and literature reviews

  • Meta-analysis: Equivalent to looking at IMDB or Rotten Tomato score for a movie to capture aggregate sentiment

 

— Literature reviews look at existing research for foundation for decision making and evaluating studies for a specific field and using term more broadly and all together themes and knowledge for strategy and decision making

  • Instead of single score or reviews, looking for themes and perspective to get sense of why people feel the way they do
    • Insights to get bigger picture

— Both about building on what we know , and focus on past research for better decisions

— Why do this?

  • Huge time and resource saver and leverage rich foundation already created and existing knowledge to refine strategy and provides valuable context and insights

— Rigor and credibility of work and creating robust patterns more than a single study on its own

  • Amplifying impact of other insights

— Three ways to do this

  • Informing new study by showing gaps and research questions and launch pad for deeper research
  • Create shared understanding for team and align with same insights and helping teams work together
  • Identify strategic opportunities and well-crafted review is means to an end, and actionable insights

— Example of this from Thumbtack and high impact research deliverable

  • Collaboration of four researcher, and insights from other internal studies

— What is Thumbtack though?

  • Tech platform to help people care for and manage their home with network of local service professionals

— Goal of the playbook to figure out what team would achieve and how to drive value for the organization, and unified perspective

  • Need resource to align around shared understanding of what they need

— Make sure strategies rooted in data and insight, rather than assumptions and stereotypes

— Helping teams connect with real people bheind numbers

— Process behind playbook

  • Define scope for foundational homeowner insights, so playbook would be long-term resource
  • Gather studies from 2018-2024, and structured database from Google Sheet, and insights themselves and homeowner resource reports
  • Synthesize insights through FigJam board and explore insights with frameworks and key takeaways
  • Structuring and refining into clear logical framework and drive empathy
    • Designer to polish playbook visually to make clear and user friendly
  • Living document and making sure it remains relevant and high impact resources

— Share quote from playbook, that he was speechless with impression and underscores why lit review and elevating to impact resource can be so powerful and foundational impact across company, and unified language for customers

  • Teams across product and operations and shared homeowner experience

— Drivers foundational lit review and principles to service professionals and broader culture of learning and alignment

— So for meta-analysis, it aggregates and statistically analyzes findings from multiple studies

  • Reveals overarching patterns and surfaces trends for studies and strengthens confidence

— Make patterns more reliable

— Answering high impact questions and strategic questions for multiple data-sets or studies

 

— Example of work from NNG of synthesizing existing research can provide new insights and four usability testing studies and none were about search, but seeing how people use it naturally like party or home improvement projects

  • Leverage large number of studies and apply same principles for trends and patterns

— Providing meta-analysis lens and entirely new questions on search behavior

— Three key goals and how people interacted with search results and pages

  • Analyzing search result complexity with images, videos, and carousels changing how attention distributed
  • Actionable insights for digital-product teams

— Methodical and deliberate and first step of defining questions and how users interact with modern search results and pages

  • Collect data: Data from usability testing studies
  • Coding specific metrics: Query length, eye-gaze patterns, etc.
  • Schema: Develop schema to allow them to capture key patterns
  • Insights: Generate insights

— Key findings from meta-analysis

  • Pinball pattern of review with people bouncing from end of page, and challenged SEO strategy, and layout for design teams

— Allow team for broad, statistically supported trends from new studies

  • New questions answered that prior studies didn’t address

— Some common challenges exist though

  • Selection Bias: Rely heavily on sources you choose to include, if not relevant or credible where
    • Clear criteria upfront
  • Overgeneralization: Stretching findings between what data supports and contextualize findings and outline them, and not apply to broadly
  • Data Variability: When synthesizing sources, differing methodologies or format, and creating schema for coding metrics
  • Resource Constraints: Processes can be time-consuming and high volume of sources, and research repository

— Leave with key idea that insights we need are at our finger trips and synthesize and maximize what already done, and fresh questions of past work, with living, evolving resource, instead of static deliverables

— Synthesize related findings, and even small efforts can align team

  • Go unlock pattern of what you already know

Q&A

  1. Example of how it created unified perspective, and lack engagement from whole team
    1. Got leadership buy-in to present at company all hands with great visibility and worked with internal comms team on push to include it in weekly company newsletter to make sure it got out there and was on same page
    2. Worked directly with comms team to be visible , and internal comms appreciated to get it out