Day 2–Experimental Research: techniques for deep, psychology-driven insights

— Thrilled to be back to share how deep insights for decision making

  • May recognize me from prior talk, but for those who don’t I’m in social psychology to answer challenging questions and same technique at Thumbtack

— Thumbtack is a platform to help homeowners figure what to do, when, and who to hire to take care of their home

— Keep this context in mind for key study from thumbtack to show power

— Which trust signals mattered most when hiring a pro?

  • 20 trust signals across eight trust related metrics

— Let’s take a step back first

  • Introduced to fantastic qual methods in research and my favorite method and experiments

— Industry standard A-B testing and experimental research for grasping the “why”behind outcomes and revealing causation to prioritize high impact product ideas to save time and resources in process

— Experiment research is a process where variables are manipulated to show a causal relationship

— Different from

  • AB Testing: Focuses on outcomes by comparing specific variants.
    • Valuable, but don’t show the why
  • Surveys: Grasp self-reported attitudes
    • But correlational, and don’t show what caused attitudes

— Experimental shows why behind attitudinal change and causation, not correlation

  • Actionable and strategic

— Two key concepts

  • Independent Variables: What you manipulate in experiment
  • Dependent Variables: Outcome you measure, as a result of the independent variable

— Relatable example of coffee improving productivity

Dependent variable is productivity for tasks completed, and independent is whether we drink coffee or not

    • Dependent variable is what we observe as a result of what variable we control

— A few core principles of experimental research

  1. Randomization: Different conditions randomly assigned to eliminate bias
    1. In the coffee and productivity experiment you randomly assign people to drink coffee or not
  2. Control Groups: Baseline for measuring impact of intervention on dependent variable
    1. People who don’t drink coffee, and whether coffee makes difference
  3. Focused Manipulation: Systematically vary independent variable to test it’s impact, and isolate effect on dependent variable
  4. Measurement of Outcomes: Essential to determine effect of manipulation and tracking how many tasks people complete in an hour

— Explore why behind behavioral outcomes and understanding drivers to get deeper pyschology driven insights and

  1. Show causation and true impact of change with confidence
  2. Allows smart prioritization to test small scale ideas focusing time and energy on what weill succceeed
  3. Maximize impact: By saving resources and changes that truly work

— Identify hypothesis and clear, testable question for problem you are trying to solve and number of tasks in an hour

  • Have someone drink coffee and measured by number of tasks completed

— Design and randomize to create conditions and random assignment to eliminate bias and show by external barriers

— Collect and analyze data and use tool like Qualtrics

— Apply appropriate statistical techniques and number of tasks completed by each group

— Interpret results: Comparing outcomes of each group and may need to rethink assumptions

— Example of Thumbtack platform

  • Trust is crucial for marketplaces like Thumbtack that involve expensive home projects

— Trust signals previously implemented haphazardly and we prioritized ones with the most promise in product testing, and were unsure what would increase user perceptions of trust in Thumbtack and its professional network

  • 20 trust signals across 8 variables

 

— Example of factorial design and bringing variables to life, and create static stimuli for pro-list and first impressions and service architecture and one signal

  • Isolating one trust signal by condition and making sure user perceptions were tied to signal

— One for house-cleaner and electrician

  • Random assignment made it true experiment and infer causation by signal being tested and single experiment

— Two hypothetical scenarios and one for housecleaner and more generalizable and advantage of approach to show how trust signals influence perception

  • Ecological validity— Hypothetical scenarios don’t tie to real world decision making
  • Needed to test in real world conditions

— Trust couldn’t capture in single measure and two layers of trust

  • Thumbtack level of trust: Confidence in screening process, quality of professionals
  • Pro-level of trust: Trust in individual to hire, like qualifications and likelihood to hire the pro

— Graph shows approach in trust between control group and each experimental condition, against statistical significance

  • High-level trends remained clear

— Half of signals had positive impact of trust in Thumbtack

  • Targeted trust building efforts and how customers perceive thumbtack

— Trust signals tested across all dependent variables and each signal impact visualized using color intensity, while white cells showed no significant affect

  • Out of signals tested

— 10 had broad impact across all eight factors, and provided clear roadmap of what to prioritize and test in product

— Valuable for several reasons

  1. Efficient. to shave time and resources compared to individual A-B tests
  2. Comprehensive and gained holistic understanding
  3. Causal insights to show what signals influenced trust
  4. Scalable Impact to show what to focus on for 2025, and one just launched in past week

— Two parts

  • Ethics especially when manipulating variables, and can impact perceptions of emotions and behaviors for ethical practice to have harm an integrity of study
  • Informed Consent— Understanding study
  • Avoiding Harm— Preventing unnecessary harm to participants in the research

— Rigor

  • Randomization to show differences are attributable to variable than internal factors
  • Adequate to ensure enough participants to detect meaningful differences between conditions
  • Robust analysis to make replicable and able to inform decision making

— Ability to uncover isnights that show real innovation and like others key is to start small with clear testable hypothesis

  • Follow one pager to take ideas to evidence, and evidence, to action

Q&A

  1. Teaching one experimental psych concept what would it be?
    1. Goes down to ‘correlation doesn’t equal causation’ and you need an experiment to show what might be causing direct relationship between those two things