AR2021-What Did I Miss? The Hidden Costs of De-prioritizing Diversity in User Research (Megan Campos, Mad*Pow)

—> Hi everyone, and I hope everyone is staying safe and healthy
—>  Today’s talk will be about the importance of diversity in UX research

 

—> This topic has its roots in my own experience in the nuances of identity, especially with her own background as being half-Puerto Rican.
  • This has impacted how I approach UX research recruitment

—> At MadPow, we are always examining practices and how they can be improved for clients and users

 

—> Now, to frame this talk:
  1. This conversation is ongoing. Our goal is to expand the scope of conversation and learn together
  1. I don’t have the solution, and we need to work together to iterate and evolve on it
  2. Diversity for UX research refers to race, gender, economic, social status, disability and other aspects of identity
  3. We need to avoid the term ‘under-represented’, as we don’t want to center dominant characteristics as the standard for our research. Mis-represented is a better word choice

 

—> How’d I get here?
  • I noticed that all screeners called for a mix of criteria like gender, race, income
  • But actual user studies had a homogeneous talent pool that was all cis-gendered, and mostly white and middle-class
—> I wanted to know the reasons why a desire diverse factors didn’t have impact on how research was conducted

 

—> Demographics matter, and shape how we experience the world
—> There are many essays, books, journal, posts that capture how the world looks different to you depending on skin color, sexuality, class status, etc.
  • From how you walk into a store, to everyday conversations, you are shaped in visible and invisible ways by your demographic make-up

 

—> For  a recent example, look at how Black populations were impacted by the coronavirus pandemic, due to pre-existing  health-care system discrimination
  • Importantly, this discrimination was visible only when stress occurred to the healthcare system in the form of the pandemic

 

—> So how do we get demographic mix for UXR efforts?
—> First, I wanted to validate to see if I was only person who saw need for changing our approach. So I ran a survey

 

—> I wanted to get a sense of how people think about demographic criteria,
  • I reached out to people who run UX research studies, and recruitment screeners for these studies
  • I also reached out to personal/professional networks

 

—> At a high-level my survey indicated that demographic criteria is less important than project criteria, or contextual criteria

 

—> Then there was a hierarchy of importance In terms of which demographic data was important for UX research
  • Race, ethnicity, gender, sex, age, location, income, education, disability, and profession
—> Race, ethnicity, gender, sex were seen as less important relative to other criteria like age, which was seen as the most important rating

 

—> Location, profession, household income, fell into the middle

 

—> Demographic data with more invisible structures was given less priority in terms of research relevance
  • The highest priorities were profession and age which clearly impact a person’s experience
  • The lowest priorities were race and gender which do not have as visible and impact
—> I began creating a ranked order to assign characteristics with higher/lower area
—> My hypothesis: Since we want to believe gender, ethnicity, and race don’t have an impact on our lived experience, and the experiences we observe in others, we have a tendency to downplay these differences
—> As a result, we are comfortable saying differences matter in things that are more obvious to us like occupation and age, than in things that are less visible like race and gender
  • Seeing race and gender impacting people’s experiences makes us see the inequality in other’s experiences,  and that makes us feel uncomfortable

 

—> So why not emphasize diversity?
  • I heard two kinds of responses for my request to recruit for diversity
    • 1) Demographics don’t shape participant behavior
    • 2) We know that demographics matter, but we can only do our best

 

—> Here are some erbatim quotes that say demographics don’t matter
  • In their view, a person’s identity and the experiences they have, doesn’t matter in a study

 

—> However, by neglecting diversity, we are giving in to existing structural and social inequality
—> We can’t expect people to have behaviors and experiences completely separate from their identities
  • And we are also missing an opportunity to serve our users

 

—> On the second belief, people stressed they did what they could to make sure participants were diverse
—> I agree with this sentiment, but shouldn’t  our “best” be open to improvement and evolution?
  • We should be able to shift our perspectives to adjust our practices accordingly
—> We should apply a willingness to improve our practice
  • Stagnation isn’t good for everyone at best, and can be detrimental to some at worst

 

—> Now, for some examples
—> We wanted to figure out what the health-care experience looked liked for independent, employer-insured adults in Massachusetts.
  • Nothing in this case study was explicitly race specific, but health-care  in the United States is stratified along racial lines

 

—> We set up a panel quota of non-white participants, which was pulled from census data estimates
  • Our recruiter struggled to get participants for quota, but we insisted on it, and we finally got six non-white participants
—> These non-white participants didn’t have a totally different experience from the white participants, but their personal stories had distinctions that reflected other trends, which included:
  • A wariness of being under-prescribed and over diagnosed
  • A fear of being unable to find Black doctors, as subjects wanted someone who could relate to them

 

—> One participant expressed a concern about being under-prescribed and over-diagnosed.
—> Another felt his doctor dismissed and ignored him due to his race.
  • Being Black influenced his health-care experience,
—> These differences weren’t rooted in profession or age, but rooted in experiences in the subject’s identities as Black men and women
  • These differences are important
—> Interviews gave us personal stories and let us craft solutions to address pain points

 

—> So what happens when don’t insist on diversity?
—> Well, we don’t know what we miss when diversity is avoided in recruitment
  • We can’t ask questions to people that were excluded, since they are out of the conversation by default
  • All we have are hypotheses

 

—> For example, in another project, our goal was assessing the comprehension and actionability of a  health insurance “welcome letter” for new members
  • The audience was 80% non-white and had a large segment of non-English speakers
—> Despite the clear need for diversity, the participant list was not reflective of the population served by the provider
  • All participants spoke English as their first language
  • The client felt it was too expensive to hire a Spanish-speaking moderator

 

—> As a result, all the people in the study understood what the insurance welcome packet was asking them do
  • The impact of homogeneous recruitment was that the client couldn’t say anything with certainty about their customer’s experience
  • They could only provide assumptions about what wasn’t heard, like cultural barriers, language comprehension issues, etc.
—> We don’t know if these cultural barriers are real though, so we can’t solve for these barriers
  • All we have are assumptions that otherize groups, and treat the group we talked to as the norm for our recommendations

 

—> So how do we ensure diversity?
—> I will discuss an initial approach and suggestions

 

—> First use a third party to get research recruits
  • Get very specific in asks for third-party recruits
    • Need to be demanding with numbers, and making sure the recruiter can deliver
—> If we accept that demographics matter, and shape experiences, we need to be specific and demanding for quotas
  • If you make it clear it’s not a suggestion, you get what you ask for
  • Recruiters should not treat this as optional

 

—> Next, try screening recruiters, knowing you want to recruit diverse participant pool.  Be upfront with recruiter and ask if they can meet your needs or not
—> If it can’t work, find someone who can execute on an address
  • Hire a recruiter who can get job done accurately
  • Recruiters will make change only when you incentivize them to do so
—> Your responsibility to make diverse number of recruits a priority

 

—> Regarding guerrilla research,  people tend reach out to people who look a lot like them.
  • You need to go outside your network, and tailor your recruitment to appeal to a broader audience

 

—> Many researchers have already done this, and I encourage you to do so

 

—> The following suggestion comes from my experience with teams who are homogeneous
  • Many of us are operating from a position of privilege as a result, and are bringing a similar perspective to the research process
—> Hiring diverse teams, can help you catch people who don’t get recruited for user research

 

—> Diversity is worth the effort
  • If demography matters, we should recruit participants who reflect the make-up of our user demographics
    • We need to be insistent on finding the pool of recruits
—> Otherwise we will get flat dimensions, and making assumptions about wants, needs, and experiences of our users

 

—> This presentation is not the end of the conversation
  • There’s no guaranteee that diverse participants will reflect our user audience
  • But if we continue to work with homogeneous recruitment pools there is a 100% chance of ignoring the needs of a diverse population

 

Q&A
  1.  When talking with specific audiences where diversity is a known systemic issue (such as college students), how do you tackle recruitment in a way that does not result in any one or two participant feeling like they are ‘tokenized’?
A: You can’t represent everyone, but see how you can do the most good with the work that you had
—> Use words that are inclusive, and be conscious of how audience wants to be spoken to
—> Questions asked should cover holistic experience, not just focusing on one characteristic
  1. Would you recommend using purposive sampling to draw more on experiences of those marginalized by various factors (i.e. over-representing them in the sample) rather than using representative sampling?
A: We are currently over-representing white/cis people, so when needs are there, it makes sense
  1. What is your process of deciding what the specific mix should look like? Does it start from understanding the product user base?
A: Start with asking if client has audience breakdown of existing user population, and understanding where the product wants to go
—> If clients don’t have the data, reference US census data as a backstop
  1. For those researchers undertaking their own recruitment, do you have any recommendations for how to reach out to misrepresented groups?
A: Caveat, in that I use third-party  recruitment and not going into space
—> Emphasize getting outside of the comfort zone, but defer to experts
  1. Any advice for B2B companies that primarily have members of a specific profession as users? Working in healthcare, I have buy-in that diversity is key for patients. But there’s this (incorrect) idea that being a physician overrides every demographic category.?
A:  Depends on scope of study. It depends whether B2B use clients to do work beforehand to pull evidence showing experience along racial/gender lines  might influence how users interact with product
—> Incorporate diversity study pool piece by piece and that will get you there
  1. I work for a company that makes apps for outdoor activities like hunting and offroading. Our customer base is mostly white men, and my research participants mostly match this. I know that adding diversity is the right thing. But how do you champion bringing in diversity if the diversity doesn’t match your customer base?
A: The wider the audience, the more you can sell, and you can get a larger purchasing audience
  • You will have to make case to support types of marketing efforts for broader user base
—> It’s been found that when you have findings that suggest solutions of smaller population, you will serve larger population as well
—> Moreover, there are places are there for a diverse recruitment pool, but people assume they don’t exist.
  • NPR podcast Code Switch helps tell story of these kinds of populations
  1. What are the known barriers that you often hear about when recruiting these populations?
—> Main barrier is recruiters with mostly white panels, as they struggle to find non-white people in New England for user research studies
—> Another barrier is strict adherence to getting representative sample
  • Putting screener in way that’s very specific and asking questions in a sensitive way