AR2021-What Did I Miss? The Hidden Costs of De-prioritizing Diversity in User Research (Megan Campos, Mad*Pow)
—> Hi everyone, and I hope everyone is staying safe and healthy
—> Today’s talk will be about the importance of diversity in UX research
—> This topic has its roots in my own experience in the nuances of identity, especially with her own background as being half-Puerto Rican.
-
This has impacted how I approach UX research recruitment
—> At MadPow, we are always examining practices and how they can be improved for clients and users
—> Now, to frame this talk:
-
This conversation is ongoing. Our goal is to expand the scope of conversation and learn together
-
I don’t have the solution, and we need to work together to iterate and evolve on it
-
Diversity for UX research refers to race, gender, economic, social status, disability and other aspects of identity
-
We need to avoid the term ‘under-represented’, as we don’t want to center dominant characteristics as the standard for our research. Mis-represented is a better word choice
—> How’d I get here?
-
I noticed that all screeners called for a mix of criteria like gender, race, income
-
But actual user studies had a homogeneous talent pool that was all cis-gendered, and mostly white and middle-class
—> I wanted to know the reasons why a desire diverse factors didn’t have impact on how research was conducted
—> Demographics matter, and shape how we experience the world
—> There are many essays, books, journal, posts that capture how the world looks different to you depending on skin color, sexuality, class status, etc.
-
From how you walk into a store, to everyday conversations, you are shaped in visible and invisible ways by your demographic make-up
—> For a recent example, look at how Black populations were impacted by the coronavirus pandemic, due to pre-existing health-care system discrimination
-
Importantly, this discrimination was visible only when stress occurred to the healthcare system in the form of the pandemic
—> So how do we get demographic mix for UXR efforts?
—> First, I wanted to validate to see if I was only person who saw need for changing our approach. So I ran a survey
—> I wanted to get a sense of how people think about demographic criteria,
-
I reached out to people who run UX research studies, and recruitment screeners for these studies
-
I also reached out to personal/professional networks
—> At a high-level my survey indicated that demographic criteria is less important than project criteria, or contextual criteria
—> Then there was a hierarchy of importance In terms of which demographic data was important for UX research
-
Race, ethnicity, gender, sex, age, location, income, education, disability, and profession
—> Race, ethnicity, gender, sex were seen as less important relative to other criteria like age, which was seen as the most important rating
—> Location, profession, household income, fell into the middle
—> Demographic data with more invisible structures was given less priority in terms of research relevance
-
The highest priorities were profession and age which clearly impact a person’s experience
-
The lowest priorities were race and gender which do not have as visible and impact
—> I began creating a ranked order to assign characteristics with higher/lower area
—> My hypothesis: Since we want to believe gender, ethnicity, and race don’t have an impact on our lived experience, and the experiences we observe in others, we have a tendency to downplay these differences
—> As a result, we are comfortable saying differences matter in things that are more obvious to us like occupation and age, than in things that are less visible like race and gender
-
Seeing race and gender impacting people’s experiences makes us see the inequality in other’s experiences, and that makes us feel uncomfortable
—> So why not emphasize diversity?
-
I heard two kinds of responses for my request to recruit for diversity
-
1) Demographics don’t shape participant behavior
-
2) We know that demographics matter, but we can only do our best
-
—> Here are some erbatim quotes that say demographics don’t matter
-
In their view, a person’s identity and the experiences they have, doesn’t matter in a study
—> However, by neglecting diversity, we are giving in to existing structural and social inequality
—> We can’t expect people to have behaviors and experiences completely separate from their identities
-
And we are also missing an opportunity to serve our users
—> On the second belief, people stressed they did what they could to make sure participants were diverse
—> I agree with this sentiment, but shouldn’t our “best” be open to improvement and evolution?
-
We should be able to shift our perspectives to adjust our practices accordingly
—> We should apply a willingness to improve our practice
-
Stagnation isn’t good for everyone at best, and can be detrimental to some at worst
—> Now, for some examples
—> We wanted to figure out what the health-care experience looked liked for independent, employer-insured adults in Massachusetts.
-
Nothing in this case study was explicitly race specific, but health-care in the United States is stratified along racial lines
—> We set up a panel quota of non-white participants, which was pulled from census data estimates
-
Our recruiter struggled to get participants for quota, but we insisted on it, and we finally got six non-white participants
—> These non-white participants didn’t have a totally different experience from the white participants, but their personal stories had distinctions that reflected other trends, which included:
-
A wariness of being under-prescribed and over diagnosed
-
A fear of being unable to find Black doctors, as subjects wanted someone who could relate to them
—> One participant expressed a concern about being under-prescribed and over-diagnosed.
—> Another felt his doctor dismissed and ignored him due to his race.
-
Being Black influenced his health-care experience,
—> These differences weren’t rooted in profession or age, but rooted in experiences in the subject’s identities as Black men and women
-
These differences are important
—> Interviews gave us personal stories and let us craft solutions to address pain points
—> So what happens when don’t insist on diversity?
—> Well, we don’t know what we miss when diversity is avoided in recruitment
-
We can’t ask questions to people that were excluded, since they are out of the conversation by default
-
All we have are hypotheses
—> For example, in another project, our goal was assessing the comprehension and actionability of a health insurance “welcome letter” for new members
-
The audience was 80% non-white and had a large segment of non-English speakers
—> Despite the clear need for diversity, the participant list was not reflective of the population served by the provider
-
All participants spoke English as their first language
-
The client felt it was too expensive to hire a Spanish-speaking moderator
—> As a result, all the people in the study understood what the insurance welcome packet was asking them do
-
The impact of homogeneous recruitment was that the client couldn’t say anything with certainty about their customer’s experience
-
They could only provide assumptions about what wasn’t heard, like cultural barriers, language comprehension issues, etc.
—> We don’t know if these cultural barriers are real though, so we can’t solve for these barriers
-
All we have are assumptions that otherize groups, and treat the group we talked to as the norm for our recommendations
—> So how do we ensure diversity?
—> I will discuss an initial approach and suggestions
—> First use a third party to get research recruits
-
Get very specific in asks for third-party recruits
-
Need to be demanding with numbers, and making sure the recruiter can deliver
-
—> If we accept that demographics matter, and shape experiences, we need to be specific and demanding for quotas
-
If you make it clear it’s not a suggestion, you get what you ask for
-
Recruiters should not treat this as optional
—> Next, try screening recruiters, knowing you want to recruit diverse participant pool. Be upfront with recruiter and ask if they can meet your needs or not
—> If it can’t work, find someone who can execute on an address
-
Hire a recruiter who can get job done accurately
-
Recruiters will make change only when you incentivize them to do so
—> Your responsibility to make diverse number of recruits a priority
—> Regarding guerrilla research, people tend reach out to people who look a lot like them.
-
You need to go outside your network, and tailor your recruitment to appeal to a broader audience
—> Many researchers have already done this, and I encourage you to do so
—> The following suggestion comes from my experience with teams who are homogeneous
-
Many of us are operating from a position of privilege as a result, and are bringing a similar perspective to the research process
—> Hiring diverse teams, can help you catch people who don’t get recruited for user research
—> Diversity is worth the effort
-
If demography matters, we should recruit participants who reflect the make-up of our user demographics
-
-
We need to be insistent on finding the pool of recruits
-
—> Otherwise we will get flat dimensions, and making assumptions about wants, needs, and experiences of our users
—> This presentation is not the end of the conversation
-
There’s no guaranteee that diverse participants will reflect our user audience
-
But if we continue to work with homogeneous recruitment pools there is a 100% chance of ignoring the needs of a diverse population
Q&A
-
When talking with specific audiences where diversity is a known systemic issue (such as college students), how do you tackle recruitment in a way that does not result in any one or two participant feeling like they are ‘tokenized’?
A: You can’t represent everyone, but see how you can do the most good with the work that you had
—> Use words that are inclusive, and be conscious of how audience wants to be spoken to
—> Questions asked should cover holistic experience, not just focusing on one characteristic
-
Would you recommend using purposive sampling to draw more on experiences of those marginalized by various factors (i.e. over-representing them in the sample) rather than using representative sampling?
A: We are currently over-representing white/cis people, so when needs are there, it makes sense
-
What is your process of deciding what the specific mix should look like? Does it start from understanding the product user base?
A: Start with asking if client has audience breakdown of existing user population, and understanding where the product wants to go
—> If clients don’t have the data, reference US census data as a backstop
-
For those researchers undertaking their own recruitment, do you have any recommendations for how to reach out to misrepresented groups?
A: Caveat, in that I use third-party recruitment and not going into space
—> Emphasize getting outside of the comfort zone, but defer to experts
-
Any advice for B2B companies that primarily have members of a specific profession as users? Working in healthcare, I have buy-in that diversity is key for patients. But there’s this (incorrect) idea that being a physician overrides every demographic category.?
A: Depends on scope of study. It depends whether B2B use clients to do work beforehand to pull evidence showing experience along racial/gender lines might influence how users interact with product
—> Incorporate diversity study pool piece by piece and that will get you there
-
I work for a company that makes apps for outdoor activities like hunting and offroading. Our customer base is mostly white men, and my research participants mostly match this. I know that adding diversity is the right thing. But how do you champion bringing in diversity if the diversity doesn’t match your customer base?
A: The wider the audience, the more you can sell, and you can get a larger purchasing audience
-
You will have to make case to support types of marketing efforts for broader user base
—> It’s been found that when you have findings that suggest solutions of smaller population, you will serve larger population as well
—> Moreover, there are places are there for a diverse recruitment pool, but people assume they don’t exist.
-
NPR podcast Code Switch helps tell story of these kinds of populations
-
What are the known barriers that you often hear about when recruiting these populations?
—> Main barrier is recruiters with mostly white panels, as they struggle to find non-white people in New England for user research studies
—> Another barrier is strict adherence to getting representative sample
-
Putting screener in way that’s very specific and asking questions in a sensitive way