AR2021-Dark Metrics- Illuminating the Negative Impact of Digital Health Design (Raven Veal, IBM Watson Health)

—> I’m excited to speak about dark metrics and negative impact of digital health design

 

—> First, a little bit about me
  • My Pinterest is worth a thousands words
—> I’m a design researcher for IBM Watson Health, and am interested in a jambalaya of different things
  • Afrofuturism and speculative design
  • Stretching out and flipping different mental concepts
—> This is due to experience, in my family and community, with healthcare

 

—> I’ve designed my conversation with you to focus on four points
  • Where the concept of dark metrics came from
  • The framework itself
  • Case studies involving dark metrics
  • A call to action and final thoughts

 

 

—> For a while, in corporate America, it was emphasized that designs needed to be data-driven
  • This is not necessarily new, but I was surprised that product metrics focused on near-term and product oriented impact
—> Example of Google Heart framework, which categorized metrics that are good from business perspective, but not necessarily for the users
  • It doesn’t acknowledge whole person, outside of the tech they use

 

 

—> I’m passionate in my role about ethical thinking and how we view data-driven decisions
— Surveillance of social media, and racial biases have accelerated, so we need to understand risks and harms for what’s done in the world
—> Consider the Institute of the Future’s Ethical OS
  • It looks at risk zones technology can have from surveillance data to algorithmic control
—> How can this thinking be applied to success metrics and failures in the field?
  • Especially healthcare, with emphasis on no harm

 

 

—> Dark metrics is a new paradigm: something created as aggregate of AI ethics, personal experiences, and equity principles
—> Focused on individual level, as that’s where we have most immediate influence and direct impact

 

 

—> In terms of dark metrics, technology can be exclusionary, addiction, distracting, or disempowering
—> We will examine how each apply in real life scenarios

 

—> Disempowerment: Does it weaken autonomy and decision making?
  • This is about power
—> When researching product or features, are we taking authority from people who will use it?
  • IBM has idea of Augmented Intelligence, rather than Artificial Intelligence
—> Example of black box AI technologies for decision support
  • Imagine a friend with cancer who’s oncologist was using a fancy AI to treat her.
  • That AI would not be transparent with how its  recommendations were provided
    • The technology can appear smarter than you, and this is dangerous with life or death decisions

 

—> For disempowerment: I have an example of my work on improving performance with students with depression
  • How to give students skills to obtain help for their condition
—> It’s easy to lean towards side of “let me do it”, if you think people are impaired by their condition
  • This can be a from of implicit bias, or paternalism
—> So evaluate the design of the product, and evaluate yourself for implicit bias
  • Is there lack of explainability for how it works?
  • Can people prevent data being collected or override data decisions made by the product?
—> If answer is not to these questions, the device is disempowering
—> We can use a validated self-efficacy scale to see how people can understand health, and do something about it

 

—> Exclusion:
  • Does it introduce prejudicial treatment or inflict such treatment against others?
—> See the AI coding of racial discrimination,
  • Example of health-care company that tried to identify high risk patients, to see who might need additional resources to need help
  • Less money spent on caring for black patients than white patients
    • The algorithm sed prior health-care costs as a proxy for predicting treatment. The people whose healthcare cost more in the past got more treatment.
  • Ruha Benjamin refers to these practices as the “New Jim Code”
    • Reinforcing racial bias in technology
—> Importantly, exclusion doesn’t have to be an explicit goal, only indifferent to negative impacts of bias

 

 

—> So what does racial equity mean, in the context of exclusion?
  • Equity is the condition that would be achieved if racial/ethnic identity didn’t impact outcomes people faced
—> Following the death of George Floyd and the riots afterwards, IBM developed technology to reduce racial health disparities
—> We were encouraged to scale concept across all projects
  • How might we support experiences that are more inclusive of customers who are racially/ethnically diverse?
  • How can products be experienced as equitable?
—> We landed on a  prototype design rubric for assessing racial bias
  • Meant to raise awareness of bias in research operations and product design
—> Examples of this type of bias include:
  • Does team invite diverse participants to scope agenda and research questions?
  • Are there appropriate depiction of black people in product, avoiding stereotypes
—> This anti-bias focus should be embedded in design practices, the way accessibility is

 

 

—> Addiction: Does technology promote excessive use or unhealthy dependency?
  • Example of gamified patient applications
—> How can we tell people are addicted to technology?
  • Diary studies where people capture how often they spend time on app
  • We define what counts as an excessive level of use for engagement
—> Look at capturing well being, to see if higher engagement correlated with things like physical stress

 

—> Example of tension between electronic health records (EHR) and patient communication
  • EHRs breaks bedside manner, and doctor’s attentiveness to patients
—> We can tell that new health technologies are not stress tested in real-world environments
  • They are more distracting than helpful

 

 

—> So how can we tell if something is distracting?

 

—> Clinical trials are experiments/observations done on medical research
  • Testing treatments for safety and effectiveness
—> Imagine if you were admitted for emergency room for something urgent, would you want to be recruited for research at that time?
  • So how to support staff for recruiting patients for clinical trial, while maintains bedside manner?
—> We fielded study observations to study people active in clinical recruitments
  • Time spent looking at recruitment technology relative to time spent with patients
  • We also looked at cognitive load people had
—> We could tell how distracting clinical recruitment was, and designed new solutions. Solutions included:
  • Creating a voice assistant
  • As well as recommended language to provide to patient

 

 

—> So, as an overview, we discussed the core terms of dark metrics, signals as potential methods to figure out the metrics, data captured for each one

 

—> This work is emerging, but take this as starting place, with the overarching principle of prioritizing potential risks and harms of technologies

 

—> So I wrote out three questions/provocations for you:
  • How might we raise awareness of dark metrics among CFTs and clients?
    • Make pre-mortems a common exercise. Ask all of what can go wrong, use framework to ideate for risk mitigation
  • How might we reimagine data collection?
    • Focus on capturing holistic insights
  • How might we tell a complete story?
    • Storytelling will be effective in making decsions
    • How to prototype narrative for negative consequences

 

 

Q&A
  1. Do you have good resources for dark metrics?
A: I do. Will put them all up on her website, and first reference is Institute of Future for risk zone framework
—> More references will be published
  1. Is Enterprise Design Racial Equity rubric available?
A: See the initial field guide and initiative listed on IBM website. https://www.ibm.com/design/racial-equity-in-design/
  1. Some products are designed to improve decision-making efficiency for workers, including when those decisions are about other human beings. What are your thoughts about the tensions between business/market priorities and Disempowerment and Exclusion?
A: Focus on principle of do-no-harm, and be transparent and honest about where you are in dark metrics, so you can work to fix things
  1. All of the 4 Dark Metrics are data centric in that they mean more user data for orgs. What has been your experience of negotiating the Metrics against this perpetual hunger for data, especially when your work is looking to pull innovation toward user advocacy and control?
A: Want to give more thought to questions, and ask more about it
  1. How can we be more proactive in identifying unintended consequences that may not be obvious initially?
A: Collaborative conversations with team, as well as with customers and research participants
  • Diverse co-creation to mitigate against biases
  • Key to reducing bias as much as possible
—> Not every diverse population will have same experience
  • Do research and get them
  1. Any input or reaction from frontline workers who deal with black-box messages?
A: Conversations in clinical trials wanted to have ability to know what’s going on
  1. Great topic & framework! I’m curious if you’ve had input/reactions from front line workers in the field? In particular thinking of community pharmacy workers who deal with many “blackbox” answers from insurance providers and end up being the messenger for a ill designed system.
A: Try to co-create as much as possible, and feel you should have perspective and point-of-view from company
  • Clients should respect your desire to do no harm
  1. If as a researcher you find a metric that is driving unintended consequences, any advice for how best to raise these concerns? Metrics are often set for the year, and it’s hard to go back to management and suggest they revisit metrics/goals.
A: With any new research sprint or product see if you can add new sub-metrics that can illuminate dark side of measuring success
  • Capturing metrics you have and what can be added and what can make metric set better