Day 2-Driving Change with CX Metrics

— Thank you so much and we are so excited to share CX metrics with you today, as we have been busy making data and driving change with it

 

 

— A business like data feels conclusive and honest, but we have found way to have data to do the following:
  • Act as a vehicle for organizational change
  • Transform our products
  • Scale design quality

 

— Our company, Mastercard, acts as key infrastructure for payments for global scale, and any small release Is huge
  • Focus on partnership, as we don’t have banking license and making new experiences possible is key for us
— Aware that products need to be reasonably good, otherwise we risk becoming a company people avoid
  • So we are focusing on implementation and usage and retention

 

— It takes a lot to make products happen at Mastercard beyond the product, engineering, design triad
  • We are one of many groups, and are part of a complex value chain

 

—  Mastercard’s history as a sales-driven org has resulted in a set of historical anti-patterns
  • A HIPPO culture
  • Customer-Led Decision Making
  • Mistaking a willingness to pilot versus commercialization

 

— HIPPO Culture: Building things because someone important had an idea
  • No insights but vibes
  • Hippos want what they want

 

— Customer-led decision making
  • Customization gone wild, and associated scope creep
  • Bloated products that didn’t meet a need

 

— Mistaking willingness to pilot versus willingness to pay versus commercialization
  • Commercializing is far more expensive than pilot test

 

— We were to encourage more useful habits, and change ways of working beyond us and our roles

 

— Distilled to ways of working going sideways, in three big ways
  • Issues with desirability
  • Unclear ideas about Value exchange
  • Lack of clarity regarding value delivery
—Differentiated experiences and data to indicate values importnat to company and our future

 

— We addressed problems though MasterCard CX metrics as standardized indicators of product quality aligned to product development framework
  • They allow proper course correction and minimize rework in product design, and allow an innovative approach
— Enable product teams to take ownership in testing and research of products, in a way they typically couldn’t

 

— Connected design quality to revenue generated by products

 

— The metrics were
  • The Concept Desirability Score
  • The Design Quality Score
  • The Product Onboarding Gold Standard
— Concept Desirability Score  helped figure out the right space MasterCard should be playing in, versus where they should focus on being  more invovled

 

— Design Quality was determined once we have market and demographic, how should product be designed for ideal sufficiency
  • When it goes out the door, from minimal market problem point of view and qualifies all touch points customers will encounter
— Onboarding Gold Standard to figure out best way partner can enable for greatest point of success

 

— The Concept Desirability Score included
  • Figuring out what to design, whether through unmet needs or fragmented markets, and how our company could make things better
  • Dos
    • Go broad and have many concepts for opportunity
    • Your chance to figure out how to dial things up and back as product will be combination of different things and run dozens of these
    • Shouldn’t replace traditional work
    • Highlight risks and areas to avoid, and deep dive with thematic observations that could otherwise take several months to surface

 

 

— For the Design Quality Score
  • Know we have target demo and market and thinking about the partners we need to get product out the door
  • Where it’s about more about adoption and use versus targeting the market
  • How does someone align their life to the product, from an end-user point of view
    • How to make transactions and flows and different touchpoint with a product and continue to benefit relationship with produc
  • Going deep on a single idea
  • Do’s and Don’ts for this include
    • Getting into cognitive models of interaction
    • Not replacement for traditional research

 

 

— We’ll dive deep on the anatomy of outputs of these scores
  • We created diagnostics grouping of info in particular subject matter and up to 200 people and pieces of subject matter relevant for interaction of product
— We asked in output: Do people understand and comprehend what they are looking at?
  • Disruptor to current way of behavior and how would they integrate it
  • Qualitative feedback through research practitioners and answering all questions and quantitative questions
— This then bubbles up to thematic area of score, which becomes something PMs and project manager get excited about
  • Thematic layer is where a high score lets you dive way down into opportunity areas
— Finally have top-level score that encompasses diagnostic and thematic levels screening
  • This provides a holistic view of the product’s overall satisfaction
— From researchers point of view we can have great product and can twist know slightly and make it better
  • Lot of internal development though

 

 

— Why all this effort?
  • To avoid project, product, portfolio managers saying, “I don’t get it” and needing someone to explain an explanation of a metric
  • Not one person who got scores felt they couldn’t act on them
    • Avoided people doing research as window-dressing and not understanding potential value offered

 

 

— Will now try to articulate these metrics into a wireframe case study
  • On left was first DQS run for a subscription service for MasterCard, which could let you aggregate all your subscriptions into one sport
    • We felt this would crush it from a design quality perspective
  • But score was low, as people felt it didn’t differentiate from an Excel spreadsheet
    • End-users didn’t see the value
— So redesigned to the wireframe on the right, and aligning product with a person’s daily life
  • If service changed pricing model or did something different, here’s something for you to pay attention to
  • The design on right score exponentially higher, and not possible without an initial DQS

 

— Main audience is product from the Product Onboarding Gold Standards
  • Not measuring product at all, but quantify all touchpoints as customers try to implement product
— Customers get list of action items to choose to improve and scores that visible and have skin in the game for executives

 

— People are incentivized for speed to market, speed to scale, and ensuring product teams can move forward instead of of fast

 

— Product success should be about making customers more successful

 

— Our products are highly configurable
  • Can we give good instructions like documentation, troubleshooting, testing configurations?
  • If teams make changes we recommend, do they improve time to revenue and make more money

 

 

— What’s easy for us to deliver might not align with what customers need

 

— Asking people to take on responsibility is tough, and with incentives
  • Easy for us to plug-in to business processes

 

— Would be remiss if I didn’t bring up Will and Jenny’s presentation for need for metrics

 

— We will talk about operationalizing these metrics through a product dev framework
  • Otherwise just another checkbox, and  the frame will be completely useless
— We aligned CX metrics with product stage gates that already existed
  • Moving from concept to prototypes, can come in with opportunity space
  • Prototype to market test, have score to fine-tune best in class design to take to market

 

 

— Here’s how CX metrics align to framework and documentation, but once we’ve unveiled CX metrics, people are very grateful to have them
  • People like concept desirability score and design quality score, and onboarding goal standards
— Imagine you are armed with studies and research, so why not make sure customers build it in the right way

 

— All three metrics are a de-facto three-legged stool

 

 

— These metrics were not an overnight process, as it started several years and not enough people were initially there, and we still needed to nudge people to better decision making
  • People without design capability required outside intervention
    • If you don’t value it, you don’t fund it
  • Partners required specific kinds of help
— Building on guardrails helped us scale and prevent misuse

 

— We previously mentioned that people perceive automation for replacement for authenticity people are excited about tools and getting product to market by using metrics
  • You couldn’t do that though
— Most of constituents once they’ve ran a few tests— recognize need for more funding the need to look at once particular piece, and view research as like a laser (not a shotgun)
  • DQS and other metrics figure out where to aim laser, and this requires coaching from the design team on proper implementation

 

— We also faced the huge risk of gaming the design system, and people completing the evaluation process in a desire to impress people in leadership

 

— Products scored in an unusually close range, and worried if all PMs lied about the metrics they had
  • They didn’t, but gold standard paradigm existed
  • We built guardrails to prevent this kind of gaming in the future, like disclosing cross-functional contributors and uploading evidence
— Needed ability to analyze data, and have someone else  quantify the situation

 

— Moral is that score are means to an end, but goals is not to create fear, but incentivize dollars for changes
  • Create air cover for teams to make improvements too

 

 

— Thank you. Any questions?

 

Q&A
  1. How well established are measurement tools across org?
— Required for all products and have been mandated across the org
  • Point made where its less about metric itself but we make it more about mandate to get people excited about metric
    • People get excited about tools after initial frustration
  1. Scores are just the beginning, so do you emphasize the need for learning and iteration?
— Product Managers are worried they might do badly
  • Executives looking at trends and big opportunities
  • Resisted setting up score, to make sure it was in service of change, instead of overwhelming the PMs
  1. Any pro/con sentiments from product team using the scores?
— Been pro across the board, as the metrics speak their language
  • As people are so excited by metric they don’t do further research, and need shepherds to fine-tune things as they go
  • Couple of cons on B2B side to not test consumer products but banking products
    • Banking products require right audience and getting data correct
      • Dial numbers down in this case, to avoid overly challenging of data
  1. How if at all, UX heuristic assessment component of metrics?
— As of night now from CDS and DQS, heuristic evaluations are further upstream from what we are doing
  • We are thinking of introducing that metric though
— We are so close at the ground, so goal is to make more money
  1. How to set up data for metrics across broad range of areas?
— Run through vendor of ours, and mostly automated
  • Provided interactive prototype to get people across task
    • Combination of automation in diagnostic theme and specific things inherent to your product
  • Taking 30 minutes to go into the field for general populations, is inexpensive compared to other kinds of research and helps product feel like they own the project
    • It has to be cheap, fast, and understandable while tying to something real
    • This leads to benchmark for data, which we can check our products against
  • Information goes into scoring dashboard and contactless transactions
  • Make it easy to use, so it leads to differentiaton
— For all of metrics, we get people interested in participating and get interested in their wins and what could be done, since executives expect improvements