Day 2 Session Notes–Meters, Miles, and Madness: New Frameworks to Measure the (Elusive) Value of DesignOps

— Thank you and to audience good morning, and welcome to the talk

 

— Start with story oaboutn outer space

  • Mars Climate Orbiter: A $500 million probe of Mars that crashed in 1999 and destroyed by Martian atmosphere, a big expensive mission that failed

— Reason behind crash was that the wrong metric units were incorrectly measured in the landing

  • Measuring wrong cost half billion
  • What and how your measure matters a lot—especially for DesignOps

— DesignOps is harder with what and how to measure is key for mission

  • Value of DesignOps itself is often hard to point out

— Measuring value of DesignOps important but complicated and measuring impact feels aspirational at best

  • Heard version of what we can do without you—but that doesn’t add up to investment proof

— According to report, 52% of DesignOps have defined metrics and outcomes

  • We need measures to address DesignOps impact

— Why?

  • DesignOps is relatively new practice and obscure discipline
  • We are busy and have small teams
  • Teams are not consistent with fluid outcomes that are not stable
  • We are using outdated metrics foisted on design and DesignOps from adjacent business practices

— Do you find hard to measure DesignOps value for these reasons?

— Design and DesignOps defined by traditional frameworks like these, but applying them to DesignOps is like trying to fit a star into a box

  • We know frameworks not good enough for DesignOps

— Measuring financial and functional benefits omits broader constellation of impact

— We can let in more light and more frameworks that have more complete vision of what DesignOps can do

  • Share three frameworks for designs value
    • Jobs To Be Done (JTBD), HEROES, and DesignOps Value Factors

— While traditional frameworks are narrow for DesignOps, we should show value in business terms and frameworks more effective to do this

— DesignOps using framework typically in product dev, but origins from business development framework

— Think of every products and service as something bought for fulfilling a particular job

  • Understand job first, and then discovery about job to build better product or service

— JTBD service can be used for better programs and processes

  • For example design teams don’t buy a drill because it is a drill, but a hole that requires management

— Five steps to follow

  • Outputs look from first four steps— finding design performers who need the job
  • Research their jobs and outcomes like, standardize findings, and ‘deliver great design’
  • Figure out main jobs versus job tasks

— Measuring what happens in the final step to take job tasks and define single measurable outcome for framework

  • Direction : Increase, reduce, maximize
  • Object: Time, confidence bugs, something quantifiable
  • Target: What job performers want to impact or accomplish

— Together they make an outcome that define what org wants to gets done, and meaningful metric to measure

— Why do this?

This introduces new opportunities for the DesignOps teams— advancing stakeholder relationship,

— When

  • Your work is mature and design jobs are already established

— What about newer work?

  • Enter the HEROES framework

— Let’s look at current challenges and opportunities in organization

  • 76% of staff get burnout
  • 60% of staff deals with unskilled work

— What do they mean?

  • Common challenges or opportunities faced by design teams and each statistic comes from heroes framework to capture impact design has in an organization

— Example of categories and associated impact questions

  • Metrics that matter to you, by asking these six framing questions and focus on areas where things are unclear

— List of metrics to help answer these questions, and tailored to fit organization

  • HEROES framework acts like questionnaire rather than step-by-step guide
  • Figure out metrics and then start working on them

— Reply in Slack what measurement feels most relevant to you right now

— Impact of Health on design org

  • Can be demanding and draining but healthy team lead to healthy org

— DesignOps makes big difference by tracking specific health metrics and supportive and balanced environment for teams

— Analyze efficacy of delivering work

  • Designers to focus on design versus work about what work
  • Every hiccup in process, and redundant flow leads to burden on team as well

— Measure success to be more effective and efficient and make huge impact here

— Measuring how prepared team are with skills and knowledge to excel in roles

  • Tool utilization rates like knowledge gained and boosting readiness by enhancing skills
  • Future-proofing design capabilities

— Products and services prove useful valuable and impactful

  • Significant role in enabling design’s success and usability score and time on task to measure impact of good design

— Ultimately work of design to tie to business outcomes

— Metrics to answer question of how products and services designed, and design choices impact ecosystems

  • Redefine value and assess impact of broader ecosystem

— Measure impact of products and metrics related to inclusive accessible and ethical design practices — vendors and tools

  • Environmental impact or diversity

— DesignOps can’t single-handedly change focus, but we can champion responsible practices in teams

— Sentiment metric to understand how design is perceived and understood

  • Sees value placed on design and impact on UX

— Perception through design leadership roles, training participation rates, budget allocation , or social media engagement

  • Design perceived by hires and organizations

— What does this look like in action?

  • Go back to summer where GenAI everywhere and all products pivoted to GenAI feature
    • Designers felt unprepared to do this
  • DesignOps worked to figure out where we could be most valuable

— For readiness, designers weren’t ready to build or understand Generative AI

  • Had Generative AI summer camp to boost fluency at lightening speed, with deep dives on topic

— Tracked attendance to gauge participation and higher engagement

— Google Meet polls to capture learning and how much participants understand topics in meeting with 72% reported

— Looked at qual and quant majors for metric of session value

  • Quotes from each session to show valuable series was to people in their own words

— After every event and sent summaries to cross-functional leadership team to tie impact to DesignOps

  • Great response from Chief Design Officer

— Just one example of one category to show value, but imagine if you went for ways to track impact across categories

— Practical starting point for relevant metrics and broader view of DesignOps value

— Right time to use this is when design lacks clear understanding of own success metrics, or inspiration

— Regardless of framework will get lot of new measure for DesignOps teams

  • Easy to say yes, but get past measure all things, but things that matter

— Where value factors come into play and measures most valuable for framework

— Many great prioritization efforts out there

  • Instead consider two factors
    • Organizational factors
    • Outcome driven factors

— Organizational factors: Measures related to organization in business

— Outcome Driven Factors: Sustained improvement over time

— Some factors are internal and include structure of design team

  • Prioritize those aligned with what team will achieve this year and what you will organize and deliver

— Measures to make design work visible to business

  • How centralized or embedded is team, which drives central outcomes

— Look at work streams to see what to adopt

— Other factors are external for design teams and huge impact on what DesignOps prioritize

  • Design teams are hiring and look at programs that support design at scale, like org satisfaction and retention
  • Contraction works with downturns, and more concerned about bottom line rather than end- experience
    • Prioritize outcomes that support business

— Inspired by Jared Spool, and measure to refine underlying outcome itself, and not just current success

  • Success: Any metric that quantifies DesignOps achievement like increasing event attendance
  • Progress: As effectiveness of DesignOps processes, and onboarding program
  • Problem-Value: Poor employee experience and operational gaps [inactive Figma licenses and work that reduces number of licenses]

— Operations value is defined by what is prioritized

— We feel narrowing outcomes will make durable and lasting

— Yes, this is outcomes, but endless justification positions DesignOps as up for debate,

  • Just do good work and that’s more important

— People will invest in what they can see directly as opposed to justification

  • Can’t expect stakeholders to invest and prioritize your team’s efforts without making your UX work visible at every level of an organization

— To borrow a quote, we choose to do measure value because it is hard

  • Prior frameworks don’t measure our progress accurately
  • So new frameworks are needed to accurately measure what’s going on and see bigger picture of value

— We have a whole book coming up on DesignOps value and and how to create value for designers and any program and product manager can order via link and can join AMA following day for conference

Q&A

  1. How would you see difference to JTBD versus OKRs from bigger strategy plan?
    1. OKRs come from business, while JTBD validate whether measures meet needs of people doing the jobs
      1. Impacting what job performers are actually trying to accomplish, and how you are moving the needle
      2. Mindset of job performer and not just business
  2. How to integrate frameworks with larger methodologies like Agile?
    1. Find that Agile, Scrum and different design methodologies are black box where work comes in and out, but if you focus on outcomes, methodologies should be compatible with dev practices
  3. Effectiveness is scored as highest HEROES element. How did you capture effectiveness in org?
  4. HEROES framework measurements also measure other things in org? Is that okay to do?
    1. Dual purpose of measurements is very valid
    2. Look at impact of design ops as broader than design organization
  5. Which measure should team-of-one prioritize while overloading overlap with what design management measures?
    1. Not one particular metric or category, but framework to figure out what’s most important to your organization and as team of one
    2. Often found going in as team-of-one effectiveness metrics are usually most foundational things and showing value in DesignOps
    3. Depends on org and what they do