Day 3- Improving Legacy Software: How Much Better Does It Have to Be?

— Hello, and welcome to my talk at improving legacy software
  • I’ve worked on legacy systems at Microsoft
— Our product teams often ask if users will like a design change to an existing piece of software
  • Will that change be better, worse, or the same for users?
— This talk outlines a set of ideas on how to get some answers

 

— Legacy software is bounded technological constraints, and it’s involves working with people to change engrained habits

 

— There’s the notion of Legacy Habits, with habits formed around software use that makes people be as efficient as possible at doing the work
  • People don’t want to change existing habits, even if the software is new

 

— There are two important ideas:
  • We need to know legacy habit paths related to software change
  • We need to know what is being measured across that change
— Let’s consider products you work on: what are the most ingrained habit paths
  • Can you point out where a feature change makes the experience better/worse/the same?

 

— But what happens when we learn how to do our work with software?
  • People form the necessary habits using the work software to get things done

— Think of the lines above as a simplified habit path

 

— During work people form habits
  • The complexity and frequency of the software workflow determines how quickly habit forms and how cemented it becomes
  • Some habit paths are non-optimal, and come from a sense of “This is what I know” and unwillingness to unlearn the existing pattern

 

— As people cement habit paths, they are difficult to unravel
  • Think of a doctor entering patient information into an electronic system
  • Or a business analyst pulling data from a data lake and creating dashboard, in a particular way
— Many paths are ingrained across every day tasks
  • Once a habit sets in, we get good at doing the work

 

— But companies must innovate and change, in various ways:
  • They must acquire new users
  • The must move fast to get ahead of competition
  • Bad product experiences need to be fixed

 

— We get at a high-level that companies need to do these things, but along the way their efforts can collide with broken, but established, habit paths
  • This weight of cognitive overload can result in a really bad day for our users
— So let’s respect legacy habits

 

— What can happen if legacy habits are not respected?
  • The proposed changes in software interrupt workflows and frustration ensues
— But let’s get concrete about what this means

 

— The habit path in the Windows Operating Systems is 27 years old
  • It’s a simple path, but great for demonstration of how a user sets up a habit path

 

— For example, let’s consider the setting habit path, which requires a user to click on a start button,  seeing a control panel, and clicking on settings
  • Imagine someone doing this every time they enter a conference room
— Windows 95, had a simple habit path line of accessing these settings

 

— The same path existed for Windows 98

 

— And in Windows 2000

 

— Then came Windows XP, which had the same entrance to the start menu, but no label for settings, which broke a five year habit path

 

— The new habit path for Windows Vista and Windows 7 was the roughly same as the path for Windows XP

 

— Then came Windows 8 in 2012
  • From 1995 to 2012 there was basic pattern
    • The start button and menu existed in bottom section of screen, and from there you accessed the control panel
  • Windows 8 broke a 15-year habit path
    • The control panel was not located where it once was

 

— Windows 8.1 had to the fix start menu problem by including the start button on bottom-left, and having two habit paths from start to settings
  • This was different from the settings habit path, and people spent the year with new settings from the side panel

 

 

— The Windows 10 software habit path leveraged prior habit paths, but broke with Windows 8

 

— Windows 11, broke with the Windows 10 path, by introducing a new version of the screen. And people had to grab onto a new path

 

— Nine habit paths have been introduced over the 27 year life of Windows so far
  • But how to measure the impact of the habit change though?

 

— How to measure the proposed changes:
  • Did the change make the user experience better?
  • Did the change change keep the experience the same?
  • Or did the change make the experience worse?

 

— We didn’t know which habits were formed by users over the years, so empirically it’s hard to see changes in user sentiment

 

— So let’s guess the changed and expected mood
  • First change was for the better, with few clicks removed
  • For the second set of changes, no change
  • The next two changes were worse, as they broke with the habit path
  • Settings in Windows 10 were better, as they had simple flow, but Windows 11 worse
— But how could we have detected this habit path?

 

— How could we have measured it?
  • Through looking at usability and detecting time on task
  • Examining cognitive load and seeing if people struggled with the change
  • Reviewing adoption speed and the number of repetitions to learn the habit
  • Viewing emotions to see facial expression
  • Attitudinal studies
  • Examining the context of use, such as having people enter conference rooms and practice setting up
— The key is to do this before/after analysis with whatever measure you need

 

— Now I will show a study to measure before/after for habit change

 

 

— This study deals with switching tabs, in a browser as they are tabbed in a browser bar
  • In this study, we wanted to see if people could stop using the task bar at the bottom and start using tab par at the top to toggle through Microsoft applications

 

— We understood people had used task bar to move to new applications
  • But people also had to switch to the tab bar for websites
— Could the habit be broken to switch from tasks to tab bar? Could people replace their old habit path with a new one?

 

— We demonstrated that people could make the switch from task bar to tab bar and number of switches that leveled off
  • People used the task bar and made more switches through tabs

 

— I hope this has gotten you thinking about the potential of the approach

 

— The goal is to identify and measure habit path in changes to legacy software

 

— But this comes with challenges

 

— First, we need a big sample to capture new/old habit paths

 

— Second, we know that screening right people for studies is important, and grasping existing user paths
  • Explore how many people take one path or another, or how long it takes to form one habit or another
— Third, we must grasp the pain from releasing old habit and adopting a new one
  • Motivation to change and self-efficacy
  • Measure impact of size or pane itself
— Fourth, context is king, and there is a huge difference between a simulated environment versus the real world.
  • The more complex the work, the more ecological validity counts
— Fifth, pathing telemetry should be added to the prototype to capture switching paths, and on software for habit paths
  • When joining events into a path, think of telemetry in systems
— Sixth, prototypes are key for capturing paths and differences between one and another
  • If the path is not too complex, building it into prototype will be helpful
  • Getting the right telemetry in place as new builds come out
  • Setting up the test environment in-house can be challenging
  • Hard to get data to triangulate behavior, attitude and emotion all at once, so be aware of what breaking habit path may do to people

 

— Not knowing how a feature change and software impacts habit can be disastrous
  • Measuring difference can capture quality of change

 

—Let’s design to respect habit paths, measure changes to capture habit paths, and monitor these changes before a new feature is released to the world

 

— I’m excited to see what you do regarding legacy habit paths

 

Q&A
  1. Are there certain habit paths that are common among industries? And if so, have industries every collectively initiated a habit change or are habit changes usually done by a single company?  
—> I think we should track these changes in habit paths as an industry
  • There is the idea of interaction pattens that come out at certain time
—> In terms of what we know about habit paths as an industry, I think we should try to figure out how to unify experiences across an industry
  1. Users may not follow every product release? How do you do comparisons for that situation? 
—> You should know the uptake of different updates, and weighing it based on the population
  • New update that doesn’t have super-fast adoption rates
  • Ask why and make a decision
    • Knowing habit paths and how to grasp differences between version 1 and version 5, will let you see how to serve needs
  1. How to trade off consistency with the past, with other consistencies, such as consistence with other tools or mobile?
—> This is a hard one, as we ran into this as we had version of Windows operating on mobile, with desktop, holo-lens
  • Knowing which audience you are reaching out to, and the habit patterns that are most familiar with them
  • Need to make sure habit carrier over to new habit content
  • Know the audience, habit paths, and probabilities of someone experiencing pain
  1. UX research is often too late and downstream for product decisions. How can we embed iterative design/prototyping so there’s less need to figure out impact? 
—> It’s always the biggest thing for UXR to be upstream
  • Be strong, and have strong voices to get in early
—> If not a possibility, figure out culture change and interactive studies
  • For every study team runs, have people collect habit path data
  • Build reporting system to capture habit paths, and that data over time will help you
  1. Would you say data collection is best served via wiki or database where people can dig into?
—> It would be great if more wiki like, as it is easily more consumable and searchable for most folks
  • But depends on how you structure data, and how people want to access it
  1. My hunch is that teams makes changes to their design to match current affordances. What is the Bbst way to see how entrenched habit is?
—> There are assumptions of everyone easily grasping an affordance, but this is not universally true
  • There is challenge of auto-save in Office web apps where some easily handled it, but others didn’t
  • Know the population, where there pain, and understanding the worst problem
    • It’s about caring for people and grasping where change will cause most pain

 

  1. I work on an enterprise team with small direct user base, and a lot of legacy software to retire. How do you think about habit formation with bespoke software?

 

—> It’s a better case to measure, as you can know the people and how they use the software to grasp what gets broken
  • Know the users. If you know who 20 people are you can grasp how they use the software the way they do
  • If you are making changes, can make changes in response to those habit changes
  1. How can we align legacy habit paths with new needs and mental models?
—> That challenge is hard, especially with acquiring new users from a different system
  • Say Microsoft wants more Mac users
  • We should be capturing deltas pattern in settings in Windows and Mac
    • Seeing where people make mistakes
    • Details will be super important to know the habit path