QuickPanel: Launch a Healthcare Exchange with No User Testing?

The Wall Street Journal just informed us that “Groups Leading Insurance Sign-Ups Haven’t Tested Program’s Web Tool” (login required, sorry). The exchange sign-up process, expected to take as long as 45 minutes, would help U.S. citizens through complex issues, such as understanding their coverage options, deductibles, and prices. But it hasn’t even been tried by the community workers whose job it is to help citizens through the process.

Still, “a state Department of Health spokesman said… that everything was ‘on track’.” Good idea? We asked some of our UX experts to weigh in on the process:

Rosenfeld Media: Should state governments have delayed launching their health exchanges to perform user testing, even if it meant missing their launch deadline?

Steve Portigal It’s interesting that in some contexts we applaud moving quickly, launching in beta, and having real users give feedback to help refine the product. Granted, there’s more at stake here for the producers (e.g., the states) and the users than there would be (say) for an app that finds dog walkers in your area, but it’s intriguing how we reframe this as a clusterfuck already. Meanwhile, if there’s a “should” here, it’s that they should have developed a plan that included time to expose the exchange signup process—in whatever form—to real people.

If just one small piece fails, that can be enough to keep someone from succeeding.

Aarron Walter: Healthcare is complicated even for the experts who drafted the legislation, so it should come as no surprise that U.S. citizens with varying levels of proficiency with technology and the English language will struggle and fail to enroll in the new ACA system. Lives are quite literally at stake.  If healthcare is to be a fundamental right of Americans, then the system that provides it needs to be usable and accessible for all on day one of the launch.

Whitney Quesenbery Was there really “no usability testing”?  Or was there not enough testing, so that each piece of the system was strong?  I know that work on the health exchanges started long ago, including some early design research and templates.  But a design framework, even with a good interaction style guide, is just the start of a large complex system, and getting all those details right is a real challenge. If just one small piece fails, that can be enough to keep someone from succeeding.

Rosenfeld Media: Just like flubbed launches, delayed launches are embarrassing. Is there a sweet spot here? In other words, at what point is continuing user testing less valuable than launching?

If there’s a “should” here, it’s that they should have developed a plan that included time to expose the exchange signup process—in whatever form—to real people.

Steve Portigal: Context is everything. If the thing you are launching is on the front page of every paper and your vociferous opponents are willing to shut the government down rather than have you launch, then you probably want to be as flawless as possible. Launch a reduced experience, but make sure it works. If you are delivering a product for a small set of elite users, you have more forgiveness baked in. Backlash is a function of hype (see: iOS7).

Whitney Quesenbery:  I’d like to suggest a different question. I don’t think there was any way that the opening was going to be delayed—that just isn’t politically feasible in the current context.  Maybe the question is what could have been done to make the first 24 hours more of a success.

The real test of the usability of the system will be what happens after the opening day (or week) rush. First, they need to solve the technical problems. The best designed site is useless if you can’t sign on or if you hit bugs. Then, we’ll be able to see if people can understand how to set up an account, compare their options, and complete an application.

I agree with Steve’s comment above that a better process for getting everyone ready—especially the front-line health workers is the big “should.”  Demos, training, and support could all have helped everyone be more confident that they knew how the exchanges would work. It’s really hard to see something for the first time on the day that you’re supposed to be helping others.

Rosenfeld Media: If you were consulting for an exchange that had launched without user testing, what advice would you be giving them this minute?

Steve Portigal: Pay my invoice ASAP. Okay, more realistically, I’d be encouraging them to develop a triage plan. What are the elements of the experience (e.g., accuracy, completion, actionable) that are essential, and how can they, even anecdotally, begin to take the temperature of those measures and to develop a way to determine iterative changes and rapidly issue incremental releases.

If healthcare is to be a fundamental right of Americans, then the system that provides it needs to be usable and accessible for all on day one of the launch.

Whitney Quesenbery: I’d want the team out in the community health centers and listening in on the support lines, observing, listening, and learning. Make sure they know where the barriers are and understand exactly what problems people are having: Where is the process is confusing? Where people don’t understand the instructions or information? Where is the interaction counterintuitive? It’s an opportunity for doing usability testing in the field and seeing first-hand how the marketplaces are working. Those insights are critical to both prioritizing the work of fixing problems and making sure any changes are done right.  The worst thing they could do is flail around. Fix the most immediate and visible problems, and then keep working on the others in a careful way.

Like what our experts had to say? Guess what: you can have them bring their brains to you. Steve Portigal, Whitney Quesenbery, and Aarron Walter are available for consulting and training through Rosenfeld Media.