Survey book of the month, February 2011

02/28/2011

This month, I’ve been working on my chapter on survey questions and revisiting my libary of books about questions. My favorite, easily, is:

“The Psychology of Survey Response” by Roger Tourangeau, Lance J. Rips and Kenneth Rasinski (2000) Cambridge University Press.

Let me take you through their main framework and some of my recent experiences with the book, and I hope that will explain why I like it so much.

The four components of the response process

The book is based around their analysis of the components of the mental processes that respondents go through as they answer a question. In Forms that work: Designing web forms for usability, I explained these steps as:

1. Read and understand the question
2. Find an answer
3. Judge whether the answer is one you want to give
4. Place the answer on the form

Tourangeau, Rips and Rasinski describe the four processes, and their sub-processes, like this:

Comprehension:

• Attend to questions and instructions
• Represent logical form of question
• Identify question focus (information sought)
• Link key terms to relevent concepts

Retrieval:

• Generate retrieval strategy and cues
• Retrieve specific, generic memories
• Fill in missing details

Judgment:

• Assess completeness and relevance of memory
• Draw inferences based on accessibility
• Integrate material retrieved
• Make estimate based on partial retrieval

Response:

• Map judgement onto response category
• Edit responses

An example of the dicussion: logical form of question

It’s a bit hard on them to present this rather complex list, full of technical terms, without their explanations – but I wanted you to realise that this is a really deep dive into questions. For example, when they are talking about the process “Represent logical form of question”. They investigate the grammar of the question, comparing easy pairs such as:

Iterrogative: “Do you have a home equity loan?”
Declarative: “You have a home equity loan”

with the more complex ‘who, what, where, when, why, how’ questions, such as analysing this question:
“When did Lydia tell Emily the telephone would be fixed?”

which has two possible interpretations of appropriate answers:

• The time when Lydia spoke to Emily
• The time when the telephone would be fixed.

This illustrates both the one of the challenges and one of the strengths of this book. You’ve got to be willing to be challenged by topics like sentence parsing, but the authors will help you along by providing lots of examples.

People are better at recalling recent, significant events

The biggest chunk of the book is devoted to ‘retrieval’, with a chapter on memory followed by three chapters investigating how people use their memories to answer three types of questions: dates and durations, factual and numerical estimates, and attitudes.

One of the crucial nuggets from these chapters: we do better at recalling events if they are recent and significant to us. Example: I know exactly what I did all day on Saturday 4th December 2010, but had to check my diary to remind myself about Thursday 3rd December 2010. Why? Because my cousin’s wonderful wedding was on the Saturday, whereas Thursday was just an ordinary working day rather like any other.

Why does that matter for us in user experience? Because we often want to know about routine events that are relatively insignificant to our users, such as browsing a familiar web site or making an everyday purchase. Big deal to the site’s designer; not such a big deal (maybe) to the user.

Judgment is about selecting an answer

The authors introduce the process of judgment in their chapter about attitudes, where they describe different models of the concept of attitude. Their preferred one includes a judgment phase, where “multiple considerations about the issue will come to mind, and the respondent will have to combine them to produce an overall judgment”.

They point out that these considerations may include:

• any existing evaluation of the issue
• new arguments and newly made judgments
• transient considerations such as social desirability.

They go on to explore these topics in greater detail in the chapter about attitudes and context effects: the way in which reported attitudes can apparently be changed by asking questions in slightly different ways or in different orders.

Response is about mapping judgments to survey answers

The final process of the four, response, is about presenting the answer in the format acceptable to the survey. In my terms: putting the answer on the form. They devote a chapter to this process, which includes a brief discussion of one of the most-researched topics in survey design: the number of points in a response scale

The final chapter about their four-element process looks at what they call ‘editing of responses’, by which they mean the way in which respondents make choices about what to reveal in their answers. You may be familiar with the concept of ‘social desirability bias’,: we like to portray ourselves as nice people and therefore tend to over-report behaviors and attitudes that we believe reflect well on us, and under-report ‘bad’ things.

A user experience example: telling someone that you don’t like their product might be seen as being an attack on them. So we find that users in usability testing struggle during their tasks, but then report satisfaction overall. Are they really satisfied, or are they editing their response to avoid appearing like nasty, attacking people?

Two technical chapters: on interview mode and cognitive models

The last two chapters move away from the main processes and look at ‘interview mode’ and ‘cognitive models’.

Interview mode is about how you deliver the survey: by telephone, mail, email, Internet or interviewer, and whether you use a computer or pen and paper. This is where the book shows its age a little: it was written in 2000, and the chapter needs to be read with that in mind.

Can I admit that I struggled a bit with the final chapter, on the application of cognitive models? It seems to be going into arguments for and against various threads in the academic discourse around surveys in the late 1990s, but as I’m not entirely familiar with those threads I found it a bit hard to follow.

So my suggestion: don’t worry too much about chapters 10 and 11. There’s more than enough in the previous nine chapters to justify reading this book.

Is satisfaction the same as happiness?

I’ve tried to give you a flavor of the big themes in the book, which I find fascinating. I also enjoy reading the finer details within those themes.

For example: One topic that’s particularly relevant to user experience: reported ‘satisfaction’. Is that different to, or the same as, ‘happiness’?

They explain that “respondents may assume that two very similar items convey different questions when they appear in the same part of the questionnaire” and go on to quote an experiment that “asked respondents to rate both their happiness and their satisfaction with their lives. In one condition, the questions – How happy are you…? and How satisfied are you…? – appeared next to each other in the same questionnaire. In a second condition, on of these questions appeared as the last item on an intial questionnaire, the second as the first item on a second questionnaire. In line with the prediction, ratings for the questions tended to be more similar when they appeared in different questionnaires than in the same one”.

So: for satisfaction and happiness in life in general, the concepts are about the same if you ask the question at different times, but different if you ask the question at the same time. (Should you wish to follow up the experiment, it’s described here: Semantic and pragmatic aspects of context effects in social and psychological research.Detail Only Available  Strack, Fritz; Schwarz, Norbert; Wänke, Michaela; Social Cognition, Vol 9(1), Spr, 1991).

Hmm. What does that mean for us in ux? I’m thinking: it seems likely that the same effect happens when you ask about satisfaction and happiness for a product or experience. If you need to distinguish between them, you’ve got to ask about both at the same time.

Overall: a book that’s academic but intriguing

Can you see why I enjoy this book so much? I find I keep reading a bit of it, start to think about the implications, want to follow up a reference – and next thing I know, an hour has gone by.

I won’t pretend it’s an easy read, but it does repay the effort.