Surveys That Work Blog

Posts written by Caroline Jarrett

  • Survey book of the month, August 2011

    Posted on

    If you love looking at wonderful information design, you’ll enjoy this book of the month:

    Looking back: a century of Dutch statistics by CBS Statistics Netherlands

    It’s in English and it’s free as a .pdf download, or you can pay a small fee for a printed copy.

    So far this year I’ve chosen survey books that are about information, either key concepts or practical how-to. This month’s book is about inspiration.

    (more…)

    Does your survey need a prenotice?

    Posted on

    Do you enjoy hunting around on the web for surveys to fill in? I do, but that’s because I’m researching this book and I believe that it’s rather unusual behavior.

    Most people need to be asked to complete a survey, so I talked about better invitations in my previous post.

    What about a prenotice? It tells your respondents in advance that a survey is on its way: a pre-invitation, perhaps.
    (more…)

    Better survey invitations and reminders

    Posted on

    Question: What’s the difference between a survey and a questionnaire?

    Answer: A survey is a process; a questionnaire is the series of questions and answers in the middle of that process.

    It’s easy to focus on the questionnaire to the exclusion of other important aspects of the survey. A couple of examples in my in-box recently reminded me about the importance of the survey invitation and the reminder.

    The untrustworthy invitation

    Here’s the top of a survey invitation that I received recently. I’ve anonymized the organization that sent it and the link to the survey itself.

    Suspicious-invitation

    For purposes of research, I tend to respond to every survey invitation I get if I possibly can, but this one shouted ‘spam’ at me. Why?

    • It’s not addressed to me. It’s addressed to #fullname#.
    • The incentive seems rather high for a survey. 25 euros (approx US $36) guaranteed? For 10 minutes?
    • And a bit further down, not shown in this screenshot, there was an address of a US corporation but the contact email address was in the Netherlands.

    I was hesitant.

    Response relies on trust, perceived effort and perceived reward

    Let’s look at it from the point of view of Dillman’s Social Exchange Theory. This says that people will respond to your survey if the perceived reward, which doesn’t have to be monetary, is in balance with the perceived effort – but only if they trust you. And if you’re exceptionally trustworthy, then being nice to you is itself a reward. Think about a mother, patiently answering a series of ‘why’ questions from a three-year-old; it’s likely that mother will keep answering until the perceived effort just becomes too much. (The most recent edition of Dillman’s book is Dillman, Smyth and Christian (2008), my January 2011 Survey Book of the Month).

    Strangely enough, if the reward is excessive for the perceived effort, then that undermines trust – exactly what happened to me in the example above. And the trust wasn’t great in the first place: that dodgy saluation and the mismatch of sender and email address also worked to undermine trust.

    The outcome? Well, in the interest of research I did click through – and the survey seemed perfectly genuine but I didn’t qualify because my business has fewer than 500 employees. I wonder what response rate they did get. Would those features that put me off have inspired trust in a busy HR manager is a large business? I suspect not.

    A reminder that looks like a reproach

    Then a couple of days later, a friend forwarded this survey reminder to me:

    Recently, we sent you an invitation to complete a Guest Satisfaction Survey concerning your stay with us at XXX YYYY, where you checked out on August 3, 2011.

    We noticed that you did not have time to complete the survey. We are concerned that you may not have responded because we have somehow failed to live up to your expectations.

    At XXXX, we are committed to providing a superior guest experience to every customer.

    Please take a few minutes to tell us how well we met your expectations.

    To complete the survey, please click on the web address below. If that does not work, please copy and paste the entire web address into the address field of your browser.

    http://survey.xxxxxx

    Thank you again for choosing XXXX. I look forward to hearing about your stay with us.

    Sincerely,

    Dave ZZZZ

    Manager,

    XXXXX Hotels

    ————
    Please do not ‘Reply’ directly to this invitation.

    Technical assistance: Should you have any problems accessing or completing this survey, please e-mail our survey vendor at hhctechsupport.xxxx.com

    To unsubscribe: We rely on feedback from guests to ensure that your hotel stay meets and exceeds your expectations. If, however, you prefer not to receive a survey invitation in the future, you may unsubscribe by clicking this link:
    http://survey.xxx

    For all other requests, please visit http://www.xxxx

    My friend forwarded it to me with the comment “How to be annoying. I didn’t know that I am somehow obligated”. Needless to say, she didn’t respond.

    Let’s look for the reward. Hmm. I can’t see one. What’s in it for my friend? It seems she’s missing out on an opportunity to complain. But she didn’t have a complaint about her stay – only about her survey reminder!

    Another way people can feel rewarded is if they’re made to feel special; if they’re picked out as one of a small group who is being asked to respond to an important survey that will influence a major business decision. So is there anything in this reminder to make her feel like an individual whose opinion is valued?

    No, rather the reverse. It says:

    At XXXX, we are committed to providing a superior guest experience to every customer.

    Maybe you read this sentence differently, but to me it has the effect of reminding her that she’s just one of the crowd, nothing special.

    And finally, look at these mixed messages:

    I look forward to hearing about your stay with us.

    Please do not ‘Reply’ directly to this invitation.

    He doesn’t really want to hear from her, because she’s not allowed to write to him. He only wants to increase the response rate on his survey.

    A survey invitation has three purposes

    Your survey invitation needs to do three things:

    1. Offer a preceived reward.
    2. Explain the amount of effort.
    3. Inspire trust.

    Offer a perceived reward

    People like to do nice things. If you can make your respondent feel
    special in some way, that can act as a reward. Obviously, a financial
    payment can work – but not if using that payment, or getting the
    payment, will be a huge hassle for the respondent.

    A delayed reward is definitely less effective than an immediate one – as I discussed in my post Do incentives help to improve response rates?

    Explain the amount of effort

    I have to admit that I’m still researching the question of explaining the amount of effort. I think two factors are operating:

    1. The possible benefit from saying how long a survey is, either in minutes or as a number of questions
    2. Widespread cynicism about the value of these promises.

    Personally, I’ve experienced questionnaires that promised to be ‘short’ but actually took me 30 minutes or more. It’s also tricky to predict how many questions a person will have to answer if your questionnaire has skips in it (places where you can routed round an inappropriate question).

    I’m hoping to find some data to tell us how long people expect a questionnaire to be, and whether that varies by topic or by the organization that’s asking.

    Features that inspire trust

    I’m still looking for research on the features of a survey invitation or reminder that inspire trust. Meanwhile, I think we can transfer ideas from research on features of web sites in general that inspire trust, such as the Stanford Guidelines for Web Credibility.

    For example:

    Design your site so it looks professional (or is appropriate for your purpose).

    If you’re doing a survey on behalf of an organization that’s known to your users, and I assume respected by them, then I interpret ‘look professional’ to mean: Make it look “designed” and make sure that it clearly communicates the brand.

    Here are two examples that I got this week. The two brands are John Lewis, middle-to-upmarket department stores, and Champneys, a spa. I’m a happy customer of both, regularly shopping with John Lewis (at the store and online), and occasionally treating myself to a visit to the spa. Here are mini versions of the home pages of their respective web sites.

     

    To my eyes, there’s a certain similiarity: both brands rely on a reasonably restrained use of a specific color palette (dark green/gray/while for John Lewis; dusky brown/light brown/gold for Champneys) to complement large, striking images.

    combined-jlewis-champneys

    Now let’s look at their two survey invitations, side by side. I’ve made them quite small so we can focus on the overall impression of the design.

    combined-jlewis-cham-invite

    Both of them look designed and professional, but I see the Champneys one as being much better at conveying the brand – and giving me the idea that filling in their survey might even be a pleasurable experience.

    Tips for better invitations and reminders

    Your user’s experience of your survey starts with the invitation. If you want to improve the response rate:

    • Look first at trust. Have you made it clear who you are? Is the person likely to know you, and if not have you made it easy for them to find out that you’re trustworthy? Does your invitation look as carefully considered and designed as the rest of your brand?
    • Then check perceived reward. Why would this person want to be nice to your organization in the context of this request?
    • Then check perceived effort. What will the respondents get in return? Is this all about you, or is there something in it for them?
    • And finally: Test! Test! Test! Make sure your testing includes the invitation and any reminders. Don’t let silly mistakes like sending your invitation to #fullname# ruin your survey.

    Using EEG in a usability test of a survey

    Posted on

    We all know that people will give up on surveys if they are too long or too boring. But exactly how long is too long? Exactly how boring is too boring?

    One of my most fascinating clients is Kantar, the market research and insight part of WPP. This is a giant business; Kantar conducts 77 million interviews a year, 34 million online. And that’s not 34 million people interacting with one single web site, it’s samples interacting with many, many questionnaires. My very conservative estimate is that there must be a minimum of 50,000 different questionnaires used each year.

    (more…)

    Ten tips for a better survey, UX Bristol

    Posted on

    Bristol is a lively city, the effective capital of the south west of the UK. Its wealth originally came from trade – rather unfortunately, considerably from the slave trade – but these days it’s mostly about aerospace, electronics and creative media.

    The Bristol Usability Group has been networking and holding evening meetings for a while now, and they decided to take the next big step and hold a one-day conference, UX Bristol.
    (more…)

    Survey that could be better; Radisson

    Posted on

    Have you ever had a survey that was sort of OK, but tripped you up with a few questions here and there?

    My friend and colleague Gerry Gaffney of Information and Design tweeted about a survey he had a few issues with and it seemed like a good time to start talking about some specific surveys and draw some general lessons from them, particularly as I was able to persuade him to write this guest post. Thanks, Gerry, and here we go.

    (more…)