{"id":192625,"date":"2023-07-17T20:51:08","date_gmt":"2023-07-17T20:51:08","guid":{"rendered":"https:\/\/staging.rm.gfolkdev.net\/?page_id=192625"},"modified":"2023-07-17T20:52:15","modified_gmt":"2023-07-17T20:52:15","slug":"sample-chapter-surveys-that-work","status":"publish","type":"page","link":"https:\/\/rosenfeldmedia.com\/sample-chapter-surveys-that-work\/","title":{"rendered":"Sample Chapter: Surveys That Work"},"content":{"rendered":"<p>This is a sample chapter from <a title=\"Caroline Jarrett\" href=\"https:\/\/staging.rm.gfolkdev.net\/people\/caroline-jarrett\/\">Caroline Jarrett<\/a>&#8216;s book\u00a0<em>Surveys That Work: A Practical Guide for Designing and Running Better Surveys<\/em>. 2021, Rosenfeld Media.<\/p>\n<h3>Chapter 1: Goals: Establish Your Goals for the Survey<\/h3>\n<p>In this chapter, you\u2019re going to think about the reason why you\u2019re doing the survey (Figure 1.1).<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192636 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.01-1024x980.jpg\" alt=\"\" width=\"960\" height=\"919\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-1024x980.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-300x287.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-768x735.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-1536x1471.jpg 1536w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-2048x1961.jpg 2048w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-180x172.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-320x306.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-480x460.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.01-600x574.jpg 600w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.1<\/strong><br \/>\n<em>It\u2019s easier to hit a target if you know which one you\u2019re aiming for.<\/em><\/p>\n<p>By the end of the chapter, you\u2019ll have turned the list of possible questions into a smaller set of questions that you need answers to.<\/p>\n<h3>Write down all your questions<\/h3>\n<p>I\u2019m going to talk about two sorts of questions for a moment:<\/p>\n<ul class=\"list-ul\">\n<li>Research questions<\/li>\n<li>Questions that you put into the questionnaire<\/li>\n<\/ul>\n<p>Research questions are the topics that you want to find out about. At this stage, they may be very precise (\u201cWhat is the resident<br \/>\npopulation of the U.S. on 1st April in the years of the U.S. Decennial Census?\u201d) or very vague (\u201cWhat can we find out about people who purchase yogurt?\u201d).<\/p>\n<p>Questions that go into the questionnaire are different; they are the ones that you\u2019ll write when you get to Chapter 3, \u201cQuestions.\u201d<\/p>\n<p>Now that I\u2019ve said that\u2014don\u2019t worry about it. At this point, you ought to have neatly defined research questions, but my experience is that I usually have a mush of draft questions, topic titles, and ideas (good and bad).<\/p>\n<p>Write down all the questions. Variety is good. Duplicates are OK.<\/p>\n<h3>Give your subconscious a chance<\/h3>\n<p>If you\u2019re working on your own, or you have the primary responsibility for the survey in a team, then try to take a decent break between two sessions of writing down questions. A night\u2019s sleep gives your subconscious a chance to work out what you really want to find out. If that isn\u2019t practical, then maybe try a walk in the fresh air, a break to chat with a friend, or anything else that might provide a pause.<\/p>\n<h3>Get plenty of suggestions for questions<\/h3>\n<p>If you\u2019re working with a team or you\u2019re in an organization, then often when word gets out that there\u2019s a survey ahead, colleagues will pile in with all sorts of suggestions for their questions. This can feel a little overwhelming at first, but it\u2019s best to encourage everyone to contribute their potential questions as early as possible so that you can carefully evaluate all of them, focus on some goals for this specific survey, and have a good selection of other questions available for follow-up surveys and other research.<\/p>\n<p>If I\u2019m too restrictive at the very beginning, I find that everyone tries to sneak just one little extra essential question into the questionnaire a day\u2014or even an hour\u2014before the fieldwork starts. By then, it is too late to test the little extra questions properly, and they could sink my whole survey.<\/p>\n<p>But while you\u2019re still establishing the goals for the survey? Great! Collect as many questions as possible. Encourage everyone to join in\u2014colleagues, stakeholders, managers, whoever you think might be interested. If you\u2019re running a workshop, give the introverts some space by having a bit of silent writing where everyone captures their individual question ideas by writing them down.<\/p>\n<p>Create a nice big spreadsheet of all the suggestions, a pile of sticky notes, or whatever idea-gathering tool works for you.<\/p>\n<p>Ideally, make it clear that there\u2019s a cutoff: suggestions before a particular date will get considered for this survey; miss the date, and they\u2019ll be deferred until the next opportunity. This helps to encourage the idea of many Light Touch Surveys.<\/p>\n<h3>Challenge your question ideas<\/h3>\n<p>When you\u2019ve gathered or created question ideas, it\u2019s time to confront them with these four detailed challenges in Figure 1.2:<\/p>\n<ul class=\"list-ul\">\n<li>What do you want to know?<\/li>\n<li>Why do you want to know?<\/li>\n<li>What decision will you make based on the answers?<\/li>\n<li>What number do you need to make the decision?<\/li>\n<\/ul>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192640 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.02-1024x576.jpg\" alt=\"\" width=\"960\" height=\"540\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-1024x576.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-300x169.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-768x432.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-1536x864.jpg 1536w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-2048x1152.jpg 2048w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-180x101.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-320x180.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-480x270.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.02-600x338.jpg 600w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.2<\/strong><br \/>\n<em>What decisions will you make based on the answers?<\/em><\/p>\n<h4><em>Ask: What do you want to know?<\/em><\/h4>\n<p>Surprisingly, I find that the question suggestions that I create or collect from colleagues often do not relate to what we want to know. Many times, I\u2019ve challenged a question by saying, \u201cOK, so you\u2019re thinking about &lt;xxx question&gt;. What do you want to know?\u201d and it turns out that there\u2019s a gap between the question and the reason for asking it.<\/p>\n<p>Probably the most common example is the question: \u201cAre you satisfied?\u201d The question is OK but very general.<\/p>\n<h4><em>Ask: Why do you want to know?<\/em><\/h4>\n<p>I\u2019m usually working with someone else when I\u2019m doing a survey. To help narrow down from \u201devery possible suggestion\u201d to a sensible set of goals for the survey, I ask \u201cWhy do you want to know the answers to these questions?\u201d and we then go on to challenge ourselves with the three questions in Figure 1.2.<\/p>\n<p>If I\u2019m on my own, then I find it helps to add \u201cthis time\u201d or \u201cright now\u201d\u2014to help me focus on the practical matter of getting my ideas down to something manageable. Come to think about it, that\u2019s not a bad idea for a team, too\u2014it helps all of them realize that they don\u2019t have to ask everyone everything all at once.<\/p>\n<p>Ask: What decision will you make based on the answers?<\/p>\n<p>If you\u2019re not going to make any decision, why are you doing the survey?<\/p>\n<p>Look very hard at each of the suggested questions and think about whether or not the answers to them will help you make a decision.<\/p>\n<p>Don\u2019t worry at this stage about the wording of the questions or whether people will want to answer them. You\u2019ll work on those topics in upcoming chapters.<\/p>\n<p>But if the answers to a question won\u2019t help you make a decision, set that question aside. Be bold! The question might be fascinating. You might be looking forward to reading the answers. But you\u2019re trying to focus really hard on making the smallest possible useful survey. You don\u2019t need to waste the question\u2014it can go into the possible suggestions for next time.<\/p>\n<p>At this point, you\u2019ll have some candidate questions where you know what decisions you\u2019ll make based on the answers.<\/p>\n<p>Ask: What number do you need to make the decision?<\/p>\n<p>In the opening chapter, \u201cDefinitions,\u201d I emphasized that a survey is a quantitative method and the result is a number. Sometimes you\u2019ll realize at this point that although you have candidate questions, you do not need numeric answers to them in order to make the decisions. That\u2019s fine, but it also means that a survey is probably not the right method for you. Your work so far will not be wasted because you can use it to prepare for a more appropriate method.<\/p>\n<h3>Choose the Most Crucial Question (MCQ)<\/h3>\n<p>If you were only allowed answers to one of your candidate questions, which would it be?<\/p>\n<p>That\u2019s your Most Crucial Question (MCQ).<\/p>\n<ul class=\"list-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"list-ul\">The<\/ul>\n<\/li>\n<\/ul>\n<p><strong>Most Crucial Question<\/strong><\/p>\n<ul class=\"list-ul\">is the one that makes a difference. It\u2019s the one that will provide essential data for decision-making.<\/ul>\n<p>You\u2019ll be able to state your question in these terms:<\/p>\n<ul class=\"list-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"list-ul\">We need to ask _______.<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"list-ul\">So that we can decide _______.<\/ul>\n<p>At this stage, don\u2019t worry if it\u2019s a Research Question (in your language, maybe even full of jargon) or the question that will go into the questionnaire (using words that are familiar to the people who will answer).<\/p>\n<h4>Test your goals: Attack your Most Crucial Question<\/h4>\n<p>Try attacking every word in your Most Crucial Question to find out what you really mean by it. Really hammer it.<\/p>\n<p>Here\u2019s an example: \u201cDo you like our magazine?\u201d<\/p>\n<ul class=\"list-ul\">\n<li>Who is \u201cyou\u201d? Purchaser, subscriber, reader, recommender, vendor, or someone else?<\/li>\n<li>What does \u201clike\u201d mean? Admire? Recommend? Plan to purchase? Actually purchase? Obsessively collect every edition? Give subscriptions as gifts?<\/li>\n<li>What do you mean by \u201cour\u201d? Us as a brand? A department? A team? As a supplier to someone else?<\/li>\n<li>What do you mean by \u201cmagazine\u201d? Every aspect of it? The paper edition? The online one? The Facebook page? The article they read most recently? Some parts, but not others? Does it matter if they\u2019ve read it or not?<\/li>\n<\/ul>\n<p>I found a great attack on a question by Annie Pettit, survey methodologist. She starts with the question:<\/p>\n<ul class=\"list-ul\">\u201cWhen was the last time you bought milk?\u201d<\/ul>\n<p>Here\u2019s how Annie attacks \u201cbought\u201d and \u201cmilk\u201d:<\/p>\n<ul class=\"list-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"list-ul\">Wait, do you care if the milk was purchased? Or could it be that we have an arrangement whereby we don\u2019t actually pay for milk? Perhaps people who live on a farm with dairy cows, or people who own a convenience store?<\/ul>\n<\/li>\n<\/ul>\n<p>Do you mean only cow milk? What about milk from goats, sheep, buffalo, camel, reindeer? Or what about milk-substitutes from nuts or plants like soy, almond, rice, and coconut that are labeled as milk? Were you really trying to figure out if we put a liquid on cereal? (Pettit, 2016)<\/p>\n<p>(And she added a whole lot more about topics, like whether or not chocolate milk counts.)<\/p>\n<h4>Decide on your defined group of people<\/h4>\n<p>When you\u2019ve really attacked your MCQ, look back and think about your \u201cdefined group of people\u201d\u2014the ones who you want to answer. Add them to your statement like this: We need to ask (people who you want to answer). The question (MCQ goes here). So that we can decide (decision goes here).<\/p>\n<p>If your defined group of people is still vague\u2014\u201ceveryone\u201d or something equally woolly\u2014then try attacking again. A strong definition of the group you want to answer at this point will help tremendously when you get to the next chapter, \u201cSample.\u201d<\/p>\n<p>But before you proceed to Chapter 2, let\u2019s pause for a moment and think about your plans.<\/p>\n<h3>Check that a survey is the right thing to do<\/h3>\n<p>Is your research question something that you must explore by asking people, or would it be better to observe them?<\/p>\n<p>Do you want to know \u201cwhy?\u201d\u2014qualitative\u2014or \u201chow many?\u201d\u2014quantitative?<\/p>\n<p>Let\u2019s look at this definition again:<\/p>\n<ul class=\"list-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"list-ul\">&gt;A<\/ul>\n<\/li>\n<\/ul>\n<p><strong>survey<\/strong><\/p>\n<ul class=\"list-ul\">is a process of asking questions that are answered by a sample of a defined group of people to get numbers that you can use to make decisions.<\/ul>\n<p>I\u2019m going to contrast that with this definition:<\/p>\n<ul class=\"list-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"list-ul\">An<\/ul>\n<\/li>\n<\/ul>\n<p><strong>interview<\/strong><\/p>\n<ul class=\"list-ul\">is a conversation where an interviewer asks questions that are answered by one person to get answers that help to understand that person\u2019s point of view, opinions, and motivations.<\/ul>\n<p>Both of them rely on asking: the interview is about \u201cwhy\u201d\u2014 qualitative\u2014and the survey is about \u201chow many\u201d\u2014quantitative, as in Figure 1.3.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192739 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.03-1024x511.jpg\" alt=\"\" width=\"960\" height=\"479\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-1024x511.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-300x150.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-768x383.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-180x90.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-320x160.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-480x240.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03-600x299.jpg 600w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.03.jpg 1088w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.3<\/strong><br \/>\n<em>Contrasting interviews as qualitative and surveys as quantitative.<\/em><\/p>\n<h3>Must your MCQ be answered by people?<\/h3>\n<p>One of my favorite questions was on a printer manufacturer\u2019s survey:<\/p>\n<ul class=\"list-ul\">\u201cHow many pages do you print in a month?\u201d<\/ul>\n<p>I had no idea. I knew the answer was more than one and less than a full box of paper because I hadn\u2019t bought a box of paper that month\u2014but I didn\u2019t feel sufficiently motivated to work out how many pages are in a full box. I guessed, wildly. Very poor data.<\/p>\n<p>The real irony, though, was that my printer was connected to their customer feedback program and was giving them the exact figure all the time: their analytics should have told them.<\/p>\n<p>Here\u2019s another example that arrived in my inbox recently:<\/p>\n<ul class=\"list-ul\">We need to ask visitors to our website whether pop-ups make them feel less like buying from us so that we can decide whether to remove pop-ups.<\/ul>\n<p>I\u2019m sure that client must have some good business reasons for using pop-ups that make them hesitate about removing them, but asking people whether they \u201cfeel like buying\u201d is a notoriously unreliable thing to do. They may feel like buying, but not actually buy, or feel unlike buying, but buy anyway. (We\u2019ll return to this topic in Chapter 3 when we look at the \u201cCurve of Prediction.\u201d)<\/p>\n<p>There\u2019s a much better quantitative method for questions like this: A\/B testing, where you publish two versions and use analytics to decide which one contributes better to the desired outcome. A\/B tests and the many other different types of analytics silently observe what people do without bothering them with questions. These are contrasted with surveys in Figure 1.4.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192740 size-full\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.04.jpg\" alt=\"\" width=\"728\" height=\"901\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.04.jpg 728w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.04-242x300.jpg 242w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.04-180x223.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.04-320x396.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.04-480x594.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.04-600x743.jpg 600w\" sizes=\"(max-width: 728px) 100vw, 728px\" \/><\/p>\n<p><strong>Figure 1.4<\/strong><br \/>\n<em>Analytics and A\/B tests are ways of observing how many people do something without asking them.<\/em><\/p>\n<h4>Do you want to find out \u201cwhy\u201d?<\/h4>\n<p>You may have spotted that we\u2019re sneaking up on the four-way matrix in Figure 1.5. The quadrant we haven\u2019t yet looked at is the top-left corner: observing to find out \u201cwhy.\u201d<\/p>\n<p>It\u2019s not always obvious why people are doing something. For example, if people tell you they can\u2019t find things on your website, then search log analytics will tell you what they are searching for\u2014but not why they are searching. Did they try searching straight away? Did they try a few clicks without success? Did they see your term for what they\u2019re searching for but not recognize it because they had something different in mind?<\/p>\n<p>Here\u2019s another MCQ that I see quite often:<\/p>\n<ul class=\"list-ul\">We need to ask visitors to our website the question: \u201cWhat do you dislike about our site?\u201d so that we can decide what to improve.<\/ul>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192741 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.05-1024x852.jpg\" alt=\"\" width=\"960\" height=\"799\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-1024x852.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-300x250.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-768x639.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-180x150.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-320x266.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-480x400.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05-600x499.jpg 600w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.05.jpg 1145w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.5<\/strong><br \/>\n<em>A matrix for choosing the right method.<\/em><\/p>\n<p>Leaving aside the problem that \u201cWhat do you dislike\u201d doesn\u2019t have a numeric answer, you\u2019ve got the more fundamental problem that there isn\u2019t a direct connection between \u201cWhat do you dislike\u201d and \u201cWhat should we improve?\u201d You need to know why people dislike something in order to get ideas about how to change it.<\/p>\n<p>You might turn to interviews, but it\u2019s unreasonable to expect most people to retain all the little details that made something easy or difficult. Observing them as they use the thing is much easier for them\u2014and much richer data for you.<\/p>\n<p>In a usability test, you can observe a participant who is tackling some tasks\u2014often in a research facility. Or you can go out to observe people in their natural setting, a field study.<\/p>\n<h4>Consider \u201cwhy\u201d alongside \u201chow many\u201d<\/h4>\n<p>A four-way matrix always makes it look as if the ideas are separate, doesn\u2019t it? Of course, in reality, the techniques complement each other.<\/p>\n<ul class=\"list-ul\">\n<li>The route in Figure 1.6 is one that I took around the matrix for a client recently.<\/li>\n<li>Analytics showed that sales of one product had dropped.<\/li>\n<li>Usability tests revealed that people thought the website was no longer maintained, so the product must also be out-of-date.<\/li>\n<li>Interviews at the same time revealed that people often left a long gap between deciding to buy the product and actually using it.<\/li>\n<li>A survey told us that the out-of-date problem was affecting more people than the wait-to-use problem.<\/li>\n<\/ul>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192742 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.06-1024x853.jpg\" alt=\"\" width=\"960\" height=\"800\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-1024x853.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-300x250.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-768x640.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-180x150.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-320x267.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-480x400.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06-600x500.jpg 600w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.06.jpg 1144w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.6<\/strong><br \/>\n<em>One of many possible routes around the matrix.<\/em><\/p>\n<p>I would love to encourage you to try some triangulation.<\/p>\n<p><strong>Triangulation<\/strong><\/p>\n<ul class=\"list-ul\">is when you use a mixture of research methods and compare the results to improve your overall insights.<\/ul>\n<h4>A draft presentation can help you decide between \u201cwhy\u201d and \u201chow many\u201d<\/h4>\n<p>A couple of years ago, I was chatting about surveys with user experience consultant Natalie Webb. Her tip was:<\/p>\n<ul class=\"list-ul\">\u201cCreate a draft of your presentation, based on the results you expect to get from your survey.\u201d<\/ul>\n<p>It seemed a strange idea to me at first, but the more I\u2019ve tried it, the more I like it as a way of testing whether I\u2019ve really thought enough about what I want to ask and whether the number that I will get as a result of my survey really will help me to make a decision\u2014the \u201cso what\u201d of surveys in Figure 1.7.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192746 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.07-1024x837.jpg\" alt=\"\" width=\"960\" height=\"785\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-1024x837.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-300x245.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-768x627.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-1536x1255.jpg 1536w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-2048x1673.jpg 2048w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-180x147.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-320x261.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-480x392.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.07-600x490.jpg 600w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.7<\/strong><br \/>\n<em>A draft presentation helps you to think about the \u201cso what?\u201d of your survey.<\/em><\/p>\n<p>I worried that by drafting the presentation first, I\u2019d be somehow constraining the direction of the research\u2014preventing my team from thinking freely about what they were doing, closing down what they might learn.<\/p>\n<p>Gradually, I realized that this is part of the power of surveys. Because you\u2019re finding out \u201chow many\u201d of something, you need to understand the \u201cwhy\u201d before you start. If you don\u2019t yet know enough about \u201cwhy,\u201d then you should be choosing to start with observation and interviews.<\/p>\n<h4>Think about what sort of number you need<\/h4>\n<p>Thinking about the \u201cso what\u201d and the number that you\u2019ll need for the decision you\u2019ll make also helps with another point to consider now: what sort of number do you need as your result? It may seem early, but statisticians will tell you that you must work out your statistical strategy before you collect the data, not afterward.<\/p>\n<p>Do you need to know the actual number of people who answer a question in a particular way? For example, when I helped with a survey about planning an office move, I wanted to know how many people said that when the office moved to the new location, their commute would become excessively long.<\/p>\n<p>Is it the proportion who answer one way rather than another? For example, I wanted to compare the proportion of people who claimed they would leave if the office moved to a new location to the proportion who said they would be likely to accept the change.<\/p>\n<p>Are you looking for a mean (the arithmetical average)? For example, I might have considered whether increasing the mean commute by more than an hour would kill the idea.<\/p>\n<p>Are you looking for a median (the value right in the middle when you place them all in order from largest to smallest)? Means can get easily distorted by one or two outlandishly large values. If one person\u2019s commute suddenly became nearly impossible\u201410 hours or more\u2014that would greatly increase the mean, but the median wouldn\u2019t be affected very much.<\/p>\n<p>And for design, I\u2019m often looking at ranges and modes. The range is the difference between the largest and the smallest values, so with a 10-hour commute and another commute that\u2019s zero because the person lived in an apartment above the possible new location, my range would be 10 hours. The mode is the most frequent value, and something that I find I have to consider very carefully for many design challenges\u2014both to design for the people who answered with the most frequent value and to make sure that I\u2019m not accidentally excluding people who don\u2019t fit \u201cthe norm\u201d for any reason.<\/p>\n<p>Or something else? You may be doing a comparative survey so you\u2019ll be considering what you want to compare from this survey to the next, or a modeling survey where you\u2019ll do all sorts of advanced statistical manipulations, or something quite different.<\/p>\n<p>Whatever you\u2019re planning to do with the answers to your survey, some careful thought at this stage about those statistics will be well worth the time you put into it\u2014and may send you back to have another review of your Most Crucial Question and how you plan to use it.<\/p>\n<h3>Determine the time you have and the help you need<\/h3>\n<p>So, you have a Most Crucial Question, you know the decision you\u2019ll make, and you\u2019ve thought a bit about the type of number you need to make that decision. It\u2019s a good moment to think about timing and who needs to be involved.<\/p>\n<p>First, think about the time available:<\/p>\n<ul class=\"list-ul\">\n<li>When do you need to have a result, and how much time can you put into it?<\/li>\n<li>If you\u2019re lucky enough to have team members to work with, how much time can they spare?<\/li>\n<li>When will you deliver the report from the survey?<\/li>\n<\/ul>\n<p>Next, think about the tools:<\/p>\n<ul class=\"list-ul\">\n<li>Do you or your organization already have a survey tool?<\/li>\n<li>Do you know how to use it?<\/li>\n<li>Will you need to buy or subscribe to one?<\/li>\n<\/ul>\n<p>Finally, and perhaps most importantly, who else is involved?<\/p>\n<ul class=\"list-ul\">\n<li>Who needs to be involved in the survey but isn\u2019t part of your team, such as the privacy or legal people?<\/li>\n<li>Who will get the results from the survey?<\/li>\n<li>Who is involved in making the decisions based on the results?<\/li>\n<\/ul>\n<h4>Interview first, survey later<\/h4>\n<p>A common mistake is to think that you\u2019ll do a survey first and then do follow-up interviews with some of the people who answer.<\/p>\n<p>The rule is: interview first, survey later. Two especially useful types of interviews are:<\/p>\n<ul class=\"list-ul\">\n<li>Interviews to find out what your defined group of people think about the topic of your survey (covered in Chapter 2)<\/li>\n<li>Cognitive interviews\u2014a special type of interview just for survey questions\u2014to help you discover whether the questions are working (Chapter 3)<\/li>\n<\/ul>\n<p>And, in fact, to get the best results from your survey, you\u2019ll complement these interviews with two other techniques from the matrix, aa noted in Figure 1.8:<\/p>\n<ul class=\"list-ul\">\n<li>Usability tests of the questionnaire (Chapter 4, \u201cQuestionnaire\u201d)<\/li>\n<li>A pilot test between the usability test and the survey itself (Chapter 5, \u201cFieldwork\u201d)<\/li>\n<\/ul>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192747 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.08-1024x853.jpg\" alt=\"\" width=\"960\" height=\"800\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-1024x853.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-300x250.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-768x640.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-180x150.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-320x267.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-480x400.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08-600x500.jpg 600w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.08.jpg 1144w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.8<\/strong><br \/>\n<em>We\u2019ll use techniques from other parts of the matrix on our way to the survey.<\/em><\/p>\n<p>If you want a couple of ideas for how to fit all those activities into the time you have available, then skip ahead to Chapter 8, \u201cThe Least You Can Do<sup>TM<\/sup>.\u201d A recent survey where I worked hard to get a single Most Crucial Question took me four days\u2014spread out over a month, admittedly, but only because I had a week\u2019s vacation in the middle.<\/p>\n<h3>What could possibly go wrong with the goals?<\/h3>\n<p>For many years, I was quite a purist about surveys. If you\u2019d asked me \u201cWhat can go wrong when choosing a goal for your survey?\u201d I\u2019d have answered, \u201cInsisting on doing a survey when it\u2019s the wrong method for the research problem.\u201d<\/p>\n<p>These days, I\u2019ve mellowed. I know that sometimes colleagues or clients will carry on with a survey for all sorts of reasons, good and bad, when it\u2019s not the ideal thing to do. If that\u2019s happening to you, don\u2019t worry. Keep making good choices, aim for a Light Touch Survey, and iterate as much as possible. No matter what the outcome is, you\u2019ll definitely learn a lot about how to do a better survey next time.<\/p>\n<p>Strictly between you and me, I\u2019ve also become more relaxed about some of the other aims of this chapter. Couldn\u2019t get down to exactly one Most Crucial Question? If you still have dozens of MCQs: definitely not. But five or six candidates for MCQ? Not so bad\u2014you can whittle them down when you start working on them in Chapter 3. Not entirely clear about the decision you\u2019ll make? Have a go, and revisit it when you\u2019ve done some more steps. You can iterate, after all.<\/p>\n<p>But I wouldn\u2019t often admit that to the team or the client because I know that when we can agree on one Most Crucial Question with a clear decision to be made, the rest of the survey process is going to be much easier and quicker. So I try pretty hard to persuade them to get there.<\/p>\n<h4>To be valid, the goals and questions must match<\/h4>\n<p>This brings me to the first of the challenges that you\u2019ll meet through the steps of the survey process. In this chapter, you\u2019ve been looking at the first tentacle of the Survey Octopus: \u201cThe reason you\u2019re doing<br \/>\nit,\u201d as shown in Figure 1.9.<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-192749 size-large\" src=\"https:\/\/staging.rm.gfolkdev.net\/wp-content\/uploads\/2023\/07\/Figure-01.09-1024x661.jpg\" alt=\"\" width=\"960\" height=\"620\" srcset=\"https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-1024x661.jpg 1024w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-300x194.jpg 300w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-768x496.jpg 768w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-180x116.jpg 180w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-320x207.jpg 320w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-480x310.jpg 480w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09-600x387.jpg 600w, https:\/\/rosenfeldmedia.com\/wp-content\/uploads\/2023\/07\/Figure-01.09.jpg 1258w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><\/p>\n<p><strong>Figure 1.9<\/strong><br \/>\n<em>Lack of validity.<\/em><\/p>\n<p>There\u2019s always an error between each tentacle and the next one. In this case, it\u2019s \u201clack of validity.\u201d<\/p>\n<p><strong>Lack of validity<\/strong><\/p>\n<ul class=\"list-ul\">happens when the questions you ask do not match the reason why you are doing the survey and what you want to ask about.<\/ul>\n<p>Or in other words:<\/p>\n<ul class=\"list-ul\">A survey is valid when the questions you ask are a good match to the reason why you are doing the survey and what you want to ask about.<\/ul>\n<p>So work really hard on the reason why you are doing it, the decision that you\u2019ll make, and that Most Crucial Question.<\/p>\n<h3>At this point, you will know<\/h3>\n<p>To have an easier ride with the next steps in the survey process, it helps a lot of at this point if you know:<\/p>\n<ul class=\"list-ul\">\n<li>The resources you have for the survey<\/li>\n<li>Who you want to answer your question\u2014your defined group of people<\/li>\n<li>The decision you\u2019ll make based on the results<\/li>\n<li>The Most Crucial Question to help you make the decision<\/li>\n<li>Whether the Most Crucial Question needs to be answered by people or not<\/li>\n<li>Whether a survey is the right thing to do<\/li>\n<\/ul>\n<p><a href=\"https:\/\/staging.rm.gfolkdev.net\/books\/surveys-that-work\/\"><em>Back to Surveys That Work<\/em><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a sample chapter from Caroline Jarrett&#8216;s book\u00a0Surveys That Work: A Practical Guide for Designing and Running Better Surveys. 2021, Rosenfeld Media. Chapter 1: Goals: Establish Your Goals for the Survey In this chapter, you\u2019re going to think about the reason why you\u2019re doing the survey (Figure 1.1). Figure 1.1 It\u2019s easier to hit &hellip; <a href=\"https:\/\/rosenfeldmedia.com\/sample-chapter-surveys-that-work\/\">Continued<\/a><\/p>\n","protected":false},"author":150108,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_seopress_robots_primary_cat":"","_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","footnotes":""},"acf":[],"_links":{"self":[{"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/pages\/192625"}],"collection":[{"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/users\/150108"}],"replies":[{"embeddable":true,"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/comments?post=192625"}],"version-history":[{"count":8,"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/pages\/192625\/revisions"}],"predecessor-version":[{"id":192754,"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/pages\/192625\/revisions\/192754"}],"wp:attachment":[{"href":"https:\/\/rosenfeldmedia.com\/wp-json\/wp\/v2\/media?parent=192625"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}