Day 3-Advanced Concept Testing Approaches to Guide Product Development and Business Decisions

— Thank you everyone, and I want to especially thank the Rosenfeld Media team


— Today, I’ll talk about an approach which can advance UXR: Concept testing
  • For some context, I have worked on both the client and agency side of research across multiple domains


— And I’ve been glad to hear about the integration of insights disciplines, as it’s been my personal experience as well
  • I’ve used qual and quant to support product development decisions
— Concept testing has been part of product development and in market research, but not in UX Research
  • So I’ll talk about it today in areas such as when to use concept testing, considerations, approaches, and the role of qual research


— Concept testing is testing ideas related to a product, configuration and features prior to product development, and communication messages prior to the product launch


— It’s used for marketing and content
  • The goal is to confirm there is need for a product idea
  • UXRs are more focused on user-experience after a product/feature idea approved
— This is use most useful before product development, to help prioritize what should be developed
  • If you don’t start with determining whether a product is needed or not, it will fail to align product with business goals and UX
  • Concept testing can help bridge gap between UXR and desired outcomes


— Now we need to understand the type of concepts we are dealing with. Difference concepts  are prioritized at different times
  • Products/Services: Describing specific product and feature functions. Or what does the product do? This determines if we should go forward with process of development
  • Positioning: Meant to look at product benefits and why someone should support or use, and can product deliver promised development
    • Meant to figure out how product will be differentiated in user’s mind from other products
    • Provides a richer vein of insights for user behavior, as people care about what features do for them
  • Advertising: How to communicate benefits of products and features to customers
    • Helps marketing teams create content as part of a larger study
    • Before expending resources in how though — we should grasp the why


— To see this come together, see this ad about attracting new customer from Quick Books
  • Quickbooks leveraged Danny DeVito to use the hurt from physical exercise to send the message that Quickbooks makes business easier
  • Through his visits, DeVito presented customers with supporting product features like expense tracking, capturing receipts, invoices and payments
  • A lot of market research went into creating commercial


— But when to use concept testing?
  • Its created when there is a decision to invest in development of new product or redesign or relaunch an existing product
— You also need evidence that the concept test is needed
  • For a new product launch we should consider why we need to monitor trend and imagine possible futures
  • For the redesign of a product category and redesign services to meet market conditions
  • Consider concept testing to ensure alignment with marketing needs


— So what to consider in a concept test:
  • Are you testing one concept or multiple ones?
  • What analytical approach are using?
  • What population is included in the study?
    • This frame influences data collection and business decisions regarding channels
  • Will the study be a branded versus unbranded blind study?
    • Branding of product can impact metrics
  • Whether is shows prices versus un-pricing, or whether the price is present or absent?
  • Whether internal or external party data will be used?
    • Risk of external norms not being specific enough


— We then have three concept testing categories
  • Monadic: Only one concept exposed to customers
  • Sequential: Multiple concepts shown in rotation to customers
  • Trade-Off:  Multiple concepts show side-by-side to customers


— Monadic concepts can cover as single product or feature
  • They are easy to implement
  • They include familiar metrics
— Drawbacks though include
  • They use rating questions, which have scale bias
  • There is no alternative to new ideas an audience is considering


— Sequential concepts are shown one at a time
  • Test products and features using a combination of texts and images and can be done within a survey tool
— Useful when we have access to a limited sample size
  • There is an inevitable interaction effect where influence of one scene influences the next scene
  • The approach relies on rating questions


— Trade-off techniques are powerful, and more so within concept testing
  • You see several concepts and are asked to make a choice
— MaxDiff Analysis contrasts the most and least important items in a concept, and asks you to make a choice
  • Works with comparing and prioritizing multiple items
  • Create idea to prioritize product feature back-log
  • Application can be anything you want to prioritize
— An example of this technique is looking at 30 features for Fitbit charge, seeing several types of screens, and ranking which one is most important, while controlling for order and interaction events


— Conjoint Analysis deals with combining product features or attributes together to present the concept to a user
  • Creates product configurations that are balanced to estimate the main effect and potential interactions on a choice
  • A user sees several screens and selections used to estimate impact level of each attribute and preference
    • A user can see the effect on samples and choice
— There are many variations  that each have their own pros and cons


— Pros for these trade-off techniques include
  • Since people are forced to select, they are easy to rate
  • There is no scale bias, as scales have different meanings in cultures
  • It lets us test concepts in an effective way
  • Main output is simulations for what-if -analysis to help us understand impact of products on preferences and likelihood for new products or services
— Cons for these trade-off techniques include
  • Cognitive burden and showing features/products
  • More specialized tools are needed to analyze results
    • Design analysis has a higher level of complexity


— All of these techniques are quantitative, but the goal to combine qual and quant techniques
  • Market research have used both to allow you to have better concepts and insights from analysis


— Qualitative Research benefits include:
  • It is exploratory and is seeing a range of behaviors, perceptions, and contexts of product use
  • It helps formulate hypotheses and capture language customers use
— I encourage you do to qualitative before quantitative research


— Cons of qualitative research include:
  • Unstructured data and small samples make it hard to project to large numbers of users
  • Good at discovering problems, but not the scale of the problems, i.e. how many people are actually affected


— Quantitative approaches have the following objectives:
  • It can help guide qualitative hypotheses and validate final concepts
  • It can find support for go/no-go decisions
— Quantitative techniques should be used after qualitative work


— Other techniques we can use outside of a controlled environment
  • A/B Testing and Web Analytics to see impact of new features
  • Digital Advertising campaigns via Facebook + Google
  • Crowd-Funding Platforms for new product s
— Pros of these methods
  • These measure actual behavior and can be effective for concept testing
— Cons of these methods
  • Same behavior can be motivated by different reasons, and hard to show insights into why a kind of behavior persists
  • Accounting for the impact of competitor actions
  • Channels lack key market information
  • It might not be good due to legal/confidentiality issues
— No method is perfect, but let’s select best the fit for problem at hand and tools we have
  • Some methods are better than others for each objective


— The goal today was to discuss concept testing for UXR, as we can get a lot from this methodology and doing this for business outcomes and making work more efficient


—Qualitative research can uncover insights. Concept testing can work to evaluate hypotheses
  • Concept testing can even begin with simple monadic tests
— As you see the value of the practice, use trade-off techniques for better insight and build the business case for value of concept testing
  • I was previously the manager of internal research team, and hybrid model worked best
  • Doing concept testing, have a clear idea of concept description and grasping what/why how to see what you are measuring



— Thank you for staying with me, and I’ll answer questions


  1. Can you elaborate on when to use monadic testing vs. multiple concept testing?
—> What are the goals: Sometimes you need to test more than one thing, while also having budget constraints
  • To save money people try to cram several concepts in one study
—> For monadic testing, you have the description where people need to pay attention and spend time on  one concept


—> Complexity of idea, timing, and team skills all determine concept choice


  1. How much training, if it any, does it take to do conjoint analysis?
—> It takes some experience and tools out there give impression that you can press a button and just do it
  • Judgment calls need to be made for it to work right


  1. Is the goal to differentiate product/features or identify what features to focus on?
—> You can do all of the above, and it depends on the research project. It can be a whole step in project, or multiple steps
  • It depends on the progress in product development


  1. How can you speed-up analysis methods?
—> There is an efficient way of doing concept testing and if you have more than 10 items that you want to check, MaxDiff works especially well, as it brings variations to large sets