Search Analytics survey results

Posted on | Leave a comment

  • To help us (Lou Rosenfeld and Rich Wiggins) gather information for our forthcoming book on local site search analytics, we invited 206 people to complete a brief survey. The survey ran from June 12 to July 24, 2006, and received 134 responses. We asked respondents the following four questions:

    1. Where did you learn about analyzing queries produced by a site’s search system?
    2. We’re surprised at how few people and organizations analyze their own site’s search queries. If you agree, why do you think it’s so uncommon?
    3. Can you recommend any useful articles, book chapters, or books that are related to analyzing search queries?
    4. Can you recommend any useful web sites or other web-based resources that are related to analyzing search queries?

    Our main goal was pretty simple: we hoped to learn about the main sources of information on search analytics. With an admittedly leading question (#2), we also wanted to see if many people would challenge our assumption that search analytics is a rare pursuit (only 5 of 134 did) and learn what the barriers were to taking advantage of search analytics.

    We’ve summarized each question’s responses by listing the most frequent answers below. Full results are available by downloading this PDF file (182 Kb). Please address questions to lou (at) louisrosenfeld (dot) com.

    “Where did you learn about analyzing queries produced by a site’s search system?” (134 responses total)

    • 55 responses: Just did it myself / on the job (unspecified) / just common sense.
    • 11: From Lou Rosenfeld / At Argus / Information Architecture for the World Wide Web / Quiet User Study.
    • 7: From 2002 IBM research / working w / IBM market research & search tech. teams / at IBM in 1995, 2000-2 / at IBM’s UX team / from Jayee Hegde.
    • 7: From speaking with peers, colleagues.
    • 5: From Avi Rappoport
    • 5: Longer ago than I can remember.
    • 4: From articles read in library school 1994 / analyzing OPAC query logs / Information retrieval courses.
    • 4: From Rich Wiggins / Accidental Thesaurus / blog.
    • 3: From usability community blogs and articles.
    • 3: From working on Ultraseek / from Walter Underwood.
    • 3: From Martin Belam’s (currybet.net) work at BBC / At BBC.
    • 3: From 2003 IA Summit (forget presenter and university) / 2002 IA Summit / ASIST IA summits.

    “We’re surprised at how few people and organizations analyze their own site’s search queries. If you agree, why do you think it’s so uncommon?” (134 responses total)

    • 27 responses: Not having or making the time or budget to do the analysis.
    • 24: Lack of easy and popular tools.
    • 23: Tension or incompatibility between those interested in analyzing content (librarians, IAs, usability folks) and IT personnel who could deliver tools or data.
    • 19: Not knowing how illuminating or useful search queries can be.
    • 14: SA is intimidating or overwhelming or dull.
    • 11: Not knowing what or how to analyze / too many choices.
    • 11: Not knowing that search logs can be analyzed to make the search engine or web site more efficient.
    • 10: Not knowing it is possible to analyze search data; not knowing how much reporting is available.
    • 9: The risk of discovering things about your site you don’t like or want to know or will have to do.
    • 8: SA requires continuously funding a human being to do the analysis rather than a one-time tool purchase
    • 7: Not wanting to spend time proactively on search if it’s working.
    • 7: People don’t have information professionals around to educate them / general lack of expertise in the field.
    • 6: With a dynamic system like a search engine (quality differs based on every query), it is hard to gauge the ROI of doing the manual work of analyzing queries.
    • 6: The search system is hidden from the site, set up and then forgotten.
    • 5: Disagree
    • 5: Confusion between transaction logging and search logging; doing neither!

    “Can you recommend any useful articles, book chapters, or books that are related to analyzing search queries?” (115 responses total)

    • 37 responses: No, I wish I could!
    • 10: A Guide to Analyzing and Optimizing Website Search Engines, by Hurol Inan
    • 5: Spink, A., & Jansen, B. J. (2004). Web Search: Public Searching of the Web. Berlin: Springer. / Other papers by Spink and Jansen / Papers by Spink.
    • 4: Web Analytics Demystified, by Eric Peterson / Web Site Measurement Hacks, Hacks #64 & 65 and The Big Book of Key Performance Indicators, by Eric T. Peterson.
    • 4: Martin Belam’s blog: www.currybet.net/articles/audiences/index.php, www.currybet.net/articles/day_in_the_life/index.php, www.currybet.net/articles/dalek/, and www.currybet.net/articles/fine_tune/index.php
    • 4: Chapter in Information Architecture for the World Wide Web (Louis Rosenfeld, Peter Morville).
    • 3: James Robertson of Step Two Design: www.steptwo.com.au/products/search/index.html, www.steptwo.com.au/papers/kmc_fixingsearch/index.html, and Step Two Designs report on improving intranet search.
    • 3: Avi Rappoport’s PowerPoint presentation “Search tools for web sites and intranets.” www.searchtools.com/
    • 2:Web Metrics, by Jim Sterne
    • 2: There is very little literature out there related to this kind of thing. / It’s still a mix between linguistics and statistics / math. Clustering of queries, disambiguation, content matching etc.
    • 2: SEO books. / There are no good books. Most Search Engine Optimization is marketing based rather than information needs based work.
    • 2: Jared Spool. “Why On-Site Searching Stinks.” www.uie.com/articles/search_stinks/ and “Getting Them to What They Want” by Jared Spool and Erik Ojakaar.
    • 2: Ashley Friedlein Maintaining and Evolving Successful Websites

    “Can you recommend any useful web sites or other web-based resources that are related to analyzing search queries?” (111 responses total)

    Miscellaneous responses

    • A number of respondents mentioned that they read through query terms and draw high-level conclusions, or do some very rudimentary programming or scripting to get at what they want.
    • Does ‘site search’ include intranets?
    • A few things that may help change this trend (of not doing SA) are (1) a processing system that groups concepts / ideas to give marketing a feel for generally what is popular by subject area and (2) demonstrated ROI on the use of this data to better target content and services online (3) an increasing focus on the ‘science’ of marketing and the importance of data in the works for the past 5 years or so now.
    • The biggest lesson learned (in implementing an analytic tool) was that without organizing the enterprises’ content, an analytic tool added little value. The results for search simply returned everything and most documentation was outdated with no version control on working documents. It also illustrated the point that finding information online was extremely difficult and taxing for the end user.
    • Liked your discussion of using the term search analytics – and based on poor quality of early search log analysis, I’d go with the newer term and hopefully better integration and understanding of real search behavior.
    • I know more individuals who do it (SA) out of curiosity than organizations who do it for business reasons.
    • I think it’s a natural extension if you’ve ever done any SEO and analyzed how the public is getting to your site and what they are most commonly searching for.
    • Through my frustrating experience with search development, I became interested in IA and eventually, targeted content delivery. Now I think that the ideal state for enterprise search is to have content come to the user based on attributes such as role, organization, task, and other tags that tell us about the individual and what they are doing.
    • I think that search optimization efforts would need to use this (SA) data and create a process in which the task is fully supported to help us understand how people think about information and navigation, but perhaps it may also help us understand a user’s level of patience with results.
    • More firms are outsourcing some of the actionable aspects of their internet presence with regard to what actions should be taken based on web analytics data. For example, PPC search campaign management, keyword discovery, landing page testing and site architecture reviews.
    • For a long time people didn’t understand the importance of search, but now they expect Google to have everything covered anyway. There has been a big chunk of education missed in the middle there.
    • The SES Latino show in Miami last week showed how the Hispanic market was less likely to use such data.
    • I was building a website for a client who was using an internal search engine that allowed him to set meta tags on individual pages to control the results. But he was not allowing standard spidered results, ONLY those manually chosen via the hand-entered meta tags. I convinced him to follow the hand-manipulated results by spidered results, so that searchers weren’t penalized if they put in terms that weren’t in the meta tags. After this I started looking at the internal search queries, and using them in our keyword choices.

    Leave a Reply