Great questions. Thanks, Caroline!
1. Our participants were looking for holiday gifts.
2. The simulation services’ websites don’t seem to mention anything about tasks (or other factors that may affect the applicability of their heatmaps).

I’d like to run a few more Web pages and full page ads through the simulations and compare them to the results from real eye tracking studies. Even though the companies claim high accuracy for websites, I believe their predictions will be better for ads.

Hi Aga,
It seems that the Tobii eye tracking results you used are only showing webpage information above the fold. And at the simulation services, you used a larger webpage image with information bellow the fold. You should crop the original webpage image and run the simulation services again.

Thanks, Alex! 3M VAS wasn't much better at predicting attention on the eBay homepage: Attention Heatmap Simulation by 3M VAS.

They claim that you can "test client-directed creative or usability decisions." Usability decisions? That's a bold statement. And their pricing is higher than their competitors'.

I liked the fact that they have a paper on their website that describes a validation study which compared their simulations to real eye tracking data. However, because scene photos and marketing material were used in the study and there were no tasks (participants were instructed to just look), the results can't be generalized to websites.

Thanks, Neil! Yes, eye movements are very task-dependent and many people bring up Yarbus' picture with the different scanpaths to demonstrate that.

What many people don't know is that there was only one person in Yarbus' study. What's more, that might have even been Yarbus himself! Regardless, there has been a lot of research since then to support task dependency of eye movements.

Some of the simulation services claim that the task doesn't matter in the initial few seconds of exposure to a visual. They say the task and other top-down processes kick in a little later. According to research, however, that's not exactly true.

Thanks for the extra credits, Rafael!

I think what we've established thus far is that because the simulation is task-independent, it may not be applicable to websites because people don't visit websites just to look at them. Instead, they typically have a (more or less specific) goal in mind.

I'm going to run some task-free stimuli and compare with my eye tracking data next week. So, stay tuned!

Thanks for sharing, Loic!

I didn't think this discussion could get any better but it just did! Thanks so much, Fabian, for all the additional information (and the extra credits, of course!).

I found the EyeQuant heatmap to be the most subtle/cautious in terms of its coloring. Because of that and because the blue shading covers a lot of areas that the other heatmaps leave blank, the EyeQuant visualization does look less inaccurate to me.

It's actually less of interest to me which simulation service is better. I'm more interested in defining the applicability of the simulations in general. It seems that many people either believe that the simulations can completely replace eye tracking or they consider the simulations to be a joke. I think there is a place for them but perhaps more on the marketing side than UX.

But the first step is to review the research that aims to determine how the simulations predicts real eye tracking data. Ideally, the research would be done with various types of stimuli (e.g., full-page ad, website) and in various contexts (e.g., specific vs. general task, stimulus-related vs. stimulus-unrelated task).

I really like the fact that you do your own research - please always feel free to email me any additional information if you can't post it here.

Dan, I could not have said it better myself. Thank you! :)

Thanks for your comment, Marc.

It is true that overt attention involves direct gaze, while covert attention is the ability to attend to something without actually looking at it. However, it is not true that overt attention depends on the task and covert attention on the stimulus characteristics. What you are referring to are top-down processes (task- and experience-dependent) and bottom-up processes (stimulus-dependent). These processes play a role in both attention types – covert and overt.

I also disagree with your claim that the simulation services are trying to simulate covert attention. Their websites indicate they are simulatingovert attention. For example:

• “Feng-GUI attention map reaches over 80% of AOI (Areas of interest) similarities to Eye and Mouse Tracking. […] We compare and measure the results with eye-tracking sessions.”
• "AttentionWizard simulates human vision during the first 5 seconds of exposure to visuals, and creates an eye-tracking heatmap based on an algorithm that predicts what a real human would be most likely to look at. […] AttentionWizard results are 75%+ correlated with eye tracking and mouse tracking approaches."
• “EyeQuant combines a decade of neuroscientific research in attention psychophysics with the most current computer models of human attention. The models are based on empirical Eye-Tracking data that has been gathered in numerous studies with over 400 human subjects.”
• “3M™ Visual Attention Service is a cutting-edge scanning tool that lets you scientifically analyze design effectiveness, based on how the average human eye responds.”

If you meant that the simulations can, for the most part, only simulate bottom-up processes, I agree with that completely.

Hi Aga,

Thanks for the post - very helpful. The follow-up discussion has been good too. A few quick points (some of them have been touched on already):

1) All algorithmic approaches focus primarily on the visual interest in the scene. When this is applied to websites, that means focusing on contrast, line orientation, areas of interesting detail, skin texture, faces, etc. Many sites include moving elements or animation, so of course static screenshots will not be able to take those into account. Motion draws huge amounts of attention.

2) In my opinion, comparisons of accuracy are not very helpful. Eye tracking is very heavily person-specific and task/intent dependent. Also, as discussed by others above, algorithms focus on initial first-impressions and not any kind of volitional actions (e.g. scrolling or reading text).

3) You need to find the right tool for the right job. Eye tracking is expensive and time consuming (equipment, recruiting of subjects, analysis of the data) but it yields very rich information. Mouse tracking is helpful too, but requires a live landing page with a decent amount of traffic to it. Algorithmic approaches are less accurate but give you instant results. They can also be used on mockups before you deploy a page live. This allows you to fine tune a design and eliminate the obvious visual attention leaks first. They are also relatiely cheap. You should be using all three approaches (sometimes in parallel depending on the circumstances).

Hope this helps. I don't have any free credits for you. But I hope that the one-cent trial for ten heatmaps will not deter you from signing up on http://AttentionWizard.com.

I'd like to chime in on #2: You really *have* to be able to measure accuracy by evaluating your tool against the Gold Standard of eye-tracking - and if your tool really works, why wouldn't you? :)

We at EyeQuant have published extensively on this topic via our associated labs at CalTech, USC and the University of Osnabrück - there's actually a very standardized way of assessing predictive performance (at least in the scientific community), here's how it works:

You've got a prediction and would like to know how well it actually predicts real behavior. First, you need a Gold Standard: how well does one eye-tracking study predict another one?

The way any serious lab will do this is to use a so-called "Receiver operating characteristic" or ROC. Bear with me, it's pretty straightforward:

In this context, a ROC curve (more specifically, the area under the ROC curve or AUC) can tell you how well one set of data (the prediction set) discriminates between fixated and non-fixated points in a second set of data (the eye-tracking set): if it's 0.5, your algorithm performs *at chance level* (not good, you're better off guessing yourself). The closer you get to 1, the better.

Now, eye-tracking itself achieves around 0.9 - 0.95 with 25 subjects predicting another 25. Obviously very good - that's why it's the Gold Standard. Also note how it interestingly doesn't get better when using more subjects!

EyeQuant achieves over 0.85 for websites - i. e., over 90% of the Gold Standard's performance.

The big difference to algorithmic approaches is that EyeQuant uses a data-driven model that is constantly fed with new eye-tracking data - the current model is based on data from over 500 subjects.

After all, developing technologies like this isn't a pet project but takes considerable resources and research. At EyeQuant we're glad to be working very closely with the three leading attention researchers in the world and we pride ourselves with laying extensive focus on the actual accuracy of our predictions.