Dario D. Salvucci

Mapping Eye Movements to Cognitive Processes Degree Type: Ph.D. in Computer Science
Advisor(s): John Anderson
Graduated: May 1999

Abstract:

Eye movements provide a rich and informative window into a person's thoughts and intentions. In recent years researchers have increasingly employed eye movements to study cognition in psychological experiments, to understand behavior in user interfaces, and even to control computers through eye-based input devices. Unfortunately, like speech and handwriting, eye movements generate vast amounts of data with significant individual variability and equipment noise. Thus, the analysis of eye-movement data -- that is, determining what people are thinking based on where they are looking -- can be extremely tedious and time-consuming. Typical eye-movement data sets are simply too large and complex to be analyzed by hand or by naive automated methods.

This thesis formalizes a new class of algorithms that provide fast and robust analysis of eye-movement data. Specifically, the thesis describes three novel algorithms for tracing eye movements -- mapping eye-movement protocols to the sequential predictions of a cognitive process model. Two algorithms, fixation tracing and point tracing, employ hidden Markov models to determine the best probabilistic interpretation of the data given the model. The third algorithm, target tracing, extends an existing tracing algorithm based on sequence matching to eye movements. The thesis also formalizes several algorithms for identifying fixations in raw eye-movement protocols and provides a working system, EyeTracer, that embodies the proposed tracing and fixation-identification algorithms.

To demonstrate the power of the proposed algorithms, the thesis applies them in three real-world domains: equation solving, reading, and eye typing. The equation-solving studies show how the algorithms can code, or interpret, eye-movement protocols as accurately as expert human coders in significantly less time. The studies also illustrate how the algorithms facilitate the prototyping and refinement of cognitive models. The reading study demonstrates how the algorithms help to evaluate and compare two existing computational models of reading and clear up temporal aspects of reading data using sequential aspects of the data. The eye-typing study shows how the algorithms can interpret eye movements in real time and help eliminate usability restrictions imposed by existing eye-based interfaces.

Thesis Committee:
John R. Anderson (Chair)
Herbert A. Simon
Kenneth R. Koedinger
Keith Rayner (UMass Amherst)

James Morris, Head, Computer Science Department
Raj Reddy Dean, School of Computer Science

Keywords:
Tracing, cognitive models, eye movements, hidden Markov models, user models, protocol analysis, sequential data analysis