Eye Movement-Based Interaction ”What You Look At Is What You Get ” (WYLAIWYG) : Eye Movement-Based Interaction ”What You Look At Is What You Get ” (WYLAIWYG) Aulikki Hyrskykari 19th January 2000 Tampere University Computer Human Interaction Group
Eye Movement-Based Interaction : Eye Movement-Based Interaction Eye on/in the interface (2)
Problems and research issues: Technological/HCI issues (2)
Processing the eye movement data (5)
Eye as a control device (2)
Command based gaze interaction (9)
Noncommand gaze interaction (3)
References
Project ideas Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Eye on/in the interface : Goal to increase the bandwidth across the channel
Eyes are extremely rapid
Target acquisition usually requires the user to look at the target first before actuating the cursor control Constrained interface
between two powerful information processors . Eye on/in the interface Eye on interface (1/2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Eye on/in the interface : Eye on/in the interface Need for keeping the hands free (or the hands can not be used for other reasons)
Increasing number of computer users suffer from RSI (repetitive stress injury)
Eye movements are natural, little conscious effort
Direction of gaze implicitly indicates the focus of attention
Eye on interface (2/2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Problems and Research Issues : Problems and Research Issues 1) Technological issues
Usability of the hardware
head mounted systems more reliable but somewhat awkward
floor mounted more comfortable but more constrained
Accuracy - need of calibration
for every user at the beginning of a task
also during the task
Costs of eye tracking (equipment)
Eye on interface (2) Problems and research issues (1/2) EM data (5) Eyes in contro l(2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Problems and Research Issues : Problems and Research Issues 2) HCI issues
Need to design and study new interaction techniques
Eyes are a perceptual device, not evolved to a control organ
people are not used to operate things by simply looking at them - if poorly done it could be very annoying
Noisy data - need to refine in order to get useful dialogue information (fixations, input tokens, intentions)
accuracy restricted by biological characteristics of the eye
Eye on interface (2) Problems and research issues (2/2) EM data (5) Eyes in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Processing the EM data : Processing the EM data
Scanpaths (3 min of raw eye data) when a subject was asked to answer to three different questions concerning the painting [Yarbus67]
The data contains jittery, errors (originating from the limited accuracy), failures of tracking …
At the lowest level the raw eye position data must be filtered and the fixations identified
When analyzing the eye movement data off-line the noisy data can be refined using different filtering algorithms before counting the fixations, in real time the analysis must be more simple Eye on interface (2) Problems and research issues (2) EM data (1/5) Eyes in control (2) Command based gaze interaction(9) Noncommand gaze interaction (9)
Slide8 : A simple algorithm for identifying the fixations in real-time [Siebert00]: 1) Fixation starts when the eye position stays within 0.5o > 100 ms (spatial and temporal thresholds filter the jitter)
2) Fixation continues as long as the position stays within 1o
3) 200 ms failures to track the eye does not terminate the fixation Eye position X-coordinates (~3 secs) time X 2/5 Processing the EM data Filtering the noisy data 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (2/5) Eyes in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Processing the EM data Scanpaths with fixations identified : Visualization of scanpaths:
circles are the fixations (center is the point of gaze during the fixation)
radius depicts the length of the fixation
lines are the saccades between fixations 3/5 Processing the EM data Scanpaths with fixations identified 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (3/5) Eyes in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Processing the EM data (4/5) Input tokens : Processing the EM data (4/5) Input tokens [Siebert00]
The fixations are then turned into input tokens
start of fixation
continuation of fixation (every 50 ms)
end of fixation
failure to locate eye position
entering monitored regions
The tokens formulate eye events
are multiplexed into the event queue stream with other input events
The eye events also carry information of the fixated screen object (using nearest neighbor approach) 4/5 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (4/5) Eyes in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Processing the EM data Deducing user’s intentions : Processing the EM data Deducing user’s intentions Objective
to refine the data further on for recognizing the user’s intentions
to implement a higher level programming interface for gaze aware applications
Eye Interpretation Engine, objective to identify such behaviors as [Edwards98]
the user is reading
just “looking around”
starts and stops searching for an object (e.g. a button)
wants to select an object
04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5/5) Eye in control (2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Eye as a control device : Eye as a control device Gaze behavior very different from other devices used for controlling computer (hand, voice, feet)
intentional control of eyes is difficult and stressful, the gaze is easily driven by external events
precise control of eyes difficult
“Midas touch” problem
Most of the time the eyes are used for obtaining information with no intent to initiate commands
Users are easily afraid of looking at the “eye active” objects or areas of the window Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (1/2) Command based gaze interaction(9) Noncommand gaze interaction (2)
Eye as a control device Jacob’s taxonomy : Eye as a control device Jacob’s taxonomy Jacob’s taxonomy of possible approaches for using gaze input in the user interface: Unnatural response Natural response Unnatural (learned) eye movement Natural eye movement A. Command based interfaces B. Noncommand interfaces C. Virtual environments 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2/2) Command based gaze interaction(9) Noncommand gaze interaction (3)
Command based gaze interaction : Even though eye movements are an old research area gaze aware applications practically do not exist
Exception: applications for disabled Command based gaze interaction Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(1/9) Noncommand gaze interaction (2) © Erica, Inc. http://www.ericainc.com
Command based gaze interaction Applications for disabled : Command based gaze interaction Applications for disabled 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(2/9) Noncommand gaze interaction (3) ©LC Technologies, Inc. http.//www.lctinc.com/doc/ecs.html
Command based gaze interaction Applications for disabled : Command based gaze interaction Applications for disabled 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(3/9) Noncommand gaze interaction (3) ©LC Technologies, Inc. http.//www.lctinc.com/doc/ecs.html
Command based gaze interaction Applications for disabled : 04/11/2007 Command based gaze interaction Applications for disabled Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(4/9) Noncommand gaze interaction (3) ©LC Technologies, Inc. http.//www.lctinc.com/doc/ecs.html
Command based gaze interaction Selection : Most obvious task is selection of an object
Midas touch problem must be resolved, different solutions, we may use
dwell time
screen buttons
eye movement (e.g. wink) for selection
hardware buttons (e.g. space bar, or mouse)
for performing the selection in the position of gaze
Experiments have proven that gaze selection is faster than mouse selection [Ware87, Jacob94, Jacob99]
Accuracy problem - target objects must not be small Command based gaze interaction Selection 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(5/9) Noncommand gaze interaction (3) ©LC Technologies, Inc. http.//www.lctinc.com/doc/ecs.html
Command based gaze interaction Selection : In separate experiments [Ware87, Jacob98] on eye selection dwell time noted the most convenient
Dwell time >150 ms.
too long, sticky feeling (especially with expert users)
too short, wrong selections
Winks have been used for implementing selection for disabled users, who can not use additional control devices
04/11/2007 Command based gaze interaction Selection 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(6/9) Noncommand gaze interaction (3)
Command based gaze interaction Selection : Quick glance menu selection method [Ohno98] Screen buttons
EyeCon - visual feedback of the selection [Glenstrup95] 04/11/2007 Command based gaze interaction Selection 04/11/2007
faster than mouse
more errors than with mouse
lack of good canceling method Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(7/9) Noncommand gaze interaction (3)
Command based gaze interaction Selection : Two different approaches were experimented with
the cursor warps to every new object the user looks at (“liberal”)
the cursor does not warp until the user actuates the cursor (“conventional”) Target The cursor is warped to eye tracking position Gaze position reported by eye tracker Eye tracking boundary with 99% confidence will be within the circle Magic pointing [Zhai99]
Combined use of gaze and mouse
gaze is used to warp the cursor to the vicinity of the target object
threshold circle, in the circle the gaze does not affect the cursor
the fine adjustment is done by the mouse Conventional way was slightly slower than plain mouse selection, but the liberal way was faster than mouse
Test persons’ reactions positive 04/11/2007 Command based gaze interaction Selection 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(8/9) Noncommand gaze interaction (3)
Command based gaze interaction - menus, dragging, scrolling, window manipulation : [Jacob98]
Gaze controlled pull down menus
using dwell time did not work out very well, the time was either too long or too prone to errors
gaze+hardware button worked better
Dragging of objects (with gaze only, with gaze + hardware button)
performed better than most of the other experiments
using the gaze + hardware button felt natural
Command based gaze interaction - menus, dragging, scrolling, window manipulation 04/11/2007 Scrolling text in a window Listener window control 04/11/2007 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(9/9) Noncommand gaze interaction (2)
Noncommand gaze interfaces : Noncommand gaze interfaces
Multimodal interfaces head towards task-oriented (and user oriented) interfaces instead of command oriented
In non-command interfaces the computer monitors the user’s actions instead of waiting user’s commands [Nielsen93]
In most cases the natural eye movement information could be valuable information for the application
Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(10) Noncommand gaze interaction (1/3)
Noncommand gaze interfaces : iEye -project (University of Tampere, SMI/Germany, GiuntiIlabs/Italy, Conexor/Espoo and University of Nottingham/England, started in January 2000) Ship database example [Jacob98] [Siebert00] Little Prince -application [Staker90]
an example of “IES-media” (interest and emotion sensitive) Noncommand gaze interfaces Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(10) Noncommand gaze interaction (2/3)
Slide25 : 3D-displays exploiting eye position recognition Noncommand gaze interfaces (3/3) Eye position recognition 04/11/2007 Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(10) Noncommand gaze interaction (3/3)
References : References [Glenstrup95] Glenstrup Arne John , Engell-Nielsen Theo, Eye Controlled Media: Present and Future State. Published as a thesis at the University of Copenhage, Institute of Computer Science. Available in http://www.diku.dk/~panic/eyegaze/article.html
[Jacob98] R.J.K. Jacob, "The Use of Eye Movements in Human-Computer Interaction Techniques: What You Look At is What You Get. ACM Transactions on Information Systems, 9 (3), 152-169, 1991. Also reprinted with commentary in Readings in Intelligent User Interfaces, ed. M.T. Maybury and W. Wahlster, Morgan Kaufmann, San Francisco, 1998, 65-83.
[Nielsen93] Nielsen Jakob, Noncommand interfaces. CACM 36, 4, 1993, 83-99.
[Ohno98] Ohno Takehiko, Features of eye gaze interface for selection tasks. APCHI’98, Japan, 1998, 176-181.
[Siebert00] Siebert Linda E. and Jacob Robert J. K.,” Evaluation of Eye Gaze Interaction,” submitted to in the Proc. of ACM CHI 2000. (available in http://www.eecs.tufts.edu/~jacob/papers/chi00.sibert.pdf )
[Staker90] Staker India and Bolt Richard A., A gaze -responsive Self -Disclosing Display. ACM CHI’90, 3-9.
[Ware87] Ware Colin and Mikaelian Harutun H., An Evaluation of an Eye Tracker as Device for Computer Input. Proc. ACM CHI'87, 183-188.
[Yarbus67] Yarbus, A. L. (1967). Eye movements during perception of complex objects, in L. A. Riggs, ed., Eye Movements and Vision, Plenum Press, New York, chapter VII, 171-196.
[Zhai99] Zhai Shumin, Morimoto Carlos, and Ihde Steven, Manual and Gaze Input Cascaded (MAGIC) pointing. In Proc. of ACM CHI 1999. 246-253. Eye on interface (2) Problems and research issues (2) EM data (5) Eye in control (2) Command based gaze interaction(10) Noncommand gaze interaction (2)
Project ideas : Project ideas Combined programming and research projects
Comparison of fixation algorithms
Implementation and Evaluation of Magic Pointing
Evaluating (some) gaze controlled interaction components (implementation and tests)
Eye behavior when watching stereograms
Programming projects
Gaze assisted word processing (selection of text)
Gaze control in window management
Implementation of real-time gaze trail visualization and playback environment
Research projects
Gaze control in virtual environments
3D Vision