Hima EkanadhamArial

An Illusory Illusion?The Role of Spatial Attention in the McGurk EffectHima Ekanadham. Guinevere Eden, Ph.D., Elizabeth Hoffman, Ph.D.Center for the Study of Learning, Georgetown University: 

An Illusory Illusion? The Role of Spatial Attention in the McGurk Effect Hima Ekanadham. Guinevere Eden, Ph.D., Elizabeth Hoffman, Ph.D. Center for the Study of Learning, Georgetown University INTRODUCTION METHODS & MATERIALS RESULTS DISCUSSION CONCLUSIONS REFERENCES What is the McGurk Effect (MGE)? The MGE is an audiovisual illusion involving the inaccurate perception of a spoken phoneme due to discrepant auditory and visual input6,7 In classic MGE studies, a viewer perceives a novel phoneme when auditory and visual inputs are discrepant e.g., s/he perceives |da| when an auditory |ba| is dubbed onto a visual |ga|6 It can be elicited from all human beings even in the face of variables such as gender, level of audio-visual synchrony and clarity of visual presentation1,4,5,8 The role of phonological perception in language acquisition is significant. Therefore, it is plausible that language acquisition ability impacts the degree of intensity of the MGE. Experimental Aim: to determine whether manipulation of another variable, eye-gaze direction, influences MGE Why? Previous studies have shown that perception of another’s gaze directs one’s own attention to the same space2,3. With this result in mind, it is possible that looking at a face whose gaze is horizontally averted could reduce a viewer’s attention toward the center of the face and therefore interfere with lip-reading. Reduced attention to the lips could decrease the visual component of phonological perception and hence attenuate or abolish MGE. Subjects: twenty native English-speaking healthy adults (aged 19-22, 10 females) Stimuli: 270 MGE stimuli that were validated in a pilot study. All stimuli were created using Adobe Premiere 6.0 from video clips of two, healthy, North American, native English-speaking adults (one female) producing the phonemes |ba|, |ta|, |va|, and |wa|. Design: This experiment is a 3 x 3 design audiovisual (A/V) phoneme pairs: gaze types: a. concordant control (CC) a. centrally fixated |ba|/|ba|; |ta|/|ta|; |va|/|va| b. discordant control (DC) b. central-averted shift |ta|/|ba|; |ta|/|va|; |ta|/|wa| c. discordant MGE (DV) c. averted-central shift |ba|/|va|; |wa|/|ta|; |wa|/|va| Procedure: On each of 270 trials, a video of a face appeared on a computer screen whose eye gaze either shifted horizontally from the center (or vice versa), or remained centrally fixated. After two seconds, the face articulated a phoneme (|va|, |ba|, |wa|, or |ta|). Each video lasted approximately three seconds. Lip movements were either concordant with the articulated phoneme (e.g., |va|, |va|),discordant without MGE (e.g., |ta|, |wa|), or discordant with MGE (e.g., |va|, |ba|). Participants indicated via keyboard press what they heard and responded as quickly as possible without compromising accuracy. Fig.2: Mean response time (RT) in ms for three A/V phoneme pair groups across three gaze conditions. Under straight gaze, the mean RTs elicited from CC, DC, and DV stimuli were 951.3 ms, 887.6 ms, and 1354.9 ms, respectively. Under a straight-to-averted shift, the mean RT elicited from CC, DC, and DV stimuli were 969.3 ms, 973.8 ms, and 1290.9 ms, respectively. Under an averted-to-straight shift, the mean RT elicited from CC, DC, and DV stimuli were 933.5 ms, 970.0 ms, and 1283.2 ms, respectively. Fig. 1: Mean accuracy (%) for three A/V pair groups across three gaze conditions. Under straight gaze, the mean accuracies elicited from CC, DC, and DV stimuli were 99.3%, 96.9%, and 22.6%, respectively. Under a straight-to-averted shift, the mean RT elicited from CC, DC, and DV stimuli were 98.1%, 5.8%, and 26.1%, respectively. Under an averted-to-straight shift, the mean RT elicited from CC, DC, and DV stimuli were 98.6%, 94.8%, and 25.9%, respectively. Table 1: F and P values from ANOVA showing the independent impact of both gaze type and A/V phoneme pair type on reaction time and accuracy percentage. What do these results mean? Consistent with previous studies, a significant MGE effect of high intensity was found. Participants were less accurate and slower in the MGE condition compared to the concordant and discordant control conditions However, contrary to initial expectations, change in gaze type did not significantly alter either RT or accuracy %. Although manipulation of eye-gaze is known to be a highly-influential social cue, it is possible that the MGE is robust enough to be impervious to changes in allocation of spatial attention. Furthermore, eye-gaze manipulation can be viewed as a minor change in the visual presentation. Previous studies have shown that the MGE is sustained even when the clarity of visual presentation is drastically diminished to the point where only a few features of the face are visible1,5. Future Directions? Currently, this experiment is being conducted on a second group of matched adult subjects who are Mandarin Chinese and English speaking bilinguals. To our knowledge, there has only been one prior study that explored the MGE in Chinese-speaking subjects and reported these subjects to have a lower MGE intensity as compared to monolingual English-speaking adults9. Future research is aimed at assessing the validity of this finding through the results that will be obtained, and furthermore, at exploring whether eye-gaze manipulation differentially effects MGE intensity in these bilinguals versus the first monolingual subject group. This research is the first to explore the potential interaction between the MGE and spatial attention. Results obtained thus far suggest that eye-gaze manipulation represents a minor disturbance to standard visual presentation that is not strong enough to diminish the robust MGE. The ongoing application of this experiment to bilingual adults, who speak both English and Chinese, will serve to be one of the first research attempts at simultaneously observing the possible interactions between language background, spatial attention, and the MGE. 1Green, K.P. & Kuhl, P. K. & Meltzoff, A. M., & Stevens, E. B. (1991). Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception and Psychophysics, 50, 524-536 2Hoffman, E.A. & Haxby, J. (2000) Distinct representations of eye gaze and identity in the distributed human neural system for face perception. Nature Neuroscience,3, 80–84. 3Hoffman, E.A. & Haxby, J. & Webb (1999) Contributions of Eye Gaze Perception to Face Recognition[Abstract], Society for Neuroscience 4Massaro, D.W. & Cohen, M.M. (1990). Perception of synthesized audible and visible speech. Psychological Science, 1, 55-63. 5Massaro, D.W. & Thompson, L.A. & Barron, B. & Laren, E.(1986) Developmental changes in visual and auditory contributions to speech perception, Journal of Experimental Child Psychology, 41(1), 93-113 6MacDonald, J. & McGurk, H. (1978). Visual influences on speech perception processes. Perception and Psychophysics. Vol 24, 253-257 7McGurk, H. & MacDonald, J. (1976). Hearing lips and seeing voices: A new illusion. Nature. Vol 264, 746-748. 8Rosenblum, L.D., Schmuckler, M.A., & Johnson, J.A. (1997). The McGurk effect in infants. Perception and Psychophysics. Vol 59(3), 347-357. 9Sekiyama, K. (1997). Cultural and linguistic factors in audiovisual speech processing: the McGurk effect in Chinese subjects. Perception and Psychophysics, Vol 59(1), 73-80.