1 National Institute for Physiological Sciences, Okazaki, Japan, 2 Biomedical Imaging Research Center, Fukui University, Fukui, Japan, 3 JST (Japan Science and Technology Corporation)/RISTEX (Research Institute of Science and Technology for Society), Kawaguchi, Japan, 4 Department of Image-based Medicine, Institute of Biomedical Research and Innovation, Kobe, Japan, 5 Department of Education, Fukui University, Fukui, Japan, 6 Department of Education, Kanazawa University, Kanazawa, Japan and 7 Graduate School of Human and Environmental Studies, Kyoto University, Kyoto, Japan
Address correspondence to Norihiro Sadato, MD, PhD, Division of Cerebral Integration, Department of Cerebral Research, National Institute for Physiological Sciences, Myodaiji, Okazaki, Aichi, 444-8585 Japan. Email: sadato{at}nips.ac.jp.
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Key Words: blood flow cortex deafness language magnetic resonance speech
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
However, the activation of the auditory cortex by visual stimuli is not specific to deaf subjects. In hearing subjects, lip-reading enhances the perception of spoken language. Combining audible speech with visible articulation movements can improve comprehension to the same extent as altering the acoustic signal-to-noise ratio by 1520 dB (Sumby and Polack, 1954). Calvert et al. (1997)
found that linguistic visual cues during lip-reading activated the auditory cortex and GTs in hearing individuals in the absence of speech sounds, whereas non-linguistic facial movement did not. The findings of Calvert et al. (1997)
suggest that this type of audio-visual integration (face/voice matching) occurs in the superior temporal cortices.
Our hypothesis was that audio-visual cross-modal plasticity occurs within areas involved in cross-modal integration. To test this, we recruited deaf and hearing subjects for a fMRI study. Three tasks were included: mouth-movement matching, random dot-movement matching, and sign-related motion matching. The mouth-movement matching task contained open-mouth movements (OPEN), representing visual phonetics, and closed-mouth movements (CLOSED), which are not phonetic. Thus, the difference between the OPEN and CLOSED conditions can be used as a marker for phonetic speech perception (lip-reading effect). Cross-modal plastic changes were evaluated by comparing the task-related activation between deaf and hearing groups during closed-mouth, random-dot and sign-related movements without lexical processing. The bimodal activation of the auditory cortex in hearing subjects suggests that visual and auditory inputs are balanced competitively (Rauschecker, 1995). Hence, with auditory deprivation, the bimodal association auditory cortex is expected to be activated by the visual processing that usually does not activate the auditory cortex. As the CLOSED task does not contain visual phonetics, we expected that it would not activate the association auditory cortex of the hearing, whereas it would activate that of the deaf subjects. The random-dot movement-matching task was included to determine whether the plastic change is specific to the biological mouth motion or related to motion in general. Sign-word/non-word discrimination tasks were also included to determine whether the plastic changes are related to the motion of the hands or linguistic processes. To this end, a direct comparison was made between the deaf signers and the hearing signers.
![]() |
Subjects and Methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Our volunteers were 19 hearing subjects (11 women and eight men) aged 36.0 ± 10.7 years (mean ± SD), and seven prelingual deaf subjects (one woman and six men) aged 31.6 ± 7.1 years. Nine of the hearing subjects and all of the deaf subjects were fluent in Japanese sign language. The deaf subjects also had been trained in lip-reading at deaf school. The characteristics of the subjects are summarized in Table 1. The subjects were all right-handed according to the Edinburgh handedness inventory (Oldfield, 1971). None of the subjects had any history of neurological or psychiatric illness, and none had any neurological deficits other than deafness. The protocol was approved by the ethical committee of Fukui Medical University, Japan, and the National Institute for Physiological Sciences. All subjects gave their written informed consent for the study.
|
This experiment consisted of three tasks: a lip-movement task, a moving-dot task and signed-word discrimination. Each task was performed in separate sessions. Hearing signers, hearing non-signers and deaf signers underwent the mouth-movement and dot-motion tasks. Hearing signers and deaf signers participated in the signing tasks. The lip-motion and dot-motion sessions were repeated twice.
Experimental Setup
Visual stimuli were displayed on a half-transparent screen hung 285 cm from the subject's eyes, and were shown via an LCD projector (Epson ELP-7200L, Tokyo, Japan) connected to a digital video camera (SONY, Tokyo, Japan). Subjects viewed the screen through a mirror. The screen covered a visual angle of 10°. The subject's right hand was placed on a button box connected to a microcomputer that recorded the subject's responses.
MRI
A time-course series of volumes was acquired using T2*-weighted gradient echo-planar imaging (EPI) sequences with a 3.0 T magnetic resonance imager (VP, General Electric, Milwaukee, WI). Each volume consisted of 28 slices, with a slice thickness of 4.0 mm and a 1.0 mm gap, and included the entire cerebral and cerebellar cortex. The time interval between two successive acquisitions of the same image was 2500 ms, the echo time was 30 ms and the flip angle was 81°. The field of view (FOV) was 19 cm. The in-plane matrix size was 64 x 64 pixels with a pixel dimension of 3.0 x 3.0 mm. Tight, but comfortable, foam padding was placed around the subject's head to minimize head movement.
Immediately after five fMRI sessions, T2-weighted fast-spin echo images were obtained from each subject using location variables that were identical to those of the EPIs used for anatomical reference. In addition, high-resolution whole-brain MRIs were obtained with a conventional T2-weighted fast-spin echo sequence. A total of 112 transaxial images were obtained. The in-plane matrix size was 256 x 256 mm, slice thickness was 1.5 mm and pixel size was 0.742 x 0.742 mm.
Face-movement Task (Fig. 1)
The event-related design consisted of three event conditions: OPEN, CLOSED and a still face. Throughout the session, the subjects fixated a small cross-hair in the center of the screen. During the OPEN condition, four consecutive frames (frame duration = 500 ms) of a face pronouncing a vowel (/a/, /i/, /u/, /e/, or /o/) were presented (Fig. 1). No auditory input was provided. Following the stimulus, we presented a question mark (?) for 400 ms. This prompted the subject to press a button with their right index finger if the movements of the lips in the first and last frames were the same, and with the middle finger if the movements were not the same. The CLOSED condition was similar to the OPEN condition, except that the mouth was always closed during the movement, which consisted of five patterns of twitches of the lips (Fig. 1). During the still-face event condition (STILL), a face without any movement was presented for 2 s followed by the 400 ms presentation of an arrow pointing to the right or left. The subject was instructed to press the button with their right index finger if the leftward arrow was shown, and with the middle finger if the rightward arrow was shown. The inter-trial interval (ITI) was fixed at 3 s. Each condition was repeated 30 times, so there were 90 events in total.
|
![]() |
The efficiency of the estimation of the CLOSEDSTILL, OPENSTILL and OPENCLOSED comparisons was evaluated by taking the inverse of the covariance of the contrast between parameter estimates (Friston et al., 1999b),
![]() |
![]() |
Dot-motion Task
The fMRI session with random-moving dot stimuli consisted of three periods of stationary dots and three periods of moving dots; each period was 30 s long, alternating stationary and moving stimuli (Fig. 1). Throughout the session, the subjects were asked to fixate a small cross-hair in the center of the screen. During the 30 s stationary period, every 3 s stationary random dots were presented for 2 s, followed by the 400 ms appearance of an arrow pointing either right or left. The frame size of the presented dots was 5°. The subject was instructed to press a button with their right index finger if the leftward arrow appeared, and with the middle finger if the rightward arrow appeared. During the 30 s moving-dot period, every 3 s four consecutive frames of coherently moving random dots were presented. In each frame, the random dots moved for 500 ms in a single horizontal direction (right or left). The dots moved at a speed of 2.5°/s. This was followed by a question mark, which was presented for 400 ms. This cued the subject to press the button with the right index finger if the direction of the movement in the first and last frames was the same, and otherwise to respond by pressing the middle finger button. Thus, the subject was required to match the movement direction of the random dots in the first and last frames. The session took 3 min, and tried twice per subject.
Sign Task
We selected 30 sign words that can be expressed by moving the hands and arms, and without facial expression. The animated sign words were generated with commercially available software (Mimehand II, Hitachi, Tokyo, Japan) implemented on a personal computer (Endevor Pro-600L, Dell, Tokyo, Japan). This software can modify all components of the movement separately. As the sign words are composed of several component movements of the hands or arms, pseudo-words can be generated by modifying a single component of each word. Thirty pseudo-words were generated from the 30 words. Non-words were generated by flipping the 30 words and 30 pseudo-words upside down. Finally, a still image was generated.
The event-related design consisted of three event conditions: word discrimination (D), non-discrimination (ND) and still (S). The order of presentation was determined to maximize the efficiency of the predefined contrasts, as in the face-movement tasks. During the D condition, a single frame (2000 ms duration) of an animated character moving her hands and arms appeared, followed by a question mark (400 ms), which cued the subject to press the button with their right index finger if the movements constituted a real word, and otherwise to press the button with the middle finger. The ND condition was similar to D, except that the reversed upside-down images were presented, and the arrows were presented for 400 ms. The subject was instructed to press the button with their right index finger if the leftward arrow appeared, and with the middle finger if the rightward arrow appeared. During the S condition, the same characters were presented but without any movement, and both flipped and non-flipped images were presented. The stimuli lasted 2 s and were followed by an arrow pointing either right or left for 400 ms. The subject was instructed to press the button with their right index finger if the leftward arrow appeared, and with the middle finger if the rightward arrow appeared. The ITI was fixed at 3 s. Each condition was repeated 60 times and there were 180 events in total. The session took 9 min, and tried once per subject.
Data Analysis
The data were analyzed using statistical parametric mapping (SPM99, Wellcome Department of Cognitive Neurology, London, UK) implemented in Matlab (Mathworks, Sherborn, MA; Friston et al., 1994, 1995a
,b
). As our scanning protocol used sequential axial-slice acquisition in ascending order, signals in different slices were measured at different time points. To correct for this, we interpolated and resampled the data (Büchel and Friston, 1997
). Following realignment, all images were co-registered to the high-resolution three-dimensional (3-D) T2-weighted MRI, using the anatomical MRI with T2-weighted spin-echo sequences from locations identical to those of the fMRI images. The parameters for affine and nonlinear transformation into a template of T2-weighted images already fitted to standard stereotaxic space (MNI template; Evans et al., 1994
) were estimated using the high-resolution 3-D T2-weighted MRI employing least-square means (Friston et al., 1995b
). The parameters were applied to the co-registered fMRI data. The anatomically normalized fMRI data were filtered using a Gaussian kernel of 8 mm (full width at half maximum) in the x, y and z axes.
Statistics
Statistical analysis was conducted at two levels. First, individual task-related activation was evaluated. Secondly, individual data were summarized and incorporated into a random-effect model so that inferences could be made at a population level (Friston et al., 1999a).
Individual Analysis
The signal was scaled proportionally by setting the whole-brain mean value to 100 arbitrary units. Proportional scaling was performed to adjust for the fluctuation of the sensitivity of the MRI scanner which affects both global and local signals. Here we assume that the global signal (which is an indirect measure of global cerebral blood flow) is unchanged and that the local change will not affect the global signal. The signal time-course of each subject was modeled with two box-car functions convolved with a hemodynamic-response function, high-pass filtering and session effect. To test hypotheses about regionally specific condition effects, the estimates for each condition were compared using linear contrasts of the matching task versus the rest period. The resulting set of voxel values for each comparison constituted a statistical parametric map (SPM) of the t statistic (SPM{t}). The SPM{t} was transformed to normal-distribution units (SPM{Z}). The threshold for the SPM{Z} of individual analyses was set at P < 0.05, correcting for multiple comparisons at the voxel level for the entire brain (Friston et al., 1996).
Group Analysis with the Random-effect Model
The weighted sum of the parameter estimates in the individual analysis constituted contrast images that were used for the group analysis (Friston et al., 1999a). The contrast images obtained by individual analysis represent the normalized task-related increment of the MR signal of each subject, comparing the matching task versus rest period. To examine the effect of the tasks by group (deaf and hearing) and the task x group interaction, the contrast images of the prelingual deaf and hearing groups were entered into a random-effect model. For the mouth-movement and dot-motion tasks, hearing signers and hearing non-signers were pooled, as there was no significant group difference. The comparison was then made between the deaf (n = 7) and hearing (n = 19) groups. For the sign tasks, the deaf subjects (n = 7) and hearing signers (n = 9) were compared. Significant signal changes for each contrast were assessed using t-statistics on a voxel-by-voxel basis (Friston et al., 1995a
). The resulting set of voxel values for each contrast constituted an SPM of the t-statistic (SPM{t}). The SPM{t} was transformed into normal-distribution units (SPM{Z}). The threshold for the SPM{Z} was set at Z > 3.09 and P < 0.05, with a correction for multiple comparisons at the cluster level for the entire brain (Friston et al., 1996
).
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
In the hearing subjects, during the CLOSED condition the matching accuracy was 93.0 ± 5.9%, and accuracy during the OPEN condition was 90.8 ± 5.5%, with no significant difference between the conditions. In the deaf subjects, during the CLOSED condition the matching accuracy was 92.7 ± 5.7%, and accuracy during the OPEN condition was 86.5 ± 9.6%, with no significant difference.
During the CLOSED condition compared with the STILL condition, activation was found in the bilateral inferior (GFi) and middle (GFm) frontal gyrus, temporo-occipital junction, pre- and post-central gyri, superior (LPs) and inferior (LPi) parietal lobules, dorsal premotor cortex (PMd), supplementary motor cortex (SMA), posterior lobe of the cerebellar hemisphere, and left basal ganglia and thalamus. The left GTs, corresponding to the planum temporale (PT), was more active during the CLOSED condition in deaf subjects compared with hearing subjects (Fig. 2). During the OPEN condition compared with the STILL condition, activation was found in almost identical regions to those found in the CLOSEDSTILL comparison. However, in the hearing subjects, when the OPEN condition was compared with the CLOSED, activation was found in the left middle and inferior frontal gyrus, the bilateral GTs along the superior temporal sulci and the SMA (Table 2, Fig. 3). Post hoc tests confirmed that the responses in the left PT of the hearing signers and non-signers were similar (P > 0.1, two-sample t-test), and that both hearing and deaf subjects showed a significant lip-reading effect (OPENCLOSED) (Fig. 2).
|
|
|
Matching accuracy in the deaf group was 87.4 ± 9.1%, and in the hearing group accuracy was 85.1 ± 12.5%, with no significant group difference (P > 0.1, two-sample t-test).
In deaf subjects, the bilateral LPi, GTs and posterior lobe of the cerebellum, the left medial prefrontal cortex and PMd, and the right GFi and GFm were activated by the dot-motion discrimination task. In hearing subjects, the bilateral LPi, LPs, GFm, PMd, inferior and middle occipital gyrus, fusiform gyrus and posterior lobe of the cerebellum, the left medial prefrontal cortex and right GTs were activated. The right GTs (corresponding to the PT) was more active during the dot-motion discrimination task in deaf subjects compared with hearing subjects (Fig. 4). This area of hearing subjects showed a significant lip-reading effect during face motion tasks (contrasting OPENCLOSED). There were no areas in which the hearing subjects showed significantly more activation than deaf subjects did.
|
The accuracy of sign discrimination was 59.8 ± 2.9% in the hearing signers and 53.6 ± 5.4% in the deaf subjects, which was an insignificant difference. The deaf subjects showed an accuracy of 84.0 ± 35.7% in response to real words, and only 35.7 ± 11.3% accuracy in response to pseudo-words. The hearing signers showed 86.7 ± 12.1% accuracy in response to true words, and 40.6 ± 16.9% in response to pseudo-words.
In the hearing signers, the sign-discrimination and non-discrimination conditions activated the SMA, bilateral GFi, bilateral PMd, LPs, LPi, GOm and GTm. Deaf subjects showed a similar activation pattern, except more prominent activation was seen in the left GTs adjacent to the transverse temporal gyrus (Fig. 5, Table 3). The left GTs was activated more prominently in the deaf than in the hearing subjects during closed-lip movement, sign discrimination and non-discrimination. This area of hearing subjects showed a significant lip-reading effect during face motion tasks (contrasting OPENCLOSED). There was a marked overlap of the foci activated by lip-reading in hearing subjects and the cross-modal plasticity in the deaf, which was revealed by the sign-word discrimination/non-discrimination and lip-movement tasks (Fig. 6). Figure 6 showed that the common area for cross-modal integration (blue, as lip-reading in the hearing) and cross-modal plasticity (yellow for sign and red for CLSOED lip movement) is in the PT adjacent to the anterior border, the sulcus behind Heschl's gyrus (Westbury et al., 1999). On the other hand, the area for the lip-reading in the hearing (blue) extended to the posterior STS area, more prominent on the left. Sign discrimination compared with non-discrimination activated the bilateral PMd, GFm and GFi of both hearing and deaf signers but not the temporal cortices, whereas no activation was observed in the PT (Fig. 7).
|
|
|
|
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Neural Substrates of Lip-reading
In this study, we used the difference between the OPEN and CLOSED conditions as a marker of phonetic speech perception (lip-reading effect). We assumed that the OPEN condition carried phonetic information even if the subjects did not understand which vowel was presented, whereas the CLOSED condition did not. This is based on the fact that visual phonetics shown by lip movements can influence speech perception without any understanding of what is presented in each modality (the McGurk effect) (McGurk and MacDonald, 1976; Driver, 1996
). We used phonetic discrimination by visual speech (OPEN) to exclude the influence of the lexicon (Bernstein et al., 2002
). To control for the perception of lip movements and working memory processes, a non-phonetic lip-movement (CLOSED) condition was included. Thus, the OPENCLOSED comparison reveals the phonetic perception of visual speech that is essential for lip-reading. In the hearing group, during the OPEN condition compared with the CLOSED condition, more prominent activation was seen in the GTs, along the STS bilaterally and in the left lateralized frontal cortex including Broca's area, which is consistent with previous results (Burt and Perrett, 1997
; Campbell et al., 2001
; Olson et al., 2002
; Calvert and Campbell, 2003
).
Broca's Area
Functional neuroimaging studies revealed activity in Broca's area during prepositional speech tasks, including reading words (Price et al., 1994), word generation (McCarthy et al., 1993
), decoding syntactically complex sentences (Just et al., 1996
) and phoneme discrimination (Zatorre et al., 1992
, 1996
). Auditory phonemic-discrimination activated secondary auditory cortices bilaterally as well as Broca's area (Zatorre et al., 1992
, 1996
). Zatorre et al. (1996)
proposed that Broca's area is recruited for fine-grained phonetic analysis by means of articulatory decoding. The phonetic analysis of speech depends not only on auditory information but also on access to information about the articulatory gestures associated with a given speech sound (Liberman and Whalen, 2000
). This access might be accomplished through a mirror system, including Broca's area, which forms a link between the observer and the actor (Rizzolatti and Arbib, 1998
; Iacoboni et al., 1999
). The mirror system might not be confined to Broca's area (Bucchino et al., 2001
). Lip-reading enhances the motor excitability of the left primary motor cortex, presumably through input from premotor areas (Watkins et al., 2003
). This could account for the activation of other left-lateralized prefrontal cortical structures, such as BA 44/6 and SMA, as parts of the mirror system.
Planum Temporale
As expected, the OPENCLOSED condition activated the part of the GTs posterior to the transverse temporal gyrus (Heschl's gyrus), corresponding to the PT. Anatomically, the anterior border of the PT is the sulcus behind Heschl's gyrus and the medial border is the point where the PT fades into the insula. The posterior border is the ascending and descending rami of the Sylvian fissures (Westbury et al., 1999). Functionally, the left PT is involved in word detection and generation, due to its ability to process rapid frequency changes (Schwartz and Tallal, 1980
; Belin et al., 1998b
). The right homologue is specialized for the discrimination of melody, pitch and sound intensity (Zattore et al., 1994; Belin et al., 1998a
). Activation of the PT by the OPENCLOSED comparison is consistent with previous work (Calvert et al., 1997
), and confirms that phonetic information encoded in lip movements (during the OPEN condition) activates the PT. Furthermore, the temporal cortical activation was limited during the CLOSED condition in this study. This supports a previous study, which found that non-phonetic lower-face movement did not activate the temporal cortex (Calvert et al., 1997
). Hence, in hearing subjects, the activation of the PT is related to the phonetic information encoded by lip movements, and thus the perception of physical visual speech stimuli (visual phonetics) (Bernstein et al., 2002
). Figure 6 shows that the area for the lip-reading in the hearing (blue) extended to the posterior STS area, more prominent on the left. This is consistent with the previous report by Puce et al. (1998)
showing that the mouth movement activates the posterior portion of the STS. The more prominent posterior extension on the left than the right is consistent with the lesion study (Campbell et al., 1986
): a patient with a left occipitotemporal lesion cannot lip-read whereas another patient with the occipitotemporal lesion on the right did not show deficit in lip-reading. Hence the posterior STS on the left may be functionally related to the nearby PT thought to be involved in lip-reading.
In the deaf group, a similar, but less significant, activation pattern in the PT was found in response to visual phonetics, and hence there is a lip-reading effect (Figs 2, 4 and 5). This might reflect the fact that, despite being profoundly deaf, the deaf subjects in the present study have undergone training in lip-reading at school.
The Effects of Auditory Deprivation
The results also indicate that the neural substrates of the cross-modal plasticity resulting from auditory deprivation lie within the PT. The PT in hearing subjects did not respond to non-phonetic lip movements (CLOSED), dot movement and sign-related hand movement without lexical meaning, whereas the PT of the deaf subjects responded to all these conditions. However, the response of the PT to visual phonetics (OPENCLOSED) was seen in the hearing subjects. As these results were found for both hearing signers and hearing non-signers, the ability to sign cannot explain this difference. Similarly, visual phonetics cannot explain these differences because the group difference was seen during the CLOSED condition. Instead, these findings suggest the PT is the common substrate of audio-visual integration during lip-reading (OPENCLOSED) in hearing subjects and audio-visual cross-modal plasticity during non-phonetic lip (CLOSED) or hand movements in deaf subjects.
Task Dependency
Lip Movement and Dot Movement
This study confirmed the results of Finney et al. (2001), who showed that visual stimulation using moving dot patterns activated the right PT of deaf subjects. This finding suggests that the auditory cortex in deaf subjects is recruited for the processing of purely visual stimuli with no lexical component. The right PT is activated by sound motion (Baumgart et al., 1999
), probably reflecting the dominance of the right hemisphere for the processing of spatial information. However, in deaf subjects, the left PT was active in response to face movement, which was not seen in hearing subjects. The left-lateralized activity might be related to the notable anatomical and functional asymmetry seen in auditory processing.
Anatomically, the PT is significantly larger in the left hemisphere, with increased cell size and density of neurons. Furthermore, there are more functionally distinct columnar systems per surface unit in the left than the right PT (Galuske et al., 2000). Functionally, the left auditory cortical areas that are optimal for speech discrimination, which is highly dependent on rapidly changing broadband sounds, have a higher degree of temporal sensitivity. By contrast, the right cortical counterpart has greater spectral sensitivity, and thus is optimal for processing the tonal patterns of music, in which small and precise changes in frequency are important (Zatorre et al., 2002
). Thus, due to auditory deprivation, in deaf subjects the processing of the rapid temporal alternations of mouth movements might be directed towards the left PT, whereas processing of the global motion of moving dots might be performed in the right PT. This functional asymmetry might also explain why the area of the right PT that was more active in deaf than hearing subjects during the dot-discrimination task showed less overlap with the area activated by OPEN versus CLOSED conditions.
Sign
The sign-word discrimination and non-discrimination conditions revealed more prominent activation in the left PT of the deaf signers than hearing signers. This is consistent with Petitto et al. (2000), who found that meaningless but linguistically organized phonetic-syllabic units in sign, as well as real signs, activated the bilateral GTs of deaf subjects. They concluded that semantic content is not an important determinant of the activity of the GTs. This study revealed that the PT was activated by not only signs and pseudo-signs, but also their upside-down reversed moving images. Thus, the linguistic component of the signs might not be causing the PT activation in the deaf subjects. The sign word-discrimination and non-discrimination tasks yielded a similar activation pattern in the left PT: both tasks activated the left PT of the deaf signers, but deactivated that of the hearing signers (Fig. 5). Furthermore, when the sign-word discrimination was compared with the non-discrimination condition, the bilateral frontal cortex was activated, but the temporal cortical areas were not (Fig. 7).
We saw a marked overlap of the enhanced activation in the left PT during sign presentation and face discrimination in the deaf subjects compared with the hearing signers. This finding suggests that the activation of the left PT in the deaf during sign presentation relates neither to sign-language comprehension nor linguistic processing. Instead, the left PT activation might be related to the analysis of the temporal segmental aspects of hand or lip movements. Lip-reading accuracy correlated negatively with sign-language usage in the deaf (Geers and Moog, 1989; Moores and Sweet, 1991
; Bernstein et al., 1998
), suggesting competition between the processes. Hence, amodal processing of rapid temporal alternation might occur in the left PT, the input to which might be visual or auditory. Although cross-modal plastic changes in the PT are not specific to linguistic processing itself, as shown in the present study, it might promote the sublexical processing of signs.
Cross-modal Plasticity in the Deaf and Lip-reading in the Hearing Share the PT
This study revealed that the primary auditory cortex (A1) is not activated by visual phonetics (lip-reading) in the hearing, which was consistent with a previous study (Bernstein et al., 2002). We also found that the plastic changes due to auditory deprivation activated the PT, as did lip-reading in the hearing subjects; the A1 was also spared. In deaf subjects, higher-order auditory areas become activated by visually presented sign language (Nishimura et al., 1999
; Petitto et al., 2000
). When the auditory nerve is stimulated by cochlear implants, the contralateral A1 becomes active (Lee et al., 2001
). The higher-order cortical areas become progressively activated only after increased hearing experience with cochlear implants (Giraud et al., 2001
). Therefore, cross-modal reorganization is likely to occur only in the higher-order auditory areas, consistent with the present findings. A previous electrophysiological study revealed no evidence of cross-modal plasticity in the A1 in congenitally deaf cats (Kral et al., 2003
). They attributed this to the different inputs that the A1 and the auditory-association cortices receive. The primary auditory cortex receives its main input from the lemniscal auditory pathway, whereas the higher-order auditory cortices receive mainly cortical and extralemniscal thalamic inputs. The extralemniscal thalamic nuclei also receive projections from other modalities (Aitkin et al., 1978
, 1981
; Robards, 1979
; Shore et al., 2000
; Schroeder et al., 2001
). The sensory representation is determined by the competitive balance of inputs from different modalities (Rauschecker, 1995
); therefore, these findings suggest that cross-modal plasticity occurs in the PT.
In conclusion, the cross-modal plastic changes as a result of auditory deprivation might be mediated, at least in part, by the neural substrates of cross-modal integration in the hearing.
![]() |
Acknowledgments |
---|
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Aitkin LM, Kenyon CE, Philpott P (1981) The representation of the auditory and somatosensory systems in the external nucleus of the cat inferior colliculus. J Comp Neurol 196:2540.[CrossRef][ISI][Medline]
Baumgart F, Gaschler-Markefski B, Woldorff MG, Heinze H-J, Scheich H (1999) A movement-sensitive area in auditory cortex. Nature 400:724726.[ISI][Medline]
Belin P, McAdams S, Smith B, Savel S, Thivard L, Samson S, Samson Y (1998a) The functional anatomy of sound intensity discrimination. J Neurosci 18:63886394.
Belin P, Zilbovicius M, Crozier S, Thivard L, Fontaine A, Masure M, Samson Y (1998b) Lateralization of speech and auditory temporal processing. J Cogn Neurosci 10:536540.[Abstract]
Bernstein LE, Demorest ME, Tucker PE (1998) What makes a good speechreader? First you have to find one. In: Hearing by eye: II. The psychology of speechreading and auditory-visual speech (Campbell R, Burnham D, Burnham D, eds), pp. 211228. Hove: Psychology Press.
Bernstein LE, Auer ET, Moor JK, Ponton CW, Don M, Singh M (2002) Visual speech perception without primary auditory cortex activation. Neuroreport 13:311315.[CrossRef][ISI][Medline]
Bucchino G, Binkofski F, Fink GR, Fadiga L, Fagossi L, Gallese V, Seitz R, Zilles K, Rizzolatti G, Freund H-J (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Europ J Neurosci 13:400404.[CrossRef][ISI][Medline]
Büchel C, Friston KJ (1997) Modulation of connectivity in visual pathways by attention: cortical interactions evaluated with structural equation modelling and fMRI. Cereb Cortex 7:768778.[Abstract]
Burt DM, Perrett DI (1997) Perceptual asymmetries in judgments of facial attractiveness, age, gender, speech and expression. Neuropsychologia 35:685693.[CrossRef][ISI][Medline]
Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997) Activation of auditory cortex during silent lipreading. Science 276:593596.
Calvert GA, Campbell R (2003) Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 15:5770.
Campbell R, Landis T, Regard M (1986) Face recognition and lipreading. A neurological dissociation. Brain 109:509521.[Abstract]
Campbell R, MacSweeney M, Surguladze S, Calvert GA, McGuire P, Suckling J, Brammer MJ, David AS (2001) Cortical substrates for the perception of face action: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cogn Brain Res 12:233243.[CrossRef][ISI][Medline]
Driver J (1996) Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature 381:6668.[CrossRef][ISI][Medline]
Evans AC, Kamber M, Collins DL, MacDonald D (1994) An MRI-based probabilistic atlas of neuroanatomy. In: Magnetic resonance scanning and epilepsy (Shorvon SD, ed.), pp. 263274. New York: Plenum Press.
Finney EM, Fine I, Dobkins KR (2001) Visual stimuli activate auditory cortex in the deaf. Nat Neurosci 4:11711173.[CrossRef][ISI][Medline]
Friston KJ, Worsley KJ, Frackowiak RSJ, Mazziotta JC, Evans AC (1994) Assessing the significance of focal activations using their spatial extent. Hum Brain Mapp 1:210220.[CrossRef]
Friston KJ, Holmes AP, Worsley KJ, Poline JB, Frith CD, Frackowiak RSJ (1995a) Statistical parametric maps in functional imaging: a general linear approach. Hum Brain Mapp 2:189210.
Friston KJ, Ashburner J, Frith CD, Heather JD, Frackowiak RSJ (1995b) Spatial registration and normalization of images. Hum Brain Mapp 2:165189.
Friston KJ, Holmes A, Poline J-B, Price CJ, Frith CD (1996) Detecting activations in PET and fMRI: levels of inference and power. Neuroimage 4:223235.[CrossRef][ISI][Medline]
Friston KJ, Holmes AP, Worsley KJ (1999a) How many subjects constitute a study? Neuroimage 10:15.[CrossRef][ISI][Medline]
Friston KJ, Zarahn E, Josephs O, Henson RNA, Dale AM (1999b) Stochastic designs in event-related fMRI. Neuroimage 10:607619.[CrossRef][ISI][Medline]
Galuske RAW, Schlote W, Bratzke H, Singer W (2000) Interhemispheric asymmetries of the modular structure in human temporal cortex. Science 289:19461949.
Geers A, Moog J (1989) Factors predictive of the development of literacy in profoundly hearing-impaired adolescents. Volta Rev 91:6986.[ISI]
Giraud AL, Price CJ, Graham JM, Truy E, Frackowiak RS (2001) Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron 30:657663.[CrossRef][ISI][Medline]
Iacoboni M, Woods RP, Brass M, Becckering H, Mazziotta JC, Rizzolatti G (1999) Cortical mechanisms of human imitation. Science 286:25262528.
Just MA, Carpenter PA, Keller TA, Eddy WF, Thulborn KR (1996) Brain activation modulated by sentence comprehension. Science 274:114116.
Kral A, Schroder J, Klinke R, Engel A (2003) Absence of cross-modal reorganization in the primary auditory cortex of congenitally deaf cats. Exp Brain Res 153:605613.[CrossRef][ISI][Medline]
Lee DS, Lee JS, Oh SH, Kim S-K, Kim J-W, Chung J-K, Lee MC, Kim CS (2001) Cross-modal plasticity and cochlear implants. Nature 409:149150.[Medline]
Liberman AM, Whalen DH (2000) On the relation of speech to language. Trends Cogn Sci 4:187196.[CrossRef][ISI][Medline]
McCarthy G, Blamire AM, Rothman DL, Gruetter R, Shulman RG (1993) Echo-planar magnetic resonance imaging studies of frontal cortex activation during word generation in humans. Proc Natl Acad Sci USA 90:49524956.
McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746748.[ISI][Medline]
Moores DF, Sweet C (1991) Factors predictive of school achievement. In: Educational and developmental aspects of deafness (Moores DF, K.P. M-O, eds), pp. 154201. Washington, DC: Gallaudet University Press.
Neville HJ, Bavelier D, Corina D, Rauschecker J, Karni A, Lalwani A, Braun A, Clark V, Jezzard P, Turner R (1998) Cerebral organization for language in deaf and hearing subjects: biological constraints and effects of experience. Proc Natl Acad Sci USA 95:922929.
Nishimura H, Hashikawa K, Doi K, Iwaki T, Watanabe Y, Kusuoka H, Nishimura T, Kubo T (1999) Sign language heard in the auditory cortex. Nature 397:116.[CrossRef][ISI][Medline]
Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97113.[CrossRef][ISI][Medline]
Olson IR, Gatenby JC, Gore JC (2002) A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Cogn Brain Res 14:129138.[CrossRef][ISI][Medline]
Petitto LA, Zatorre RJ, Gauna K, Nikelski EJ, Dostie D, Evans AC (2000) Speech-like cerebral activity in profoundly deaf people processing signed languages: implication for the neural basis of human language. Proc Natl Acad Sci USA 97:1396113966.
Price CJ, Wise RJ, Watson JD, Patterson K, Howard D, Frackowiak RS (1994) Brain activity during reading. The effects of exposure duration and task. Brain 117:12551269.[Abstract]
Puce A, Allison T, Bentin S, Gore JC, McCarthy G (1998) Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci 18:21882199.
Rauschecker JP (1995) Compensatory plasticity and sensory substitution in the cerebral cortex. Trends Neurosci 18:3643.
Rizzolatti G, Arbib A (1998) Language within our grasp. Trends Neurosci 21:188194.[CrossRef][ISI][Medline]
Robards MJ (1979) Somatic neurons in the brainstem and neocortex projecting to the external nucleus of the inferior colliculus: an anatomical study in the opossum. J Comp Neurol 184:547565.[CrossRef][ISI][Medline]
Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85:13221327.
Schwartz JH, Tallal P (1980) Rate of acoustic change may underlie hemispheric specialization for speech perception. Science 207:13801381.[ISI][Medline]
Shore SE, Vass Z, Wys NL, Altschuler RA (2000) Trigeminal ganglion innervates the auditory brainstem. J Comp Neurol 419:271285.[CrossRef][ISI][Medline]
Sumby WH, Polack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212215.[ISI]
Watkins KE, Strafella AP, Paus T (2003) Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 41:989994.[CrossRef][ISI][Medline]
Westbury CF, Zatorre RJ, Evans AC (1999) Quantifying variability in the planum temporale: a probability map. Cereb Cortex 9:392405.
Zatorre RJ, Evans AC, Meyer E, Gjedde A (1992) Lateralization of phonetic and pitch discrimination in speech processing. Science 256:846849.[ISI][Medline]
Zatorre RC, Evans AC, Meyer E (1994) Neural mechanisms underlying melodic perception and memory for pitch. J Neurosci 14:19081919.[Abstract]
Zatorre RJ, Meyer E, Gjedde A, Evans AC (1996) PET studies of phonetic processing of speech: review, replication, and reanalysis. Cereb Cortex 6:2130.[Abstract]
Zatorre RJ, Belin P, Penhune VB (2002) Structure and function of auditory cortex: music and speech. Trends Cogn Sci 6:3746.[CrossRef][ISI][Medline]
|