Publications by Year: 2012

Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.

We aimed to determine whether semantic relatedness between an incoming word and its preceding context can override expectations based on two types of stored knowledge: real-world knowledge about the specific events and states conveyed by a verb, and the verb’s broader selection restrictions on the animacy of its argument. We recorded event-related potentials on post-verbal Agent arguments as participants read and made plausibility judgments about passive English sentences. The N400 evoked by incoming animate Agent arguments that violated expectations based on real-world event/state knowledge, was strongly attenuated when they were semantically related to the context. In contrast, semantic relatedness did not modulate the N400 evoked by inanimate Agent arguments that violated the preceding verb’s animacy selection restrictions. These findings suggest that, under these task and experimental conditions, semantic relatedness can facilitate processing of post-verbal animate arguments that violate specific expectations based on real-world event/state knowledge, but only when the semantic features of these arguments match the coarser-grained animacy restrictions of the verb. Animacy selection restriction violations also evoked a P600 effect, which was not modulated by semantic relatedness, suggesting that it was triggered by propositional impossibility. Together, these data indicate that the brain distinguishes between real-world event/state knowledge and animacy-based selection restrictions during online processing.

Active reading requires coordination between frequent eye movements (saccades) and short fixations in text. Yet, the impact of saccades on word processing remains unknown, as neuroimaging studies typically employ constant eye fixation. Here we investigate eye-movement effects on word recognition processes in healthy human subjects using anatomically constrained magnetoencephalography, psychophysical measurements, and saccade detection in real time. Word recognition was slower and brain responses were reduced to words presented early versus late after saccades, suggesting an overall transient impairment of word processing after eye movements. Response reductions occurred early in visual cortices and later in language regions, where they colocalized with repetition priming effects. Qualitatively similar effects occurred when words appeared early versus late after background movement that mimicked saccades, suggesting that retinal motion contributes to postsaccadic inhibition. Further, differences in postsaccadic and background-movement effects suggest that central mechanisms also contribute to postsaccadic modulation. Together, these results suggest a complex interplay between visual and central saccadic mechanisms during reading.

We measured Event-Related Potentials (ERPs) and naming times to picture targets preceded by masked words (stimulus onset asynchrony: 80 ms) that shared one of three different types of relationship with the names of the pictures: (1) Identity related, in which the prime was the name of the picture ("socks" - ), (2) Phonemic Onset related, in which the initial segment of the prime was the same as the name of the picture ("log" - ), and (3) Semantically related in which the prime was a co-category exemplar and associated with the name of the picture ("cake" - ). Each type of related picture target was contrasted with an Unrelated picture target, resulting in a 3×2 design that crossed Relationship Type between the word and the target picture (Identity, Phonemic Onset and Semantic) with Relatedness (Related and Unrelated). Modulation of the N400 component to related (versus unrelated) pictures was taken to reflect semantic processing at the interface between the picture’s conceptual features and its lemma, while naming times reflected the end product of all stages of processing. Both attenuation of the N400 and shorter naming times were observed to pictures preceded by Identity related (versus Unrelated) words. No ERP effects within 600 ms, but shorter naming times, were observed to pictures preceded by Phonemic Onset related (versus Unrelated) words. An attenuated N400 (electrophysiological semantic priming) but longer naming times (behavioral semantic interference) were observed to pictures preceded by Semantically related (versus Unrelated) words. These dissociations between ERP modulation and naming times suggest that (a) phonemic onset priming occurred late, during encoding of the articulatory response, and (b) semantic behavioral interference was not driven by competition at the lemma level of representation, but rather occurred at a later stage of production.

Accurately communicating self-relevant and emotional information is a vital function of language, but we have little idea about how these factors impact normal discourse comprehension. In an event-related potential (ERP) study, we fully crossed self-relevance and emotion in a discourse context. Two-sentence social vignettes were presented either in the third or the second person (previous work has shown that this influences the perspective from which mental models are built). ERPs were time-locked to a critical word toward the end of the second sentence which was pleasant, neutral, or unpleasant (e.g., A man knocks on Sandra’s/your hotel room door. She/You see(s) that he has agift/tray/gunin his hand.). We saw modulation of early components (P1, N1, and P2) by self-relevance, suggesting that a self-relevant context can lead to top-down attentional effects during early stages of visual processing. Unpleasant words evoked a larger late positivity than pleasant words, which evoked a larger positivity than neutral words, indicating that, regardless of self-relevance, emotional words are assessed as motivationally significant, triggering additional or deeper processing at post-lexical stages. Finally, self-relevance and emotion interacted on the late positivity: a larger late positivity was evoked by neutral words in self-relevant, but not in non-self-relevant, contexts. This may reflect prolonged attempts to disambiguate the emotional valence of ambiguous stimuli that are relevant to the self. More broadly, our findings suggest that the assessment of emotion and self-relevance are not independent, but rather that they interactively influence one another during word-by-word language comprehension.