Date of Award
5-11-2025
Date Published
June 2025
Degree Type
Thesis
Degree Name
Master of Science (MS)
Department
Psychology
Advisor(s)
Natalie Russo
Second Advisor
Tanya Eckert
Keywords
audiovisual;MMNs;multisensory integration;N400;semantic congruence
Subject Categories
Psychology | Social and Behavioral Sciences
Abstract
Multisensory integration is a pervasive phenomenon in daily life, where our senses (commonly visual and auditory) demonstrate a high degree of interdependence in information processing. This integration is often regarded as automatic and pre-attentional. The Mismatch Negativity (MMN) is an event-related potential (ERP) component that captures the difference between novel stimuli (deviant, unexpected trials with low frequency) and regularly occurring stimuli (standard, expected trials with high frequency). MMN can detect unexpected sensory inputs and provide information about the timing of how the brain processes information from different modalities. However, limited research has explored MMN’s application in multisensory processing, with most studies focusing on speech perception or letter-sound combinations. The main objective of this study was to assess the extent to which multisensory perceptual processing relied on sensory modalities and semantic congruency by examining MMN effects in semantically meaningful audiovisual contexts. 18 participants completed an oddball task in which animal sounds (ribbit, meow) and pictures (frog, cat) were paired and presented simultaneously. The standard stimulus (meowcat) occurred in 70% of trials, while the three deviants (meowfrog, ribbitcat, ribbitfrog) each occurred in 10% of trials. Three difference waves were calculated by subtracting the standards from the deviants. Results revealed robust visual and audiovisual MMNs within the 100–250 ms time window. These two MMNs exhibited similar onset latency and mean amplitude, though the audiovisual MMN had an earlier offset latency than the visual MMN. A two-way repeated measures ANOVA found a significant main effect of the visual factor for mean amplitude. This indicates a strong Colavita visual dominance effect, and it may lead to a delay in peak timing for the auditory difference wave as evidenced by an N2-like ERP. Additionally, an interaction for 50% fractional area latency indicated a significant difference in processing cost for two semantically incongruent stimuli compared to two semantically congruent stimuli. To further investigate the role of semantic congruence in modulating multisensory processing, difference waves were calculated in the 300–500 ms time window to examine brain responses to semantic violations (e.g., ribbitcat-minus-ribbitfrog). N400 effects were found for both auditory and audiovisual violations. This research suggests a nuanced, two-phase processing pattern for audiovisual deviants, involving a rapid initial detection followed by a later semantic analysis. Processing time windows for visual and auditory deviants were distinctly separated, perhaps reflecting a dominant modality first to efficiently process sensory inputs across modalities in this semantic multisensory context. This study not only highlights the brain’s dynamic adaptability in processing various types of sensory input but also underscores the complex interplay between sensory modalities and semantic contexts in multisensory integration.
Access
Open Access
Recommended Citation
Chen, Xiaoxiao, "INVESTIGATING CROSS-MODAL TEMPORAL DYNAMICS AND SEMANTIC INCONGRUENCY IN MULTISENSORY INTEGRATION" (2025). Theses - ALL. 911.
https://surface.syr.edu/thesis/911
