For decades, scientists thought that in order to read other people’s emotions, the brain processed sensory information from faces and voices in separate areas. But new research shows otherwise—seeing an emotive face lights up the brain’s voice-processing centers, and vice versa.
In everyday life, “we don’t only see a face that is sad,” explains Stefania Mattioni , the study’s lead author and a cognitive neuroscience postdoctoral researcher at Ghent University in Belgium. “We normally also hear the sound of the person crying, for instance. So, the brain is actually structured to interact with this kind of multisensory stimulation.”
For the study, Mattioni’s team played clips of actors portraying different emotions, such as happiness, disgust, and fear,to 24 participants who were hooked up to a magnetic resonance imaging (MRI) machine. Some of the clips showed both the actors’ faces and voices; others only included video or audio.
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Audio-only clips activated brain regions known to process faces, and video-only clips activated brain regions that process voices. In fact, each individual emotion activated both the face and voice brain regions in similar ways, regardless of whether the emotion was presented visually or auditorily, Mattioni says. This ability may help the brain understand situations in which you can’t see a person but can hear them, such as when talking on the phone—or vice versa.
“It seems that these regions are more guided by the emotion than by the modality,” Mattioni says.
Mattioni’s team also found that people’s brains react differently when they can see and hear another person speak. “The brain is doing more than adding two inputs together,” explains Stefania Benetti, an assistant professor of psychology and cognitive science at the University of Trento in Italy, who was not involved in the study. “Emotion recognition, in other words, seems to depend on a multisensory network that integrates what we see and hear.”
“From an evolutionary and behavioral standpoint, combining sight and sound provides a huge advantage,” Benetti says. “It reduces ambiguity and allows us to recognize emotions faster and more accurately, especially in noisy or complex environments.”
The findings could eventually help scientists develop interventions for people who struggle with social cues. They also support previous research on brain plasticity that has found that in people with early deafness, voice-selective brain regions adapt to process facial expressions. Similarly, in people who are blind, face-selective regions of the brain adapt to process vocal expressions, Benetti says. These regions may be able to adapt in these ways because their main aim is to process emotions, not single senses, she says.
The study was published in the Journal of Neuroscience.
Subscribe to Support Independent Journalism
Great science journalism requires human expertise, time, effort and creativity. And it costs money. That’s why I and the journalists here at Scientific American hope you’ll join our community.
When you subscribe, you are supporting staff and freelance journalists who are passionate about telling science stories that are true, important and compelling. Our editors and reporters are often experts in their fields, which means they understand the nuances of big discoveries and can untangle the breakthroughs from the hype. With a subscription, you are also supporting rigorous fact-checking to ensure the words we publish are precise and accurate. And you’re supporting original illustrations, graphics and photos that bring you closer to an advanced laboratory, an ice sheet in Antarctica or a space mission in orbit. You’re helping us craft other types of high-quality journalism as well: Our newsletters are carefully written, edited and curated by staffers you have or will come to know and love. Our Science Quickly podcast is based on original reporting, collaboration with editors and scientists and exacting production.
Subscriptions keep this engine running so we can continue to deliver thoughtful, rigorous and independent science journalism to you. In an era of viral misinformation, this work is crucial. If you value what we do, I hope you’ll consider joining us as a subscriber
