Brain scans can reveal what song you’re listening to
- Brain activity can be decoded to reconstruct sensory experiences and certain mental states.
- Decoding acoustic information aids development of speech prostheses for paralyzed patients, but typically requires invasive recording methods.
- A new study records and decodes acoustic information non-invasively.
Neural decoding is a subfield of neuroscience that uses machine learning algorithms to infer or reconstruct mental states or sensory information from brain activity recorded by various neuroimaging methods.
Most decoding studies employ functional magnetic resonance imaging (fMRI), which records changes in cerebral blood flow as a proxy for brain activity. In the past decade, researchers have decoded fMRI data to detect individual memories and to reconstruct what a person is hearing from the brain’s responses to speech or what they are seeing while watching movies. Perhaps the most spectacular example is found in a 2013 study which used fMRI to decode the visual imagery that occurs in dreams.
Mind reading
fMRI can record activity across the whole brain at high spatial resolution, but cannot record brain activity over very short timescales. Electroencephalography (EEG), which records brain wave activity via electrodes attached to the scalp, has a much higher temporal resolution, but has poorer spatial resolution, making it difficult to record and decode accurately from anywhere other than the surface of the brain. Combining the two allows researchers to take advantage of each technology’s benefits, and that is what Ian Daly from the University of Sussex did to decode and reconstruct individual pieces of music that people are listening to.
He first used a combination of fMRI and EEG to record brain activity from 18 participants as they listened to 36 different pieces of music, and he identified several brain regions that respond selectively to music: the cerebellum, the parahippocampal gyrus, and areas within the left and right temporal lobes.
The combined fMRI-EEG data enabled Daly to identify individual pieces of music the participants were listening to with 72% accuracy. He then recorded data from 20 additional participants with EEG alone, using the first dataset to better target the source of the signals. Decoding the EEG data also enabled him to identify individual pieces of music, albeit with a lower accuracy (59%), perhaps because of significant variations in how the participants’ brains responded to the music they heard. (Although tempo is known to influence the EEG signal, it did not affect the results in this case, suggesting that decoding does not rely solely on this property of music.)
Giving speech to the speechless
The ability to decode acoustic information from brain activity aids the development of brain-computer interfaces that restore communication in patients who suffer stroke or other neurological conditions that can cause paralysis. But most attempts to decode acoustic information used fMRI, or invasive methods that involve placing electrodes directly onto the surface of the brain, both of which are impractical for these purposes.
Daly built on a method that was first used several years ago to synthesize speech from brain activity recorded invasively, and adapted it to decode complex acoustic information from data collected non-invasively. He is now planning to use EEG to decode heard and imagined speech.
