Decoding the signals of speech
Ann was a high school math teacher in Canada before her stroke in 2005. In 2020, she described her life since in a paper she wrote, painstakingly typing letter-by-letter, for a psychology class.
“Locked-in syndrome, or LIS, is just like it sounds,” she wrote. “You’re fully cognizant, you have full sensation, all five senses work, but you are locked inside a body where no muscles work. I learned to breathe on my own again, I now have full neck movement, my laugh returned, I can cry and read and over the years my smile has returned, and I am able to wink and say a few words.”
As she recovered, she realized she could use her own experiences to help others, and she now aspires to become a counselor in a physical rehabilitation facility.
“I want patients there to see me and know their lives are not over now,” she wrote. “I want to show them that disabilities don’t need to stop us or slow us down.”
She learned about Chang’s study in 2021 after reading about a paralyzed man named Pancho, who helped the team translate his brain signals into text as he attempted to speak. He had also experienced a brainstem stroke many years earlier, and it wasn’t clear if his brain could still signal the movements for speech. It’s not enough just to think about something; a person has to actually attempt to speak for the system to pick it up. Pancho became the first person living with paralysis to demonstrate that it was possible to decode speech-brain signals into full words.
With Ann, Chang’s team attempted something even more ambitious: decoding her brain signals into the richness of speech, along with the movements that animate a person’s face during conversation.
To do this, the team implanted a paper-thin rectangle of 253 electrodes onto the surface of her brain over areas they previously discovered were critical for speech. The electrodes intercepted the brain signals that, if not for the stroke, would have gone to muscles in Ann’s lips, tongue, jaw and larynx, as well as her face. A cable, plugged into a port fixed to Ann’s head, connected the electrodes to a bank of computers.
For weeks, Ann worked with the team to train the system’s artificial intelligence algorithms to recognize her unique brain signals for speech. This involved repeating different phrases from a 1,024-word conversational vocabulary over and over again until the computer recognized the brain activity patterns associated with all the basic sounds of speech.
Ann worked with the team on training the AI algorithm to recognize her brain signals associated with phonemes, the sub-units of speech that form spoken words. Photo by Noah Berger
