Errors on the voice side follow this path back to the decoderTraining so there is no need to pair neural signals with voice annotations
Drag horizontally to see the full process →
Click on any stop to see what this step is doing
01cortical signalECoG
Electrode arrays are applied to the surface of the cortex to record high gamma activity while speaking. Each subject has a different electrode position, which is the first problem to be dealt with for this type of work.
02decoderDECODER
Translate the high gamma signal into a set of speech parameters. In the paper, the 3D ResNet, Swin, and LSTM structures were compared, and the correlation coefficient between 3D ResNet and the original spectrum was the highest, reaching 0.804.
03Voice parametersPARAMS
The output is the amount of pitch, loudness, resonance peaks, and turbidity that people can read. Each can be checked, or it can be questioned as to why it took this value.
04Differentiable synthesizerSYNTH
Re-synthesize the speech with this set of parameters. This step is minimal, so the error on the speech side can be transmitted back to the decoder all the way, and the neural signals in pairs do not need to be trained to hard align with the speech marker.
05SpectrumSPECTROGRAM
The synthetic spectrum was verbatim compared with the actual spectrum spoken by the subjects, and the harmonic structure, starting and ending moments, and turbidity changes were all correct. 48 subjects were stably reproduced.
First author: Yifei Chen (Algorithm Engineer) · Corresponding author: Xupeng Chen · Both affiliated with Dimension Gate
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)IROS 2026 · AcceptedarXiv:2603.04277
A deterministic geometric perception skill for embodied agents: detect common vehicles in metadata-free aerial imagery, recover ground sample distance, and return a confidence score before metric decisions are made. Yifei Chen is first author; Xupeng Chen is corresponding author. Both are affiliated with Dimension Gate.
Xupeng Chen, Ran Wang, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker
An ECoG decoder that translates cortical signals into interpretable speech parameters, paired with a differentiable speech synthesizer. Reproducible across 48 participants; the 3D ResNet decoder reaches PCC 0.804 against the original spectrogram, and stays high under causal-only operation as required for real-time neural prostheses.
Drag horizontally to view full page →
Fig. 1Closed loop of the whole frame: The ECoG decoder translates the high gamma signal of the cortex into an interpretable set of speech parameters, which can then be reduced to spectrum by the differentiable speech synthesizer. The synthesizer is differentiable, so the loss of the speech side can be transmitted back to the decoder - training does not require a pair of "neural signal — speech" annotations to hard align.Original picture ↗·CC BY 4.0
Drag horizontally to view full page →
Fig. 2c–dAscending is the spectrum of the eight words actually spoken by the subject, and descending is the spectrum decoded only by the cortical signal. The harmonic structure, start and end time, and turbidity change were all correct - the correlation coefficient between the 3D ResNet decoder and the original spectrum reached 0.804 on 48 subjects.Original picture ↗·CC BY 4.0
Drag horizontally to view full page →
Fig. 4Covering a certain cortex, how much quality is lost in decoding - the more it falls, the darker the color. The hot zone is concentrated in the sensory motor cortex and the superior temporal gyrus, and is stable under both decoders, causal and non-causal settings. This conclusion does not rely on model selection, indicating that the model has learned something that makes physiological sense.Original picture ↗·CC BY 4.0
Asks why telling a vision-language model where to look often makes its answer worse. Benchmarks frontier VLMs on medical grounding, separates the two failure modes — the box lands in the wrong place, or the box is right but cropping to it throws away the context the answer needed — and shows what fine-tuning does and does not fix.
MedVRAG retrieves over ~350K page images from the literature rather than over plain text, then decides for itself whether the evidence it pulled is enough — and goes back for more if it is not. Most questions settle in one round; the hard ones are exactly the ones that need a second and third.
Surface EMG from the face and neck, compared across spoken, mimed and silently imagined speech. Asks how much of the articulatory signal survives when no sound is produced — the question any silent-speech interface has to answer before it can work.
Decodes what a person is looking at from fMRI, with one model shared across subjects instead of one model per brain. A token-merging encoder maps voxels into a CLIP embedding space, so adding a new subject costs a fraction of the parameters a per-subject model would.
Predicts which cortical sites will arrest speech when stimulated — the mapping a surgeon currently has to establish by stimulating the awake patient site by site. Same signal, read ahead of time instead of found the hard way.
Chain-of-thought prompting is supposed to make models better. On instruction following it frequently makes them worse: the model reasons its way past a constraint it would have obeyed if asked directly. Measures where the trade-off bites and what mitigates it.
Junbo Chen, Xupeng Chen, Ran Wang, Chenqian Le, Amirhossein Khalilian-Gourtani, Erika Jensen, Patricia Dugan, Werner Doyle, Orrin Devinsky, Daniel Friedman, Adeen Flinker, Yao Wang
SwinTW, a transformer that works with arbitrarily positioned electrodes by using their 3D cortical coordinates rather than a 2D grid. Subject-specific models on low-density 8×8 ECoG reach PCC 0.817 across 43 participants, freeing future speech prostheses from grid-only electrode layouts.
Injects the region a clinician would actually look at straight into the visual token stream, so the model reasons over the lesion instead of over the whole slide. The line of work that the 2026 grounding benchmark above grew out of, and argued with.
Amirhossein Khalilian-Gourtani, Ran Wang, Xupeng Chen, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker
Proceedings of the National Academy of Sciences (PNAS)Vol. 121, No. 50, e2404121121doi:10.1073/pnas.2404121121
Identifies a reproducible corollary discharge source in ventral speech motor cortex that fires before articulation and predicts how strongly auditory cortex is suppressed during speech — how the brain tells its own voice apart from the world.
Ran Wang, Xupeng Chen, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker
Proceedings of the National Academy of Sciences (PNAS)Vol. 120, No. 42, e2300255120doi:10.1073/pnas.2300255120
Uses causal, anticausal and noncausal convolutional architectures to disentangle motor control from sensory processing during speech, revealing a mixed feedforward/feedback recruitment across perisylvian cortex.
Ran Wang, Xupeng Chen, Amirhossein Khalilian-Gourtani, Zhaoxi Chen, Leyao Yu, Adeen Flinker, Yao Wang
IEEE International Symposium on Biomedical Imaging (ISBI)
Applies GAN transfer learning to decode heard speech from cortical recordings, addressing the scarcity of paired neural-speech data.
The 3 original drawings on the page are taken from the papers listed above, all of which are open access, licensed under CC BY 4.0, and the signed and licensed links are shown in the drawings. The layout is scaled by the width of the layout, and only the sub-panel of the individual diagrams has been taken, and the pixel color has not been changed. The remaining papers are copyrighted by the publisher and are not authorized to be reproduced.
From Papers to Products
Academic research and implementation
01
Leave unsaid words
Research
During his PhD, CTO and Chief Scientist Xupeng Chen focused on decoding speech from neural signals, producing interpretable speech outputs that reproduce stably. We carry the same direction forward: recovering thoughts that have formed before a person speaks. That work is part of Dimension Gate’s technical DNA.
fall on the product
This is where our phrase “eight bits per second” comes from. Today's products do not reach the cortex, but the direction is the same: MinuteX first retains the whole part of the speech, and then pushes it where it is vague, half said, and did not have time to say it at that time.
In the past two years, this line has been multimodal: the model has been answered correctly, which does not mean that it has seen the right place. We separated and evaluated "where to look" and "what to answer" separately, and found that the frontier model was often wrong in medical images, but the answer was very similar; the retrieval was also changed from plain text to page image retrieval, so that the evidence could be pointed at.
fall on the product
Any conclusion, as long as it is read from the image, cannot bypass the same question: it must be able to point back to which piece it saw. This is the origin of the set of things traced by the evidence in the product. Can't trace the conclusion, we don't let it appear on the interface.
Being able to reason does not mean being able to understand words
Research
We want the model to think before answering, but this makes instruction compliance worse. This article reveals how the model's excessive reasoning time overturns the original constraints and how to alleviate the cognitive load caused by the model's excessive reasoning.
fall on the product
In a multi-agent system, one agent can act on its own and pass errors down the chain. We therefore make correction explicit: CONFIRM, CORRECT, DISMISS, and UPSERT. This is how frontier research becomes product practice.
