# [7th place solution] Mamba + GRU + KenLM (with code)

[7th Place Solution] Mamba + GRU + KenLM (with code)
First off, thanks to Nicholas Card, the UC Davis Neuroprosthetics Lab, and the organizers for this fascinating competition. It has always been my dream to explore how ML can apply to clinical settings. Huge thanks to my teammate
[@Kyle](https://www.kaggle.com/Kyle)
Hui.
🔗 Resources
📖 Detailed Writeup (Medium):
[Click Here](https://medium.com/@jackson3b04/7th-place-solution-mamba-gru-kenlm-with-code-brain-to-text-25-00f1c69dcd0d)
💻 Full Code (GitHub):
[Click Here](https://github.com/greentree327/brain-to-text-mamba-decoder)
If you find this writeup or the code useful, please support us by
starring the GitHub repo
and leaving a
clap/comment on Medium
!
TL;DR
Our solution is a hybrid ensemble of
SoftWindow Bi-Mamba
and
GRU
models. We focused heavily on memory efficiency (fitting a custom KenLM into 19GB RAM vs 300GB baseline) and a
Context-Aware Inference Pipeline (LISA)
that gates LLM rescoring based on signal confidence.
1. The Challenge
We faced severe overfitting (0.0 training loss vs high validation loss) and strict compute limits (Single A100 on Colab). We needed a model light on RAM but heavy on regularization.
2. The Solution: Hybrid Architecture
We found
Mamba
captured long-range semantic context while
GRU
stabilized short-range acoustic modeling. Their error patterns were highly uncorrelated.
Bi-directional Mamba:
3-layer input compression with heavy dropout.
Soft Sliding Window:
Biased SSM parameters to prioritize short-term memory.
Temporal Consistency Loss:
A custom loss minimizing the difference between adjacent day matrices to handle neural drift.
3. Memory Optimization: Custom KenLM
Standard 5-gram models (~300GB) were impossible. We compiled a custom
flashlight decoder
with
kTrieMaxLabel=14
, fitting a robust 4-gram model into just
19GB RAM
.
4. Inference Pipeline (LISA)
We didn't rely on a single model. Our
LISA (LLM-Integrated Scoring Aggregation)
pipeline routes samples dynamically:
Logit Averaging:
We averaged logits within model families to create constructive interference (signal amplification).
Gating:
We calculated a normalized n-gram score for every sentence.
High Confidence (≥ -3.76):
Routed to
Mistral-7B
for semantic rescoring.
Low Confidence (< -3.76):
Bypassed the LLM to avoid hallucinations on random sequences.
TTA:
Batched Test-Time Adaptation on high-confidence predictions.
5. Results
This approach allowed us to decode both coherent speech and random patterns with high fidelity, achieving 7th place despite hardware constraints.
Again, if you enjoyed this summary, please check out the full detailed breakdown on
[Medium](https://medium.com/@jackson3b04/7th-place-solution-mamba-gru-kenlm-with-code-brain-to-text-25-00f1c69dcd0d)
and star the
[GitHub repo](https://github.com/greentree327/brain-to-text-mamba-decoder)
.