# Ensemble with Weighted MBR for Akkadian Translation

Overview
Our final solution was based on an ensemble of two ByT5 translation models combined with a Minimum Bayes Risk (MBR) decoding framework.
Instead of selecting the highest-probability output from a single model, we generated a diverse pool of translation candidates from multiple decoding strategies and selected the candidate with the strongest consensus across the pool.
The key idea was that candidate diversity and consensus-based selection produced more robust translations than relying solely on beam search.
Competition Challenges
The Deep Past Initiative Machine Translation task involves translating Akkadian transliterations into English.
Several characteristics made the problem challenging:
Low-resource parallel data.
Significant transliteration variability.
Damaged tablet markers and gap annotations.
Special symbols and determinatives.
Fractional and commodity notations.
OCR and formatting inconsistencies.
Because of these issues, preprocessing and postprocessing played an important role in overall performance.
Data Normalization
Before inference, transliterations were normalized to reduce notation variability.
The preprocessing pipeline included:
ASCII transliteration normalization.
Unicode diacritic conversion.
Gap marker standardization.
Determinative normalization.
Fraction normalization.
Whitespace cleanup.
Symbol harmonization.
Examples:
sz → š
s, → ṣ
t, → ṭ
Gap indicators from multiple formats were unified into a single
<gap>
token.
This reduced fragmentation in the input space and improved consistency between similar texts.
Model Ensemble
The final system used two independently trained ByT5 models:
Model A: akkadian-optimized
Model B: akkadian-mbr
For every test sample, both models generated candidate translations.
Using multiple models reduced model-specific biases and increased candidate diversity.
Candidate Generation
Instead of generating a single translation, each model produced multiple candidates.
Beam Search
For each model:
num_beams = 8
beam candidates = 4
Beam search provided high-probability translations.
Multi-Temperature Sampling
Additional candidates were generated using stochastic decoding:
Temperature = 0.60
Temperature = 0.80
Temperature = 1.05
Two samples were generated for each temperature.
This produced:
4 beam candidates
6 sampled candidates
------------------
10 candidates/model
With two models:
20 candidates/sample
(before deduplication)
The purpose was to create a heterogeneous candidate pool from which MBR could select the most reliable translation.
Postprocessing
After generation, translations were normalized using a rule-based postprocessing pipeline.
Important corrections included:
Host-recommended fixes
5/12 shekel → ⅓ shekel 15 grains
Preservation of parentheses
Curly quotes converted to straight quotes
ḫ/Ḫ normalized to h/H
Noise removal
Duplicate words
Duplicate punctuation
Stray OCR marks
Invalid markup symbols
Special care was taken to preserve meaningful content while removing formatting noise.
Minimum Bayes Risk Decoding
The core of the solution was a weighted MBR selection procedure.
Instead of selecting the candidate with the highest model probability, we selected the candidate with the highest agreement with all other candidates.
Candidate Similarity Metrics
Each candidate pair was evaluated using:
Metric
Weight
chrF++
0.55
BLEU
0.25
Jaccard similarity
0.20
Additionally, a length-consistency bonus was applied.
The final score was:
MBR Score =
0.55 × chrF++
+ 0.25 × BLEU
+ 0.20 × Jaccard
+ Length Bonus
The candidate with the highest consensus score was selected as the final translation.
Why MBR Worked
Beam search often produced several translations that were very similar but differed in wording.
Sampling generated more diverse alternatives.
MBR acted as a consensus mechanism:
weak outliers were rejected,
unstable generations were penalized,
translations supported by multiple decoding paths were preferred.
This generally improved robustness compared with selecting the top beam candidate directly.
Computational Optimizations
To keep inference efficient:
BF16 mixed precision was enabled.
BetterTransformer acceleration was used when available.
Bucket batching reduced padding overhead.
Adaptive beam sizing was applied for shorter inputs.
Checkpointing was used during large-scale inference.
These optimizations allowed larger candidate pools without exceeding resource constraints.
Final Pipeline
Input Transliteration
          ↓
Preprocessing
          ↓
Model A (ByT5)
          ↓
Beam + Sampling Candidates
          ↓

Model B (ByT5)
          ↓
Beam + Sampling Candidates
          ↓

Candidate Pool Merge
          ↓
Postprocessing
          ↓
Weighted MBR Selection
          ↓
Final Translation
Lessons Learned
The most important lesson from this competition was that candidate diversity can be as valuable as model quality.
The largest gains came from:
Strong preprocessing.
Diverse candidate generation.
Consensus-based MBR decoding.
Rather than relying on a single best prediction, aggregating evidence from multiple models and decoding strategies produced more stable translations and better overall performance.