# 160th Place Solution: +218 Shake Up

Thank you to the competition hosts for this fascinating challenge and to all participants for the insightful discussions throughout.
Private LB: 36
|
+222 shake-up (382nd public -> 160th private)
The core of this solution was
data engineering
-- extracting parallel corpora from scholarly PDFs using Gemini Vision, which expanded the training set from ~1,500 to ~36,750 rows. On the modeling side: mT5-large 4-fold ensemble with MBR decoding.
Data Engineering
With only ~1,500 document-level pairs, the competition was a
data problem
. The AKT publication PDFs contained thousands of transliteration-translation pairs buried in scholarly text.
Sentence-first extraction
: Gemini extracted each PDF page as a list of sentence-level pairs, not a single document blob. From these, I produced two views:
Sentence rows
: individual pairs (the biggest data multiplier -- 1 tablet becomes ~8 rows)
Document rows
: sentences concatenated back into a single row
Both views were concatenated into a single training set.
<eos>
separators were replaced with periods so the model could leverage its English sentence boundary understanding.
Pipeline
![url](assets/3aa411a62cfa40c98920ced0.png)
Final Dataset Breakdown
Source
Rows
AKT volumes (12 vols, sentence+doc split)
23,963
CAD dictionary (13 volumes)
4,059
Secondary scholarly sources
3,630
ICK IV 1998
1,857
Larsen 2002
1,681
Subtotal (publications_csv)
35,190
Competition train.csv
1,561
Total training pool
~36,750
Model & Training
Started with
ByT5-base
(byte-level, no OOV issues) but switched to
mT5
for speed and vocabulary coverage. mT5's subword tokenizer produces ~4x shorter sequences than ByT5's byte-level encoding, enabling ensemble + MBR within Kaggle's 9-hour time limit.
Scale-up
: mT5-base (LB 33.1) -> mT5-large (LB 34.6 single, 36.0 ensemble). BF16 on vast.ai GPUs.
Parameter
Value
Model
mT5-large
Optimizer
AdamW, cosine LR 5e-5, 5% warmup
Batch
8
Epochs
20-40, early stopping patience 5
Label smoothing
0.1
Precision
BF16
Validation
series_group_kfold (5 folds, leave-series-out)
Validation strategy
:
series_group_kfold
(5 folds, leave-series-out) -- a GroupKFold where each fold holds out an entire publication series (e.g., AKT 6a). This prevents series-level data leakage, since documents within the same series share vocabulary, formulaic expressions, and proper nouns.
Inference Pipeline
MBR Decoding
For each source, generated 10 candidates per model:
4 beam search
(num_beams=4, length_penalty=1.08)
6 sampling
(temperatures 0.60/0.80/1.05, 2 each, top_p=0.92)
GeoMeanMBRSelector
picks the candidate with highest average pairwise
sqrt(BLEU * chrF++)
against all others.
4-Fold Ensemble + Char Constraint
5-fold CV but only 4 folds used for ensemble -- 5 models × 10 candidates × ~4,000 sentences exceeded Kaggle's 9-hour time limit. Merged pools from 4 mT5-large folds (40 candidates total).
CharConstraintProcessor
masks disallowed characters during beam search (prevents CJK/Cyrillic generation from mT5).
Preprocessing
Source (Akkadian):
ATF to Unicode:
sz
->
š
,
s,
->
ṣ
,
t,
->
ṭ
, vowel+index to accents (
a2
->
á
,
a3
->
à
)
ḫ
/
Ḫ
->
h
/
H
(confirmed by host)
Gap unification:
[...]
,
[x x x]
,
...
->
<gap>
, with deduplication
Determinative conversion:
(d)
->
{d}
Remove scribal notation (
!
,
?
) and brackets (keep content)
Decimal to fraction:
0.5
->
1/2
Target (English):
Gap normalization, fraction canonicalization (shekel-specific rules from host)
Commodity replacement (
-gold
->
pasallum gold
), grammar marker removal
Post-Processing
Gap unification,
PN
->
<gap>
Commodity replacement (
-gold
->
pasallum gold
)
Fraction canonicalization (shekel-specific rules)
UNK character replacement and fraction rendering normalization to match test format
PN fuzzy matching against onomasticon