# Silver Medal Solution[29th]: ByT5-XL Custom CPT with Dictionary Data + KL-Anchor

Summary
Model
: ByT5-XL (3.7B) with custom Continued Pre-Training (CPT) on Akkadian domain data (eSAD/CAD dictionaries + onomasticon + published texts) followed by fine-tuning →
LB = 37.6
Key insight
: CV and LB are inversely correlated — fine-tuning consistently improves CV but worsens LB, due to a fundamental distribution mismatch between train and hidden test data
Novel approach
: KL-anchored LoRA adaptation that preserves the base model's translation ability while adapting to domain-specific patterns
Over 40 experiments conducted; sharing technical findings that I hope are useful to the community
Shared notebook
:
[expJ00 FT Infer](https://www.kaggle.com/code/naotonishida/expj00-ft-infer)
(inference) +
[expJ00 FT DL](https://www.kaggle.com/code/naotonishida/expj00-ft-dl)
(model downloader)
Model & Training Pipeline
Base Model
google/byt5-xl (3.7B parameters) — a byte-level T5 model with 36 encoder layers and 12 decoder layers.
Starting from a publicly available CPT checkpoint that had already been pre-trained on Akkadian transliterations for 400 steps.
Stage 1: Continued Pre-Training (CPT)
Task
: ByT5 span corruption (noise_density=0.15, mean_noise_span_length=20 bytes)
CPT Corpus
(79,604 texts, 7.3 MB — transliteration only, no English):
Source
Entries
Notes
published_texts
~8,000 docs
Transliterations from ORACC/CDLI
eSAD dictionary
2,645 entries
Akkadian headwords & transliterations extracted from PDF
CAD dictionary
8,682 entries
Chicago Assyrian Dictionary, all volumes, extracted from 782MB PDF
onomasticon
5,595 entries
Proper noun transliterations
Sentences_Oare
~9,700 texts
Sentence-level transliterations
eBL_Dictionary, OA_Lexicon
~50K+ entries
Additional lexical resources
Hyperparameters
:
1,600 steps, LR = 5e-5, effective batch size = 16
BF16 precision (critical for ByT5 — FP16 causes NaN divergence)
max_length = 512 bytes
Final loss: 1.12
Training time: ~27 min on RTX PRO 6000 (96GB)
Stage 2: Fine-Tuning (FT)
Data: train.csv (1,561 pairs) + synthetic translations (7,660 pairs generated by the base model on published_texts)
5 epochs, LR = 1e-5, max_length = 256
num_beams = 4, length_penalty = 1.3
Best checkpoint selected by CV score (step 800)
Stage 3: Inference
BF16 on T4 16GB (model uses ~7.5GB VRAM)
MBR decoding: 2 beam candidates + 1 sampling candidate, chrF++ consensus
Minimal post-processing only (subscript normalization, gap normalization, whitespace cleanup)
Utility notebook pattern: downloader (internet=ON) → inference (internet=OFF, kernel_sources)
Key Finding: The CV-LB Gap
This was the most important and frustrating discovery of this competition.
Fine-tuning consistently improves CV but worsens LB.
Experiment
Description
CV
LB
CV-LB Gap
Base model (no FT)
ByT5-XL, zero-shot
12.4
38.0
-25.6
+ CPT 400 + FT 5ep
Published texts CPT
40.5
35.0
5.5
+ CPT 1600 + FT (eSAD+CAD, English included)
expJ00
43.7
37.6
6.1
+ CPT 1600 + FT (translit-only)
expJ01
44.2
36.7
7.5
+ LoRA + KL distillation
expK01
—
37.1
—
ByT5-base FT (data aug)
Small model, heavy FT
53.3
29.2
24.1
Root cause
: The training data consists of ~1,500 Old Assyrian commercial texts (contracts, debt notes, trade correspondence), while the hidden test appears to contain sentence-level samples from diverse genres (letters, legal texts, formal documents). Fine-tuning on train data specializes the model for one narrow domain, hurting generalization.
The paradox
: CPT-only (without FT) cannot translate at all — it outputs span corruption responses (
<extra_id_0>
+ Akkadian text). But FT, while enabling translation, inevitably overfits to the training distribution. This structural dilemma defined my entire competition experience.
What Worked
1. Model Scale (580M → 3.7B): +14.4 LB points
The single biggest improvement. ByT5-XL (3.7B) at LB=38.0 vs ByT5-base (580M) at LB=23.6. The larger model's broader pre-training knowledge provides much better zero-shot translation ability.
2. BF16 Precision (mandatory for ByT5)
FP16 causes NaN divergence in ByT5 due to its narrow dynamic range (max ~65,504). BF16 shares FP32's dynamic range (max ~3.4e38) while using only 2 bytes per parameter. This is not optional — FP16 gives Score=0.54 vs BF16's Score=25.95.
3. Translit-Only CPT Data (no English)
Including English dictionary definitions in CPT data hurts performance (CV 43.7 vs 44.2 for translit-only). Span corruption is a monolingual task — English text is noise. Extract only Akkadian headwords and transliterations from dictionaries.
4. Dictionary & Lexicon Data for CPT
Adding eSAD, CAD, and onomasticon transliterations improved the model's Akkadian vocabulary coverage. The CPT corpus grew from ~8K to ~80K texts, and the model learned rarer cuneiform sign readings and proper noun patterns.
5. Minimal Post-Processing
Aggressive post-processing (fraction conversion, PN normalization, Roman numeral conversion, annotation removal) consistently hurt LB scores. The optimal pipeline is minimal: subscript digit conversion + gap normalization + whitespace cleanup only.
6. LoRA + KL Distillation (Anchored Adaptation)
To mitigate the FT-induced LB degradation, I developed a KL-anchored LoRA approach:
Teacher
: Base model (frozen)
Student
: Base model + LoRA (0.24% trainable params)
LoRA targets
: Encoder self-attention (q, v) + Decoder cross-attention (q, v)
Frozen
: Decoder self-attention + lm_head
Loss
:
0.7 × denoise + 0.2 × KL_distill + 0.1 × anchor_CE
This achieved LB=37.1, improving +0.4 over standard FT (36.7), confirming that preserving the base model's translation distribution helps. However, it still couldn't surpass the base model's 38.0.
7. MBR Decoding
Minimum Bayes Risk decoding with chrF++ scoring. For the 3.7B model, a lightweight config works best: 2 beam candidates + 1 sampling candidate (3 total). Larger candidate pools don't justify the compute cost at this model scale.
What Didn't Work
Approach
Result
Why
Aggressive fine-tuning
CV=53.3, LB=29.2
Overfits to train distribution
English-mixed CPT data
CV=43.7 (vs 44.2 translit-only)
English is noise for span corruption
Heavy post-processing
LB -0.2 to -0.4
GT format doesn't match assumed patterns
Flan-T5 Philologist refinement
LB=21.1
Destroys proper nouns (karum → Krum)
Inference parameter tuning
LB ±0.0
Already near-optimal with default settings
CPT-only (no FT)
Cannot translate
Outputs span corruption responses only
NLLB 3.3B
LB=31.9
No Akkadian-specific pre-training
ByT5-small/base models
LB=23-26
Insufficient model capacity
Longer sequences (max512, max1024)
CV drops significantly
Model trained on shorter sequences
Technical Details
ByT5 Span Corruption for CPT
ByT5 operates at the byte level, so the standard T5 span corruption parameters need adjustment. Use
mean_noise_span_length=20
(bytes, not tokens — standard T5 uses 3 tokens ≈ 15 bytes).
noise_density=0.15
works well.
BF16 on T4 16GB
ByT5-XL (3.7B) in BF16 uses only ~7.5GB VRAM, fitting comfortably on T4 with room for beam search KV cache. Save models in BF16 format to avoid FP16 NaN issues during loading:
model = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
model.save_pretrained(output_path)  # Saved as BF16
Utility Notebook Pattern
For internet-disabled competitions:
Downloader notebook
(internet=ON): Download from HuggingFace, save BF16 model to
/kaggle/working/
Inference notebook
(internet=OFF): Reference downloader output via
kernel_sources
, load model from
/kaggle/input/notebooks/<owner>/<slug>/
Use
glob
to discover
config.json
path rather than hardcoding — Kaggle's mount paths can vary.
Synthetic Data Generation
Generated ~7,660 translation pairs by running the base ByT5-XL model on published_texts transliterations. These synthetic pairs, combined with the original train.csv, provided the FT training data. Quality is reasonable since the base model already has strong Akkadian translation ability from its original training.
Reflections
The biggest lesson:
in low-resource translation with distribution mismatch, a strong pre-trained model with minimal adaptation beats heavy fine-tuning.
The base ByT5-XL model (LB=38.0) outperformed every fine-tuned variant I produced. If I could do it over, I would focus more on:
Data curation for CPT that better matches the hidden test distribution
Even lighter adaptation methods (prompt tuning, adapter layers with stronger regularization)
Ensemble of CPT-only models with different data subsets
The CV-LB gap is a known challenge in NLP competitions but rarely this extreme. The key takeaway:
when your CV and LB diverge, trust LB and investigate the distribution gap rather than optimizing CV.
Acknowledgments
Thanks to the Deep Past Initiative organizers for this fascinating competition — working with Akkadian is a unique and intellectually rewarding challenge. Thanks also to the Kaggle community for sharing models and insights throughout the competition.