Shared notebook: expJ00 FT Infer (inference) + expJ00 FT DL (model downloader)
\ngoogle/byt5-xl (3.7B parameters) — a byte-level T5 model with 36 encoder layers and 12 decoder layers.\nStarting from a publicly available CPT checkpoint that had already been pre-trained on Akkadian transliterations for 400 steps.
\nTask: ByT5 span corruption (noise_density=0.15, mean_noise_span_length=20 bytes)
\nCPT Corpus (79,604 texts, 7.3 MB — transliteration only, no English):
\n| Source | \nEntries | \nNotes | \n
|---|---|---|
| published_texts | \n~8,000 docs | \nTransliterations from ORACC/CDLI | \n
| eSAD dictionary | \n2,645 entries | \nAkkadian headwords & transliterations extracted from PDF | \n
| CAD dictionary | \n8,682 entries | \nChicago Assyrian Dictionary, all volumes, extracted from 782MB PDF | \n
| onomasticon | \n5,595 entries | \nProper noun transliterations | \n
| Sentences_Oare | \n~9,700 texts | \nSentence-level transliterations | \n
| eBL_Dictionary, OA_Lexicon | \n~50K+ entries | \nAdditional lexical resources | \n
Hyperparameters:
\nThis was the most important and frustrating discovery of this competition. Fine-tuning consistently improves CV but worsens LB.
\n| Experiment | \nDescription | \nCV | \nLB | \nCV-LB Gap | \n
|---|---|---|---|---|
| Base model (no FT) | \nByT5-XL, zero-shot | \n12.4 | \n38.0 | \n-25.6 | \n
| + CPT 400 + FT 5ep | \nPublished texts CPT | \n40.5 | \n35.0 | \n5.5 | \n
| + CPT 1600 + FT (eSAD+CAD, English included) | \nexpJ00 | \n43.7 | \n37.6 | \n6.1 | \n
| + CPT 1600 + FT (translit-only) | \nexpJ01 | \n44.2 | \n36.7 | \n7.5 | \n
| + LoRA + KL distillation | \nexpK01 | \n— | \n37.1 | \n— | \n
| ByT5-base FT (data aug) | \nSmall model, heavy FT | \n53.3 | \n29.2 | \n24.1 | \n
Root cause: The training data consists of ~1,500 Old Assyrian commercial texts (contracts, debt notes, trade correspondence), while the hidden test appears to contain sentence-level samples from diverse genres (letters, legal texts, formal documents). Fine-tuning on train data specializes the model for one narrow domain, hurting generalization.
\nThe paradox: CPT-only (without FT) cannot translate at all — it outputs span corruption responses (<extra_id_0> + Akkadian text). But FT, while enabling translation, inevitably overfits to the training distribution. This structural dilemma defined my entire competition experience.
The single biggest improvement. ByT5-XL (3.7B) at LB=38.0 vs ByT5-base (580M) at LB=23.6. The larger model's broader pre-training knowledge provides much better zero-shot translation ability.
\nFP16 causes NaN divergence in ByT5 due to its narrow dynamic range (max ~65,504). BF16 shares FP32's dynamic range (max ~3.4e38) while using only 2 bytes per parameter. This is not optional — FP16 gives Score=0.54 vs BF16's Score=25.95.
\nIncluding English dictionary definitions in CPT data hurts performance (CV 43.7 vs 44.2 for translit-only). Span corruption is a monolingual task — English text is noise. Extract only Akkadian headwords and transliterations from dictionaries.
\nAdding eSAD, CAD, and onomasticon transliterations improved the model's Akkadian vocabulary coverage. The CPT corpus grew from ~8K to ~80K texts, and the model learned rarer cuneiform sign readings and proper noun patterns.
\nAggressive post-processing (fraction conversion, PN normalization, Roman numeral conversion, annotation removal) consistently hurt LB scores. The optimal pipeline is minimal: subscript digit conversion + gap normalization + whitespace cleanup only.
\nTo mitigate the FT-induced LB degradation, I developed a KL-anchored LoRA approach:
\n0.7 × denoise + 0.2 × KL_distill + 0.1 × anchor_CEThis achieved LB=37.1, improving +0.4 over standard FT (36.7), confirming that preserving the base model's translation distribution helps. However, it still couldn't surpass the base model's 38.0.
\nMinimum Bayes Risk decoding with chrF++ scoring. For the 3.7B model, a lightweight config works best: 2 beam candidates + 1 sampling candidate (3 total). Larger candidate pools don't justify the compute cost at this model scale.
\n| Approach | \nResult | \nWhy | \n
|---|---|---|
| Aggressive fine-tuning | \nCV=53.3, LB=29.2 | \nOverfits to train distribution | \n
| English-mixed CPT data | \nCV=43.7 (vs 44.2 translit-only) | \nEnglish is noise for span corruption | \n
| Heavy post-processing | \nLB -0.2 to -0.4 | \nGT format doesn't match assumed patterns | \n
| Flan-T5 Philologist refinement | \nLB=21.1 | \nDestroys proper nouns (karum → Krum) | \n
| Inference parameter tuning | \nLB ±0.0 | \nAlready near-optimal with default settings | \n
| CPT-only (no FT) | \nCannot translate | \nOutputs span corruption responses only | \n
| NLLB 3.3B | \nLB=31.9 | \nNo Akkadian-specific pre-training | \n
| ByT5-small/base models | \nLB=23-26 | \nInsufficient model capacity | \n
| Longer sequences (max512, max1024) | \nCV drops significantly | \nModel trained on shorter sequences | \n
ByT5 operates at the byte level, so the standard T5 span corruption parameters need adjustment. Use mean_noise_span_length=20 (bytes, not tokens — standard T5 uses 3 tokens ≈ 15 bytes). noise_density=0.15 works well.
ByT5-XL (3.7B) in BF16 uses only ~7.5GB VRAM, fitting comfortably on T4 with room for beam search KV cache. Save models in BF16 format to avoid FP16 NaN issues during loading:
\nmodel = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)\nmodel.save_pretrained(output_path) # Saved as BF16\nFor internet-disabled competitions:
\n/kaggle/working/kernel_sources, load model from /kaggle/input/notebooks/<owner>/<slug>/Use glob to discover config.json path rather than hardcoding — Kaggle's mount paths can vary.
Generated ~7,660 translation pairs by running the base ByT5-XL model on published_texts transliterations. These synthetic pairs, combined with the original train.csv, provided the FT training data. Quality is reasonable since the base model already has strong Akkadian translation ability from its original training.
\nThe biggest lesson: in low-resource translation with distribution mismatch, a strong pre-trained model with minimal adaptation beats heavy fine-tuning. The base ByT5-XL model (LB=38.0) outperformed every fine-tuned variant I produced. If I could do it over, I would focus more on:
\nThe CV-LB gap is a known challenge in NLP competitions but rarely this extreme. The key takeaway: when your CV and LB diverge, trust LB and investigate the distribution gap rather than optimizing CV.
\nThanks to the Deep Past Initiative organizers for this fascinating competition — working with Akkadian is a unique and intellectually rewarding challenge. Thanks also to the Kaggle community for sharing models and insights throughout the competition.
", "messages": [ { "id": 3428358, "postDate": "2026-03-25T10:51:15.030Z", "content": "Shared notebook: expJ00 FT Infer (inference) + expJ00 FT DL (model downloader)
\ngoogle/byt5-xl (3.7B parameters) — a byte-level T5 model with 36 encoder layers and 12 decoder layers.\nStarting from a publicly available CPT checkpoint that had already been pre-trained on Akkadian transliterations for 400 steps.
\nTask: ByT5 span corruption (noise_density=0.15, mean_noise_span_length=20 bytes)
\nCPT Corpus (79,604 texts, 7.3 MB — transliteration only, no English):
\n| Source | \nEntries | \nNotes | \n
|---|---|---|
| published_texts | \n~8,000 docs | \nTransliterations from ORACC/CDLI | \n
| eSAD dictionary | \n2,645 entries | \nAkkadian headwords & transliterations extracted from PDF | \n
| CAD dictionary | \n8,682 entries | \nChicago Assyrian Dictionary, all volumes, extracted from 782MB PDF | \n
| onomasticon | \n5,595 entries | \nProper noun transliterations | \n
| Sentences_Oare | \n~9,700 texts | \nSentence-level transliterations | \n
| eBL_Dictionary, OA_Lexicon | \n~50K+ entries | \nAdditional lexical resources | \n
Hyperparameters:
\nThis was the most important and frustrating discovery of this competition. Fine-tuning consistently improves CV but worsens LB.
\n| Experiment | \nDescription | \nCV | \nLB | \nCV-LB Gap | \n
|---|---|---|---|---|
| Base model (no FT) | \nByT5-XL, zero-shot | \n12.4 | \n38.0 | \n-25.6 | \n
| + CPT 400 + FT 5ep | \nPublished texts CPT | \n40.5 | \n35.0 | \n5.5 | \n
| + CPT 1600 + FT (eSAD+CAD, English included) | \nexpJ00 | \n43.7 | \n37.6 | \n6.1 | \n
| + CPT 1600 + FT (translit-only) | \nexpJ01 | \n44.2 | \n36.7 | \n7.5 | \n
| + LoRA + KL distillation | \nexpK01 | \n— | \n37.1 | \n— | \n
| ByT5-base FT (data aug) | \nSmall model, heavy FT | \n53.3 | \n29.2 | \n24.1 | \n
Root cause: The training data consists of ~1,500 Old Assyrian commercial texts (contracts, debt notes, trade correspondence), while the hidden test appears to contain sentence-level samples from diverse genres (letters, legal texts, formal documents). Fine-tuning on train data specializes the model for one narrow domain, hurting generalization.
\nThe paradox: CPT-only (without FT) cannot translate at all — it outputs span corruption responses (<extra_id_0> + Akkadian text). But FT, while enabling translation, inevitably overfits to the training distribution. This structural dilemma defined my entire competition experience.
The single biggest improvement. ByT5-XL (3.7B) at LB=38.0 vs ByT5-base (580M) at LB=23.6. The larger model's broader pre-training knowledge provides much better zero-shot translation ability.
\nFP16 causes NaN divergence in ByT5 due to its narrow dynamic range (max ~65,504). BF16 shares FP32's dynamic range (max ~3.4e38) while using only 2 bytes per parameter. This is not optional — FP16 gives Score=0.54 vs BF16's Score=25.95.
\nIncluding English dictionary definitions in CPT data hurts performance (CV 43.7 vs 44.2 for translit-only). Span corruption is a monolingual task — English text is noise. Extract only Akkadian headwords and transliterations from dictionaries.
\nAdding eSAD, CAD, and onomasticon transliterations improved the model's Akkadian vocabulary coverage. The CPT corpus grew from ~8K to ~80K texts, and the model learned rarer cuneiform sign readings and proper noun patterns.
\nAggressive post-processing (fraction conversion, PN normalization, Roman numeral conversion, annotation removal) consistently hurt LB scores. The optimal pipeline is minimal: subscript digit conversion + gap normalization + whitespace cleanup only.
\nTo mitigate the FT-induced LB degradation, I developed a KL-anchored LoRA approach:
\n0.7 × denoise + 0.2 × KL_distill + 0.1 × anchor_CEThis achieved LB=37.1, improving +0.4 over standard FT (36.7), confirming that preserving the base model's translation distribution helps. However, it still couldn't surpass the base model's 38.0.
\nMinimum Bayes Risk decoding with chrF++ scoring. For the 3.7B model, a lightweight config works best: 2 beam candidates + 1 sampling candidate (3 total). Larger candidate pools don't justify the compute cost at this model scale.
\n| Approach | \nResult | \nWhy | \n
|---|---|---|
| Aggressive fine-tuning | \nCV=53.3, LB=29.2 | \nOverfits to train distribution | \n
| English-mixed CPT data | \nCV=43.7 (vs 44.2 translit-only) | \nEnglish is noise for span corruption | \n
| Heavy post-processing | \nLB -0.2 to -0.4 | \nGT format doesn't match assumed patterns | \n
| Flan-T5 Philologist refinement | \nLB=21.1 | \nDestroys proper nouns (karum → Krum) | \n
| Inference parameter tuning | \nLB ±0.0 | \nAlready near-optimal with default settings | \n
| CPT-only (no FT) | \nCannot translate | \nOutputs span corruption responses only | \n
| NLLB 3.3B | \nLB=31.9 | \nNo Akkadian-specific pre-training | \n
| ByT5-small/base models | \nLB=23-26 | \nInsufficient model capacity | \n
| Longer sequences (max512, max1024) | \nCV drops significantly | \nModel trained on shorter sequences | \n
ByT5 operates at the byte level, so the standard T5 span corruption parameters need adjustment. Use mean_noise_span_length=20 (bytes, not tokens — standard T5 uses 3 tokens ≈ 15 bytes). noise_density=0.15 works well.
ByT5-XL (3.7B) in BF16 uses only ~7.5GB VRAM, fitting comfortably on T4 with room for beam search KV cache. Save models in BF16 format to avoid FP16 NaN issues during loading:
\nmodel = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)\nmodel.save_pretrained(output_path) # Saved as BF16\nFor internet-disabled competitions:
\n/kaggle/working/kernel_sources, load model from /kaggle/input/notebooks/<owner>/<slug>/Use glob to discover config.json path rather than hardcoding — Kaggle's mount paths can vary.
Generated ~7,660 translation pairs by running the base ByT5-XL model on published_texts transliterations. These synthetic pairs, combined with the original train.csv, provided the FT training data. Quality is reasonable since the base model already has strong Akkadian translation ability from its original training.
\nThe biggest lesson: in low-resource translation with distribution mismatch, a strong pre-trained model with minimal adaptation beats heavy fine-tuning. The base ByT5-XL model (LB=38.0) outperformed every fine-tuned variant I produced. If I could do it over, I would focus more on:
\nThe CV-LB gap is a known challenge in NLP competitions but rarely this extreme. The key takeaway: when your CV and LB diverge, trust LB and investigate the distribution gap rather than optimizing CV.
\nThanks to the Deep Past Initiative organizers for this fascinating competition — working with Akkadian is a unique and intellectually rewarding challenge. Thanks also to the Kaggle community for sharing models and insights throughout the competition.
", "rawMarkdown": "## Summary\n\n- **Model**: ByT5-XL (3.7B) with custom Continued Pre-Training (CPT) on Akkadian domain data (eSAD/CAD dictionaries + onomasticon + published texts) followed by fine-tuning → **LB = 37.6**\n- **Key insight**: CV and LB are inversely correlated — fine-tuning consistently improves CV but worsens LB, due to a fundamental distribution mismatch between train and hidden test data\n- **Novel approach**: KL-anchored LoRA adaptation that preserves the base model's translation ability while adapting to domain-specific patterns\n- Over 40 experiments conducted; sharing technical findings that I hope are useful to the community\n\n**Shared notebook**: [expJ00 FT Infer](https://www.kaggle.com/code/naotonishida/expj00-ft-infer) (inference) + [expJ00 FT DL](https://www.kaggle.com/code/naotonishida/expj00-ft-dl) (model downloader)\n\n## Model & Training Pipeline\n\n### Base Model\n\ngoogle/byt5-xl (3.7B parameters) — a byte-level T5 model with 36 encoder layers and 12 decoder layers.\nStarting from a publicly available CPT checkpoint that had already been pre-trained on Akkadian transliterations for 400 steps.\n\n### Stage 1: Continued Pre-Training (CPT)\n\n**Task**: ByT5 span corruption (noise_density=0.15, mean_noise_span_length=20 bytes)\n\n**CPT Corpus** (79,604 texts, 7.3 MB — transliteration only, no English):\n\n| Source | Entries | Notes |\n|--------|---------|-------|\n| published_texts | ~8,000 docs | Transliterations from ORACC/CDLI |\n| eSAD dictionary | 2,645 entries | Akkadian headwords & transliterations extracted from PDF |\n| CAD dictionary | 8,682 entries | Chicago Assyrian Dictionary, all volumes, extracted from 782MB PDF |\n| onomasticon | 5,595 entries | Proper noun transliterations |\n| Sentences_Oare | ~9,700 texts | Sentence-level transliterations |\n| eBL_Dictionary, OA_Lexicon | ~50K+ entries | Additional lexical resources |\n\n**Hyperparameters**:\n- 1,600 steps, LR = 5e-5, effective batch size = 16\n- BF16 precision (critical for ByT5 — FP16 causes NaN divergence)\n- max_length = 512 bytes\n- Final loss: 1.12\n- Training time: ~27 min on RTX PRO 6000 (96GB)\n\n### Stage 2: Fine-Tuning (FT)\n\n- Data: train.csv (1,561 pairs) + synthetic translations (7,660 pairs generated by the base model on published_texts)\n- 5 epochs, LR = 1e-5, max_length = 256\n- num_beams = 4, length_penalty = 1.3\n- Best checkpoint selected by CV score (step 800)\n\n### Stage 3: Inference\n\n- BF16 on T4 16GB (model uses ~7.5GB VRAM)\n- MBR decoding: 2 beam candidates + 1 sampling candidate, chrF++ consensus\n- Minimal post-processing only (subscript normalization, gap normalization, whitespace cleanup)\n- Utility notebook pattern: downloader (internet=ON) → inference (internet=OFF, kernel_sources)\n\n## Key Finding: The CV-LB Gap\n\nThis was the most important and frustrating discovery of this competition. **Fine-tuning consistently improves CV but worsens LB.**\n\n| Experiment | Description | CV | LB | CV-LB Gap |\n|------------|-------------|----|----|-----------|\n| Base model (no FT) | ByT5-XL, zero-shot | 12.4 | **38.0** | -25.6 |\n| + CPT 400 + FT 5ep | Published texts CPT | 40.5 | 35.0 | 5.5 |\n| + CPT 1600 + FT (eSAD+CAD, English included) | expJ00 | 43.7 | **37.6** | 6.1 |\n| + CPT 1600 + FT (translit-only) | expJ01 | **44.2** | 36.7 | 7.5 |\n| + LoRA + KL distillation | expK01 | — | 37.1 | — |\n| ByT5-base FT (data aug) | Small model, heavy FT | **53.3** | 29.2 | 24.1 |\n\n**Root cause**: The training data consists of ~1,500 Old Assyrian commercial texts (contracts, debt notes, trade correspondence), while the hidden test appears to contain sentence-level samples from diverse genres (letters, legal texts, formal documents). Fine-tuning on train data specializes the model for one narrow domain, hurting generalization.\n\n**The paradox**: CPT-only (without FT) cannot translate at all — it outputs span corruption responses (`