Final: Private 38.9 / Public 38.3 (Version 137)
\nSolo entry, single GPU on AutoDL, ByT5-base throughout.
\n ┌──────────────────┐ ┌───────────────────┐ ┌────────────────┐\n │ train.csv │ │ published_texts │ │ Gemini API │\n │ (1500 labeled) │ │ (unlabeled) │ │ augmentation │\n └────────┬─────────┘ └────────┬──────────┘ └───────┬────────┘\n │ │ │\n ▼ ▼ ▼\n ┌──────────────────┐ ┌───────────────────┐ ┌────────────────┐\n │ DeepSeek split │ │ Gemini pseudo- │ │ Synthetic │\n │ + clean/align │ │ label translation │ │ ~7800 samples │\n └────────┬─────────┘ └────────┬──────────┘ └───────┬────────┘\n │ │ │\n └────────────────┬───────┘─────────────────────────┘\n ▼\n ┌─────────────────────┐\n │ Combined ~30k rows │\n │ weighted sampling │\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ ByT5-base training │\n │ 5 epochs, single GPU│\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ Epoch 3-5 avg │\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ Inference │\n │ beam=4, post-proc │\n └─────────────────────┘\nMany samples in train.csv are paragraph-level (multiple sentences concatenated), while the test set is evaluated at sentence level. Used DeepSeek to split paragraphs into sentences, then cleaned out samples with abnormal IO length ratios and empty fields. This produced ~6750 rows of gold data and gave ~+2 points improvement.
\nUsed Gemini-3 Pro to split and translate published_texts.csv at sentence level. Iterated over multiple rounds, expanding to ~40k rows of silver data. This was the single biggest improvement at ~+4 points.
\nFor each paragraph in the training set, asked Gemini to generate a semantically related but different paragraph (e.g. reply letter, counter-claim, follow-up transaction) as a transliteration-translation pair. Produced ~7800 augmented samples, contributing ~+0.5 points.
\nModel: google/byt5-base (~580M params). Byte-level tokenization, naturally suited for the special characters and diacritics in Akkadian transliteration.
\n~ as placeholder, restored to <gap> at inference<big gap> → <gap>, remove parenthetical content, number normalization| Method | \nImprovement | \nNotes | \n
|---|---|---|
| Subscript removal | \n+0.7 | \nSubscripts only mark cuneiform sign variants, not pronunciation | \n
| Sentence splitting | \n+2.0 | \nParagraph → sentence level, aligned with test set granularity | \n
| Published_texts pseudo-labels (Gemini) | \n+4.0 | \n~40k silver rows, single biggest gain | \n
| Gold data cleaning + val set recovery | \n+0.5 | \nIO ratio filtering, folding val set into training | \n
| Gemini synthetic augmentation + weighted training | \n+0.5 | \n~7800 augmented samples, gold/silver weighting | \n
| Checkpoint averaging | \n+0.1 | \nEpoch 3-5 parameter averaging, reduces variance | \n
Overall trajectory: baseline 31.1 → splitting 33.8 → pseudo-labels 37.5 → refinement 38.3 → final 38.9 Private (V137)
\nFinal: Private 38.9 / Public 38.3 (Version 137) | ~140 submissions, 50+ experiments, 4 weeks, solo
", "messages": [ { "id": 3427555, "postDate": "2026-03-24T08:34:47.527Z", "content": "Final: Private 38.9 / Public 38.3 (Version 137)
\nSolo entry, single GPU on AutoDL, ByT5-base throughout.
\n ┌──────────────────┐ ┌───────────────────┐ ┌────────────────┐\n │ train.csv │ │ published_texts │ │ Gemini API │\n │ (1500 labeled) │ │ (unlabeled) │ │ augmentation │\n └────────┬─────────┘ └────────┬──────────┘ └───────┬────────┘\n │ │ │\n ▼ ▼ ▼\n ┌──────────────────┐ ┌───────────────────┐ ┌────────────────┐\n │ DeepSeek split │ │ Gemini pseudo- │ │ Synthetic │\n │ + clean/align │ │ label translation │ │ ~7800 samples │\n └────────┬─────────┘ └────────┬──────────┘ └───────┬────────┘\n │ │ │\n └────────────────┬───────┘─────────────────────────┘\n ▼\n ┌─────────────────────┐\n │ Combined ~30k rows │\n │ weighted sampling │\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ ByT5-base training │\n │ 5 epochs, single GPU│\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ Epoch 3-5 avg │\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ Inference │\n │ beam=4, post-proc │\n └─────────────────────┘\nMany samples in train.csv are paragraph-level (multiple sentences concatenated), while the test set is evaluated at sentence level. Used DeepSeek to split paragraphs into sentences, then cleaned out samples with abnormal IO length ratios and empty fields. This produced ~6750 rows of gold data and gave ~+2 points improvement.
\nUsed Gemini-3 Pro to split and translate published_texts.csv at sentence level. Iterated over multiple rounds, expanding to ~40k rows of silver data. This was the single biggest improvement at ~+4 points.
\nFor each paragraph in the training set, asked Gemini to generate a semantically related but different paragraph (e.g. reply letter, counter-claim, follow-up transaction) as a transliteration-translation pair. Produced ~7800 augmented samples, contributing ~+0.5 points.
\nModel: google/byt5-base (~580M params). Byte-level tokenization, naturally suited for the special characters and diacritics in Akkadian transliteration.
\n~ as placeholder, restored to <gap> at inference<big gap> → <gap>, remove parenthetical content, number normalization| Method | \nImprovement | \nNotes | \n
|---|---|---|
| Subscript removal | \n+0.7 | \nSubscripts only mark cuneiform sign variants, not pronunciation | \n
| Sentence splitting | \n+2.0 | \nParagraph → sentence level, aligned with test set granularity | \n
| Published_texts pseudo-labels (Gemini) | \n+4.0 | \n~40k silver rows, single biggest gain | \n
| Gold data cleaning + val set recovery | \n+0.5 | \nIO ratio filtering, folding val set into training | \n
| Gemini synthetic augmentation + weighted training | \n+0.5 | \n~7800 augmented samples, gold/silver weighting | \n
| Checkpoint averaging | \n+0.1 | \nEpoch 3-5 parameter averaging, reduces variance | \n
Overall trajectory: baseline 31.1 → splitting 33.8 → pseudo-labels 37.5 → refinement 38.3 → final 38.9 Private (V137)
\nFinal: Private 38.9 / Public 38.3 (Version 137) | ~140 submissions, 50+ experiments, 4 weeks, solo
", "rawMarkdown": "# ByT5-Base + Gemini Augmentation, No Ensemble\n\n**Final: Private 38.9 / Public 38.3 (Version 137)**\n\nSolo entry, single GPU on AutoDL, ByT5-base throughout.\n\n---\n\n## 1. Overview\n\n```\n ┌──────────────────┐ ┌───────────────────┐ ┌────────────────┐\n │ train.csv │ │ published_texts │ │ Gemini API │\n │ (1500 labeled) │ │ (unlabeled) │ │ augmentation │\n └────────┬─────────┘ └────────┬──────────┘ └───────┬────────┘\n │ │ │\n ▼ ▼ ▼\n ┌──────────────────┐ ┌───────────────────┐ ┌────────────────┐\n │ DeepSeek split │ │ Gemini pseudo- │ │ Synthetic │\n │ + clean/align │ │ label translation │ │ ~7800 samples │\n └────────┬─────────┘ └────────┬──────────┘ └───────┬────────┘\n │ │ │\n └────────────────┬───────┘─────────────────────────┘\n ▼\n ┌─────────────────────┐\n │ Combined ~30k rows │\n │ weighted sampling │\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ ByT5-base training │\n │ 5 epochs, single GPU│\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ Epoch 3-5 avg │\n └──────────┬──────────┘\n ▼\n ┌─────────────────────┐\n │ Inference │\n │ beam=4, post-proc │\n └─────────────────────┘\n```\n\n## 2. Data Engineering\n\n### 2.1 Sentence Splitting\n\nMany samples in train.csv are paragraph-level (multiple sentences concatenated), while the test set is evaluated at sentence level. Used **DeepSeek** to split paragraphs into sentences, then cleaned out samples with abnormal IO length ratios and empty fields. This produced ~**6750 rows** of gold data and gave ~+2 points improvement.\n\n### 2.2 Pseudo-labeling Published Texts\n\nUsed **Gemini-3 Pro** to split and translate published_texts.csv at sentence level. Iterated over multiple rounds, expanding to ~40k rows of silver data. This was the single biggest improvement at ~**+4 points**.\n\n### 2.3 Gemini Synthetic Augmentation\n\nFor each paragraph in the training set, asked Gemini to generate a semantically related but different paragraph (e.g. reply letter, counter-claim, follow-up transaction) as a transliteration-translation pair. Produced ~**7800** augmented samples, contributing ~+0.5 points.\n\n### 2.4 Preprocessing\n\n- **Subscript character removal:** Akkadian transliterations contain Unicode subscript characters (e.g. ₂, ₃). Based on information from Gemini, these subscripts only mark different cuneiform sign variants in the original writing and do not affect pronunciation — and Akkadian word meaning depends solely on pronunciation. Removing them improved training results. Not sure how other participants handled this.\n- Other preprocessing (gap normalization, parenthetical removal, number/fraction normalization, etc.) followed ideas shared in public discussions.\n\n## 3. Training\n\nModel: **google/byt5-base** (~580M params). Byte-level tokenization, naturally suited for the special characters and diacritics in Akkadian transliteration.\n\n- **Optimizer:** AdamW, weight_decay=0.01\n- **LR:** 3e-4, Warmup (6%) + Cosine Decay\n- **Precision:** BF16\n- **Max Length:** 512\n- **Epochs:** 5\n- **Weighted training:** gold data weighted higher than silver data in loss computation\n- **Checkpoint averaging:** averaged parameters from epochs 3-5. Observed ~+0.1 LB improvement in a couple of A/B tests. Not sure if optimal, but makes sense for reducing variance in epoch selection.\n\n## 4. Inference & Post-processing\n\n- Beam search, num_beams=4\n- Gap compression: trained with `~` as placeholder, restored to `