Final scores: Public 24.87 / Private 24.41 (NLLB-200 1.3B)
\nThis write-up focuses primarily on debugging, preprocessing, and decoding diagnostics rather than leaderboard optimization alone.
\nNot a top finish, but I learned a lot about what breaks silently in low-resource MT pipelines. Sharing the diagnostics because I think some of these findings are useful regardless of approach.
\nr=32, alpha=64)q_proj, k_proj, v_proj, out_proj, fc1, fc2transformers==4.38.2 (pinned — more on why below)<gap> Token Was Being Silently Deleted During DecodeThis turned out to be the most impactful bug in the pipeline.
\nThe <gap> token appears in ~50% of validation references. My model was predicting effectively zero gaps for weeks. I assumed the model simply had not learned the token yet.
Then I ran this diagnostic:
\ngap_id = tokenizer.convert_tokens_to_ids('<gap>')\nprint('Is special token:', gap_id in tokenizer.all_special_ids)\n\n# Encode a test sentence with <gap>\ntest = \"1 mina of silver <gap> Puzur-Aššur <gap> has paid\"\n\nenc = tokenizer(test, return_tensors='pt')\n\ndec_skip = tokenizer.decode(\n enc['input_ids'][0],\n skip_special_tokens=True\n)\n\ndec_keep = tokenizer.decode(\n enc['input_ids'][0],\n skip_special_tokens=False\n)\n\nprint('With skip_special_tokens=True :', dec_skip)\nprint('With skip_special_tokens=False:', dec_keep)\nOutput:
\nIs special token: True\n\nWith skip_special_tokens=True :\n1 mina of silver Puzur-Aššur has paid\n\nWith skip_special_tokens=False:\n1 mina of silver <gap> Puzur-Aššur <gap> has paid\nThe model was predicting gaps correctly during training.
\nBut because <gap> had been registered as an additional_special_token, every predicted gap was being removed during decoding with:
skip_special_tokens=True\nThe evaluation pipeline never saw a single gap token.
\n# BROKEN\n# <gap> added as special token -> stripped during decode\n\nspecial_tokens = {\n 'additional_special_tokens': [SRC_LANG, '<gap>']\n}\n\ntokenizer.add_special_tokens(special_tokens)\n\n\n# FIXED\n# <gap> added as regular token -> survives decode\n\ntokenizer.add_tokens(['<gap>'] + extra_tokens)\n\nspecial_tokens = {\n 'additional_special_tokens': [SRC_LANG]\n}\n\ntokenizer.add_special_tokens(special_tokens)\n\n\n# FIXED DECODE\n# manually strip only structural tokens\n\npred_texts = tokenizer.batch_decode(\n outputs,\n skip_special_tokens=False\n)\n\nfor tok in [\n '</s>',\n '<s>',\n '<pad>',\n '<unk>',\n 'eng_Latn',\n 'akk_Latn'\n]:\n pred_texts = [\n p.replace(tok, '')\n for p in pred_texts\n ]\nThis fix alone appeared to improve the geometric mean by roughly +1.7 on validation and leaderboard experiments.
\nNo retraining required — only fixing the decode pipeline.
\nAfter fixing decoding, I expected <gap> predictions to appear naturally.
They still mostly did not.
\nI ran a gap alignment diagnostic on the merged training data:
\nsrc_gap = train_df['text_akk'].str.contains(\n '<gap>',\n na=False\n)\n\ntgt_gap = train_df['text_en'].str.contains(\n '<gap>',\n na=False\n)\n\nprint('Source has <gap>:', src_gap.sum())\nprint('Target has <gap>:', tgt_gap.sum())\n\nprint(\n 'Source ONLY (problem):',\n (src_gap & ~tgt_gap).sum()\n)\nOutput:
\nSource has <gap>: 920 rows\nTarget has <gap>: 686 rows\nSource ONLY (problem): 329 rows\n329 rows contained <gap> in the source but not in the target.
The model learned the statistically safer strategy:
\n\n\nnever output a gap
\n
because a substantial fraction of source-gap examples had no target-side gap.
\ndef postprocess_gap_alignment(\n src_text,\n pred_text\n):\n\n src_gaps = src_text.count('<gap>')\n pred_gaps = pred_text.count('<gap>')\n\n if src_gaps == 0:\n return pred_text\n\n if pred_gaps >= src_gaps * 0.5:\n return pred_text\n\n missing = src_gaps - pred_gaps\n words = pred_text.split()\n\n positions = [\n max(\n 0,\n int(len(words) * (i + 1) / (missing + 1))\n )\n for i in range(missing)\n ]\n\n for offset, pos in enumerate(positions):\n words.insert(pos + offset, '<gap>')\n\n return ' '.join(words)\nI tested four postprocess strategies on identical raw predictions:
\n| Strategy | \nGaps Injected | \nGeo Mean | \n
|---|---|---|
| None | \n0 | \n26.71 | \n
| Conservative | \n82 | \n27.24 | \n
| Half | \n~200 | \n27.89 | \n
| Aggressive (0.5 threshold) | \n420 | \n28.45 | \n
Aggressive injection improved chrF++ overlap enough to offset BLEU penalties from over-injection.
\n<gap> as a regular token from the beginningIn transformers==5.x (Kaggle default), Group Beam Search was moved out of core and requires dynamic loading:
ValueError:\nGroup Beam Search requires trust_remote_code=True\nOffline Kaggle inference could not resolve the external repository.
\nAfter multiple failed workarounds, the cleanest solution was:
\ntransformers==4.38.2\nwhere Group Beam Search still exists natively.
\nsubprocess.run([\n sys.executable,\n '-m',\n 'pip',\n 'install',\n '--no-index',\n '--find-links=/kaggle/input/datasets/amritha18/transformers-4-38-offline-wheels',\n '--force-reinstall',\n '--no-deps',\n '--quiet',\n 'transformers==4.38.2',\n 'tokenizers==0.15.2',\n 'peft==0.10.0',\n], check=True)\n| Decoding | \nGeo Mean | \nNotes | \n
|---|---|---|
| Standard beam | \n~22–23 | \nShort outputs | \n
| Group beam | \n~24–25 | \nLonger, more complete | \n
It helped noticeably on long tablet translations where standard beam tended to collapse into short generic phrases.
\nThe training set is largely document-level while the test set behaves more like sentence-level translation.
\nThis mismatch caused:
\nThe solution was reconstructing sentence-level pairs using:
\nSentences_Oare_FirstWord_LinNum.csv\nFor tablets containing um-ma, I split around the epistolary opening.
def split_on_umma(\n akk_text,\n en_text\n):\n\n akk_parts = re.split(\n r'(?=\\bum-ma\\b)',\n akk_text.strip()\n )\n\n akk_parts = [\n p.strip()\n for p in akk_parts\n if len(p.strip()) > 10\n ]\n\n en_parts = re.split(\n r'(?<=[.!?])\\s+(?=[A-Z])',\n en_text.strip()\n )\n\n en_parts = [\n p.strip()\n for p in en_parts\n if len(p.strip()) > 10\n ]\n\n if (\n len(akk_parts) == len(en_parts)\n and len(akk_parts) > 1\n ):\n return list(zip(akk_parts, en_parts))\n\n return None\n| Stage | \nPairs | \n
|---|---|
| Raw train.csv | \n1,561 | \n
| After sentence splitting | \n~1,600 | \n
| + published_texts.csv | \n+3,522 | \n
| Final filtered dataset | \n5,009 | \n
min_new_tokens Needed Careful TuningTo reduce truncation after headers, I experimented with min_new_tokens.
| Config | \nGeo Mean | \nBrevity Penalty | \n
|---|---|---|
| None | \n27.24 | \n0.890 | \n
min_new_tokens=50 | \n27.24 | \n0.890 | \n
min_new_tokens=30 | \n28.24 | \n0.934 | \n
+ aggressive gap postprocess | \n29.55 | \n0.998 | \n
min_new_tokens=50 surprisingly had almost no effect.
Most headers are only ~8–12 tokens, so the model still stopped immediately after formulaic openings.
\nmin_new_tokens=30 pushed generation past the opening formulas into actual content.
Very small leaderboard movement despite much larger compute cost.
\nThe bottleneck appeared to be:
\nrather than model size.
\nAdding ~900 filtered ORACC pairs increased validation loss:
\nBaseline (no ORACC): val_loss = 1.575\nWith ORACC: val_loss = 1.686\nLikely causes:
\nI built ~11k proper noun constraints from:
\nOA_Lexicon_eBL.csv\nusing:
\nforce_words_ids\nIn practice, constrained decoding failed on most generations and frequently reverted to unconstrained fallback.
\nN-best reranking by proper noun overlap helped slightly, but forced decoding itself was not useful.
\nThe most interesting diagnostic was the divergence between chrF++ and BLEU.
\n| Metric | \nValidation | \nLeaderboard | \n
|---|---|---|
| BLEU | \n20.00 | \n~12 | \n
| chrF++ | \n43.66 | \n~21 | \n
chrF++ rewards partial character overlap.
\nBLEU rewards exact lexical matches.
\nThe model learned approximate transliteration structure but struggled with exact Akkadian word reconstruction.
\nThis appears closely related to subword tokenization fragmentation.
\nExample:
\niš-qú-ul\nbecomes fragmented into multiple unrelated subword pieces rather than a single coherent morphological unit.
\nByte-level architectures such as ByT5 avoid fixed vocabularies entirely.
\nPotential advantages:
\nGiven the chrF++ / BLEU divergence pattern, byte-level models may ultimately be a better fit for Akkadian MT than subword-based architectures.
\nFix tokenizer diagnostics immediately instead of assuming the model failed to learn <gap>
Mine publication PDFs earlier for additional parallel data
Test ByT5 much earlier as a baseline architecture
Domain-check external corpora before merging datasets
Focus more on alignment consistency than decoding tricks
Happy to discuss any of the diagnostics — especially the <gap> token issue since I suspect others may have encountered similar behavior without realizing it.
*Full technical report: https://github.com/Amrith182007/kaggle-deep-past-challenge-akkadian-mt*
", "messages": [ { "id": 3458490, "postDate": "2026-05-16T00:26:29.103Z", "content": "Final scores: Public 24.87 / Private 24.41 (NLLB-200 1.3B)
\nThis write-up focuses primarily on debugging, preprocessing, and decoding diagnostics rather than leaderboard optimization alone.
\nNot a top finish, but I learned a lot about what breaks silently in low-resource MT pipelines. Sharing the diagnostics because I think some of these findings are useful regardless of approach.
\nr=32, alpha=64)q_proj, k_proj, v_proj, out_proj, fc1, fc2transformers==4.38.2 (pinned — more on why below)<gap> Token Was Being Silently Deleted During DecodeThis turned out to be the most impactful bug in the pipeline.
\nThe <gap> token appears in ~50% of validation references. My model was predicting effectively zero gaps for weeks. I assumed the model simply had not learned the token yet.
Then I ran this diagnostic:
\ngap_id = tokenizer.convert_tokens_to_ids('<gap>')\nprint('Is special token:', gap_id in tokenizer.all_special_ids)\n\n# Encode a test sentence with <gap>\ntest = \"1 mina of silver <gap> Puzur-Aššur <gap> has paid\"\n\nenc = tokenizer(test, return_tensors='pt')\n\ndec_skip = tokenizer.decode(\n enc['input_ids'][0],\n skip_special_tokens=True\n)\n\ndec_keep = tokenizer.decode(\n enc['input_ids'][0],\n skip_special_tokens=False\n)\n\nprint('With skip_special_tokens=True :', dec_skip)\nprint('With skip_special_tokens=False:', dec_keep)\nOutput:
\nIs special token: True\n\nWith skip_special_tokens=True :\n1 mina of silver Puzur-Aššur has paid\n\nWith skip_special_tokens=False:\n1 mina of silver <gap> Puzur-Aššur <gap> has paid\nThe model was predicting gaps correctly during training.
\nBut because <gap> had been registered as an additional_special_token, every predicted gap was being removed during decoding with:
skip_special_tokens=True\nThe evaluation pipeline never saw a single gap token.
\n# BROKEN\n# <gap> added as special token -> stripped during decode\n\nspecial_tokens = {\n 'additional_special_tokens': [SRC_LANG, '<gap>']\n}\n\ntokenizer.add_special_tokens(special_tokens)\n\n\n# FIXED\n# <gap> added as regular token -> survives decode\n\ntokenizer.add_tokens(['<gap>'] + extra_tokens)\n\nspecial_tokens = {\n 'additional_special_tokens': [SRC_LANG]\n}\n\ntokenizer.add_special_tokens(special_tokens)\n\n\n# FIXED DECODE\n# manually strip only structural tokens\n\npred_texts = tokenizer.batch_decode(\n outputs,\n skip_special_tokens=False\n)\n\nfor tok in [\n '</s>',\n '<s>',\n '<pad>',\n '<unk>',\n 'eng_Latn',\n 'akk_Latn'\n]:\n pred_texts = [\n p.replace(tok, '')\n for p in pred_texts\n ]\nThis fix alone appeared to improve the geometric mean by roughly +1.7 on validation and leaderboard experiments.
\nNo retraining required — only fixing the decode pipeline.
\nAfter fixing decoding, I expected <gap> predictions to appear naturally.
They still mostly did not.
\nI ran a gap alignment diagnostic on the merged training data:
\nsrc_gap = train_df['text_akk'].str.contains(\n '<gap>',\n na=False\n)\n\ntgt_gap = train_df['text_en'].str.contains(\n '<gap>',\n na=False\n)\n\nprint('Source has <gap>:', src_gap.sum())\nprint('Target has <gap>:', tgt_gap.sum())\n\nprint(\n 'Source ONLY (problem):',\n (src_gap & ~tgt_gap).sum()\n)\nOutput:
\nSource has <gap>: 920 rows\nTarget has <gap>: 686 rows\nSource ONLY (problem): 329 rows\n329 rows contained <gap> in the source but not in the target.
The model learned the statistically safer strategy:
\n\n\nnever output a gap
\n
because a substantial fraction of source-gap examples had no target-side gap.
\ndef postprocess_gap_alignment(\n src_text,\n pred_text\n):\n\n src_gaps = src_text.count('<gap>')\n pred_gaps = pred_text.count('<gap>')\n\n if src_gaps == 0:\n return pred_text\n\n if pred_gaps >= src_gaps * 0.5:\n return pred_text\n\n missing = src_gaps - pred_gaps\n words = pred_text.split()\n\n positions = [\n max(\n 0,\n int(len(words) * (i + 1) / (missing + 1))\n )\n for i in range(missing)\n ]\n\n for offset, pos in enumerate(positions):\n words.insert(pos + offset, '<gap>')\n\n return ' '.join(words)\nI tested four postprocess strategies on identical raw predictions:
\n| Strategy | \nGaps Injected | \nGeo Mean | \n
|---|---|---|
| None | \n0 | \n26.71 | \n
| Conservative | \n82 | \n27.24 | \n
| Half | \n~200 | \n27.89 | \n
| Aggressive (0.5 threshold) | \n420 | \n28.45 | \n
Aggressive injection improved chrF++ overlap enough to offset BLEU penalties from over-injection.
\n<gap> as a regular token from the beginningIn transformers==5.x (Kaggle default), Group Beam Search was moved out of core and requires dynamic loading:
ValueError:\nGroup Beam Search requires trust_remote_code=True\nOffline Kaggle inference could not resolve the external repository.
\nAfter multiple failed workarounds, the cleanest solution was:
\ntransformers==4.38.2\nwhere Group Beam Search still exists natively.
\nsubprocess.run([\n sys.executable,\n '-m',\n 'pip',\n 'install',\n '--no-index',\n '--find-links=/kaggle/input/datasets/amritha18/transformers-4-38-offline-wheels',\n '--force-reinstall',\n '--no-deps',\n '--quiet',\n 'transformers==4.38.2',\n 'tokenizers==0.15.2',\n 'peft==0.10.0',\n], check=True)\n| Decoding | \nGeo Mean | \nNotes | \n
|---|---|---|
| Standard beam | \n~22–23 | \nShort outputs | \n
| Group beam | \n~24–25 | \nLonger, more complete | \n
It helped noticeably on long tablet translations where standard beam tended to collapse into short generic phrases.
\nThe training set is largely document-level while the test set behaves more like sentence-level translation.
\nThis mismatch caused:
\nThe solution was reconstructing sentence-level pairs using:
\nSentences_Oare_FirstWord_LinNum.csv\nFor tablets containing um-ma, I split around the epistolary opening.
def split_on_umma(\n akk_text,\n en_text\n):\n\n akk_parts = re.split(\n r'(?=\\bum-ma\\b)',\n akk_text.strip()\n )\n\n akk_parts = [\n p.strip()\n for p in akk_parts\n if len(p.strip()) > 10\n ]\n\n en_parts = re.split(\n r'(?<=[.!?])\\s+(?=[A-Z])',\n en_text.strip()\n )\n\n en_parts = [\n p.strip()\n for p in en_parts\n if len(p.strip()) > 10\n ]\n\n if (\n len(akk_parts) == len(en_parts)\n and len(akk_parts) > 1\n ):\n return list(zip(akk_parts, en_parts))\n\n return None\n| Stage | \nPairs | \n
|---|---|
| Raw train.csv | \n1,561 | \n
| After sentence splitting | \n~1,600 | \n
| + published_texts.csv | \n+3,522 | \n
| Final filtered dataset | \n5,009 | \n
min_new_tokens Needed Careful TuningTo reduce truncation after headers, I experimented with min_new_tokens.
| Config | \nGeo Mean | \nBrevity Penalty | \n
|---|---|---|
| None | \n27.24 | \n0.890 | \n
min_new_tokens=50 | \n27.24 | \n0.890 | \n
min_new_tokens=30 | \n28.24 | \n0.934 | \n
+ aggressive gap postprocess | \n29.55 | \n0.998 | \n
min_new_tokens=50 surprisingly had almost no effect.
Most headers are only ~8–12 tokens, so the model still stopped immediately after formulaic openings.
\nmin_new_tokens=30 pushed generation past the opening formulas into actual content.
Very small leaderboard movement despite much larger compute cost.
\nThe bottleneck appeared to be:
\nrather than model size.
\nAdding ~900 filtered ORACC pairs increased validation loss:
\nBaseline (no ORACC): val_loss = 1.575\nWith ORACC: val_loss = 1.686\nLikely causes:
\nI built ~11k proper noun constraints from:
\nOA_Lexicon_eBL.csv\nusing:
\nforce_words_ids\nIn practice, constrained decoding failed on most generations and frequently reverted to unconstrained fallback.
\nN-best reranking by proper noun overlap helped slightly, but forced decoding itself was not useful.
\nThe most interesting diagnostic was the divergence between chrF++ and BLEU.
\n| Metric | \nValidation | \nLeaderboard | \n
|---|---|---|
| BLEU | \n20.00 | \n~12 | \n
| chrF++ | \n43.66 | \n~21 | \n
chrF++ rewards partial character overlap.
\nBLEU rewards exact lexical matches.
\nThe model learned approximate transliteration structure but struggled with exact Akkadian word reconstruction.
\nThis appears closely related to subword tokenization fragmentation.
\nExample:
\niš-qú-ul\nbecomes fragmented into multiple unrelated subword pieces rather than a single coherent morphological unit.
\nByte-level architectures such as ByT5 avoid fixed vocabularies entirely.
\nPotential advantages:
\nGiven the chrF++ / BLEU divergence pattern, byte-level models may ultimately be a better fit for Akkadian MT than subword-based architectures.
\nFix tokenizer diagnostics immediately instead of assuming the model failed to learn <gap>
Mine publication PDFs earlier for additional parallel data
Test ByT5 much earlier as a baseline architecture
Domain-check external corpora before merging datasets
Focus more on alignment consistency than decoding tricks
Happy to discuss any of the diagnostics — especially the <gap> token issue since I suspect others may have encountered similar behavior without realizing it.
*Full technical report: https://github.com/Amrith182007/kaggle-deep-past-challenge-akkadian-mt*
", "rawMarkdown": "# Deep Past Challenge — NLLB Post-Mortem and Debugging Notes\n\nFinal scores: **Public 24.87 / Private 24.41** (NLLB-200 1.3B)\n\nThis write-up focuses primarily on debugging, preprocessing, and decoding diagnostics rather than leaderboard optimization alone.\n\nNot a top finish, but I learned a lot about what breaks silently in low-resource MT pipelines. Sharing the diagnostics because I think some of these findings are useful regardless of approach.\n\n---\n\n## The Setup\n\n- Base model: NLLB-200 distilled 600M → later 1.3B\n- Fine-tuning: LoRA (`r=32`, `alpha=64`)\n- Target modules: `q_proj`, `k_proj`, `v_proj`, `out_proj`, `fc1`, `fc2`\n- Training pairs: started at 1,561 → ended at ~5,700 after data expansion\n- Hardware: P100 (600M), TPU v5e-8 (1.3B)\n- Framework: `transformers==4.38.2` (pinned — more on why below)\n\n---\n\n# Finding 1: The `