# What Actually Moved the Needle (and What Didn't) _NLLB-200 +LORA Deep Past Challenge — NLLB Post-Mortem and Debugging Notes Final scores: Public 24.87 / Private 24.41 (NLLB-200 1.3B) This write-up focuses primarily on debugging, preprocessing, and decoding diagnostics rather than leaderboard optimization alone. Not a top finish, but I learned a lot about what breaks silently in low-resource MT pipelines. Sharing the diagnostics because I think some of these findings are useful regardless of approach. The Setup Base model: NLLB-200 distilled 600M → later 1.3B Fine-tuning: LoRA ( r=32 , alpha=64 ) Target modules: q_proj , k_proj , v_proj , out_proj , fc1 , fc2 Training pairs: started at 1,561 → ended at ~5,700 after data expansion Hardware: P100 (600M), TPU v5e-8 (1.3B) Framework: transformers==4.38.2 (pinned — more on why below) Finding 1: The Token Was Being Silently Deleted During Decode This turned out to be the most impactful bug in the pipeline. The token appears in ~50% of validation references. My model was predicting effectively zero gaps for weeks. I assumed the model simply had not learned the token yet. Then I ran this diagnostic: gap_id = tokenizer.convert_tokens_to_ids('') print('Is special token:', gap_id in tokenizer.all_special_ids) # Encode a test sentence with test = "1 mina of silver Puzur-Aššur has paid" enc = tokenizer(test, return_tensors='pt') dec_skip = tokenizer.decode( enc['input_ids'][0], skip_special_tokens=True ) dec_keep = tokenizer.decode( enc['input_ids'][0], skip_special_tokens=False ) print('With skip_special_tokens=True :', dec_skip) print('With skip_special_tokens=False:', dec_keep) Output: Is special token: True With skip_special_tokens=True : 1 mina of silver Puzur-Aššur has paid With skip_special_tokens=False: 1 mina of silver Puzur-Aššur has paid The model was predicting gaps correctly during training. But because had been registered as an additional_special_token , every predicted gap was being removed during decoding with: skip_special_tokens=True The evaluation pipeline never saw a single gap token. The Fix # BROKEN # added as special token -> stripped during decode special_tokens = { 'additional_special_tokens': [SRC_LANG, ''] } tokenizer.add_special_tokens(special_tokens) # FIXED # added as regular token -> survives decode tokenizer.add_tokens([''] + extra_tokens) special_tokens = { 'additional_special_tokens': [SRC_LANG] } tokenizer.add_special_tokens(special_tokens) # FIXED DECODE # manually strip only structural tokens pred_texts = tokenizer.batch_decode( outputs, skip_special_tokens=False ) for tok in [ '', '', '', '', 'eng_Latn', 'akk_Latn' ]: pred_texts = [ p.replace(tok, '') for p in pred_texts ] This fix alone appeared to improve the geometric mean by roughly +1.7 on validation and leaderboard experiments. No retraining required — only fixing the decode pipeline. Finding 2: Even After the Fix, the Model Still Avoided Predicting Gaps After fixing decoding, I expected predictions to appear naturally. They still mostly did not. I ran a gap alignment diagnostic on the merged training data: src_gap = train_df['text_akk'].str.contains( '', na=False ) tgt_gap = train_df['text_en'].str.contains( '', na=False ) print('Source has :', src_gap.sum()) print('Target has :', tgt_gap.sum()) print( 'Source ONLY (problem):', (src_gap & ~tgt_gap).sum() ) Output: Source has : 920 rows Target has : 686 rows Source ONLY (problem): 329 rows 329 rows contained in the source but not in the target. The model learned the statistically safer strategy: never output a gap because a substantial fraction of source-gap examples had no target-side gap. Postprocessing Band-Aid def postprocess_gap_alignment( src_text, pred_text ): src_gaps = src_text.count('') pred_gaps = pred_text.count('') if src_gaps == 0: return pred_text if pred_gaps >= src_gaps * 0.5: return pred_text missing = src_gaps - pred_gaps words = pred_text.split() positions = [ max( 0, int(len(words) * (i + 1) / (missing + 1)) ) for i in range(missing) ] for offset, pos in enumerate(positions): words.insert(pos + offset, '') return ' '.join(words) I tested four postprocess strategies on identical raw predictions: Strategy Gaps Injected Geo Mean None 0 26.71 Conservative 82 27.24 Half ~200 27.89 Aggressive (0.5 threshold) 420 28.45 Aggressive injection improved chrF++ overlap enough to offset BLEU penalties from over-injection. Real fix for future work Retrain with as a regular token from the beginning Clean up source-only gap alignment rows Finding 3: Group Beam Search Required Downgrading Transformers In transformers==5.x (Kaggle default), Group Beam Search was moved out of core and requires dynamic loading: ValueError: Group Beam Search requires trust_remote_code=True Offline Kaggle inference could not resolve the external repository. After multiple failed workarounds, the cleanest solution was: transformers==4.38.2 where Group Beam Search still exists natively. Offline Inference Install subprocess.run([ sys.executable, '-m', 'pip', 'install', '--no-index', '--find-links=/kaggle/input/datasets/amritha18/transformers-4-38-offline-wheels', '--force-reinstall', '--no-deps', '--quiet', 'transformers==4.38.2', 'tokenizers==0.15.2', 'peft==0.10.0', ], check=True) Did Group Beam Search Help? Decoding Geo Mean Notes Standard beam ~22–23 Short outputs Group beam ~24–25 Longer, more complete It helped noticeably on long tablet translations where standard beam tended to collapse into short generic phrases. Finding 4: Document-Level Training vs Sentence-Level Test The training set is largely document-level while the test set behaves more like sentence-level translation. This mismatch caused: truncation after formulaic openings hallucination loops brevity penalties incomplete generations The solution was reconstructing sentence-level pairs using: Sentences_Oare_FirstWord_LinNum.csv For tablets containing um-ma , I split around the epistolary opening. Sentence Reconstruction def split_on_umma( akk_text, en_text ): akk_parts = re.split( r'(?=\bum-ma\b)', akk_text.strip() ) akk_parts = [ p.strip() for p in akk_parts if len(p.strip()) > 10 ] en_parts = re.split( r'(?<=[.!?])\s+(?=[A-Z])', en_text.strip() ) en_parts = [ p.strip() for p in en_parts if len(p.strip()) > 10 ] if ( len(akk_parts) == len(en_parts) and len(akk_parts) > 1 ): return list(zip(akk_parts, en_parts)) return None Data Expansion Stage Pairs Raw train.csv 1,561 After sentence splitting ~1,600 + published_texts.csv +3,522 Final filtered dataset 5,009 Finding 5: min_new_tokens Needed Careful Tuning To reduce truncation after headers, I experimented with min_new_tokens . Config Geo Mean Brevity Penalty None 27.24 0.890 min_new_tokens=50 27.24 0.890 min_new_tokens=30 28.24 0.934 + aggressive gap postprocess 29.55 0.998 min_new_tokens=50 surprisingly had almost no effect. Most headers are only ~8–12 tokens, so the model still stopped immediately after formulaic openings. min_new_tokens=30 pushed generation past the opening formulas into actual content. What Did Not Help 1. NLLB-200 1.3B vs 600M Very small leaderboard movement despite much larger compute cost. The bottleneck appeared to be: data quality representation tokenization rather than model size. 2. ORACC External Data Adding ~900 filtered ORACC pairs increased validation loss: Baseline (no ORACC): val_loss = 1.575 With ORACC: val_loss = 1.686 Likely causes: dialect mismatch Neo-Assyrian vs Old Assyrian inconsistent gap alignment different scribal conventions 3. Forced Proper-Noun Decoding I built ~11k proper noun constraints from: OA_Lexicon_eBL.csv using: force_words_ids In practice, constrained decoding failed on most generations and frequently reverted to unconstrained fallback. N-best reranking by proper noun overlap helped slightly, but forced decoding itself was not useful. chrF++ vs BLEU — The Diagnostic That Explained the Ceiling The most interesting diagnostic was the divergence between chrF++ and BLEU. Metric Validation Leaderboard BLEU 20.00 ~12 chrF++ 43.66 ~21 chrF++ rewards partial character overlap. BLEU rewards exact lexical matches. The model learned approximate transliteration structure but struggled with exact Akkadian word reconstruction. This appears closely related to subword tokenization fragmentation. Example: iš-qú-ul becomes fragmented into multiple unrelated subword pieces rather than a single coherent morphological unit. Why ByT5 May Be Better Byte-level architectures such as ByT5 avoid fixed vocabularies entirely. Potential advantages: no BPE fragmentation direct character modeling stronger handling of rare morphology better transliteration robustness Given the chrF++ / BLEU divergence pattern, byte-level models may ultimately be a better fit for Akkadian MT than subword-based architectures. Things I Would Do Differently Fix tokenizer diagnostics immediately instead of assuming the model failed to learn Mine publication PDFs earlier for additional parallel data Test ByT5 much earlier as a baseline architecture Domain-check external corpora before merging datasets Focus more on alignment consistency than decoding tricks Lessons Learned Silent preprocessing and decoding bugs can matter more than model scale. Alignment consistency mattered more than aggressive augmentation. chrF++ and BLEU diverged significantly on Akkadian transliterations. Domain mismatch in external corpora can hurt more than small dataset size. Byte-level architectures such as ByT5 may ultimately be a better fit for Akkadian MT than subword-based models. Happy to discuss any of the diagnostics — especially the token issue since I suspect others may have encountered similar behavior without realizing it. *Full technical report: [https://github.com/Amrith182007/kaggle-deep-past-challenge-akkadian-mt*](https://github.com/Amrith182007/kaggle-deep-past-challenge-akkadian-mt*)