Transparency: This is a synthesis of community discussions and public notebooks, not original research. I'm currently ranked at 35.1 score and learning from the community myself.
\nTiming Note: Many insights here are from early-to-mid January discussions. \nThe community's understanding has evolved since then. If you're in the top 50, you probably know more than what's written here.
\nTarget Audience: This consolidation is aimed at competitors who are:
\nWhat I HAVE tested personally: AKK-300m failure, bucket batching, length penalty variations, the 35.1 optimization stack.
\nWhat I'm synthesizing from others: Training recipes, data quality insights, overfitting analysis.
\nRequest to Top Competitors: If anything here is outdated or wrong, please comment! I want this to be accurate and helpful, not misleading.
\nPurpose of This Post
\nAfter weeks of testing, collaboration, and learning from the community, I wanted to consolidate what's actually working (and what's not) to help everyone still pushing for improvements. This is a synthesis of public discussions, shared notebooks, and systematic testing.
\nTL;DR: We're at an interesting inflection point. The public model ceiling is ~35.1, and breaking past it requires either custom training data, novel techniques, or collaboration. Here's what we know.
\nCurrent State of the Leaderboard
\nThe 35.1 Plateau:
\nbyt5-akkadian-optimized-34x) with minor variationsScore Distribution:
\nCritical Insight from Aaron Bornstein:
\n\n\n\"High scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain\"
\n
Public LB (34% of test) ≠ Private LB (66% of test). Conservative, semantically-accurate approaches may outperform pattern-matching models on final evaluation.
\nProven Optimization Stack (34.9 → 35.1)
\nbyt5-akkadian-optimized-34x (34.9)IMPORTANT CORRECTION (Thanks to Musa Peker for catching this):
\nThis is actually an ENSEMBLE of three models, not a single model:
\n/kaggle/input/byt5-akk-gap-sentence-v4-cp-final (likely Assia's)/kaggle/input/byt5-akkadian-model/kaggle/input/byt5-base-big-data2Implication: The 34.9 baseline is achieved through ensemble combination, not a \nsingle checkpoint. This means:
\nCredit: Assia Benkedia for the ensemble approach, plus the contributors of the \nthree base models\nAvailability: Kaggle dataset final-byt5
class BucketBatchSampler(Sampler):\n \"\"\"Groups samples by similar length to minimize padding\"\"\"\n def __init__(self, lengths, batch_size, num_buckets=4):\n self.lengths = lengths\n self.batch_size = batch_size\n self.num_buckets = num_buckets\n\n # Sort indices by length\n sorted_indices = sorted(range(len(lengths)), key=lambda i: lengths[i])\n\n # Divide into buckets\n bucket_size = len(sorted_indices) // num_buckets\n self.buckets = []\n for i in range(num_buckets):\n start = i * bucket_size\n end = start + bucket_size if i < num_buckets - 1 else len(sorted_indices)\n self.buckets.append(sorted_indices[start:end])\n\n def __iter__(self):\n for bucket in self.buckets:\n indices = bucket.copy()\n for i in range(0, len(indices), self.batch_size):\n yield indices[i:i + self.batch_size]\n\n def __len__(self):\n return sum(len(bucket) // self.batch_size for bucket in self.buckets)\nCONFIG = {\n \"model\": \"byt5-akkadian-optimized-34x\",\n \"num_beams\": 8,\n \"max_new_tokens\": 512,\n \"length_penalty\": 1.3, # or 1.5\n \"early_stopping\": True,\n \"batch_size\": 8,\n \"use_bucket_batching\": True,\n \"num_buckets\": 4\n}\nImplementation outline:
\n# Generate 15 diverse candidates (temperature=0.7)\n# Generate 5 beam search candidates\n# Score each candidate against all others using chrF++\n# Select candidate with highest average similarity\nlength_penalty=1.788, beams=6 → validation score 25.97length_penalty=1.5, beams=8 → validation score 25.58<big_gap> tokens (complete garbage)From Angantyr:
\nHost Confirmation:
\n\n\n\"By no means an error-free dataset. Many of these databases are worked on by students with little oversight\"
\n
From Aaron Bornstein's Systematic Perturbation Testing:
\nFrom Musa Peker:
\nCRITICAL FINDING:
\n\"\"\"\nASSIA'S MODEL + BUCKET BATCHING + OPTIMAL LENGTH PENALTY\nExpected Score: 35.1\n\nNOTEBOOK SETUP:\n- Accelerator: GPU T4 x2\n- Internet: OFF \n- Datasets: deep-past-initiative-machine-translation, final-byt5\n\"\"\"\n\nimport re\nimport pandas as pd\nimport torch\nfrom torch.utils.data import Dataset, DataLoader, Sampler\nfrom transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n# ============================================================\n# CONFIGURATION\n# ============================================================\nCONFIG = {\n \"test_path\": \"/kaggle/input/deep-past-initiative-machine-translation/test.csv\",\n \"model_path\": \"/kaggle/input/final-byt5/byt5-akkadian-optimized-34x\",\n \"device\": torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\"),\n \"max_length\": 512,\n \"batch_size\": 8,\n \"num_buckets\": 4,\n\n \"generation\": {\n \"num_beams\": 8,\n \"max_new_tokens\": 512,\n \"length_penalty\": 1.3, # or 1.5\n \"early_stopping\": True\n }\n}\n\n# ============================================================\n# BUCKET BATCH SAMPLER (SERA'S OPTIMIZATION)\n# ============================================================\nclass BucketBatchSampler(Sampler):\n def __init__(self, lengths, batch_size, num_buckets=4):\n self.lengths = lengths\n self.batch_size = batch_size\n self.num_buckets = num_buckets\n\n sorted_indices = sorted(range(len(lengths)), key=lambda i: lengths[i])\n\n bucket_size = len(sorted_indices) // num_buckets\n self.buckets = []\n for i in range(num_buckets):\n start = i * bucket_size\n end = start + bucket_size if i < num_buckets - 1 else len(sorted_indices)\n self.buckets.append(sorted_indices[start:end])\n\n def __iter__(self):\n for bucket in self.buckets:\n indices = bucket.copy()\n for i in range(0, len(indices), self.batch_size):\n yield indices[i:i + self.batch_size]\n\n def __len__(self):\n return sum(len(bucket) // self.batch_size for bucket in self.buckets)\n\n# ============================================================\n# PREPROCESSING & POSTPROCESSING\n# ============================================================\ndef preprocess_input(text):\n \"\"\"Minimal preprocessing for test data\"\"\"\n if pd.isna(text):\n return \"\"\n text = str(text)\n text = re.sub(r'(\\.{3,}|…+)', '<big_gap>', text)\n text = re.sub(r'(xx+|\\s+x\\s+)', '<gap>', text)\n return text\n\ndef postprocess_output(text):\n \"\"\"Clean model output - proven effective stack\"\"\"\n if not isinstance(text, str) or not text.strip():\n return \"\"\n\n # 1. Normalize ḫ/Ḫ → h/H (test set uses h)\n text = text.replace('ḫ', 'h').replace('Ḫ', 'H')\n\n # 2. Subscript numbers → regular\n subscript_map = str.maketrans(\"₀₁₂₃₄₅₆₇₈₉\", \"0123456789\")\n text = text.translate(subscript_map)\n\n # 3. Normalize gaps in output\n text = re.sub(r'(\\[x\\]|\\(x\\)|\\bx\\b)', '<gap>', text, flags=re.I)\n text = re.sub(r'(\\.{3,}|…)', '<big_gap>', text)\n\n # 4. Remove scribal annotations\n text = re.sub(r'\\((fem|plur|pl|sing|\\?|!)\\.?\\)', '', text, flags=re.I)\n\n # 5. Protect gap markers, remove forbidden chars\n text = text.replace('<gap>', '\\x00GAP\\x00')\n text = text.replace('<big_gap>', '\\x00BIG\\x00')\n forbidden = '!?()\"—–<>⌈⌋⌊[]+ʾ/;'\n text = text.translate(str.maketrans('', '', forbidden))\n text = text.replace('\\x00GAP\\x00', ' <gap> ')\n text = text.replace('\\x00BIG\\x00', ' <big_gap> ')\n\n # 6. Unicode fractions\n text = re.sub(r'(\\d+)\\.5\\b', r'\\1 ½', text)\n text = re.sub(r'(\\d+)\\.25\\b', r'\\1 ¼', text)\n text = re.sub(r'(\\d+)\\.75\\b', r'\\1 ¾', text)\n\n # 7. Remove word repetitions\n text = re.sub(r'\\b(\\w+)(?:\\s+\\1\\b)+', r'\\1', text)\n\n # 8. Final cleanup\n text = re.sub(r'\\s+', ' ', text).strip()\n\n return text\n\n# ============================================================\n# DATASET CLASS\n# ============================================================\nclass AkkadianDataset(Dataset):\n def __init__(self, df):\n self.ids = df['id'].tolist()\n self.texts = [\n \"translate Akkadian to English: \" + str(t) \n for t in df['transliteration']\n ]\n self.lengths = [len(t.split()) for t in self.texts]\n\n def __len__(self):\n return len(self.ids)\n\n def __getitem__(self, idx):\n return self.ids[idx], self.texts[idx]\n\n# ============================================================\n# MAIN INFERENCE\n# ============================================================\nprint(\"Loading test data...\")\ntest_df = pd.read_csv(CONFIG['test_path'])\ntest_df['transliteration'] = test_df['transliteration'].apply(preprocess_input)\n\nprint(\"Loading model...\")\nmodel = AutoModelForSeq2SeqLM.from_pretrained(CONFIG['model_path']).to(CONFIG['device']).eval()\ntokenizer = AutoTokenizer.from_pretrained(CONFIG['model_path'])\n\nprint(\"Creating dataset with bucket sampling...\")\ndataset = AkkadianDataset(test_df)\nsampler = BucketBatchSampler(dataset.lengths, CONFIG['batch_size'], CONFIG['num_buckets'])\n\ndef collate_fn(batch):\n ids = [item[0] for item in batch]\n texts = [item[1] for item in batch]\n encoded = tokenizer(texts, max_length=CONFIG['max_length'], \n padding=True, truncation=True, return_tensors=\"pt\")\n return ids, encoded\n\ndataloader = DataLoader(dataset, batch_sampler=sampler, collate_fn=collate_fn, num_workers=2)\n\nprint(\"Running inference...\")\npredictions = []\nwith torch.inference_mode():\n for batch_ids, encoded_inputs in dataloader:\n outputs = model.generate(\n input_ids=encoded_inputs.input_ids.to(CONFIG['device']),\n attention_mask=encoded_inputs.attention_mask.to(CONFIG['device']),\n **CONFIG['generation']\n )\n decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)\n cleaned = [postprocess_output(text) for text in decoded]\n predictions.extend(zip(batch_ids, cleaned))\n\n# Create submission\nsubmission = pd.DataFrame(predictions, columns=['id', 'translation'])\nsubmission.to_csv('submission.csv', index=False)\nprint(f\"Done! Saved {len(submission)} translations. Expected score: 35.1\")\nYou need ONE of these:
\nCustom Training Data (3,000-5,000 clean Old Assyrian sentence pairs)
\nNovel Inference Techniques
\nCollaboration
\nFrom Jean-Louis Roy (#5) - Validated Recipe:
\nEPOCHS = 10-15 # NOT 26+\nMAX_LENGTH = 512 # NOT 256\nBATCH_SIZE = 4\nGRADIENT_ACCUMULATION_STEPS = 4\nLEARNING_RATE = 5e-5\nFP16 = False # Critical - prevents NaN errors\nFrom Jack (#2) - CORRECTED (Thanks to hongan for catching this):
\nWhat I originally wrote was WRONG. Jack clarified his actual approach:
\n✅ What Jack actually does:
\n❌ What Jack does NOT do:
\nKey quote (Feb 2026):\n\"I really started to gain in score when I manually reviewed (briefly - still \nfinding things I've missed) each and every document in the train.csv.\"
\nAbout sentence alignment:\n\"Sentence alignment didn't work for me because my sentences sucked 😭\" \n(This was about HIS implementation, not the approach itself)
\nCritical insight:\n\"If your formatting is off, especially on translations, nothing else you do \nwill matter since your model isn't matching the desired output format.\"
\nFrom 耶✌ :
\nSentences_Oare_FirstWord_LinNum.csv improved score +0.5train.csv (1,561 samples) - Original competition dataSentences_Oare_FirstWord_LinNum.csv (9,782 alignments) - If verified against first_word matchingpublished_texts.csv (7,953 samples) - Transliterations only, unknown time period mixGold (Top 15): Requires custom data OR collaboration OR breakthrough technique
\nSilver (Top 50): Achievable with proven optimization stack (35.1-35.3)
\n(Top 200): Public model baseline (34.9-35.0)
\nKey Decision: Private LB shake-up probability ~60% (per Aaron Bornstein's analysis). Conservative, semantically-accurate approaches may outperform pattern-matching models on final evaluation.
\nMajor Contributors (in alphabetical order):
\nHost (DeepPast Initiative) - For hosting this fascinating competition and providing clarifications throughout
\nIf I missed giving proper credit to anyone or misattributed any techniques, please let me know in the comments and I'll update immediately. This synthesis is based on public discussions and notebooks—if you contributed something I didn't mention, that's my oversight, not intentional!
\nGood luck to everyone in the final push! 51 days to go.
", "messages": [ { "id": 3403575, "postDate": "2026-02-08T19:51:26.817Z", "content": "Transparency: This is a synthesis of community discussions and public notebooks, not original research. I'm currently ranked at 35.1 score and learning from the community myself.
\nTiming Note: Many insights here are from early-to-mid January discussions. \nThe community's understanding has evolved since then. If you're in the top 50, you probably know more than what's written here.
\nTarget Audience: This consolidation is aimed at competitors who are:
\nWhat I HAVE tested personally: AKK-300m failure, bucket batching, length penalty variations, the 35.1 optimization stack.
\nWhat I'm synthesizing from others: Training recipes, data quality insights, overfitting analysis.
\nRequest to Top Competitors: If anything here is outdated or wrong, please comment! I want this to be accurate and helpful, not misleading.
\nPurpose of This Post
\nAfter weeks of testing, collaboration, and learning from the community, I wanted to consolidate what's actually working (and what's not) to help everyone still pushing for improvements. This is a synthesis of public discussions, shared notebooks, and systematic testing.
\nTL;DR: We're at an interesting inflection point. The public model ceiling is ~35.1, and breaking past it requires either custom training data, novel techniques, or collaboration. Here's what we know.
\nCurrent State of the Leaderboard
\nThe 35.1 Plateau:
\nbyt5-akkadian-optimized-34x) with minor variationsScore Distribution:
\nCritical Insight from Aaron Bornstein:
\n\n\n\"High scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain\"
\n
Public LB (34% of test) ≠ Private LB (66% of test). Conservative, semantically-accurate approaches may outperform pattern-matching models on final evaluation.
\nProven Optimization Stack (34.9 → 35.1)
\nbyt5-akkadian-optimized-34x (34.9)IMPORTANT CORRECTION (Thanks to Musa Peker for catching this):
\nThis is actually an ENSEMBLE of three models, not a single model:
\n/kaggle/input/byt5-akk-gap-sentence-v4-cp-final (likely Assia's)/kaggle/input/byt5-akkadian-model/kaggle/input/byt5-base-big-data2Implication: The 34.9 baseline is achieved through ensemble combination, not a \nsingle checkpoint. This means:
\nCredit: Assia Benkedia for the ensemble approach, plus the contributors of the \nthree base models\nAvailability: Kaggle dataset final-byt5
class BucketBatchSampler(Sampler):\n \"\"\"Groups samples by similar length to minimize padding\"\"\"\n def __init__(self, lengths, batch_size, num_buckets=4):\n self.lengths = lengths\n self.batch_size = batch_size\n self.num_buckets = num_buckets\n\n # Sort indices by length\n sorted_indices = sorted(range(len(lengths)), key=lambda i: lengths[i])\n\n # Divide into buckets\n bucket_size = len(sorted_indices) // num_buckets\n self.buckets = []\n for i in range(num_buckets):\n start = i * bucket_size\n end = start + bucket_size if i < num_buckets - 1 else len(sorted_indices)\n self.buckets.append(sorted_indices[start:end])\n\n def __iter__(self):\n for bucket in self.buckets:\n indices = bucket.copy()\n for i in range(0, len(indices), self.batch_size):\n yield indices[i:i + self.batch_size]\n\n def __len__(self):\n return sum(len(bucket) // self.batch_size for bucket in self.buckets)\nCONFIG = {\n \"model\": \"byt5-akkadian-optimized-34x\",\n \"num_beams\": 8,\n \"max_new_tokens\": 512,\n \"length_penalty\": 1.3, # or 1.5\n \"early_stopping\": True,\n \"batch_size\": 8,\n \"use_bucket_batching\": True,\n \"num_buckets\": 4\n}\nImplementation outline:
\n# Generate 15 diverse candidates (temperature=0.7)\n# Generate 5 beam search candidates\n# Score each candidate against all others using chrF++\n# Select candidate with highest average similarity\nlength_penalty=1.788, beams=6 → validation score 25.97length_penalty=1.5, beams=8 → validation score 25.58<big_gap> tokens (complete garbage)From Angantyr:
\nHost Confirmation:
\n\n\n\"By no means an error-free dataset. Many of these databases are worked on by students with little oversight\"
\n
From Aaron Bornstein's Systematic Perturbation Testing:
\nFrom Musa Peker:
\nCRITICAL FINDING:
\n\"\"\"\nASSIA'S MODEL + BUCKET BATCHING + OPTIMAL LENGTH PENALTY\nExpected Score: 35.1\n\nNOTEBOOK SETUP:\n- Accelerator: GPU T4 x2\n- Internet: OFF \n- Datasets: deep-past-initiative-machine-translation, final-byt5\n\"\"\"\n\nimport re\nimport pandas as pd\nimport torch\nfrom torch.utils.data import Dataset, DataLoader, Sampler\nfrom transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n# ============================================================\n# CONFIGURATION\n# ============================================================\nCONFIG = {\n \"test_path\": \"/kaggle/input/deep-past-initiative-machine-translation/test.csv\",\n \"model_path\": \"/kaggle/input/final-byt5/byt5-akkadian-optimized-34x\",\n \"device\": torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\"),\n \"max_length\": 512,\n \"batch_size\": 8,\n \"num_buckets\": 4,\n\n \"generation\": {\n \"num_beams\": 8,\n \"max_new_tokens\": 512,\n \"length_penalty\": 1.3, # or 1.5\n \"early_stopping\": True\n }\n}\n\n# ============================================================\n# BUCKET BATCH SAMPLER (SERA'S OPTIMIZATION)\n# ============================================================\nclass BucketBatchSampler(Sampler):\n def __init__(self, lengths, batch_size, num_buckets=4):\n self.lengths = lengths\n self.batch_size = batch_size\n self.num_buckets = num_buckets\n\n sorted_indices = sorted(range(len(lengths)), key=lambda i: lengths[i])\n\n bucket_size = len(sorted_indices) // num_buckets\n self.buckets = []\n for i in range(num_buckets):\n start = i * bucket_size\n end = start + bucket_size if i < num_buckets - 1 else len(sorted_indices)\n self.buckets.append(sorted_indices[start:end])\n\n def __iter__(self):\n for bucket in self.buckets:\n indices = bucket.copy()\n for i in range(0, len(indices), self.batch_size):\n yield indices[i:i + self.batch_size]\n\n def __len__(self):\n return sum(len(bucket) // self.batch_size for bucket in self.buckets)\n\n# ============================================================\n# PREPROCESSING & POSTPROCESSING\n# ============================================================\ndef preprocess_input(text):\n \"\"\"Minimal preprocessing for test data\"\"\"\n if pd.isna(text):\n return \"\"\n text = str(text)\n text = re.sub(r'(\\.{3,}|…+)', '<big_gap>', text)\n text = re.sub(r'(xx+|\\s+x\\s+)', '<gap>', text)\n return text\n\ndef postprocess_output(text):\n \"\"\"Clean model output - proven effective stack\"\"\"\n if not isinstance(text, str) or not text.strip():\n return \"\"\n\n # 1. Normalize ḫ/Ḫ → h/H (test set uses h)\n text = text.replace('ḫ', 'h').replace('Ḫ', 'H')\n\n # 2. Subscript numbers → regular\n subscript_map = str.maketrans(\"₀₁₂₃₄₅₆₇₈₉\", \"0123456789\")\n text = text.translate(subscript_map)\n\n # 3. Normalize gaps in output\n text = re.sub(r'(\\[x\\]|\\(x\\)|\\bx\\b)', '<gap>', text, flags=re.I)\n text = re.sub(r'(\\.{3,}|…)', '<big_gap>', text)\n\n # 4. Remove scribal annotations\n text = re.sub(r'\\((fem|plur|pl|sing|\\?|!)\\.?\\)', '', text, flags=re.I)\n\n # 5. Protect gap markers, remove forbidden chars\n text = text.replace('<gap>', '\\x00GAP\\x00')\n text = text.replace('<big_gap>', '\\x00BIG\\x00')\n forbidden = '!?()\"—–<>⌈⌋⌊[]+ʾ/;'\n text = text.translate(str.maketrans('', '', forbidden))\n text = text.replace('\\x00GAP\\x00', ' <gap> ')\n text = text.replace('\\x00BIG\\x00', ' <big_gap> ')\n\n # 6. Unicode fractions\n text = re.sub(r'(\\d+)\\.5\\b', r'\\1 ½', text)\n text = re.sub(r'(\\d+)\\.25\\b', r'\\1 ¼', text)\n text = re.sub(r'(\\d+)\\.75\\b', r'\\1 ¾', text)\n\n # 7. Remove word repetitions\n text = re.sub(r'\\b(\\w+)(?:\\s+\\1\\b)+', r'\\1', text)\n\n # 8. Final cleanup\n text = re.sub(r'\\s+', ' ', text).strip()\n\n return text\n\n# ============================================================\n# DATASET CLASS\n# ============================================================\nclass AkkadianDataset(Dataset):\n def __init__(self, df):\n self.ids = df['id'].tolist()\n self.texts = [\n \"translate Akkadian to English: \" + str(t) \n for t in df['transliteration']\n ]\n self.lengths = [len(t.split()) for t in self.texts]\n\n def __len__(self):\n return len(self.ids)\n\n def __getitem__(self, idx):\n return self.ids[idx], self.texts[idx]\n\n# ============================================================\n# MAIN INFERENCE\n# ============================================================\nprint(\"Loading test data...\")\ntest_df = pd.read_csv(CONFIG['test_path'])\ntest_df['transliteration'] = test_df['transliteration'].apply(preprocess_input)\n\nprint(\"Loading model...\")\nmodel = AutoModelForSeq2SeqLM.from_pretrained(CONFIG['model_path']).to(CONFIG['device']).eval()\ntokenizer = AutoTokenizer.from_pretrained(CONFIG['model_path'])\n\nprint(\"Creating dataset with bucket sampling...\")\ndataset = AkkadianDataset(test_df)\nsampler = BucketBatchSampler(dataset.lengths, CONFIG['batch_size'], CONFIG['num_buckets'])\n\ndef collate_fn(batch):\n ids = [item[0] for item in batch]\n texts = [item[1] for item in batch]\n encoded = tokenizer(texts, max_length=CONFIG['max_length'], \n padding=True, truncation=True, return_tensors=\"pt\")\n return ids, encoded\n\ndataloader = DataLoader(dataset, batch_sampler=sampler, collate_fn=collate_fn, num_workers=2)\n\nprint(\"Running inference...\")\npredictions = []\nwith torch.inference_mode():\n for batch_ids, encoded_inputs in dataloader:\n outputs = model.generate(\n input_ids=encoded_inputs.input_ids.to(CONFIG['device']),\n attention_mask=encoded_inputs.attention_mask.to(CONFIG['device']),\n **CONFIG['generation']\n )\n decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)\n cleaned = [postprocess_output(text) for text in decoded]\n predictions.extend(zip(batch_ids, cleaned))\n\n# Create submission\nsubmission = pd.DataFrame(predictions, columns=['id', 'translation'])\nsubmission.to_csv('submission.csv', index=False)\nprint(f\"Done! Saved {len(submission)} translations. Expected score: 35.1\")\nYou need ONE of these:
\nCustom Training Data (3,000-5,000 clean Old Assyrian sentence pairs)
\nNovel Inference Techniques
\nCollaboration
\nFrom Jean-Louis Roy (#5) - Validated Recipe:
\nEPOCHS = 10-15 # NOT 26+\nMAX_LENGTH = 512 # NOT 256\nBATCH_SIZE = 4\nGRADIENT_ACCUMULATION_STEPS = 4\nLEARNING_RATE = 5e-5\nFP16 = False # Critical - prevents NaN errors\nFrom Jack (#2) - CORRECTED (Thanks to hongan for catching this):
\nWhat I originally wrote was WRONG. Jack clarified his actual approach:
\n✅ What Jack actually does:
\n❌ What Jack does NOT do:
\nKey quote (Feb 2026):\n\"I really started to gain in score when I manually reviewed (briefly - still \nfinding things I've missed) each and every document in the train.csv.\"
\nAbout sentence alignment:\n\"Sentence alignment didn't work for me because my sentences sucked 😭\" \n(This was about HIS implementation, not the approach itself)
\nCritical insight:\n\"If your formatting is off, especially on translations, nothing else you do \nwill matter since your model isn't matching the desired output format.\"
\nFrom 耶✌ :
\nSentences_Oare_FirstWord_LinNum.csv improved score +0.5train.csv (1,561 samples) - Original competition dataSentences_Oare_FirstWord_LinNum.csv (9,782 alignments) - If verified against first_word matchingpublished_texts.csv (7,953 samples) - Transliterations only, unknown time period mixGold (Top 15): Requires custom data OR collaboration OR breakthrough technique
\nSilver (Top 50): Achievable with proven optimization stack (35.1-35.3)
\n(Top 200): Public model baseline (34.9-35.0)
\nKey Decision: Private LB shake-up probability ~60% (per Aaron Bornstein's analysis). Conservative, semantically-accurate approaches may outperform pattern-matching models on final evaluation.
\nMajor Contributors (in alphabetical order):
\nHost (DeepPast Initiative) - For hosting this fascinating competition and providing clarifications throughout
\nIf I missed giving proper credit to anyone or misattributed any techniques, please let me know in the comments and I'll update immediately. This synthesis is based on public discussions and notebooks—if you contributed something I didn't mention, that's my oversight, not intentional!
\nGood luck to everyone in the final push! 51 days to go.
", "rawMarkdown": "## ⚠️ Important Context\n\n**Transparency:** This is a synthesis of community discussions and public notebooks, not original research. I'm currently ranked at 35.1 score and learning from the community myself.\n\n**Timing Note:** Many insights here are from early-to-mid January discussions. \nThe community's understanding has evolved since then. **If you're in the top 50, you probably know more than what's written here.**\n\n**Target Audience:** This consolidation is aimed at competitors who are:\n- New to the competition\n- Stuck at baseline scores (30-35)\n- Looking for a starting point before diving deeper\n\n**What I HAVE tested personally:** AKK-300m failure, bucket batching, length penalty variations, the 35.1 optimization stack.\n\n**What I'm synthesizing from others:** Training recipes, data quality insights, overfitting analysis.\n\n**Request to Top Competitors:** If anything here is outdated or wrong, please comment! I want this to be accurate and helpful, not misleading.\n\n**Purpose of This Post**\n\nAfter weeks of testing, collaboration, and learning from the community, I wanted to consolidate what's actually working (and what's not) to help everyone still pushing for improvements. This is a synthesis of public discussions, shared notebooks, and systematic testing.\n\n**TL;DR:** We're at an interesting inflection point. The public model ceiling is ~35.1, and breaking past it requires either custom training data, novel techniques, or collaboration. Here's what we know.\n\n\nCurrent State of the Leaderboard\n\n**The 35.1 Plateau:**\n- 100s of competitors clustered at exactly 35.1\n- **Why:** Everyone using the same model (Assia Benkedia's `byt5-akkadian-optimized-34x`) with minor variations\n- **Implication:** To break 35.5+, you need something different\n\n**Score Distribution:**\n- Top tier (36.5-38.1): Likely custom training on private Old Assyrian data\n- Competitive cluster (35.1-36.4): Public model + optimizations \n- Copy-paste crowd (34.9-35.0): Direct Assia model usage\n- Baseline methods (<34.9): Older approaches\n\n**Critical Insight from Aaron Bornstein:**\n> \"High scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain\"\n\nPublic LB (34% of test) ≠ Private LB (66% of test). Conservative, semantically-accurate approaches may outperform pattern-matching models on final evaluation.\n\nProven Optimization Stack (34.9 → 35.1)\n\n### 1. Base \"Model\": Assia Benkedia's `byt5-akkadian-optimized-34x` (34.9)\n\n**IMPORTANT CORRECTION** (Thanks to Musa Peker for catching this):\n\nThis is actually an **ENSEMBLE** of three models, not a single model:\n- `/kaggle/input/byt5-akk-gap-sentence-v4-cp-final` (likely Assia's)\n- `/kaggle/input/byt5-akkadian-model`\n- `/kaggle/input/byt5-base-big-data2`\n\n**Implication:** The 34.9 baseline is achieved through ensemble combination, not a \nsingle checkpoint. This means:\n- Single model ceiling is probably ~33-34\n- To beat 35.5, you need either a better base model OR different ensemble components\n- The 35.1 \"plateau\" makes sense - everyone using the same 3-model ensemble + minor optimizations\n\n**Credit:** Assia Benkedia for the ensemble approach, plus the contributors of the \nthree base models\n**Availability:** Kaggle dataset `final-byt5`\n\n### 2. Bucket Batching (+0.1) \n- **Credit:** Sera Ria Gomes \n- **What it does:** Groups similar-length samples together in batches\n- **Why it works:** Reduces padding waste by 20-40% → model sees less noise\n- **Implementation:**\n\n```python\nclass BucketBatchSampler(Sampler):\n \"\"\"Groups samples by similar length to minimize padding\"\"\"\n def __init__(self, lengths, batch_size, num_buckets=4):\n self.lengths = lengths\n self.batch_size = batch_size\n self.num_buckets = num_buckets\n \n # Sort indices by length\n sorted_indices = sorted(range(len(lengths)), key=lambda i: lengths[i])\n \n # Divide into buckets\n bucket_size = len(sorted_indices) // num_buckets\n self.buckets = []\n for i in range(num_buckets):\n start = i * bucket_size\n end = start + bucket_size if i < num_buckets - 1 else len(sorted_indices)\n self.buckets.append(sorted_indices[start:end])\n \n def __iter__(self):\n for bucket in self.buckets:\n indices = bucket.copy()\n for i in range(0, len(indices), self.batch_size):\n yield indices[i:i + self.batch_size]\n \n def __len__(self):\n return sum(len(bucket) // self.batch_size for bucket in self.buckets)\n```\n\n### 3. Length Penalty = 1.3-1.5 (+0.1)\n- **Credit:** manwithacat (discovered optimal range)\n- **Why it works:** Encourages longer, more complete translations\n- **Testing results:**\n - 1.3: 35.1 ✅ (conservative)\n - 1.5: 35.1 ✅ (also works) \n - 1.7: 35.0 ❌ (too aggressive)\n\n### 4. Optimal Configuration\n\n```python\nCONFIG = {\n \"model\": \"byt5-akkadian-optimized-34x\",\n \"num_beams\": 8,\n \"max_new_tokens\": 512,\n \"length_penalty\": 1.3, # or 1.5\n \"early_stopping\": True,\n \"batch_size\": 8,\n \"use_bucket_batching\": True,\n \"num_buckets\": 4\n}\n```\n\n---\n\n## Novel Technique: MBR Decoding (Untested Potential)\n\n- **Credit:** Hikari_30 \n- **What it does:** Minimum Bayes Risk - generates 20 diverse candidates, scores each against all others using chrF++, selects consensus winner\n- **Why it could help:** \n - Reduces hallucination risk (single beam search can hallucinate)\n - Combines diversity (sampling) + quality (beam search)\n - No training required - pure inference technique\n - Validated in NMT research literature\n- **Cost:** ~3-4x slower inference\n- **Status:** Only 1 person testing it publicly - could improve private LB robustness\n\n**Implementation outline:**\n```python\n# Generate 15 diverse candidates (temperature=0.7)\n# Generate 5 beam search candidates\n# Score each candidate against all others using chrF++\n# Select candidate with highest average similarity\n```\n\n---\n\n## ❌ Failed Optimizations (Save Your Time!)\n\n### 1. Onomasticon Name Replacement (0.0 gain)\n- 5,973 Akkadian→English name mappings\n- Expected: +0.5 to +1.0 (host claimed \"biggest single improvement\")\n- **Actual: 34.9 → 34.9** (no change)\n- Why: Model already outputs English names correctly\n\n### 2. Submission Blending (0.0 gain) \n- Attempted: Blend Assia's 34.9 with older 34.6 ensemble\n- Result: Assia's model dominated 100%\n- Conclusion: Need equal-quality models to blend effectively\n\n### 3. Hyperparameter Tuning via Optuna (Minimal Impact)\n- **Credit:** HARUKI HARADA\n- Tested 20 combinations on validation set\n- Best found: `length_penalty=1.788, beams=6` → validation score 25.97\n- Baseline: `length_penalty=1.5, beams=8` → validation score 25.58\n- **BUT:** They still used baseline for submission (didn't trust Optuna)\n- Conclusion: Limited impact (~0.4 max), not worth compute time\n\n### 4. AKK-300m Model (Complete Failure - AVOID!)\n- **Credit:** Thalesian/Akkademia Project \n- Tested Feb 8, 2026\n- **Result:** Outputs only `I made a lot of mistakes in my preprocessing early on.. I’m not totally sure when that quote is from but it was probably early.
\nI’d still argue 95% of this will be preprocessing/formatting your transliterations/translations.. but, I’d follow the recommended guidelines. Also, sentence alignment didn’t work for me because my sentences sucked 😂
\nIf your formatting is off, especially on translations, nothing else you do will matter since your model isn’t matching the desired output format.
\nI would start off with minimal changes to the train.csv but you should 100% test different things and ensure meaning is aligned in your samples as well. I really started to gain in score when I manually reviewed (briefly - still finding things I’ve missed) each and every document in the train.csv.
", "rawMarkdown": "I made a lot of mistakes in my preprocessing early on.. I’m not totally sure when that quote is from but it was probably early.\n\nI’d still argue 95% of this will be preprocessing/formatting your transliterations/translations.. but, I’d follow the recommended guidelines. Also, sentence alignment didn’t work for me because my sentences sucked 😂 \n\nIf your formatting is off, especially on translations, nothing else you do will matter since your model isn’t matching the desired output format. \n\nI would start off with minimal changes to the train.csv but you should 100% test different things and ensure meaning is aligned in your samples as well. I really started to gain in score when I manually reviewed (briefly - still finding things I’ve missed) each and every document in the train.csv. ", "votes": 5 }, { "id": 3403583, "postDate": "2026-02-08T20:31:11.240Z", "content": "Great synthesis! This is a very comprehensive summary of the current state of the competition. Thanks for putting this together.
\nI just wanted to add a small technical correction regarding the \"Base Model\" section (Assia Benkedia's byt5-akkadian-optimized-34x). Upon inspecting the model files and configuration, it appears this is actually an ensemble technique rather than a single model.
\nThe code references a list of BASE_MODELS that includes:
\n/kaggle/input/byt5-akk-gap-sentence-v4-cp-final (Likely Assia's)\n/kaggle/input/byt5-akkadian-model\n/kaggle/input/byt5-base-big-data2
\nSo, the 34.9+ baseline is achieved by ensembling these public models. It’s worth acknowledging that the performance comes from combining the work of multiple contributors/models, not just a single checkpoint.
\nThanks again for the great overview!
", "rawMarkdown": "Great synthesis! This is a very comprehensive summary of the current state of the competition. Thanks for putting this together.\n\nI just wanted to add a small technical correction regarding the \"Base Model\" section (Assia Benkedia's byt5-akkadian-optimized-34x). Upon inspecting the model files and configuration, it appears this is actually an ensemble technique rather than a single model.\n\nThe code references a list of BASE_MODELS that includes:\n\n/kaggle/input/byt5-akk-gap-sentence-v4-cp-final (Likely Assia's)\n/kaggle/input/byt5-akkadian-model\n/kaggle/input/byt5-base-big-data2\n\nSo, the 34.9+ baseline is achieved by ensembling these public models. It’s worth acknowledging that the performance comes from combining the work of multiple contributors/models, not just a single checkpoint.\n\nThanks again for the great overview!", "votes": 3, "replies": [ { "id": 3403590, "postDate": "2026-02-08T20:41:47.077Z", "content": "Thank you for the correction.
\nThis is a great info! This also explains why submission blending didn't work for me (I was trying to blend an ensemble with another ensemble)
\nQuick clarifications:
\nUpdated the post with this info. Really appreciate this technical detail!
", "rawMarkdown": "Thank you for the correction. \n\nThis is a great info! This also explains why submission blending didn't work for me (I was trying to blend an ensemble with another ensemble)\n\nQuick clarifications:\n1. Is the ensemble using simple averaging, or weighted voting?\n2. Do you know if the 3 base models were trained on different data splits, or different approaches?\n3. Does this mean a well-trained single model could potentially reach around 34 on its own?\n\nUpdated the post with this info. Really appreciate this technical detail!", "votes": 1, "replies": [ { "id": 3403592, "postDate": "2026-02-08T20:48:14.643Z", "content": "This is not a classic inference-time ensemble; instead, it merges the model weights.\nThe parameters of three different ByT5 checkpoints were averaged using Weighted Parameter Averaging with predefined ratios, producing a single new model.
", "rawMarkdown": "This is not a classic inference-time ensemble; instead, it merges the model weights.\nThe parameters of three different ByT5 checkpoints were averaged using Weighted Parameter Averaging with predefined ratios, producing a single new model.", "votes": 1 }, { "id": 3404133, "postDate": "2026-02-09T23:20:45.603Z", "content": "Thanks so much for the technical details about the ensemble! I updated my post to mention it's weighted parameter averaging(without specific weights or code though).
\nThis explains why submission blending didn't work for me(I was trying to ensemble an already-ensembled model with another ensemble).
\nOne question if you don't mind: do you know if the 3 base models were trained on different data or just different hyperparameters/random seeds? Trying to figure out if I should be aiming for diversity in training data or diversity in model architecture for my own attempts.
", "rawMarkdown": "Thanks so much for the technical details about the ensemble! I updated my post to mention it's weighted parameter averaging(without specific weights or code though).\n\nThis explains why submission blending didn't work for me(I was trying to ensemble an already-ensembled model with another ensemble). \n\nOne question if you don't mind: do you know if the 3 base models were trained on different data or just different hyperparameters/random seeds? Trying to figure out if I should be aiming for diversity in training data or diversity in model architecture for my own attempts." }, { "id": 3406687, "postDate": "2026-02-16T10:59:31.230Z", "content": "I don’t have definitive information on whether these models were trained on different datasets or whether the differences arise from architectural choices, hyperparameters, or random seeds.
", "rawMarkdown": "I don’t have definitive information on whether these models were trained on different datasets or whether the differences arise from architectural choices, hyperparameters, or random seeds." } ] } ] }, { "id": 3403581, "postDate": "2026-02-08T20:22:37.560Z", "content": "respect the effort. However, I believe many of your quotes and suggestions are not entirely accurate, mainly because they are comments made 2 months ago (!) and I believe we, as a community, have gained a better understanding of the data since then… IMO you need to be prepared to spend many more hours on this task in order to beat the public model and understand why you are doing better, but i could be wrong😀 And until you have gone through the train data at least once, i would refrain from giving any advice to other people…
", "rawMarkdown": "respect the effort. However, I believe many of your quotes and suggestions are not entirely accurate, mainly because they are comments made 2 months ago (!) and I believe we, as a community, have gained a better understanding of the data since then... IMO you need to be prepared to spend many more hours on this task in order to beat the public model and understand why you are doing better, but i could be wrong😀 And until you have gone through the train data at least once, i would refrain from giving any advice to other people...", "votes": 3, "replies": [ { "id": 3403585, "postDate": "2026-02-08T20:32:55.077Z", "content": "Really appreciate the feedback! You're absolutely right that:
\nQuick question, what have you learned in the past 2 months that contradicts or updates the insights here? I'm genuinely curious what the top competitors know now that we didn't know in January. \nHappy to update the post with newer findings!
", "rawMarkdown": "Really appreciate the feedback! You're absolutely right that:\n\n1. I haven't yet gone through the training data line-by-line myself\n2. many of these insights are from early jan discussions (2 months old)\n3. im synthesizing community knowledge, not presenting original research\nMy intent was to consolidate scattered information(added additional context to the post) for people entering the competition now, but I should have been clearer about what I've personally tested vs. what I'm reporting from others.\n\nQuick question, what have you learned in the past 2 months that contradicts or updates the insights here? I'm genuinely curious what the top competitors know now that we didn't know in January. \nHappy to update the post with newer findings!", "replies": [ { "id": 3403591, "postDate": "2026-02-08T20:44:07.267Z", "content": "for example, i believe the \"insight\" you are giving here is very far from being true:
\n\n", "rawMarkdown": "for example, i believe the \"insight\" you are giving here is very far from being true:\n>\"95% is preprocessing. Every attempt to follow instructions for cleaning data led to worse performance.\"\nKeep training data as close to original as possible\nHeavy preprocessing ONLY for test/inference data\nAdding sentence-level data from published_texts.csv made his model WORSE" }, { "id": 3403658, "postDate": "2026-02-09T02:35:20.423Z", "content": "\"95% is preprocessing. Every attempt to follow instructions for cleaning data led to worse performance.\"\n Keep training data as close to original as possible\n Heavy preprocessing ONLY for test/inference data\n Adding sentence-level data from published_texts.csv made his model WORSE
\n
Yes, this is off. I was goin through it at the time.
", "rawMarkdown": "Yes, this is off. I was goin through it at the time. " }, { "id": 3404132, "postDate": "2026-02-09T23:17:17.263Z", "content": "Yeah, good catch on that one.
\nTurns out I completely misread what Jack meant. Pretty much the opposite of what I wrote lol.
\nUpdated the post with the correction.
", "rawMarkdown": "Yeah, good catch on that one.\n\nTurns out I completely misread what Jack meant. Pretty much the opposite of what I wrote lol.\n\nUpdated the post with the correction. " } ] } ] }, { "id": 3403740, "postDate": "2026-02-09T07:21:50.197Z", "content": "I unintentionally made one of my notebooks public. While it wasn't my initial intention, I would be happy if it could contribute to the community's learning and discussion.
\nCurrently, I suspect that the top scores on the Public LB might be overfitting. Therefore, I believe the key to success in this competition—beyond just improving raw model performance—is focusing on robustness. In other words, our goal should be to minimize the \"shake\" and ensure the model generalizes well to the private data.
\nP.S. We are looking for teammates! If you are passionate about LLMs or interested in translation tasks, let’s team up! Please feel free to reach out.
\nHappy Kaggling!
", "rawMarkdown": "I unintentionally made one of my notebooks public. While it wasn't my initial intention, I would be happy if it could contribute to the community's learning and discussion.\n\nCurrently, I suspect that the top scores on the Public LB might be overfitting. Therefore, I believe the key to success in this competition—beyond just improving raw model performance—is focusing on robustness. In other words, our goal should be to minimize the \"shake\" and ensure the model generalizes well to the private data.\n\nP.S. We are looking for teammates! If you are passionate about LLMs or interested in translation tasks, let’s team up! Please feel free to reach out.\n\nHappy Kaggling!", "votes": 2, "replies": [ { "id": 3403818, "postDate": "2026-02-09T10:36:01.090Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3403897, "postDate": "2026-02-09T13:52:49.133Z", "content": "Currently, I suspect that your scores on the Public LB might be underfitting.
", "rawMarkdown": "Currently, I suspect that your scores on the Public LB might be underfitting.", "votes": 2 } ] } ], "comments": [ { "id": 3403657, "author_name": "Jack", "author_url": "", "post_date": "2026-02-09T02:34:55.993000", "content": "I made a lot of mistakes in my preprocessing early on.. I’m not totally sure when that quote is from but it was probably early.
\nI’d still argue 95% of this will be preprocessing/formatting your transliterations/translations.. but, I’d follow the recommended guidelines. Also, sentence alignment didn’t work for me because my sentences sucked 😂
\nIf your formatting is off, especially on translations, nothing else you do will matter since your model isn’t matching the desired output format.
\nI would start off with minimal changes to the train.csv but you should 100% test different things and ensure meaning is aligned in your samples as well. I really started to gain in score when I manually reviewed (briefly - still finding things I’ve missed) each and every document in the train.csv.
", "votes": 5, "replies": [] }, { "id": 3403583, "author_name": "Musa Peker", "author_url": "", "post_date": "2026-02-08T20:31:11.240000", "content": "Great synthesis! This is a very comprehensive summary of the current state of the competition. Thanks for putting this together.
\nI just wanted to add a small technical correction regarding the \"Base Model\" section (Assia Benkedia's byt5-akkadian-optimized-34x). Upon inspecting the model files and configuration, it appears this is actually an ensemble technique rather than a single model.
\nThe code references a list of BASE_MODELS that includes:
\n/kaggle/input/byt5-akk-gap-sentence-v4-cp-final (Likely Assia's)\n/kaggle/input/byt5-akkadian-model\n/kaggle/input/byt5-base-big-data2
\nSo, the 34.9+ baseline is achieved by ensembling these public models. It’s worth acknowledging that the performance comes from combining the work of multiple contributors/models, not just a single checkpoint.
\nThanks again for the great overview!
", "votes": 3, "replies": [ { "id": 3403590, "author_name": "Prayag Patel", "author_url": "", "post_date": "2026-02-08T20:41:47.077000", "content": "Thank you for the correction.
\nThis is a great info! This also explains why submission blending didn't work for me (I was trying to blend an ensemble with another ensemble)
\nQuick clarifications:
\nUpdated the post with this info. Really appreciate this technical detail!
", "votes": 1, "replies": [ { "id": 3403592, "author_name": "Musa Peker", "author_url": "", "post_date": "2026-02-08T20:48:14.643000", "content": "This is not a classic inference-time ensemble; instead, it merges the model weights.\nThe parameters of three different ByT5 checkpoints were averaged using Weighted Parameter Averaging with predefined ratios, producing a single new model.
", "votes": 1, "replies": [] }, { "id": 3404133, "author_name": "Prayag Patel", "author_url": "", "post_date": "2026-02-09T23:20:45.603000", "content": "Thanks so much for the technical details about the ensemble! I updated my post to mention it's weighted parameter averaging(without specific weights or code though).
\nThis explains why submission blending didn't work for me(I was trying to ensemble an already-ensembled model with another ensemble).
\nOne question if you don't mind: do you know if the 3 base models were trained on different data or just different hyperparameters/random seeds? Trying to figure out if I should be aiming for diversity in training data or diversity in model architecture for my own attempts.
", "votes": 0, "replies": [] }, { "id": 3406687, "author_name": "Musa Peker", "author_url": "", "post_date": "2026-02-16T10:59:31.230000", "content": "I don’t have definitive information on whether these models were trained on different datasets or whether the differences arise from architectural choices, hyperparameters, or random seeds.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3403581, "author_name": "hongan", "author_url": "", "post_date": "2026-02-08T20:22:37.560000", "content": "respect the effort. However, I believe many of your quotes and suggestions are not entirely accurate, mainly because they are comments made 2 months ago (!) and I believe we, as a community, have gained a better understanding of the data since then… IMO you need to be prepared to spend many more hours on this task in order to beat the public model and understand why you are doing better, but i could be wrong😀 And until you have gone through the train data at least once, i would refrain from giving any advice to other people…
", "votes": 3, "replies": [ { "id": 3403585, "author_name": "Prayag Patel", "author_url": "", "post_date": "2026-02-08T20:32:55.077000", "content": "Really appreciate the feedback! You're absolutely right that:
\nQuick question, what have you learned in the past 2 months that contradicts or updates the insights here? I'm genuinely curious what the top competitors know now that we didn't know in January. \nHappy to update the post with newer findings!
", "votes": 0, "replies": [ { "id": 3403591, "author_name": "hongan", "author_url": "", "post_date": "2026-02-08T20:44:07.267000", "content": "for example, i believe the \"insight\" you are giving here is very far from being true:
\n\n", "votes": 0, "replies": [] }, { "id": 3403658, "author_name": "Jack", "author_url": "", "post_date": "2026-02-09T02:35:20.423000", "content": "\"95% is preprocessing. Every attempt to follow instructions for cleaning data led to worse performance.\"\n Keep training data as close to original as possible\n Heavy preprocessing ONLY for test/inference data\n Adding sentence-level data from published_texts.csv made his model WORSE
\n
Yes, this is off. I was goin through it at the time.
", "votes": 0, "replies": [] }, { "id": 3404132, "author_name": "Prayag Patel", "author_url": "", "post_date": "2026-02-09T23:17:17.263000", "content": "Yeah, good catch on that one.
\nTurns out I completely misread what Jack meant. Pretty much the opposite of what I wrote lol.
\nUpdated the post with the correction.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3403740, "author_name": "Hikari_30", "author_url": "", "post_date": "2026-02-09T07:21:50.197000", "content": "I unintentionally made one of my notebooks public. While it wasn't my initial intention, I would be happy if it could contribute to the community's learning and discussion.
\nCurrently, I suspect that the top scores on the Public LB might be overfitting. Therefore, I believe the key to success in this competition—beyond just improving raw model performance—is focusing on robustness. In other words, our goal should be to minimize the \"shake\" and ensure the model generalizes well to the private data.
\nP.S. We are looking for teammates! If you are passionate about LLMs or interested in translation tasks, let’s team up! Please feel free to reach out.
\nHappy Kaggling!
", "votes": 2, "replies": [ { "id": 3403818, "author_name": "", "author_url": "", "post_date": "2026-02-09T10:36:01.090000", "content": "", "votes": 0, "replies": [] }, { "id": 3403897, "author_name": "Jack", "author_url": "", "post_date": "2026-02-09T13:52:49.133000", "content": "Currently, I suspect that your scores on the Public LB might be underfitting.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "3403575": "## ⚠️ Important Context\n\n**Transparency:** This is a synthesis of community discussions and public notebooks, not original research. I'm currently ranked at 35.1 score and learning from the community myself.\n\n**Timing Note:** Many insights here are from early-to-mid January discussions. \nThe community's understanding has evolved since then. **If you're in the top 50, you probably know more than what's written here.**\n\n**Target Audience:** This consolidation is aimed at competitors who are:\n- New to the competition\n- Stuck at baseline scores (30-35)\n- Looking for a starting point before diving deeper\n\n**What I HAVE tested personally:** AKK-300m failure, bucket batching, length penalty variations, the 35.1 optimization stack.\n\n**What I'm synthesizing from others:** Training recipes, data quality insights, overfitting analysis.\n\n**Request to Top Competitors:** If anything here is outdated or wrong, please comment! I want this to be accurate and helpful, not misleading.\n\n**Purpose of This Post**\n\nAfter weeks of testing, collaboration, and learning from the community, I wanted to consolidate what's actually working (and what's not) to help everyone still pushing for improvements. This is a synthesis of public discussions, shared notebooks, and systematic testing.\n\n**TL;DR:** We're at an interesting inflection point. The public model ceiling is ~35.1, and breaking past it requires either custom training data, novel techniques, or collaboration. Here's what we know.\n\n\nCurrent State of the Leaderboard\n\n**The 35.1 Plateau:**\n- 100s of competitors clustered at exactly 35.1\n- **Why:** Everyone using the same model (Assia Benkedia's `byt5-akkadian-optimized-34x`) with minor variations\n- **Implication:** To break 35.5+, you need something different\n\n**Score Distribution:**\n- Top tier (36.5-38.1): Likely custom training on private Old Assyrian data\n- Competitive cluster (35.1-36.4): Public model + optimizations \n- Copy-paste crowd (34.9-35.0): Direct Assia model usage\n- Baseline methods (<34.9): Older approaches\n\n**Critical Insight from Aaron Bornstein:**\n> \"High scores on the visible 33% of the test set are likely to degrade significantly on the hidden 67% if they are out of domain\"\n\nPublic LB (34% of test) ≠ Private LB (66% of test). Conservative, semantically-accurate approaches may outperform pattern-matching models on final evaluation.\n\nProven Optimization Stack (34.9 → 35.1)\n\n### 1. Base \"Model\": Assia Benkedia's `byt5-akkadian-optimized-34x` (34.9)\n\n**IMPORTANT CORRECTION** (Thanks to Musa Peker for catching this):\n\nThis is actually an **ENSEMBLE** of three models, not a single model:\n- `/kaggle/input/byt5-akk-gap-sentence-v4-cp-final` (likely Assia's)\n- `/kaggle/input/byt5-akkadian-model`\n- `/kaggle/input/byt5-base-big-data2`\n\n**Implication:** The 34.9 baseline is achieved through ensemble combination, not a \nsingle checkpoint. This means:\n- Single model ceiling is probably ~33-34\n- To beat 35.5, you need either a better base model OR different ensemble components\n- The 35.1 \"plateau\" makes sense - everyone using the same 3-model ensemble + minor optimizations\n\n**Credit:** Assia Benkedia for the ensemble approach, plus the contributors of the \nthree base models\n**Availability:** Kaggle dataset `final-byt5`\n\n### 2. Bucket Batching (+0.1) \n- **Credit:** Sera Ria Gomes \n- **What it does:** Groups similar-length samples together in batches\n- **Why it works:** Reduces padding waste by 20-40% → model sees less noise\n- **Implementation:**\n\n```python\nclass BucketBatchSampler(Sampler):\n \"\"\"Groups samples by similar length to minimize padding\"\"\"\n def __init__(self, lengths, batch_size, num_buckets=4):\n self.lengths = lengths\n self.batch_size = batch_size\n self.num_buckets = num_buckets\n \n # Sort indices by length\n sorted_indices = sorted(range(len(lengths)), key=lambda i: lengths[i])\n \n # Divide into buckets\n bucket_size = len(sorted_indices) // num_buckets\n self.buckets = []\n for i in range(num_buckets):\n start = i * bucket_size\n end = start + bucket_size if i < num_buckets - 1 else len(sorted_indices)\n self.buckets.append(sorted_indices[start:end])\n \n def __iter__(self):\n for bucket in self.buckets:\n indices = bucket.copy()\n for i in range(0, len(indices), self.batch_size):\n yield indices[i:i + self.batch_size]\n \n def __len__(self):\n return sum(len(bucket) // self.batch_size for bucket in self.buckets)\n```\n\n### 3. Length Penalty = 1.3-1.5 (+0.1)\n- **Credit:** manwithacat (discovered optimal range)\n- **Why it works:** Encourages longer, more complete translations\n- **Testing results:**\n - 1.3: 35.1 ✅ (conservative)\n - 1.5: 35.1 ✅ (also works) \n - 1.7: 35.0 ❌ (too aggressive)\n\n### 4. Optimal Configuration\n\n```python\nCONFIG = {\n \"model\": \"byt5-akkadian-optimized-34x\",\n \"num_beams\": 8,\n \"max_new_tokens\": 512,\n \"length_penalty\": 1.3, # or 1.5\n \"early_stopping\": True,\n \"batch_size\": 8,\n \"use_bucket_batching\": True,\n \"num_buckets\": 4\n}\n```\n\n---\n\n## Novel Technique: MBR Decoding (Untested Potential)\n\n- **Credit:** Hikari_30 \n- **What it does:** Minimum Bayes Risk - generates 20 diverse candidates, scores each against all others using chrF++, selects consensus winner\n- **Why it could help:** \n - Reduces hallucination risk (single beam search can hallucinate)\n - Combines diversity (sampling) + quality (beam search)\n - No training required - pure inference technique\n - Validated in NMT research literature\n- **Cost:** ~3-4x slower inference\n- **Status:** Only 1 person testing it publicly - could improve private LB robustness\n\n**Implementation outline:**\n```python\n# Generate 15 diverse candidates (temperature=0.7)\n# Generate 5 beam search candidates\n# Score each candidate against all others using chrF++\n# Select candidate with highest average similarity\n```\n\n---\n\n## ❌ Failed Optimizations (Save Your Time!)\n\n### 1. Onomasticon Name Replacement (0.0 gain)\n- 5,973 Akkadian→English name mappings\n- Expected: +0.5 to +1.0 (host claimed \"biggest single improvement\")\n- **Actual: 34.9 → 34.9** (no change)\n- Why: Model already outputs English names correctly\n\n### 2. Submission Blending (0.0 gain) \n- Attempted: Blend Assia's 34.9 with older 34.6 ensemble\n- Result: Assia's model dominated 100%\n- Conclusion: Need equal-quality models to blend effectively\n\n### 3. Hyperparameter Tuning via Optuna (Minimal Impact)\n- **Credit:** HARUKI HARADA\n- Tested 20 combinations on validation set\n- Best found: `length_penalty=1.788, beams=6` → validation score 25.97\n- Baseline: `length_penalty=1.5, beams=8` → validation score 25.58\n- **BUT:** They still used baseline for submission (didn't trust Optuna)\n- Conclusion: Limited impact (~0.4 max), not worth compute time\n\n### 4. AKK-300m Model (Complete Failure - AVOID!)\n- **Credit:** Thalesian/Akkademia Project \n- Tested Feb 8, 2026\n- **Result:** Outputs only `