{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":121150,"databundleVersionId":15061024,"isSourceIdPinned":false,"sourceType":"competition"},{"sourceId":14374989,"sourceType":"datasetVersion","datasetId":9147887},{"sourceId":14376272,"sourceType":"datasetVersion","datasetId":9181082},{"sourceId":14398637,"sourceType":"datasetVersion","datasetId":9195725},{"sourceId":14410665,"sourceType":"datasetVersion","datasetId":9203761}],"dockerImageVersionId":31260,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🏺 Complete Guide: Akkadian Translation with ByT5 Ensemble\n\n## Deep Past Challenge - Machine Translation Tutorial\n\n**Author:** Arda Yıldız \n**Last Updated:** January 2026  \n**Local Validation Score:** ~51 Combined (BLEU × chrF++ geometric mean)\n\n---\n\n### 📚 What You'll Learn\n\n1. **Understanding Akkadian Transliteration** - How 4,000-year-old texts are represented\n2. **Why ByT5 Works** - Byte-level tokenization for morphologically complex languages\n3. **Preprocessing Techniques** - Handling determinatives, gaps, and special characters\n4. **Multi-Model Ensemble** - Combining 4 fine-tuned models for better accuracy\n5. **Weighted Voting** - Intelligent prediction combination\n\n---\n\n### 🎯 Competition Overview\n\n**Task:** Translate Old Assyrian Akkadian transliterations → English\n\n**Challenge:** \n- Low-resource language (~1,500 training samples)\n- Morphologically complex (one word = multiple English words)\n- Special formatting: determinatives `{d}`, `{ki}`, gaps `...`, `xx`\n\n**Metric:** `sqrt(BLEU × chrF++)` - Geometric mean\n\n---\n\n### 🙏 If this notebook helps you, please upvote! ⬆️","metadata":{}},{"cell_type":"markdown","source":"## 📦 1. Setup and Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport math\nimport pandas as pd\nimport numpy as np\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForSeq2SeqLM\nfrom tqdm.auto import tqdm\nfrom collections import Counter\nimport warnings\nwarnings.filterwarnings('ignore')\n\nprint(\"🏺 Akkadian Translation - 4 Model Ensemble\")\nprint(\"=\"*50)\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(f\"Device: {device}\")\nif device == \"cuda\":\n    print(f\"GPU: {torch.cuda.get_device_name(0)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T21:59:59.928831Z","iopub.execute_input":"2026-01-18T21:59:59.929216Z","iopub.status.idle":"2026-01-18T22:00:18.356168Z","shell.execute_reply.started":"2026-01-18T21:59:59.929167Z","shell.execute_reply":"2026-01-18T22:00:18.355539Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🔤 2. Understanding Akkadian Transliteration\n\nAkkadian texts use a unique representation system:\n\n| Element | Example | Meaning |\n|---------|---------|--------|\n| **Syllables** | `a-na a-bi-ia` | \"to my father\" |\n| **Determinatives** | `{d}UTU` | Divine name (Shamash) |\n| **Place markers** | `a-šur{ki}` | Geographic location (Ashur) |\n| **Person markers** | `{m}i-ku-num` | Male personal name |\n| **Gaps** | `...` or `xx` | Missing/illegible text |\n| **Logograms** | `KÙ.BABBAR` | Sumerian word for \"silver\" |\n\n### Why ByT5?\n\nByT5 uses **byte-level tokenization** which is perfect for:\n- Special characters (ú, š, ṭ, etc.)\n- Syllable boundaries (hyphens)\n- Mixed scripts (Akkadian + Sumerian logograms)","metadata":{}},{"cell_type":"code","source":"# Load and explore data\ntrain_df = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/test.csv\")\n\nprint(f\"📊 Dataset Statistics:\")\nprint(f\"   Training samples: {len(train_df):,}\")\nprint(f\"   Test samples: {len(test_df):,}\")\n\nprint(f\"\\n📝 Sample Akkadian Text:\")\nprint(f\"   {train_df.iloc[0]['transliteration'][:100]}...\")\nprint(f\"\\n📝 English Translation:\")\nprint(f\"   {train_df.iloc[0]['translation'][:100]}...\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:00:18.357518Z","iopub.execute_input":"2026-01-18T22:00:18.357913Z","iopub.status.idle":"2026-01-18T22:00:18.448796Z","shell.execute_reply.started":"2026-01-18T22:00:18.357891Z","shell.execute_reply":"2026-01-18T22:00:18.448242Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧹 3. Preprocessing\n\nKey preprocessing steps for Akkadian:\n\n1. **Gap Normalization:** `...` → `<big_gap>`, `xx` → `<gap>`\n2. **Preserve Determinatives:** Keep `{d}`, `{ki}`, `{m}` as-is\n3. **Task Prefix:** Add \"translate Akkadian to English: \"","metadata":{}},{"cell_type":"code","source":"import re\n\ndef preprocess_akkadian(text):\n    \"\"\"Preprocess Akkadian transliteration\"\"\"\n    if pd.isna(text):\n        return \"\"\n    text = str(text)\n    \n    # Normalize gaps\n    text = re.sub(r'\\.{3,}', '<big_gap>', text)\n    text = re.sub(r'x{2,}', '<gap>', text)\n    \n    # Clean whitespace\n    text = re.sub(r'\\s+', ' ', text).strip()\n    \n    return text\n\n# Test preprocessing\nsample = \"um-ma kà-ru-um... a-na a-bi-ia qi2-bi2-ma\"\nprint(f\"Original: {sample}\")\nprint(f\"Processed: {preprocess_akkadian(sample)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:00:18.449489Z","iopub.execute_input":"2026-01-18T22:00:18.449740Z","iopub.status.idle":"2026-01-18T22:00:18.455134Z","shell.execute_reply.started":"2026-01-18T22:00:18.449707Z","shell.execute_reply":"2026-01-18T22:00:18.454469Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🤖 4. Model Configuration\n\nWe use **4 fine-tuned ByT5 models** with different training strategies:\n\n| Model | Description | Validation Score |\n|-------|-------------|------------------|\n| `byt5-rag-akkadian-v1` | RAG-enhanced training | 50.93 |\n| `train-gap-all-2` | Gap-aware training | 50.40 |\n| `byt5-akkadian-model` | Standard fine-tuning | 48.70 |\n| `byt5-base-big-data2` | Extended data training | 42.85 |\n\n**Why Ensemble?** Each model captures different patterns. Combining them reduces individual errors.","metadata":{}},{"cell_type":"code","source":"# Model configurations with validation-based weights\nMODELS = {\n    \"byt5-rag-akkadian-v1\": {\n        \"path\": \"/kaggle/input/byt5-rag-akkadian-v1\",\n        \"weight\": 50.93,\n        \"description\": \"RAG-enhanced ByT5\"\n    },\n    \"train-gap-all-2\": {\n        \"path\": \"/kaggle/input/train-gap-all-2/byt5-base-akkadian_gap_setence2\",\n        \"weight\": 50.40,\n        \"description\": \"Gap-aware training\"\n    },\n    \"byt5-akkadian-model\": {\n        \"path\": \"/kaggle/input/byt5-akkadian-model\",\n        \"weight\": 48.70,\n        \"description\": \"Standard fine-tuned\"\n    },\n    \"byt5-base-big-data2\": {\n        \"path\": \"/kaggle/input/byt5-base-big-data2\",\n        \"weight\": 42.85,\n        \"description\": \"Extended data\"\n    },\n}\n\n# Verify all models are accessible\nprint(\"🔍 Checking model availability:\")\navailable_models = {}\nfor name, info in MODELS.items():\n    exists = os.path.exists(info[\"path\"])\n    status = \"✅\" if exists else \"❌\"\n    print(f\"   {status} {name} ({info['description']})\")\n    if exists:\n        available_models[name] = info\n\nprint(f\"\\n📊 {len(available_models)}/{len(MODELS)} models available\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:00:18.456611Z","iopub.execute_input":"2026-01-18T22:00:18.456918Z","iopub.status.idle":"2026-01-18T22:00:18.477601Z","shell.execute_reply.started":"2026-01-18T22:00:18.456899Z","shell.execute_reply":"2026-01-18T22:00:18.477073Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ⚙️ 5. Inference Function\n\n**Generation Parameters Explained:**\n\n- `max_new_tokens=256`: Allow longer translations\n- `num_beams=5`: Beam search for better quality\n- `repetition_penalty=1.2`: Prevent repetitive output\n- `no_repeat_ngram_size=3`: Block repeated 3-grams","metadata":{}},{"cell_type":"code","source":"def generate_predictions(model_path, test_df, device, batch_size=1):\n    \"\"\"\n    Generate translations from a single model.\n    \n    Args:\n        model_path: Path to the fine-tuned model\n        test_df: DataFrame with 'transliteration' column\n        device: 'cuda' or 'cpu'\n    \n    Returns:\n        List of English translations\n    \"\"\"\n    # Load model and tokenizer\n    tokenizer = AutoTokenizer.from_pretrained(model_path)\n    model = AutoModelForSeq2SeqLM.from_pretrained(model_path)\n    model = model.to(device)\n    model.eval()\n    \n    predictions = []\n    prefix = \"translate Akkadian to English: \"\n    \n    for idx, row in tqdm(test_df.iterrows(), total=len(test_df), desc=\"Translating\"):\n        # Preprocess input\n        src = prefix + preprocess_akkadian(row['transliteration'])\n        \n        # Tokenize\n        inputs = tokenizer(\n            src,\n            max_length=256,\n            truncation=True,\n            return_tensors=\"pt\"\n        ).to(device)\n        \n        # Generate\n        with torch.no_grad():\n            outputs = model.generate(\n                **inputs,\n                max_new_tokens=256,\n                num_beams=5,\n                repetition_penalty=1.2,\n                no_repeat_ngram_size=3,\n                early_stopping=True\n            )\n        \n        # Decode\n        pred = tokenizer.decode(outputs[0], skip_special_tokens=True)\n        predictions.append(pred)\n    \n    # Free memory\n    del model, tokenizer\n    gc.collect()\n    torch.cuda.empty_cache()\n    \n    return predictions","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:00:18.479158Z","iopub.execute_input":"2026-01-18T22:00:18.479447Z","iopub.status.idle":"2026-01-18T22:00:18.495441Z","shell.execute_reply.started":"2026-01-18T22:00:18.479419Z","shell.execute_reply":"2026-01-18T22:00:18.494785Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🎯 6. Generate Predictions from All Models\n\nWe run inference on each model sequentially to manage GPU memory.","metadata":{}},{"cell_type":"code","source":"# Store predictions from each model\nall_predictions = {}\n\nfor name, info in available_models.items():\n    print(f\"\\n{'='*60}\")\n    print(f\"🤖 Model: {name}\")\n    print(f\"   {info['description']}\")\n    print(f\"   Weight: {info['weight']}\")\n    print(f\"{'='*60}\")\n    \n    preds = generate_predictions(info[\"path\"], test_df, device)\n    all_predictions[name] = preds\n    \n    print(f\"\\n✅ Generated {len(preds)} predictions\")\n    print(f\"   Sample: {preds[0][:80]}...\")\n\nprint(f\"\\n🎉 All {len(all_predictions)} models completed!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:00:18.496166Z","iopub.execute_input":"2026-01-18T22:00:18.496377Z","iopub.status.idle":"2026-01-18T22:02:51.197465Z","shell.execute_reply.started":"2026-01-18T22:00:18.496359Z","shell.execute_reply":"2026-01-18T22:02:51.196784Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🗳️ 7. Ensemble - Weighted Voting\n\n**Strategy: Weighted Majority Voting**\n\nFor each position in the output:\n1. Collect the word from each model\n2. Weight each word by the model's validation score\n3. Select the word with highest total weight\n\nThis combines the strengths of all models while giving more influence to better-performing ones.","metadata":{}},{"cell_type":"code","source":"def ensemble_weighted_voting(all_preds, weights):\n    \"\"\"\n    Combine predictions using weighted majority voting.\n    \n    For each word position, select the word with highest weighted votes.\n    \"\"\"\n    n_samples = len(list(all_preds.values())[0])\n    model_names = list(all_preds.keys())\n    \n    # Normalize weights to sum to 1\n    total_weight = sum(weights[m] for m in model_names)\n    norm_weights = {m: weights[m] / total_weight for m in model_names}\n    \n    print(f\"📊 Normalized weights:\")\n    for m, w in norm_weights.items():\n        print(f\"   {m}: {w:.3f}\")\n    \n    final_predictions = []\n    \n    for i in tqdm(range(n_samples), desc=\"Ensembling\"):\n        # Get predictions for this sample\n        preds_i = {m: all_preds[m][i] for m in model_names}\n        \n        # Tokenize into words\n        words_by_model = {m: preds_i[m].split() for m in model_names}\n        max_len = max(len(w) for w in words_by_model.values())\n        \n        # Vote for each position\n        result = []\n        for pos in range(max_len):\n            word_votes = {}\n            for m in model_names:\n                if pos < len(words_by_model[m]):\n                    word = words_by_model[m][pos]\n                    word_votes[word] = word_votes.get(word, 0) + norm_weights[m]\n            \n            if word_votes:\n                best_word = max(word_votes.keys(), key=lambda w: word_votes[w])\n                result.append(best_word)\n        \n        final_predictions.append(' '.join(result))\n    \n    return final_predictions\n\n# Apply ensemble\nweights = {name: info[\"weight\"] for name, info in available_models.items()}\nfinal_predictions = ensemble_weighted_voting(all_predictions, weights)\n\nprint(f\"\\n✅ Ensemble complete: {len(final_predictions)} predictions\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:02:51.198411Z","iopub.execute_input":"2026-01-18T22:02:51.198916Z","iopub.status.idle":"2026-01-18T22:02:51.218875Z","shell.execute_reply.started":"2026-01-18T22:02:51.198892Z","shell.execute_reply":"2026-01-18T22:02:51.218055Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📝 8. Create Submission","metadata":{}},{"cell_type":"code","source":"# Create submission DataFrame\nsubmission = pd.DataFrame({\n    'id': test_df['id'],\n    'translation': final_predictions\n})\n\n# Save\nsubmission.to_csv('submission.csv', index=False)\n\nprint(\"=\"*60)\nprint(\"✅ SUBMISSION CREATED!\")\nprint(\"=\"*60)\nprint(f\"File: submission.csv\")\nprint(f\"Rows: {len(submission):,}\")\n\n# Show samples\nprint(\"\\n📝 Sample Predictions:\")\nfor i in range(min(5, len(submission))):\n    print(f\"\\n[{i+1}]\")\n    print(f\"   Input: {test_df.iloc[i]['transliteration'][:60]}...\")\n    print(f\"   Output: {final_predictions[i][:80]}...\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:02:51.220021Z","iopub.execute_input":"2026-01-18T22:02:51.220390Z","iopub.status.idle":"2026-01-18T22:02:51.244436Z","shell.execute_reply.started":"2026-01-18T22:02:51.220362Z","shell.execute_reply":"2026-01-18T22:02:51.243873Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📊 9. Comparison: Individual vs Ensemble\n\nLet's compare predictions from individual models vs our ensemble.","metadata":{}},{"cell_type":"code","source":"# Visual comparison\nsample_idx = 0\nprint(f\"📊 Comparison for sample #{sample_idx}:\")\nprint(f\"\\nInput:\")\nprint(f\"   {test_df.iloc[sample_idx]['transliteration'][:100]}...\")\n\nprint(f\"\\nIndividual Model Outputs:\")\nfor name in all_predictions.keys():\n    print(f\"\\n   [{name}]:\")\n    print(f\"   {all_predictions[name][sample_idx][:100]}...\")\n\nprint(f\"\\n🏆 ENSEMBLE Output:\")\nprint(f\"   {final_predictions[sample_idx][:100]}...\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-01-18T22:02:51.245432Z","iopub.execute_input":"2026-01-18T22:02:51.245846Z","iopub.status.idle":"2026-01-18T22:02:51.262732Z","shell.execute_reply.started":"2026-01-18T22:02:51.245817Z","shell.execute_reply":"2026-01-18T22:02:51.262061Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 💡 10. Tips for Improvement\n\n### What worked well:\n- ✅ **ByT5 byte-level tokenization** - handles special characters naturally\n- ✅ **Ensemble of diverse models** - reduces individual errors\n- ✅ **Weighted voting** - gives more influence to better models\n\n### Potential improvements:\n- 📈 **Postprocessing:** Capitalize proper nouns using onomasticon\n- 📈 **Data augmentation:** Inject synthetic gaps, vary word separators\n- 📈 **Larger models:** Try ByT5-large (requires more GPU memory)\n- 📈 **RAG integration:** Retrieve similar examples for context\n\n### Competition-specific tips:\n- 🎯 **Preserve determinatives:** `{d}`, `{ki}`, `{m}` are important markers\n- 🎯 **Handle gaps gracefully:** Don't hallucinate missing text\n- 🎯 **Focus on chrF++:** Word overlap matters more than exact n-gram match\n\n---\n\n## 🙏 Acknowledgments\n\nThanks to the Deep Past Initiative for this fascinating challenge and to the community for sharing their models:\n- `manwithacat/byt5-rag-akkadian-v1`\n- `llkh0a/byt5-akkadian-model`\n- `jeanjean111/byt5-base-big-data2`\n- `qifeihhh666/train-gap-all-2`\n\n---\n\n### 🙏 If this notebook helped you, please upvote! ⬆️\n\n**Good luck with your submissions!** 🏺","metadata":{}}]}