{"cells":[{"cell_type":"markdown","metadata":{},"source":"# 🏺 Deep Past Challenge - Baseline Starter Notebook\n\n**Competition:** [Deep Past Challenge - Translate Akkadian to English](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation)\n\n**Goal:** Build neural machine translation models to convert Old Assyrian (Akkadian) transliterations into English.\n\n**Evaluation:** Geometric Mean of BLEU and chrF++ scores"},{"cell_type":"markdown","metadata":{},"source":"## 📋 Table of Contents\n\n1. [Setup & Imports](#setup)\n2. [Data Loading & Exploration](#data)\n3. [Text Preprocessing](#preprocessing)\n4. [Model: Fine-tuning mT5 for Translation](#model)\n5. [Training](#training)\n6. [Inference & Submission](#submission)"},{"cell_type":"markdown","metadata":{},"source":"<a id=\"setup\"></a>\n## 1. Setup & Imports"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Install required packages\n!pip install -q transformers datasets sacrebleu sentencepiece accelerate"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"import os\nimport re\nimport pandas as pd\nimport numpy as np\nfrom pathlib import Path\n\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\n\nfrom transformers import (\n    MT5ForConditionalGeneration,\n    MT5Tokenizer,\n    TrainingArguments,\n    Trainer,\n    DataCollatorForSeq2Seq\n)\nfrom datasets import Dataset as HFDataset\nimport sacrebleu\n\n# Set random seeds for reproducibility\ntorch.manual_seed(42)\nnp.random.seed(42)\n\n# Check GPU availability\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using device: {device}\")\nif torch.cuda.is_available():\n    print(f\"GPU: {torch.cuda.get_device_name(0)}\")"},{"cell_type":"markdown","metadata":{},"source":"<a id=\"data\"></a>\n## 2. Data Loading & Exploration"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Load competition data\nDATA_PATH = Path(\"/kaggle/input/deep-past-initiative-machine-translation/\")\n\n# Main training data\ntrain_df = pd.read_csv(DATA_PATH / \"train.csv\")\ntest_df = pd.read_csv(DATA_PATH / \"test.csv\")\nsample_submission = pd.read_csv(DATA_PATH / \"sample_submission.csv\")\n\nprint(\"=== TRAINING DATA ===\")\nprint(f\"Shape: {train_df.shape}\")\nprint(f\"\\nColumns: {list(train_df.columns)}\")\nprint(f\"\\nMissing values:\\n{train_df.isnull().sum()}\")\nprint(f\"\\nDuplicated rows: {train_df.duplicated().sum()}\")\n\nprint(\"\\n=== TEST DATA ===\")\nprint(f\"Shape: {test_df.shape}\")\nprint(f\"\\nColumns: {list(test_df.columns)}\")"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Display sample data\nprint(\"=== SAMPLE TRANSLITERATION & TRANSLATION ===\\n\")\nfor i in range(3):\n    print(f\"Example {i+1}:\")\n    print(f\"OARE ID: {train_df.iloc[i]['oare_id']}\")\n    print(f\"Transliteration: {train_df.iloc[i]['transliteration'][:200]}...\")\n    print(f\"Translation: {train_df.iloc[i]['translation'][:200]}...\")\n    print(\"-\" * 80)"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Text length analysis\ntrain_df['transliteration_len'] = train_df['transliteration'].str.len()\ntrain_df['translation_len'] = train_df['translation'].str.len()\n\nprint(\"=== TEXT LENGTH STATISTICS ===\\n\")\nprint(\"Transliteration (Akkadian):\")\nprint(train_df['transliteration_len'].describe())\nprint(\"\\nTranslation (English):\")\nprint(train_df['translation_len'].describe())"},{"cell_type":"markdown","metadata":{},"source":"<a id=\"preprocessing\"></a>\n## 3. Text Preprocessing\n\nHandle special Akkadian transliteration formatting:\n- Remove modern scribal notation (!, ?, /, :)\n- Normalize brackets and gaps\n- Handle determinatives in curly brackets\n- Normalize special characters (ḫ → h, š, ṣ, ṭ)"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def clean_transliteration(text):\n    \"\"\"\n    Clean Akkadian transliteration text.\n    \"\"\"\n    if pd.isna(text):\n        return \"\"\n    \n    # Convert to string\n    text = str(text)\n    \n    # Remove modern scribal notations\n    text = re.sub(r'!', '', text)  # Certain reading\n    text = re.sub(r'\\?', '', text)  # Uncertain reading\n    text = re.sub(r'/', ' ', text)  # Line divider\n    text = re.sub(r'[:.]', ' ', text)  # Word dividers\n    \n    # Normalize gaps\n    text = re.sub(r'\\[x\\]', '<gap>', text)  # Single sign gap\n    text = re.sub(r'\\[…\\s*…\\]', '<big_gap>', text)  # Large gap\n    text = re.sub(r'…', '<big_gap>', text)  # Ellipsis gaps\n    \n    # Remove partially broken sign markers but keep content\n    text = re.sub(r'˹|˺', '', text)\n    \n    # Normalize special characters\n    text = text.replace('ḫ', 'h').replace('Ḫ', 'H')\n    text = text.replace('ʾ', \"'\")\n    \n    # Keep determinatives in curly brackets as they carry meaning\n    # Keep square brackets for broken text indicators\n    \n    # Normalize whitespace\n    text = ' '.join(text.split())\n    \n    return text.strip()\n\n\ndef clean_translation(text):\n    \"\"\"\n    Clean English translation text.\n    \"\"\"\n    if pd.isna(text):\n        return \"\"\n    \n    text = str(text)\n    \n    # Remove modern scribal notations\n    text = re.sub(r'!', '', text)\n    text = re.sub(r'\\?', '', text)\n    text = re.sub(r'<|>', '', text)  # Remove pointy brackets but keep content\n    \n    # Normalize whitespace\n    text = ' '.join(text.split())\n    \n    return text.strip()"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Apply preprocessing\nprint(\"Applying text preprocessing...\")\n\ntrain_df['transliteration_clean'] = train_df['transliteration'].apply(clean_transliteration)\ntrain_df['translation_clean'] = train_df['translation'].apply(clean_translation)\ntest_df['transliteration_clean'] = test_df['transliteration'].apply(clean_transliteration)\n\n# Show before/after\nprint(\"\\n=== PREPROCESSING EXAMPLES ===\\n\")\nfor i in range(2):\n    print(f\"Example {i+1}:\")\n    print(f\"BEFORE: {train_df.iloc[i]['transliteration'][:150]}...\")\n    print(f\"AFTER:  {train_df.iloc[i]['transliteration_clean'][:150]}...\")\n    print(f\"TRANS:  {train_df.iloc[i]['translation_clean'][:150]}...\")\n    print(\"-\" * 80)"},{"cell_type":"markdown","metadata":{},"source":"<a id=\"model\"></a>\n## 4. Model: Fine-tuning mT5 for Translation\n\nWe'll use **mT5** (multilingual T5) which is well-suited for low-resource translation tasks.\n\nAlternative models to try:\n- `google/mt5-base` or `google/mt5-large`\n- `facebook/nllb-200-distilled-600M` (No Language Left Behind)\n- `Helsinki-NLP/opus-mt-ine-en` (OPUS-MT for Indo-European languages)"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Model configuration\nMODEL_NAME = \"google/mt5-small\"  # Change to mt5-base or mt5-large for better results\nMAX_SOURCE_LENGTH = 512\nMAX_TARGET_LENGTH = 256\nBATCH_SIZE = 8\nEPOCHS = 5\nLEARNING_RATE = 5e-5"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Load tokenizer and model\nprint(f\"Loading {MODEL_NAME}...\")\ntokenizer = MT5Tokenizer.from_pretrained(MODEL_NAME)\nmodel = MT5ForConditionalGeneration.from_pretrained(MODEL_NAME)\n\n# Move to GPU\nmodel = model.to(device)\n\nprint(f\"Model parameters: {sum(p.numel() for p in model.parameters()) / 1e6:.1f}M\")"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Create dataset class\nclass TranslationDataset(Dataset):\n    def __init__(self, df, tokenizer, max_source_len=512, max_target_len=256):\n        self.df = df\n        self.tokenizer = tokenizer\n        self.max_source_len = max_source_len\n        self.max_target_len = max_target_len\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        row = self.df.iloc[idx]\n        \n        # Add task prefix for mT5\n        source_text = \"translate Akkadian to English: \" + row['transliteration_clean']\n        target_text = row['translation_clean']\n        \n        # Tokenize\n        source_encoding = self.tokenizer(\n            source_text,\n            max_length=self.max_source_len,\n            padding='max_length',\n            truncation=True,\n            return_tensors='pt'\n        )\n        \n        target_encoding = self.tokenizer(\n            target_text,\n            max_length=self.max_target_len,\n            padding='max_length',\n            truncation=True,\n            return_tensors='pt'\n        )\n        \n        labels = target_encoding['input_ids']\n        labels[labels == self.tokenizer.pad_token_id] = -100\n        \n        return {\n            'input_ids': source_encoding['input_ids'].flatten(),\n            'attention_mask': source_encoding['attention_mask'].flatten(),\n            'labels': labels.flatten()\n        }"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Create datasets\nfrom sklearn.model_selection import train_test_split\n\n# Split for validation\ntrain_split, val_split = train_test_split(\n    train_df, test_size=0.1, random_state=42\n)\n\nprint(f\"Training samples: {len(train_split)}\")\nprint(f\"Validation samples: {len(val_split)}\")\n\n# Create PyTorch datasets\ntrain_dataset = TranslationDataset(train_split, tokenizer, MAX_SOURCE_LENGTH, MAX_TARGET_LENGTH)\nval_dataset = TranslationDataset(val_split, tokenizer, MAX_SOURCE_LENGTH, MAX_TARGET_LENGTH)"},{"cell_type":"markdown","metadata":{},"source":"<a id=\"training\"></a>\n## 5. Training"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Define training arguments\ntraining_args = TrainingArguments(\n    output_dir=\"/kaggle/working/results\",\n    num_train_epochs=EPOCHS,\n    per_device_train_batch_size=BATCH_SIZE,\n    per_device_eval_batch_size=BATCH_SIZE,\n    learning_rate=LEARNING_RATE,\n    weight_decay=0.01,\n    warmup_steps=100,\n    logging_dir=\"/kaggle/working/logs\",\n    logging_steps=10,\n    evaluation_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    load_best_model_at_end=True,\n    metric_for_best_model=\"eval_loss\",\n    greater_is_better=False,\n    save_total_limit=2,\n    fp16=torch.cuda.is_available(),  # Mixed precision training\n    report_to=\"none\"\n)\n\n# Data collator\ndata_collator = DataCollatorForSeq2Seq(\n    tokenizer=tokenizer,\n    model=model,\n    padding=True\n)"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Initialize trainer\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    eval_dataset=val_dataset,\n    data_collator=data_collator,\n)\n\nprint(\"Starting training...\")\ntrainer.train()"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Save the model\nmodel.save_pretrained(\"/kaggle/working/mt5-akkadian-model\")\ntokenizer.save_pretrained(\"/kaggle/working/mt5-akkadian-model\")\nprint(\"Model saved!\")"},{"cell_type":"markdown","metadata":{},"source":"<a id=\"submission\"></a>\n## 6. Inference & Submission"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def translate_batch(texts, model, tokenizer, batch_size=16):\n    \"\"\"\n    Translate a batch of Akkadian transliterations to English.\n    \"\"\"\n    translations = []\n    \n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i + batch_size]\n        \n        # Add task prefix\n        inputs = [\"translate Akkadian to English: \" + text for text in batch]\n        \n        # Tokenize\n        encoded = tokenizer(\n            inputs,\n            max_length=MAX_SOURCE_LENGTH,\n            padding=True,\n            truncation=True,\n            return_tensors='pt'\n        ).to(device)\n        \n        # Generate\n        with torch.no_grad():\n            outputs = model.generate(\n                **encoded,\n                max_length=MAX_TARGET_LENGTH,\n                num_beams=4,\n                early_stopping=True\n            )\n        \n        # Decode\n        batch_translations = tokenizer.batch_decode(outputs, skip_special_tokens=True)\n        translations.extend(batch_translations)\n        \n        if (i // batch_size + 1) % 10 == 0:\n            print(f\"Processed {min(i + batch_size, len(texts))}/{len(texts)}...\")\n    \n    return translations"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Generate translations for test set\nprint(\"Generating translations for test set...\")\ntest_texts = test_df['transliteration_clean'].tolist()\ntest_translations = translate_batch(test_texts, model, tokenizer)\n\n# Create submission\nsubmission = pd.DataFrame({\n    'id': test_df['id'],\n    'translation': test_translations\n})\n\n# Save submission\nsubmission.to_csv(\"/kaggle/working/submission.csv\", index=False)\nprint(f\"\\nSubmission saved! Shape: {submission.shape}\")\nprint(\"\\nFirst 3 predictions:\")\nprint(submission.head(3))"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"# Validate submission format\nprint(\"\\n=== SUBMISSION VALIDATION ===\")\nprint(f\"Shape matches sample: {submission.shape == sample_submission.shape}\")\nprint(f\"Columns match: {list(submission.columns) == list(sample_submission.columns)}\")\nprint(f\"No missing values: {submission.isnull().sum().sum() == 0}\")\nprint(f\"All IDs present: {set(submission['id']) == set(sample_submission['id'])}\")"},{"cell_type":"markdown","metadata":{},"source":"## 🚀 Next Steps to Improve\n\n1. **Data Augmentation**: Mine additional parallel data from `publications.csv`\n2. **Better Model**: Try `google/mt5-base`, `mt5-large`, or `facebook/nllb-200-distilled-600M`\n3. **Hyperparameter Tuning**: Adjust learning rate, batch size, epochs\n4. **Advanced Preprocessing**: Handle determinatives more carefully\n5. **Ensemble**: Combine multiple model predictions\n6. **Post-processing**: Use the lexicon for proper noun handling\n\n## 📚 Resources\n\n- Competition: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation\n- OARE Database: https://deeppast.org/oare\n- eBL Dictionary: https://www.ebl.lmu.de/dictionary\n- CDLI: https://cdli.earth/"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.0"},"kaggle":{"accelerator":"gpu","competition":"deep-past-initiative-machine-translation"}},"nbformat":4,"nbformat_minor":4}