{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":121150,"databundleVersionId":14976537,"sourceType":"competition"},{"sourceId":14190587,"sourceType":"datasetVersion","datasetId":9048186}],"dockerImageVersionId":31236,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🏛️ Deep Past Challenge: Baseline Model\n\n## Translating 4,000-Year-Old Assyrian Tablets with Neural Machine Translation\n\nWelcome to this baseline notebook for the Deep Past Challenge! In this competition, we're building machine translation systems to decode ancient Assyrian merchant records—clay tablets that have sat untranslated in museum drawers for over a century.\n\n### What This Notebook Covers\n\n1. **Setup & Environment Fixes** - Handle common Kaggle environment issues\n2. **Data Exploration** - Understanding our ancient texts\n3. **Text Preprocessing** - Handling transliteration formatting conventions\n4. **Data Validation** - Ensuring our pipeline is working correctly\n5. **Model Training** - Fine-tuning a multilingual transformer (mT5)\n6. **Inference & Submission** - Generating translations for the test set\n7. **Ideas for Improvement** - Next steps to beat this baseline\n\n---\n\n### About Old Assyrian\n\nOld Assyrian is an early form of Akkadian, the oldest documented Semitic language. The texts in this competition are **transliterations**—the cuneiform signs have been converted to Latin script by scholars. Our task is to translate these transliterations into English.\n\n**Key Challenges:**\n- **Low-resource language**: Only ~1,500 training examples with translations\n- **Morphological complexity**: A single Akkadian word can encode what takes multiple English words\n- **Alignment mismatch**: Training data is document-level, test data is sentence-level\n- **Specialized notation**: Transliterations include scribal conventions that need preprocessing","metadata":{}},{"cell_type":"markdown","source":"## 1. Setup & Environment Fixes\n\nFirst, let's fix common Kaggle environment issues before importing libraries.","metadata":{}},{"cell_type":"code","source":"# Suppress TensorFlow/CUDA warnings (these are harmless but noisy)\nimport os\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\nos.environ['CUDA_LAUNCH_BLOCKING'] = '0'\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:03:22.300471Z","iopub.execute_input":"2025-12-17T21:03:22.301092Z","iopub.status.idle":"2025-12-17T21:03:22.307976Z","shell.execute_reply.started":"2025-12-17T21:03:22.301064Z","shell.execute_reply":"2025-12-17T21:03:22.307353Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport re\nfrom pathlib import Path\nfrom tqdm.auto import tqdm\n\n# Transformers & ML\nfrom transformers import (\n    AutoTokenizer, \n    AutoModelForSeq2SeqLM,\n    Seq2SeqTrainingArguments,\n    Seq2SeqTrainer,\n    DataCollatorForSeq2Seq\n)\nfrom datasets import Dataset\nimport torch\n\n# Set random seeds for reproducibility\nSEED = 42\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\nif torch.cuda.is_available():\n    torch.cuda.manual_seed_all(SEED)\n\n# Check for GPU\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(f\"Using device: {device}\")\nif torch.cuda.is_available():\n    print(f\"GPU: {torch.cuda.get_device_name(0)}\")\n    print(f\"GPU Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:03:27.265166Z","iopub.execute_input":"2025-12-17T21:03:27.265486Z","iopub.status.idle":"2025-12-17T21:03:38.506507Z","shell.execute_reply.started":"2025-12-17T21:03:27.265463Z","shell.execute_reply":"2025-12-17T21:03:38.505809Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Configuration - Memory-optimized for Kaggle GPUs (T4/P100 with ~16GB VRAM)\nclass Config:\n    # Paths\n    DATA_DIR = Path('/kaggle/input/deep-past-initiative-machine-translation')\n    OUTPUT_DIR = Path('/kaggle/working')\n    MODEL_DIR = OUTPUT_DIR / 'model'\n    \n    # Model\n    MODEL_NAME = '/kaggle/input/googlemt5-small'  # Multilingual T5 - good for low-resource translation\n    MAX_SOURCE_LENGTH = 128  # Reduced for memory efficiency\n    MAX_TARGET_LENGTH = 256  # Reduced for memory efficiency\n    \n    # Training - Memory-optimized settings\n    BATCH_SIZE = 4\n    GRADIENT_ACCUMULATION = 2  # Effective batch size = 4 * 2 = 8\n    LEARNING_RATE = 3e-4\n    NUM_EPOCHS = 5\n    WARMUP_STEPS = 50\n    \n    # Inference\n    NUM_BEAMS = 4\n    INFERENCE_BATCH_SIZE = 8\n    \nconfig = Config()\n\n# Create output directories\nconfig.MODEL_DIR.mkdir(parents=True, exist_ok=True)\nprint(f\"Output directory: {config.OUTPUT_DIR}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:12:48.796572Z","iopub.execute_input":"2025-12-17T21:12:48.796859Z","iopub.status.idle":"2025-12-17T21:12:48.802793Z","shell.execute_reply.started":"2025-12-17T21:12:48.796839Z","shell.execute_reply":"2025-12-17T21:12:48.802148Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Data Loading & Exploration","metadata":{}},{"cell_type":"code","source":"# Load the data\ntrain_df = pd.read_csv(config.DATA_DIR / 'train.csv')\ntest_df = pd.read_csv(config.DATA_DIR / 'test.csv')\n\nprint(\"=\" * 60)\nprint(\"DATA LOADED\")\nprint(\"=\" * 60)\nprint(f\"Training samples: {len(train_df):,}\")\nprint(f\"Test samples: {len(test_df):,}\")\nprint(f\"Unique test documents: {test_df['text_id'].nunique()}\")\nprint(f\"\\nTraining columns: {train_df.columns.tolist()}\")\nprint(f\"Test columns: {test_df.columns.tolist()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:25.532750Z","iopub.execute_input":"2025-12-17T21:13:25.533295Z","iopub.status.idle":"2025-12-17T21:13:25.576224Z","shell.execute_reply.started":"2025-12-17T21:13:25.533266Z","shell.execute_reply":"2025-12-17T21:13:25.575523Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check for missing values\nprint(\"\\n\" + \"=\" * 60)\nprint(\"MISSING VALUES CHECK\")\nprint(\"=\" * 60)\nprint(\"\\nTraining data:\")\nprint(train_df.isnull().sum())\nprint(\"\\nTest data:\")\nprint(test_df.isnull().sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:25.680385Z","iopub.execute_input":"2025-12-17T21:13:25.680602Z","iopub.status.idle":"2025-12-17T21:13:25.687335Z","shell.execute_reply.started":"2025-12-17T21:13:25.680585Z","shell.execute_reply":"2025-12-17T21:13:25.686678Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Preview training data\nprint(\"=\" * 60)\nprint(\"TRAINING DATA SAMPLE\")\nprint(\"=\" * 60)\ntrain_df.head(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:25.833470Z","iopub.execute_input":"2025-12-17T21:13:25.833684Z","iopub.status.idle":"2025-12-17T21:13:25.841803Z","shell.execute_reply.started":"2025-12-17T21:13:25.833668Z","shell.execute_reply":"2025-12-17T21:13:25.841200Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Look at an example transliteration and its translation\nidx = 0\nprint(\"=\" * 60)\nprint(f\"EXAMPLE {idx}: FULL TEXT\")\nprint(\"=\" * 60)\nprint(\"\\nTRANSLITERATION (Akkadian):\")\nprint(\"-\" * 40)\nprint(train_df.iloc[idx]['transliteration'][:800])\nprint(\"\\n\")\nprint(\"TRANSLATION (English):\")\nprint(\"-\" * 40)\nprint(train_df.iloc[idx]['translation'][:800])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:26.124458Z","iopub.execute_input":"2025-12-17T21:13:26.124913Z","iopub.status.idle":"2025-12-17T21:13:26.130354Z","shell.execute_reply.started":"2025-12-17T21:13:26.124891Z","shell.execute_reply":"2025-12-17T21:13:26.129727Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Text length analysis\ntrain_df['src_words'] = train_df['transliteration'].fillna('').str.split().str.len()\ntrain_df['tgt_words'] = train_df['translation'].fillna('').str.split().str.len()\n\nprint(\"=\" * 60)\nprint(\"TEXT LENGTH STATISTICS\")\nprint(\"=\" * 60)\nprint(f\"\\nSource (Akkadian) words:\")\nprint(f\"  Mean: {train_df['src_words'].mean():.1f}\")\nprint(f\"  Median: {train_df['src_words'].median():.1f}\")\nprint(f\"  Max: {train_df['src_words'].max()}\")\nprint(f\"\\nTarget (English) words:\")\nprint(f\"  Mean: {train_df['tgt_words'].mean():.1f}\")\nprint(f\"  Median: {train_df['tgt_words'].median():.1f}\")\nprint(f\"  Max: {train_df['tgt_words'].max()}\")\nprint(f\"\\nExpansion ratio (target/source): {train_df['tgt_words'].mean() / train_df['src_words'].mean():.2f}x\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:26.313284Z","iopub.execute_input":"2025-12-17T21:13:26.313798Z","iopub.status.idle":"2025-12-17T21:13:26.347760Z","shell.execute_reply.started":"2025-12-17T21:13:26.313778Z","shell.execute_reply":"2025-12-17T21:13:26.347068Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Preview test data\nprint(\"=\" * 60)\nprint(\"TEST DATA SAMPLE\")\nprint(\"=\" * 60)\ntest_df.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:26.456556Z","iopub.execute_input":"2025-12-17T21:13:26.457117Z","iopub.status.idle":"2025-12-17T21:13:26.465029Z","shell.execute_reply.started":"2025-12-17T21:13:26.457097Z","shell.execute_reply":"2025-12-17T21:13:26.464377Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Key Observations\n\n1. **Training data is document-level**: Each row is a full tablet with many lines\n2. **Test data is sentence-level**: Each row is a single sentence from a tablet\n3. **English translations are longer**: Target text has more words than source (Akkadian is morphologically dense)\n4. **Special formatting**: Notice hyphens (syllable separators), curly brackets (determinatives), and capitalization conventions","metadata":{}},{"cell_type":"markdown","source":"## 3. Text Preprocessing\n\nBased on the [Dataset Instructions](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions), we need to clean up modern scribal notations while preserving meaningful information.","metadata":{}},{"cell_type":"code","source":"def preprocess_transliteration(text):\n    \"\"\"\n    Clean transliteration text following dataset guidelines.\n    \n    Key transformations:\n    - Remove modern scribal notations (!, ?, /, :, .)\n    - Clean up brackets while preserving determinatives {}\n    - Standardize gaps and breaks\n    - Normalize whitespace\n    \"\"\"\n    if pd.isna(text) or text is None:\n        return \"\"\n    \n    text = str(text).strip()\n    if not text:\n        return \"\"\n    \n    # Remove line numbers (e.g., \"1.\", \"5.\", \"10'.\", \"15''.\")\n    text = re.sub(r\"^\\d+['\\']*\\.?\\s*\", \"\", text, flags=re.MULTILINE)\n    \n    # Remove certain readings markers\n    text = text.replace('!', '')  # Certain reading\n    text = text.replace('?', '')  # Questionable reading\n    \n    # Remove word/line dividers\n    text = text.replace('/', ' ')  # Line divider\n    text = text.replace(':', ' ')  # Word divider (Old Assyrian convention)\n    \n    # Handle scribal insertions - keep the text but remove brackets\n    text = re.sub(r'<<([^>]*)>>', r'\\1', text)  # <<text>> -> text (do this first)\n    text = re.sub(r'<([^<>]*)>', r'\\1', text)   # <text> -> text\n    \n    # Handle partially broken signs - remove half brackets\n    text = text.replace('˹', '').replace('˺', '')\n    \n    # Handle clearly broken signs - remove square brackets but keep content\n    text = re.sub(r'\\[([^\\[\\]]*)\\]', r'\\1', text)\n    \n    # Replace gaps and breaks with standardized tokens\n    text = re.sub(r'\\.\\.+', ' <gap> ', text)  # Multiple dots indicate breaks\n    text = re.sub(r'x{2,}', ' <gap> ', text)   # Multiple x's indicate breaks\n    \n    # Keep curly brackets for determinatives {d}, {ki}, etc.\n    # These are important semantic markers!\n    \n    # Normalize whitespace\n    text = ' '.join(text.split())\n    \n    return text.strip()\n\n\ndef preprocess_translation(text):\n    \"\"\"\n    Clean translation text.\n    Translations are generally cleaner but may have some artifacts.\n    \"\"\"\n    if pd.isna(text) or text is None:\n        return \"\"\n    \n    text = str(text).strip()\n    if not text:\n        return \"\"\n    \n    # Remove line numbers if present\n    text = re.sub(r\"^\\d+['\\']*\\.?\\s*\", \"\", text, flags=re.MULTILINE)\n    \n    # Normalize whitespace\n    text = ' '.join(text.split())\n    \n    return text.strip()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:26.931834Z","iopub.execute_input":"2025-12-17T21:13:26.932398Z","iopub.status.idle":"2025-12-17T21:13:26.939654Z","shell.execute_reply.started":"2025-12-17T21:13:26.932367Z","shell.execute_reply":"2025-12-17T21:13:26.938946Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Test preprocessing on sample texts\nprint(\"=\" * 60)\nprint(\"PREPROCESSING TESTS\")\nprint(\"=\" * 60)\n\ntest_cases = [\n    \"1. um-ma {d}A-šùr-ma-lik-ma 2. a-na Puzur₄-A-šùr [qi₂-bi₂]-ma\",\n    \"ša! a-wi-lim? / KÙ.BABBAR : ša {ki}Ka-ni-iš\",\n    \"<<erasure>> text <insertion> ... broken...\",\n]\n\nfor i, sample in enumerate(test_cases):\n    print(f\"\\nTest {i+1}:\")\n    print(f\"  Original: {sample}\")\n    print(f\"  Cleaned:  {preprocess_transliteration(sample)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:27.119360Z","iopub.execute_input":"2025-12-17T21:13:27.119931Z","iopub.status.idle":"2025-12-17T21:13:27.124791Z","shell.execute_reply.started":"2025-12-17T21:13:27.119908Z","shell.execute_reply":"2025-12-17T21:13:27.124062Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Apply preprocessing to all data\nprint(\"\\nApplying preprocessing to training data...\")\ntrain_df['src_clean'] = train_df['transliteration'].apply(preprocess_transliteration)\ntrain_df['tgt_clean'] = train_df['translation'].apply(preprocess_translation)\n\nprint(\"Applying preprocessing to test data...\")\ntest_df['src_clean'] = test_df['transliteration'].apply(preprocess_transliteration)\n\nprint(\"✓ Preprocessing complete!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:27.265304Z","iopub.execute_input":"2025-12-17T21:13:27.265985Z","iopub.status.idle":"2025-12-17T21:13:27.342887Z","shell.execute_reply.started":"2025-12-17T21:13:27.265961Z","shell.execute_reply":"2025-12-17T21:13:27.342361Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Data Validation\n\n**Critical step!** Let's verify our preprocessing didn't break anything and our data is ready for training.","metadata":{}},{"cell_type":"code","source":"print(\"=\" * 60)\nprint(\"DATA VALIDATION\")\nprint(\"=\" * 60)\n\n# Check for empty strings after preprocessing\nempty_src = (train_df['src_clean'] == '').sum()\nempty_tgt = (train_df['tgt_clean'] == '').sum()\n\nprint(f\"\\nEmpty source texts after preprocessing: {empty_src}\")\nprint(f\"Empty target texts after preprocessing: {empty_tgt}\")\n\n# Remove any rows with empty source or target\ninitial_count = len(train_df)\ntrain_df = train_df[(train_df['src_clean'] != '') & (train_df['tgt_clean'] != '')].reset_index(drop=True)\nremoved_count = initial_count - len(train_df)\n\nif removed_count > 0:\n    print(f\"⚠️ Removed {removed_count} rows with empty text\")\nprint(f\"✓ Final training samples: {len(train_df)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:27.635109Z","iopub.execute_input":"2025-12-17T21:13:27.635611Z","iopub.status.idle":"2025-12-17T21:13:27.644633Z","shell.execute_reply.started":"2025-12-17T21:13:27.635588Z","shell.execute_reply":"2025-12-17T21:13:27.643804Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Preview cleaned data\nprint(\"\\n\" + \"=\" * 60)\nprint(\"CLEANED DATA SAMPLES\")\nprint(\"=\" * 60)\n\nfor i in range(min(3, len(train_df))):\n    print(f\"\\n--- Sample {i+1} ---\")\n    print(f\"Source (first 200 chars):\")\n    print(f\"  {repr(train_df.iloc[i]['src_clean'][:200])}\")\n    print(f\"Target (first 200 chars):\")\n    print(f\"  {repr(train_df.iloc[i]['tgt_clean'][:200])}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:27.823622Z","iopub.execute_input":"2025-12-17T21:13:27.824248Z","iopub.status.idle":"2025-12-17T21:13:27.829588Z","shell.execute_reply.started":"2025-12-17T21:13:27.824225Z","shell.execute_reply":"2025-12-17T21:13:27.828832Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Prepare Data for Training\n\nWe'll format our data for the mT5 model. For translation tasks, we prepend a task prefix to help the model understand what we want it to do.","metadata":{}},{"cell_type":"code","source":"# Task prefix for mT5\nTASK_PREFIX = \"translate Akkadian to English: \"\n\ndef prepare_training_examples(df):\n    \"\"\"Prepare training examples with task prefix.\"\"\"\n    sources = [TASK_PREFIX + str(src) for src in df['src_clean'].tolist()]\n    targets = [str(tgt) for tgt in df['tgt_clean'].tolist()]\n    return {\n        'source': sources,\n        'target': targets\n    }\n\n# Create training dataset\ntrain_data = prepare_training_examples(train_df)\ntrain_dataset = Dataset.from_dict(train_data)\n\nprint(f\"Training examples: {len(train_dataset)}\")\nprint(f\"\\nSample input (first 300 chars):\")\nprint(f\"  {train_dataset[0]['source'][:300]}\")\nprint(f\"\\nSample target (first 300 chars):\")\nprint(f\"  {train_dataset[0]['target'][:300]}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:28.999974Z","iopub.execute_input":"2025-12-17T21:13:29.000480Z","iopub.status.idle":"2025-12-17T21:13:29.028764Z","shell.execute_reply.started":"2025-12-17T21:13:29.000455Z","shell.execute_reply":"2025-12-17T21:13:29.028019Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load tokenizer and model\nprint(f\"Loading model: {config.MODEL_NAME}\")\n\ntokenizer = AutoTokenizer.from_pretrained(config.MODEL_NAME)\nmodel = AutoModelForSeq2SeqLM.from_pretrained(config.MODEL_NAME)\n\nprint(f\"✓ Model loaded successfully\")\nprint(f\"  Parameters: {model.num_parameters():,}\")\nprint(f\"  Vocab size: {tokenizer.vocab_size:,}\")\nprint(f\"  Pad token ID: {tokenizer.pad_token_id}\")\nprint(f\"  EOS token ID: {tokenizer.eos_token_id}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:31.461932Z","iopub.execute_input":"2025-12-17T21:13:31.462541Z","iopub.status.idle":"2025-12-17T21:13:43.082780Z","shell.execute_reply.started":"2025-12-17T21:13:31.462517Z","shell.execute_reply":"2025-12-17T21:13:43.082082Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Test tokenization on a single example BEFORE batch processing\nprint(\"=\" * 60)\nprint(\"TOKENIZATION TEST\")\nprint(\"=\" * 60)\n\ntest_source = train_dataset[0]['source']\ntest_target = train_dataset[0]['target']\n\n# Tokenize source\nsrc_tokens = tokenizer(\n    test_source,\n    max_length=config.MAX_SOURCE_LENGTH,\n    truncation=True,\n    padding='max_length',\n    return_tensors='pt'\n)\n\n# Tokenize target (using text_target parameter - correct way for mT5)\ntgt_tokens = tokenizer(\n    text_target=test_target,\n    max_length=config.MAX_TARGET_LENGTH,\n    truncation=True,\n    padding='max_length',\n    return_tensors='pt'\n)\n\nprint(f\"\\nSource tokenization:\")\nprint(f\"  Shape: {src_tokens['input_ids'].shape}\")\nprint(f\"  First 20 token IDs: {src_tokens['input_ids'][0][:20].tolist()}\")\nprint(f\"  Non-padding tokens: {(src_tokens['input_ids'][0] != tokenizer.pad_token_id).sum().item()}\")\n\nprint(f\"\\nTarget tokenization:\")\nprint(f\"  Shape: {tgt_tokens['input_ids'].shape}\")\nprint(f\"  First 20 token IDs: {tgt_tokens['input_ids'][0][:20].tolist()}\")\nprint(f\"  Non-padding tokens: {(tgt_tokens['input_ids'][0] != tokenizer.pad_token_id).sum().item()}\")\n\n# Decode back to verify\ndecoded_src = tokenizer.decode(src_tokens['input_ids'][0], skip_special_tokens=True)\ndecoded_tgt = tokenizer.decode(tgt_tokens['input_ids'][0], skip_special_tokens=True)\nprint(f\"\\nDecoded source (first 100 chars): {decoded_src[:100]}\")\nprint(f\"Decoded target (first 100 chars): {decoded_tgt[:100]}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:47.928924Z","iopub.execute_input":"2025-12-17T21:13:47.929186Z","iopub.status.idle":"2025-12-17T21:13:47.949027Z","shell.execute_reply.started":"2025-12-17T21:13:47.929157Z","shell.execute_reply":"2025-12-17T21:13:47.948388Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def tokenize_function(examples):\n    \"\"\"\n    Tokenize source and target texts.\n    \n    IMPORTANT: \n    - Use text_target parameter (not as_target_tokenizer which is deprecated)\n    - Replace padding tokens with -100 in labels so they're ignored in loss\n    \"\"\"\n    # Tokenize inputs\n    model_inputs = tokenizer(\n        examples['source'],\n        max_length=config.MAX_SOURCE_LENGTH,\n        truncation=True,\n        padding='max_length'\n    )\n    \n    # Tokenize targets using text_target parameter\n    labels = tokenizer(\n        text_target=examples['target'],\n        max_length=config.MAX_TARGET_LENGTH,\n        truncation=True,\n        padding='max_length'\n    )\n    \n    # CRITICAL: Replace padding token id with -100 so it's ignored in loss computation\n    # This is essential - without it, the model tries to predict padding tokens\n    labels_with_ignore = []\n    for label_ids in labels['input_ids']:\n        labels_with_ignore.append([\n            token_id if token_id != tokenizer.pad_token_id else -100 \n            for token_id in label_ids\n        ])\n    \n    model_inputs['labels'] = labels_with_ignore\n    return model_inputs","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:50.405481Z","iopub.execute_input":"2025-12-17T21:13:50.406133Z","iopub.status.idle":"2025-12-17T21:13:50.411097Z","shell.execute_reply.started":"2025-12-17T21:13:50.406109Z","shell.execute_reply":"2025-12-17T21:13:50.410347Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Tokenize dataset\nprint(\"Tokenizing dataset...\")\ntokenized_train = train_dataset.map(\n    tokenize_function, \n    batched=True,\n    batch_size=100,\n    remove_columns=['source', 'target'],\n    desc=\"Tokenizing\"\n)\nprint(\"✓ Tokenization complete!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:52.858468Z","iopub.execute_input":"2025-12-17T21:13:52.859013Z","iopub.status.idle":"2025-12-17T21:13:55.255313Z","shell.execute_reply.started":"2025-12-17T21:13:52.858978Z","shell.execute_reply":"2025-12-17T21:13:55.254489Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# CRITICAL: Verify tokenized data is valid\nprint(\"=\" * 60)\nprint(\"TOKENIZED DATA VALIDATION\")\nprint(\"=\" * 60)\n\nsample = tokenized_train[0]\nprint(f\"\\nInput IDs length: {len(sample['input_ids'])}\")\nprint(f\"Labels length: {len(sample['labels'])}\")\n\n# Count non-padding tokens in input\nnon_padding_input = sum(1 for x in sample['input_ids'] if x != tokenizer.pad_token_id)\nprint(f\"Non-padding input tokens: {non_padding_input}\")\n\n# Count non-ignored labels (should be > 0!)\nnon_ignored_labels = sum(1 for x in sample['labels'] if x != -100)\nprint(f\"Non-ignored labels (-100): {non_ignored_labels}\")\n\nif non_ignored_labels == 0:\n    print(\"\\n⚠️ CRITICAL WARNING: All labels are -100!\")\n    print(\"   This will cause loss = 0 and no learning!\")\n    print(\"   Check the tokenization function.\")\nelse:\n    print(f\"\\n✓ Labels look valid! Model will learn from {non_ignored_labels} tokens.\")\n\n# Decode labels to verify content\nvalid_label_ids = [x for x in sample['labels'] if x != -100]\ndecoded_labels = tokenizer.decode(valid_label_ids, skip_special_tokens=True)\nprint(f\"\\nDecoded target (first 150 chars): {decoded_labels[:150]}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:13:58.024254Z","iopub.execute_input":"2025-12-17T21:13:58.024887Z","iopub.status.idle":"2025-12-17T21:13:58.034371Z","shell.execute_reply.started":"2025-12-17T21:13:58.024863Z","shell.execute_reply":"2025-12-17T21:13:58.033758Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check a few more samples to be safe\nprint(\"\\nValidating multiple samples...\")\nall_valid = True\nfor i in range(min(10, len(tokenized_train))):\n    sample = tokenized_train[i]\n    non_ignored = sum(1 for x in sample['labels'] if x != -100)\n    if non_ignored == 0:\n        print(f\"  ⚠️ Sample {i}: 0 valid labels!\")\n        all_valid = False\n\nif all_valid:\n    print(\"✓ All samples validated successfully!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:14:00.172853Z","iopub.execute_input":"2025-12-17T21:14:00.173555Z","iopub.status.idle":"2025-12-17T21:14:00.182079Z","shell.execute_reply.started":"2025-12-17T21:14:00.173531Z","shell.execute_reply":"2025-12-17T21:14:00.181276Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Model Training","metadata":{}},{"cell_type":"code","source":"# Data collator for dynamic padding during training\ndata_collator = DataCollatorForSeq2Seq(\n    tokenizer=tokenizer,\n    model=model,\n    padding=True,\n    label_pad_token_id=-100  # Ensure padding in labels is ignored\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:14:02.809474Z","iopub.execute_input":"2025-12-17T21:14:02.810241Z","iopub.status.idle":"2025-12-17T21:14:02.813957Z","shell.execute_reply.started":"2025-12-17T21:14:02.810216Z","shell.execute_reply":"2025-12-17T21:14:02.813197Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Training arguments - optimized for Kaggle GPU memory\ntraining_args = Seq2SeqTrainingArguments(\n    output_dir=str(config.MODEL_DIR),\n    \n    # Training hyperparameters\n    num_train_epochs=config.NUM_EPOCHS,\n    per_device_train_batch_size=config.BATCH_SIZE,\n    gradient_accumulation_steps=config.GRADIENT_ACCUMULATION,\n    learning_rate=config.LEARNING_RATE,\n    warmup_steps=config.WARMUP_STEPS,\n    weight_decay=0.01,\n    \n    # Memory optimization\n    fp16=torch.cuda.is_available(),       # Mixed precision training\n    gradient_checkpointing=True,           # Trade compute for memory\n    optim=\"adafactor\",                     # Memory-efficient optimizer\n    \n    # Logging & saving\n    logging_steps=25,\n    logging_first_step=True,\n    save_strategy='epoch',\n    save_total_limit=2,\n    \n    # Misc\n    predict_with_generate=True,\n    generation_max_length=config.MAX_TARGET_LENGTH,\n    report_to='none',  # Disable wandb/tensorboard for Kaggle\n    seed=SEED,\n    \n    # Disable evaluation during training (we don't have a val set in this baseline)\n    eval_strategy='no'\n)\n\nprint(\"Training configuration:\")\nprint(f\"  Epochs: {config.NUM_EPOCHS}\")\nprint(f\"  Batch size: {config.BATCH_SIZE}\")\nprint(f\"  Gradient accumulation: {config.GRADIENT_ACCUMULATION}\")\nprint(f\"  Effective batch size: {config.BATCH_SIZE * config.GRADIENT_ACCUMULATION}\")\nprint(f\"  Learning rate: {config.LEARNING_RATE}\")\nprint(f\"  Mixed precision (fp16): {torch.cuda.is_available()}\")\nprint(f\"  Gradient checkpointing: True\")\nprint(f\"  Optimizer: Adafactor\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:14:13.785550Z","iopub.execute_input":"2025-12-17T21:14:13.785807Z","iopub.status.idle":"2025-12-17T21:14:13.824713Z","shell.execute_reply.started":"2025-12-17T21:14:13.785789Z","shell.execute_reply":"2025-12-17T21:14:13.824098Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Initialize trainer\ntrainer = Seq2SeqTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized_train,\n    data_collator=data_collator,\n    tokenizer=tokenizer\n)\n\nprint(\"\\n\" + \"=\" * 60)\nprint(\"STARTING TRAINING\")\nprint(\"=\" * 60)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:14:22.393544Z","iopub.execute_input":"2025-12-17T21:14:22.394209Z","iopub.status.idle":"2025-12-17T21:14:22.974157Z","shell.execute_reply.started":"2025-12-17T21:14:22.394187Z","shell.execute_reply":"2025-12-17T21:14:22.973543Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train the model\ntrain_result = trainer.train()\n\nprint(\"\\n\" + \"=\" * 60)\nprint(\"TRAINING COMPLETE\")\nprint(\"=\" * 60)\nprint(f\"\\nTraining loss: {train_result.training_loss:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T21:14:25.981766Z","iopub.execute_input":"2025-12-17T21:14:25.982035Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Save the trained model\nmodel_save_path = config.MODEL_DIR / 'final'\ntrainer.save_model(str(model_save_path))\ntokenizer.save_pretrained(str(model_save_path))\nprint(f\"✓ Model saved to {model_save_path}\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. Inference & Submission","metadata":{}},{"cell_type":"code","source":"def generate_translations(texts, model, tokenizer, batch_size=8):\n    \"\"\"\n    Generate translations for a list of texts.\n    \n    Args:\n        texts: List of source texts (already preprocessed)\n        model: Trained translation model\n        tokenizer: Associated tokenizer\n        batch_size: Batch size for inference\n    \n    Returns:\n        List of translated texts\n    \"\"\"\n    model.eval()\n    translations = []\n    \n    # Add task prefix\n    prefixed_texts = [TASK_PREFIX + str(text) for text in texts]\n    \n    # Process in batches with progress bar\n    for i in tqdm(range(0, len(prefixed_texts), batch_size), desc=\"Translating\"):\n        batch = prefixed_texts[i:i + batch_size]\n        \n        # Tokenize\n        inputs = tokenizer(\n            batch,\n            max_length=config.MAX_SOURCE_LENGTH,\n            truncation=True,\n            padding=True,\n            return_tensors='pt'\n        ).to(model.device)\n        \n        # Generate\n        with torch.no_grad():\n            outputs = model.generate(\n                **inputs,\n                max_length=config.MAX_TARGET_LENGTH,\n                num_beams=config.NUM_BEAMS,\n                early_stopping=True,\n                no_repeat_ngram_size=3,\n                length_penalty=1.0\n            )\n        \n        # Decode\n        decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)\n        translations.extend(decoded)\n    \n    return translations","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Quick sanity check: translate a few training examples\nprint(\"=\" * 60)\nprint(\"SANITY CHECK: Translating Training Samples\")\nprint(\"=\" * 60)\n\nsample_sources = train_df['src_clean'].head(3).tolist()\nsample_targets = train_df['tgt_clean'].head(3).tolist()\nsample_translations = generate_translations(sample_sources, model, tokenizer)\n\nfor i in range(3):\n    print(f\"\\n--- Sample {i+1} ---\")\n    print(f\"Source (first 100): {sample_sources[i][:100]}...\")\n    print(f\"Reference (first 100): {sample_targets[i][:100]}...\")\n    print(f\"Generated (first 100): {sample_translations[i][:100]}...\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Generate translations for test set\nprint(\"\\n\" + \"=\" * 60)\nprint(\"GENERATING TEST SET TRANSLATIONS\")\nprint(\"=\" * 60)\n\ntest_translations = generate_translations(\n    test_df['src_clean'].tolist(),\n    model,\n    tokenizer,\n    batch_size=config.INFERENCE_BATCH_SIZE\n)\n\nprint(f\"\\n✓ Generated {len(test_translations)} translations\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Preview test translations\nprint(\"=\" * 60)\nprint(\"SAMPLE TEST TRANSLATIONS\")\nprint(\"=\" * 60)\n\nfor i in range(min(5, len(test_df))):\n    print(f\"\\n--- Test Sample {i+1} ---\")\n    print(f\"Source: {test_df.iloc[i]['transliteration'][:150]}...\")\n    print(f\"Translation: {test_translations[i][:200]}\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create submission file\nsubmission = pd.DataFrame({\n    'id': test_df['id'],\n    'translation': test_translations\n})\n\n# Handle any empty translations\nempty_count = (submission['translation'] == '').sum() + submission['translation'].isna().sum()\nif empty_count > 0:\n    print(f\"⚠️ Found {empty_count} empty translations, filling with placeholder\")\n    submission['translation'] = submission['translation'].fillna('Translation unavailable')\n    submission['translation'] = submission['translation'].replace('', 'Translation unavailable')\n\n# Validate submission\nprint(\"\\n\" + \"=\" * 60)\nprint(\"SUBMISSION VALIDATION\")\nprint(\"=\" * 60)\nprint(f\"Submission shape: {submission.shape}\")\nprint(f\"Expected shape: ({len(test_df)}, 2)\")\nprint(f\"Columns: {submission.columns.tolist()}\")\nprint(f\"Null values: {submission.isnull().sum().sum()}\")\nprint(f\"Empty strings: {(submission['translation'] == '').sum()}\")\n\n# Preview\nsubmission.head()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Save submission\nsubmission_path = config.OUTPUT_DIR / 'submission.csv'\nsubmission.to_csv(submission_path, index=False)\n\nprint(f\"\\n✓ Submission saved to {submission_path}\")\n\n# Verify the saved file\nsaved_sub = pd.read_csv(submission_path)\nprint(f\"✓ Verified: {len(saved_sub)} rows saved\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **Below I have summarized my ideas for improvement. I will be happy to hear from you in comments section!**","metadata":{}},{"cell_type":"markdown","source":"## 9. Ideas for Improvement \n\nThis baseline gets you started, but there's plenty of room for improvement. Here are some ideas:\n\n### Data Augmentation & Additional Training Data\n- **Extract translations from publications.csv**: The supplemental data contains ~880 scholarly publications with translations. Aligning these with the transliterations could significantly expand training data.\n- **Use the lexicon**: `OA_Lexicon_eBL.csv` maps words to their dictionary forms—useful for handling vocabulary coverage.\n- **Back-translation**: Generate synthetic training data by translating English texts back to Akkadian-like format.\n\n### Model Architecture\n- **Try larger models**: mT5-base or mT5-large (if resources allow)\n- **Specialized translation models**: NLLB-200 is designed specifically for translation\n- **Ensemble multiple models**: Combine predictions from different architectures\n\n### Training Strategies\n- **Sentence-level training**: Split document-level training data into sentences to match test format\n- **Curriculum learning**: Start with shorter/simpler texts, then progress to longer ones\n- **Multi-task learning**: Add auxiliary tasks like language modeling or word alignment\n- **Create a validation set**: Hold out some training data to tune hyperparameters\n\n### Preprocessing Improvements\n- **Better gap handling**: Develop more sophisticated strategies for broken text\n- **Determinative expansion**: Expand determinatives to their meanings (e.g., {d} → 'divine')\n- **Subword tokenization**: Consider custom BPE/SentencePiece trained on Akkadian\n\n### Inference Enhancements\n- **Constrained decoding**: Use the lexicon to constrain name translations\n- **Document context**: Provide context from surrounding sentences in the same tablet\n- **Post-processing**: Fix common errors, ensure proper noun handling\n- **Beam search tuning**: Experiment with different beam sizes and length penalties","metadata":{}}]}