{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":121150,"databundleVersionId":14976537,"sourceType":"competition"}],"dockerImageVersionId":31236,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🏛️ Deep Past: Character-Level Translation from Scratch\n\n### **Introduction**\nThe task is to translate ancient Old Assyrian cuneiform texts into English. Standard word-level models often struggle here because the vocabulary is huge (due to complex inflections) and the training data is small (~1,500 aligned pairs).\n\n\n### **The Strategy: Why Character-Level?**\nInstead of using a standard Word-Piece tokenizer or a pretrained BERT model, this notebook implements a **Sequence-to-Sequence Transformer** that operates on **individual characters**.\n\n#### **1. Handling Morphology & Unknowns**\nAkkadian words are dense. `A single word can encode a sentence's worth of meaning in English`.\n* **Word-Level Problem:** If the test set contains a proper noun or verb form we haven't seen, a word-level model outputs `<UNK>`, destroying the translation.\n* **Char-Level Solution:** A char-level model learns phonetics. Even if it has never seen the name *Aššur-rabi* before, it can learn to copy the sequence `A-š-š-u-r` from source to target because it understands the *letters*, not just whole words.\n\n#### **2. Robustness to \"Garbage\"**\nThe source texts are full of breaks, lacunae, and scribal damage:\n* `[x]` (small break)\n* `...` (large break)\n* Broken signs like `x` or `X`\n\nA standard tokenizer might split `[x]` into `['[', 'x', ']']` or treat it as noise. A character-level model simply learns that the sequence `<gap>` usually maps to an English word like \"broken\" or is skipped entirely, without breaking the tokenization pipeline.\n\n\n### **Architecture Overview**\n* **Input:** Character indices (Vocabulary size: ~80 chars for Akkadian, ~90 for English).\n* **Model:** 4-layer Transformer Encoder-Decoder.\n* **Output:** Character-by-character generation (Greedy Search).\n\n---","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.nn.utils.rnn import pad_sequence\n\nimport pandas as pd\nimport numpy as np\nimport math\nimport random\nimport re\nimport time\nfrom tqdm import tqdm","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:52.018141Z","iopub.execute_input":"2025-12-17T12:47:52.018399Z","iopub.status.idle":"2025-12-17T12:47:53.954377Z","shell.execute_reply.started":"2025-12-17T12:47:52.018371Z","shell.execute_reply":"2025-12-17T12:47:53.953542Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 1. Configuration","metadata":{}},{"cell_type":"code","source":"class Config:\n    SEED = 42\n    MAX_LEN = 256          # Max sequence length. Inputs longer than this are truncated.\n    BATCH_SIZE = 64\n    EPOCHS = 20\n    LEARNING_RATE = 0.0005\n    \n    # Model Hyperparameters\n    D_MODEL = 256\n    NHEAD = 4\n    NUM_LAYERS = 4\n    DROPOUT = 0.1\n    \n    DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n\ndef seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed(seed)\n\nseed_everything(Config.SEED)\nprint(f\"Using device: {Config.DEVICE}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:53.955314Z","iopub.execute_input":"2025-12-17T12:47:53.955718Z","iopub.status.idle":"2025-12-17T12:47:54.019709Z","shell.execute_reply.started":"2025-12-17T12:47:53.955686Z","shell.execute_reply":"2025-12-17T12:47:54.018783Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Data Cleaning & Preprocessing","metadata":{}},{"cell_type":"code","source":"def clean_akkadian(text):\n    if pd.isna(text): return \"\"\n    # Standardize Gaps & Remove Noise\n    text = text.replace(\"[x]\", \"<gap>\")\n    text = text.replace(\"...\", \"<big_gap>\")\n    text = text.replace(\"[... ...]\", \"<big_gap>\")\n    for char in [\"!\", \"?\", \"/\", \":\", \"=\"]:\n        text = text.replace(char, \"\")\n    text = text.replace(\"<\", \"\").replace(\">\", \"\")\n    text = text.replace(\"[\", \"\").replace(\"]\", \"\")\n    text = \" \".join(text.split())\n    return text\n\ndef clean_english(text):\n    if pd.isna(text): return \"\"\n    text = text.replace(\"“\", '\"').replace(\"”\", '\"').replace(\"’\", \"'\")\n    text = \" \".join(text.split())\n    return text\n\n# Load & Clean Data\ntrain_df = pd.read_csv('/kaggle/input/deep-past-initiative-machine-translation/train.csv')\ntest_df = pd.read_csv('/kaggle/input/deep-past-initiative-machine-translation/test.csv')\n\ntrain_df['transliteration'] = train_df['transliteration'].apply(clean_akkadian)\ntrain_df['translation'] = train_df['translation'].apply(clean_english)\ntest_df['transliteration'] = test_df['transliteration'].apply(clean_akkadian)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:54.020727Z","iopub.execute_input":"2025-12-17T12:47:54.021282Z","iopub.status.idle":"2025-12-17T12:47:54.114430Z","shell.execute_reply.started":"2025-12-17T12:47:54.021259Z","shell.execute_reply":"2025-12-17T12:47:54.113839Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Vocabulary Builder","metadata":{}},{"cell_type":"code","source":"class CharTokenizer:\n    def __init__(self):\n        self.char2idx = {\"<pad>\": 0, \"<sos>\": 1, \"<eos>\": 2, \"<unk>\": 3}\n        self.idx2char = {0: \"<pad>\", 1: \"<sos>\", 2: \"<eos>\", 3: \"<unk>\"}\n        \n    def fit(self, texts):\n        unique_chars = set()\n        for text in texts: unique_chars.update(text)\n        for char in sorted(unique_chars):\n            if char not in self.char2idx:\n                idx = len(self.char2idx)\n                self.char2idx[char] = idx\n                self.idx2char[idx] = char\n                \n    def encode(self, text):\n        return [self.char2idx.get(c, 3) for c in text]\n    \n    def decode(self, indices):\n        chars = []\n        for idx in indices:\n            if isinstance(idx, torch.Tensor): idx = idx.item()\n            if idx == 2: break\n            if idx not in [0, 1, 2]: chars.append(self.idx2char.get(idx, \"\"))\n        return \"\".join(chars)\n    \n    def __len__(self): return len(self.char2idx)\n\nsrc_tokenizer = CharTokenizer()\ntgt_tokenizer = CharTokenizer()\n\nall_src_text = pd.concat([train_df['transliteration'], test_df['transliteration']]).tolist()\nall_tgt_text = train_df['translation'].tolist()\n\nsrc_tokenizer.fit(all_src_text)\ntgt_tokenizer.fit(all_tgt_text)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:54.115315Z","iopub.execute_input":"2025-12-17T12:47:54.115599Z","iopub.status.idle":"2025-12-17T12:47:54.143572Z","shell.execute_reply.started":"2025-12-17T12:47:54.115566Z","shell.execute_reply":"2025-12-17T12:47:54.142932Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Dataset","metadata":{}},{"cell_type":"code","source":"class TranslationDataset(Dataset):\n    def __init__(self, df, src_tok, tgt_tok=None, mode='train'):\n        self.df = df\n        self.src_tok = src_tok\n        self.tgt_tok = tgt_tok\n        self.mode = mode\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        src_text = self.df.iloc[idx]['transliteration']\n        src_indices = [1] + self.src_tok.encode(src_text) + [2]\n        \n        # --- TRUNCATION FIX ---\n        if len(src_indices) > Config.MAX_LEN:\n            src_indices = src_indices[:Config.MAX_LEN-1] + [2]\n        \n        if self.mode == 'train':\n            tgt_text = self.df.iloc[idx]['translation']\n            tgt_indices = [1] + self.tgt_tok.encode(tgt_text) + [2]\n            if len(tgt_indices) > Config.MAX_LEN:\n                tgt_indices = tgt_indices[:Config.MAX_LEN-1] + [2]\n            return torch.tensor(src_indices), torch.tensor(tgt_indices)\n        else:\n            return torch.tensor(src_indices)\n\ndef collate_fn(batch):\n    if isinstance(batch[0], tuple):\n        src_batch, tgt_batch = zip(*batch)\n        src_padded = pad_sequence(src_batch, batch_first=True, padding_value=0)\n        tgt_padded = pad_sequence(tgt_batch, batch_first=True, padding_value=0)\n        return src_padded, tgt_padded\n    else:\n        src_padded = pad_sequence(batch, batch_first=True, padding_value=0)\n        return src_padded","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:54.144308Z","iopub.execute_input":"2025-12-17T12:47:54.144557Z","iopub.status.idle":"2025-12-17T12:47:54.153410Z","shell.execute_reply.started":"2025-12-17T12:47:54.144537Z","shell.execute_reply":"2025-12-17T12:47:54.152623Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Transformer Model","metadata":{}},{"cell_type":"code","source":"class PositionalEncoding(nn.Module):\n    def __init__(self, d_model, max_len=5000):\n        super().__init__()\n        pe = torch.zeros(max_len, d_model)\n        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n        pe[:, 0::2] = torch.sin(position * div_term)\n        pe[:, 1::2] = torch.cos(position * div_term)\n        self.register_buffer('pe', pe.unsqueeze(0))\n\n    def forward(self, x):\n        return x + self.pe[:, :x.size(1)]\n\nclass AkkadianTransformer(nn.Module):\n    def __init__(self, src_vocab, tgt_vocab, config):\n        super().__init__()\n        self.src_embedding = nn.Embedding(src_vocab, config.D_MODEL)\n        self.tgt_embedding = nn.Embedding(tgt_vocab, config.D_MODEL)\n        self.pos_encoder = PositionalEncoding(config.D_MODEL, config.MAX_LEN)\n        self.transformer = nn.Transformer(\n            d_model=config.D_MODEL, nhead=config.NHEAD,\n            num_encoder_layers=config.NUM_LAYERS, num_decoder_layers=config.NUM_LAYERS,\n            dim_feedforward=config.D_MODEL * 4, dropout=config.DROPOUT, batch_first=True\n        )\n        self.fc_out = nn.Linear(config.D_MODEL, tgt_vocab)\n        \n    def forward(self, src, tgt):\n        tgt_mask = self.transformer.generate_square_subsequent_mask(tgt.size(1)).to(Config.DEVICE)\n        src_key_padding_mask = (src == 0).to(Config.DEVICE)\n        tgt_key_padding_mask = (tgt == 0).to(Config.DEVICE)\n        src = self.pos_encoder(self.src_embedding(src))\n        tgt = self.pos_encoder(self.tgt_embedding(tgt))\n        output = self.transformer(\n            src, tgt, tgt_mask=tgt_mask,\n            src_key_padding_mask=src_key_padding_mask,\n            tgt_key_padding_mask=tgt_key_padding_mask\n        )\n        return self.fc_out(output)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:54.155463Z","iopub.execute_input":"2025-12-17T12:47:54.155715Z","iopub.status.idle":"2025-12-17T12:47:54.168798Z","shell.execute_reply.started":"2025-12-17T12:47:54.155695Z","shell.execute_reply":"2025-12-17T12:47:54.168028Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Training","metadata":{}},{"cell_type":"code","source":"train_data = train_df.sample(frac=0.9, random_state=Config.SEED)\nval_data = train_df.drop(train_data.index)\ntrain_loader = DataLoader(TranslationDataset(train_data, src_tokenizer, tgt_tokenizer), \n                         batch_size=Config.BATCH_SIZE, shuffle=True, collate_fn=collate_fn)\nval_loader = DataLoader(TranslationDataset(val_data, src_tokenizer, tgt_tokenizer), \n                       batch_size=Config.BATCH_SIZE, shuffle=False, collate_fn=collate_fn)\n\nmodel = AkkadianTransformer(len(src_tokenizer), len(tgt_tokenizer), Config).to(Config.DEVICE)\noptimizer = optim.Adam(model.parameters(), lr=Config.LEARNING_RATE)\ncriterion = nn.CrossEntropyLoss(ignore_index=0)\n\nprint(\"Starting Training...\")\nfor epoch in range(Config.EPOCHS):\n    model.train()\n    total_loss = 0\n    for src, tgt in tqdm(train_loader, desc=f\"Epoch {epoch+1}\"):\n        src, tgt = src.to(Config.DEVICE), tgt.to(Config.DEVICE)\n        tgt_input, tgt_output = tgt[:, :-1], tgt[:, 1:]\n        optimizer.zero_grad()\n        output = model(src, tgt_input)\n        loss = criterion(output.reshape(-1, output.shape[-1]), tgt_output.reshape(-1))\n        loss.backward()\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n        optimizer.step()\n        total_loss += loss.item()\n    print(f\"Loss: {total_loss/len(train_loader):.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:47:54.169825Z","iopub.execute_input":"2025-12-17T12:47:54.170106Z","iopub.status.idle":"2025-12-17T12:51:22.186107Z","shell.execute_reply.started":"2025-12-17T12:47:54.170067Z","shell.execute_reply":"2025-12-17T12:51:22.185311Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. Prediction","metadata":{}},{"cell_type":"code","source":"def predict_greedy(text, model, max_length=Config.MAX_LEN):\n    model.eval()\n    with torch.no_grad():\n        src_indices = [1] + src_tokenizer.encode(text) + [2]\n        \n        # --- FIX 2: TRUNCATE INFERENCE INPUT ---\n        if len(src_indices) > Config.MAX_LEN:\n            src_indices = src_indices[:Config.MAX_LEN-1] + [2]\n            \n        src = torch.tensor(src_indices).unsqueeze(0).to(Config.DEVICE)\n        tgt_indices = [1]\n        \n        for _ in range(max_length - 1): # Ensure we don't exceed MAX_LEN\n            tgt = torch.tensor(tgt_indices).unsqueeze(0).to(Config.DEVICE)\n            output = model(src, tgt)\n            predicted_token = output[0, -1, :].argmax().item()\n            if predicted_token == 2: break\n            tgt_indices.append(predicted_token)\n            \n        return tgt_tokenizer.decode(tgt_indices)\n\nprint(\"Generating Predictions...\")\npredictions = []\nfor text in tqdm(test_df['transliteration'], desc=\"Predicting\"):\n    predictions.append(predict_greedy(text, model))\n\nsubmission_df = pd.DataFrame({'id': test_df['id'], 'translation': predictions})\nsubmission_df.to_csv('submission.csv', index=False)\nprint(\"submission.csv saved successfully!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-17T12:51:22.187046Z","iopub.execute_input":"2025-12-17T12:51:22.187545Z","iopub.status.idle":"2025-12-17T12:51:30.166721Z","shell.execute_reply.started":"2025-12-17T12:51:22.187519Z","shell.execute_reply":"2025-12-17T12:51:30.166078Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}