{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":121150,"databundleVersionId":14976537,"sourceType":"competition"}],"dockerImageVersionId":31236,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"1. Overview\n\n- The Deep Past Challenge presents a classic \"Low-Resource Language\" problem: we have only ~1,500 high-quality translation pairs for a 4,000-year-old language (Old Assyrian). Standard Transformer architectures (like T5) are pre-trained on modern languages (English, French, etc.) and lack the inherent understanding of Akkadian morphology, syntax, and unique scribal conventions.\n\n- This notebook implements Phase 1: Domain Adaptation (Continued Pre-training). Before teaching the model how to translate, we first teach it how to read and structure the Akkadian language using the provided ~8,000 unlabelled transliterations.\n\n2. Strategic Objectives\n- The primary goal of this phase is to move the model's internal representation (Embedding space) from modern languages toward the Old Assyrian domain. We achieve this through three world-class data science pillars:\n\n- Akkadian-Aware Tokenization: We inject 15+ ancient determinatives (e.g., {d} for divinity, {ki} for location) and scribal markers (<gap>) as Special Tokens. This prevents the tokenizer from fragmenting these critical semantic units into meaningless sub-characters.\n\n- Span Corruption (Self-Supervised Learning): Using T5’s specific denoising objective, the model learns to reconstruct missing \"spans\" of Akkadian text. This process forces the model to learn the statistical regularities, verb conjugations, and contextual flow of the dialect.\n\n- Mitigating Data Scarcity: By pre-training on 8,000 documents, the model gains \"linguistic intuition.\" This significantly reduces the risk of overfitting during the Phase 2 fine-tuning on the limited 1,500 translation pairs.\n\n3. Expected Impact\n- By the end of this phase, we will have an \"Akkadian-literate T5 Base.\" This specialized foundation is expected to:\n\n- Increase the BLEU and chrF++ scores significantly compared to vanilla T5.\n\n- Provide better stability during fine-tuning.\n\n- Enhance the model's ability to handle \"Broken text\" and \"Gaps\" through learned contextual inference.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport torch\nimport re\nimport os\nfrom dataclasses import dataclass\nfrom typing import Any, Dict, List\nfrom datasets import Dataset\nfrom transformers import (\n    T5Tokenizer, \n    T5ForConditionalGeneration, \n    T5Config,\n    Trainer, \n    TrainingArguments\n)\n\n# ==========================================\n# 1. Initialize Tokenizer FIRST\n# ==========================================\nMODEL_CHECKPOINT = \"t5-base\"\ntokenizer = T5Tokenizer.from_pretrained(MODEL_CHECKPOINT)\n\n# Add Akkadian Determinatives as Special Tokens\nAKKADIAN_SPECIAL_TOKENS = [\n    '{d}', '{ki}', '{lu2}', '{e2}', '{uru}', '{kur}', '{mi}', '{m}', \n    '{gesh}', '{tug2}', '{dub}', '{id2}', '{mushen}', '{na4}', '{kush}', '{u2}',\n    '<gap>', '<big_gap>'\n]\ntokenizer.add_special_tokens({'additional_special_tokens': AKKADIAN_SPECIAL_TOKENS})\n\n# ==========================================\n# 2. Define Custom Data Collator\n# ==========================================\n@dataclass\nclass AkkadianT5DataCollator:\n    \"\"\"\n    Custom Collator for T5 Span Corruption.\n    Ensures 'tokenizer' is accessible and special tokens are protected.\n    \"\"\"\n    tokenizer: Any\n    noise_density: float = 0.15\n\n    def __call__(self, examples: List[Dict[str, Any]]) -> Dict[str, Any]:\n        # Batching and Padding\n        batch = {k: [ex[k] for ex in examples] for k in examples[0].keys()}\n        batch = self.tokenizer.pad(batch, return_tensors=\"pt\")\n        input_ids = batch[\"input_ids\"]\n        \n        # Create mask (15% noise)\n        probability_matrix = torch.full(input_ids.shape, self.noise_density)\n        \n        # Prevent masking of PAD, EOS, and our custom Determinatives\n        special_tokens_mask = [\n            self.tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) \n            for val in input_ids.tolist()\n        ]\n        probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)\n        \n        masked_indices = torch.bernoulli(probability_matrix).bool()\n        \n        # Prepare labels (T5 learns to predict only the masked parts)\n        labels = input_ids.clone()\n        labels[~masked_indices] = -100  # Ignore non-masked tokens in loss calculation\n\n        batch[\"labels\"] = labels\n        return batch\n\n# Initialize the collator with our newly defined tokenizer\ndata_collator = AkkadianT5DataCollator(tokenizer=tokenizer)\n\n# ==========================================\n# 3. Load Data & Prepare Model\n# ==========================================\n# (Assuming your cleaning functions are already defined in previous cells)\ndf_published = pd.read_csv('/kaggle/input/deep-past-initiative-machine-translation/published_texts.csv')\ntexts = df_published['transliteration'].dropna().sample(min(8000, len(df_published))).tolist()\n\ndef tokenize_fn(examples):\n    return tokenizer(examples[\"text\"], truncation=True, max_length=128)\n\ndataset = Dataset.from_pandas(pd.DataFrame({'text': texts}))\ntokenized_dataset = dataset.map(tokenize_fn, batched=True, remove_columns=[\"text\"])\n\n# Load Model and Resize\nmodel = T5ForConditionalGeneration.from_pretrained(MODEL_CHECKPOINT)\nmodel.resize_token_embeddings(len(tokenizer))\n\n# ==========================================\n# 4. Start Training\n# ==========================================\ntraining_args = TrainingArguments(\n    output_dir=\"./t5-akkadian-pretrain\",\n    per_device_train_batch_size=8,\n    num_train_epochs=5,\n    learning_rate=1e-4,\n    fp16=True,\n    logging_steps=100,\n    save_total_limit=1,\n    report_to=\"none\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized_dataset,\n    data_collator=data_collator,\n)\n\ntrainer.train()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-12-18T07:49:06.507326Z","iopub.execute_input":"2025-12-18T07:49:06.507938Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}