{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":121150,"databundleVersionId":15061024,"sourceType":"competition"},{"sourceId":14188526,"sourceType":"datasetVersion","datasetId":9046477},{"sourceId":14219211,"sourceType":"datasetVersion","datasetId":9070334}],"dockerImageVersionId":31234,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import shutil; shutil.copy(\"/kaggle/input/figures/Image13.png\", \"/kaggle/working/Image13.png\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-23T15:24:40.228666Z","iopub.execute_input":"2025-12-23T15:24:40.229003Z","iopub.status.idle":"2025-12-23T15:24:40.240169Z","shell.execute_reply.started":"2025-12-23T15:24:40.228971Z","shell.execute_reply":"2025-12-23T15:24:40.239243Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n\n
![](\"Image13.png\")
\n
\n
\n
\n
\n
\n Data Analysis Overview\n
\n
\n This notebook analyzes the linguistic and statistical properties of the Deep Past Initiative dataset\n (Old Assyrian transliterations → English) to understand what drives performance and where noise enters the pipeline.\n
\n
\n
\n
\n
\n What we aim to learn\n
\n
\n - Data structure: document lengths, segmentation behavior, and outliers.
\n - Lexical repetition & sparsity: frequent formulaic tokens vs. rare forms.
\n - Train–test overlap: potential leakage signals (retrieval-like shortcuts).
\n - Generalization needs: whether success depends on memorization/retrieval or true translation robustness.
\n
\n\n
\n
\n Why this matters\n
\n
\n These findings directly guide model choice (character/byte-level vs. subword), normalization,\n alignment strategy, and whether retrieval-style baselines are competitive.\n
\n
\n
\n
\n
\n
\n
\n Length Statistics & Outliers (Document → Segments)\n
\n
\n Exploratory analysis of document length, segmentation quality, and extreme cases.\n
\n
\n
\n
\n
\n Key observation\n
\n\n
\n The source segmentation currently appears to be failing:\n src_n_lines = 1 for all samples (std = 0; min = max = 1).\n This indicates each transliteration document is treated as a single segment, while the English side is split into\n multiple sentences (median ≈ 4; max = 111). As a consequence, alignment often becomes\n 1 source chunk → many target sentences, introducing noise during training.\n
\n
\n
{'='*80}
\n
{html.escape(title)}
\n
{'='*80}
\n
\n
row_idx={int(ridx)}
\n
\n
\n
\n
\n
\n
\n
\n 2) Token & Special-Symbol Analysis (Transliteration)\n
\n
\n We quantify the frequency and distribution of unknown markers, determinatives, numbers, punctuation, and other special symbols\n to assess data quality and potential sources of training noise.\n
\n
\n
\n
\n
\n Main finding\n
\n\n
\n The transliteration text is strongly formulaic (high-frequency tokens such as a-na, ša, um-ma, DUMU),\n while also containing substantial symbolic structure.\n Hyphens dominate the character stream (~18%), numeric tokens are frequent (~7%),\n and unknown/gap markers (x, …) are moderate on average (~3.9%) but produce severe outliers\n (some documents exceed 50% unknown-rate).\n
\n\n
\n
\n Implication for modeling\n
\n
\n These statistics motivate (i) light normalization for dashes/ellipses and long decimals, and (ii) filtering or down-weighting\n high-unknown documents to reduce noisy supervision during seq2seq fine-tuning.\n
\n
\n
\n
\n
\n
\n
\n Alignment quality: what the diagnostics reveal\n
\n
\n The length-ratio and mismatch distributions highlight a small but harmful set of noisy supervision cases.\n
\n
\n
\n
\n
\n Key finding\n
\n\n
\n Most pairs fall in a reasonable regime where English is longer than transliteration (ratio < 1).\n However, a clear long tail of extreme src/tgt ratios is driven by targets that are not full translations,\n e.g., single-name outputs (“Lullu,”) or placeholder sentences (“too broken for translation”).\n These samples create misleading supervision: the model is encouraged to map long sources to trivial targets.\n
\n\n
\n
\n Practical implication\n
\n
\n Before fine-tuning, filter or down-weight high-noise documents:\n (i) placeholder targets, (ii) very short targets, and/or (iii) extreme length ratios.\n This typically reduces alignment noise and stabilizes seq2seq training.\n
\n
\n
\n
\n
\n
\n
\n 4) Baseline Diagnostics: Typical Error Patterns (ByT5)\n
\n
\n We group baseline failures into a small set of recurring error types to guide data cleaning,\n alignment choices, and decoding constraints.\n
\n
\n
\n
\n
Key finding
\n
\n Many references are extremely long (hundreds of words), suggesting that part of the validation set still\n reflects document-level targets or noisy alignments rather than sentence-like pairs.\n This can cause mode collapse (very short outputs like “From.”) and unstable learning.\n
\n
\n
Error categories observed
\n
\n - Severe under-translation: outputs collapse to 1–3 words (template prefixes).
\n - Numbers & units drift: missing or altered quantities (high impact for economic/legal texts).
\n - Hallucination / off-target: repeated names, formulaic phrases unrelated to the source.
\n - Other / mild: partial paraphrases with lost details (often improves with cleaner alignment).
\n
\n