{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":121150,"databundleVersionId":14976537,"sourceType":"competition"}],"dockerImageVersionId":31192,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on 16 December, 2025. By Prata, Marília (mpwolke)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\n#Two lines Required to Plot Plotly\nimport plotly.io as pio\npio.renderers.default = 'iframe'\n\nimport plotly.graph_objs as go\nimport plotly.offline as py\nimport plotly.express as px\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.004355Z","iopub.execute_input":"2025-12-18T12:42:34.005521Z","iopub.status.idle":"2025-12-18T12:42:34.016790Z","shell.execute_reply.started":"2025-12-18T12:42:34.005454Z","shell.execute_reply":"2025-12-18T12:42:34.016011Z"},"_kg_hide-input":true,"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Akkadian to English with neural machine translation\n\nArticle Citation: Gutherz G, Gordin S, Sáenz L, Levy O, Berant J. Translating Akkadian to English with neural machine translation. PNAS Nexus. 2023 May 2;2(5):pgad096. doi: 10.1093/pnasnexus/pgad096. PMID: 37143863; PMCID: PMC10153418.\n\n\"Cuneiform is one of the earliest writing systems in recorded human history (ca. 3,400 BCE–75 CE). Hundreds of thousands of such texts were found over the last two centuries, most of which are written in Sumerian and Akkadian. The authors show the high potential in assisting scholars and interested laypeople alike, by using natural language processing (NLP) methods such as convolutional neural networks (CNN), to automatically translate Akkadian from cuneiform Unicode glyphs directly to English (C2E: Cuneiform to English Task) and from transliteration to English (T2E: Transliteration to English Task).\"\n\n\"The authors showed that high-quality translations can be obtained when translating directly from cuneiform to English, as they got 36.52 and 37.47 Best Bilingual Evaluation Understudy 4 **(BLEU4)** scores for C2E (Cuneiform to English Task) and T2E (Transliteration to English Task), respectively. For C2E, their model was better than the translation memory baseline in 9.43, and for T2E, the difference was even higher and stands at 13.96. The model achieves best results in short- and medium-length sentences (c. 118 or less characters).\"\n\n\"As the number of digitized texts grows, the model can be improved by further training as part of a human-in-the-loop system which corrects the results.\"\n\n\"For the purposes of C2E, the signs of each text were encoded as strings of Unicode cuneiform glyphs generated by the Cuneify tool.\"\n\n\"For the Cuneiform script (C2E), the authors used character-based **tokenization** with a small vocabulary of characters. For transliteration (T2E), they used BytePair Encoding (BPE) with the SentencePiece package, where the size of the vocabulary for the transliteration and English was set to 1,000 and 10,000, respectively.\"\n\n### Fairseq\n\n\"The authors used Fairseq. Fairseq is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks.'\n\n\"There are 2 types of hallucinations: an intrinsic hallucination means that the model has produced a translation error using the input present in the source, and an extrinsic hallucination means that the model has produced a translation error that ignores the input present in the source. Both types of hallucinations could be semantically and syntactically correct, i.e. a fluent translation, yet always lack equivalence to the source\"\n\n\"The best results overall were to be found in the short- and middle-length sentences. Longer sources produced more hallucinations or missing translations in the NMT results. This is promising for the usage in realistic scenarios, since all cuneiform texts are divided into manageable lines on the clay tablet.\"\n\n\"The number of characters on an inscribed **clay tablet** can vary from period to period (signs in the **Old Babylonian period** are bigger than in the **Neo-Babylonian period**) and from **genre to genre**. Also, the number of columns by which a tablet is divided will determine the number of characters. Even in a single tablet, the number can vary to fill the space in the line.\"\n\n\"For future use in a human-in-the-loop system, the authors would define each text line on the tablet as a unit for translation.\"\n\n\"The usage of standard sentence aligners such as the **Gale–Church algorithm** can be considered as a first step of the process to handle the alignment problem of the data set.\"\n\n### Python Package Akkademia\n\n\"The input data for C2E could originate from already existing OCR tools in the **Babylonian Engine (CuRe Demo and Decuneify**), which would even allow a layperson to produce an NMT with the online notebook accompanying that publication, as well as part of the **python package Akkademia** (Akkadian).\"\n\nhttps://pmc.ncbi.nlm.nih.gov/articles/PMC10153418/\n\nhttps://github.com/cdli-gh/Machine-Translation\n\nhttps://github.com/gaigutherz/Akkademia","metadata":{}},{"cell_type":"markdown","source":"![](https://localfonts.eu/wp-content/uploads/2021/05/Akkadian_950x475_0.jpg)localfonts","metadata":{}},{"cell_type":"markdown","source":"## Compettion citation:\n\n@misc{deep-past-initiative-machine-translation,\n\n    author = {Abdulla, F. and Agarwal, R. and Anderson, A. and Barjamovic, G. and Lassen, A. and Ryan Holbrook and María Cruz},\n    \n    title = {Deep Past Challenge - Translate Akkadian to English},\n    year = {2025},\n    \n    howpublished = {\\url{https://kaggle.com/competitions/deep-past-initiative-machine-translation}},\n    note = {Kaggle}\n}","metadata":{}},{"cell_type":"markdown","source":"## Load submission file\n\nOnly 4 rows.","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/sample_submission.csv\")\nsub.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.018594Z","iopub.execute_input":"2025-12-18T12:42:34.018886Z","iopub.status.idle":"2025-12-18T12:42:34.079173Z","shell.execute_reply.started":"2025-12-18T12:42:34.018864Z","shell.execute_reply":"2025-12-18T12:42:34.078252Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## train file\n\n\"About 1500 transliterations of Old Assyrian texts from the original excavated tablets each with a translation into English.\"\n\n**oare_id** - Identifier in the **Old Assyrian Research Environment (OARE)** database. Uniquely identifies each text.\n\n**transliteration** - An Akkadian transliteration of the original tablet text.\n\n**translation** - A corresponding English translation.\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/train.csv\")\ntrain.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.080311Z","iopub.execute_input":"2025-12-18T12:42:34.080730Z","iopub.status.idle":"2025-12-18T12:42:34.135665Z","shell.execute_reply.started":"2025-12-18T12:42:34.080700Z","shell.execute_reply":"2025-12-18T12:42:34.134702Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#The 1558 transliteration\n\nprint(train['transliteration'][1558])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.136690Z","iopub.execute_input":"2025-12-18T12:42:34.137029Z","iopub.status.idle":"2025-12-18T12:42:34.143679Z","shell.execute_reply.started":"2025-12-18T12:42:34.137000Z","shell.execute_reply":"2025-12-18T12:42:34.142622Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Split document on whitespace","metadata":{}},{"cell_type":"code","source":"# Split document on whitespace\n#produced a list of tokens such as words, dates, numbers, and special characters\n\nprint(train['transliteration'][1558].split())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.146222Z","iopub.execute_input":"2025-12-18T12:42:34.146568Z","iopub.status.idle":"2025-12-18T12:42:34.159448Z","shell.execute_reply.started":"2025-12-18T12:42:34.146545Z","shell.execute_reply":"2025-12-18T12:42:34.158528Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Number of tokens","metadata":{}},{"cell_type":"code","source":"# Count the number of tokens in document\nlen(train['transliteration'][1558].split())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.160388Z","iopub.execute_input":"2025-12-18T12:42:34.160677Z","iopub.status.idle":"2025-12-18T12:42:34.175612Z","shell.execute_reply.started":"2025-12-18T12:42:34.160657Z","shell.execute_reply":"2025-12-18T12:42:34.174633Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## apply () function to count the number of tokens","metadata":{}},{"cell_type":"code","source":"#apply () function to count the number of tokens\ntrain['transliteration'].apply(lambda x: len(str(x).split()))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.176532Z","iopub.execute_input":"2025-12-18T12:42:34.176829Z","iopub.status.idle":"2025-12-18T12:42:34.205551Z","shell.execute_reply.started":"2025-12-18T12:42:34.176800Z","shell.execute_reply":"2025-12-18T12:42:34.204552Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Token counts in Transliterations","metadata":{}},{"cell_type":"code","source":"# Plot the number of tokens per transliterations\n\n# Barplot and font specifications\nbarplot_specs = {\"color\": \"mediumpurple\", \"alpha\": 0.7, \"edgecolor\": \"grey\"}\nlabel_specs   = {\"fontsize\": 12}\ntitle_specs   = {\"fontsize\": 14, \"fontweight\": \"bold\", \"y\": 1.01}\n\nplt.figure(figsize=(8,4))\n\nplt.hist(train['transliteration'].apply(lambda x:len(str(x).split())), **barplot_specs)\nplt.xlabel('Nbr of tokens', **label_specs)\nplt.ylabel('Nbr of transliterations',**label_specs)\nplt.title('Distribution of token counts in transliterations',**title_specs)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.206578Z","iopub.execute_input":"2025-12-18T12:42:34.207024Z","iopub.status.idle":"2025-12-18T12:42:34.473496Z","shell.execute_reply.started":"2025-12-18T12:42:34.206997Z","shell.execute_reply":"2025-12-18T12:42:34.472756Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Describe method\n\nWe got the shortest (min), the longest (max) and median (50%).","metadata":{}},{"cell_type":"code","source":"#shortest transliteration has 3 (three) tokens (min)\n#the longest has 187 tokens (max)\n#The median number of tokens per book description is 49 (50%)\n\ntrain['transliteration'].apply(lambda x: len(str(x).split())).describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.474331Z","iopub.execute_input":"2025-12-18T12:42:34.474701Z","iopub.status.idle":"2025-12-18T12:42:34.498017Z","shell.execute_reply.started":"2025-12-18T12:42:34.474679Z","shell.execute_reply":"2025-12-18T12:42:34.497164Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sample = train['transliteration'][1558]\nprint(sample)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.498900Z","iopub.execute_input":"2025-12-18T12:42:34.499181Z","iopub.status.idle":"2025-12-18T12:42:34.515840Z","shell.execute_reply.started":"2025-12-18T12:42:34.499161Z","shell.execute_reply":"2025-12-18T12:42:34.514927Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## split the document\n\nStart by splitting the document on whitespace. Then count the instances of each token to find the most frequent one.","metadata":{}},{"cell_type":"code","source":"# Split text on whitespace\nprint(sample.split()[0:20])\n\n# Note: we show only the first 200 tokens","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.517274Z","iopub.execute_input":"2025-12-18T12:42:34.517667Z","iopub.status.idle":"2025-12-18T12:42:34.532688Z","shell.execute_reply.started":"2025-12-18T12:42:34.517632Z","shell.execute_reply":"2025-12-18T12:42:34.531670Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import re\n\n# Replace punctuation marks (,.()) with whitespace\nclean_sample = re.sub(r\"[,.()]\", # regex pattern\n                      \" \",       # replace with whitespace\n                      sample     # string\n                     )\n\n# Replace single hyphens with whitespace\nclean_sample = re.sub(r\"\\s-\\s\",      # regex pattern\n                      \" \",           # replace with whitespace\n                      clean_sample   # string\n                     )\n\n# Print clean_sample\nprint(clean_sample)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.533691Z","iopub.execute_input":"2025-12-18T12:42:34.534030Z","iopub.status.idle":"2025-12-18T12:42:34.549743Z","shell.execute_reply.started":"2025-12-18T12:42:34.533987Z","shell.execute_reply":"2025-12-18T12:42:34.548781Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Clean_sample\n\nSplit the clean_sample on whitespace. Unfortunately, I still see single hiphens : (","metadata":{}},{"cell_type":"code","source":"# Split clean_sample on whitespace\nprint(clean_sample.split()[0:20])\n\n# Note: we show only the first 20 tokens","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.550837Z","iopub.execute_input":"2025-12-18T12:42:34.551169Z","iopub.status.idle":"2025-12-18T12:42:34.566026Z","shell.execute_reply.started":"2025-12-18T12:42:34.551141Z","shell.execute_reply":"2025-12-18T12:42:34.565009Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Most common tokens","metadata":{}},{"cell_type":"code","source":"from collections import Counter\n\n# Print top-10 most frequent tokens in clean_sample\nCounter(clean_sample.split()).most_common(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.569377Z","iopub.execute_input":"2025-12-18T12:42:34.569731Z","iopub.status.idle":"2025-12-18T12:42:34.586681Z","shell.execute_reply.started":"2025-12-18T12:42:34.569707Z","shell.execute_reply":"2025-12-18T12:42:34.585693Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Lowercase","metadata":{}},{"cell_type":"code","source":"# Case folding\nprocessed_sample = clean_sample.lower()\n\nprint(\"Convert to lowercase: \\n\\n{}\\n\".format(processed_sample))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.587688Z","iopub.execute_input":"2025-12-18T12:42:34.588024Z","iopub.status.idle":"2025-12-18T12:42:34.604796Z","shell.execute_reply.started":"2025-12-18T12:42:34.587987Z","shell.execute_reply":"2025-12-18T12:42:34.603802Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## NLTK Stopwords","metadata":{}},{"cell_type":"code","source":"# Load English stopwords from nltk\n\nimport nltk\n\nnltk.download(\"stopwords\", quiet=True)\n\nfrom nltk.corpus import stopwords\n\nstopwords_english = stopwords.words(\"english\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:34.605781Z","iopub.execute_input":"2025-12-18T12:42:34.606059Z","iopub.status.idle":"2025-12-18T12:42:35.415294Z","shell.execute_reply.started":"2025-12-18T12:42:34.606039Z","shell.execute_reply":"2025-12-18T12:42:35.414515Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Print stopwords\n\nprint(\"{} stopwords in NLTK's list.\\n\".format(len(stopwords_english)))\nprint(stopwords_english)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:35.416110Z","iopub.execute_input":"2025-12-18T12:42:35.416369Z","iopub.status.idle":"2025-12-18T12:42:35.421102Z","shell.execute_reply.started":"2025-12-18T12:42:35.416350Z","shell.execute_reply":"2025-12-18T12:42:35.420339Z"},"_kg_hide-output":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Remove stop words using list comprehensions","metadata":{}},{"cell_type":"code","source":"# Use a list comprehension to remove stopwords from processed_sample\n\nprocessed_sample_tokens = [tok for tok in processed_sample.split() if tok not in stopwords_english]\nprint(processed_sample_tokens)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:35.421965Z","iopub.execute_input":"2025-12-18T12:42:35.422259Z","iopub.status.idle":"2025-12-18T12:42:35.441594Z","shell.execute_reply.started":"2025-12-18T12:42:35.422227Z","shell.execute_reply":"2025-12-18T12:42:35.440673Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Recount words after removals","metadata":{}},{"cell_type":"code","source":"# Print top-10 most frequent tokens in processed_sample_tokens\n\nCounter(processed_sample_tokens).most_common(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:35.442626Z","iopub.execute_input":"2025-12-18T12:42:35.442960Z","iopub.status.idle":"2025-12-18T12:42:35.460435Z","shell.execute_reply.started":"2025-12-18T12:42:35.442926Z","shell.execute_reply":"2025-12-18T12:42:35.459545Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Combination of all previous cleaning steps","metadata":{}},{"cell_type":"code","source":"def preprocess_text(text):\n    \n    '''\n    This function preprocesses raw samples of text:\n    - Converts to lowercase\n    - Replaces common punctuation marks with whitespace\n    - Removes stop words\n    - Splits text on whitespace\n    \n    INPUT: \n    - text: \"raw\" text (string)\n    \n    OUTPUT:\n    - processed_sample_tokens: list of tokens (list of strings)\n    '''\n    \n    # Convert to lowercase\n    text = str (text) #Add that line to avoid float has no attribute lower\n    clean_text = text.lower()\n    \n    # Replace common punctuation marks with whitespace\n    clean_text = re.sub(r\"[,.()]\",\" \", clean_text)\n    \n    # Replace single hyphens with whitespace\n    clean_text = re.sub(r\"\\s-\\s\",\" \", clean_text)\n    \n    # Remove stop words and split on whitespace\n    processed_sample_tokens = [tok for tok in clean_text.split() if tok not in stopwords_english]\n    \n    return processed_sample_tokens","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:35.461428Z","iopub.execute_input":"2025-12-18T12:42:35.461807Z","iopub.status.idle":"2025-12-18T12:42:35.475090Z","shell.execute_reply.started":"2025-12-18T12:42:35.461776Z","shell.execute_reply":"2025-12-18T12:42:35.474344Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## apply () method","metadata":{}},{"cell_type":"code","source":"# Alternative (using the apply method)\n\ntrain['clean_transliteration'] = train['transliteration'].apply(lambda x: preprocess_text(x))\n\n# Show an example of preprocessed and tokenized text\ntrain.sample()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:35.476016Z","iopub.execute_input":"2025-12-18T12:42:35.476317Z","iopub.status.idle":"2025-12-18T12:42:35.809274Z","shell.execute_reply.started":"2025-12-18T12:42:35.476284Z","shell.execute_reply":"2025-12-18T12:42:35.808392Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## List comprehensions","metadata":{}},{"cell_type":"code","source":"# another alternative (using list comprehensions)\n\npreprocessed_transliterations = [preprocess_text(doc) for doc in train['transliteration']]\n\n# Show an example of processed and tokenized text\nprint(preprocessed_transliterations[0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:35.810074Z","iopub.execute_input":"2025-12-18T12:42:35.810330Z","iopub.status.idle":"2025-12-18T12:42:36.118935Z","shell.execute_reply.started":"2025-12-18T12:42:35.810301Z","shell.execute_reply":"2025-12-18T12:42:36.118010Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Import Gensim and Phrases\n\nfrom gensim.models import Phrases","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:42:36.119796Z","iopub.execute_input":"2025-12-18T12:42:36.120032Z","iopub.status.idle":"2025-12-18T12:43:19.891356Z","shell.execute_reply.started":"2025-12-18T12:42:36.120013Z","shell.execute_reply":"2025-12-18T12:43:19.890361Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Find common collocations in book descriptions\n\n# Create an instance of the Phrases model\nn_gram_model = Phrases(preprocessed_transliterations,       # Expected input format: list of tokenized texts\n                       min_count=5,                       # Keeps only collocations with at least min_counts \n                       delimiter='_') #Original is delimiter=b'_' With b we have error # Glue character used to join collocation tokens","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:19.892371Z","iopub.execute_input":"2025-12-18T12:43:19.893001Z","iopub.status.idle":"2025-12-18T12:43:20.019049Z","shell.execute_reply.started":"2025-12-18T12:43:19.892975Z","shell.execute_reply":"2025-12-18T12:43:20.018101Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Apply n_gram_model","metadata":{}},{"cell_type":"code","source":"# Apply Phrases to transliterations\n\npreprocessed_transliterations_collocations = [n_gram_model[doc] for doc in preprocessed_transliterations]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.020196Z","iopub.execute_input":"2025-12-18T12:43:20.020547Z","iopub.status.idle":"2025-12-18T12:43:20.162759Z","shell.execute_reply.started":"2025-12-18T12:43:20.020516Z","shell.execute_reply":"2025-12-18T12:43:20.161769Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Add processed press releases (with collocations) to dataframe\n\ntrain['clean_transliteration'] = preprocessed_transliterations_collocations","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.163713Z","iopub.execute_input":"2025-12-18T12:43:20.163998Z","iopub.status.idle":"2025-12-18T12:43:20.171467Z","shell.execute_reply.started":"2025-12-18T12:43:20.163978Z","shell.execute_reply":"2025-12-18T12:43:20.170608Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Print one example to see how it looks like.\n\nThe fastidious hiphens remain there.","metadata":{}},{"cell_type":"code","source":"print(preprocessed_transliterations_collocations[0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.172343Z","iopub.execute_input":"2025-12-18T12:43:20.172602Z","iopub.status.idle":"2025-12-18T12:43:20.189332Z","shell.execute_reply.started":"2025-12-18T12:43:20.172584Z","shell.execute_reply":"2025-12-18T12:43:20.188449Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Find top-10 most common words\nw = Counter(preprocessed_transliterations_collocations[0]).most_common(10)\n\n# Plots top words\n\n# Barplot and font specifications\nbarplot_specs   = {\"color\": \"mediumpurple\", \"alpha\": 0.7, \"edgecolor\": \"grey\"}\ntitle_specs     = {\"fontsize\": 14, \"fontweight\": \"bold\", \"y\": 1.2}\nsuptitle_specs  = {\"fontsize\": 11}\nlabel_specs     = {\"fontsize\": 13}\n\nplt.figure(figsize=(15,3))\nplt.bar(*zip(*w), **barplot_specs)\nplt.title('Most common words in transliteration') \n#plt.suptitle(train.iloc[0,0],**suptitle_specs)\nplt.ylabel('Word counts', **label_specs)\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.190354Z","iopub.execute_input":"2025-12-18T12:43:20.190753Z","iopub.status.idle":"2025-12-18T12:43:20.392687Z","shell.execute_reply.started":"2025-12-18T12:43:20.190720Z","shell.execute_reply":"2025-12-18T12:43:20.391692Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### How that above would help translating Akkadian to English?\n\nProbably not, I don't know Akkadian and even programming too.","metadata":{}},{"cell_type":"markdown","source":"## test file\n\n\"When your submission is scored, this example test data will be replaced with the full test set. There are about 4000 sentences in the test data from about 400 unique documents. Note that while the training data has translations aligned at the document level, the test data has translations aligned at the sentence level.\"\n\n**id** - A unique identifier for each sentence.\n\n**text_id** - A unique identifier for each document.\n\n**line_start, line_end** - Denotes the boundaries of the sentence within the original tablet. Orders the sentences within the document. Note that this field has str type with values like 1, 1', or 1''. See the note on line numbers under Modern Scribal Notations in the Dataset Instructions.\n\n**transliteration** - An Akkadian transliteration of the original tablet text. Your goal is to produce the corresponding translation.\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/data","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/test.csv\")\ntest.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.393711Z","iopub.execute_input":"2025-12-18T12:43:20.394265Z","iopub.status.idle":"2025-12-18T12:43:20.408924Z","shell.execute_reply.started":"2025-12-18T12:43:20.394243Z","shell.execute_reply":"2025-12-18T12:43:20.408284Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Bibliography file\n\nBibliogr.: Recommended for scientific/indexing (ISO 4). ","metadata":{}},{"cell_type":"code","source":"bibliogr = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/bibliography.csv\")\nbibliogr.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.409750Z","iopub.execute_input":"2025-12-18T12:43:20.410047Z","iopub.status.idle":"2025-12-18T12:43:20.432374Z","shell.execute_reply.started":"2025-12-18T12:43:20.410019Z","shell.execute_reply":"2025-12-18T12:43:20.431660Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Publications csv file\n\n\"Contains the raw text of about 880 scholarly publications containing translations from Old Assyrian into multiple modern languages. The texts were produced via OCR with LLM postprocessing. You may attempt to extract these translations and align them with the transliterations in published_texts.csv. Note that the translations often are given in a language other than English.\"\n\n\n**pdf_name** - The name of the PDF file from which the text was extracted.\n\n**page** - The page number where the given text occured.\n\n**page_text** - The text of the article itself.\n\n**has_akkadian** - Whether or not the text contains Akkadian transliterations.\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/data","metadata":{}},{"cell_type":"code","source":"pub = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/publications.csv\")\npub","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:20.433131Z","iopub.execute_input":"2025-12-18T12:43:20.433355Z","iopub.status.idle":"2025-12-18T12:43:30.143974Z","shell.execute_reply.started":"2025-12-18T12:43:20.433336Z","shell.execute_reply":"2025-12-18T12:43:30.143113Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10,4))\nsns.countplot(data=pub, x='has_akkadian', order=pub['has_akkadian'].value_counts().index)\n#plt.xticks(rotation=45)\nplt.title(\"Texts that have Akkadian transliterations\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:30.144847Z","iopub.execute_input":"2025-12-18T12:43:30.145101Z","iopub.status.idle":"2025-12-18T12:43:30.305877Z","shell.execute_reply.started":"2025-12-18T12:43:30.145080Z","shell.execute_reply":"2025-12-18T12:43:30.305118Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### OA_Lexicon_eBL file\n\n\"OA_Lexicon_eBL.csv - This file contains a list of all the Old Assyrian words in transliteration with their lexical equivalents (that is, how they are found in a dictionary). The links included are to an online Akkadian dictionary hosted by **LMU (Ludwig Maximilian University of Munich**), the **electronic Babylonian Library (eBL)**.\"\n\n**type** - Type of word (e.g. word, PN = person name, GN = geographic name).\n\n**form** - String-literal word, as found in transliteration.\n\n**norm** - Normalized form, with hyphens removed and vowel length indications.\nlexeme - Lemmatized form, as found in a dictionary.\n\n**eBL** - URL of the online dictionaries in the **electronic Babylonian Library (eBL)**.\n\n**I_IV** - Roman numeral designation of the homonym lexemes, corresponding to the Concise Dictionary of Akkadian (CDA) found in the eBL.\n\n**A_D** - Alphabetic designation of the homonym lexemes, corresponding to the Chicago Assyrian Dictionary.\n\n**Female(f)** - Designation for female gender.\n\n**Alt_lex** - Alternative normalizations.\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/data","metadata":{}},{"cell_type":"code","source":"lex = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/OA_Lexicon_eBL.csv\")\nlex.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:30.306734Z","iopub.execute_input":"2025-12-18T12:43:30.307088Z","iopub.status.idle":"2025-12-18T12:43:30.438896Z","shell.execute_reply.started":"2025-12-18T12:43:30.307059Z","shell.execute_reply":"2025-12-18T12:43:30.437717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(8,4))\nsns.countplot(x='Female(f)', hue='type', data=lex, order=sorted(lex['Female(f)'].astype(str).unique()))\nplt.title('Female type of words')\n#plt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:30.439881Z","iopub.execute_input":"2025-12-18T12:43:30.440417Z","iopub.status.idle":"2025-12-18T12:43:30.628996Z","shell.execute_reply.started":"2025-12-18T12:43:30.440393Z","shell.execute_reply":"2025-12-18T12:43:30.628222Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### published_texts file","metadata":{}},{"cell_type":"code","source":"text = pd.read_csv(\"/kaggle/input/deep-past-initiative-machine-translation/published_texts.csv\")\ntext.head(2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-12-18T12:43:30.629903Z","iopub.execute_input":"2025-12-18T12:43:30.630354Z","iopub.status.idle":"2025-12-18T12:43:30.939445Z","shell.execute_reply.started":"2025-12-18T12:43:30.630326Z","shell.execute_reply":"2025-12-18T12:43:30.938615Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Draft Session: 3h:30m","metadata":{}},{"cell_type":"markdown","source":"## Ancient knowledge accessible to the public.\n\n## By the rivers of Babylon (In 1979, Disco times I thought it was just a Song)\n\n## Only in 2025, I realized It's a Psalm 137!\n\nThere we sat down\n\nWe wept\n\nWhen we remembered **Zion**\n\nWhen the wicked carried us away in captivity\n\nRequired from us a song\n\nNow, how shall we sing the Lord's song\n\nIn a strange land?\n\nLet the words of our mouth\n\nAnd the meditations of our heart\n\n**Be acceptable in Thy sight**\n\nHere tonight\n\nSong writers:  T. McNaughton / George Reyam / Frank Farian / Brent Dowe","metadata":{}},{"cell_type":"markdown","source":"### Boney M. - Rivers of Babylon (Sopot Festival 1979)\n\n### Why there is a musician dressed as an arab? \n\n### Disco times All is fair, vibrant and glamorous. Like Babylonians and Akkadian language.  \n\n\"No, ancient Babylonians were not Arabs; they were indigenous Mesopotamians speaking Akkadian (a Semitic language), while Arabs emerged later from the Arabian Peninsula.\"\n\n<iframe width=\"690\" height=\"518\" src=\"https://www.youtube.com/embed/l3QxT-w3WMo\" title=\"Boney M. - Rivers of Babylon (Sopot Festival 1979)\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen></iframe>\n\nhttps://www.youtube.com/watch?v=l3QxT-w3WMo","metadata":{}}]}