![\"example](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F18505507%2F05787bbe3e387d112d2845c6df35a3f0%2F4.png?generation=1774319930773772&alt=media\")
First of all, I’d like to thank my teammate @jackvd for providing high-quality data and clever data cleaning methods, as well as for constantly bringing fresh ideas to the table. I also want to thank @marisyn for providing numerous excellent APIs and handling the PDF extraction work! And all personnel from the host organization, DPI.
\nOur solution focuses on three key areas: data, training, and inference.
\nThe organizers provided a large number of PDFs (There is also a Larsen 2002 in here.), so our first challenge was figuring out how to effectively extract text from them.
\nAfter reviewing these PDFs, we have concluded that they can primarily be categorized into three types: image, text, and broken_text:
\nImage type: Although these have an OCR text layer, OCR artifacts become clearly visible when zooming in on the PDF. This category includes: AKT 1 1990, AKT 11a, AKT 2 1995, AKT 3 1995, AKT 4 2006, AKT 6a, AKT 6b, AKT 6c, AKT 6d, AKT 6e, and AKT_12.
Text type: Zooming in does not degrade the clarity of the PDF in any way. Furthermore, the text layer extracted using PyMuPDF has an exact one-to-one correspondence with what is visible to the naked eye. For example, using this code to handle one-to-one character error issues:
# AKT 5 2008\n_MACRON = {\n \"a\": \"ā\", \"e\": \"ē\", \"i\": \"ī\", \"u\": \"ū\",\n \"A\": \"Ā\", \"E\": \"Ē\", \"I\": \"Ī\", \"U\": \"Ū\",\n}\n_MARKERS_CLASS = r\"[\\u00a3\\u2039\\u2122]+\" # £‹™ (allow multi)\n_MARKER_VOWEL_RE = re.compile(_MARKERS_CLASS + r\"([aeiuAEIU])\")\n\ndef apply_akt5_vowel_marker_rule(text: str) -> str:\n text = _MARKER_VOWEL_RE.sub(lambda m: _MACRON[m.group(1)], text)\n text = re.sub(_MARKERS_CLASS, \"\", text)\n return text\nThis category includes: AKT 10, AKT 11b, AKT 5 2008, AKT 7b, and AKT 9a.
\n
This category includes: AKT 8 2015 and AKT 7a.
\nFor the broken_text type, we will directly treat them as the image type.
\nOnce the file type is identified, we pass text-based PDFs directly to the LLM API for processing, where it extracts the corresponding document pairs (which are later split into sentences). The method involves using Regular Expressions (Regex) to identify heading blocks, then passing both the headings and their corresponding text blocks to a LLM api for analysis and the extraction of document pairs (including error correction).
\nFor image-based PDFs, our workflow is as follows: \nwe first utilize the open-source GLM-OCR method, featuring custom parameter tuning and parallelization enhancements. Then we can get this:
\n{\n \"p-1_f-0\": \"1. kt 94/k 1263\",\n \"p-1_f-1\": \"14 1/2 GÍN KB \"KI\" I-lí-dan\",\n \"p-1_f-2\": \"14 1/2 shekels of silver is owed by Ili-dan\",\n \"p-1_f-3\": \"DUMU Šu-ki-tim 2 ma-na 10 GÍN / KU\",\n \"p-1_f-4\": \"son of Sukkutum. 2 minas 10 shekels of\",\n ...\n}\n| \n | page_num | \nfield_num | \nx | \ny | \nw | \nh | \nx_pad | \ny_pad | \nw_pad | \nh_pad | \n
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n1 | \n-1 | \n0 | \n0 | \n2075 | \n2809 | \n0 | \n0 | \n2075 | \n2809 | \n
| 1 | \n1 | \n0 | \n864 | \n253 | \n378 | \n50 | \n852 | \n241 | \n402 | \n74 | \n
| 2 | \n1 | \n1 | \n166 | \n403 | \n533 | \n48 | \n154 | \n391 | \n557 | \n72 | \n
| 3 | \n1 | \n2 | \n1167 | \n410 | \n746 | \n52 | \n1155 | \n398 | \n770 | \n76 | \n
| 4 | \n1 | \n3 | \n162 | \n470 | \n773 | \n49 | \n150 | \n458 | \n797 | \n73 | \n
| … | \n\n | \n | \n | \n | \n | \n | \n | \n | \n | \n |
Next, we use Regex to match the headings again. The LLM API then processes these headings and images to generate text. Finally, we run a second pass with the image and text together for fine-grained correction, ensuring a high degree of accuracy.
\nSome of these PDFs are in Turkish. We're using a combination of LLM APIs with dictionaries & Turkish translation to translate them into English for training.
\nRegarding train.csv, I have to mention the OARE website. For each sample, we visit https://oare.byu.edu/epigraphies/{oare_id}.
We discovered that clicking on a specific word on this page displays its definition. Consequently, we scraped the site to obtain the meanings of these words. While not every word has a corresponding definition, the vast majority do, which is more than sufficient for our needs.
\nAs a result, we obtained a dictionary covering most of the words in train.csv.
\nHowever, it is important to note that almost all documents in train.csv originate from AKT 5 2008, AKT 6a, AKT 6b, AKT 6c, AKT 6d, AKT 6e, and AKT 8 2015. To improve accuracy, we performed similarity matching between the entries in train.csv and the text extracted from these PDF sources. This allowed us to retrieve the \"ground truth\" or correct form of the documents, as the data quality within the original PDFs is significantly higher. (This involved a lot of manual fixes.)
\nFollowing these procedures, we used that dictionary to give the LLM a general understanding of what transliteration means, then followed the English sentence breaks to eventually obtain sentence-level parallel corpora. (The alignment in this step might not be perfect, so we have implemented corresponding AUG to address this.)
\nSimilar to train.csv, almost all samples can be found on the OARE website. However, since translations were unavailable, we used an LLM API to create pseudo-labels for these samples. The process is roughly as follows: we prompt the LLM to translate the original text using the dictionary we retrieved from the OARE site.
\nexample (The prompt also includes several constraints, such as JSON formatting, mana 60 conversion, …):
\nkey_prompt = \"\"\"\nKIŠIB a-šùr-ma-lik DUMU i-na-a a-na a-šùr-na-da DUMU <gap> ù en-um-a-šùr DUMU <gap> a-pu-tum a-na a-wa-at ṭup-pì-im iḫ-da <gap> ša ú-<gap> ḫa-sí-sú <gap>\n\n{KIŠIB|form:Noun|1. seal, cylinder seal 2. seal impression produced by a cylinder seal 3. sealed clay tablet (legal or administrative document, also letter)}\n{a-šùr-ma-lik|form:Noun|1. Aššur is counselor}\n{DUMU|form:Noun|1. son}\n{i-na-a|form:Noun|nan}\n{a-na|form:Preposition|1. to, for, up to, toward, against, upon}\n{a-šùr-na-da|form:Noun|1. Praise Aššur!}\n{ù|form:Conjunction|1. and}\n{en-um-a-šùr|form:Noun|1. Mercy O Aššur}\n{a-pu-tum|form:Interjection|1. urgent!, please!}\n{a-wa-at|form:Noun|1. word, matter, lawsuit 2. pl. awâtum - stipulation, rule}\n{ṭup-pì-im|form:Noun|1. (inscribed) tablet (of clay, rarely of other materials) 2. board, flat surface}\n{iḫ-da|form:Finite verb|1. to attend, to watch, to pay attention, to do something carefully 2. to draw someone's attention to something, to alert, to ask someone to pay attention, to take care}\n{ša|form:Pronoun|1. relative pronoun}\n\"\"\"\nFor this file, we provide a fixed number of pages to the LLM API to extract sentences and other elements. While this yields a significant amount of data, it is difficult to achieve document-level alignment.
\nWith that, we have obtained a parallel corpus dataset; the remaining task is now focused on data cleaning.
\nActually, the organizers updated the data several times. The train.csv I used in Step 1 to match against the PDF data was the second version, which was much better normalized.
For preprocessing, here’s the main stuff (mainly for pdf-data or LLM results, The second version of train.csv is already quite clean.):
\nGeneral Foundation Processing (Applied to Both Sides)
\nTransliteration Text Side\nThe primary objective of the source text side is to eliminate typographical variations within the sign system and to standardize determinatives for divine names, toponyms, and similar entities.
\n<gap>, and strictly clean up extraneous spaces and punctuation before and after the <gap> marker.transliteration_map, and correct specific spellings of personal names and anomalous annotations.Translation Side\nThe primary objective of the translation side is to enhance the natural linguistic fluency of the text, eliminating alternative options and academic jargon left by translators.
\ntranslation_map, expand complex fractional weights unique to antiquity (e.g., 12 5/12 shekels) into more readable text (e.g., 12 ⅓ shekels 15 grains).<gap> to ensure strict alignment with the source text side in subsequent machine learning or text comparison processes.Actually, this part skips over a considerable amount of manual effort—things like manually checking for anomalies, using regex for matching, and then incorporating those into the cleaning rules, this is our final clean func:
\n\nThis technique aims to address two primary issues:
\nSpecifically, we implemented the following mechanisms:
\nrandom_ctx_prob = 0.20) of bypassing contiguous context and instead randomly sampling and concatenating sentences from the same article. This compels the model to learn not only the local context but also the global topic, preventing the rote memorization of absolute sentence orders.def _build_k_cdf(self):\n \"\"\"\n Build a CDF over K in [1, k_max].\n - zipf: P(K) ∝ 1/K (heavy-tail, good length coverage)\n - geom_mix: (1-α)·Geom(p) + α·Zipf(1/K)\n \"\"\"\n keys = list(range(1, self.k_max + 1))\n\n if self.k_sampling == \"geom_mix\":\n # geometric over {1..k_max}: p*(1-p)^(k-1)\n geom, zipf = [self.geom_p * ((1.0 - self.geom_p) ** (k - 1)) for k in keys], [1.0 / k for k in keys]\n probs = [(1.0 - self.long_tail_mix) * g + self.long_tail_mix * z for g, z in zip(geom, zipf)]\n else:\n # default: Zipf / 1-K\n probs = [1.0 / k for k in keys]\n\n s = sum(probs); probs = [p / s for p in probs]\n cdf = list(accumulate(probs))\n cdf[-1] = 1.0\n return keys, cdf\n\ndef _sample_K(self):\n r = random.random()\n i = bisect_left(self._k_cdf, r)\n if i >= len(self._k_keys): i = len(self._k_keys) - 1\n return int(self._k_keys[i])\n\n@staticmethod\ndef _sample_LR(K: int):\n # L ~ Binomial(K-1, 0.5), R = K-1-L\n if K <= 1: return 0, 0\n L = int(np.random.binomial(K - 1, 0.5))\n R = int((K - 1) - L)\n return L, R\nFor a given input x, it is passed through the network twice (due to Dropout, the two forward propagation results, y_1 and y_2, will exhibit minor variances). In addition to calculating the standard cross-entropy loss, the symmetric Kullback-Leibler (KL) divergence between these two output distributions (logits) is computed. This enforces output consistency in the presence of internal noise (Dropout), significantly mitigating overfitting.
\nThis involves identifying the \"worst-case perturbations\" within the continuous embedding space and injecting them into the input, ensuring that the model can still produce accurate predictions even under slight adversarial perturbations.
\nExponential Moving Average, a commonly adopted technique in deep learning competitions to improve model generalization and stability.
\nIncorporating these complex components necessitates engineering optimizations, which include:
\nFinal parameters:
\nk_max=10\nk_sampling=\"zipf\"\ngeom_p=0.35\nrandom_ctx_prob=0.20\nRDROP_ALPHA = 0.8\nPGD_ENABLE = True\nPGD_STEPS = 3\nPGD_EPS = 0.5\nPGD_ALPHA = 0.2\nPGD_EMB_KEYWORDS = (\"shared\", \"embed_tokens\", \"embeddings\")\nThis is our primary experimental pipeline (the optimal path for single-mode optimization).\nBy using different subsets of data, we trained a variety of models. Generally, it works like this:
\n| Model | \nMethod | \nLB | \nPB | \n
|---|---|---|---|
| byt5-large | \nv1 No external data (only aligned and cleaned train.csv) | \n34.0 | \n34.3 | \n
| byt5-large | \nv2 Added Turkish-to-English data | \n36.9 | \n37.0 | \n
| byt5-large | \nv3 Added high quality publications data | \n38.7 | \n38.9 | \n
| byt5-large | \nv4 Pre-trained for 5 epochs using all publications data before formal training (filtered by quality) | \n38.6 | \n39.5 | \n
| byt5-large | \nv5 Pre-trained for 5 epochs using all publications data and published_texts pseudo-labels before formal training (filtered by quality) | \n38.9 | \n39.3 | \n
| byt5-base | \nv6 Formally trained using publications data, published_texts pseudo-labels, alongside main data | \n39.3 | \n39.9 | \n
Initially, we thought the extra data was too noisy and left it out of the formal training phase, but it actually performed better once we included it—probably just because of the large volume.
\nOur final submission consists of 15 models, primarily detailed as follows:
\n| Model Description | \nLB | \nPB | \n
|---|---|---|
| Large model trained on data subset v3 | \n38.7 | \n38.9 | \n
| Large model trained on data subset v3 with name augmentation | \n37.8 | \n38.8 | \n
| XL model trained on data subset v3 | \n37.9 | \n38.3 | \n
| Large model trained on data subset v4 | \n38.6 | \n39.5 | \n
| Large model trained on data subset v5 | \n38.9 | \n39.3 | \n
| Base model trained on data subset v5 | \n38.8 | \n39.0 | \n
| Base model trained on data subset v6 | \n39.2 | \n39.5 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v1) | \n39.3 | \n39.9 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v2) | \n38.3 | \n39.2 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v3) | \n38.2 | \n39.5 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v4) | \n38.4 | \n39.6 | \n
We performed 5-fold cross-validation training based on our best-performing model on the Public Leaderboard (LB score 39.3: Base model trained on data subset v6 with additional filtering and cleaning (v1)). Although these models were not submitted individually, they were also incorporated into the MBR (Minimum Bayes Risk) ensemble.
\nTo accelerate inference, we restricted the output to four results per model using a beam size of 4 (beam=4), without applying any sampling methods (our experiments indicated that utilizing sampling within the MBR process leads to a decrease in score).
The models were grouped and deployed across two T4 GPUs for execution.
\n![\"15models\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F18505507%2F2d32d5e4df1b2bbafc1e969535c8f85f%2Fall_models.png?generation=1774331390404829&alt=media\")
Final Score: 40.7 | Execution Time: 208m 50s
\ntranslate Akkadian to English. Translate TARGET only. Use CONTEXT only as reference.\nTARGET:\nki-ma mup-pì-ni ta-áa-me-a-ni a-ma-kam lu a-na aí-mì-im a-na É.GAL-lim i-dí-in lu té-ra-at É.GAL-lim ú-kà-lim lu na-aí-ma a-dí-ni lá i-dí-in ma-lá KÙ.AN na-áa-ú ni-bi„-it a-aí-im au-um-au ú au-mì a-bi„-au i-na mup-pì-im lu-up-ta-nim-ma ia-tí aí-ip-ri-ni aé-bi„-lá-nim\n\nCONTEXT BEFORE:\num-ma kà-ru-um kà-ni-ia-ma a-na aa-qí-il… da-tim aí-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi„-ma mup-pu-um aa a-lim(ki) i-li-kam i-na mup-pì-im aa a-lim(ki) ia-tù u„-mì-im a-nim ma-ma-an KÙ.AN i-aa-ú-mu-ni i-na né-mì-lim da-aùr ú-lá e-WA ia-ra-tí-au kà-ru-um kà-ni-ia i-lá-qé\n\nCONTEXT AFTER:\nme-+e-er mup-pì-ni a-na kà-ar kà-ar-ma ú wa-bar-ra-tim aé-bi„-lá KÙ.AN lu a-na DAM.GÀR-ru-tim i-dí-in au-mì a-wi-lim lu-up-ta-nim\nData augmentation on names and dictionaries in the source text during training\nRandomly altering the capitalization of names, and using a dictionary to randomly select a norm form for many-to-one norm -> lexeme mappings. \nThe single-model score didn't improve much, but we still included these in our ensemble.
Weight fusion / averaging\nThis showed almost no improvement and even resulted in lower scores than the original single model. After a few attempts, we stopped trying this and switched to using MBR (Minimum Bayes Risk) for ensembling instead.
Setting a very long maxlen and training at the document level\nUnsurprisingly, the results were poor because the test set is evaluated at the sentence level.
Using Large Language Models (Qwen, LLaMA, etc.)\nThis was likely due to the tokenizer or the fact that my data wasn't clean enough in the early stages of the competition. Without applying any special tricks, the performance of a trained 14B model was only average. However, I was very glad to see top-ranking participants successfully taking on the challenge using LLMs—an excellent solution!
Bidirectional training (adding akk->eng)\nThere wasn't much progress in terms of single-model improvement. We didn't have extra submission quotas to test this before ensembling, but perhaps including it in the ensemble could have provided a slight boost to the final results.
First of all, I’d like to thank my teammate @jackvd for providing high-quality data and clever data cleaning methods, as well as for constantly bringing fresh ideas to the table. I also want to thank @marisyn for providing numerous excellent APIs and handling the PDF extraction work! And all personnel from the host organization, DPI.
\nOur solution focuses on three key areas: data, training, and inference.
\nThe organizers provided a large number of PDFs (There is also a Larsen 2002 in here.), so our first challenge was figuring out how to effectively extract text from them.
\nAfter reviewing these PDFs, we have concluded that they can primarily be categorized into three types: image, text, and broken_text:
\nImage type: Although these have an OCR text layer, OCR artifacts become clearly visible when zooming in on the PDF. This category includes: AKT 1 1990, AKT 11a, AKT 2 1995, AKT 3 1995, AKT 4 2006, AKT 6a, AKT 6b, AKT 6c, AKT 6d, AKT 6e, and AKT_12.
Text type: Zooming in does not degrade the clarity of the PDF in any way. Furthermore, the text layer extracted using PyMuPDF has an exact one-to-one correspondence with what is visible to the naked eye. For example, using this code to handle one-to-one character error issues:
# AKT 5 2008\n_MACRON = {\n \"a\": \"ā\", \"e\": \"ē\", \"i\": \"ī\", \"u\": \"ū\",\n \"A\": \"Ā\", \"E\": \"Ē\", \"I\": \"Ī\", \"U\": \"Ū\",\n}\n_MARKERS_CLASS = r\"[\\u00a3\\u2039\\u2122]+\" # £‹™ (allow multi)\n_MARKER_VOWEL_RE = re.compile(_MARKERS_CLASS + r\"([aeiuAEIU])\")\n\ndef apply_akt5_vowel_marker_rule(text: str) -> str:\n text = _MARKER_VOWEL_RE.sub(lambda m: _MACRON[m.group(1)], text)\n text = re.sub(_MARKERS_CLASS, \"\", text)\n return text\nThis category includes: AKT 10, AKT 11b, AKT 5 2008, AKT 7b, and AKT 9a.
\n
This category includes: AKT 8 2015 and AKT 7a.
\nFor the broken_text type, we will directly treat them as the image type.
\nOnce the file type is identified, we pass text-based PDFs directly to the LLM API for processing, where it extracts the corresponding document pairs (which are later split into sentences). The method involves using Regular Expressions (Regex) to identify heading blocks, then passing both the headings and their corresponding text blocks to a LLM api for analysis and the extraction of document pairs (including error correction).
\nFor image-based PDFs, our workflow is as follows: \nwe first utilize the open-source GLM-OCR method, featuring custom parameter tuning and parallelization enhancements. Then we can get this:
\n{\n \"p-1_f-0\": \"1. kt 94/k 1263\",\n \"p-1_f-1\": \"14 1/2 GÍN KB \"KI\" I-lí-dan\",\n \"p-1_f-2\": \"14 1/2 shekels of silver is owed by Ili-dan\",\n \"p-1_f-3\": \"DUMU Šu-ki-tim 2 ma-na 10 GÍN / KU\",\n \"p-1_f-4\": \"son of Sukkutum. 2 minas 10 shekels of\",\n ...\n}\n| \n | page_num | \nfield_num | \nx | \ny | \nw | \nh | \nx_pad | \ny_pad | \nw_pad | \nh_pad | \n
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n1 | \n-1 | \n0 | \n0 | \n2075 | \n2809 | \n0 | \n0 | \n2075 | \n2809 | \n
| 1 | \n1 | \n0 | \n864 | \n253 | \n378 | \n50 | \n852 | \n241 | \n402 | \n74 | \n
| 2 | \n1 | \n1 | \n166 | \n403 | \n533 | \n48 | \n154 | \n391 | \n557 | \n72 | \n
| 3 | \n1 | \n2 | \n1167 | \n410 | \n746 | \n52 | \n1155 | \n398 | \n770 | \n76 | \n
| 4 | \n1 | \n3 | \n162 | \n470 | \n773 | \n49 | \n150 | \n458 | \n797 | \n73 | \n
| … | \n\n | \n | \n | \n | \n | \n | \n | \n | \n | \n |
Next, we use Regex to match the headings again. The LLM API then processes these headings and images to generate text. Finally, we run a second pass with the image and text together for fine-grained correction, ensuring a high degree of accuracy.
\nSome of these PDFs are in Turkish. We're using a combination of LLM APIs with dictionaries & Turkish translation to translate them into English for training.
\nRegarding train.csv, I have to mention the OARE website. For each sample, we visit https://oare.byu.edu/epigraphies/{oare_id}.
We discovered that clicking on a specific word on this page displays its definition. Consequently, we scraped the site to obtain the meanings of these words. While not every word has a corresponding definition, the vast majority do, which is more than sufficient for our needs.
\nAs a result, we obtained a dictionary covering most of the words in train.csv.
\nHowever, it is important to note that almost all documents in train.csv originate from AKT 5 2008, AKT 6a, AKT 6b, AKT 6c, AKT 6d, AKT 6e, and AKT 8 2015. To improve accuracy, we performed similarity matching between the entries in train.csv and the text extracted from these PDF sources. This allowed us to retrieve the \"ground truth\" or correct form of the documents, as the data quality within the original PDFs is significantly higher. (This involved a lot of manual fixes.)
\nFollowing these procedures, we used that dictionary to give the LLM a general understanding of what transliteration means, then followed the English sentence breaks to eventually obtain sentence-level parallel corpora. (The alignment in this step might not be perfect, so we have implemented corresponding AUG to address this.)
\nSimilar to train.csv, almost all samples can be found on the OARE website. However, since translations were unavailable, we used an LLM API to create pseudo-labels for these samples. The process is roughly as follows: we prompt the LLM to translate the original text using the dictionary we retrieved from the OARE site.
\nexample (The prompt also includes several constraints, such as JSON formatting, mana 60 conversion, …):
\nkey_prompt = \"\"\"\nKIŠIB a-šùr-ma-lik DUMU i-na-a a-na a-šùr-na-da DUMU <gap> ù en-um-a-šùr DUMU <gap> a-pu-tum a-na a-wa-at ṭup-pì-im iḫ-da <gap> ša ú-<gap> ḫa-sí-sú <gap>\n\n{KIŠIB|form:Noun|1. seal, cylinder seal 2. seal impression produced by a cylinder seal 3. sealed clay tablet (legal or administrative document, also letter)}\n{a-šùr-ma-lik|form:Noun|1. Aššur is counselor}\n{DUMU|form:Noun|1. son}\n{i-na-a|form:Noun|nan}\n{a-na|form:Preposition|1. to, for, up to, toward, against, upon}\n{a-šùr-na-da|form:Noun|1. Praise Aššur!}\n{ù|form:Conjunction|1. and}\n{en-um-a-šùr|form:Noun|1. Mercy O Aššur}\n{a-pu-tum|form:Interjection|1. urgent!, please!}\n{a-wa-at|form:Noun|1. word, matter, lawsuit 2. pl. awâtum - stipulation, rule}\n{ṭup-pì-im|form:Noun|1. (inscribed) tablet (of clay, rarely of other materials) 2. board, flat surface}\n{iḫ-da|form:Finite verb|1. to attend, to watch, to pay attention, to do something carefully 2. to draw someone's attention to something, to alert, to ask someone to pay attention, to take care}\n{ša|form:Pronoun|1. relative pronoun}\n\"\"\"\nFor this file, we provide a fixed number of pages to the LLM API to extract sentences and other elements. While this yields a significant amount of data, it is difficult to achieve document-level alignment.
\nWith that, we have obtained a parallel corpus dataset; the remaining task is now focused on data cleaning.
\nActually, the organizers updated the data several times. The train.csv I used in Step 1 to match against the PDF data was the second version, which was much better normalized.
For preprocessing, here’s the main stuff (mainly for pdf-data or LLM results, The second version of train.csv is already quite clean.):
\nGeneral Foundation Processing (Applied to Both Sides)
\nTransliteration Text Side\nThe primary objective of the source text side is to eliminate typographical variations within the sign system and to standardize determinatives for divine names, toponyms, and similar entities.
\n<gap>, and strictly clean up extraneous spaces and punctuation before and after the <gap> marker.transliteration_map, and correct specific spellings of personal names and anomalous annotations.Translation Side\nThe primary objective of the translation side is to enhance the natural linguistic fluency of the text, eliminating alternative options and academic jargon left by translators.
\ntranslation_map, expand complex fractional weights unique to antiquity (e.g., 12 5/12 shekels) into more readable text (e.g., 12 ⅓ shekels 15 grains).<gap> to ensure strict alignment with the source text side in subsequent machine learning or text comparison processes.Actually, this part skips over a considerable amount of manual effort—things like manually checking for anomalies, using regex for matching, and then incorporating those into the cleaning rules, this is our final clean func:
\n\nThis technique aims to address two primary issues:
\nSpecifically, we implemented the following mechanisms:
\nrandom_ctx_prob = 0.20) of bypassing contiguous context and instead randomly sampling and concatenating sentences from the same article. This compels the model to learn not only the local context but also the global topic, preventing the rote memorization of absolute sentence orders.def _build_k_cdf(self):\n \"\"\"\n Build a CDF over K in [1, k_max].\n - zipf: P(K) ∝ 1/K (heavy-tail, good length coverage)\n - geom_mix: (1-α)·Geom(p) + α·Zipf(1/K)\n \"\"\"\n keys = list(range(1, self.k_max + 1))\n\n if self.k_sampling == \"geom_mix\":\n # geometric over {1..k_max}: p*(1-p)^(k-1)\n geom, zipf = [self.geom_p * ((1.0 - self.geom_p) ** (k - 1)) for k in keys], [1.0 / k for k in keys]\n probs = [(1.0 - self.long_tail_mix) * g + self.long_tail_mix * z for g, z in zip(geom, zipf)]\n else:\n # default: Zipf / 1-K\n probs = [1.0 / k for k in keys]\n\n s = sum(probs); probs = [p / s for p in probs]\n cdf = list(accumulate(probs))\n cdf[-1] = 1.0\n return keys, cdf\n\ndef _sample_K(self):\n r = random.random()\n i = bisect_left(self._k_cdf, r)\n if i >= len(self._k_keys): i = len(self._k_keys) - 1\n return int(self._k_keys[i])\n\n@staticmethod\ndef _sample_LR(K: int):\n # L ~ Binomial(K-1, 0.5), R = K-1-L\n if K <= 1: return 0, 0\n L = int(np.random.binomial(K - 1, 0.5))\n R = int((K - 1) - L)\n return L, R\nFor a given input x, it is passed through the network twice (due to Dropout, the two forward propagation results, y_1 and y_2, will exhibit minor variances). In addition to calculating the standard cross-entropy loss, the symmetric Kullback-Leibler (KL) divergence between these two output distributions (logits) is computed. This enforces output consistency in the presence of internal noise (Dropout), significantly mitigating overfitting.
\nThis involves identifying the \"worst-case perturbations\" within the continuous embedding space and injecting them into the input, ensuring that the model can still produce accurate predictions even under slight adversarial perturbations.
\nExponential Moving Average, a commonly adopted technique in deep learning competitions to improve model generalization and stability.
\nIncorporating these complex components necessitates engineering optimizations, which include:
\nFinal parameters:
\nk_max=10\nk_sampling=\"zipf\"\ngeom_p=0.35\nrandom_ctx_prob=0.20\nRDROP_ALPHA = 0.8\nPGD_ENABLE = True\nPGD_STEPS = 3\nPGD_EPS = 0.5\nPGD_ALPHA = 0.2\nPGD_EMB_KEYWORDS = (\"shared\", \"embed_tokens\", \"embeddings\")\nThis is our primary experimental pipeline (the optimal path for single-mode optimization).\nBy using different subsets of data, we trained a variety of models. Generally, it works like this:
\n| Model | \nMethod | \nLB | \nPB | \n
|---|---|---|---|
| byt5-large | \nv1 No external data (only aligned and cleaned train.csv) | \n34.0 | \n34.3 | \n
| byt5-large | \nv2 Added Turkish-to-English data | \n36.9 | \n37.0 | \n
| byt5-large | \nv3 Added high quality publications data | \n38.7 | \n38.9 | \n
| byt5-large | \nv4 Pre-trained for 5 epochs using all publications data before formal training (filtered by quality) | \n38.6 | \n39.5 | \n
| byt5-large | \nv5 Pre-trained for 5 epochs using all publications data and published_texts pseudo-labels before formal training (filtered by quality) | \n38.9 | \n39.3 | \n
| byt5-base | \nv6 Formally trained using publications data, published_texts pseudo-labels, alongside main data | \n39.3 | \n39.9 | \n
Initially, we thought the extra data was too noisy and left it out of the formal training phase, but it actually performed better once we included it—probably just because of the large volume.
\nOur final submission consists of 15 models, primarily detailed as follows:
\n| Model Description | \nLB | \nPB | \n
|---|---|---|
| Large model trained on data subset v3 | \n38.7 | \n38.9 | \n
| Large model trained on data subset v3 with name augmentation | \n37.8 | \n38.8 | \n
| XL model trained on data subset v3 | \n37.9 | \n38.3 | \n
| Large model trained on data subset v4 | \n38.6 | \n39.5 | \n
| Large model trained on data subset v5 | \n38.9 | \n39.3 | \n
| Base model trained on data subset v5 | \n38.8 | \n39.0 | \n
| Base model trained on data subset v6 | \n39.2 | \n39.5 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v1) | \n39.3 | \n39.9 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v2) | \n38.3 | \n39.2 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v3) | \n38.2 | \n39.5 | \n
| Base model trained on data subset v6 with additional filtering and cleaning (v4) | \n38.4 | \n39.6 | \n
We performed 5-fold cross-validation training based on our best-performing model on the Public Leaderboard (LB score 39.3: Base model trained on data subset v6 with additional filtering and cleaning (v1)). Although these models were not submitted individually, they were also incorporated into the MBR (Minimum Bayes Risk) ensemble.
\nTo accelerate inference, we restricted the output to four results per model using a beam size of 4 (beam=4), without applying any sampling methods (our experiments indicated that utilizing sampling within the MBR process leads to a decrease in score).
The models were grouped and deployed across two T4 GPUs for execution.
\n
Final Score: 40.7 | Execution Time: 208m 50s
\ntranslate Akkadian to English. Translate TARGET only. Use CONTEXT only as reference.\nTARGET:\nki-ma mup-pì-ni ta-áa-me-a-ni a-ma-kam lu a-na aí-mì-im a-na É.GAL-lim i-dí-in lu té-ra-at É.GAL-lim ú-kà-lim lu na-aí-ma a-dí-ni lá i-dí-in ma-lá KÙ.AN na-áa-ú ni-bi„-it a-aí-im au-um-au ú au-mì a-bi„-au i-na mup-pì-im lu-up-ta-nim-ma ia-tí aí-ip-ri-ni aé-bi„-lá-nim\n\nCONTEXT BEFORE:\num-ma kà-ru-um kà-ni-ia-ma a-na aa-qí-il… da-tim aí-ip-ri-ni kà-ar kà-ar-ma ú wa-bar-ra-tim qí-bi„-ma mup-pu-um aa a-lim(ki) i-li-kam i-na mup-pì-im aa a-lim(ki) ia-tù u„-mì-im a-nim ma-ma-an KÙ.AN i-aa-ú-mu-ni i-na né-mì-lim da-aùr ú-lá e-WA ia-ra-tí-au kà-ru-um kà-ni-ia i-lá-qé\n\nCONTEXT AFTER:\nme-+e-er mup-pì-ni a-na kà-ar kà-ar-ma ú wa-bar-ra-tim aé-bi„-lá KÙ.AN lu a-na DAM.GÀR-ru-tim i-dí-in au-mì a-wi-lim lu-up-ta-nim\nData augmentation on names and dictionaries in the source text during training\nRandomly altering the capitalization of names, and using a dictionary to randomly select a norm form for many-to-one norm -> lexeme mappings. \nThe single-model score didn't improve much, but we still included these in our ensemble.
Weight fusion / averaging\nThis showed almost no improvement and even resulted in lower scores than the original single model. After a few attempts, we stopped trying this and switched to using MBR (Minimum Bayes Risk) for ensembling instead.
Setting a very long maxlen and training at the document level\nUnsurprisingly, the results were poor because the test set is evaluated at the sentence level.
Using Large Language Models (Qwen, LLaMA, etc.)\nThis was likely due to the tokenizer or the fact that my data wasn't clean enough in the early stages of the competition. Without applying any special tricks, the performance of a trained 14B model was only average. However, I was very glad to see top-ranking participants successfully taking on the challenge using LLMs—an excellent solution!
Bidirectional training (adding akk->eng)\nThere wasn't much progress in terms of single-model improvement. We didn't have extra submission quotas to test this before ensembling, but perhaps including it in the ensemble could have provided a slight boost to the final results.
Congratulations! The thoroughness and patience you demonstrated in data processing and augmentation are truly amazing ! I’ve learned so much from it.
", "rawMarkdown": "Congratulations! The thoroughness and patience you demonstrated in data processing and augmentation are truly amazing ! I’ve learned so much from it.", "votes": null }, { "id": "3427513", "postDate": "03/24/2026 07:20:42", "content": "True, this is actually just the tip of the iceberg. The countless hours of manual work, trial and error, and constant tweaking my teammates and I went through didn't even make it into this writeup.
", "rawMarkdown": "True, this is actually just the tip of the iceberg. The countless hours of manual work, trial and error, and constant tweaking my teammates and I went through didn't even make it into this writeup.", "votes": null }, { "id": "3427514", "postDate": "03/24/2026 07:21:23", "content": "Congratulations! Excellent solution sharing.
", "rawMarkdown": "Congratulations! Excellent solution sharing.", "votes": null }, { "id": "3427520", "postDate": "03/24/2026 07:25:30", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "3427532", "postDate": "03/24/2026 07:49:01", "content": "I'm glad you found the OCR solution useful!
\nKudos on the great job of testing so many options and getting a 6th place gold!
", "rawMarkdown": "I'm glad you found the OCR solution useful!\n\nKudos on the great job of testing so many options and getting a 6th place gold!", "votes": null }, { "id": "3427533", "postDate": "03/24/2026 07:50:01", "content": "Your code has been a huge help to us! We’re using LLMs to refine the details after the initial extraction.
", "rawMarkdown": "Your code has been a huge help to us! We’re using LLMs to refine the details after the initial extraction.", "votes": null }, { "id": "3427595", "postDate": "03/24/2026 09:23:39", "content": "Congratulations to your team on winning the gold medal. The amount of engineering work is truly astonishing, and congratulations as well on your promotion to Master!
", "rawMarkdown": "Congratulations to your team on winning the gold medal. The amount of engineering work is truly astonishing, and congratulations as well on your promotion to Master!", "votes": null }, { "id": "3427597", "postDate": "03/24/2026 09:25:35", "content": "Thank you! I hope you become a master soon, too!
", "rawMarkdown": "Thank you! I hope you become a master soon, too!", "votes": null }, { "id": "3427675", "postDate": "03/24/2026 11:47:32", "content": "Brilliant work！What a fucking feat for 3G！
", "rawMarkdown": "Brilliant work！What a fucking feat for 3G！", "votes": null }, { "id": "3427988", "postDate": "03/24/2026 18:52:27", "content": "Congrats @wscmisx !!
", "rawMarkdown": "Congrats @wscmisx !!", "votes": null }, { "id": "3428078", "postDate": "03/24/2026 23:25:21", "content": "Thank you so much bro🥰
", "rawMarkdown": "Thank you so much bro🥰", "votes": null }, { "id": "3430291", "postDate": "03/28/2026 07:31:02", "content": "Wow, Thats a genius thinking. Well deserved brother. Congratulations 🎊
", "rawMarkdown": "Wow, Thats a genius thinking. Well deserved brother. Congratulations 🎊", "votes": null } ], "comments": [ { "id": 3427499, "author_name": "brucewu1200", "author_url": "", "post_date": "03/24/2026 06:51:14", "content": "Congratulations! The thoroughness and patience you demonstrated in data processing and augmentation are truly amazing ! I’ve learned so much from it.
", "votes": null, "replies": [ { "id": 3427513, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 07:20:42", "content": "True, this is actually just the tip of the iceberg. The countless hours of manual work, trial and error, and constant tweaking my teammates and I went through didn't even make it into this writeup.
", "votes": null, "replies": [] } ] }, { "id": 3427514, "author_name": "yaokaixinya123", "author_url": "", "post_date": "03/24/2026 07:21:23", "content": "Congratulations! Excellent solution sharing.
", "votes": null, "replies": [ { "id": 3427520, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 07:25:30", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 3427532, "author_name": "angantyr", "author_url": "", "post_date": "03/24/2026 07:49:01", "content": "I'm glad you found the OCR solution useful!
\nKudos on the great job of testing so many options and getting a 6th place gold!
", "votes": null, "replies": [ { "id": 3427533, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 07:50:01", "content": "Your code has been a huge help to us! We’re using LLMs to refine the details after the initial extraction.
", "votes": null, "replies": [] } ] }, { "id": 3427595, "author_name": "gunianxue", "author_url": "", "post_date": "03/24/2026 09:23:39", "content": "Congratulations to your team on winning the gold medal. The amount of engineering work is truly astonishing, and congratulations as well on your promotion to Master!
", "votes": null, "replies": [ { "id": 3427597, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 09:25:35", "content": "Thank you! I hope you become a master soon, too!
", "votes": null, "replies": [] } ] }, { "id": 3427675, "author_name": "yxw666", "author_url": "", "post_date": "03/24/2026 11:47:32", "content": "Brilliant work！What a fucking feat for 3G！
", "votes": null, "replies": [] }, { "id": 3427988, "author_name": "alejopaullier", "author_url": "", "post_date": "03/24/2026 18:52:27", "content": "Congrats @wscmisx !!
", "votes": null, "replies": [ { "id": 3428078, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 23:25:21", "content": "Thank you so much bro🥰
", "votes": null, "replies": [] } ] }, { "id": 3430291, "author_name": "koushikrudra", "author_url": "", "post_date": "03/28/2026 07:31:02", "content": "Wow, Thats a genius thinking. Well deserved brother. Congratulations 🎊
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3427476": "First of all, I’d like to thank my teammate [@jackvd](https://www.kaggle.com/jackvd) for providing high-quality data and clever data cleaning methods, as well as for constantly bringing fresh ideas to the table. I also want to thank [@marisyn](https://www.kaggle.com/marisyn) for providing numerous excellent APIs and handling the PDF extraction work! And all personnel from the host organization, DPI.\n\nOur solution focuses on three key areas: data, training, and inference.\n\n# 1. Data Extract\n\n## PDF Data\n\nThe organizers provided a large number of [PDFs](https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf) (There is also a *Larsen 2002* in [here](https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources).), so our first challenge was figuring out how to effectively extract text from them.\n\nAfter reviewing these PDFs, we have concluded that they can primarily be categorized into three types: image, text, and broken_text:\n\n- **Image type**: Although these have an OCR text layer, OCR artifacts become clearly visible when zooming in on the PDF. This category includes: AKT 1 1990, AKT 11a, AKT 2 1995, AKT 3 1995, AKT 4 2006, AKT 6a, AKT 6b, AKT 6c, AKT 6d, AKT 6e, and AKT_12.\n\n- **Text type**: Zooming in does not degrade the clarity of the PDF in any way. Furthermore, the text layer extracted using PyMuPDF has an exact one-to-one correspondence with what is visible to the naked eye. For example, using this code to handle one-to-one character error issues:\n```python\n# AKT 5 2008\n_MACRON = {\n \"a\": \"ā\", \"e\": \"ē\", \"i\": \"ī\", \"u\": \"ū\",\n \"A\": \"Ā\", \"E\": \"Ē\", \"I\": \"Ī\", \"U\": \"Ū\",\n}\n_MARKERS_CLASS = r\"[\\u00a3\\u2039\\u2122]+\" # £‹™ (allow multi)\n_MARKER_VOWEL_RE = re.compile(_MARKERS_CLASS + r\"([aeiuAEIU])\")\n\ndef apply_akt5_vowel_marker_rule(text: str) -> str:\n text = _MARKER_VOWEL_RE.sub(lambda m: _MACRON[m.group(1)], text)\n text = re.sub(_MARKERS_CLASS, \"\", text)\n return text\n```\nThis category includes: AKT 10, AKT 11b, AKT 5 2008, AKT 7b, and AKT 9a.\n\n- **Broken_text type**: Similar to the text type, zooming in does not affect clarity. However, the text layer extracted using PyMuPDF exhibits a one-to-many or many-to-many correspondence with what is visually observed. \n\n\n\n\nThis category includes: AKT 8 2015 and AKT 7a.\n\nFor the broken_text type, we will directly treat them as the image type.\n\nOnce the file type is identified, we pass text-based PDFs directly to the LLM API for processing, where it extracts the corresponding document pairs (which are later split into sentences). The method involves using Regular Expressions (Regex) to identify heading blocks, then passing both the headings and their corresponding text blocks to a LLM api for analysis and the extraction of document pairs (including error correction).\n\nFor image-based PDFs, our workflow is as follows: \nwe first utilize the [open-source GLM-OCR method](https://www.kaggle.com/code/angantyr/text-extraction-from-pdf-docs-using-glm-oc), featuring custom parameter tuning and parallelization enhancements. Then we can get this:\n```python\n{\n \"p-1_f-0\": \"1. kt 94/k 1263\",\n \"p-1_f-1\": \"14 1/2 GÍN KB \"KI\" I-lí-dan\",\n \"p-1_f-2\": \"14 1/2 shekels of silver is owed by Ili-dan\",\n \"p-1_f-3\": \"DUMU Šu-ki-tim 2 ma-na 10 GÍN / KU\",\n \"p-1_f-4\": \"son of Sukkutum. 2 minas 10 shekels of\",\n ...\n}\n```\n| | page_num | field_num | x | y | w | h | x_pad | y_pad | w_pad | h_pad |\n|---:|-----------:|------------:|-----:|----:|-----:|-----:|--------:|--------:|--------:|--------:|\n| 0 | 1 | -1 | 0 | 0 | 2075 | 2809 | 0 | 0 | 2075 | 2809 |\n| 1 | 1 | 0 | 864 | 253 | 378 | 50 | 852 | 241 | 402 | 74 |\n| 2 | 1 | 1 | 166 | 403 | 533 | 48 | 154 | 391 | 557 | 72 |\n| 3 | 1 | 2 | 1167 | 410 | 746 | 52 | 1155 | 398 | 770 | 76 |\n| 4 | 1 | 3 | 162 | 470 | 773 | 49 | 150 | 458 | 797 | 73 |\n| ... |\n\n