{ "id": 664177, "title": "Data Update", "url": "/competitions/deep-past-initiative-machine-translation/discussion/664177", "author_name": "Ryan Holbrook", "post_date": "2025-12-22T21:06:30.727000", "votes": 44, "comment_count": 33, "views": 0, "content": "

Hi everyone,

\n

We just published several new supplemental data files. We'll try to add more comprehensive documentation for these in the coming days, but for now:

\n\n

We hope you find these useful!

", "messages": [ { "id": 3380710, "postDate": "2025-12-22T21:06:30.727Z", "content": "

Hi everyone,

\n

We just published several new supplemental data files. We'll try to add more comprehensive documentation for these in the coming days, but for now:

\n\n

We hope you find these useful!

", "rawMarkdown": "Hi everyone,\n\nWe just published several new supplemental data files. We'll try to add more comprehensive documentation for these in the coming days, but for now:\n- `eBL_Dictionary.csv` The complete dictionary of Akkadian words from the eBL database. It collects the data provided by the URLs at `eBL` in the `OA_Lexicon_eBL.csv` file. (No need to scrape the website now!)\n- `resources.csv` A list of resources that might be used for additional data.\n- `Sentences_Oare_FirstWord_LinNum.csv` An aid to aligning translations at the sentence level for the data in `train.csv`. Indicates the first word of each sentence and its location on the tablet.\n\nWe hope you find these useful!", "votes": 44 }, { "id": 3381146, "postDate": "2025-12-23T21:08:33.173Z", "content": "

@ryanholbrook Is the Dataset Instructions available provided correct?

\n

https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions

\n

For deity / divinity, it reports it should be preceded by {d} but it seems to (d) instead?

\n
\n

{d} = dingir ‘god, deity’ — d preceding non-human divine actors

\n
\n

For instance: 004a7dbd-57ce-46f8-9691-409be61c676e

\n\n

Also, it reports:

\n\n

But we also have […] alone and for x, we could have x x or x x x and more.

\n

Could you clarify?

\n

Thanks.

", "rawMarkdown": "@ryanholbrook Is the Dataset Instructions available provided correct?\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\n\nFor deity / divinity, it reports it should be preceded by `{d}` but it seems to `(d)` instead?\n\n>{d} = dingir ‘god, deity’ — d preceding non-human divine actors\n\nFor instance: 004a7dbd-57ce-46f8-9691-409be61c676e\n\n- KIŠIB ma-nu-ba-lúm-a-šur DUMU ṣí-lá-**(d)**IM KIŠIB šu-**(d)**EN.LÍL DUMU ma-nu-ki-a-šur KIŠIB MAN-a-šur DUMU a-ta-a\n- Seal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.\n\nAlso, it reports:\n- [ x ] gap\n- … big_gap\n- [… …] big_gap\n\nBut we also have [...] alone and for x, we could have x x or x x x and more.\n\nCould you clarify?\n\nThanks.\n", "votes": 4, "replies": [ { "id": 3381381, "postDate": "2025-12-24T11:59:28.223Z", "content": "

@mpware \nShall we team-up?

", "rawMarkdown": "@mpware \nShall we team-up? ", "votes": -1, "replies": [ { "id": 3381789, "postDate": "2025-12-25T14:33:44.073Z", "content": "

No, it's too early.

", "rawMarkdown": "No, it's too early.", "votes": 1 }, { "id": 3381805, "postDate": "2025-12-25T15:15:30.580Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3381470, "postDate": "2025-12-24T17:24:58.340Z", "content": "

Thank you for posting this MPWARE. You're correct in identifying some inconsistencies in the dataset, where in some cases the determinative has parentheses instead of curly brackets. Also some of the broken signs x x slipped through our normalization efforts. We recommend making those replacements for consistency. Thank you again for your post!

", "rawMarkdown": "Thank you for posting this MPWARE. You're correct in identifying some inconsistencies in the dataset, where in some cases the determinative has parentheses instead of curly brackets. Also some of the broken signs x x slipped through our normalization efforts. We recommend making those replacements for consistency. Thank you again for your post! ", "votes": 1, "replies": [ { "id": 3381788, "postDate": "2025-12-25T14:32:51.190Z", "content": "

Thanks, in fact in train.csv it's () and in published_texts.csv it's {}. The published_texts.csv \"transliteration\" from \"transliteration_orig\" has gap+ big_gap alignment wrong for around 200 rows but it's not a real problem. Even a few \"transliteration_orig\" already come with big_gap, for instance: bf95c240-a277-4427-ae93-cc5602b44fdf

\n
\n

{large break} … x … ú-lá-pá-at … ni lá i-ta-na x na-ma … a-wa-tim … a-bi a-ta be-li a-ta … a-dí … x É ba … tí x … < big_gap > {large break}

\n
", "rawMarkdown": "Thanks, in fact in train.csv it's `()` and in published_texts.csv it's `{}`. The published_texts.csv \"transliteration\" from \"transliteration_orig\" has `gap`+ `big_gap` alignment wrong for around 200 rows but it's not a real problem. Even a few \"transliteration_orig\" already come with big_gap, for instance: bf95c240-a277-4427-ae93-cc5602b44fdf\n\n>{large break} … x … ú-lá-pá-at … ni lá i-ta-na x na-ma … a-wa-tim … a-bi a-ta be-li a-ta … a-dí … x É ba … tí x … < big_gap > {large break}", "votes": 3 } ] } ] }, { "id": 3381012, "postDate": "2025-12-23T15:18:22.470Z", "content": "

@ryanholbrook @deeppast \nI noticed that after matching text_uuid from Sentences_Oare_FirstWord_LinNum.csv with oare_id from train.csv, only 253 out of 1,561 documents (16.2%) have sentence-level alignment data available. The remaining 1,308 documents from train.csv don't have corresponding entries in Sentences_Oare.

\n

Are there plans to provide sentence-level alignment for the remaining train documents?\nThank you for your help!

", "rawMarkdown": "@ryanholbrook @deeppast \nI noticed that after matching text_uuid from Sentences_Oare_FirstWord_LinNum.csv with oare_id from train.csv, only 253 out of 1,561 documents (16.2%) have sentence-level alignment data available. The remaining 1,308 documents from train.csv don't have corresponding entries in Sentences_Oare.\n\nAre there plans to provide sentence-level alignment for the remaining train documents?\nThank you for your help!", "votes": 1, "replies": [ { "id": 3381053, "postDate": "2025-12-23T16:37:42.393Z", "content": "

Unfortunately, what we posted is all that exists at this time. A significant part of this challenge is to assemble all useful datasets for the purpose of machine translation tools for Old Assyrian. This collective effort will be a novel contribution and a first for this dialect of Akkadian. Thank you for your collective efforts toward this goal!

", "rawMarkdown": "Unfortunately, what we posted is all that exists at this time. A significant part of this challenge is to assemble all useful datasets for the purpose of machine translation tools for Old Assyrian. This collective effort will be a novel contribution and a first for this dialect of Akkadian. Thank you for your collective efforts toward this goal!", "votes": 2 }, { "id": 3381167, "postDate": "2025-12-23T23:25:10.593Z", "content": "

Did you try using those for training? Adding the sentences made my model perform much worse. Every attempt to follow the instructions for cleaning the data or adding to the data has led to worse performance. It’s a bit frustrating

", "rawMarkdown": "Did you try using those for training? Adding the sentences made my model perform much worse. Every attempt to follow the instructions for cleaning the data or adding to the data has led to worse performance. It’s a bit frustrating", "votes": 3, "replies": [ { "id": 3381178, "postDate": "2025-12-24T00:52:45.053Z", "content": "

I have also been encountering this problem. It may be because the test set is derived from unpublished content—no published texts are used for the test data(reply)—so the translated content differs from the published Old Assyrian texts, and there may even be newly translated content.

\n

Therefore, my suggestion is to avoid modifying the training set data as much as possible. At most, we can convert the document-level aligned data in the training set to sentence-level alignment (implemented via the newly added Sentences_Oare_FirstWord_LinNum.csv). After all, the maximum input and output length of currently publicly available models is 512, and I am now trying out this new idea.Good luck:)

", "rawMarkdown": "I have also been encountering this problem. It may be because the test set is derived from unpublished content—**no published texts are used for the test data**[(reply)](https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663998#3381032)—so the translated content differs from the published Old Assyrian texts, and there may even be newly translated content. \n\nTherefore, my suggestion is to avoid modifying the training set data as much as possible. At most, we can convert the **document-level aligned** data in the training set to **sentence-level alignment** (implemented via the newly added **Sentences_Oare_FirstWord_LinNum.csv**). After all, the maximum input and output length of currently publicly available models is 512, and I am now trying out this new idea.Good luck:)", "votes": 2 }, { "id": 3381490, "postDate": "2025-12-24T18:19:47.010Z", "content": "

That’s what I was saying… I tried it. It reduced lb. I tried using sentences in training and using them as validation - both led to worse performance. I can’t seem to identify a CV-LB correlation which certainly makes things harder. Replacing [x] with and all that led to worse lb. Everything that I’ve tried as recommended has led to worse lb.

", "rawMarkdown": "That’s what I was saying… I tried it. It reduced lb. I tried using sentences in training and using them as validation - both led to worse performance. I can’t seem to identify a CV-LB correlation which certainly makes things harder. Replacing [x] with and all that led to worse lb. Everything that I’ve tried as recommended has led to worse lb. ", "votes": 1 }, { "id": 3381500, "postDate": "2025-12-24T18:44:48.980Z", "content": "

Thanks for this clarification. The data cleaning recommendations came from our previous attempts to try to normalize identified instances of broken text. Over the years there have been a number of different markings which specialists have used to mark broken passages, and because we assumed these many different signs for breaks would pose a problem for MT, we thought that using only two signs would help clean up the problem. But of course, the problem stems from a much more difficult issue to begin with, in that when there is broken text, which is listed with any of the signs for broken text (e.g. x, [x], …, […], etc.), there is no clear way to determine what is missing, at least not currently. This has been attempted before as a masked language modeling task (https://aclanthology.org/2021.emnlp-main.384/), but we felt it would be too difficult to try to do both tasks in the same challenge.

", "rawMarkdown": "Thanks for this clarification. The data cleaning recommendations came from our previous attempts to try to normalize identified instances of broken text. Over the years there have been a number of different markings which specialists have used to mark broken passages, and because we assumed these many different signs for breaks would pose a problem for MT, we thought that using only two signs would help clean up the problem. But of course, the problem stems from a much more difficult issue to begin with, in that when there is broken text, which is listed with any of the signs for broken text (e.g. x, [x], ..., [...], etc.), there is no clear way to determine what is missing, at least not currently. This has been attempted before as a masked language modeling task (https://aclanthology.org/2021.emnlp-main.384/), but we felt it would be too difficult to try to do both tasks in the same challenge.", "votes": 2 }, { "id": 3381506, "postDate": "2025-12-24T19:13:47.240Z", "content": "

I wonder if you've noticed that in sentence alignment, the starting position of the first given word does not take the numerical values into account, which prevents the sentences from being aligned properly. I'm currently implementing sentence alignment manually, and this might be the reason for the score drop.

", "rawMarkdown": "I wonder if you've noticed that in sentence alignment, the starting position of the first given word does not take the **numerical values** into account, which prevents the sentences from being aligned properly. I'm currently implementing sentence alignment manually, and this might be the reason for the score drop.", "votes": 2 }, { "id": 3381512, "postDate": "2025-12-24T19:34:38.567Z", "content": "

I wrote some code to do it for me but manually checked some samples and they looked good. I believe this competition will be decided by domain knowledge + manual work and not as much code.

", "rawMarkdown": "I wrote some code to do it for me but manually checked some samples and they looked good. I believe this competition will be decided by domain knowledge + manual work and not as much code. ", "votes": 2 }, { "id": 3381514, "postDate": "2025-12-24T19:41:25.417Z", "content": "

Don’t take my comment as assuming the recommendations are bad as it’s likely just my implementation of them. The problem is very complex! It will require a bit more study on my part to understand the domain enough to make any changes to the text. I think the best thing for me start with is looking through every transliteration -> translation pair.. once I do that I think I’ll have a better idea of how to alter things.

", "rawMarkdown": "Don’t take my comment as assuming the recommendations are bad as it’s likely just my implementation of them. The problem is very complex! It will require a bit more study on my part to understand the domain enough to make any changes to the text. I think the best thing for me start with is looking through every transliteration -> translation pair.. once I do that I think I’ll have a better idea of how to alter things.", "votes": 2 }, { "id": 3389533, "postDate": "2026-01-11T11:48:07.477Z", "content": "

@deeppast . the url of the link with additional \")\" added about masked language modeling task (https://aclanthology.org/2021.emnlp-main.384/),

\n

Please try to add a space after the url: https://aclanthology.org/2021.emnlp-main.384/ )

", "rawMarkdown": "@deeppast . the url of the link with additional \")\" added about masked language modeling task (https://aclanthology.org/2021.emnlp-main.384/), \n\nPlease try to add a space after the url: https://aclanthology.org/2021.emnlp-main.384/ )", "votes": 1 } ] }, { "id": 3381557, "postDate": "2025-12-24T22:18:53.790Z", "content": "

I discovered that a big part of this is that some of the english translations don't actually align with the akkadian text they contain more english than akkadian

", "rawMarkdown": "I discovered that a big part of this is that some of the english translations don't actually align with the akkadian text they contain more english than akkadian\n\n", "votes": 1, "replies": [ { "id": 3381561, "postDate": "2025-12-24T22:36:34.217Z", "content": "

I checked lengths of Akkadian and English sentences after alignment and removed all where there was no match either way. The training data has some heavily imbalanced pairs as well that, if removed, worsen my LB. I forked a top performing public notebook for these experiments where I only changed training data and validation data. I’m going to read some related research and see where I’m going wrong

", "rawMarkdown": "I checked lengths of Akkadian and English sentences after alignment and removed all where there was no match either way. The training data has some heavily imbalanced pairs as well that, if removed, worsen my LB. I forked a top performing public notebook for these experiments where I only changed training data and validation data. I’m going to read some related research and see where I’m going wrong ", "votes": 1 }, { "id": 3381792, "postDate": "2025-12-25T14:40:42.327Z", "content": "

You can identifiy more than 1000 suspicious rows with bad transliteration / english length ratio.\nEverything every far from the red line looks bad if you inspect them.

\n

\"\"

", "rawMarkdown": "You can identifiy more than 1000 suspicious rows with bad transliteration / english length ratio.\nEverything every far from the red line looks bad if you inspect them.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F58bf4ee671d5c67420c51066472526ce%2Fbadratio.png?generation=1766673556347876&alt=media)\n\n", "votes": 2 }, { "id": 3382947, "postDate": "2025-12-28T22:02:58.443Z", "content": "

I wouldn’t expect there to be 1-1 length though since Akkadian is very morphological and dense relative to English

", "rawMarkdown": "I wouldn’t expect there to be 1-1 length though since Akkadian is very morphological and dense relative to English " }, { "id": 3382952, "postDate": "2025-12-28T22:18:46.187Z", "content": "

Using a rough estimate gave me around 163 incomplete translations: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663849

\n

It's not a guarantee but some are definitely incomplete. The case of long translation for shorter transliterations I have not verified yet.

\n

\"\"

\n

EDIT: For comparison purposes I measure the length in the number of words by joining signs at hyphens and dots (-, .)/separating at spaces, and removing empty strings ('').

", "rawMarkdown": "Using a rough estimate gave me around 163 incomplete translations: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663849\n\nIt's not a guarantee but some are definitely incomplete. The case of long translation for shorter transliterations I have not verified yet.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2259646%2Fdf56e43e2eb4040ae000181209301043%2FTransliteration%20length%20vs.%20translation%20matters.png?generation=1766960230886815&alt=media)\n\n\nEDIT: For comparison purposes I measure the length in the number of words by joining signs at hyphens and dots (-, .)/separating at spaces, and removing empty strings ('').", "votes": 1 }, { "id": 3383945, "postDate": "2025-12-31T06:55:43.320Z", "content": "

I assumed length ratio filter of 2 (Akkadian to English) and removed these incomplete translation pairs but LB dropped by 1 whole point

", "rawMarkdown": "I assumed length ratio filter of 2 (Akkadian to English) and removed these incomplete translation pairs but LB dropped by 1 whole point " }, { "id": 3383992, "postDate": "2025-12-31T09:11:02.257Z", "content": "

Have you tried to filter out the incomplete translations? I think 2:1 may be a bit too rough but 1.626 - some-B could be better.

\n

There may also be the issue of underrepresentation of some syllables as some are pretty rare.

", "rawMarkdown": "Have you tried to filter out the incomplete translations? I think 2:1 may be a bit too rough but 1.626 - _some-B_ could be better.\n\nThere may also be the issue of underrepresentation of some syllables as some are pretty rare." }, { "id": 3384002, "postDate": "2025-12-31T09:29:02.387Z", "content": "

Yeah , I have only tried 2:1 ratio till now , will check 1.6 also

", "rawMarkdown": "Yeah , I have only tried 2:1 ratio till now , will check 1.6 also", "votes": 1 } ] }, { "id": 3389203, "postDate": "2026-01-10T14:15:22.257Z", "content": "

Can Sentences_Oare_FirstWord_LinNum.csv really help?

\n

Even the 253 sounds bad:

\n\n
\n

Based on a detailed inspection of the two candidate columns in Sentences_Oare_FirstWord_Recompose.csv, translation is significantly better and more reliable than translation_eng_recompose: Logical and Mathematical Errors, Repetitions and \"Stuttering\", Incomplete Sentences …

\n
", "rawMarkdown": "Can Sentences_Oare_FirstWord_LinNum.csv really help? \n\nEven the 253 sounds bad:\n- 5 with NaN translation\n- Sentence length 95th percentile does not match to test set.\n- If you recompose translation sentences then it does not match to \"translation\" column from train.csv\n- If you prompt an LLM to compare translations from train.csv and recomposed from provided sentences (translation_eng_recompose), you will get:\n\n>Based on a detailed inspection of the two candidate columns in Sentences_Oare_FirstWord_Recompose.csv, **translation** is significantly better and more reliable than **translation_eng_recompose**: Logical and Mathematical Errors, Repetitions and \"Stuttering\", Incomplete Sentences ...", "votes": 1 } ] }, { "id": 3380740, "postDate": "2025-12-22T23:51:43.590Z", "content": "

Thank you for releasing the new supplemental datasets — they are extremely helpful.

\n

I noticed that TPU acceleration is currently disabled for this competition in Kaggle notebooks. I would like to ask whether it would be possible to enable TPU support.

", "rawMarkdown": "Thank you for releasing the new supplemental datasets — they are extremely helpful.\n\nI noticed that TPU acceleration is currently disabled for this competition in Kaggle notebooks. I would like to ask whether it would be possible to enable TPU support.", "votes": 1 }, { "id": 3383842, "postDate": "2025-12-30T22:54:59.533Z", "content": "

As per Sentences_Oare_FirstWord_LinNum.csv, sentence_uuid = \"f3e05936-d8c0-4a2b-bb21-3079a9ab90ae\"\n(text_uuid: \"3edc936a-c851-4ea9-9d0c-35388bc61288\") has first_word_number=35 (\"i-na\"). However on checking the oare_id, I see \"i-na\" at positions 30 and 36 (29 and 35 with 0-indexing). Is there a mistake in the dataset or am I missing something with counting the words? The transliteration text is as follows:

\n
a-na … ù a-lá-ḫi-im qí-bi-ma um-ma en-um-a-šùr-ma a-na a-lá-ḫi-im qí-bi-ma a-dí ša um-me-a-ni ša a-bi-ni tí-ir-tí a-dí ma-lá ù ší-ni-šu i-li-kà-kum-ma nu-za-kà-nim ù a-na … x ša qá-sú-nu iš-ku-nu-ni i-na na-áp-tù ḫa-ra-nim ú x x i-na wa-ar-ki-tim ki-ma ṣú-ḫa-ru iš-tù ša-lá-tù-ar im-qú-tù-ni-ni-ma KÙ.BABBAR mì-ma lá ub-lu-ni-ni ú a-ṣé-er x x x tù-ša-áš-ni-ú x x x x-ni um-me-a-x … (2 broken lines) (2 broken lines) (3 broken lines) ù wa-ar-kà-sà ša ni-ša-me-ú … URUDU i-sà-ri-du-šu-ma a-na KÙ.BABBAR i-tù-ar a-wa-tum iš-tí ma-ma-ḫi-ir i-bi-tí-iq KÙ.BABBAR 1 GÍN ša ú-šé-ṣí-ú i ma-aḫ-ri-šu-nu a-ša-kán URUDU ša ri šu 2 ANŠE.ḪI.A ṣa-lá-mì 10 i-lá-tim ú ú-nu-sú-nu šé-ep-IŠTAR i-ra-dí-a-ku-nu-tí a-ṣé-er x ma-na ša a-ma-kam i-za-zu-ni x 3 GÚ URUDU KI ma-ma-ḫi-ir ša ku-a-tí x SIG₅ ša e-zi-ba-ku-nu-tí-ni [...]\n
\n

I am not able to align the first word number with the first_word spelling for about 20 to 25 sentences. Any hints as to what's going on here @deeppast ? Should I be collapsing all the multiple x's or something like that?

", "rawMarkdown": "As per `Sentences_Oare_FirstWord_LinNum.csv`, `sentence_uuid = \"f3e05936-d8c0-4a2b-bb21-3079a9ab90ae\"`\n(`text_uuid: \"3edc936a-c851-4ea9-9d0c-35388bc61288\"`) has `first_word_number=35` (\"i-na\"). However on checking the oare_id, I see \"i-na\" at positions 30 and 36 (29 and 35 with 0-indexing). Is there a mistake in the dataset or am I missing something with counting the words? The transliteration text is as follows:\n```\na-na … ù a-lá-ḫi-im qí-bi-ma um-ma en-um-a-šùr-ma a-na a-lá-ḫi-im qí-bi-ma a-dí ša um-me-a-ni ša a-bi-ni tí-ir-tí a-dí ma-lá ù ší-ni-šu i-li-kà-kum-ma nu-za-kà-nim ù a-na … x ša qá-sú-nu iš-ku-nu-ni i-na na-áp-tù ḫa-ra-nim ú x x i-na wa-ar-ki-tim ki-ma ṣú-ḫa-ru iš-tù ša-lá-tù-ar im-qú-tù-ni-ni-ma KÙ.BABBAR mì-ma lá ub-lu-ni-ni ú a-ṣé-er x x x tù-ša-áš-ni-ú x x x x-ni um-me-a-x … (2 broken lines) (2 broken lines) (3 broken lines) ù wa-ar-kà-sà ša ni-ša-me-ú … URUDU i-sà-ri-du-šu-ma a-na KÙ.BABBAR i-tù-ar a-wa-tum iš-tí ma-ma-ḫi-ir i-bi-tí-iq KÙ.BABBAR 1 GÍN ša ú-šé-ṣí-ú i ma-aḫ-ri-šu-nu a-ša-kán URUDU ša ri šu 2 ANŠE.ḪI.A ṣa-lá-mì 10 i-lá-tim ú ú-nu-sú-nu šé-ep-IŠTAR i-ra-dí-a-ku-nu-tí a-ṣé-er x ma-na ša a-ma-kam i-za-zu-ni x 3 GÚ URUDU KI ma-ma-ḫi-ir ša ku-a-tí x SIG₅ ša e-zi-ba-ku-nu-tí-ni [...]\n```\nI am not able to align the first word number with the first_word spelling for about 20 to 25 sentences. Any hints as to what's going on here @deeppast ? Should I be collapsing all the multiple x's or something like that?", "votes": 2, "replies": [ { "id": 3383994, "postDate": "2025-12-31T09:14:17.433Z", "content": "

The gaps should be collapsed to <gap> (with x, [x], etc.) and <big_gap> (with several x, … or a mixture of those) as mentioned in \"Gaps, damage markers, and parallel alignment\": https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209.

\n

EDIT: I started digging into the issue and see something wrong probably with the gap normalization.

", "rawMarkdown": "The gaps should be collapsed to `` (with x, [x], etc.) and `` (with several x, ... or a mixture of those) as mentioned in \"Gaps, damage markers, and parallel alignment\": https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209.\n\nEDIT: I started digging into the issue and see something wrong probably with the gap normalization.", "replies": [ { "id": 3384055, "postDate": "2025-12-31T11:37:42.203Z", "content": "

Thanks. I am aware of that discussion. It still won’t find „i-na“ at position 35 the way i see it. Did it work for you @angantyr ?

", "rawMarkdown": "Thanks. I am aware of that discussion. It still won’t find „i-na“ at position 35 the way i see it. Did it work for you @angantyr ?" }, { "id": 3384121, "postDate": "2025-12-31T14:50:58.807Z", "content": "

I don't think its the gap normalisation (alone). There are more errors even in this small dataset.\nFor example, one of the first word spellings are \"i-dí-in=(d)MAR.TU\". That should be a \"-\" and not a \"=\" in that string. \nThen there's the case of \"iš-tù\" which doesn't occur as a word on its own but is a part of a word at that very same location…

", "rawMarkdown": "I don't think its the gap normalisation (alone). There are more errors even in this small dataset.\nFor example, one of the first word spellings are \"i-dí-in=(d)MAR.TU\". That should be a \"-\" and not a \"=\" in that string. \nThen there's the case of \"iš-tù\" which doesn't occur as a word on its own but is a part of a word at that very same location..." }, { "id": 3384284, "postDate": "2025-12-31T21:55:15.777Z", "content": "

It does not, no… I'm beginning to think that for some texts we have an \"incomplete transliteration\" issue, analogous to the \"incomplete translation\".

\n

The problem concerns 160 sentences but spread over only 32 texts. I'm digging into it further to see if there is some part of the issue we can deal with the current texts.

", "rawMarkdown": "It does not, no... I'm beginning to think that for some texts we have an \"incomplete transliteration\" issue, analogous to the \"incomplete translation\".\n\nThe problem concerns 160 sentences but spread over only 32 texts. I'm digging into it further to see if there is some part of the issue we can deal with the current texts." }, { "id": 3384304, "postDate": "2026-01-01T00:11:56.550Z", "content": "

@einherjer @deeppast \nI made a simple notebook to analyse the issue: https://www.kaggle.com/code/angantyr/deep-past-2025-1st-word-in-sentence-idx-analysis

\n

From what I see the problem is two-fold:

\n
    \n
  1. misaligned texts (8)
    2. \n
  2. incomplete transliterations (32)
    3. \n
\n

The first is the case reported by @einherjer. I believe the problem to come from the gap normalization. The problem here is that the string representations, e.g., (2 broken lines) take a total of 3 indices instead of 1 as <big_gap>. Accounting for that results in shift of ù from id=67 by 15 to id=52, which is ok with the Sentences_Oare_FirstWord_LinNum.csv.

\n

The second one is the case of incomplete transliterations for 32 texts whose oare_id coincide with the Sentences_Oare_FirstWord_LinNum.csv. As these are a part of a bigger 'tail' on the word length chart I suspect that many of closer points may have similar issues.

\n

\"\"

", "rawMarkdown": "@einherjer @deeppast \nI made a simple notebook to analyse the issue: https://www.kaggle.com/code/angantyr/deep-past-2025-1st-word-in-sentence-idx-analysis\n\nFrom what I see the problem is two-fold:\n1. misaligned texts (8)\n2. incomplete transliterations (32)\n\nThe first is the case reported by @einherjer. I believe the problem to come from the gap normalization. The problem here is that the string representations, e.g., (2 broken lines) take a total of 3 indices instead of 1 as ``. Accounting for that results in shift of `ù` from id=67 by 15 to id=52, which is ok with the `Sentences_Oare_FirstWord_LinNum.csv`.\n\nThe second one is the case of incomplete transliterations for 32 texts whose `oare_id` coincide with the `Sentences_Oare_FirstWord_LinNum.csv`. As these are a part of a bigger 'tail' on the word length chart I suspect that many of closer points may have similar issues.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2259646%2F76b7e27cfe9a289ae92074ab79fa4181%2FSentences_OARE_word_match_issues.png?generation=1767226186052883&alt=media)" } ] } ] }, { "id": 3389548, "postDate": "2026-01-11T12:12:15.353Z", "content": "

Thank you for releasing the new supplemental datasets

", "rawMarkdown": "Thank you for releasing the new supplemental datasets", "votes": -5 } ], "comments": [ { "id": 3381146, "author_name": "MPWARE", "author_url": "", "post_date": "2025-12-23T21:08:33.173000", "content": "

@ryanholbrook Is the Dataset Instructions available provided correct?

\n

https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions

\n

For deity / divinity, it reports it should be preceded by {d} but it seems to (d) instead?

\n
\n

{d} = dingir ‘god, deity’ — d preceding non-human divine actors

\n
\n

For instance: 004a7dbd-57ce-46f8-9691-409be61c676e

\n
    \n
  • KIŠIB ma-nu-ba-lúm-a-šur DUMU ṣí-lá-(d)IM KIŠIB šu-(d)EN.LÍL DUMU ma-nu-ki-a-šur KIŠIB MAN-a-šur DUMU a-ta-a
    • \n
  • Seal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.
    • \n
\n

Also, it reports:

\n
    \n
  • [ x ] gap
    • \n
  • … big_gap
    • \n
  • [… …] big_gap
    • \n
\n

But we also have […] alone and for x, we could have x x or x x x and more.

\n

Could you clarify?

\n

Thanks.

", "votes": 4, "replies": [ { "id": 3381381, "author_name": "", "author_url": "", "post_date": "2025-12-24T11:59:28.223000", "content": "

@mpware \nShall we team-up?

", "votes": -1, "replies": [ { "id": 3381789, "author_name": "MPWARE", "author_url": "", "post_date": "2025-12-25T14:33:44.073000", "content": "

No, it's too early.

", "votes": 1, "replies": [] }, { "id": 3381805, "author_name": "", "author_url": "", "post_date": "2025-12-25T15:15:30.580000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3381470, "author_name": "Adam Anderson", "author_url": "", "post_date": "2025-12-24T17:24:58.340000", "content": "

Thank you for posting this MPWARE. You're correct in identifying some inconsistencies in the dataset, where in some cases the determinative has parentheses instead of curly brackets. Also some of the broken signs x x slipped through our normalization efforts. We recommend making those replacements for consistency. Thank you again for your post!

", "votes": 1, "replies": [ { "id": 3381788, "author_name": "MPWARE", "author_url": "", "post_date": "2025-12-25T14:32:51.190000", "content": "

Thanks, in fact in train.csv it's () and in published_texts.csv it's {}. The published_texts.csv \"transliteration\" from \"transliteration_orig\" has gap+ big_gap alignment wrong for around 200 rows but it's not a real problem. Even a few \"transliteration_orig\" already come with big_gap, for instance: bf95c240-a277-4427-ae93-cc5602b44fdf

\n
\n

{large break} … x … ú-lá-pá-at … ni lá i-ta-na x na-ma … a-wa-tim … a-bi a-ta be-li a-ta … a-dí … x É ba … tí x … < big_gap > {large break}

\n
", "votes": 3, "replies": [] } ] } ] }, { "id": 3381012, "author_name": "automatylicza", "author_url": "", "post_date": "2025-12-23T15:18:22.470000", "content": "

@ryanholbrook @deeppast \nI noticed that after matching text_uuid from Sentences_Oare_FirstWord_LinNum.csv with oare_id from train.csv, only 253 out of 1,561 documents (16.2%) have sentence-level alignment data available. The remaining 1,308 documents from train.csv don't have corresponding entries in Sentences_Oare.

\n

Are there plans to provide sentence-level alignment for the remaining train documents?\nThank you for your help!

", "votes": 1, "replies": [ { "id": 3381053, "author_name": "Adam Anderson", "author_url": "", "post_date": "2025-12-23T16:37:42.393000", "content": "

Unfortunately, what we posted is all that exists at this time. A significant part of this challenge is to assemble all useful datasets for the purpose of machine translation tools for Old Assyrian. This collective effort will be a novel contribution and a first for this dialect of Akkadian. Thank you for your collective efforts toward this goal!

", "votes": 2, "replies": [] }, { "id": 3381167, "author_name": "Jack", "author_url": "", "post_date": "2025-12-23T23:25:10.593000", "content": "

Did you try using those for training? Adding the sentences made my model perform much worse. Every attempt to follow the instructions for cleaning the data or adding to the data has led to worse performance. It’s a bit frustrating

", "votes": 3, "replies": [ { "id": 3381178, "author_name": "耶✌", "author_url": "", "post_date": "2025-12-24T00:52:45.053000", "content": "

I have also been encountering this problem. It may be because the test set is derived from unpublished content—no published texts are used for the test data(reply)—so the translated content differs from the published Old Assyrian texts, and there may even be newly translated content.

\n

Therefore, my suggestion is to avoid modifying the training set data as much as possible. At most, we can convert the document-level aligned data in the training set to sentence-level alignment (implemented via the newly added Sentences_Oare_FirstWord_LinNum.csv). After all, the maximum input and output length of currently publicly available models is 512, and I am now trying out this new idea.Good luck:)

", "votes": 2, "replies": [] }, { "id": 3381490, "author_name": "Jack", "author_url": "", "post_date": "2025-12-24T18:19:47.010000", "content": "

That’s what I was saying… I tried it. It reduced lb. I tried using sentences in training and using them as validation - both led to worse performance. I can’t seem to identify a CV-LB correlation which certainly makes things harder. Replacing [x] with and all that led to worse lb. Everything that I’ve tried as recommended has led to worse lb.

", "votes": 1, "replies": [] }, { "id": 3381500, "author_name": "Adam Anderson", "author_url": "", "post_date": "2025-12-24T18:44:48.980000", "content": "

Thanks for this clarification. The data cleaning recommendations came from our previous attempts to try to normalize identified instances of broken text. Over the years there have been a number of different markings which specialists have used to mark broken passages, and because we assumed these many different signs for breaks would pose a problem for MT, we thought that using only two signs would help clean up the problem. But of course, the problem stems from a much more difficult issue to begin with, in that when there is broken text, which is listed with any of the signs for broken text (e.g. x, [x], …, […], etc.), there is no clear way to determine what is missing, at least not currently. This has been attempted before as a masked language modeling task (https://aclanthology.org/2021.emnlp-main.384/), but we felt it would be too difficult to try to do both tasks in the same challenge.

", "votes": 2, "replies": [] }, { "id": 3381506, "author_name": "耶✌", "author_url": "", "post_date": "2025-12-24T19:13:47.240000", "content": "

I wonder if you've noticed that in sentence alignment, the starting position of the first given word does not take the numerical values into account, which prevents the sentences from being aligned properly. I'm currently implementing sentence alignment manually, and this might be the reason for the score drop.

", "votes": 2, "replies": [] }, { "id": 3381512, "author_name": "Jack", "author_url": "", "post_date": "2025-12-24T19:34:38.567000", "content": "

I wrote some code to do it for me but manually checked some samples and they looked good. I believe this competition will be decided by domain knowledge + manual work and not as much code.

", "votes": 2, "replies": [] }, { "id": 3381514, "author_name": "Jack", "author_url": "", "post_date": "2025-12-24T19:41:25.417000", "content": "

Don’t take my comment as assuming the recommendations are bad as it’s likely just my implementation of them. The problem is very complex! It will require a bit more study on my part to understand the domain enough to make any changes to the text. I think the best thing for me start with is looking through every transliteration -> translation pair.. once I do that I think I’ll have a better idea of how to alter things.

", "votes": 2, "replies": [] }, { "id": 3389533, "author_name": "kebenny", "author_url": "", "post_date": "2026-01-11T11:48:07.477000", "content": "

@deeppast . the url of the link with additional \")\" added about masked language modeling task (https://aclanthology.org/2021.emnlp-main.384/),

\n

Please try to add a space after the url: https://aclanthology.org/2021.emnlp-main.384/ )

", "votes": 1, "replies": [] } ] }, { "id": 3381557, "author_name": "Aaron Bornstein", "author_url": "", "post_date": "2025-12-24T22:18:53.790000", "content": "

I discovered that a big part of this is that some of the english translations don't actually align with the akkadian text they contain more english than akkadian

", "votes": 1, "replies": [ { "id": 3381561, "author_name": "Jack", "author_url": "", "post_date": "2025-12-24T22:36:34.217000", "content": "

I checked lengths of Akkadian and English sentences after alignment and removed all where there was no match either way. The training data has some heavily imbalanced pairs as well that, if removed, worsen my LB. I forked a top performing public notebook for these experiments where I only changed training data and validation data. I’m going to read some related research and see where I’m going wrong

", "votes": 1, "replies": [] }, { "id": 3381792, "author_name": "MPWARE", "author_url": "", "post_date": "2025-12-25T14:40:42.327000", "content": "

You can identifiy more than 1000 suspicious rows with bad transliteration / english length ratio.\nEverything every far from the red line looks bad if you inspect them.

\n

\"\"

", "votes": 2, "replies": [] }, { "id": 3382947, "author_name": "Aaron Bornstein", "author_url": "", "post_date": "2025-12-28T22:02:58.443000", "content": "

I wouldn’t expect there to be 1-1 length though since Akkadian is very morphological and dense relative to English

", "votes": 0, "replies": [] }, { "id": 3382952, "author_name": "Angantyr", "author_url": "", "post_date": "2025-12-28T22:18:46.187000", "content": "

Using a rough estimate gave me around 163 incomplete translations: https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/663849

\n

It's not a guarantee but some are definitely incomplete. The case of long translation for shorter transliterations I have not verified yet.

\n

\"\"

\n

EDIT: For comparison purposes I measure the length in the number of words by joining signs at hyphens and dots (-, .)/separating at spaces, and removing empty strings ('').

", "votes": 1, "replies": [] }, { "id": 3383945, "author_name": "Hrithik Reddy", "author_url": "", "post_date": "2025-12-31T06:55:43.320000", "content": "

I assumed length ratio filter of 2 (Akkadian to English) and removed these incomplete translation pairs but LB dropped by 1 whole point

", "votes": 0, "replies": [] }, { "id": 3383992, "author_name": "Angantyr", "author_url": "", "post_date": "2025-12-31T09:11:02.257000", "content": "

Have you tried to filter out the incomplete translations? I think 2:1 may be a bit too rough but 1.626 - some-B could be better.

\n

There may also be the issue of underrepresentation of some syllables as some are pretty rare.

", "votes": 0, "replies": [] }, { "id": 3384002, "author_name": "Hrithik Reddy", "author_url": "", "post_date": "2025-12-31T09:29:02.387000", "content": "

Yeah , I have only tried 2:1 ratio till now , will check 1.6 also

", "votes": 1, "replies": [] } ] }, { "id": 3389203, "author_name": "MPWARE", "author_url": "", "post_date": "2026-01-10T14:15:22.257000", "content": "

Can Sentences_Oare_FirstWord_LinNum.csv really help?

\n

Even the 253 sounds bad:

\n
    \n
  • 5 with NaN translation
    • \n
  • Sentence length 95th percentile does not match to test set.
    • \n
  • If you recompose translation sentences then it does not match to \"translation\" column from train.csv
    • \n
  • If you prompt an LLM to compare translations from train.csv and recomposed from provided sentences (translation_eng_recompose), you will get:
    • \n
\n
\n

Based on a detailed inspection of the two candidate columns in Sentences_Oare_FirstWord_Recompose.csv, translation is significantly better and more reliable than translation_eng_recompose: Logical and Mathematical Errors, Repetitions and \"Stuttering\", Incomplete Sentences …

\n
", "votes": 1, "replies": [] } ] }, { "id": 3380740, "author_name": "Fernando Rodrigues", "author_url": "", "post_date": "2025-12-22T23:51:43.590000", "content": "

Thank you for releasing the new supplemental datasets — they are extremely helpful.

\n

I noticed that TPU acceleration is currently disabled for this competition in Kaggle notebooks. I would like to ask whether it would be possible to enable TPU support.

", "votes": 1, "replies": [] }, { "id": 3383842, "author_name": "einherjer", "author_url": "", "post_date": "2025-12-30T22:54:59.533000", "content": "

As per Sentences_Oare_FirstWord_LinNum.csv, sentence_uuid = \"f3e05936-d8c0-4a2b-bb21-3079a9ab90ae\"\n(text_uuid: \"3edc936a-c851-4ea9-9d0c-35388bc61288\") has first_word_number=35 (\"i-na\"). However on checking the oare_id, I see \"i-na\" at positions 30 and 36 (29 and 35 with 0-indexing). Is there a mistake in the dataset or am I missing something with counting the words? The transliteration text is as follows:

\n
a-na … ù a-lá-ḫi-im qí-bi-ma um-ma en-um-a-šùr-ma a-na a-lá-ḫi-im qí-bi-ma a-dí ša um-me-a-ni ša a-bi-ni tí-ir-tí a-dí ma-lá ù ší-ni-šu i-li-kà-kum-ma nu-za-kà-nim ù a-na … x ša qá-sú-nu iš-ku-nu-ni i-na na-áp-tù ḫa-ra-nim ú x x i-na wa-ar-ki-tim ki-ma ṣú-ḫa-ru iš-tù ša-lá-tù-ar im-qú-tù-ni-ni-ma KÙ.BABBAR mì-ma lá ub-lu-ni-ni ú a-ṣé-er x x x tù-ša-áš-ni-ú x x x x-ni um-me-a-x … (2 broken lines) (2 broken lines) (3 broken lines) ù wa-ar-kà-sà ša ni-ša-me-ú … URUDU i-sà-ri-du-šu-ma a-na KÙ.BABBAR i-tù-ar a-wa-tum iš-tí ma-ma-ḫi-ir i-bi-tí-iq KÙ.BABBAR 1 GÍN ša ú-šé-ṣí-ú i ma-aḫ-ri-šu-nu a-ša-kán URUDU ša ri šu 2 ANŠE.ḪI.A ṣa-lá-mì 10 i-lá-tim ú ú-nu-sú-nu šé-ep-IŠTAR i-ra-dí-a-ku-nu-tí a-ṣé-er x ma-na ša a-ma-kam i-za-zu-ni x 3 GÚ URUDU KI ma-ma-ḫi-ir ša ku-a-tí x SIG₅ ša e-zi-ba-ku-nu-tí-ni [...]\n
\n

I am not able to align the first word number with the first_word spelling for about 20 to 25 sentences. Any hints as to what's going on here @deeppast ? Should I be collapsing all the multiple x's or something like that?

", "votes": 2, "replies": [ { "id": 3383994, "author_name": "Angantyr", "author_url": "", "post_date": "2025-12-31T09:14:17.433000", "content": "

The gaps should be collapsed to <gap> (with x, [x], etc.) and <big_gap> (with several x, … or a mixture of those) as mentioned in \"Gaps, damage markers, and parallel alignment\": https://www.kaggle.com/competitions/deep-past-initiative-machine-translation/discussion/665209.

\n

EDIT: I started digging into the issue and see something wrong probably with the gap normalization.

", "votes": 0, "replies": [ { "id": 3384055, "author_name": "einherjer", "author_url": "", "post_date": "2025-12-31T11:37:42.203000", "content": "

Thanks. I am aware of that discussion. It still won’t find „i-na“ at position 35 the way i see it. Did it work for you @angantyr ?

", "votes": 0, "replies": [] }, { "id": 3384121, "author_name": "einherjer", "author_url": "", "post_date": "2025-12-31T14:50:58.807000", "content": "

I don't think its the gap normalisation (alone). There are more errors even in this small dataset.\nFor example, one of the first word spellings are \"i-dí-in=(d)MAR.TU\". That should be a \"-\" and not a \"=\" in that string. \nThen there's the case of \"iš-tù\" which doesn't occur as a word on its own but is a part of a word at that very same location…

", "votes": 0, "replies": [] }, { "id": 3384284, "author_name": "Angantyr", "author_url": "", "post_date": "2025-12-31T21:55:15.777000", "content": "

It does not, no… I'm beginning to think that for some texts we have an \"incomplete transliteration\" issue, analogous to the \"incomplete translation\".

\n

The problem concerns 160 sentences but spread over only 32 texts. I'm digging into it further to see if there is some part of the issue we can deal with the current texts.

", "votes": 0, "replies": [] }, { "id": 3384304, "author_name": "Angantyr", "author_url": "", "post_date": "2026-01-01T00:11:56.550000", "content": "

@einherjer @deeppast \nI made a simple notebook to analyse the issue: https://www.kaggle.com/code/angantyr/deep-past-2025-1st-word-in-sentence-idx-analysis

\n

From what I see the problem is two-fold:

\n
    \n
  1. misaligned texts (8)
    2. \n
  2. incomplete transliterations (32)
    3. \n
\n

The first is the case reported by @einherjer. I believe the problem to come from the gap normalization. The problem here is that the string representations, e.g., (2 broken lines) take a total of 3 indices instead of 1 as <big_gap>. Accounting for that results in shift of ù from id=67 by 15 to id=52, which is ok with the Sentences_Oare_FirstWord_LinNum.csv.

\n

The second one is the case of incomplete transliterations for 32 texts whose oare_id coincide with the Sentences_Oare_FirstWord_LinNum.csv. As these are a part of a bigger 'tail' on the word length chart I suspect that many of closer points may have similar issues.

\n

\"\"

", "votes": 0, "replies": [] } ] } ] }, { "id": 3389548, "author_name": "Khushi Yadav", "author_url": "", "post_date": "2026-01-11T12:12:15.353000", "content": "

Thank you for releasing the new supplemental datasets

", "votes": -5, "replies": [] } ], "raw_markdown_by_id": { "3380710": "Hi everyone,\n\nWe just published several new supplemental data files. We'll try to add more comprehensive documentation for these in the coming days, but for now:\n- `eBL_Dictionary.csv` The complete dictionary of Akkadian words from the eBL database. It collects the data provided by the URLs at `eBL` in the `OA_Lexicon_eBL.csv` file. (No need to scrape the website now!)\n- `resources.csv` A list of resources that might be used for additional data.\n- `Sentences_Oare_FirstWord_LinNum.csv` An aid to aligning translations at the sentence level for the data in `train.csv`. Indicates the first word of each sentence and its location on the tablet.\n\nWe hope you find these useful!", "3381146": "@ryanholbrook Is the Dataset Instructions available provided correct?\n\nhttps://www.kaggle.com/competitions/deep-past-initiative-machine-translation/overview/dataset-instructions\n\nFor deity / divinity, it reports it should be preceded by `{d}` but it seems to `(d)` instead?\n\n>{d} = dingir ‘god, deity’ — d preceding non-human divine actors\n\nFor instance: 004a7dbd-57ce-46f8-9691-409be61c676e\n\n- KIŠIB ma-nu-ba-lúm-a-šur DUMU ṣí-lá-**(d)**IM KIŠIB šu-**(d)**EN.LÍL DUMU ma-nu-ki-a-šur KIŠIB MAN-a-šur DUMU a-ta-a\n- Seal of Mannum-balum-Aššur son of Ṣilli-Adad, seal of Šu-Illil son of Mannum-kī-Aššur, seal of Puzur-Aššur son of Ataya.\n\nAlso, it reports:\n- [ x ] gap\n- … big_gap\n- [… …] big_gap\n\nBut we also have [...] alone and for x, we could have x x or x x x and more.\n\nCould you clarify?\n\nThanks.\n", "3381012": "@ryanholbrook @deeppast \nI noticed that after matching text_uuid from Sentences_Oare_FirstWord_LinNum.csv with oare_id from train.csv, only 253 out of 1,561 documents (16.2%) have sentence-level alignment data available. The remaining 1,308 documents from train.csv don't have corresponding entries in Sentences_Oare.\n\nAre there plans to provide sentence-level alignment for the remaining train documents?\nThank you for your help!", "3380740": "Thank you for releasing the new supplemental datasets — they are extremely helpful.\n\nI noticed that TPU acceleration is currently disabled for this competition in Kaggle notebooks. I would like to ask whether it would be possible to enable TPU support.", "3383842": "As per `Sentences_Oare_FirstWord_LinNum.csv`, `sentence_uuid = \"f3e05936-d8c0-4a2b-bb21-3079a9ab90ae\"`\n(`text_uuid: \"3edc936a-c851-4ea9-9d0c-35388bc61288\"`) has `first_word_number=35` (\"i-na\"). However on checking the oare_id, I see \"i-na\" at positions 30 and 36 (29 and 35 with 0-indexing). Is there a mistake in the dataset or am I missing something with counting the words? The transliteration text is as follows:\n```\na-na … ù a-lá-ḫi-im qí-bi-ma um-ma en-um-a-šùr-ma a-na a-lá-ḫi-im qí-bi-ma a-dí ša um-me-a-ni ša a-bi-ni tí-ir-tí a-dí ma-lá ù ší-ni-šu i-li-kà-kum-ma nu-za-kà-nim ù a-na … x ša qá-sú-nu iš-ku-nu-ni i-na na-áp-tù ḫa-ra-nim ú x x i-na wa-ar-ki-tim ki-ma ṣú-ḫa-ru iš-tù ša-lá-tù-ar im-qú-tù-ni-ni-ma KÙ.BABBAR mì-ma lá ub-lu-ni-ni ú a-ṣé-er x x x tù-ša-áš-ni-ú x x x x-ni um-me-a-x … (2 broken lines) (2 broken lines) (3 broken lines) ù wa-ar-kà-sà ša ni-ša-me-ú … URUDU i-sà-ri-du-šu-ma a-na KÙ.BABBAR i-tù-ar a-wa-tum iš-tí ma-ma-ḫi-ir i-bi-tí-iq KÙ.BABBAR 1 GÍN ša ú-šé-ṣí-ú i ma-aḫ-ri-šu-nu a-ša-kán URUDU ša ri šu 2 ANŠE.ḪI.A ṣa-lá-mì 10 i-lá-tim ú ú-nu-sú-nu šé-ep-IŠTAR i-ra-dí-a-ku-nu-tí a-ṣé-er x ma-na ša a-ma-kam i-za-zu-ni x 3 GÚ URUDU KI ma-ma-ḫi-ir ša ku-a-tí x SIG₅ ša e-zi-ba-ku-nu-tí-ni [...]\n```\nI am not able to align the first word number with the first_word spelling for about 20 to 25 sentences. Any hints as to what's going on here @deeppast ? Should I be collapsing all the multiple x's or something like that?", "3389548": "Thank you for releasing the new supplemental datasets" } }