![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Ffce4ceaac6bb12a8fffd668816bc945f%2Fpage54.png?generation=1769862895279329&alt=media\")
Discussions are so quiet since Kaggle removed the Discussions rewards, that's sad … Anyway, I'm trying to use Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf provided as additional data to train with more transliterations / translations.
\nI faced some issues: First, it's difficult to use a PDF library to read content safely as we cannot capture unicode characters. If you inspect some sections, for instance:
\n\n\npage 51 - CCT 5, 6a, Line 3:
\n
From text extracted from PDF:
\nAlmost real text extracted by a multi-modal model:
\nLook at the translation differences: Pnsu vs Pušu
\n\n\nAnother example, page 54 line 12:
\n
From text extracted from PDF:
\nAlmost real text extracted by a multi-modal model:
\nLook at the garbage transliteration :\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A vs 8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A
\nEven parsing with a multi-modal model is far from being 100% correct. Second, the sentences alignment in some letter looks weird such as:
\n\n\npage 58: TC 3,95 line 16:
\n
ha-am-ša-tim šu-ma => If
\n{\"transliteration\": \"lu-up-ta a-na 25\", \"translation\": \"to run for 25 weeks.\"},\n# 15\n{\"transliteration\": \"ha-am-ša-tim šu-ma\", \"translation\": \"If\"},\n{\"transliteration\": \"lá i-ma-ga-ar-ku-nu\", \"translation\": \"he does not agree with you,\"},\n{\"transliteration\": \"a-na 30 lu 35 ha-am-ša-/tim\", \"translation\": \"then draw it up for 30 or 35 weeks.\"},\n{\"transliteration\": \"lu-up-ta tup-pu-šu\", \"translation\": \"Certify his tablet\"},\n{\"transliteration\": \"hi-ir-ma-ma a-na A-šur-na-/da\", \"translation\": \"and entrust it to Aššur-nāda.\"},\nI'm sharing some of my parsed letters.
", "messages": [ { "id": 3396734, "postDate": "2026-01-25T18:18:59.313Z", "content": "Discussions are so quiet since Kaggle removed the Discussions rewards, that's sad … Anyway, I'm trying to use Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf provided as additional data to train with more transliterations / translations.
\nI faced some issues: First, it's difficult to use a PDF library to read content safely as we cannot capture unicode characters. If you inspect some sections, for instance:
\n\n\npage 51 - CCT 5, 6a, Line 3:
\n
From text extracted from PDF:
\nAlmost real text extracted by a multi-modal model:
\nLook at the translation differences: Pnsu vs Pušu
\n\n\nAnother example, page 54 line 12:
\n
From text extracted from PDF:
\nAlmost real text extracted by a multi-modal model:
\nLook at the garbage transliteration :\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A vs 8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A
\nEven parsing with a multi-modal model is far from being 100% correct. Second, the sentences alignment in some letter looks weird such as:
\n\n\npage 58: TC 3,95 line 16:
\n
ha-am-ša-tim šu-ma => If
\n{\"transliteration\": \"lu-up-ta a-na 25\", \"translation\": \"to run for 25 weeks.\"},\n# 15\n{\"transliteration\": \"ha-am-ša-tim šu-ma\", \"translation\": \"If\"},\n{\"transliteration\": \"lá i-ma-ga-ar-ku-nu\", \"translation\": \"he does not agree with you,\"},\n{\"transliteration\": \"a-na 30 lu 35 ha-am-ša-/tim\", \"translation\": \"then draw it up for 30 or 35 weeks.\"},\n{\"transliteration\": \"lu-up-ta tup-pu-šu\", \"translation\": \"Certify his tablet\"},\n{\"transliteration\": \"hi-ir-ma-ma a-na A-šur-na-/da\", \"translation\": \"and entrust it to Aššur-nāda.\"},\nI'm sharing some of my parsed letters.
", "rawMarkdown": "Discussions are so quiet since Kaggle removed the Discussions rewards, that's sad ... Anyway, I'm trying to use **Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf** provided as additional data to train with more transliterations / translations.\n\nI faced some issues: First, it's difficult to use a PDF library to read content safely as we cannot capture unicode characters. If you inspect some sections, for instance:\n\n> page 51 - CCT 5, 6a, Line 3:\n\nFrom text extracted from PDF:\n- Transliteration: Pu-su-ki-in ù A-sùr-ta-ak-/la-ku\n- Translation: Pnsu-kén and Aššur-taldâku:\n\nAlmost real text extracted by a multi-modal model:\n- Transliteration: Pu-šu-ki-in ù A-šùr-ta-ak-/la-ku\n- Translation: Pušu-kēn and Aššur-taklāku:\n\nLook at the translation differences: **Pnsu** vs **Pušu**\n\n>Another example, page 54 line 12:\n\nFrom text extracted from PDF:\n- Transliteration: :\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A\n- Translation: These 8 textiles plus 15 textiles\n\nAlmost real text extracted by a multi-modal model:\n- Transliteration: 8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A\n- Translation: These 8 textiles plus 15 textiles\n\nLook at the garbage transliteration **:\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A** vs **8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A**\n\nEven parsing with a multi-modal model is far from being 100% correct. Second, the sentences alignment in some letter looks weird such as:\n\n>page 58: TC 3,95 line 16:\n\nha-am-ša-tim šu-ma => If\n\n```python\n{\"transliteration\": \"lu-up-ta a-na 25\", \"translation\": \"to run for 25 weeks.\"},\n# 15\n{\"transliteration\": \"ha-am-ša-tim šu-ma\", \"translation\": \"If\"},\n{\"transliteration\": \"lá i-ma-ga-ar-ku-nu\", \"translation\": \"he does not agree with you,\"},\n{\"transliteration\": \"a-na 30 lu 35 ha-am-ša-/tim\", \"translation\": \"then draw it up for 30 or 35 weeks.\"},\n{\"transliteration\": \"lu-up-ta tup-pu-šu\", \"translation\": \"Certify his tablet\"},\n{\"transliteration\": \"hi-ir-ma-ma a-na A-šur-na-/da\", \"translation\": \"and entrust it to Aššur-nāda.\"},\n```\n\nI'm sharing some of my [parsed letters](https://www.kaggle.com/datasets/mpware/dpc-larsen). \n\n\n", "votes": 22 }, { "id": 3398938, "postDate": "2026-01-29T23:55:01.633Z", "content": "Great work, and thanks for keeping things rolling! Note that all the PDFs that have been provided so far already have transliterations in the publications.csv - what is missing are the translations for these texts. So you don't need to redo all the transliteration text, you just need to get the corresponding translations.
\nYou're right to identify that the alignment is an issue, which is why looking at the line numbers as they correspond to the translations are somewhat helpful, but also not exactly in alignment. For example the last word of the previous sentence was \"ha-am-ša-tim\" and the first word of the next sentence began \"šu-ma\".
\nUsing the dictionary for lemmatizing should be helpful if you're matching on the sense of the Akkadian with the English translations, although this type of lemmatization is very advanced work.
", "rawMarkdown": "Great work, and thanks for keeping things rolling! Note that all the PDFs that have been provided so far already have transliterations in the publications.csv - what is missing are the translations for these texts. So you don't need to redo all the transliteration text, you just need to get the corresponding translations.\n\nYou're right to identify that the alignment is an issue, which is why looking at the line numbers as they correspond to the translations are somewhat helpful, but also not exactly in alignment. For example the last word of the previous sentence was \"ha-am-ša-tim\" and the first word of the next sentence began \"šu-ma\". \n\n- šu-ma = šumma = if (https://www.ebl.lmu.de/dictionary/%C5%A1umma%20I)\n- ha-am-ša-tim = hamuštum = group of 5 (https://www.ebl.lmu.de/dictionary?origin=CDA&word=hamu%C5%A1tum)\n\nUsing the dictionary for lemmatizing should be helpful if you're matching on the sense of the Akkadian with the English translations, although this type of lemmatization is very advanced work.", "votes": 3, "replies": [ { "id": 3399879, "postDate": "2026-01-31T12:50:10.847Z", "content": "@deeppast Do you think transliteration in published.csv are safe? IMO it's difficult to be used due to what I've explained in my post, parsed PDF does not reflect the printed transliteration by design. PDF parser cannot be used safely for \"Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf\":
\nPage 54, line 12:
\n\n\nLook at the garbage transliteration :\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A instead of 8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A
\n
published.csv:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fbc2b5d6c06196a45c18e1a9745a8c22a%2Fpage54_zoom.png?generation=1769863618633916&alt=media\")
Printed text:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fb35cc081d7d0bfb15330a8171975ff6a%2Fpage54_1.png?generation=1769863159484143&alt=media\")
This is what I get by another method:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F25727492a276cfd10638a0577c903b14%2Fprint_54.png?generation=1769863649188496&alt=media\")
Any thoughts?
\nIf the hidden set contains such garbage then just let us know because I should not spend more time to capture real transliteration/english but just garbage transliteration/english pairs.
", "rawMarkdown": "@deeppast Do you think transliteration in published.csv are safe? IMO it's difficult to be used due to what I've explained in my post, parsed PDF does not reflect the printed transliteration by design. PDF parser cannot be used safely for \"Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf\":\n\nPage 54, line 12:\n\n>Look at the garbage transliteration :\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A **instead of** 8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A\n\n**published.csv:**\n\n\n\n\n\n**Printed text:**\n\n\n\n\nThis is what I get by another method:\n\n\n\nAny thoughts?\n\nIf the hidden set contains such garbage then just let us know because I should not spend more time to capture real transliteration/english but just garbage transliteration/english pairs.", "votes": 1, "replies": [ { "id": 3402372, "postDate": "2026-02-05T22:04:22.037Z", "content": "Re-OCR sounds a good way to get clean transliteration.\nI've tried a few OCR models, DeepSeek-OCR-2 is not bad but far from the majors.
", "rawMarkdown": "Re-OCR sounds a good way to get clean transliteration.\nI've tried a few OCR models, [DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) is not bad but far from the majors.", "votes": 1 }, { "id": 3402474, "postDate": "2026-02-06T06:13:27.623Z", "content": "Have you extracted the image layer from the PDF prior to the OCR? I found that most of the text in the text layer is likely already an OCR attempt, sometimes broken so any text-type parser will work with dummy data underneath.
\nI tried DeepSeek-OCR-2 as well and can confirm it was ok, but failed to capture some letters. The determinatives will be a problem too.
", "rawMarkdown": "Have you extracted the image layer from the PDF prior to the OCR? I found that most of the text in the text layer is likely already an OCR attempt, sometimes broken so any text-type parser will work with dummy data underneath.\n\nI tried DeepSeek-OCR-2 as well and can confirm it was ok, but failed to capture some letters. The determinatives will be a problem too.", "votes": 1 }, { "id": 3402797, "postDate": "2026-02-06T21:11:12.810Z", "content": "\nYes, I've tried both:
\nOption #2 is worse than Option #1 for me with DeepSeek-OCR-2. I've also tried different kind of image preprocessing.
\nHere is an example for 16cct32b for option #1:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fa2dc4081d1fab7f8812b3fa453f01b01%2Fdeepseek-ocr-2.png?generation=1770411279442676&alt=media\")
In fact English is quite perfect but Akkadian is 5-10% bad. I get better results with GLM.
\nSame example with GLM, better but not good.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F4b3d1ade91eb94a4a59621161dbbadd7%2Fglm-ocr.png?generation=1770411669878255&alt=media\")
Same example with the majors multi-modals, not perfect but much better:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2F61cf2df1adedc4fae2e1a8802cfe69eb%2Fmajors-ocr.png?generation=1770412064000002&alt=media\")
What do you have for letter 16 CCT 3, 2b page 73? Is it better than my experiments?
", "rawMarkdown": "@angantyr \n\nYes, I've tried both: \n1. Render PDF and take screenshot high res (700 DPI)\n2. Use directly embedded images\n\nOption #2 is worse than Option #1 for me with DeepSeek-OCR-2. I've also tried different kind of image preprocessing. \n\nHere is an example for 16cct32b for option #1:\n\n\n\nIn fact English is quite perfect but Akkadian is 5-10% bad. I get better results with [GLM](https://huggingface.co/zai-org/GLM-OCR).\n\nSame example with GLM, better but not good.\n\n\nSame example with the majors multi-modals, not perfect but much better:\n\n\n\nWhat do you have for letter 16 CCT 3, 2b page 73? Is it better than my experiments?", "votes": 1 }, { "id": 3402958, "postDate": "2026-02-07T08:53:46.750Z", "content": "I get nothing close to the GLM output, which looks almost perfect -- the sub/superscript parsing is pretty neat. I tried several lighter models with poor results. Gemma-3 much heavier than Deepseek-OCR-2 performed worse.
\nAll of the above needed Google Colab's A100 to work in real time and it's too costly for me. I'll give Unsloth-GLM a try, probably with some image preprocessing to ease capture of emphasized consonants.
\nThank you for the inspiration and insights. It makes the acquisition and cleaning much less frustrating.
\nI'll share my results after the tests, regardless of the quality.
", "rawMarkdown": "I get nothing close to the GLM output, which looks almost perfect -- the sub/superscript parsing is pretty neat. I tried several lighter models with poor results. Gemma-3 much heavier than Deepseek-OCR-2 performed worse.\n\nAll of the above needed Google Colab's A100 to work in real time and it's too costly for me. I'll give Unsloth-GLM a try, probably with some image preprocessing to ease capture of emphasized consonants.\n\nThank you for the inspiration and insights. It makes the acquisition and cleaning much less frustrating.\n\nI'll share my results after the tests, regardless of the quality.", "votes": 1 }, { "id": 3403975, "postDate": "2026-02-09T16:53:06.947Z", "content": "@mpware I come bearing several good and a single bad news.
\nThe bad news is that I fail to reproduce the text in its intact form. The parser fails to account for emphasized letters (ṣ and ṭ) and sometimes even ù. I'll have to check if it was unsloth/GLM-OCR or even the original zai-org/GLM-OCR, that was trained on several languages, all of which do not use these letters. If so, we may have to resort to fine-tuning.
The good news is, plenty, so lets start with the biggest surprise. The end result is pretty good. For 16. CCT 3,2b it correctly captured all subcripts and superscripts, albeit sometimes in html
There still are a few mistakes but I'm happy with it so far. The image fragments were not preprocessed. I;m getting worse results with preprocessing but maybe because the pipeline is too aggressive where the GLM-OCR is pretty sensitive. I'll experiment with that a bit more. For reasons unknown the script won't capture row number 5.
\nThe good new is, I partially wrote/vibed a fragmentation script, which cuts the original image into small chunks. This makes it possible to run on T4 machine and processing fragments from page 73 took around 2 minutes. I'm hopeful that this will enable more people to join the hunt. I'll do some cleaning and share the working example when it's done.
", "rawMarkdown": "@mpware I come bearing several good and a single bad news.\n\nThe bad news is that I fail to reproduce the text in its intact form. The parser fails to account for emphasized letters (ṣ and ṭ) and sometimes even ù. I'll have to check if it was `unsloth/GLM-OCR` or even the original `zai-org/GLM-OCR`, that was trained on several languages, all of which do not use these letters. If so, we may have to resort to fine-tuning.\n\nThe good news is, plenty, so lets start with the biggest surprise. The end result is pretty good. For `16. CCT 3,2b` it correctly captured all subcripts and superscripts, albeit sometimes in html or tags and sometimes in LaTeX $$ formula notation. This can be dealt with.\n\n\nThere still are a few mistakes but I'm happy with it so far. The image fragments were not preprocessed. I;m getting worse results with preprocessing but maybe because the pipeline is too aggressive where the GLM-OCR is pretty sensitive. I'll experiment with that a bit more. For reasons unknown the script won't capture row number 5.\n\nThe good new is, I partially wrote/vibed a fragmentation script, which cuts the original image into small chunks. This makes it possible to run on T4 machine and processing fragments from page 73 took around 2 minutes. I'm hopeful that this will enable more people to join the hunt. I'll do some cleaning and share the working example when it's done.", "votes": 1 }, { "id": 3404079, "postDate": "2026-02-09T21:11:34.517Z", "content": "I've completed the RE-OCR, that's not perfect. I had to clean-up and re-introduce the {d} and {ki}. I've also re-aligned for byT5 and this is what I'm getting:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Ff3f6ddc67feeba53444cb9e49020237f%2Fre-orc.png?generation=1770671415316918&alt=media\")
I need to complete some experiments but currently the best single model with this RE-OCR is CV=51 and LB=35.6
", "rawMarkdown": "I've completed the RE-OCR, that's not perfect. I had to clean-up and re-introduce the {d} and {ki}. I've also re-aligned for byT5 and this is what I'm getting:\n\n\nI need to complete some experiments but currently the best single model with this RE-OCR is CV=51 and LB=35.6", "votes": 1 }, { "id": 3404301, "postDate": "2026-02-10T08:39:39.797Z", "content": "Looks good. What was your CV/LB prior to the inclusion of Larsen data?
\nDoes \"sentence aligned\" equal \"line aligned\"?
", "rawMarkdown": "Looks good. What was your CV/LB prior to the inclusion of Larsen data?\n\nDoes \"sentence aligned\" equal \"line aligned\"?" } ] } ] }, { "id": 3398026, "postDate": "2026-01-28T13:17:40.093Z", "content": "I am trying to work with the:
\nLarsen 2010 - The Archive of the Šalim-Aššur Family, Vol. 1.
\nHowever, it is extremely difficult unless I review each translation manually. The transliteration is ok because I can find in the published_texts.csv.
\nWe could make a team, I could clean the translation manually and you (anyone) do the model😀!
", "rawMarkdown": "I am trying to work with the:\n\nLarsen 2010 - The Archive of the Šalim-Aššur Family, Vol. 1. \n\nHowever, it is extremely difficult unless I review each translation manually. The transliteration is ok because I can find in the published_texts.csv.\n\nWe could make a team, I could clean the translation manually and you (anyone) do the model😀!", "votes": 1, "replies": [ { "id": 3398828, "postDate": "2026-01-29T18:47:45.500Z", "content": "Where is this one? Any link?
", "rawMarkdown": "Where is this one? Any link?", "replies": [ { "id": 3398918, "postDate": "2026-01-29T22:35:07.977Z", "content": "https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf/data?select=AKT+6a.pdf
", "rawMarkdown": "https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf/data?select=AKT+6a.pdf", "votes": 1 } ] } ] }, { "id": 3396901, "postDate": "2026-01-26T05:45:32.290Z", "content": "Hi @mpware \nCan you drop in the url or file of the pdf? I can't find a way to open it.
", "rawMarkdown": "Hi @mpware \nCan you drop in the url or file of the pdf? I can't find a way to open it.", "votes": -1, "replies": [ { "id": 3396962, "postDate": "2026-01-26T07:54:04.410Z", "content": "", "rawMarkdown": "https://www.kaggle.com/datasets/deeppast/old-assyrian-grammars-and-other-resources/data?select=Larsen+2002+-+The+Assur-nada+Archive.+PIHANS+96+2002.pdf", "replies": [ { "id": 3397153, "postDate": "2026-01-26T15:24:35.520Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3404340, "postDate": "2026-02-10T10:46:42.003Z", "content": "I just published the notebook
\nhttps://www.kaggle.com/code/seraquevence/dpc-increase-the-train-data-v02
\nI hope it helps in yours strategies. Regards.
", "rawMarkdown": "I just published the notebook\n\nhttps://www.kaggle.com/code/seraquevence/dpc-increase-the-train-data-v02\n\nI hope it helps in yours strategies. Regards." }, { "id": 3397624, "postDate": "2026-01-27T17:31:46.350Z", "content": "Extraction is difficult but feasible. Diacritics and ˹ ˺ are my main pain points.. the extractions will definitely require some cleaning but should be pretty helpful once usable.
", "rawMarkdown": "Extraction is difficult but feasible. Diacritics and ˹ ˺ are my main pain points.. the extractions will definitely require some cleaning but should be pretty helpful once usable. ", "replies": [ { "id": 3406021, "postDate": "2026-02-14T10:57:11.067Z", "content": "those were supposed to be removed before training if I'm not wrong, so whats causing the pain?
", "rawMarkdown": "those were supposed to be removed before training if I'm not wrong, so whats causing the pain?\n" } ] }, { "id": 3396794, "postDate": "2026-01-25T21:24:24.113Z", "content": "I was convinced that people just lost interest.
\nI'm yet to start the fine-tuning -- still patching the parser for the transliterations, mainly from train.csv.
Great work, and thanks for keeping things rolling! Note that all the PDFs that have been provided so far already have transliterations in the publications.csv - what is missing are the translations for these texts. So you don't need to redo all the transliteration text, you just need to get the corresponding translations.
\nYou're right to identify that the alignment is an issue, which is why looking at the line numbers as they correspond to the translations are somewhat helpful, but also not exactly in alignment. For example the last word of the previous sentence was \"ha-am-ša-tim\" and the first word of the next sentence began \"šu-ma\".
\nUsing the dictionary for lemmatizing should be helpful if you're matching on the sense of the Akkadian with the English translations, although this type of lemmatization is very advanced work.
", "votes": 3, "replies": [ { "id": 3399879, "author_name": "MPWARE", "author_url": "", "post_date": "2026-01-31T12:50:10.847000", "content": "@deeppast Do you think transliteration in published.csv are safe? IMO it's difficult to be used due to what I've explained in my post, parsed PDF does not reflect the printed transliteration by design. PDF parser cannot be used safely for \"Larsen 2002 - The Assur-nada Archive. PIHANS 96 2002.pdf\":
\nPage 54, line 12:
\n\n\nLook at the garbage transliteration :\"TOI\"a-ni-a-tim ù 15 TÙI0JI01A instead of 8 TÚG a-ni-ú-tim ù 15 TÚG.HI.A
\n
published.csv:
\n
Printed text:
\n
This is what I get by another method:\n
Any thoughts?
\nIf the hidden set contains such garbage then just let us know because I should not spend more time to capture real transliteration/english but just garbage transliteration/english pairs.
", "votes": 1, "replies": [ { "id": 3402372, "author_name": "MPWARE", "author_url": "", "post_date": "2026-02-05T22:04:22.037000", "content": "Re-OCR sounds a good way to get clean transliteration.\nI've tried a few OCR models, DeepSeek-OCR-2 is not bad but far from the majors.
", "votes": 1, "replies": [] }, { "id": 3402474, "author_name": "Angantyr", "author_url": "", "post_date": "2026-02-06T06:13:27.623000", "content": "Have you extracted the image layer from the PDF prior to the OCR? I found that most of the text in the text layer is likely already an OCR attempt, sometimes broken so any text-type parser will work with dummy data underneath.
\nI tried DeepSeek-OCR-2 as well and can confirm it was ok, but failed to capture some letters. The determinatives will be a problem too.
", "votes": 1, "replies": [] }, { "id": 3402797, "author_name": "MPWARE", "author_url": "", "post_date": "2026-02-06T21:11:12.810000", "content": "\nYes, I've tried both:
\nOption #2 is worse than Option #1 for me with DeepSeek-OCR-2. I've also tried different kind of image preprocessing.
\nHere is an example for 16cct32b for option #1:
\n
In fact English is quite perfect but Akkadian is 5-10% bad. I get better results with GLM.
\nSame example with GLM, better but not good.\n
Same example with the majors multi-modals, not perfect but much better:\n
What do you have for letter 16 CCT 3, 2b page 73? Is it better than my experiments?
", "votes": 1, "replies": [] }, { "id": 3402958, "author_name": "Angantyr", "author_url": "", "post_date": "2026-02-07T08:53:46.750000", "content": "I get nothing close to the GLM output, which looks almost perfect -- the sub/superscript parsing is pretty neat. I tried several lighter models with poor results. Gemma-3 much heavier than Deepseek-OCR-2 performed worse.
\nAll of the above needed Google Colab's A100 to work in real time and it's too costly for me. I'll give Unsloth-GLM a try, probably with some image preprocessing to ease capture of emphasized consonants.
\nThank you for the inspiration and insights. It makes the acquisition and cleaning much less frustrating.
\nI'll share my results after the tests, regardless of the quality.
", "votes": 1, "replies": [] }, { "id": 3403975, "author_name": "Angantyr", "author_url": "", "post_date": "2026-02-09T16:53:06.947000", "content": "@mpware I come bearing several good and a single bad news.
\nThe bad news is that I fail to reproduce the text in its intact form. The parser fails to account for emphasized letters (ṣ and ṭ) and sometimes even ù. I'll have to check if it was unsloth/GLM-OCR or even the original zai-org/GLM-OCR, that was trained on several languages, all of which do not use these letters. If so, we may have to resort to fine-tuning.
The good news is, plenty, so lets start with the biggest surprise. The end result is pretty good. For 16. CCT 3,2b it correctly captured all subcripts and superscripts, albeit sometimes in html
There still are a few mistakes but I'm happy with it so far. The image fragments were not preprocessed. I;m getting worse results with preprocessing but maybe because the pipeline is too aggressive where the GLM-OCR is pretty sensitive. I'll experiment with that a bit more. For reasons unknown the script won't capture row number 5.
\nThe good new is, I partially wrote/vibed a fragmentation script, which cuts the original image into small chunks. This makes it possible to run on T4 machine and processing fragments from page 73 took around 2 minutes. I'm hopeful that this will enable more people to join the hunt. I'll do some cleaning and share the working example when it's done.
", "votes": 1, "replies": [] }, { "id": 3404079, "author_name": "MPWARE", "author_url": "", "post_date": "2026-02-09T21:11:34.517000", "content": "I've completed the RE-OCR, that's not perfect. I had to clean-up and re-introduce the {d} and {ki}. I've also re-aligned for byT5 and this is what I'm getting:\n
I need to complete some experiments but currently the best single model with this RE-OCR is CV=51 and LB=35.6
", "votes": 1, "replies": [] }, { "id": 3404301, "author_name": "Angantyr", "author_url": "", "post_date": "2026-02-10T08:39:39.797000", "content": "Looks good. What was your CV/LB prior to the inclusion of Larsen data?
\nDoes \"sentence aligned\" equal \"line aligned\"?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3398026, "author_name": "FML", "author_url": "", "post_date": "2026-01-28T13:17:40.093000", "content": "I am trying to work with the:
\nLarsen 2010 - The Archive of the Šalim-Aššur Family, Vol. 1.
\nHowever, it is extremely difficult unless I review each translation manually. The transliteration is ok because I can find in the published_texts.csv.
\nWe could make a team, I could clean the translation manually and you (anyone) do the model😀!
", "votes": 1, "replies": [ { "id": 3398828, "author_name": "MPWARE", "author_url": "", "post_date": "2026-01-29T18:47:45.500000", "content": "Where is this one? Any link?
", "votes": 0, "replies": [ { "id": 3398918, "author_name": "FML", "author_url": "", "post_date": "2026-01-29T22:35:07.977000", "content": "https://www.kaggle.com/datasets/deeppast/old-assyrian-kltepe-tablets-in-pdf/data?select=AKT+6a.pdf
", "votes": 1, "replies": [] } ] } ] }, { "id": 3396901, "author_name": "", "author_url": "", "post_date": "2026-01-26T05:45:32.290000", "content": "Hi @mpware \nCan you drop in the url or file of the pdf? I can't find a way to open it.
", "votes": -1, "replies": [ { "id": 3396962, "author_name": "MPWARE", "author_url": "", "post_date": "2026-01-26T07:54:04.410000", "content": "", "votes": 0, "replies": [ { "id": 3397153, "author_name": "", "author_url": "", "post_date": "2026-01-26T15:24:35.520000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3404340, "author_name": "FML", "author_url": "", "post_date": "2026-02-10T10:46:42.003000", "content": "I just published the notebook
\nhttps://www.kaggle.com/code/seraquevence/dpc-increase-the-train-data-v02
\nI hope it helps in yours strategies. Regards.
", "votes": 0, "replies": [] }, { "id": 3397624, "author_name": "Jack", "author_url": "", "post_date": "2026-01-27T17:31:46.350000", "content": "Extraction is difficult but feasible. Diacritics and ˹ ˺ are my main pain points.. the extractions will definitely require some cleaning but should be pretty helpful once usable.
", "votes": 0, "replies": [ { "id": 3406021, "author_name": "9Hash", "author_url": "", "post_date": "2026-02-14T10:57:11.067000", "content": "those were supposed to be removed before training if I'm not wrong, so whats causing the pain?
", "votes": 0, "replies": [] } ] }, { "id": 3396794, "author_name": "Angantyr", "author_url": "", "post_date": "2026-01-25T21:24:24.113000", "content": "I was convinced that people just lost interest.
\nI'm yet to start the fine-tuning -- still patching the parser for the transliterations, mainly from train.csv.