Hi everyone,\nI've been trying to use the \"OA_Lexicon_eBL.csv\" lexicon for post-processing to verify personal names in translations. However, I noticed that the norm field in the lexicon often differs from the actual spelling used in the ground truth translations.
\nExamples:\nSource (transliteration) Lexicon norm Ground Truth spelling
\nStatistics:
\nQuestions:
\nHi everyone,\nI've been trying to use the \"OA_Lexicon_eBL.csv\" lexicon for post-processing to verify personal names in translations. However, I noticed that the norm field in the lexicon often differs from the actual spelling used in the ground truth translations.
\nExamples:\nSource (transliteration) Lexicon norm Ground Truth spelling
\nStatistics:
\nQuestions:
\nDo you mean the differences between normalized words and the translations? From what I understand, normalization process gives you the reconstructed pronounciation in Old Assyrian, while the translation gives you the translated version, expected in the `translation' field.
\nSeveral transliterations can result in the same normalization, though there shouldn't be normalizations resulting in different translations.
", "rawMarkdown": "Do you mean the differences between normalized words and the translations? From what I understand, normalization process gives you the reconstructed pronounciation in Old Assyrian, while the translation gives you the translated version, expected in the `translation' field.\n\nSeveral transliterations can result in the same normalization, though there shouldn't be normalizations resulting in different translations.", "votes": null }, { "id": "3382925", "postDate": "12/28/2025 20:27:06", "content": "Yes, Angantyr is correct, and your questions are worth addressing one by one:
\n(1) Spelling variation exists at both the transliteration and normalization (unfortunately). For example we find: \nAhu-waqar = A-hu-wa-qar & A-hu-qar\nAhuwaqar = A-hu-wa-qar & A-hu-qar
\nThis discrepency is because there are many scholars all over the world doing their best to normalize the names on these texts, but there are not yet firmly established conventions for every name encountered on the texts. I realize this is a big issue, since most of the OOV (out of vocabulary) words are going to be proper nouns / names. So I'm working on getting a list of spellings and normalizations together for the supplemental data.
\n(2) Most of the names are provided in the OA_Lexicon_eBL.csv and given a \"PN\" type (instead of \"Word\"). And I will share a more robust onomasticon (list of names) shortly.
\n(3) All the resources we provided are what is currently available in digital format. There are lists of personal names in book publications in the index of the major text editions, and specialist studies (e.g. https://archive.org/details/veenhof-1972-assyrian-trade/page/n515/mode/2up). But these are rarely accessible online as far as I'm aware.
", "rawMarkdown": "Yes, Angantyr is correct, and your questions are worth addressing one by one:\n\n(1) Spelling variation exists at both the transliteration and normalization (unfortunately). For example we find: \nAhu-waqar = A-hu-wa-qar & A-hu-qar\nAhuwaqar = A-hu-wa-qar & A-hu-qar\n\nThis discrepency is because there are many scholars all over the world doing their best to normalize the names on these texts, but there are not yet firmly established conventions for every name encountered on the texts. I realize this is a big issue, since most of the OOV (out of vocabulary) words are going to be proper nouns / names. So I'm working on getting a list of spellings and normalizations together for the supplemental data.\n\n(2) Most of the names are provided in the OA_Lexicon_eBL.csv and given a \"PN\" type (instead of \"Word\"). And I will share a more robust onomasticon (list of names) shortly.\n\n(3) All the resources we provided are what is currently available in digital format. There are lists of personal names in book publications in the index of the major text editions, and specialist studies (e.g. https://archive.org/details/veenhof-1972-assyrian-trade/page/n515/mode/2up). But these are rarely accessible online as far as I'm aware.", "votes": null }, { "id": "3382997", "postDate": "12/29/2025 04:34:30", "content": "OK. Thank you for your reply.
", "rawMarkdown": "OK. Thank you for your reply.", "votes": null } ], "comments": [ { "id": 3382863, "author_name": "angantyr", "author_url": "", "post_date": "12/28/2025 18:24:46", "content": "Do you mean the differences between normalized words and the translations? From what I understand, normalization process gives you the reconstructed pronounciation in Old Assyrian, while the translation gives you the translated version, expected in the `translation' field.
\nSeveral transliterations can result in the same normalization, though there shouldn't be normalizations resulting in different translations.
", "votes": null, "replies": [] }, { "id": 3382925, "author_name": "deeppast", "author_url": "", "post_date": "12/28/2025 20:27:06", "content": "Yes, Angantyr is correct, and your questions are worth addressing one by one:
\n(1) Spelling variation exists at both the transliteration and normalization (unfortunately). For example we find: \nAhu-waqar = A-hu-wa-qar & A-hu-qar\nAhuwaqar = A-hu-wa-qar & A-hu-qar
\nThis discrepency is because there are many scholars all over the world doing their best to normalize the names on these texts, but there are not yet firmly established conventions for every name encountered on the texts. I realize this is a big issue, since most of the OOV (out of vocabulary) words are going to be proper nouns / names. So I'm working on getting a list of spellings and normalizations together for the supplemental data.
\n(2) Most of the names are provided in the OA_Lexicon_eBL.csv and given a \"PN\" type (instead of \"Word\"). And I will share a more robust onomasticon (list of names) shortly.
\n(3) All the resources we provided are what is currently available in digital format. There are lists of personal names in book publications in the index of the major text editions, and specialist studies (e.g. https://archive.org/details/veenhof-1972-assyrian-trade/page/n515/mode/2up). But these are rarely accessible online as far as I'm aware.
", "votes": null, "replies": [ { "id": 3382997, "author_name": "ruichardliu", "author_url": "", "post_date": "12/29/2025 04:34:30", "content": "OK. Thank you for your reply.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3382852": "Hi everyone,\nI've been trying to use the \"OA_Lexicon_eBL.csv\" lexicon for post-processing to verify personal names in translations. However, I noticed that the norm field in the lexicon often differs from the actual spelling used in the ground truth translations.\n\nExamples:\nSource (transliteration) Lexicon norm Ground Truth spelling\n1. šu-ku-bi-im Šu-Kūbim Šu-Kūbum\n2. šu-ta-mu-zi Šu-Tammazi Šu-Tammuzī\n3. e-lá-a Elā(ya) Elaya\n4. a-bi₄-lá Abela Abila\n\nStatistics:\n- Total personal names detected in training data: 11,381\n- Exact match with lexicon norm: 37.2%\n- Fuzzy match (85% similarity): 73.3%\n- Still unmatched after fuzzy matching: 26.7% (~3,041 names)\n\nQuestions:\n1. Is this spelling variation intentional? (e.g., different scholarly conventions)\n2. Is there a recommended way to use the lexicon for name verification/correction?\n3. Are there any additional resources that map between these spelling variants?", "3382863": "Do you mean the differences between normalized words and the translations? From what I understand, normalization process gives you the reconstructed pronounciation in Old Assyrian, while the translation gives you the translated version, expected in the `translation' field.\n\nSeveral transliterations can result in the same normalization, though there shouldn't be normalizations resulting in different translations.", "3382925": "Yes, Angantyr is correct, and your questions are worth addressing one by one:\n\n(1) Spelling variation exists at both the transliteration and normalization (unfortunately). For example we find: \nAhu-waqar = A-hu-wa-qar & A-hu-qar\nAhuwaqar = A-hu-wa-qar & A-hu-qar\n\nThis discrepency is because there are many scholars all over the world doing their best to normalize the names on these texts, but there are not yet firmly established conventions for every name encountered on the texts. I realize this is a big issue, since most of the OOV (out of vocabulary) words are going to be proper nouns / names. So I'm working on getting a list of spellings and normalizations together for the supplemental data.\n\n(2) Most of the names are provided in the OA_Lexicon_eBL.csv and given a \"PN\" type (instead of \"Word\"). And I will share a more robust onomasticon (list of names) shortly.\n\n(3) All the resources we provided are what is currently available in digital format. There are lists of personal names in book publications in the index of the major text editions, and specialist studies (e.g. https://archive.org/details/veenhof-1972-assyrian-trade/page/n515/mode/2up). But these are rarely accessible online as far as I'm aware.", "3382997": "OK. Thank you for your reply." }, "source": "meta" }