Since lots of the vocabs inside the transliterations are PN, GN entities, I tried a few ways to leverage the lexicon
\nNone of them works, and the hallucination does not go away, an example masking translation
\n<PN> <PN> qí-bi-ma um-ma <PN> šál-ma-ku i <GN> wa-áš-ba-ku <PN> <GN> 1 ku-ta-nam i-ṣé-ri-a il₅-qé iš-tù <GN> a-dí <GN> 0.5 <PN> 2 GÍN.TA da-tum <PN> ANŠE ik-šu-ud a-ḫi <PN> we-da-ku mu-ša-ki-il₅ ANŠE ú ša ba-áb-tí ú-šé-ṣa-a-ni ú-lá i-šu 2 ṣú-ḫa-ri ša li-ba-kà i-de₈-ú šu-up-ra-ma iš-tí-in ANŠE.ḪI.A \nmodel inferred
\n'To <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN>\nWonder any one have good ways to use the lexicon and remove the hallucination.
", "messages": [ { "id": 3390369, "postDate": "2026-01-13T03:00:48.723Z", "content": "Since lots of the vocabs inside the transliterations are PN, GN entities, I tried a few ways to leverage the lexicon
\nNone of them works, and the hallucination does not go away, an example masking translation
\n<PN> <PN> qí-bi-ma um-ma <PN> šál-ma-ku i <GN> wa-áš-ba-ku <PN> <GN> 1 ku-ta-nam i-ṣé-ri-a il₅-qé iš-tù <GN> a-dí <GN> 0.5 <PN> 2 GÍN.TA da-tum <PN> ANŠE ik-šu-ud a-ḫi <PN> we-da-ku mu-ša-ki-il₅ ANŠE ú ša ba-áb-tí ú-šé-ṣa-a-ni ú-lá i-šu 2 ṣú-ḫa-ri ša li-ba-kà i-de₈-ú šu-up-ra-ma iš-tí-in ANŠE.ḪI.A \nmodel inferred
\n'To <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN> son of <PN>\nWonder any one have good ways to use the lexicon and remove the hallucination.
", "rawMarkdown": "Since lots of the vocabs inside the transliterations are PN, GN entities, I tried a few ways to leverage the lexicon\n1. Use normalised form in the transliteration, expecting the model to learn that these names are normalised not translated.\n2. UseThis is a very common hallucination. It makes sense when you consider that often these people are mentioned with their patronymics (son of PN) and the occur in multiples, so the most likely next word would be another PN (with their patronymic). To avoid this, you may want to add numbers to the PNs, so you can limit the number of repeats. To PN1, PN2, PN3, from PN4, PN5, PN6. Also the patronymics are rare in the introduction of a letter (mostly used for when there are two people with the same name), whereas they should be expected in lists of witnesses (following the logogram: IGI) at the end of a contract / loan.
\nAnd just for clarification:\nPN = proper noun, but more specifically, person name\nGN = geographic name
\n(see full list of proper noun tags here: https://oracc.museum.upenn.edu/doc/help/languages/propernouns/index.html)
", "rawMarkdown": "This is a very common hallucination. It makes sense when you consider that often these people are mentioned with their patronymics (son of PN) and the occur in multiples, so the most likely next word would be another PN (with their patronymic). To avoid this, you may want to add numbers to the PNs, so you can limit the number of repeats. To PN1, PN2, PN3, from PN4, PN5, PN6. Also the patronymics are rare in the introduction of a letter (mostly used for when there are two people with the same name), whereas they should be expected in lists of witnesses (following the logogram: IGI) at the end of a contract / loan.\n\nAnd just for clarification:\nPN = proper noun, but more specifically, person name\nGN = geographic name\n\n(see full list of proper noun tags here: https://oracc.museum.upenn.edu/doc/help/languages/propernouns/index.html)", "votes": 1 }, { "id": 3390618, "postDate": "2026-01-13T14:31:54.383Z", "content": "The way you could use it depends on the model. In your case, I would say the problem is with the way you tag your forms.
\nI don't know your method specifically, so I tried to parse your example in my mind but it doesn't work, as you have two <PN> tags in a row or <PN> <GN> tags and I'm unsure what that means and I'm sure the model is confused about it too. On top of that, you have the tags around a number which is also clearly wrong. Also note that I found a few examples where a form could correspond to both a PN/GN or to an actual word.
For decoding, consider utilizing beam search and repetition penalty. Or have a look into n-gram size penalty, I forgot the exact argument name in transformers but it basically penalizes repetitions of the style you have. Then again, i don't think it's decoding-only problem in this example.
I think you're talking about this
\n no_repeat_ngram_size=4,\n # repetition_penalty=1.,\n # length_penalty=0.7,\n early_stopping=True,\n # top_k=5, \n # top_p=0.95,\n num_beams=4\nBut overall, I think it's more of my method problem. I skim through all the words in transliteration and match the word with the type in the lexicon data set.
", "rawMarkdown": "I think you're talking about this\n```\n no_repeat_ngram_size=4,\n # repetition_penalty=1.,\n # length_penalty=0.7,\n early_stopping=True,\n # top_k=5, \n # top_p=0.95,\n num_beams=4\n\n```\n\nBut overall, I think it's more of my method problem. I skim through all the words in transliteration and match the word with the type in the lexicon data set.", "replies": [ { "id": 3391856, "postDate": "2026-01-15T20:02:42.417Z", "content": "Yes this can help but if you are seeing such repetitions it means your model is not converging its a sign that its overfitting to shallow heuristics like that \"PN son of PN\" is a common string in the english translations so if it generates this string loss goes down vs generating pure noise and it has over converged on that signal.
", "rawMarkdown": "Yes this can help but if you are seeing such repetitions it means your model is not converging its a sign that its overfitting to shallow heuristics like that \"PN son of PN\" is a common string in the english translations so if it generates this string loss goes down vs generating pure noise and it has over converged on that signal.", "votes": 1 } ] } ] }, { "id": 3391852, "postDate": "2026-01-15T19:59:02.443Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3390644, "author_name": "Adam Anderson", "author_url": "", "post_date": "2026-01-13T15:38:34.873000", "content": "This is a very common hallucination. It makes sense when you consider that often these people are mentioned with their patronymics (son of PN) and the occur in multiples, so the most likely next word would be another PN (with their patronymic). To avoid this, you may want to add numbers to the PNs, so you can limit the number of repeats. To PN1, PN2, PN3, from PN4, PN5, PN6. Also the patronymics are rare in the introduction of a letter (mostly used for when there are two people with the same name), whereas they should be expected in lists of witnesses (following the logogram: IGI) at the end of a contract / loan.
\nAnd just for clarification:\nPN = proper noun, but more specifically, person name\nGN = geographic name
\n(see full list of proper noun tags here: https://oracc.museum.upenn.edu/doc/help/languages/propernouns/index.html)
", "votes": 1, "replies": [] }, { "id": 3390618, "author_name": "Alexandru Craevschi", "author_url": "", "post_date": "2026-01-13T14:31:54.383000", "content": "The way you could use it depends on the model. In your case, I would say the problem is with the way you tag your forms.
\nI don't know your method specifically, so I tried to parse your example in my mind but it doesn't work, as you have two <PN> tags in a row or <PN> <GN> tags and I'm unsure what that means and I'm sure the model is confused about it too. On top of that, you have the tags around a number which is also clearly wrong. Also note that I found a few examples where a form could correspond to both a PN/GN or to an actual word.
For decoding, consider utilizing beam search and repetition penalty. Or have a look into n-gram size penalty, I forgot the exact argument name in transformers but it basically penalizes repetitions of the style you have. Then again, i don't think it's decoding-only problem in this example.
I think you're talking about this
\n no_repeat_ngram_size=4,\n # repetition_penalty=1.,\n # length_penalty=0.7,\n early_stopping=True,\n # top_k=5, \n # top_p=0.95,\n num_beams=4\nBut overall, I think it's more of my method problem. I skim through all the words in transliteration and match the word with the type in the lexicon data set.
", "votes": 0, "replies": [ { "id": 3391856, "author_name": "Aaron Bornstein", "author_url": "", "post_date": "2026-01-15T20:02:42.417000", "content": "Yes this can help but if you are seeing such repetitions it means your model is not converging its a sign that its overfitting to shallow heuristics like that \"PN son of PN\" is a common string in the english translations so if it generates this string loss goes down vs generating pure noise and it has over converged on that signal.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3391852, "author_name": "", "author_url": "", "post_date": "2026-01-15T19:59:02.443000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3390369": "Since lots of the vocabs inside the transliterations are PN, GN entities, I tried a few ways to leverage the lexicon\n1. Use normalised form in the transliteration, expecting the model to learn that these names are normalised not translated.\n2. Use