insights from https://github.com/gaigutherz/Akkademia/blob/master/akkadian/ -> https://claude.ai/chat/ae982bc5-0d2e-41ea-a35c-c5e5f2dfccf3
\nInsights from the Akkademia Codebase & PNAS Paper for the Deep Past Challenge\nHere's what you can extract and apply from the Akkademia project:
\nAkkademia uses HMM, MEMM and BiLSTM neural networks for transliteration, achieving accuracy rates of 89.5% with HMM, 94% with MEMM, and 96.7% with BiLSTM on the trained corpora. GitHub But the translation work (the 2023 PNAS Nexus paper) used something different — a Fairseq convolutional model (fconv) for the NMT task, as CNNs have been shown to work well for low-resource and character-level machine translation, with shorter training time compared with transformers. PubMed Central
\nKey takeaway for you: Their SOTA results used CNNs, not transformers. This is a 2023 result. Modern transformers (especially ByT5, mT5, or NLLB) with proper fine-tuning should be able to beat 37.47 BLEU4. The competition is essentially asking you to outperform this benchmark on a different (Old Assyrian) corpus.
\nThe repo supports two tasks:
\nInterestingly, though they anticipated the results of T2E to be better, there was little to no substantial degradation of the results compared with C2E. PubMed Central This means going directly from cuneiform Unicode to English is viable — you don't necessarily need a two-stage transliteration → translation pipeline. If the competition data includes Unicode cuneiform, consider training C2E directly.
\nFrom the paper and codebase:
\nTakeaway: Apply the same cleaning rigorously to the competition data. Also, a cleaned version of the Akkademia training corpus is available as a HuggingFace dataset in the veezbo/akkadian_english_corpus repo GitHub — use it for supplementary pre-training.
\nFor cuneiform (C2E), they used character-based tokenization with a small vocabulary of 400. For transliteration (T2E), they used BytePair Encoding (BPE) with SentencePiece, vocabulary sizes of 1,000 for transliteration and 10,000 for English. PubMed Central
\nInsight for the competition:
\nThe more formulaic the genre of the source, the more accurate the translation. Administrative and divinatory texts tend to be very formulaic. PubMed Central The competition specifically targets Old Assyrian business records — contracts, letters, loans, receipts — which are highly formulaic.
\nHowever, the great majority of Akkademia's training data is Neo-Assyrian (7,327 out of 8,056 tablets), with only 122 Old Assyrian texts. PubMed Central This is a critical domain mismatch — the model was trained overwhelmingly on Neo-Assyrian data, but the competition wants Old Assyrian.
\nWhat this means for you:
\nThe model produces both intrinsic hallucinations (factual but incorrectly phrased) and extrinsic hallucinations (fabricated content not in the source). PubMed Central From the actual output examples in the repo, we can see cases where the model:
\nTakeaway: Add constrained decoding or length penalties. For proper nouns, consider a copy mechanism or dictionary-based post-processing. For the chrF++ component of your score, getting names character-close matters.
\nTheir tuned hyperparameters: architecture fconv, dropout 0.1, label smoothed cross-entropy criterion, label smoothing 0.1, NAG optimizer, clip-norm 0.1, fixed learning rate scheduler, force-anneal at 50, max-tokens 4000, learning rate 0.1. PubMed Central
\nIf you're using Fairseq as a starting point, these are your baseline settings. But I'd recommend switching to a modern framework (HuggingFace Transformers) and using their seq2seq pipeline with the following models as strong starting points: google/byt5-large, google/mt5-base, or facebook/nllb-200-distilled-600M.
\nThe Akkademia team's CNN-based approach from 2023 is your floor. To beat it on Old Assyrian:
\ninsights from https://github.com/gaigutherz/Akkademia/blob/master/akkadian/ -> https://claude.ai/chat/ae982bc5-0d2e-41ea-a35c-c5e5f2dfccf3
\nInsights from the Akkademia Codebase & PNAS Paper for the Deep Past Challenge\nHere's what you can extract and apply from the Akkademia project:
\nAkkademia uses HMM, MEMM and BiLSTM neural networks for transliteration, achieving accuracy rates of 89.5% with HMM, 94% with MEMM, and 96.7% with BiLSTM on the trained corpora. GitHub But the translation work (the 2023 PNAS Nexus paper) used something different — a Fairseq convolutional model (fconv) for the NMT task, as CNNs have been shown to work well for low-resource and character-level machine translation, with shorter training time compared with transformers. PubMed Central
\nKey takeaway for you: Their SOTA results used CNNs, not transformers. This is a 2023 result. Modern transformers (especially ByT5, mT5, or NLLB) with proper fine-tuning should be able to beat 37.47 BLEU4. The competition is essentially asking you to outperform this benchmark on a different (Old Assyrian) corpus.
\nThe repo supports two tasks:
\nInterestingly, though they anticipated the results of T2E to be better, there was little to no substantial degradation of the results compared with C2E. PubMed Central This means going directly from cuneiform Unicode to English is viable — you don't necessarily need a two-stage transliteration → translation pipeline. If the competition data includes Unicode cuneiform, consider training C2E directly.
\nFrom the paper and codebase:
\nTakeaway: Apply the same cleaning rigorously to the competition data. Also, a cleaned version of the Akkademia training corpus is available as a HuggingFace dataset in the veezbo/akkadian_english_corpus repo GitHub — use it for supplementary pre-training.
\nFor cuneiform (C2E), they used character-based tokenization with a small vocabulary of 400. For transliteration (T2E), they used BytePair Encoding (BPE) with SentencePiece, vocabulary sizes of 1,000 for transliteration and 10,000 for English. PubMed Central
\nInsight for the competition:
\nThe more formulaic the genre of the source, the more accurate the translation. Administrative and divinatory texts tend to be very formulaic. PubMed Central The competition specifically targets Old Assyrian business records — contracts, letters, loans, receipts — which are highly formulaic.
\nHowever, the great majority of Akkademia's training data is Neo-Assyrian (7,327 out of 8,056 tablets), with only 122 Old Assyrian texts. PubMed Central This is a critical domain mismatch — the model was trained overwhelmingly on Neo-Assyrian data, but the competition wants Old Assyrian.
\nWhat this means for you:
\nThe model produces both intrinsic hallucinations (factual but incorrectly phrased) and extrinsic hallucinations (fabricated content not in the source). PubMed Central From the actual output examples in the repo, we can see cases where the model:
\nTakeaway: Add constrained decoding or length penalties. For proper nouns, consider a copy mechanism or dictionary-based post-processing. For the chrF++ component of your score, getting names character-close matters.
\nTheir tuned hyperparameters: architecture fconv, dropout 0.1, label smoothed cross-entropy criterion, label smoothing 0.1, NAG optimizer, clip-norm 0.1, fixed learning rate scheduler, force-anneal at 50, max-tokens 4000, learning rate 0.1. PubMed Central
\nIf you're using Fairseq as a starting point, these are your baseline settings. But I'd recommend switching to a modern framework (HuggingFace Transformers) and using their seq2seq pipeline with the following models as strong starting points: google/byt5-large, google/mt5-base, or facebook/nllb-200-distilled-600M.
\nThe Akkademia team's CNN-based approach from 2023 is your floor. To beat it on Old Assyrian:
\nThank you for Akkademia Codebase @jamesmcguigan
", "rawMarkdown": "Thank you for Akkademia Codebase @jamesmcguigan", "votes": null }, { "id": "3417599", "postDate": "03/05/2026 19:01:02", "content": "Wonderful write up thank you. I wonder how much overlap there is between this competition's data and the HuggingFace veezbo/akkadian_english_corpus data.
", "rawMarkdown": "Wonderful write up thank you. I wonder how much overlap there is between this competition's data and the HuggingFace veezbo/akkadian_english_corpus data.", "votes": null }, { "id": "3417607", "postDate": "03/05/2026 19:13:59", "content": "There is no overlap at all, in terms of text used. The Akkademia dataset used Neo-Assyrian texts, which are 1000 years removed in time from the Old Assyrian texts.
", "rawMarkdown": "There is no overlap at all, in terms of text used. The Akkademia dataset used Neo-Assyrian texts, which are 1000 years removed in time from the Old Assyrian texts.", "votes": null }, { "id": "3417610", "postDate": "03/05/2026 19:16:07", "content": "I will be very interested to see your results of this training process. From a few experiments I've tested, there is not enough overlap between the Neo-Assyrian texts used in the Akkademia workflow, and those used in this competition. Old Assyrian and Neo-Assyrian are separated by about 1000 years.
", "rawMarkdown": "I will be very interested to see your results of this training process. From a few experiments I've tested, there is not enough overlap between the Neo-Assyrian texts used in the Akkademia workflow, and those used in this competition. Old Assyrian and Neo-Assyrian are separated by about 1000 years.", "votes": null }, { "id": "3417793", "postDate": "03/06/2026 08:43:57", "content": "Would this hold for names, too? I foudn some words from train.csv did occur in RINAP project glossaries. So if not as a main source, could they serve as an additional resource of personal names, several words, etc.?
\nAnother question comes to mind. Sumerian was still used in some form long after it died as a spoken language. Do you think Middle-Assyrian and Neo-Assyrian additions are more likely to help or add unnecessary noise to the data? Similr to how words in many other languages shifft meaning over time.
", "rawMarkdown": "Would this hold for names, too? I foudn some words from train.csv did occur in RINAP project glossaries. So if not as a main source, could they serve as an additional resource of personal names, several words, etc.?\n\nAnother question comes to mind. Sumerian was still used in some form long after it died as a spoken language. Do you think Middle-Assyrian and Neo-Assyrian additions are more likely to help or add unnecessary noise to the data? Similr to how words in many other languages shifft meaning over time.", "votes": null } ], "comments": [ { "id": 3407819, "author_name": "navneetbende", "author_url": "", "post_date": "02/19/2026 08:02:21", "content": "Thank you for Akkademia Codebase @jamesmcguigan
", "votes": null, "replies": [] }, { "id": 3417599, "author_name": "shwesh", "author_url": "", "post_date": "03/05/2026 19:01:02", "content": "Wonderful write up thank you. I wonder how much overlap there is between this competition's data and the HuggingFace veezbo/akkadian_english_corpus data.
", "votes": null, "replies": [ { "id": 3417607, "author_name": "deeppast", "author_url": "", "post_date": "03/05/2026 19:13:59", "content": "There is no overlap at all, in terms of text used. The Akkademia dataset used Neo-Assyrian texts, which are 1000 years removed in time from the Old Assyrian texts.
", "votes": null, "replies": [] } ] }, { "id": 3417610, "author_name": "deeppast", "author_url": "", "post_date": "03/05/2026 19:16:07", "content": "I will be very interested to see your results of this training process. From a few experiments I've tested, there is not enough overlap between the Neo-Assyrian texts used in the Akkademia workflow, and those used in this competition. Old Assyrian and Neo-Assyrian are separated by about 1000 years.
", "votes": null, "replies": [ { "id": 3417793, "author_name": "angantyr", "author_url": "", "post_date": "03/06/2026 08:43:57", "content": "Would this hold for names, too? I foudn some words from train.csv did occur in RINAP project glossaries. So if not as a main source, could they serve as an additional resource of personal names, several words, etc.?
\nAnother question comes to mind. Sumerian was still used in some form long after it died as a spoken language. Do you think Middle-Assyrian and Neo-Assyrian additions are more likely to help or add unnecessary noise to the data? Similr to how words in many other languages shifft meaning over time.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3407195": "insights from https://github.com/gaigutherz/Akkademia/blob/master/akkadian/ -> https://claude.ai/chat/ae982bc5-0d2e-41ea-a35c-c5e5f2dfccf3\n\nInsights from the Akkademia Codebase & PNAS Paper for the Deep Past Challenge\nHere's what you can extract and apply from the Akkademia project:\n\n# 1. Architecture & Models\n\nAkkademia uses HMM, MEMM and BiLSTM neural networks for transliteration, achieving accuracy rates of 89.5% with HMM, 94% with MEMM, and 96.7% with BiLSTM on the trained corpora. GitHub But the translation work (the 2023 PNAS Nexus paper) used something different — a Fairseq convolutional model (fconv) for the NMT task, as CNNs have been shown to work well for low-resource and character-level machine translation, with shorter training time compared with transformers. PubMed Central\n\nKey takeaway for you: Their SOTA results used CNNs, not transformers. This is a 2023 result. Modern transformers (especially ByT5, mT5, or NLLB) with proper fine-tuning should be able to beat 37.47 BLEU4. The competition is essentially asking you to outperform this benchmark on a different (Old Assyrian) corpus.\n\n# 2. Two Translation Pipelines — and Why C2E ≈ T2E\n\nThe repo supports two tasks:\n\n- C2E (Cuneiform Unicode → English): Character-based tokenization with a small vocabulary of ~400 characters\n- T2E (Transliteration → English): BPE tokenization via SentencePiece — 1,000 vocab for transliteration, 10,000 for English\n\nInterestingly, though they anticipated the results of T2E to be better, there was little to no substantial degradation of the results compared with C2E. PubMed Central This means going directly from cuneiform Unicode to English is viable — you don't necessarily need a two-stage transliteration → translation pipeline. If the competition data includes Unicode cuneiform, consider training C2E directly.\n\n# 3. Data Pipeline & Preprocessing (Critical for the Competition)\nFrom the paper and codebase:\n\n- Training data: They used ORACC corpora — RINAP, RIAo, RIBo, SAAo, and Suhu — totaling 8,056 tablets and ~56,160 sentences for T2E. PubMed Central\n- Data split: 90% train / 5% validation / 5% test\n- Sentence definition: They treat each line as an independent \"sentence\" — even if it's a single word, phrase, or group of phrases. The shortest sentences are 3 characters, the longest 237, with a median of 118 characters. \nPubMed Central\n- Cleaning steps applied:\n - Verified no text appears more than once (deduplication)\n - Removed examples without proper translation (flagged \"No translation possible,\" \"No translation warranted,\" \"broken for translation\")\n - Kept examples with restorations (partially reconstructed text)\n - Handled alignment mismatches between Akkadian lines and English translations\n\n\nTakeaway: Apply the same cleaning rigorously to the competition data. Also, a cleaned version of the Akkademia training corpus is available as a HuggingFace dataset in the veezbo/akkadian_english_corpus repo GitHub — use it for supplementary pre-training.\n\n# 4. Tokenization Strategy Matters Enormously\nFor cuneiform (C2E), they used character-based tokenization with a small vocabulary of 400. For transliteration (T2E), they used BytePair Encoding (BPE) with SentencePiece, vocabulary sizes of 1,000 for transliteration and 10,000 for English. PubMed Central\n\nInsight for the competition:\n\n- The tiny source vocabulary (400 for cuneiform, 1,000 for transliteration) suggests that byte-level or character-level models (ByT5) are a natural fit, since they inherently handle small-vocabulary scripts well.\n- The 10:1 ratio of English-to-source vocab size reflects the asymmetry — Akkadian has a compact sign system while English output needs rich vocabulary. Consider asymmetric architectures or tuning BPE sizes differently for source vs. target.\n\n# 5. Genre Sensitivity & the Old Assyrian Angle\nThe more formulaic the genre of the source, the more accurate the translation. Administrative and divinatory texts tend to be very formulaic. PubMed Central The competition specifically targets Old Assyrian business records — contracts, letters, loans, receipts — which are highly formulaic.\n\nHowever, the great majority of Akkademia's training data is Neo-Assyrian (7,327 out of 8,056 tablets), with only 122 Old Assyrian texts. PubMed Central This is a critical domain mismatch — the model was trained overwhelmingly on Neo-Assyrian data, but the competition wants Old Assyrian.\n\nWhat this means for you:\n\n- Pre-training on Akkademia data gives you Akkadian language structure, but you need to fine-tune aggressively on the competition's Old Assyrian corpus\n- Old Assyrian has distinct orthographic conventions and vocabulary — expect significant distribution shift\n- The formulaic nature of business records is your advantage: learn the templates (\"X minas of silver,\" \"sealed by Y,\" \"in the presence of Z\")\n\n# 6. The Hallucination Problem\nThe model produces both intrinsic hallucinations (factual but incorrectly phrased) and extrinsic hallucinations (fabricated content not in the source). PubMed Central From the actual output examples in the repo, we can see cases where the model:\n\n- Substitutes deity names (e.g., \"Nikkal\" for \"Ninegal\")\n- Fabricates content when input is fragmentary (fills gaps with plausible-sounding but wrong text)\n- Truncates long translations\n\nTakeaway: Add constrained decoding or length penalties. For proper nouns, consider a copy mechanism or dictionary-based post-processing. For the chrF++ component of your score, getting names character-close matters.\n\n# 7. Key Hyperparameters from Their Best Run\n\nTheir tuned hyperparameters: architecture fconv, dropout 0.1, label smoothed cross-entropy criterion, label smoothing 0.1, NAG optimizer, clip-norm 0.1, fixed learning rate scheduler, force-anneal at 50, max-tokens 4000, learning rate 0.1. PubMed Central\n\nIf you're using Fairseq as a starting point, these are your baseline settings. But I'd recommend switching to a modern framework (HuggingFace Transformers) and using their seq2seq pipeline with the following models as strong starting points: google/byt5-large, google/mt5-base, or facebook/nllb-200-distilled-600M.\n8. Supplementary Data Sources to Mine\nFrom the paper and related repos:\n\n- ORACC (oracc.org): RINAP, RIAo, RIBo, SAAo, Suhu — all have parallel transliteration + English\n- veezbo/akkadian_english_corpus: Pre-cleaned Akkademia data on HuggingFace\n- Electronic Babylonian Library (eBL): Additional cuneiform data\n- Cuneiform Digital Library Initiative (CDLI): cdli.ucla.edu — metadata and some translations\n- cuneiform_to_unicode_fixed.csv in the Akkademia repo — mapping table for cuneiform sign → Unicode, useful for any preprocessing pipeline\n\n# Bottom Line: Competitive Strategy\n\nThe Akkademia team's CNN-based approach from 2023 is your floor. To beat it on Old Assyrian:\n\n- Pre-train on the full Akkademia/ORACC corpus (mostly Neo-Assyrian, but it teaches Akkadian structure)\n- Fine-tune on the competition's Old Assyrian business records\n- Use a modern architecture — ByT5-large or mT5 will likely outperform fconv\n- Exploit formulaic patterns — Old Assyrian business records are highly templated\n- Post-process — dictionary lookup for proper nouns, length normalization, consistent punctuation to maximize both BLEU and chrF++", "3407819": "Thank you for Akkademia Codebase @jamesmcguigan", "3417599": "Wonderful write up thank you. I wonder how much overlap there is between this competition's data and the HuggingFace veezbo/akkadian_english_corpus data.", "3417607": "There is no overlap at all, in terms of text used. The Akkademia dataset used Neo-Assyrian texts, which are 1000 years removed in time from the Old Assyrian texts.", "3417610": "I will be very interested to see your results of this training process. From a few experiments I've tested, there is not enough overlap between the Neo-Assyrian texts used in the Akkademia workflow, and those used in this competition. Old Assyrian and Neo-Assyrian are separated by about 1000 years.", "3417793": "Would this hold for names, too? I foudn some words from train.csv did occur in RINAP project glossaries. So if not as a main source, could they serve as an additional resource of personal names, several words, etc.?\n\nAnother question comes to mind. Sumerian was still used in some form long after it died as a spoken language. Do you think Middle-Assyrian and Neo-Assyrian additions are more likely to help or add unnecessary noise to the data? Similr to how words in many other languages shifft meaning over time." }, "source": "meta" }