This was an interesting one. Thanks to the organizers for putting together this competition. The model I submitted was trained in two stages: continued pre-training (CPT) on a large domain corpus, followed directly by group relative policy optimization (GRPO). No supervised fine-tuning (SFT) step was used.
\nTraining on train.csv directly gave mixed results due to data quality issues, so most of the work went into building a cleaner, larger corpus from external sources instead. The training corpus was assembled from several publicly available Old Assyrian and Akkadian sources:
And pretty much every other publicly available Old Assyrian/Akkadian corpus I could find. Data collection alone took up roughly the first two months of the competition. The raw sources contained Old Assyrian sentences translated into some combination of English, French, Turkish, and German; DeepSeek was used to fill in the missing translations for each sentence, bringing the total to ~1 million sentence pairs after augmentation. About half of those pairs were Old Assyrian; the remainder covered closely related Akkadian dialects.
\nSome work was done by hand, but after that DeepSeek and Gemini were used to:
\nDeepSeek was also used to modernize some older translations to better match the style found in train.csv, though it's unclear whether this helped or hurt — the test set style was an unknown.
Rule-based preprocessing was applied to both source (transliteration) and target (translation) sides:
\n<gap> markers for damaged or missing textThe base Qwen3-8B model was adapted to the Old Assyrian domain via causal language modeling on the full cleaned corpus (~250M tokens) that I had assembled.
\nGRPO was applied directly on the CPT checkpoint, skipping SFT entirely. The full dataset was filtered using a combination of heuristics (such as relative sentence length ratios) and LLM-as-judge quality scores, then deduplicated using MinHash, and only the ~30k highest-quality Old Assyrian–English sentence pairs were retained for this step. The primary reward signal was the competition metric: geometric mean of sentence-level BLEU and chrF++. Additional penalties were added for length divergence and poor whitespace ratios to keep outputs well-formed.
\nTraining went surprisingly smoothly. The model learned to use its stop token and started producing reasonably formatted translations after ~70 steps. Metrics plateaued around step 1500, though I let it run to 3000.
\nCandidates were generated with vLLM. Final selection used length-normalized average log-probability — picking whichever candidate the model was most confident in.
\nPost-processing included forbidden character removal, name normalization to match source-side canonical forms, and number consistency checks. These steps were mostly put in place for earlier models that needed them more — the final model rarely triggered them — but were kept as a safeguard.
\numma X-ma ana Y qibīma to seed the decoder with From X to Y:), and a statistical method that mined the training data for recurring source prefix → target prefix correspondences (longest match first, filtered by frequency and consistency). Both showed local promise, but inference crashes on Kaggle ruled it out of the final submission.This was an interesting one. Thanks to the organizers for putting together this competition. The model I submitted was trained in two stages: continued pre-training (CPT) on a large domain corpus, followed directly by group relative policy optimization (GRPO). No supervised fine-tuning (SFT) step was used.
\nTraining on train.csv directly gave mixed results due to data quality issues, so most of the work went into building a cleaner, larger corpus from external sources instead. The training corpus was assembled from several publicly available Old Assyrian and Akkadian sources:
And pretty much every other publicly available Old Assyrian/Akkadian corpus I could find. Data collection alone took up roughly the first two months of the competition. The raw sources contained Old Assyrian sentences translated into some combination of English, French, Turkish, and German; DeepSeek was used to fill in the missing translations for each sentence, bringing the total to ~1 million sentence pairs after augmentation. About half of those pairs were Old Assyrian; the remainder covered closely related Akkadian dialects.
\nSome work was done by hand, but after that DeepSeek and Gemini were used to:
\nDeepSeek was also used to modernize some older translations to better match the style found in train.csv, though it's unclear whether this helped or hurt — the test set style was an unknown.
Rule-based preprocessing was applied to both source (transliteration) and target (translation) sides:
\n<gap> markers for damaged or missing textThe base Qwen3-8B model was adapted to the Old Assyrian domain via causal language modeling on the full cleaned corpus (~250M tokens) that I had assembled.
\nGRPO was applied directly on the CPT checkpoint, skipping SFT entirely. The full dataset was filtered using a combination of heuristics (such as relative sentence length ratios) and LLM-as-judge quality scores, then deduplicated using MinHash, and only the ~30k highest-quality Old Assyrian–English sentence pairs were retained for this step. The primary reward signal was the competition metric: geometric mean of sentence-level BLEU and chrF++. Additional penalties were added for length divergence and poor whitespace ratios to keep outputs well-formed.
\nTraining went surprisingly smoothly. The model learned to use its stop token and started producing reasonably formatted translations after ~70 steps. Metrics plateaued around step 1500, though I let it run to 3000.
\nCandidates were generated with vLLM. Final selection used length-normalized average log-probability — picking whichever candidate the model was most confident in.
\nPost-processing included forbidden character removal, name normalization to match source-side canonical forms, and number consistency checks. These steps were mostly put in place for earlier models that needed them more — the final model rarely triggered them — but were kept as a safeguard.
\numma X-ma ana Y qibīma to seed the decoder with From X to Y:), and a statistical method that mined the training data for recurring source prefix → target prefix correspondences (longest match first, filtered by frequency and consistency). Both showed local promise, but inference crashes on Kaggle ruled it out of the final submission.Thank you for your solution. May I ask, after step 1 (CPT), how will you merge the LoRa into the base model to obtain the CPT model?
", "rawMarkdown": "Thank you for your solution. May I ask, after step 1 (CPT), how will you merge the LoRa into the base model to obtain the CPT model?", "votes": null }, { "id": "3427713", "postDate": "03/24/2026 12:53:22", "content": "I had some issues merging at that step. In the end here is what worked:
\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n model_name = adapter_path,\n max_seq_length = 2048,\n dtype = torch.bfloat16,\n load_in_4bit = False,\n device_map = \"cpu\",\n)\n\nmodel = model.merge_and_unload()\nmodel.config.tie_word_embeddings = False\n\nmodel.save_pretrained(save_path)\ntokenizer.save_pretrained(save_path)\nThank you for your solution. May I ask, after step 1 (CPT), how will you merge the LoRa into the base model to obtain the CPT model?
", "votes": null, "replies": [ { "id": 3427713, "author_name": "epiktistes", "author_url": "", "post_date": "03/24/2026 12:53:22", "content": "I had some issues merging at that step. In the end here is what worked:
\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n model_name = adapter_path,\n max_seq_length = 2048,\n dtype = torch.bfloat16,\n load_in_4bit = False,\n device_map = \"cpu\",\n)\n\nmodel = model.merge_and_unload()\nmodel.config.tie_word_embeddings = False\n\nmodel.save_pretrained(save_path)\ntokenizer.save_pretrained(save_path)\n