![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9464013%2F4a9d01f5e770301882b404b8abc1b056%2F.PNG?generation=1774332818086912&alt=media\")
Hi Kaggle
\nThanks so much for the host and I am very delighted to join the competition. We are the lucky team and did not shake down.\nBig Thanks my teammates zxc and Baiph, we work very hard and share the equal credits to the final results
\nFirst of all, I did not think the quality of training data is not as good as expected. We need lots of work to clean and format training data. In addition, we need to extract or pseudo-label from publication.csv or published_texts.csv. The test and train data varies lots and cv is not stable
\nThe test data is only 4000 sentences we can apply online leanring training strategy to improve the model performance further.
\nAs for data, I used Gemini 3.1 with web tool to generate external data.
\nAs for model selection, we used LLM model as the backbone we tried qwne3,qwen2.5, phi-4, byt5
\nAs for training strategy, total pipeline can be divided into three stages, SFT (knowledge distial )-> RL (reward hacking ) -> Online Learning (fit test distribution), we found that online learning only worked if model archs are very different.
\nI used Gemini 3.1 flash to extract all the sentence level transliteration-translation pairs. The instructions are shown as following
\nprompt = f\"\"\"\n Help me extract Old Assyrian transliteration and its translation\n\n Rules:\n 1. Output a valid JSON strings.\n 2. Only complete sentences need be extracted\n 3. Break both the Akkadian transliteration and the matching English translation into sentences and align them pairwise.\n 4. clean OCR mistakes in sentence_transliteration and sentence_translation\n 5. remove unnecessary characters sentence_transliteration and sentence_translation\n\nInput:\n{page_text}\n\nThe output should only be in json format like the below\n{output_example}\"\"\"\nI did use the above data as sft data, but Qwen got lots of Hallucination issues. Maybe it need to be further clean or formating
\nI used above data from Step1 to build the doc database, and use Qwen3 8B Embedding to retrieval top 3 samples and concat them togther to generate more relable translation .
\n
After that I used it to SFT data, Qwen3 14B model got private 37.9 and public 36.1
\nI used the same apporach RAG to clean train.csv, but I filter out the bad data using the following prompt, only bad data need to be regenerated.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9464013%2F7d6e379d19a5a6e828285be7763ba64b%2F.PNG?generation=1774333256653579&alt=media\")
After that I used it to RL data, Qwen3 14B model got private 38.1 and public 36.9
\nMy teammates think online learning can overcome the shift of data distribution between train and test data
\nWe utilized Qwen3-14B, Qwen2.5-14B, and Phi-4 to generate pseudo-labels for the entire test dataset. Subsequently, we trained a ByT5-Large model and used its inference results to further fine-tune the LLMs. This cross-model iterative process was repeated to progressively enhance performance. Notably, we observed that self-training (e.g., training an LLM on LLM outputs or ByT5 on ByT5 outputs) yielded marginal improvements. Significant gains were only achieved through heterogeneous iteration—alternating between different model architectures. Finally, we ensembled the predictions from these 'online-trained' models to produce our final submission.
\nHope you enjoy our solutions.
", "messages": [ { "id": "3427492", "postDate": "03/24/2026 06:32:27", "content": "Hi Kaggle
\nThanks so much for the host and I am very delighted to join the competition. We are the lucky team and did not shake down.\nBig Thanks my teammates zxc and Baiph, we work very hard and share the equal credits to the final results
\nFirst of all, I did not think the quality of training data is not as good as expected. We need lots of work to clean and format training data. In addition, we need to extract or pseudo-label from publication.csv or published_texts.csv. The test and train data varies lots and cv is not stable
\nThe test data is only 4000 sentences we can apply online leanring training strategy to improve the model performance further.
\nAs for data, I used Gemini 3.1 with web tool to generate external data.
\nAs for model selection, we used LLM model as the backbone we tried qwne3,qwen2.5, phi-4, byt5
\nAs for training strategy, total pipeline can be divided into three stages, SFT (knowledge distial )-> RL (reward hacking ) -> Online Learning (fit test distribution), we found that online learning only worked if model archs are very different.
\nI used Gemini 3.1 flash to extract all the sentence level transliteration-translation pairs. The instructions are shown as following
\nprompt = f\"\"\"\n Help me extract Old Assyrian transliteration and its translation\n\n Rules:\n 1. Output a valid JSON strings.\n 2. Only complete sentences need be extracted\n 3. Break both the Akkadian transliteration and the matching English translation into sentences and align them pairwise.\n 4. clean OCR mistakes in sentence_transliteration and sentence_translation\n 5. remove unnecessary characters sentence_transliteration and sentence_translation\n\nInput:\n{page_text}\n\nThe output should only be in json format like the below\n{output_example}\"\"\"\nI did use the above data as sft data, but Qwen got lots of Hallucination issues. Maybe it need to be further clean or formating
\nI used above data from Step1 to build the doc database, and use Qwen3 8B Embedding to retrieval top 3 samples and concat them togther to generate more relable translation .
\n
After that I used it to SFT data, Qwen3 14B model got private 37.9 and public 36.1
\nI used the same apporach RAG to clean train.csv, but I filter out the bad data using the following prompt, only bad data need to be regenerated.
\n
After that I used it to RL data, Qwen3 14B model got private 38.1 and public 36.9
\nMy teammates think online learning can overcome the shift of data distribution between train and test data
\nWe utilized Qwen3-14B, Qwen2.5-14B, and Phi-4 to generate pseudo-labels for the entire test dataset. Subsequently, we trained a ByT5-Large model and used its inference results to further fine-tune the LLMs. This cross-model iterative process was repeated to progressively enhance performance. Notably, we observed that self-training (e.g., training an LLM on LLM outputs or ByT5 on ByT5 outputs) yielded marginal improvements. Significant gains were only achieved through heterogeneous iteration—alternating between different model architectures. Finally, we ensembled the predictions from these 'online-trained' models to produce our final submission.
\nHope you enjoy our solutions.
", "rawMarkdown": "Hi Kaggle\n\nThanks so much for the host and I am very delighted to join the competition. We are the lucky team and did not shake down.\nBig Thanks my teammates zxc and Baiph, we work very hard and share the equal credits to the final results\n\n## Overview of our solution\nFirst of all, I did not think the quality of training data is not as good as expected. We need lots of work to clean and format training data. In addition, we need to extract or pseudo-label from publication.csv or published_texts.csv. The test and train data varies lots and cv is not stable \n\nThe test data is only 4000 sentences we can apply online leanring training strategy to improve the model performance further.\n\nAs for data, I used Gemini 3.1 with web tool to generate external data.\n\nAs for model selection, we used LLM model as the backbone we tried qwne3,qwen2.5, phi-4, byt5\n\nAs for training strategy, total pipeline can be divided into three stages, SFT (knowledge distial )-> RL (reward hacking ) -> Online Learning (fit test distribution), we found that online learning only worked if model archs are very different.\n\n## Step1: Generate data from publication.csv\nI used Gemini 3.1 flash to extract all the sentence level transliteration-translation pairs. The instructions are shown as following\n\n\n prompt = f\"\"\"\n Help me extract Old Assyrian transliteration and its translation\n \n Rules:\n 1. Output a valid JSON strings.\n 2. Only complete sentences need be extracted\n 3. Break both the Akkadian transliteration and the matching English translation into sentences and align them pairwise.\n 4. clean OCR mistakes in sentence_transliteration and sentence_translation\n 5. remove unnecessary characters sentence_transliteration and sentence_translation\n\n Input:\n {page_text}\n\n The output should only be in json format like the below\n {output_example}\"\"\"\n\n\n\n\nI did use the above data as sft data, but Qwen got lots of Hallucination issues. Maybe it need to be further clean or formating\n\n\n## Step2: Pseudo-labelling for published_texts.csv\nI used above data from Step1 to build the doc database, and use Qwen3 8B Embedding to retrieval top 3 samples and concat them togther to generate more relable translation .\n\n\n\nAfter that I used it to SFT data, Qwen3 14B model got private 37.9 and public 36.1\n\n## Step3: Clean train.csv and RL\nI used the same apporach RAG to clean train.csv, but I filter out the bad data using the following prompt, only bad data need to be regenerated.\n\n\n\nAfter that I used it to RL data, Qwen3 14B model got private 38.1 and public 36.9\n\n## Step4: Online learning\nMy teammates think online learning can overcome the shift of data distribution between train and test data\n\nWe utilized Qwen3-14B, Qwen2.5-14B, and Phi-4 to generate **pseudo-labels** for the entire test dataset. Subsequently, we trained a ByT5-Large model and used its inference results to further fine-tune the LLMs. This **cross-model iterative process** was repeated to progressively enhance performance. Notably, we observed that **self-training** (e.g., training an LLM on LLM outputs or ByT5 on ByT5 outputs) yielded marginal improvements. Significant gains were only achieved through **heterogeneous iteration**—alternating between different model architectures. Finally, we ensembled the predictions from these 'online-trained' models to produce our final submission.\n\nHope you enjoy our solutions.", "votes": null }, { "id": "3427580", "postDate": "03/24/2026 09:09:58", "content": "I have one question: How did you fit all online-trained models in Kaggle's 2×T4 inference budget?
", "rawMarkdown": "I have one question: How did you fit all online-trained models in Kaggle's 2×T4 inference budget?", "votes": null }, { "id": "3427585", "postDate": "03/24/2026 09:16:47", "content": "A genuine question (as I've spent most of the time trying to align chunks using all kind of heurstics and back-translations): were we allowed to use closed source commercial models for preparing training dataset? I see most of top solutions used those for dataset chunking/alignment.
\nps: and another genuine naive question: were we allowed to manually (by hands) fix / chunk the training dataset?
", "rawMarkdown": "A genuine question (as I've spent most of the time trying to align chunks using all kind of heurstics and back-translations): were we allowed to use closed source commercial models for preparing training dataset? I see most of top solutions used those for dataset chunking/alignment.\n\nps: and another genuine naive question: were we allowed to manually (by hands) fix / chunk the training dataset?", "votes": null }, { "id": "3427587", "postDate": "03/24/2026 09:20:03", "content": "Inspired by the Jigsaw competition, you can use the Unsloth framework to train 14B-sized models on a T4 GPU.
", "rawMarkdown": "Inspired by the Jigsaw competition, you can use the Unsloth framework to train 14B-sized models on a T4 GPU.", "votes": null }, { "id": "3427588", "postDate": "03/24/2026 09:20:05", "content": "the hoster did not ban any commercial LLM
", "rawMarkdown": "the hoster did not ban any commercial LLM", "votes": null }, { "id": "3427590", "postDate": "03/24/2026 09:21:17", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F805012%2F4b745247e8c33f344adfabba1154cd81%2Fpikachu-pika.png?generation=1774344075820065&alt=media\")
I think we must face reality: we are living in the era of tokens. If this kind of content is banned, does that mean using Claude Code to write programs would also be disallowed?
", "rawMarkdown": "I think we must face reality: we are living in the era of tokens. If this kind of content is banned, does that mean using Claude Code to write programs would also be disallowed?", "votes": null }, { "id": "3427602", "postDate": "03/24/2026 09:33:23", "content": "I probably agree with you. I haven't participated in the competitions for a few years so I [mistakenly] assumed no commercial/hand labelling is allowed by any means so that's why I'm asking.
", "rawMarkdown": "I probably agree with you. I haven't participated in the competitions for a few years so I [mistakenly] assumed no commercial/hand labelling is allowed by any means so that's why I'm asking.", "votes": null }, { "id": "3427604", "postDate": "03/24/2026 09:35:11", "content": "Yeah, many NLP competitions in recent years involve using LLM API for data augmentation or processing. I actually learned about these techniques while studying their solutions.
", "rawMarkdown": "Yeah, many NLP competitions in recent years involve using LLM API for data augmentation or processing. I actually learned about these techniques while studying their solutions.", "votes": null }, { "id": "3427611", "postDate": "03/24/2026 09:49:40", "content": "but, you really can distill a commercial LLM into a smaller model by labelling an extra dataset (definitely not for this competition, but let's assume a toxicity classification)? are there any constraints by Rules on this?
", "rawMarkdown": "but, you really can distill a commercial LLM into a smaller model by labelling an extra dataset (definitely not for this competition, but let's assume a toxicity classification)? are there any constraints by Rules on this?", "votes": null }, { "id": "3427616", "postDate": "03/24/2026 09:53:20", "content": "In terms of distillation, I personally feel that commercial model providers might have restrictions in place—you'd need to check their specific Terms of Service (ToS) to be sure.
", "rawMarkdown": "In terms of distillation, I personally feel that commercial model providers might have restrictions in place—you'd need to check their specific Terms of Service (ToS) to be sure.", "votes": null }, { "id": "3428579", "postDate": "03/25/2026 17:13:47", "content": "Fabulous solution! Really seeing the gaps we missed this competition. Enhancing the data with large commercial models was not something I thought would work so well. Clearly Akkadian is far more researched than I understood! Great work!
", "rawMarkdown": "Fabulous solution! Really seeing the gaps we missed this competition. Enhancing the data with large commercial models was not something I thought would work so well. Clearly Akkadian is far more researched than I understood! Great work!", "votes": null }, { "id": "3428583", "postDate": "03/25/2026 17:21:11", "content": "I really appreciate your solution—it is both concise and elegant. I’d be interested to know how many rounds of heterogeneous iteration your LLM and ByT5 models underwent, how effective the online learning proved to be, and whether you have conducted any corresponding ablation studies.
", "rawMarkdown": "I really appreciate your solution—it is both concise and elegant. I’d be interested to know how many rounds of heterogeneous iteration your LLM and ByT5 models underwent, how effective the online learning proved to be, and whether you have conducted any corresponding ablation studies.", "votes": null }, { "id": "3429651", "postDate": "03/27/2026 03:21:58", "content": "Thank you for recognizing our approach. We conducted a total of three iterations: Initial LLM outputs → ByT5 → LLM → ByT5. Based on our experiments on the Public Leaderboard, the initial score before online learning was 37.3. Directly applying these results to the LLM for online learning yielded a score of 37.5. However, using the same results to train ByT5 via online learning significantly improved the score to 38.4. Subsequently, we used the ByT5-refined outputs to perform online learning on the LLM, achieving a Public LB score of 38.3.
", "rawMarkdown": "Thank you for recognizing our approach. We conducted a total of three iterations: Initial LLM outputs → ByT5 → LLM → ByT5. Based on our experiments on the Public Leaderboard, the initial score before online learning was 37.3. Directly applying these results to the LLM for online learning yielded a score of 37.5. However, using the same results to train ByT5 via online learning significantly improved the score to 38.4. Subsequently, we used the ByT5-refined outputs to perform online learning on the LLM, achieving a Public LB score of 38.3.", "votes": null } ], "comments": [ { "id": 3427580, "author_name": "kingkong153", "author_url": "", "post_date": "03/24/2026 09:09:58", "content": "I have one question: How did you fit all online-trained models in Kaggle's 2×T4 inference budget?
", "votes": null, "replies": [ { "id": 3427587, "author_name": "zxc123cc", "author_url": "", "post_date": "03/24/2026 09:20:03", "content": "Inspired by the Jigsaw competition, you can use the Unsloth framework to train 14B-sized models on a T4 GPU.
", "votes": null, "replies": [] } ] }, { "id": 3427585, "author_name": "yaroshevskiy", "author_url": "", "post_date": "03/24/2026 09:16:47", "content": "A genuine question (as I've spent most of the time trying to align chunks using all kind of heurstics and back-translations): were we allowed to use closed source commercial models for preparing training dataset? I see most of top solutions used those for dataset chunking/alignment.
\nps: and another genuine naive question: were we allowed to manually (by hands) fix / chunk the training dataset?
", "votes": null, "replies": [ { "id": 3427588, "author_name": "leehann", "author_url": "", "post_date": "03/24/2026 09:20:05", "content": "the hoster did not ban any commercial LLM
", "votes": null, "replies": [ { "id": 3427590, "author_name": "yaroshevskiy", "author_url": "", "post_date": "03/24/2026 09:21:17", "content": "
I think we must face reality: we are living in the era of tokens. If this kind of content is banned, does that mean using Claude Code to write programs would also be disallowed?
", "votes": null, "replies": [ { "id": 3427602, "author_name": "yaroshevskiy", "author_url": "", "post_date": "03/24/2026 09:33:23", "content": "I probably agree with you. I haven't participated in the competitions for a few years so I [mistakenly] assumed no commercial/hand labelling is allowed by any means so that's why I'm asking.
", "votes": null, "replies": [ { "id": 3427604, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 09:35:11", "content": "Yeah, many NLP competitions in recent years involve using LLM API for data augmentation or processing. I actually learned about these techniques while studying their solutions.
", "votes": null, "replies": [ { "id": 3427611, "author_name": "yaroshevskiy", "author_url": "", "post_date": "03/24/2026 09:49:40", "content": "but, you really can distill a commercial LLM into a smaller model by labelling an extra dataset (definitely not for this competition, but let's assume a toxicity classification)? are there any constraints by Rules on this?
", "votes": null, "replies": [ { "id": 3427616, "author_name": "wscmisx", "author_url": "", "post_date": "03/24/2026 09:53:20", "content": "In terms of distillation, I personally feel that commercial model providers might have restrictions in place—you'd need to check their specific Terms of Service (ToS) to be sure.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3428579, "author_name": "cody11null", "author_url": "", "post_date": "03/25/2026 17:13:47", "content": "Fabulous solution! Really seeing the gaps we missed this competition. Enhancing the data with large commercial models was not something I thought would work so well. Clearly Akkadian is far more researched than I understood! Great work!
", "votes": null, "replies": [] }, { "id": 3428583, "author_name": "zhuyifanss", "author_url": "", "post_date": "03/25/2026 17:21:11", "content": "I really appreciate your solution—it is both concise and elegant. I’d be interested to know how many rounds of heterogeneous iteration your LLM and ByT5 models underwent, how effective the online learning proved to be, and whether you have conducted any corresponding ablation studies.
", "votes": null, "replies": [ { "id": 3429651, "author_name": "zxc123cc", "author_url": "", "post_date": "03/27/2026 03:21:58", "content": "Thank you for recognizing our approach. We conducted a total of three iterations: Initial LLM outputs → ByT5 → LLM → ByT5. Based on our experiments on the Public Leaderboard, the initial score before online learning was 37.3. Directly applying these results to the LLM for online learning yielded a score of 37.5. However, using the same results to train ByT5 via online learning significantly improved the score to 38.4. Subsequently, we used the ByT5-refined outputs to perform online learning on the LLM, achieving a Public LB score of 38.3.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3427492": "Hi Kaggle\n\nThanks so much for the host and I am very delighted to join the competition. We are the lucky team and did not shake down.\nBig Thanks my teammates zxc and Baiph, we work very hard and share the equal credits to the final results\n\n## Overview of our solution\nFirst of all, I did not think the quality of training data is not as good as expected. We need lots of work to clean and format training data. In addition, we need to extract or pseudo-label from publication.csv or published_texts.csv. The test and train data varies lots and cv is not stable \n\nThe test data is only 4000 sentences we can apply online leanring training strategy to improve the model performance further.\n\nAs for data, I used Gemini 3.1 with web tool to generate external data.\n\nAs for model selection, we used LLM model as the backbone we tried qwne3,qwen2.5, phi-4, byt5\n\nAs for training strategy, total pipeline can be divided into three stages, SFT (knowledge distial )-> RL (reward hacking ) -> Online Learning (fit test distribution), we found that online learning only worked if model archs are very different.\n\n## Step1: Generate data from publication.csv\nI used Gemini 3.1 flash to extract all the sentence level transliteration-translation pairs. The instructions are shown as following\n\n\n prompt = f\"\"\"\n Help me extract Old Assyrian transliteration and its translation\n \n Rules:\n 1. Output a valid JSON strings.\n 2. Only complete sentences need be extracted\n 3. Break both the Akkadian transliteration and the matching English translation into sentences and align them pairwise.\n 4. clean OCR mistakes in sentence_transliteration and sentence_translation\n 5. remove unnecessary characters sentence_transliteration and sentence_translation\n\n Input:\n {page_text}\n\n The output should only be in json format like the below\n {output_example}\"\"\"\n\n\n\n\nI did use the above data as sft data, but Qwen got lots of Hallucination issues. Maybe it need to be further clean or formating\n\n\n## Step2: Pseudo-labelling for published_texts.csv\nI used above data from Step1 to build the doc database, and use Qwen3 8B Embedding to retrieval top 3 samples and concat them togther to generate more relable translation .\n\n\n\nAfter that I used it to SFT data, Qwen3 14B model got private 37.9 and public 36.1\n\n## Step3: Clean train.csv and RL\nI used the same apporach RAG to clean train.csv, but I filter out the bad data using the following prompt, only bad data need to be regenerated.\n\n\n\nAfter that I used it to RL data, Qwen3 14B model got private 38.1 and public 36.9\n\n## Step4: Online learning\nMy teammates think online learning can overcome the shift of data distribution between train and test data\n\nWe utilized Qwen3-14B, Qwen2.5-14B, and Phi-4 to generate **pseudo-labels** for the entire test dataset. Subsequently, we trained a ByT5-Large model and used its inference results to further fine-tune the LLMs. This **cross-model iterative process** was repeated to progressively enhance performance. Notably, we observed that **self-training** (e.g., training an LLM on LLM outputs or ByT5 on ByT5 outputs) yielded marginal improvements. Significant gains were only achieved through **heterogeneous iteration**—alternating between different model architectures. Finally, we ensembled the predictions from these 'online-trained' models to produce our final submission.\n\nHope you enjoy our solutions.", "3427580": "I have one question: How did you fit all online-trained models in Kaggle's 2×T4 inference budget?", "3427585": "A genuine question (as I've spent most of the time trying to align chunks using all kind of heurstics and back-translations): were we allowed to use closed source commercial models for preparing training dataset? I see most of top solutions used those for dataset chunking/alignment.\n\nps: and another genuine naive question: were we allowed to manually (by hands) fix / chunk the training dataset?", "3427587": "Inspired by the Jigsaw competition, you can use the Unsloth framework to train 14B-sized models on a T4 GPU.", "3427588": "the hoster did not ban any commercial LLM", "3427590": "", "3427594": "I think we must face reality: we are living in the era of tokens. If this kind of content is banned, does that mean using Claude Code to write programs would also be disallowed?", "3427602": "I probably agree with you. I haven't participated in the competitions for a few years so I [mistakenly] assumed no commercial/hand labelling is allowed by any means so that's why I'm asking.", "3427604": "Yeah, many NLP competitions in recent years involve using LLM API for data augmentation or processing. I actually learned about these techniques while studying their solutions.", "3427611": "but, you really can distill a commercial LLM into a smaller model by labelling an extra dataset (definitely not for this competition, but let's assume a toxicity classification)? are there any constraints by Rules on this?", "3427616": "In terms of distillation, I personally feel that commercial model providers might have restrictions in place—you'd need to check their specific Terms of Service (ToS) to be sure.", "3428579": "Fabulous solution! Really seeing the gaps we missed this competition. Enhancing the data with large commercial models was not something I thought would work so well. Clearly Akkadian is far more researched than I understood! Great work!", "3428583": "I really appreciate your solution—it is both concise and elegant. I’d be interested to know how many rounds of heterogeneous iteration your LLM and ByT5 models underwent, how effective the online learning proved to be, and whether you have conducted any corresponding ablation studies.", "3429651": "Thank you for recognizing our approach. We conducted a total of three iterations: Initial LLM outputs → ByT5 → LLM → ByT5. Based on our experiments on the Public Leaderboard, the initial score before online learning was 37.3. Directly applying these results to the LLM for online learning yielded a score of 37.5. However, using the same results to train ByT5 via online learning significantly improved the score to 38.4. Subsequently, we used the ByT5-refined outputs to perform online learning on the LLM, achieving a Public LB score of 38.3." }, "source": "meta" }