![\"Bild1\"](\"https://i.ibb.co/HKxsfYG/Bild1.png\")
First of all, I would like to thank Kaggle and the competitions hosts for hosting this interesting nlp challenge. I like the trend, that we see more and more nlp competitions. I need a short break but will join Jigsaw for sure 😉.
\nWe realized early in the competition that the local cv is not really meaningful. We decided the first time in a competition to focus on the public leaderboard instead on the cv but keeping diversity all the time to avoid a shakedown. Diversity was already a key for our gold medal in the CommonLit challenge. As all teams we started with XLM Roberta Large. We pushed it to 0.785 but didn’t manage to improve it further (1).
\n
After that we had obviously a breakthrough. We managed to ensemble models of different architectures. We mapped the logits to the character level to combine outputs of different tokenizer sizes (link to the notebook the end of my post).
\nAfter that we ensembled our Roberta Large with Muril Base (0.732) and scored 0.792 (2). We implemented a normalization of the logits to take different scales into account and scored 0.795 (3).
\nNow a link to Muril Large was posted in the forums. We struggled to finetune it but finally created a model with 0.784 public score. We ensembled it with our Roberta Large and had a big jump to 0.808 (4). An interesting sidenote is that this model would have score 0.776 on the private lb already.
\nNow we added Rembert (0.774 single score) (5) and XLM Info Large (6). In the last days of the competition, we spent a lot of time finetuning the weights of our ensemble (7). We also added the Wechsel-models with a small share in the end (8) and ended up with 0.821 public lb score.
Our final ensemble was:
\nRembert Large - 16.8%
\nMuril Large - 29.4%
\nRoberta Large - 44.1%
\nXlm Info Large - 5.7%
\nWechsel - 4%
I'm happy with the result, even though we were out of the top 5 in the end. Because of the small data, the huge gap between cv and lb, it felt right to focus on the public lb. I expected a much higher correlation between private lb and public lb than between private lb and cv. I have checked it for our submissions and feel vindicated:
\n![\"Bild2\"](\"https://i.ibb.co/7SXN57d/Bild2.png\")
It was a pleasure to tackle this problem with you guys. I will try to be more active in the forums next time and wish you all a good time.
\nThanks to my awesome teammate @chamecall
\nLink to our notebook:
\nhttps://www.kaggle.com/benbla/fork-of-chaiiqa-inference-modelsensembling-eba980
Congrats on gold @benbla !✨😄
", "votes": 1, "replies": [ { "id": 1591392, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2021-11-22T09:33:22.087000", "content": "Thank you :)
", "votes": 0, "replies": [] } ] }, { "id": 1590693, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-21T15:06:35.827000", "content": "Congratulations on your achievements.
\nCongratulations~
", "votes": 1, "replies": [ { "id": 1591393, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2021-11-22T09:33:28.917000", "content": "Thanks! :)
", "votes": 0, "replies": [] } ] }, { "id": 1589770, "author_name": "hiromu", "author_url": "", "post_date": "2021-11-20T14:57:55.537000", "content": "Congrats!!
\nCould you tell me which datasets did you use?
We just used the public available additional data (mlqa, xquad and the 7.92 mb additional tamil data which was shared some weeks ago) and the regular training data.
", "votes": 1, "replies": [] }, { "id": 1592187, "author_name": "hiromu", "author_url": "", "post_date": "2021-11-23T00:36:23.650000", "content": "Thank you so much!
", "votes": 0, "replies": [] } ] }, { "id": 1586236, "author_name": "DrHB", "author_url": "", "post_date": "2021-11-17T23:08:19.843000", "content": "Nice and congrats on Gold!
", "votes": 1, "replies": [ { "id": 1586647, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2021-11-18T07:18:31.590000", "content": "Thank you :)
", "votes": 0, "replies": [] } ] }, { "id": 1586209, "author_name": "Darek KÅ‚eczek", "author_url": "", "post_date": "2021-11-17T22:23:54.183000", "content": "Congrats! Indeed diversity was one of the keys to a good score here!
", "votes": 2, "replies": [ { "id": 1586645, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2021-11-18T07:17:02.647000", "content": "Yeah, architecture diversity seems always to be one of the success factors. Anyway, there was a 10 fold muril ensemble in the top 5 as well but I think generally it makes sense to use different architectures. My feeling was that the xlm roberta large models in my ensemble were quite similar to each other. I mean even the .732 muril base improved lb score already.
", "votes": 0, "replies": [] } ] } ], "comment_count": 11, "messages": [ { "id": 1585828, "postDate": "2021-11-17T15:23:03.357Z", "content": "First of all, I would like to thank Kaggle and the competitions hosts for hosting this interesting nlp challenge. I like the trend, that we see more and more nlp competitions. I need a short break but will join Jigsaw for sure 😉.
\nWe realized early in the competition that the local cv is not really meaningful. We decided the first time in a competition to focus on the public leaderboard instead on the cv but keeping diversity all the time to avoid a shakedown. Diversity was already a key for our gold medal in the CommonLit challenge. As all teams we started with XLM Roberta Large. We pushed it to 0.785 but didn’t manage to improve it further (1).
\n
After that we had obviously a breakthrough. We managed to ensemble models of different architectures. We mapped the logits to the character level to combine outputs of different tokenizer sizes (link to the notebook the end of my post).
\nAfter that we ensembled our Roberta Large with Muril Base (0.732) and scored 0.792 (2). We implemented a normalization of the logits to take different scales into account and scored 0.795 (3).
\nNow a link to Muril Large was posted in the forums. We struggled to finetune it but finally created a model with 0.784 public score. We ensembled it with our Roberta Large and had a big jump to 0.808 (4). An interesting sidenote is that this model would have score 0.776 on the private lb already.
\nNow we added Rembert (0.774 single score) (5) and XLM Info Large (6). In the last days of the competition, we spent a lot of time finetuning the weights of our ensemble (7). We also added the Wechsel-models with a small share in the end (8) and ended up with 0.821 public lb score.
Our final ensemble was:
\nRembert Large - 16.8%
\nMuril Large - 29.4%
\nRoberta Large - 44.1%
\nXlm Info Large - 5.7%
\nWechsel - 4%
I'm happy with the result, even though we were out of the top 5 in the end. Because of the small data, the huge gap between cv and lb, it felt right to focus on the public lb. I expected a much higher correlation between private lb and public lb than between private lb and cv. I have checked it for our submissions and feel vindicated:
\n
It was a pleasure to tackle this problem with you guys. I will try to be more active in the forums next time and wish you all a good time.
\nThanks to my awesome teammate @chamecall
\nLink to our notebook:
\nhttps://www.kaggle.com/benbla/fork-of-chaiiqa-inference-modelsensembling-eba980
\n \nAfter that we had obviously a breakthrough. We managed to **ensemble models of different architectures**. We mapped the logits to the character level to combine outputs of different tokenizer sizes (link to the notebook the end of my post).\n\nAfter that we ensembled our Roberta Large with Muril Base (0.732) and scored **0.792 (2)**. We implemented a normalization of the logits to take different scales into account and scored **0.795 (3)**.\n\nNow a link to Muril Large was posted in the forums. We struggled to finetune it but finally created a model with 0.784 public score. We ensembled it with our Roberta Large and had a big jump to **0.808 (4)**. An interesting sidenote is that this model would have score 0.776 on the private lb already. \nNow we added Rembert (0.774 single score) (5) and XLM Info Large (6). In the last days of the competition, we spent a lot of time finetuning the weights of our ensemble (7). We also added the Wechsel-models with a small share in the end (8) and **ended up with 0.821 public lb score**. \n\nOur final ensemble was:\nRembert Large - 16.8%\nMuril Large - 29.4%\nRoberta Large - 44.1%\nXlm Info Large - 5.7%\nWechsel - 4%\n\nI'm happy with the result, even though we were out of the top 5 in the end. Because of the small data, the huge gap between cv and lb, **it felt right to focus on the public lb**. I expected a much higher correlation between private lb and public lb than between private lb and cv. I have checked it for our submissions and feel vindicated:\n\n \n\nIt was a pleasure to tackle this problem with you guys. I will try to be more active in the forums next time and wish you all a good time.\n\nThanks to my awesome teammate @chamecall \n\nLink to our notebook:\nhttps://www.kaggle.com/benbla/fork-of-chaiiqa-inference-modelsensembling-eba980",
"votes": 15
},
{
"id": 1591243,
"postDate": "2021-11-22T05:46:28.763Z",
"content": "Congrats on gold @benbla !✨😄
", "rawMarkdown": "Congrats on gold @benbla !✨😄", "votes": 1, "replies": [ { "id": 1591392, "postDate": "2021-11-22T09:33:22.087Z", "content": "Thank you :)
", "rawMarkdown": "Thank you :)" } ] }, { "id": 1590693, "postDate": "2021-11-21T15:06:35.827Z", "content": "Congratulations on your achievements.
\nCongratulations~
", "rawMarkdown": "Congratulations on your achievements.\n\nCongratulations~", "votes": 1, "replies": [ { "id": 1591393, "postDate": "2021-11-22T09:33:28.917Z", "content": "Thanks! :)
", "rawMarkdown": "Thanks! :)" } ] }, { "id": 1589770, "postDate": "2021-11-20T14:57:55.537Z", "content": "Congrats!!
\nCould you tell me which datasets did you use?
We just used the public available additional data (mlqa, xquad and the 7.92 mb additional tamil data which was shared some weeks ago) and the regular training data.
", "rawMarkdown": "We just used the public available additional data (mlqa, xquad and the 7.92 mb additional tamil data which was shared some weeks ago) and the regular training data.", "votes": 1 }, { "id": 1592187, "postDate": "2021-11-23T00:36:23.650Z", "content": "Thank you so much!
", "rawMarkdown": "Thank you so much!" } ] }, { "id": 1586236, "postDate": "2021-11-17T23:08:19.843Z", "content": "Nice and congrats on Gold!
", "rawMarkdown": "Nice and congrats on Gold!", "votes": 1, "replies": [ { "id": 1586647, "postDate": "2021-11-18T07:18:31.590Z", "content": "Thank you :)
", "rawMarkdown": "Thank you :)" } ] }, { "id": 1586209, "postDate": "2021-11-17T22:23:54.183Z", "content": "Congrats! Indeed diversity was one of the keys to a good score here!
", "rawMarkdown": "Congrats! Indeed diversity was one of the keys to a good score here!", "votes": 2, "replies": [ { "id": 1586645, "postDate": "2021-11-18T07:17:02.647Z", "content": "Yeah, architecture diversity seems always to be one of the success factors. Anyway, there was a 10 fold muril ensemble in the top 5 as well but I think generally it makes sense to use different architectures. My feeling was that the xlm roberta large models in my ensemble were quite similar to each other. I mean even the .732 muril base improved lb score already.
", "rawMarkdown": "Yeah, architecture diversity seems always to be one of the success factors. Anyway, there was a 10 fold muril ensemble in the top 5 as well but I think generally it makes sense to use different architectures. My feeling was that the xlm roberta large models in my ensemble were quite similar to each other. I mean even the .732 muril base improved lb score already." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:23:16Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7403583, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/sophie-6th-place-solution-with-a-visualization-of-" }