Thank kaggle stuff holds this great competition. Thank the great community provides so many fantastic notebooks and discussion. Finally thank my teammate @guohey and @thousandtie for accompanying me on this unforgettable 10-day India trip.
\nOur solution is mainly the topic: trust the cv and fuse models with as much different as possible.
\nWe can't find a way to ensemble rembert and xlmr, so we decided to use only Rembert in one final submit and XLMR in the other.
Why do we decide to Violently fusion of different models
\nBecause we found that adding data can't improve performance of single model in both CV and LB.
Why do we decide to trust CV more than LB?
\nActually, there's a bit gamble. We can't get a better LB score than the 792 submission before the last 2 days. Besides, no matter how I blend my own model with 792, even if I give my model a small weight, it's worse than 792. I calculate the cv and lb of each model in the 792 submission, finding the fold 0 and fold 1 have poor lb and cv, while the last 3 folds have great cv and great lb. However, without using the poor first 2 fols, its LB score drops greatly. Reading the training notebook of the first 2 folds, I found nothing special.
\nTaking the above experimental results and analysis into consideration, I would rather believe the 792 submission overfit LB. So instead of trying to beat the public 792 on LB, I chose to use my CV's better model to ensemble and prayed for its shake up on PLB.
\nLuckily, this gamble paid off.
How do we make models different?
\nWe take different seed and different data.
\nFor example, the 12 folds rembert submission consists of the following models:
How do we trust our cv?
\nWe train 5 fold of different data and seed each time. After we got weight, we calculate their CV. We only use CV better than 0.68 to ensemble.
What can be improved in the future？
\nWe can't find a way to evaluate the cv of our ensemble submission, because there are different seeds, and each seed only use 20%-80% model.
\nIn fact our best submission has a 0.760 PLB, which is really close to gold. But its LB is worse than our best 795 rembert submission, so we didn't choose it. Maybe with a better cv strategy, we have a chance to get a gold medal.
\nIf you have any ideas to calculate cv of the ensemble output, please discuss with me!
Thank you for your reading, feel free to comment and upvote!
", "comments": [ { "id": 1583774, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T03:59:21.930000", "content": "Congratulations @forcewithme and team! I saw you at the same public LB with me. Happy to see that we both jumped upward on private LB.
\nSimilar to you, i computed the CV score for the first 2 folds of the public 0.792 notebook. Their CV score was low, but none-the-less, if you remove or change them the LB score decrease. This confused me (and made me suspicious). So I also built a strong CV model (i.e. \"trust your cv\") without using the public notebook as my second submission. And it did better on private LB!
", "votes": 1, "replies": [ { "id": 1583781, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-16T04:05:12.870000", "content": "Congratulations @cdeotte your another silver medal. It seems like we have almost the same experience and strategy in this game🤝.
", "votes": 1, "replies": [] } ] }, { "id": 1583772, "author_name": "Ctrl_CV", "author_url": "", "post_date": "2021-11-16T03:56:56.020000", "content": "Good job! very impressive works!👍
", "votes": 2, "replies": [] }, { "id": 1584523, "author_name": "Mathurin Ache", "author_url": "", "post_date": "2021-11-16T15:14:41.080000", "content": "Thank you @forcewithme for your solution overview!
", "votes": 0, "replies": [ { "id": 1584599, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-16T16:02:54.760000", "content": "You are welcome! Thank you for reading.😄
", "votes": 0, "replies": [] } ] }, { "id": 1584070, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T08:43:55.933000", "content": "Congrats @forcewithme ! I always like to read solutions claiming \"trust your cv\". That's the right thing to do! I want to ask you, I read that you variated your training data, but what about your validation one? Did it variate too based on your training data or did you keep it fixed (only original training data, for instance).
", "votes": 0, "replies": [ { "id": 1584099, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-16T09:14:17.303000", "content": "Thank you @lucamassaron . I only validate on the training dataset(keep it fixed), no external data in validation.
", "votes": 1, "replies": [] }, { "id": 1584112, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T09:27:06.490000", "content": "It confirms the same strategy thay we also used, thank you!
", "votes": 0, "replies": [] }, { "id": 1584121, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-16T09:34:57.357000", "content": "You are welcome😁
", "votes": 0, "replies": [] } ] } ], "comment_count": 9, "messages": [ { "id": 1583769, "postDate": "2021-11-16T03:52:16.763Z", "content": "Thank kaggle stuff holds this great competition. Thank the great community provides so many fantastic notebooks and discussion. Finally thank my teammate @guohey and @thousandtie for accompanying me on this unforgettable 10-day India trip.
\nOur solution is mainly the topic: trust the cv and fuse models with as much different as possible.
\nWe can't find a way to ensemble rembert and xlmr, so we decided to use only Rembert in one final submit and XLMR in the other.
Why do we decide to Violently fusion of different models
\nBecause we found that adding data can't improve performance of single model in both CV and LB.
Why do we decide to trust CV more than LB?
\nActually, there's a bit gamble. We can't get a better LB score than the 792 submission before the last 2 days. Besides, no matter how I blend my own model with 792, even if I give my model a small weight, it's worse than 792. I calculate the cv and lb of each model in the 792 submission, finding the fold 0 and fold 1 have poor lb and cv, while the last 3 folds have great cv and great lb. However, without using the poor first 2 fols, its LB score drops greatly. Reading the training notebook of the first 2 folds, I found nothing special.
\nTaking the above experimental results and analysis into consideration, I would rather believe the 792 submission overfit LB. So instead of trying to beat the public 792 on LB, I chose to use my CV's better model to ensemble and prayed for its shake up on PLB.
\nLuckily, this gamble paid off.
How do we make models different?
\nWe take different seed and different data.
\nFor example, the 12 folds rembert submission consists of the following models:
How do we trust our cv?
\nWe train 5 fold of different data and seed each time. After we got weight, we calculate their CV. We only use CV better than 0.68 to ensemble.
What can be improved in the future？
\nWe can't find a way to evaluate the cv of our ensemble submission, because there are different seeds, and each seed only use 20%-80% model.
\nIn fact our best submission has a 0.760 PLB, which is really close to gold. But its LB is worse than our best 795 rembert submission, so we didn't choose it. Maybe with a better cv strategy, we have a chance to get a gold medal.
\nIf you have any ideas to calculate cv of the ensemble output, please discuss with me!
Thank you for your reading, feel free to comment and upvote!
", "rawMarkdown": "Thank kaggle stuff holds this great competition. Thank the great community provides so many fantastic notebooks and discussion. Finally thank my teammate @guohey and @thousandtie for accompanying me on this unforgettable 10-day India trip.\n\nOur solution is mainly the topic: trust the cv and fuse models with as much different as possible.\nWe can't find a way to ensemble rembert and xlmr, so we decided to use only Rembert in one final submit and XLMR in the other.\n\n1. Why do we decide to Violently fusion of different models\nBecause we found that adding data can't improve performance of single model in both CV and LB.\n\n2. Why do we decide to trust CV more than LB?\nActually, there's a bit gamble. We can't get a better LB score than the 792 submission before the last 2 days. Besides, no matter how I blend my own model with 792, even if I give my model a small weight, it's worse than 792. I calculate the cv and lb of each model in the 792 submission, finding the fold 0 and fold 1 have poor lb and cv, while the last 3 folds have great cv and great lb. However, without using the poor first 2 fols, its LB score drops greatly. Reading the training notebook of the first 2 folds, I found nothing special. \nTaking the above experimental results and analysis into consideration, I would rather believe the 792 submission overfit LB. So instead of trying to beat the public 792 on LB, I chose to use my CV's better model to ensemble and prayed for its shake up on PLB.\nLuckily, this gamble paid off.\n\n3. How do we make models different?\nWe take different seed and different data. \nFor example, the 12 folds rembert submission consists of the following models:\n- 2 fold with **only hindi external data** and **seed 2021** (LB 782)\n- 3 fold with **hindi and tamil external data** which is popular in community and **seed 42** (LB776)\n- 4 fold with the same data of the above one and **seed 1111** (haven't got LB, we didn't have enough submission)\n- 2 fold with the data of the above one **adding a sample of 15k @zzy990106 external tamil data** and seed 99.\nWe tried some other combinations, sampling more external data and replace it with some folds in the seed1111 part. But this one plays best in LB, so we use it for final submision of rembert. \nWe take the same principle in xlmr, but maybe we use some/all folds of the public 792 models in our xlmr ensemble, this strategy didn't work in xlmr.(No matter how I blend with 792, it is worse in Lb.)\n\n4. How do we trust our cv?\nWe train 5 fold of different data and seed each time. After we got weight, we calculate their CV. We only use CV better than 0.68 to ensemble.\n\n5. What can be improved in the future？\nWe can't find a way to evaluate the cv of our ensemble submission, because there are different seeds, and each seed only use 20%-80% model.\nIn fact our best submission has a 0.760 PLB, which is really close to gold. But its LB is worse than our best 795 rembert submission, so we didn't choose it. Maybe with a better cv strategy, we have a chance to get a gold medal.\nIf you have any ideas to calculate cv of the ensemble output, please discuss with me!\n\nThank you for your reading, feel free to comment and upvote!\n \n\n", "votes": 16 }, { "id": 1583774, "postDate": "2021-11-16T03:59:21.930Z", "content": "Congratulations @forcewithme and team! I saw you at the same public LB with me. Happy to see that we both jumped upward on private LB.
\nSimilar to you, i computed the CV score for the first 2 folds of the public 0.792 notebook. Their CV score was low, but none-the-less, if you remove or change them the LB score decrease. This confused me (and made me suspicious). So I also built a strong CV model (i.e. \"trust your cv\") without using the public notebook as my second submission. And it did better on private LB!
", "rawMarkdown": "Congratulations @forcewithme and team! I saw you at the same public LB with me. Happy to see that we both jumped upward on private LB.\n\nSimilar to you, i computed the CV score for the first 2 folds of the public 0.792 notebook. Their CV score was low, but none-the-less, if you remove or change them the LB score decrease. This confused me (and made me suspicious). So I also built a strong CV model (i.e. \"trust your cv\") without using the public notebook as my second submission. And it did better on private LB!", "votes": 1, "replies": [ { "id": 1583781, "postDate": "2021-11-16T04:05:12.870Z", "content": "Congratulations @cdeotte your another silver medal. It seems like we have almost the same experience and strategy in this game🤝.
", "rawMarkdown": "Congratulations @cdeotte your another silver medal. It seems like we have almost the same experience and strategy in this game🤝. ", "votes": 1 } ] }, { "id": 1583772, "postDate": "2021-11-16T03:56:56.020Z", "content": "Good job! very impressive works!👍
", "rawMarkdown": "Good job! very impressive works!👍", "votes": 2 }, { "id": 1584523, "postDate": "2021-11-16T15:14:41.080Z", "content": "Thank you @forcewithme for your solution overview!
", "rawMarkdown": "Thank you @forcewithme for your solution overview!", "replies": [ { "id": 1584599, "postDate": "2021-11-16T16:02:54.760Z", "content": "You are welcome! Thank you for reading.😄
", "rawMarkdown": "You are welcome! Thank you for reading.😄" } ] }, { "id": 1584070, "postDate": "2021-11-16T08:43:55.933Z", "content": "Congrats @forcewithme ! I always like to read solutions claiming \"trust your cv\". That's the right thing to do! I want to ask you, I read that you variated your training data, but what about your validation one? Did it variate too based on your training data or did you keep it fixed (only original training data, for instance).
", "rawMarkdown": "Congrats @forcewithme ! I always like to read solutions claiming \"trust your cv\". That's the right thing to do! I want to ask you, I read that you variated your training data, but what about your validation one? Did it variate too based on your training data or did you keep it fixed (only original training data, for instance).", "replies": [ { "id": 1584099, "postDate": "2021-11-16T09:14:17.303Z", "content": "Thank you @lucamassaron . I only validate on the training dataset(keep it fixed), no external data in validation.
", "rawMarkdown": "Thank you @lucamassaron . I only validate on the training dataset(keep it fixed), no external data in validation. ", "votes": 1 }, { "id": 1584112, "postDate": "2021-11-16T09:27:06.490Z", "content": "It confirms the same strategy thay we also used, thank you!
", "rawMarkdown": "It confirms the same strategy thay we also used, thank you!" }, { "id": 1584121, "postDate": "2021-11-16T09:34:57.357Z", "content": "You are welcome😁
", "rawMarkdown": "You are welcome😁" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:20:50Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7539053, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/21-place-solution-trust-cv-differential-fusion" }