Thank you Kaggle and host for a fun and interesting competition.
\nThank you @kishalmandal for the wonderful public notebooks. I joined a few days ago and immediately studied Kishal's top scoring public notebooks.
\nMy first final submission was my maximum public LB of 0.795. I took Kishal's LB 0.792 public notebook here and changed the weights to
start_logits = (start_logits1 + 0.85*start_logits2 + 0.85*start_logits3\n + start_logits4+ 1.7*start_logits5 )/5.4\nend_logits = (end_logits1 + 0.85*end_logits2 + 0.85*end_logits3 \n + end_logits4 + 1.7*end_logits5)/5.4\nThis achieved public LB 0.795 which was 60th place on public LB. But when i calculated the local CV score, it was only CV 0.677. This was very concerning. (And as it turns out, this submission had private LB 0.718 and had 728th place 😮 so it was overfitting public LB)
For my second final submission, I downloaded @kishalmandal train notebooks here and here. I optimized the hyperparameters, learning rates, and learning schedules and achieved local CV 0.697.
Next, I adjusted the popular public post process. I changed the first line which uses logits to the second line which uses the Bayesian product of probabilities (and gives start position a slight emphasis over end position).
\n\"score\": start_logits[start_index] + end_logits[end_index]\n\nfrom scipy.special import expit #THIS IS SIGMOID FUNCTION\n\"score\": expit(1.2*start_logits[start_index]) \\\n * expit(end_logits[end_index])\nThis boosted CV by +0.003, public LB by +0.003, and private LB by +0.004!
\nMy best final submission turned out to be my second submission which was my best CV and not my best LB. This model had CV 0.700, Public LB 0.784, and Private LB 0.744. Once again, \"trust you CV\" was the best strategy!
\nMy solution is only XLM-Roberta. After reading top solutions, it appears that ensemble with Muril-Large (and maybe Rembert) was the secret sauce to achieving a higher LB score! And TydiQA external dataset.
Congratulations !
\nGood Job!
", "votes": 1, "replies": [] }, { "id": 1589818, "author_name": "Old Monk", "author_url": "", "post_date": "2021-11-20T15:48:20.303000", "content": "Congratulation @cdeotte ! Thanks for sharing your solution!
", "votes": 1, "replies": [] }, { "id": 1585163, "author_name": "adldotori", "author_url": "", "post_date": "2021-11-17T06:10:22.093000", "content": "Wow… This is so efficient solutions. I learned a lot. thanks!
", "votes": 1, "replies": [ { "id": 1586429, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-18T04:24:50.177000", "content": "Thanks Adldotori
", "votes": 0, "replies": [] } ] }, { "id": 1584290, "author_name": "Mikhail Savin", "author_url": "", "post_date": "2021-11-16T12:23:48.397000", "content": "@cdeotte congrats!
\nCan i ask about this moment?
\n\n\nMy first final submission was my maximum public LB of 0.795. I took Kishal's LB 0.792 public notebook here and changed the weights
\n
And how you came to this decision?
\nThank you in advance
", "votes": 1, "replies": [ { "id": 1584686, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T17:29:22.777000", "content": "Thanks. Are you asking how I found the optimal LB weights? I explain the procedure in one of the other comments.
", "votes": 1, "replies": [] }, { "id": 1585142, "author_name": "Mikhail Savin", "author_url": "", "post_date": "2021-11-17T05:51:04.567000", "content": "Oh, thank you!
", "votes": 1, "replies": [] } ] }, { "id": 1583846, "author_name": "", "author_url": "", "post_date": "2021-11-16T05:23:40.003000", "content": "You make it look easy..
", "votes": 1, "replies": [ { "id": 1584045, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T08:27:31.513000", "content": "Thanks Yam
", "votes": 0, "replies": [] } ] }, { "id": 1583794, "author_name": "Saiyan Warrior", "author_url": "", "post_date": "2021-11-16T04:15:52.750000", "content": "Hi @cdeotte congrats on your win. Can you tell us how did you decide to change weights?
\nstart_logits = (start_logits1 + 0.85*start_logits2 + 0.85*start_logits3\n + start_logits4+ 1.7*start_logits5 )/5.4\nend_logits = (end_logits1 + 0.85*end_logits2 + 0.85*end_logits3 \n + end_logits4 + 1.7*end_logits5)/5.4\nThanks
", "votes": 1, "replies": [ { "id": 1583797, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T04:25:01.303000", "content": "On first day, i make 5 submissions using the five weights below and get respective LB scores:
\nFrom above, i see that the strongest LB fold is fold 5 and the weakest LB folds are folds 2,3. So on second day I try to strengthen fold 5:
\nNext, I try to weaken folds 2,3 (and decrease fold 5 to keep in correct proportion):
\nNote that this is a \"trust your LB\" optimization and works in some competitions, but didn't work in this competition.
", "votes": 2, "replies": [] } ] }, { "id": 1583917, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-11-16T06:27:34.540000", "content": "Congratulations @cdeotte on results!
", "votes": 2, "replies": [ { "id": 1584044, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T08:27:19.837000", "content": "Thanks. Congratulations to you and your team for maintaining your great result on both public and private LB
", "votes": 0, "replies": [] } ] }, { "id": 1583852, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-11-16T05:27:47.640000", "content": "Hi @cdeotte great work , can you tell me the intuition behind this
\n\n\n\"score\": expit(1.2*start_logits[start_index]) \\
\n
\n * expit(end_logits[end_index])
Thanks
", "votes": 2, "replies": [ { "id": 1583860, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T05:36:12.023000", "content": "Logits are log odds (like \"there is log(3 to 1) chance this horse will win the race\"). And sigmoid(logits) are probabilities (like \"there is 75% chance this horse will win\"). After we convert to probabilties then if start has for example probability 0.8 and end has probability 0.6. Then the probability that this combination of start and end is the correct pair is 0.8 * 0.6 = 0.48 by probability product rule. There's a 48% chance. To me, the product of probabilities is more meaningful (and interpretable) score than adding logits (which i'm not sure what adding logits means).
Next i give start a weight of 1.2 because i think it is more important to have correct start than correct end. And this was verified by evaluating CV scores.
(Note expit above is sigmoid)
After reading top solutions it appears that using Muril-Large was one of the magics! What was Muril-Large CV score? (and perhaps using Rembert was helpful too).
Muril Large and TYDI QA Dataset
", "votes": 0, "replies": [] }, { "id": 1583771, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T03:52:57.807000", "content": "What is TYDI QA Dataset?
The one that has similar collection procedure like chaii dataset, showing in the \"Data\" description. It is also a great Magic according to other top solutions: )
", "votes": 1, "replies": [] }, { "id": 1585116, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-17T05:04:57.037000", "content": "Ah yes, that makes sense because the description page says that our competition data was prepared the same way as TydiQA dataset. So that external dataset should help our models learn the format of the data (even if it is different languages).
", "votes": 0, "replies": [] } ] }, { "id": 1583588, "author_name": "Kishal Mandal", "author_url": "", "post_date": "2021-11-16T00:39:20.317000", "content": "Thanks for the mention. I also tried using the sigmoid function on the logits to find the score and do a amjority voting among models. Couldn't finish. But to my surprise another single model (5folds) outperformed and could have been the 35th place solution :) My bad I didn't select any and two of my highest scoring notebooks got selected. So never trust LB (that had 0.75 LB score)
", "votes": 2, "replies": [ { "id": 1583597, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T00:46:25.067000", "content": "@kishalmandal In my early days, i trusted too many LB models and got shake down a lot. Now in my 2nd and 3rd year on Kaggle, i always build a \"trust your cv\" model. And in most competitions this model outperforms my \"trust your LB\" model.
\nThanks for your wonderful notebooks. This was a difficult competition to understand the data, understand external datasets, understand CV-LB gap, and other stuff. Your public notebooks helped everyone.
", "votes": 5, "replies": [] }, { "id": 1583602, "author_name": "Kishal Mandal", "author_url": "", "post_date": "2021-11-16T00:52:55.810000", "content": "I will keep that in mind. This was my first competition though :) Learned a lesson. Hope won't do the same blunder again. Thanks. I like your notebooks and all solutions and learn a lot from them. And I am big fan of yours :)
", "votes": 1, "replies": [] } ] }, { "id": 1590360, "author_name": "Parv Yadav", "author_url": "", "post_date": "2021-11-21T07:02:18.637000", "content": "Learnt a lot .Thanks @cdeotte !😄
", "votes": 0, "replies": [] } ], "comment_count": 24, "messages": [ { "id": 1583580, "postDate": "2021-11-16T00:33:31.570Z", "content": "Thank you Kaggle and host for a fun and interesting competition.
\nThank you @kishalmandal for the wonderful public notebooks. I joined a few days ago and immediately studied Kishal's top scoring public notebooks.
\nMy first final submission was my maximum public LB of 0.795. I took Kishal's LB 0.792 public notebook here and changed the weights to
start_logits = (start_logits1 + 0.85*start_logits2 + 0.85*start_logits3\n + start_logits4+ 1.7*start_logits5 )/5.4\nend_logits = (end_logits1 + 0.85*end_logits2 + 0.85*end_logits3 \n + end_logits4 + 1.7*end_logits5)/5.4\nThis achieved public LB 0.795 which was 60th place on public LB. But when i calculated the local CV score, it was only CV 0.677. This was very concerning. (And as it turns out, this submission had private LB 0.718 and had 728th place 😮 so it was overfitting public LB)
For my second final submission, I downloaded @kishalmandal train notebooks here and here. I optimized the hyperparameters, learning rates, and learning schedules and achieved local CV 0.697.
Next, I adjusted the popular public post process. I changed the first line which uses logits to the second line which uses the Bayesian product of probabilities (and gives start position a slight emphasis over end position).
\n\"score\": start_logits[start_index] + end_logits[end_index]\n\nfrom scipy.special import expit #THIS IS SIGMOID FUNCTION\n\"score\": expit(1.2*start_logits[start_index]) \\\n * expit(end_logits[end_index])\nThis boosted CV by +0.003, public LB by +0.003, and private LB by +0.004!
\nMy best final submission turned out to be my second submission which was my best CV and not my best LB. This model had CV 0.700, Public LB 0.784, and Private LB 0.744. Once again, \"trust you CV\" was the best strategy!
\nMy solution is only XLM-Roberta. After reading top solutions, it appears that ensemble with Muril-Large (and maybe Rembert) was the secret sauce to achieving a higher LB score! And TydiQA external dataset.
Congratulations !
\nGood Job!
", "rawMarkdown": "Congratulations !\n\nGood Job!", "votes": 1 }, { "id": 1589818, "postDate": "2021-11-20T15:48:20.303Z", "content": "Congratulation @cdeotte ! Thanks for sharing your solution!
", "rawMarkdown": "Congratulation @cdeotte ! Thanks for sharing your solution!", "votes": 1 }, { "id": 1585163, "postDate": "2021-11-17T06:10:22.093Z", "content": "Wow… This is so efficient solutions. I learned a lot. thanks!
", "rawMarkdown": "Wow... This is so efficient solutions. I learned a lot. thanks!", "votes": 1, "replies": [ { "id": 1586429, "postDate": "2021-11-18T04:24:50.177Z", "content": "Thanks Adldotori
", "rawMarkdown": "Thanks Adldotori " } ] }, { "id": 1584290, "postDate": "2021-11-16T12:23:48.397Z", "content": "@cdeotte congrats!
\nCan i ask about this moment?
\n\n\nMy first final submission was my maximum public LB of 0.795. I took Kishal's LB 0.792 public notebook here and changed the weights
\n
And how you came to this decision?
\nThank you in advance
", "rawMarkdown": "@cdeotte congrats!\n\nCan i ask about this moment?\n> My first final submission was my maximum public LB of 0.795. I took Kishal's LB 0.792 public notebook here and **changed the weights**\n\nAnd how you came to this decision? \n - is this intuition with experience?\n - accurate math calculation? \n - mixed? \n\nThank you in advance ", "votes": 1, "replies": [ { "id": 1584686, "postDate": "2021-11-16T17:29:22.777Z", "content": "Thanks. Are you asking how I found the optimal LB weights? I explain the procedure in one of the other comments.
", "rawMarkdown": "Thanks. Are you asking how I found the optimal LB weights? I explain the procedure in one of the other comments.", "votes": 1 }, { "id": 1585142, "postDate": "2021-11-17T05:51:04.567Z", "content": "Oh, thank you!
", "rawMarkdown": "Oh, thank you!", "votes": 1 } ] }, { "id": 1583846, "postDate": "2021-11-16T05:23:40.003Z", "content": "You make it look easy..
", "rawMarkdown": "You make it look easy.. ", "votes": 1, "replies": [ { "id": 1584045, "postDate": "2021-11-16T08:27:31.513Z", "content": "Thanks Yam
", "rawMarkdown": "Thanks Yam" } ] }, { "id": 1583794, "postDate": "2021-11-16T04:15:52.750Z", "content": "Hi @cdeotte congrats on your win. Can you tell us how did you decide to change weights?
\nstart_logits = (start_logits1 + 0.85*start_logits2 + 0.85*start_logits3\n + start_logits4+ 1.7*start_logits5 )/5.4\nend_logits = (end_logits1 + 0.85*end_logits2 + 0.85*end_logits3 \n + end_logits4 + 1.7*end_logits5)/5.4\nThanks
", "rawMarkdown": "Hi @cdeotte congrats on your win. Can you tell us how did you decide to change weights?\n```\nstart_logits = (start_logits1 + 0.85*start_logits2 + 0.85*start_logits3\n + start_logits4+ 1.7*start_logits5 )/5.4\nend_logits = (end_logits1 + 0.85*end_logits2 + 0.85*end_logits3 \n + end_logits4 + 1.7*end_logits5)/5.4\n```\nThanks ", "votes": 1, "replies": [ { "id": 1583797, "postDate": "2021-11-16T04:25:01.303Z", "content": "On first day, i make 5 submissions using the five weights below and get respective LB scores:
\nFrom above, i see that the strongest LB fold is fold 5 and the weakest LB folds are folds 2,3. So on second day I try to strengthen fold 5:
\nNext, I try to weaken folds 2,3 (and decrease fold 5 to keep in correct proportion):
\nNote that this is a \"trust your LB\" optimization and works in some competitions, but didn't work in this competition.
", "rawMarkdown": "On first day, i make 5 submissions using the five weights below and get respective LB scores:\n* 2,1,1,1,1 - LB 0.789\n* 1,2,1,1,1 - LB 0.787\n* 1,1,2,1,1 - LB 0.786\n* 1,1,1,2,1 - LB 0.789\n* 1,1,1,1,2 - LB 0.794\n\nFrom above, i see that the strongest LB fold is fold 5 and the weakest LB folds are folds 2,3. So on second day I try to strengthen fold 5:\n* 1,1,1,1,3 - LB 0.791\n\nNext, I try to weaken folds 2,3 (and decrease fold 5 to keep in correct proportion):\n* 1, 0.85, 0.85, 1, 1.7 - LB 0.795\n\nNote that this is a \"trust your LB\" optimization and works in some competitions, but didn't work in this competition.", "votes": 2 } ] }, { "id": 1583917, "postDate": "2021-11-16T06:27:34.540Z", "content": "Congratulations @cdeotte on results!
", "rawMarkdown": "Congratulations @cdeotte on results! ", "votes": 2, "replies": [ { "id": 1584044, "postDate": "2021-11-16T08:27:19.837Z", "content": "Thanks. Congratulations to you and your team for maintaining your great result on both public and private LB
", "rawMarkdown": "Thanks. Congratulations to you and your team for maintaining your great result on both public and private LB" } ] }, { "id": 1583852, "postDate": "2021-11-16T05:27:47.640Z", "content": "Hi @cdeotte great work , can you tell me the intuition behind this
\n\n\n\"score\": expit(1.2*start_logits[start_index]) \\
\n
\n * expit(end_logits[end_index])
Thanks
", "rawMarkdown": "Hi @cdeotte great work , can you tell me the intuition behind this \n\n> \"score\": expit(1.2*start_logits[start_index]) \\\n * expit(end_logits[end_index])\n\nThanks", "votes": 2, "replies": [ { "id": 1583860, "postDate": "2021-11-16T05:36:12.023Z", "content": "Logits are log odds (like \"there is log(3 to 1) chance this horse will win the race\"). And sigmoid(logits) are probabilities (like \"there is 75% chance this horse will win\"). After we convert to probabilties then if start has for example probability 0.8 and end has probability 0.6. Then the probability that this combination of start and end is the correct pair is 0.8 * 0.6 = 0.48 by probability product rule. There's a 48% chance. To me, the product of probabilities is more meaningful (and interpretable) score than adding logits (which i'm not sure what adding logits means).
Next i give start a weight of 1.2 because i think it is more important to have correct start than correct end. And this was verified by evaluating CV scores.
(Note expit above is sigmoid)
After reading top solutions it appears that using Muril-Large was one of the magics! What was Muril-Large CV score? (and perhaps using Rembert was helpful too).
Muril Large and TYDI QA Dataset
", "rawMarkdown": "Muril Large and TYDI QA Dataset " }, { "id": 1583771, "postDate": "2021-11-16T03:52:57.807Z", "content": "What is TYDI QA Dataset?
The one that has similar collection procedure like chaii dataset, showing in the \"Data\" description. It is also a great Magic according to other top solutions: )
", "rawMarkdown": "The one that has similar collection procedure like chaii dataset, showing in the \"Data\" description. It is also a great Magic according to other top solutions: )", "votes": 1 }, { "id": 1585116, "postDate": "2021-11-17T05:04:57.037Z", "content": "Ah yes, that makes sense because the description page says that our competition data was prepared the same way as TydiQA dataset. So that external dataset should help our models learn the format of the data (even if it is different languages).
", "rawMarkdown": "Ah yes, that makes sense because the description page says that our competition data was prepared the same way as TydiQA dataset. So that external dataset should help our models learn the format of the data (even if it is different languages)." } ] }, { "id": 1583588, "postDate": "2021-11-16T00:39:20.317Z", "content": "Thanks for the mention. I also tried using the sigmoid function on the logits to find the score and do a amjority voting among models. Couldn't finish. But to my surprise another single model (5folds) outperformed and could have been the 35th place solution :) My bad I didn't select any and two of my highest scoring notebooks got selected. So never trust LB (that had 0.75 LB score)
", "rawMarkdown": "Thanks for the mention. I also tried using the sigmoid function on the logits to find the score and do a amjority voting among models. Couldn't finish. But to my surprise another single model (5folds) outperformed and could have been the 35th place solution :) My bad I didn't select any and two of my highest scoring notebooks got selected. So never trust LB (that had 0.75 LB score)", "votes": 2, "replies": [ { "id": 1583597, "postDate": "2021-11-16T00:46:25.067Z", "content": "@kishalmandal In my early days, i trusted too many LB models and got shake down a lot. Now in my 2nd and 3rd year on Kaggle, i always build a \"trust your cv\" model. And in most competitions this model outperforms my \"trust your LB\" model.
\nThanks for your wonderful notebooks. This was a difficult competition to understand the data, understand external datasets, understand CV-LB gap, and other stuff. Your public notebooks helped everyone.
", "rawMarkdown": "@kishalmandal In my early days, i trusted too many LB models and got shake down a lot. Now in my 2nd and 3rd year on Kaggle, i always build a \"trust your cv\" model. And in most competitions this model outperforms my \"trust your LB\" model.\n\nThanks for your wonderful notebooks. This was a difficult competition to understand the data, understand external datasets, understand CV-LB gap, and other stuff. Your public notebooks helped everyone.", "votes": 5 }, { "id": 1583602, "postDate": "2021-11-16T00:52:55.810Z", "content": "I will keep that in mind. This was my first competition though :) Learned a lesson. Hope won't do the same blunder again. Thanks. I like your notebooks and all solutions and learn a lot from them. And I am big fan of yours :)
", "rawMarkdown": "I will keep that in mind. This was my first competition though :) Learned a lesson. Hope won't do the same blunder again. Thanks. I like your notebooks and all solutions and learn a lot from them. And I am big fan of yours :)", "votes": 1 } ] }, { "id": 1590360, "postDate": "2021-11-21T07:02:18.637Z", "content": "Learnt a lot .Thanks @cdeotte !😄
", "rawMarkdown": "Learnt a lot .Thanks @cdeotte !😄" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:20:57Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7671758, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/chris-deotte-36th-place-trust-your-cv-optimize-pp" }