![\"\"](\"https://i.imgur.com/sUBxnBI.png\")
I'd like to thank Kaggle and CommonLit for hosting this competition, as well has everyone here who has shared such great discussions and notebooks. I'm still in shock as I've always dreamed of a solo gold but never thought I'd actually get here!
\nThis was my second NLP competitions after the Tweet Sentiment Extraction competition last year. There was some similarity between the two, namely a) a small dataset and b) a fairly noisy target. The trick with the TSE competition was to use a 5-fold CV but to average multiple seeds (I used 5 for a total of 25 runs per experiment, which I would then average).
\nTo create my 5 folds, I used Abhishek's binning and stratification method. On average I was seeing about 0.01 gap between CV & public LB, but the correlation was good. I didn't bother submitting the majority of my single models as I was happy with my CV setup and I'm lazy 😂️.
\nMy best performing model was a deberta-large with a CV of 0.4738 (average of 5 seeds).
For all of my transformer models I used the AttentionBlock from this notebook, but applied it to the sequence last hidden state of the transformer.
n_hidden = self.config.hidden_size # from HF config\nself.seq_attn_head = nn.Sequential(\n nn.LayerNorm(n_hidden),\n AttentionBlock(n_hidden, n_hidden, 1),\n nn.Linear(n_hidden + n_additional_features, 2 if kl_loss else 1),\n)\nI also added a couple of features using textstat, flesch_reading_ease and smog_index and concatenated them to the output of the AttentionBlock before the regression head. These brought a tiny but repeatable improvement to my CV, but I'm not sure if this had any major impact on the final performance. I did a test during the late stages of the competition where I concatenated predictions of other models, and this didn't work any better than the ensemble methods below, which makes me think these features were doing very little.
I started with all of my models and iteratively removed the worst models until I could get down to the GPU time limit. (I found that this worked better than greedily adding models). I think this worked well in the end since the selection process would select models that will help the overall ensemble instead of models with the best CV.
\nI used 3 main methods for ensembling:
RidgeCV (which uses LeaveOneOut cross validation)BayesianRidgeRegression (with a LOO CV loop)My final ensembles usually had about 38 models to fit into the 3 hour run time and had quite a range of model architectures and CV scores.
\nI chose the RidgeCV & BayesianRidgeRegression methods as my final 2 submissions with the former scoring 0.447.
\nInterestingly, the Netflix method scored 0.446, but I had more faith in the CV scores from LOO CV, as I was worried about leakage using the Netflix method.
| Model | \nCV | \n
|---|---|
| microsoft/deberta-large | \n0.484005 | \n
| microsoft/deberta-base | \n0.486186 | \n
| deepset/roberta-base-squad2 | \n0.495078 | \n
| deepset/roberta-base-squad2 | \n0.501851 | \n
| deepset/roberta-large-squad2 | \n0.482696 | \n
| distilroberta-base | \n0.505654 | \n
| funnel-transformer/large-base | \n0.495666 | \n
| bert-large-uncased | \n0.524264 | \n
| bert-large-uncased | \n0.538173 | \n
| facebook/bart-base | \n0.534491 | \n
| facebook/bart-base | \n0.556662 | \n
| facebook/bart-large | \n0.543126 | \n
| facebook/bart-large | \n0.526896 | \n
| roberta-base | \n0.49835 | \n
| microsoft/deberta-large | \n0.474008 | \n
| microsoft/deberta-large | \n0.471707 | \n
| distilroberta-base | \n0.502802 | \n
| albert-large-v2 | \n0.545104 | \n
| albert-large-v2 | \n0.506411 | \n
| funnel-transformer/large-base | \n0.493789 | \n
| funnel-transformer/large-base | \n0.491098 | \n
| microsoft/deberta-base | \n0.499093 | \n
| microsoft/deberta-base | \n0.509152 | \n
| deepset/roberta-base-squad2 | \n0.490829 | \n
| deepset/roberta-large-squad2 | \n0.489583 | \n
| deepset/roberta-large-squad2 | \n0.494185 | \n
| funnel-transformer/large-base | \n0.523333 | \n
| albert-large-v2 | \n0.508817 | \n
| albert-large-v2 | \n0.529876 | \n
| sentence-transformers/LaBSE | \n0.525731 | \n
| microsoft/deberta-large | \n0.471197 | \n
| microsoft/deberta-large | \n0.475243 | \n
| bert-large-cased-whole-word-masking | \n0.514011 | \n
| bert-large-cased | \n0.505559 | \n
| xlm-roberta-large | \n0.505576 | \n
| facebook/bart-base | \n0.530183 | \n
| google/electra-large-discriminator | \n0.514033 | \n
| sentence-transformers/paraphrase-mpnet-base-v2 | \n0.510096 | \n
Code repository and submission notebook is now public :)
", "votes": 5, "replies": [] }, { "id": 1448914, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-04T19:10:39.200000", "content": "Congratulations @anjum48 Great job! Can you explain how to do LeaveOneOut cross validation? If the train data has 2500 rows, doesn't that require building 2500 models? (i.e. for each row, you build a model using all data except that row, and infer that row with that model).
Thanks very much @cdeotte! Normally when ensembling I try to keep the same folds for each model so that I can use the OOF predictions to create a blending model. In this case, since I had models trained with different seeds for CV this made things a little tricky and I wasn't sure if a normal 5 fold CV would be reliable.
\nMy default method was to build a dataset using all my OOF predictions and then use RidgeCV. If cv=None, by default it uses an efficient Leave-One-Out Cross-Validation (the code is here and it's quite interesting to see how it works). My understanding is that it uses LOO CV to select the best alpha and then fit the weights using the full X dataset. The CV score is then the best_score_ attribute (which is negative MSE).
I also tried to replicate this with other sklearn linear models, but using a manual LeaveOneOut CV loop, which was slower, but still gave me good CV/LB correlation. It was these CV scores I used to iteratively remove models for the final ensemble. In the final submission notebook I would just use fit & predict using the OOFs for the chosen models.
| Model | \nCV | \nPublic | \nPrivate | \n
|---|---|---|---|
| BayesianRidge | \n0.44256 | \n0.452 | \n0.448 | \n
| RidgeCV | \n0.44254 | \n0.451 | \n0.447 | \n
Cool. Thanks for the code link. Great job!
", "votes": 1, "replies": [] } ] }, { "id": 1444733, "author_name": "Mohammed Deeb", "author_url": "", "post_date": "2021-08-04T06:47:35.620000", "content": "Congrats. We know how much effort you have done to reach this solution. Many methods have been tested to be in the top 1% in any competitions. Great job. 💪😎
", "votes": 1, "replies": [] }, { "id": 1429713, "author_name": "Miguel González González", "author_url": "", "post_date": "2021-08-03T10:54:41.083000", "content": "Impressive! Congrats and thanks for sharing. The /kaggle/working trick is something I didn't know!
", "votes": 1, "replies": [] }, { "id": 1429270, "author_name": "Tim Yee", "author_url": "", "post_date": "2021-08-03T10:29:32.433000", "content": "Congrats on solo gold and great solution!
\nDid you average 25 models (5 seeds x 5 folds) and then perform RidgeCV w/ LOO? I had a lot of success stacking with Ridge early on and I found that averaging 5 folds prior to the ensembling stage performed better on LB and I later attributed it to leaking useful information from validation to training.
", "votes": 1, "replies": [ { "id": 1429602, "author_name": "datasaurus", "author_url": "", "post_date": "2021-08-03T10:49:27.123000", "content": "Thanks Tim! I averaged the 5 folds prior to the ensemble and treated the 5 seeds as independent models for the purpose of the ensemble (but only selected the models that improved the ensemble). This meant there were 38 * 5 = 195 model checkpoints.
\nMy final RidgeCV score was 0.44254 with an alpha of 0.01
", "votes": 2, "replies": [] } ] }, { "id": 1429167, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-08-03T10:23:37.620000", "content": "How did you handled the limited private storage offered by Kaggle? I guess 36 models should occupy more than 100Gb.
", "votes": 1, "replies": [ { "id": 1429500, "author_name": "datasaurus", "author_url": "", "post_date": "2021-08-03T10:44:06.213000", "content": "I used the method described by Nischay Dhankhar in this discussion.
\nEssentially anything that gets saved into the /kaggle/output/ section of a notebook doesn't count towards your 100GB (seems like a weird loophole but it works). I just connected my Google Cloud Bucket and downloaded my model weights to that folder
", "votes": 2, "replies": [] }, { "id": 1429534, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-08-03T10:45:44.357000", "content": "If you save them as output of a notebook there isn't a limit
", "votes": 0, "replies": [] }, { "id": 1430276, "author_name": "Tim Yee", "author_url": "", "post_date": "2021-08-03T11:28:42.573000", "content": "Not long ago, the notebook output limit was 5GB. I remember very clearly from Pulmonary Embolism because I tried to use 40 notebooks to create resized images for the competition.
", "votes": 0, "replies": [] }, { "id": 1433295, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-08-03T13:54:08.310000", "content": "That's a really nice trick! Thank you.
", "votes": 0, "replies": [] } ] }, { "id": 1450799, "author_name": "spy1234567", "author_url": "", "post_date": "2021-08-05T07:18:26.570000", "content": "Hi, @anjum48 , Congratulations! Thanks for your sharing.
\nI have a question: you said that you use 5 fold and each has 5 seed, this means that you have 25 trained models for a single model structure, after training, how did you make your prediction? to use the result of all of the 25 models or just use the result of the best model of these 25 models?
Hi @spy1234567, yes I used 25 model checkpoints to determine if a change to heads/features/hyperparams etc. actually made an improvement that was worth keeping.
\nBut for my final prediction, I treated each of the 5 folds as a single model and averaged the predictions. So in the table above, you can see some models appear multiple times due to being from different seeds (or some other change).
", "votes": 0, "replies": [] }, { "id": 1451798, "author_name": "spy1234567", "author_url": "", "post_date": "2021-08-05T12:43:51.310000", "content": "Hi, @anjum48 , do you mean that you treat 5 folds (each has 5 seeds) as 25 single model and choose the best model among them?
", "votes": 0, "replies": [] }, { "id": 1452034, "author_name": "datasaurus", "author_url": "", "post_date": "2021-08-05T14:01:35.770000", "content": "Perhaps a picture will help :)
\nEach \"timestamp\" or row is a single model run with a single seed run for 5 folds. Each \"group name\" is a group of 5 identical models run with different seeds.
\nI would take the mean RMSE of the group (25 model checkpoints) to determine if a change made something better or worse. This is how I calculated my CV score.
\nBut I would treat each timestamp as a unique model in my ensemble (averaging the 5 folds), but only using that model if it improved my ensemble CV.
\n
@anjum48 thanks again for your patience and detailed reply!👍
", "votes": 1, "replies": [] } ] }, { "id": 1443594, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-04T03:31:57.367000", "content": "Congrats @anjum48 on 4th place. Well done
", "votes": 2, "replies": [] }, { "id": 1426586, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T07:42:27.390000", "content": "great job. I could see by the count of your submissions during the competition that you seem to be fairly confident of your CV..
\nAlso interesting to see you experiments with kl_loss
", "votes": 2, "replies": [ { "id": 1427485, "author_name": "datasaurus", "author_url": "", "post_date": "2021-08-03T08:29:03.767000", "content": "Thank you! I actually forgot to mention KL Loss above.
\nI tried the loss from this discussion but didn't really see gains on CV. However, there were some models which were difficult to train (the loss would suddenly jump up after a couple of epochs) and this loss function appeared to help slightly. MSE was my default loss for most models though
", "votes": 2, "replies": [] } ] }, { "id": 1426127, "author_name": "Salim Khazem", "author_url": "", "post_date": "2021-08-03T07:18:59.550000", "content": "Congratulations, thanks for sharing your approach
", "votes": 2, "replies": [] }, { "id": 1455843, "author_name": "Amol Ambkar", "author_url": "", "post_date": "2021-08-06T17:51:32.410000", "content": "Great One…
", "votes": 0, "replies": [] }, { "id": 1453440, "author_name": "lyo_gavin", "author_url": "", "post_date": "2021-08-05T21:20:13.367000", "content": "Congrats! Great work! Learned a lot. Your method is probably the most replicable solution independent to the specific task.
", "votes": 0, "replies": [] }, { "id": 1451855, "author_name": "arpita dutta", "author_url": "", "post_date": "2021-08-05T12:59:17.030000", "content": "Congratulations! Thank you for providing this information.
", "votes": 0, "replies": [] }, { "id": 1444881, "author_name": "Shubham Singh", "author_url": "", "post_date": "2021-08-04T07:14:50.567000", "content": "Congrats and thanks for sharing .
", "votes": 1, "replies": [] }, { "id": 1427874, "author_name": "SSinyu", "author_url": "", "post_date": "2021-08-03T08:52:42.010000", "content": "Thank you for sharing, Congratulations!
", "votes": 1, "replies": [] }, { "id": 1453084, "author_name": "Shubham Chaurasia", "author_url": "", "post_date": "2021-08-05T18:54:13.587000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1451677, "author_name": "Devashree Madhugiri", "author_url": "", "post_date": "2021-08-05T12:02:18.717000", "content": "Congratulations! Thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 29, "messages": [ { "id": 1425964, "postDate": "2021-08-03T07:11:25.213Z", "content": "I'd like to thank Kaggle and CommonLit for hosting this competition, as well has everyone here who has shared such great discussions and notebooks. I'm still in shock as I've always dreamed of a solo gold but never thought I'd actually get here!
\nThis was my second NLP competitions after the Tweet Sentiment Extraction competition last year. There was some similarity between the two, namely a) a small dataset and b) a fairly noisy target. The trick with the TSE competition was to use a 5-fold CV but to average multiple seeds (I used 5 for a total of 25 runs per experiment, which I would then average).
\nTo create my 5 folds, I used Abhishek's binning and stratification method. On average I was seeing about 0.01 gap between CV & public LB, but the correlation was good. I didn't bother submitting the majority of my single models as I was happy with my CV setup and I'm lazy 😂️.
\nMy best performing model was a deberta-large with a CV of 0.4738 (average of 5 seeds).
For all of my transformer models I used the AttentionBlock from this notebook, but applied it to the sequence last hidden state of the transformer.
n_hidden = self.config.hidden_size # from HF config\nself.seq_attn_head = nn.Sequential(\n nn.LayerNorm(n_hidden),\n AttentionBlock(n_hidden, n_hidden, 1),\n nn.Linear(n_hidden + n_additional_features, 2 if kl_loss else 1),\n)\nI also added a couple of features using textstat, flesch_reading_ease and smog_index and concatenated them to the output of the AttentionBlock before the regression head. These brought a tiny but repeatable improvement to my CV, but I'm not sure if this had any major impact on the final performance. I did a test during the late stages of the competition where I concatenated predictions of other models, and this didn't work any better than the ensemble methods below, which makes me think these features were doing very little.
I started with all of my models and iteratively removed the worst models until I could get down to the GPU time limit. (I found that this worked better than greedily adding models). I think this worked well in the end since the selection process would select models that will help the overall ensemble instead of models with the best CV.
\nI used 3 main methods for ensembling:
RidgeCV (which uses LeaveOneOut cross validation)BayesianRidgeRegression (with a LOO CV loop)My final ensembles usually had about 38 models to fit into the 3 hour run time and had quite a range of model architectures and CV scores.
\nI chose the RidgeCV & BayesianRidgeRegression methods as my final 2 submissions with the former scoring 0.447.
\nInterestingly, the Netflix method scored 0.446, but I had more faith in the CV scores from LOO CV, as I was worried about leakage using the Netflix method.
| Model | \nCV | \n
|---|---|
| microsoft/deberta-large | \n0.484005 | \n
| microsoft/deberta-base | \n0.486186 | \n
| deepset/roberta-base-squad2 | \n0.495078 | \n
| deepset/roberta-base-squad2 | \n0.501851 | \n
| deepset/roberta-large-squad2 | \n0.482696 | \n
| distilroberta-base | \n0.505654 | \n
| funnel-transformer/large-base | \n0.495666 | \n
| bert-large-uncased | \n0.524264 | \n
| bert-large-uncased | \n0.538173 | \n
| facebook/bart-base | \n0.534491 | \n
| facebook/bart-base | \n0.556662 | \n
| facebook/bart-large | \n0.543126 | \n
| facebook/bart-large | \n0.526896 | \n
| roberta-base | \n0.49835 | \n
| microsoft/deberta-large | \n0.474008 | \n
| microsoft/deberta-large | \n0.471707 | \n
| distilroberta-base | \n0.502802 | \n
| albert-large-v2 | \n0.545104 | \n
| albert-large-v2 | \n0.506411 | \n
| funnel-transformer/large-base | \n0.493789 | \n
| funnel-transformer/large-base | \n0.491098 | \n
| microsoft/deberta-base | \n0.499093 | \n
| microsoft/deberta-base | \n0.509152 | \n
| deepset/roberta-base-squad2 | \n0.490829 | \n
| deepset/roberta-large-squad2 | \n0.489583 | \n
| deepset/roberta-large-squad2 | \n0.494185 | \n
| funnel-transformer/large-base | \n0.523333 | \n
| albert-large-v2 | \n0.508817 | \n
| albert-large-v2 | \n0.529876 | \n
| sentence-transformers/LaBSE | \n0.525731 | \n
| microsoft/deberta-large | \n0.471197 | \n
| microsoft/deberta-large | \n0.475243 | \n
| bert-large-cased-whole-word-masking | \n0.514011 | \n
| bert-large-cased | \n0.505559 | \n
| xlm-roberta-large | \n0.505576 | \n
| facebook/bart-base | \n0.530183 | \n
| google/electra-large-discriminator | \n0.514033 | \n
| sentence-transformers/paraphrase-mpnet-base-v2 | \n0.510096 | \n
Code repository and submission notebook is now public :)
", "rawMarkdown": "Code repository and submission notebook is now public :)", "votes": 5 }, { "id": 1448914, "postDate": "2021-08-04T19:10:39.200Z", "content": "Congratulations @anjum48 Great job! Can you explain how to do LeaveOneOut cross validation? If the train data has 2500 rows, doesn't that require building 2500 models? (i.e. for each row, you build a model using all data except that row, and infer that row with that model).
Thanks very much @cdeotte! Normally when ensembling I try to keep the same folds for each model so that I can use the OOF predictions to create a blending model. In this case, since I had models trained with different seeds for CV this made things a little tricky and I wasn't sure if a normal 5 fold CV would be reliable.
\nMy default method was to build a dataset using all my OOF predictions and then use RidgeCV. If cv=None, by default it uses an efficient Leave-One-Out Cross-Validation (the code is here and it's quite interesting to see how it works). My understanding is that it uses LOO CV to select the best alpha and then fit the weights using the full X dataset. The CV score is then the best_score_ attribute (which is negative MSE).
I also tried to replicate this with other sklearn linear models, but using a manual LeaveOneOut CV loop, which was slower, but still gave me good CV/LB correlation. It was these CV scores I used to iteratively remove models for the final ensemble. In the final submission notebook I would just use fit & predict using the OOFs for the chosen models.
| Model | \nCV | \nPublic | \nPrivate | \n
|---|---|---|---|
| BayesianRidge | \n0.44256 | \n0.452 | \n0.448 | \n
| RidgeCV | \n0.44254 | \n0.451 | \n0.447 | \n
Cool. Thanks for the code link. Great job!
", "rawMarkdown": "Cool. Thanks for the code link. Great job!", "votes": 1 } ] }, { "id": 1444733, "postDate": "2021-08-04T06:47:35.620Z", "content": "Congrats. We know how much effort you have done to reach this solution. Many methods have been tested to be in the top 1% in any competitions. Great job. 💪😎
", "rawMarkdown": "Congrats. We know how much effort you have done to reach this solution. Many methods have been tested to be in the top 1% in any competitions. Great job. 💪😎", "votes": 1 }, { "id": 1429713, "postDate": "2021-08-03T10:54:41.083Z", "content": "Impressive! Congrats and thanks for sharing. The /kaggle/working trick is something I didn't know!
", "rawMarkdown": "Impressive! Congrats and thanks for sharing. The /kaggle/working trick is something I didn't know!", "votes": 1 }, { "id": 1429270, "postDate": "2021-08-03T10:29:32.433Z", "content": "Congrats on solo gold and great solution!
\nDid you average 25 models (5 seeds x 5 folds) and then perform RidgeCV w/ LOO? I had a lot of success stacking with Ridge early on and I found that averaging 5 folds prior to the ensembling stage performed better on LB and I later attributed it to leaking useful information from validation to training.
", "rawMarkdown": "Congrats on solo gold and great solution! \n\nDid you average 25 models (5 seeds x 5 folds) and then perform RidgeCV w/ LOO? I had a lot of success stacking with Ridge early on and I found that averaging 5 folds prior to the ensembling stage performed better on LB and I later attributed it to leaking useful information from validation to training.", "votes": 1, "replies": [ { "id": 1429602, "postDate": "2021-08-03T10:49:27.123Z", "content": "Thanks Tim! I averaged the 5 folds prior to the ensemble and treated the 5 seeds as independent models for the purpose of the ensemble (but only selected the models that improved the ensemble). This meant there were 38 * 5 = 195 model checkpoints.
\nMy final RidgeCV score was 0.44254 with an alpha of 0.01
", "rawMarkdown": "Thanks Tim! I averaged the 5 folds prior to the ensemble and treated the 5 seeds as independent models for the purpose of the ensemble (but only selected the models that improved the ensemble). This meant there were 38 * 5 = 195 model checkpoints.\n\nMy final RidgeCV score was 0.44254 with an alpha of 0.01", "votes": 2 } ] }, { "id": 1429167, "postDate": "2021-08-03T10:23:37.620Z", "content": "How did you handled the limited private storage offered by Kaggle? I guess 36 models should occupy more than 100Gb.
", "rawMarkdown": "How did you handled the limited private storage offered by Kaggle? I guess 36 models should occupy more than 100Gb.", "votes": 1, "replies": [ { "id": 1429500, "postDate": "2021-08-03T10:44:06.213Z", "content": "I used the method described by Nischay Dhankhar in this discussion.
\nEssentially anything that gets saved into the /kaggle/output/ section of a notebook doesn't count towards your 100GB (seems like a weird loophole but it works). I just connected my Google Cloud Bucket and downloaded my model weights to that folder
", "rawMarkdown": "I used the method described by Nischay Dhankhar in this [discussion](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/249689).\n\nEssentially anything that gets saved into the /kaggle/output/ section of a notebook doesn't count towards your 100GB (seems like a weird loophole but it works). I just connected my Google Cloud Bucket and downloaded my model weights to that folder", "votes": 2 }, { "id": 1429534, "postDate": "2021-08-03T10:45:44.357Z", "content": "If you save them as output of a notebook there isn't a limit
", "rawMarkdown": "If you save them as output of a notebook there isn't a limit" }, { "id": 1430276, "postDate": "2021-08-03T11:28:42.573Z", "content": "Not long ago, the notebook output limit was 5GB. I remember very clearly from Pulmonary Embolism because I tried to use 40 notebooks to create resized images for the competition.
", "rawMarkdown": "Not long ago, the notebook output limit was 5GB. I remember very clearly from Pulmonary Embolism because I tried to use 40 notebooks to create resized images for the competition." }, { "id": 1433295, "postDate": "2021-08-03T13:54:08.310Z", "content": "That's a really nice trick! Thank you.
", "rawMarkdown": "That's a really nice trick! Thank you." } ] }, { "id": 1450799, "postDate": "2021-08-05T07:18:26.570Z", "content": "Hi, @anjum48 , Congratulations! Thanks for your sharing.
\nI have a question: you said that you use 5 fold and each has 5 seed, this means that you have 25 trained models for a single model structure, after training, how did you make your prediction? to use the result of all of the 25 models or just use the result of the best model of these 25 models?
Hi @spy1234567, yes I used 25 model checkpoints to determine if a change to heads/features/hyperparams etc. actually made an improvement that was worth keeping.
\nBut for my final prediction, I treated each of the 5 folds as a single model and averaged the predictions. So in the table above, you can see some models appear multiple times due to being from different seeds (or some other change).
", "rawMarkdown": "Hi @spy1234567, yes I used 25 model checkpoints to determine if a change to heads/features/hyperparams etc. actually made an improvement that was worth keeping.\n\nBut for my final prediction, I treated each of the 5 folds as a single model and averaged the predictions. So in the table above, you can see some models appear multiple times due to being from different seeds (or some other change)." }, { "id": 1451798, "postDate": "2021-08-05T12:43:51.310Z", "content": "Hi, @anjum48 , do you mean that you treat 5 folds (each has 5 seeds) as 25 single model and choose the best model among them?
", "rawMarkdown": "Hi, @anjum48 , do you mean that you treat 5 folds (each has 5 seeds) as 25 single model and choose the best model among them?" }, { "id": 1452034, "postDate": "2021-08-05T14:01:35.770Z", "content": "Perhaps a picture will help :)
\nEach \"timestamp\" or row is a single model run with a single seed run for 5 folds. Each \"group name\" is a group of 5 identical models run with different seeds.
\nI would take the mean RMSE of the group (25 model checkpoints) to determine if a change made something better or worse. This is how I calculated my CV score.
\nBut I would treat each timestamp as a unique model in my ensemble (averaging the 5 folds), but only using that model if it improved my ensemble CV.
\n
@anjum48 thanks again for your patience and detailed reply!👍
", "rawMarkdown": "@anjum48 thanks again for your patience and detailed reply!👍", "votes": 1 } ] }, { "id": 1443594, "postDate": "2021-08-04T03:31:57.367Z", "content": "Congrats @anjum48 on 4th place. Well done
", "rawMarkdown": "Congrats @anjum48 on 4th place. Well done", "votes": 2 }, { "id": 1426586, "postDate": "2021-08-03T07:42:27.390Z", "content": "great job. I could see by the count of your submissions during the competition that you seem to be fairly confident of your CV..
\nAlso interesting to see you experiments with kl_loss
", "rawMarkdown": "great job. I could see by the count of your submissions during the competition that you seem to be fairly confident of your CV..\n\nAlso interesting to see you experiments with kl_loss", "votes": 2, "replies": [ { "id": 1427485, "postDate": "2021-08-03T08:29:03.767Z", "content": "Thank you! I actually forgot to mention KL Loss above.
\nI tried the loss from this discussion but didn't really see gains on CV. However, there were some models which were difficult to train (the loss would suddenly jump up after a couple of epochs) and this loss function appeared to help slightly. MSE was my default loss for most models though
", "rawMarkdown": "Thank you! I actually forgot to mention KL Loss above.\n\nI tried the loss from [this discussion](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/239421#1310794) but didn't really see gains on CV. However, there were some models which were difficult to train (the loss would suddenly jump up after a couple of epochs) and this loss function appeared to help slightly. MSE was my default loss for most models though", "votes": 2 } ] }, { "id": 1426127, "postDate": "2021-08-03T07:18:59.550Z", "content": "Congratulations, thanks for sharing your approach
", "rawMarkdown": "Congratulations, thanks for sharing your approach ", "votes": 2 }, { "id": 1455843, "postDate": "2021-08-06T17:51:32.410Z", "content": "Great One…
", "rawMarkdown": "Great One...\n" }, { "id": 1453440, "postDate": "2021-08-05T21:20:13.367Z", "content": "Congrats! Great work! Learned a lot. Your method is probably the most replicable solution independent to the specific task.
", "rawMarkdown": "Congrats! Great work! Learned a lot. Your method is probably the most replicable solution independent to the specific task." }, { "id": 1451855, "postDate": "2021-08-05T12:59:17.030Z", "content": "Congratulations! Thank you for providing this information.
", "rawMarkdown": "Congratulations! Thank you for providing this information." }, { "id": 1444881, "postDate": "2021-08-04T07:14:50.567Z", "content": "Congrats and thanks for sharing .
", "rawMarkdown": "Congrats and thanks for sharing .", "votes": 1 }, { "id": 1427874, "postDate": "2021-08-03T08:52:42.010Z", "content": "Thank you for sharing, Congratulations!
", "rawMarkdown": "Thank you for sharing, Congratulations!", "votes": 1 }, { "id": 1453084, "postDate": "2021-08-05T18:54:13.587Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1451677, "postDate": "2021-08-05T12:02:18.717Z", "content": "Congratulations! Thanks for sharing.
", "rawMarkdown": "Congratulations! Thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:46:32Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6823320, "solution_url": "/competitions/commonlitreadabilityprize/writeups/datasaurus-4th-place-solution-0-447" }