![](\"https://i.imgur.com/ANcnHNt.png\")
First of all, we would like to thanks Kaggle & CommonLit for hosting this interesting competition, this has been an interesting journey for us! Also congratulations to all winning teams!
\nAnd special thanks to @rhtsingh, @maunish, and @andretugan for their knowledgeable and resourceful notebooks. Since our team is fairly new to transformers, leveraging the resources from public notebooks definitely helps in our journey!
\nOur final submission consists of 9 transformer models, and our final solution is based on our 2nd-stage model Gaussian Process Regression (GPR) [1] which we use to correct overconfident transformer predictions due to the small training data. This can be summarized in two steps.
\n
We scraped data from the “url_legal” information which is available in the training data and only use those with licenses of “CC BY” and “CC BY-SA”.
\nLike every other team, we first started out fine-tuning transformers model and focus on increasing the single model LB score by training different transformers for diversity. Since our goal is to use GPR to improve the transformer predictions, we need robust transformer models that can generate robust embeddings.
\nOur final submission consists of the following 9 transformer models and all models are trained using 5-fold stratified CV with attention head.
\n| Model | \nCV | \nPublic | \nPrivate | \nNotes | \n
|---|---|---|---|---|
| roberta-large | \n0.492 | \n0.471 | \n0.471 | \nmlm on training set | \n
| microsoft/deberta-large | \n0.485 | \n0.474 | \n0.476 | \nmlm on training set | \n
| xlnet-large-cased | \n0.494 | \n0.475 | \n0.476 | \n- | \n
| deepset/roberta-large-squad2 | \n0.488 | \n0.464 | \n0.467 | \n- | \n
| deepset/roberta-large-squad2 | \n0.484 | \n0.466 | \n0.464 | \nmlm on train set and external data | \n
| allenai/longformer-large-4096-finetuned-triviaqa | \n0.489 | \n0.467 | \n0.47 | \n- | \n
| valhalla/bart-large-finetuned-squadv1 | \n0.471 | \n0.462 | \n0.466 | \nmlm on train set and external data, remove dropout | \n
| microsoft/deberta-large-mnli | \n0.469 | \n0.462 | \n0.469 | \nmlm on train set and external data, remove dropout | \n
| ahotrod/electra_large_discriminator_squad2_512 | \n0.477 | \n0.468 | \n0.468 | \nremove dropout | \n
The model selection is based on the individual LB score (preferably <= 0.475)
\nThe following are the fine-tuned strategy for the transformer models
\nWe are aware of the limited training samples ~2.8k, hence we immediately thought of Gaussian process regression (GPR). Given the constraint of limited training samples, using Bayesian techniques such as GPR could potentially help us make a better approximation/prediction. Since Bayesian/Probabilistic model are proven to performs relatively well in small setting, and it also has a strong notion of uncertainty. The uncertainty is important here as we know the distribution of the training and test data are different, and we do not want the transformer to make overconfident predictions, hence GPR alleviates overconfident predictions from transformers. A NN with one hidden layer of infinite width is basically a Gaussian Processes.
\nThe below figure shows the deberta-large model predictions for 100 randomly sampled points. The green curve is the deberta-large predictions, and the red curve is the GPR predictions. We can see that deberta-large will try to fit all the points as closely as possible (Red circle), however, GPR will avoid overconfident predictions and will regress to the mean of nearby samples when it is not confident, thus providing reliable uncertainty estimation.
\n![](\"https://i.imgur.com/yMlaKpG.png\")
We use GPR to correct overconfident transformer predictions. Doing this allows us to further improved the single model score significantly by training a GPR model using all the OOF embeddings extracted from the last transformer layer (attention-head).
\nThe following submission shows that using deberta-large trained with pseudo-labeling (PL) can achieve a single private score of 0.459, but with GPR it can achieve 0.453 alone!
\n| Model | \nCV | \nPublic | \nPrivate | \nGPR Public | \nGPR private | \nNotes | \n
|---|---|---|---|---|---|---|
| deberta-large | \n0.462 | \n0.465 | \n0.459 | \n0.457 | \n0.453 | \ntrain with pseudo labeling | \n
Unfortunately, we didn't include any model trained with pseudo-labeling since many of our PL-trained models has bad single/ensemble LB score as compared to the model trained without pseudo-labeling..
\nThis leads us to believe the LB has a different data distribution as compared to the training data, hence we use all the OOF embeddings for training and only validate the individual GPR model performance on LB.
\nFrom the transformer models section above, the 9 models were selected in terms of their individual LB scores and correlations for diversity. We extracted the attention head embeddings which is of size 1024 for each model, hence 1024*9 embeddings for 9 models. We concatenate all 1024*9 embeddings and train all samples using GPR model with an initial RBF kernel hyperparameters of lengthscale=3, variance=1, noise=1, and run for 1000 steps using stochastic variational inference (SVI). Since we are doing inference with GPR, we are able to learn the GPR/kernel hyperparameters directly from the embeddings itself with minimal fine-tuning.
\nOur final submission with GPR model achieved public 0.448, private 0.448
\nDropout removal
\nThe remove dropout note in the table above refers to removing dropout in the transformer architecture, instead of the dropout placed in the output head. We were inspired by this kernel. It is a very interesting idea to remove it as the default dropout of 0.1 is used in nearly every transformer model. Previously, we can see that the validation loss is very unstable, which is why a small evaluation interval is commonly used. But after removing dropout in the transformers, the val loss became much more steady, and the converged points were usually much better. The cv score improved by a large margin, but the public LB doesn't reflect the change accordingly. Later we found more and more people used this strategy, so we just adopted it in some of our models.
Pseudo labelling
\nWe scraped the data from the same url in the training data and labelled them with an ensemble model which scored LB 0.450. We didn't train on the pseudo data first and finetune with the competition data. Instead, during each fold training, we only added the pseudo data which have the same id as the training data in that fold, to avoid potential data leakage. With this method, the CV score improved a lot but the public LB didn't agree. Though we didn't found data leakage in the cv, we abandoned the models trained with pseudo labelling. It turned out that it works well for the private set, and we should've trusted the cv more. The public test set is only ~600 samples so it doesn't reflect the data distribution in the private set well.
Pretrained models
\nWe found that some of the models further pretrained on question answering datasets are better for finetuning in this competition such as squad and triviaqa. So it's always better to try different pretrained models when using transformer models.
Gaussian process regression
\nThis is the most important ensembling/preprocessing/correction steps for our solution. All of our single transformer models improved significantly with gains from 0.002 - 0.01 after training a GPR model using the attention-head embedding.
Extreme data and post-processing
\nWe found that the data with the targets < -3 or > 1 are the ones that make the training unstable. There are only 131 of these samples in the training set. Their standard errors are also usually higher than the others. By removing these samples, the cv improved a lot. And we found that there are also some patterns in the targets of the extreme data that we can use for post-processing, only if we can detect these extreme samples. But we failed to build a model to achieve that.
Using different heads
\nWe tried training transformers using different heads, e.g. cls, mean pooling, pooler, concatenate last 4/8/12 layers, mean max pooling. But attention head works best for us, hence we stick with attention-head for all our models.
Different optimization strategies
\nInitially, we also tried mixout regularization and multi-sample dropout, doing this stabilizes training, but the individual LB score is bad. And in the end, the transformers w/o dropout works better.
We should have trusted our CV for the models trained with pseudo-labeling. Our best submission has public 0.452, private 0.446 which is achieved by using only 6 models trained with pseudo-labeling. Unfortunately, we didn't select that for our final submission since the public LB is bad..
\nReferences:
\n[1] https://pyro.ai/examples/gp.html
I would like to share some more insight about our journey.
\nDropout removal
\nThe remove dropout note in the table above refers to removing dropout in the transformer architecture, instead of the dropout placed in the output head. We were inspired by this kernel. It is a very interesting idea to remove it as the default dropout of 0.1 is used in nearly every transformer model. Previously, we can see that the validation loss is very unstable, which is why a small evaluation interval is commonly used. But after removing dropout in the transformers, the val loss became much more steady, and the converged points were usually much better. The cv score improved by a large margin, but the public LB doesn't reflect the change accordingly. Later we found more and more people used this strategy, so we just adopted it in some of our models.
Pseudo labelling
\nWe scraped the data from the same url in the training data and labelled them with an ensemble model which scored LB 0.450. We didn't train on the pseudo data first and finetune with the competition data. Instead, during each fold training, we only added the pseudo data which have the same id as the training data in that fold, to avoid potential data leakage. With this method, the CV score improved a lot but the public LB didn't agree. Though we didn't found data leakage in the cv, we abandoned the models trained with pseudo labelling. It turned out that it works well for the private set, and we should've trusted the cv more. The public test set is only ~600 samples so it doesn't reflect the data distribution in the private set well.
Pretrained models
\nWe found that some of the models further pretrained on question answering datasets are better for finetuning in this competition such as squad and triviaqa. So it's always better to try different pretrained models when using transformer models.
Extreme data and post-processing
\nWe found that the data with the targets < -3 or > 1 are the ones that make the training unstable. There are only 131 these samples in the training set. Their standard errors are also usually higher than the others. By removing these samples, the cv improved a lot. And we found that there are also some patterns in the targets of the extreme data that we can use for post-processing, only if we can detect these extreme samples. But we failed to build a model to achieve that.
Thanks @xfffrank for your great contribution to the team! I've added the additional information to our post :)
", "votes": 1, "replies": [] }, { "id": 1454160, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-06T05:43:16.397000", "content": "actually last comment is very interesting. Do you think it would have made sense to use MAE for these type of cases and MSE for rest and then combine the two for a consolidated loss. For a while I toyed with the idea, but couldn't take it forward due to paucity of time, but I strongly felt that accounting for those extremes somehow would make he training more stable, but just couldn't find a right approach (I had thought to do this by making pred=label where label=extreme value in MSE loss calculation and pred=label where label !=extreme value in MAE loss calculation and add the 2 losses but maybe this messes gradients?)
", "votes": 1, "replies": [] }, { "id": 1454254, "author_name": "Frank", "author_url": "", "post_date": "2021-08-06T06:16:17.247000", "content": "@allohvk Thanks for the comment. You mean using MSE loss for extreme data and MAE loss for the rest, or the opposite? Given the same predictions, the MSE loss gives smaller values but actually its gradients are 2x larger than MAE loss, so this could potentially help the learning of extreme data?
\nI think those samples are not quite learnable as their standard errors are very high. The model can only learn the general trend and during inference, I observed that it would give more conservative predictions. I tried one experiment where I put all of the extreme data at the front in each training epoch, and it clearly showed that they totally destroyed what the model has learned.
", "votes": 1, "replies": [] }, { "id": 1454526, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-06T08:20:44.607000", "content": "@xfffrank Thanks for your feedback. I meant opposite - use MAE for extremes and MSE for rest. My logic for same is that I felt we should not try too much to learn extreme values else this will corrupt the learning of regular values. By \"extreme\" we could mean the upper & lower ends of the target data or the high standard error records or a mix of both. Discarding the extremes completely could be one solution. But I was thinking if there is another way.
\nInstead of discarding completely, for these particular extreme records can we use a loss function which gives little lesser impact? So hence MAE for such cases and MSE for regular cases. This way extreme/outlier data won't effect the regular row data training too much.
\nOne way to implement this would be:
\nBut this is theoritical. not sure if this kind of approach works. Maybe I will experiment it and share further of it works.
", "votes": 1, "replies": [] }, { "id": 1454588, "author_name": "Frank", "author_url": "", "post_date": "2021-08-06T08:57:20.360000", "content": "@allohvk Cool. That's an interesting approach. Worth a try!
\nJust to let you know, I tried discarding either the upper & lower ends of the target data or the high std data (std > 0.6). The cv improved but public LB dropped. And I just checked the private LB, training without high std data (std > 0.6) worsened the public LB by 0.004 but improved the private LB by 0.003 :)
", "votes": 1, "replies": [] }, { "id": 1454641, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-06T09:23:24.130000", "content": ":) very tricky. It is one of those decisions where we have to take a leap of faith in our CV :)
", "votes": 1, "replies": [] }, { "id": 1533895, "author_name": "Allohvk", "author_url": "", "post_date": "2021-10-04T12:51:01.327000", "content": "@xfffrank In case you are interested in further analysis of the BT loss - https://www.kaggle.com/c/commonlitreadabilityprize/discussion/276356
", "votes": 1, "replies": [] }, { "id": 1535705, "author_name": "Frank", "author_url": "", "post_date": "2021-10-06T05:54:01.450000", "content": "Thanks for sharing! @allohvk I'll read it later.
", "votes": 1, "replies": [] } ] }, { "id": 1444884, "author_name": "Minglei Wang", "author_url": "", "post_date": "2021-08-04T07:15:04.830000", "content": "Thanks @xyzdivergence for sharing the ideas and congrats on the win.
\nI had tried to use GPR to ensemble the prediction targets from different models instead of averaging them. It improved on the training set but not on LB. Ensemble the embeddings is really a good idea.
\nSince each of your model uses 5-fold, should you have 1024x5x9 embeddings on the test set? or did you average 5 fold embeddings as the one of the model and the concatenate the 9s together?
Yes, you are right. In total, we have 1024x5x9 embeddings on the test set, and we took the average of the 5-folds GPR prediction as one final prediction.
\nE.g. Assuming the test size is 7
\nembeddings1 = np.concatenate((embedding1_roberta_large), .., (embeddings1_deberta_large), axis=1) # (7, 9216)\n...\nembeddings5 = np.concatenate((embedding5_roberta_large), .., (embeddings5_deberta_large), axis=1) # (7, 9216)\n\nembeddings = [embeddings1, ..., embeddings5] # e.g. [(7, 9216), ..., (7, 9216)]\n\ngpr_preds_concat = [gpr_inference(x) for x in embeddings] # list of gpr predictions array each of size (7,) e.g. [(7,), ..., (7,)]\n\nfinal_gpr_preds = np.mean(gpr_preds_concat, axis=0) # (7,)\nI believe that other than GPR u can use the concatenate embeddings for other models as well e.g. SVR. However, I am not sure how well it will perform since we only tested on GPR as the training dataset is small and seems more logical for our approach.
", "votes": 2, "replies": [] }, { "id": 1446022, "author_name": "Minglei Wang", "author_url": "", "post_date": "2021-08-04T10:48:15.217000", "content": "Thank you. That's a lot clearer. I tried SVR and Boost but unfortunately that was applied on the targets. Will try your technique next time.
", "votes": 0, "replies": [] } ] }, { "id": 1442217, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-03T22:03:46.060000", "content": "Congratulations. Great idea to concatenate all your models' embeddings together and train one meta model.
", "votes": 1, "replies": [ { "id": 1443248, "author_name": "Kelvin", "author_url": "", "post_date": "2021-08-04T02:18:12.593000", "content": "Thanks @cdeotte and congratulations to you too! We learnt a lot from you as well!
", "votes": 1, "replies": [] } ] }, { "id": 1442076, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2021-08-03T21:39:53.783000", "content": "Congratulations on the gold finish! Did you use the AttentionHead with dropout and another Linear layer similar to the notebook or jus the AttentionHead?
", "votes": 1, "replies": [ { "id": 1443273, "author_name": "Kelvin", "author_url": "", "post_date": "2021-08-04T02:24:10.410000", "content": "Thanks @trushk for your comment :). Yes our transformer models uses AttentionHead with dropout and added another linear layer similar to the public notebook https://www.kaggle.com/maunish/clrp-pytorch-roberta-finetune.
", "votes": 1, "replies": [] } ] }, { "id": 1433171, "author_name": "over watch", "author_url": "", "post_date": "2021-08-03T13:48:20.087000", "content": "Congratulations to top finishes in the competition!
\nThe Gaussian process regression looks like such an interesting way to ensemble models. Does your team plan to share the code public?
", "votes": 2, "replies": [ { "id": 1443280, "author_name": "Kelvin", "author_url": "", "post_date": "2021-08-04T02:26:21.860000", "content": "Thank you for your comment :). Yes, we plan to share our code, currently, we are in the midst of tidying up. Will share our training/inference code soon!
", "votes": 1, "replies": [] } ] }, { "id": 1973981, "author_name": "blAckPearL1376", "author_url": "", "post_date": "2022-10-06T02:19:43.027000", "content": "Tks for sharing！
", "votes": 0, "replies": [] }, { "id": 1952128, "author_name": "WANG ZHIJIAN@NUS", "author_url": "", "post_date": "2022-09-23T13:14:25.613000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 20, "messages": [ { "id": 1429879, "postDate": "2021-08-03T11:04:51.573Z", "content": "First of all, we would like to thanks Kaggle & CommonLit for hosting this interesting competition, this has been an interesting journey for us! Also congratulations to all winning teams!
\nAnd special thanks to @rhtsingh, @maunish, and @andretugan for their knowledgeable and resourceful notebooks. Since our team is fairly new to transformers, leveraging the resources from public notebooks definitely helps in our journey!
\nOur final submission consists of 9 transformer models, and our final solution is based on our 2nd-stage model Gaussian Process Regression (GPR) [1] which we use to correct overconfident transformer predictions due to the small training data. This can be summarized in two steps.
\n
We scraped data from the “url_legal” information which is available in the training data and only use those with licenses of “CC BY” and “CC BY-SA”.
\nLike every other team, we first started out fine-tuning transformers model and focus on increasing the single model LB score by training different transformers for diversity. Since our goal is to use GPR to improve the transformer predictions, we need robust transformer models that can generate robust embeddings.
\nOur final submission consists of the following 9 transformer models and all models are trained using 5-fold stratified CV with attention head.
\n| Model | \nCV | \nPublic | \nPrivate | \nNotes | \n
|---|---|---|---|---|
| roberta-large | \n0.492 | \n0.471 | \n0.471 | \nmlm on training set | \n
| microsoft/deberta-large | \n0.485 | \n0.474 | \n0.476 | \nmlm on training set | \n
| xlnet-large-cased | \n0.494 | \n0.475 | \n0.476 | \n- | \n
| deepset/roberta-large-squad2 | \n0.488 | \n0.464 | \n0.467 | \n- | \n
| deepset/roberta-large-squad2 | \n0.484 | \n0.466 | \n0.464 | \nmlm on train set and external data | \n
| allenai/longformer-large-4096-finetuned-triviaqa | \n0.489 | \n0.467 | \n0.47 | \n- | \n
| valhalla/bart-large-finetuned-squadv1 | \n0.471 | \n0.462 | \n0.466 | \nmlm on train set and external data, remove dropout | \n
| microsoft/deberta-large-mnli | \n0.469 | \n0.462 | \n0.469 | \nmlm on train set and external data, remove dropout | \n
| ahotrod/electra_large_discriminator_squad2_512 | \n0.477 | \n0.468 | \n0.468 | \nremove dropout | \n
The model selection is based on the individual LB score (preferably <= 0.475)
\nThe following are the fine-tuned strategy for the transformer models
\nWe are aware of the limited training samples ~2.8k, hence we immediately thought of Gaussian process regression (GPR). Given the constraint of limited training samples, using Bayesian techniques such as GPR could potentially help us make a better approximation/prediction. Since Bayesian/Probabilistic model are proven to performs relatively well in small setting, and it also has a strong notion of uncertainty. The uncertainty is important here as we know the distribution of the training and test data are different, and we do not want the transformer to make overconfident predictions, hence GPR alleviates overconfident predictions from transformers. A NN with one hidden layer of infinite width is basically a Gaussian Processes.
\nThe below figure shows the deberta-large model predictions for 100 randomly sampled points. The green curve is the deberta-large predictions, and the red curve is the GPR predictions. We can see that deberta-large will try to fit all the points as closely as possible (Red circle), however, GPR will avoid overconfident predictions and will regress to the mean of nearby samples when it is not confident, thus providing reliable uncertainty estimation.
\n
We use GPR to correct overconfident transformer predictions. Doing this allows us to further improved the single model score significantly by training a GPR model using all the OOF embeddings extracted from the last transformer layer (attention-head).
\nThe following submission shows that using deberta-large trained with pseudo-labeling (PL) can achieve a single private score of 0.459, but with GPR it can achieve 0.453 alone!
\n| Model | \nCV | \nPublic | \nPrivate | \nGPR Public | \nGPR private | \nNotes | \n
|---|---|---|---|---|---|---|
| deberta-large | \n0.462 | \n0.465 | \n0.459 | \n0.457 | \n0.453 | \ntrain with pseudo labeling | \n
Unfortunately, we didn't include any model trained with pseudo-labeling since many of our PL-trained models has bad single/ensemble LB score as compared to the model trained without pseudo-labeling..
\nThis leads us to believe the LB has a different data distribution as compared to the training data, hence we use all the OOF embeddings for training and only validate the individual GPR model performance on LB.
\nFrom the transformer models section above, the 9 models were selected in terms of their individual LB scores and correlations for diversity. We extracted the attention head embeddings which is of size 1024 for each model, hence 1024*9 embeddings for 9 models. We concatenate all 1024*9 embeddings and train all samples using GPR model with an initial RBF kernel hyperparameters of lengthscale=3, variance=1, noise=1, and run for 1000 steps using stochastic variational inference (SVI). Since we are doing inference with GPR, we are able to learn the GPR/kernel hyperparameters directly from the embeddings itself with minimal fine-tuning.
\nOur final submission with GPR model achieved public 0.448, private 0.448
\nDropout removal
\nThe remove dropout note in the table above refers to removing dropout in the transformer architecture, instead of the dropout placed in the output head. We were inspired by this kernel. It is a very interesting idea to remove it as the default dropout of 0.1 is used in nearly every transformer model. Previously, we can see that the validation loss is very unstable, which is why a small evaluation interval is commonly used. But after removing dropout in the transformers, the val loss became much more steady, and the converged points were usually much better. The cv score improved by a large margin, but the public LB doesn't reflect the change accordingly. Later we found more and more people used this strategy, so we just adopted it in some of our models.
Pseudo labelling
\nWe scraped the data from the same url in the training data and labelled them with an ensemble model which scored LB 0.450. We didn't train on the pseudo data first and finetune with the competition data. Instead, during each fold training, we only added the pseudo data which have the same id as the training data in that fold, to avoid potential data leakage. With this method, the CV score improved a lot but the public LB didn't agree. Though we didn't found data leakage in the cv, we abandoned the models trained with pseudo labelling. It turned out that it works well for the private set, and we should've trusted the cv more. The public test set is only ~600 samples so it doesn't reflect the data distribution in the private set well.
Pretrained models
\nWe found that some of the models further pretrained on question answering datasets are better for finetuning in this competition such as squad and triviaqa. So it's always better to try different pretrained models when using transformer models.
Gaussian process regression
\nThis is the most important ensembling/preprocessing/correction steps for our solution. All of our single transformer models improved significantly with gains from 0.002 - 0.01 after training a GPR model using the attention-head embedding.
Extreme data and post-processing
\nWe found that the data with the targets < -3 or > 1 are the ones that make the training unstable. There are only 131 of these samples in the training set. Their standard errors are also usually higher than the others. By removing these samples, the cv improved a lot. And we found that there are also some patterns in the targets of the extreme data that we can use for post-processing, only if we can detect these extreme samples. But we failed to build a model to achieve that.
Using different heads
\nWe tried training transformers using different heads, e.g. cls, mean pooling, pooler, concatenate last 4/8/12 layers, mean max pooling. But attention head works best for us, hence we stick with attention-head for all our models.
Different optimization strategies
\nInitially, we also tried mixout regularization and multi-sample dropout, doing this stabilizes training, but the individual LB score is bad. And in the end, the transformers w/o dropout works better.
We should have trusted our CV for the models trained with pseudo-labeling. Our best submission has public 0.452, private 0.446 which is achieved by using only 6 models trained with pseudo-labeling. Unfortunately, we didn't select that for our final submission since the public LB is bad..
\nReferences:
\n[1] https://pyro.ai/examples/gp.html
\n\n\n### External data\nWe scraped data from the “url_legal” information which is available in the training data and only use those with licenses of “CC BY” and “CC BY-SA”.\n\n#### Transformers model\nLike every other team, we first started out fine-tuning transformers model and focus on increasing the single model LB score by training different transformers for diversity. Since our goal is to use GPR to improve the transformer predictions, we need robust transformer models that can generate robust embeddings.\n\nOur final submission consists of the following 9 transformer models and all models are trained using 5-fold stratified CV with attention head.\n| Model | CV | Public | Private | Notes\n| --- | --- | --- | --- | --- |\n| roberta-large | 0.492 | 0.471 | 0.471| mlm on training set\n| microsoft/deberta-large | 0.485 |0.474 | 0.476| mlm on training set\n| xlnet-large-cased | 0.494 | 0.475 | 0.476| -\n| deepset/roberta-large-squad2 | 0.488 | 0.464 | 0.467| -\n| deepset/roberta-large-squad2 | 0.484 | 0.466 | 0.464 | mlm on train set and external data\n| allenai/longformer-large-4096-finetuned-triviaqa | 0.489 | 0.467 | 0.47 | -\n| valhalla/bart-large-finetuned-squadv1 | 0.471 | 0.462 | 0.466 | mlm on train set and external data, remove dropout\n| microsoft/deberta-large-mnli | 0.469 | 0.462 | 0.469 | mlm on train set and external data, remove dropout\n| ahotrod/electra_large_discriminator_squad2_512 | 0.477 | 0.468 | 0.468 | remove dropout\n\nThe model selection is based on the individual LB score (preferably <= 0.475)\n\n### Fine-tune\nThe following are the fine-tuned strategy for the transformer models\n- Epochs: 5\n- Batch Size: 8\n- Optimizer: AdamW with SWA\n- LR Scheduler: Linear learning rate scheduler with 6% warmup steps\n- Learning rate: 2e-5\n- Layer reinitialisation: last 3 transformer layers (reference: https://www.kaggle.com/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning)\nUse attention head. (reference: https://www.kaggle.com/maunish/clrp-pytorch-roberta-finetune)\n- Evaluating for every n steps: 10\n\n### Gaussian Process Regression (GPR)\nWe are aware of the limited training samples ~2.8k, hence we immediately thought of Gaussian process regression (GPR). Given the constraint of limited training samples, using Bayesian techniques such as GPR could potentially help us make a better approximation/prediction. Since Bayesian/Probabilistic model are proven to performs relatively well in small setting, and it also has a strong notion of uncertainty. The uncertainty is important here as we know the distribution of the training and test data are different, and we do not want the transformer to make overconfident predictions, hence GPR alleviates overconfident predictions from transformers. **A NN with one hidden layer of infinite width is basically a Gaussian Processes.**\n\nThe below figure shows the deberta-large model predictions for 100 randomly sampled points. The green curve is the deberta-large predictions, and the red curve is the GPR predictions. We can see that deberta-large will try to fit all the points as closely as possible (Red circle), however, GPR will avoid overconfident predictions and will regress to the mean of nearby samples when it is not confident, thus providing reliable uncertainty estimation.\n\n
\n\n### Gaussian process regression (GPR) as 2nd-stage model\nWe use GPR to correct overconfident transformer predictions. Doing this allows us to further improved the single model score significantly by training a GPR model using all the OOF embeddings extracted from the last transformer layer (attention-head). \n\nThe following submission shows that using deberta-large trained with pseudo-labeling (PL) can achieve a single private score of 0.459, but with GPR it can achieve 0.453 alone!\n| Model | CV | Public | Private | GPR Public | GPR private | Notes\n| --- | --- | --- | --- | --- | --- | ---\n| deberta-large | 0.462 | 0.465 | 0.459 | 0.457 | 0.453 | train with pseudo labeling\n\nUnfortunately, we didn't include any model trained with pseudo-labeling since many of our PL-trained models has bad single/ensemble LB score as compared to the model trained without pseudo-labeling.. \n\nThis leads us to believe the LB has a different data distribution as compared to the training data, hence we use all the OOF embeddings for training and only validate the individual GPR model performance on LB. \n\n### Training GPR using all oof embeddings\nFrom the transformer models section above, the 9 models were selected in terms of their individual LB scores and correlations for diversity. We extracted the attention head embeddings which is of size 1024 for each model, hence 1024\\*9 embeddings for 9 models. We concatenate all 1024\\*9 embeddings and train all samples using GPR model with an initial RBF kernel hyperparameters of lengthscale=3, variance=1, noise=1, and run for 1000 steps using stochastic variational inference (SVI). Since we are doing inference with GPR, we are able to learn the GPR/kernel hyperparameters directly from the embeddings itself with minimal fine-tuning.\n\nOur final submission with GPR model achieved public 0.448, private 0.448\n\n### What worked\nDropout removal\nThe `remove dropout` note in the table above refers to removing dropout in the transformer architecture, instead of the dropout placed in the output head. We were inspired by this kernel. It is a very interesting idea to remove it as the default dropout of 0.1 is used in nearly every transformer model. Previously, we can see that the validation loss is very unstable, which is why a small evaluation interval is commonly used. But after removing dropout in the transformers, the val loss became much more steady, and the converged points were usually much better. The cv score improved by a large margin, but the public LB doesn't reflect the change accordingly. Later we found more and more people used this strategy, so we just adopted it in some of our models.\n\nPseudo labelling\nWe scraped the data from the same url in the training data and labelled them with an ensemble model which scored LB 0.450. We didn't train on the pseudo data first and finetune with the competition data. Instead, during each fold training, we only added the pseudo data which have the same id as the training data in that fold, to avoid potential data leakage. With this method, the CV score improved a lot but the public LB didn't agree. Though we didn't found data leakage in the cv, we abandoned the models trained with pseudo labelling. It turned out that it works well for the private set, and we should've trusted the cv more. The public test set is only ~600 samples so it doesn't reflect the data distribution in the private set well.\n\nPretrained models\nWe found that some of the models further pretrained on question answering datasets are better for finetuning in this competition such as `squad` and `triviaqa`. So it's always better to try different pretrained models when using transformer models.\n\nGaussian process regression\nThis is the most important ensembling/preprocessing/correction steps for our solution. All of our single transformer models improved significantly with gains from 0.002 - 0.01 after training a GPR model using the attention-head embedding.\n\n### What didn't work\nExtreme data and post-processing\nWe found that the data with the targets < -3 or > 1 are the ones that make the training unstable. There are only 131 of these samples in the training set. Their standard errors are also usually higher than the others. By removing these samples, the cv improved a lot. And we found that there are also some patterns in the targets of the extreme data that we can use for post-processing, only if we can detect these extreme samples. But we failed to build a model to achieve that.\n\nUsing different heads\nWe tried training transformers using different heads, e.g. cls, mean pooling, pooler, concatenate last 4/8/12 layers, mean max pooling. But attention head works best for us, hence we stick with attention-head for all our models.\n\nDifferent optimization strategies\nInitially, we also tried mixout regularization and multi-sample dropout, doing this stabilizes training, but the individual LB score is bad. And in the end, the transformers w/o dropout works better.\n\n### Final words\nWe should have trusted our CV for the models trained with pseudo-labeling. Our best submission has public 0.452, private 0.446 which is achieved by using only 6 models trained with pseudo-labeling. Unfortunately, we didn't select that for our final submission since the public LB is bad..\n\nReferences:\n[1] https://pyro.ai/examples/gp.html",
"votes": 69
},
{
"id": 1444311,
"postDate": "2021-08-04T05:37:01.043Z",
"content": "I would like to share some more insight about our journey.
\nDropout removal
\nThe remove dropout note in the table above refers to removing dropout in the transformer architecture, instead of the dropout placed in the output head. We were inspired by this kernel. It is a very interesting idea to remove it as the default dropout of 0.1 is used in nearly every transformer model. Previously, we can see that the validation loss is very unstable, which is why a small evaluation interval is commonly used. But after removing dropout in the transformers, the val loss became much more steady, and the converged points were usually much better. The cv score improved by a large margin, but the public LB doesn't reflect the change accordingly. Later we found more and more people used this strategy, so we just adopted it in some of our models.
Pseudo labelling
\nWe scraped the data from the same url in the training data and labelled them with an ensemble model which scored LB 0.450. We didn't train on the pseudo data first and finetune with the competition data. Instead, during each fold training, we only added the pseudo data which have the same id as the training data in that fold, to avoid potential data leakage. With this method, the CV score improved a lot but the public LB didn't agree. Though we didn't found data leakage in the cv, we abandoned the models trained with pseudo labelling. It turned out that it works well for the private set, and we should've trusted the cv more. The public test set is only ~600 samples so it doesn't reflect the data distribution in the private set well.
Pretrained models
\nWe found that some of the models further pretrained on question answering datasets are better for finetuning in this competition such as squad and triviaqa. So it's always better to try different pretrained models when using transformer models.
Extreme data and post-processing
\nWe found that the data with the targets < -3 or > 1 are the ones that make the training unstable. There are only 131 these samples in the training set. Their standard errors are also usually higher than the others. By removing these samples, the cv improved a lot. And we found that there are also some patterns in the targets of the extreme data that we can use for post-processing, only if we can detect these extreme samples. But we failed to build a model to achieve that.
Thanks @xfffrank for your great contribution to the team! I've added the additional information to our post :)
", "rawMarkdown": "Thanks @xfffrank for your great contribution to the team! I've added the additional information to our post :)", "votes": 1 }, { "id": 1454160, "postDate": "2021-08-06T05:43:16.397Z", "content": "actually last comment is very interesting. Do you think it would have made sense to use MAE for these type of cases and MSE for rest and then combine the two for a consolidated loss. For a while I toyed with the idea, but couldn't take it forward due to paucity of time, but I strongly felt that accounting for those extremes somehow would make he training more stable, but just couldn't find a right approach (I had thought to do this by making pred=label where label=extreme value in MSE loss calculation and pred=label where label !=extreme value in MAE loss calculation and add the 2 losses but maybe this messes gradients?)
", "rawMarkdown": "actually last comment is very interesting. Do you think it would have made sense to use MAE for these type of cases and MSE for rest and then combine the two for a consolidated loss. For a while I toyed with the idea, but couldn't take it forward due to paucity of time, but I strongly felt that accounting for those extremes somehow would make he training more stable, but just couldn't find a right approach (I had thought to do this by making pred=label where label=extreme value in MSE loss calculation and pred=label where label !=extreme value in MAE loss calculation and add the 2 losses but maybe this messes gradients?)", "votes": 1 }, { "id": 1454254, "postDate": "2021-08-06T06:16:17.247Z", "content": "@allohvk Thanks for the comment. You mean using MSE loss for extreme data and MAE loss for the rest, or the opposite? Given the same predictions, the MSE loss gives smaller values but actually its gradients are 2x larger than MAE loss, so this could potentially help the learning of extreme data?
\nI think those samples are not quite learnable as their standard errors are very high. The model can only learn the general trend and during inference, I observed that it would give more conservative predictions. I tried one experiment where I put all of the extreme data at the front in each training epoch, and it clearly showed that they totally destroyed what the model has learned.
", "rawMarkdown": "@allohvk Thanks for the comment. You mean using MSE loss for extreme data and MAE loss for the rest, or the opposite? Given the same predictions, the MSE loss gives smaller values but actually its gradients are 2x larger than MAE loss, so this could potentially help the learning of extreme data?\n\nI think those samples are not quite learnable as their standard errors are very high. The model can only learn the general trend and during inference, I observed that it would give more conservative predictions. I tried one experiment where I put all of the extreme data at the front in each training epoch, and it clearly showed that they totally destroyed what the model has learned.", "votes": 1 }, { "id": 1454526, "postDate": "2021-08-06T08:20:44.607Z", "content": "@xfffrank Thanks for your feedback. I meant opposite - use MAE for extremes and MSE for rest. My logic for same is that I felt we should not try too much to learn extreme values else this will corrupt the learning of regular values. By \"extreme\" we could mean the upper & lower ends of the target data or the high standard error records or a mix of both. Discarding the extremes completely could be one solution. But I was thinking if there is another way.
\nInstead of discarding completely, for these particular extreme records can we use a loss function which gives little lesser impact? So hence MAE for such cases and MSE for regular cases. This way extreme/outlier data won't effect the regular row data training too much.
\nOne way to implement this would be:
\nBut this is theoritical. not sure if this kind of approach works. Maybe I will experiment it and share further of it works.
", "rawMarkdown": "@xfffrank Thanks for your feedback. I meant opposite - use MAE for extremes and MSE for rest. My logic for same is that I felt we should not try too much to learn extreme values else this will corrupt the learning of regular values. By \"extreme\" we could mean the upper & lower ends of the target data or the high standard error records or a mix of both. Discarding the extremes completely could be one solution. But I was thinking if there is another way. \n\nInstead of discarding completely, for these particular extreme records can we use a loss function which gives little lesser impact? So hence MAE for such cases and MSE for regular cases. This way extreme/outlier data won't effect the regular row data training too much.\n\nOne way to implement this would be: \n- If (extreme>): make pred=label, then feed to MSE loss calculation (so all extremes rows have zero loss)\n- else: make pred=label for all non-extremes, then feed to MAE loss calculation (so all non-extreme rows have zero loss)\n- Add the two losses to arrive at final loss\n\nBut this is theoritical. not sure if this kind of approach works. Maybe I will experiment it and share further of it works.", "votes": 1 }, { "id": 1454588, "postDate": "2021-08-06T08:57:20.360Z", "content": "@allohvk Cool. That's an interesting approach. Worth a try!
\nJust to let you know, I tried discarding either the upper & lower ends of the target data or the high std data (std > 0.6). The cv improved but public LB dropped. And I just checked the private LB, training without high std data (std > 0.6) worsened the public LB by 0.004 but improved the private LB by 0.003 :)
", "rawMarkdown": "@allohvk Cool. That's an interesting approach. Worth a try!\n\nJust to let you know, I tried discarding either the upper & lower ends of the target data or the high std data (std > 0.6). The cv improved but public LB dropped. And I just checked the private LB, training without high std data (std > 0.6) worsened the public LB by 0.004 but improved the private LB by 0.003 :)", "votes": 1 }, { "id": 1454641, "postDate": "2021-08-06T09:23:24.130Z", "content": ":) very tricky. It is one of those decisions where we have to take a leap of faith in our CV :)
", "rawMarkdown": ":) very tricky. It is one of those decisions where we have to take a leap of faith in our CV :)", "votes": 1 }, { "id": 1533895, "postDate": "2021-10-04T12:51:01.327Z", "content": "@xfffrank In case you are interested in further analysis of the BT loss - https://www.kaggle.com/c/commonlitreadabilityprize/discussion/276356
", "rawMarkdown": "@xfffrank In case you are interested in further analysis of the BT loss - https://www.kaggle.com/c/commonlitreadabilityprize/discussion/276356", "votes": 1 }, { "id": 1535705, "postDate": "2021-10-06T05:54:01.450Z", "content": "Thanks for sharing! @allohvk I'll read it later.
", "rawMarkdown": "Thanks for sharing! @allohvk I'll read it later.", "votes": 1 } ] }, { "id": 1444884, "postDate": "2021-08-04T07:15:04.830Z", "content": "Thanks @xyzdivergence for sharing the ideas and congrats on the win.
\nI had tried to use GPR to ensemble the prediction targets from different models instead of averaging them. It improved on the training set but not on LB. Ensemble the embeddings is really a good idea.
\nSince each of your model uses 5-fold, should you have 1024x5x9 embeddings on the test set? or did you average 5 fold embeddings as the one of the model and the concatenate the 9s together?
Yes, you are right. In total, we have 1024x5x9 embeddings on the test set, and we took the average of the 5-folds GPR prediction as one final prediction.
\nE.g. Assuming the test size is 7
\nembeddings1 = np.concatenate((embedding1_roberta_large), .., (embeddings1_deberta_large), axis=1) # (7, 9216)\n...\nembeddings5 = np.concatenate((embedding5_roberta_large), .., (embeddings5_deberta_large), axis=1) # (7, 9216)\n\nembeddings = [embeddings1, ..., embeddings5] # e.g. [(7, 9216), ..., (7, 9216)]\n\ngpr_preds_concat = [gpr_inference(x) for x in embeddings] # list of gpr predictions array each of size (7,) e.g. [(7,), ..., (7,)]\n\nfinal_gpr_preds = np.mean(gpr_preds_concat, axis=0) # (7,)\nI believe that other than GPR u can use the concatenate embeddings for other models as well e.g. SVR. However, I am not sure how well it will perform since we only tested on GPR as the training dataset is small and seems more logical for our approach.
", "rawMarkdown": "Yes, you are right. In total, we have 1024x5x9 embeddings on the test set, and we took the average of the 5-folds GPR prediction as one final prediction.\n\nE.g. Assuming the test size is 7\n```\nembeddings1 = np.concatenate((embedding1_roberta_large), .., (embeddings1_deberta_large), axis=1) # (7, 9216)\n...\nembeddings5 = np.concatenate((embedding5_roberta_large), .., (embeddings5_deberta_large), axis=1) # (7, 9216)\n\nembeddings = [embeddings1, ..., embeddings5] # e.g. [(7, 9216), ..., (7, 9216)]\n\ngpr_preds_concat = [gpr_inference(x) for x in embeddings] # list of gpr predictions array each of size (7,) e.g. [(7,), ..., (7,)]\n\nfinal_gpr_preds = np.mean(gpr_preds_concat, axis=0) # (7,)\n```\n\nI believe that other than GPR u can use the concatenate embeddings for other models as well e.g. SVR. However, I am not sure how well it will perform since we only tested on GPR as the training dataset is small and seems more logical for our approach.", "votes": 2 }, { "id": 1446022, "postDate": "2021-08-04T10:48:15.217Z", "content": "Thank you. That's a lot clearer. I tried SVR and Boost but unfortunately that was applied on the targets. Will try your technique next time.
", "rawMarkdown": "Thank you. That's a lot clearer. I tried SVR and Boost but unfortunately that was applied on the targets. Will try your technique next time. " } ] }, { "id": 1442217, "postDate": "2021-08-03T22:03:46.060Z", "content": "Congratulations. Great idea to concatenate all your models' embeddings together and train one meta model.
", "rawMarkdown": "Congratulations. Great idea to concatenate all your models' embeddings together and train one meta model. ", "votes": 1, "replies": [ { "id": 1443248, "postDate": "2021-08-04T02:18:12.593Z", "content": "Thanks @cdeotte and congratulations to you too! We learnt a lot from you as well!
", "rawMarkdown": "Thanks @cdeotte and congratulations to you too! We learnt a lot from you as well!", "votes": 1 } ] }, { "id": 1442076, "postDate": "2021-08-03T21:39:53.783Z", "content": "Congratulations on the gold finish! Did you use the AttentionHead with dropout and another Linear layer similar to the notebook or jus the AttentionHead?
", "rawMarkdown": "Congratulations on the gold finish! Did you use the AttentionHead with dropout and another Linear layer similar to the notebook or jus the AttentionHead? \n", "votes": 1, "replies": [ { "id": 1443273, "postDate": "2021-08-04T02:24:10.410Z", "content": "Thanks @trushk for your comment :). Yes our transformer models uses AttentionHead with dropout and added another linear layer similar to the public notebook https://www.kaggle.com/maunish/clrp-pytorch-roberta-finetune.
", "rawMarkdown": "Thanks @trushk for your comment :). Yes our transformer models uses AttentionHead with dropout and added another linear layer similar to the public notebook https://www.kaggle.com/maunish/clrp-pytorch-roberta-finetune.", "votes": 1 } ] }, { "id": 1433171, "postDate": "2021-08-03T13:48:20.087Z", "content": "Congratulations to top finishes in the competition!
\nThe Gaussian process regression looks like such an interesting way to ensemble models. Does your team plan to share the code public?
", "rawMarkdown": "Congratulations to top finishes in the competition!\n\nThe Gaussian process regression looks like such an interesting way to ensemble models. Does your team plan to share the code public?", "votes": 2, "replies": [ { "id": 1443280, "postDate": "2021-08-04T02:26:21.860Z", "content": "Thank you for your comment :). Yes, we plan to share our code, currently, we are in the midst of tidying up. Will share our training/inference code soon!
", "rawMarkdown": "Thank you for your comment :). Yes, we plan to share our code, currently, we are in the midst of tidying up. Will share our training/inference code soon!", "votes": 1 } ] }, { "id": 1973981, "postDate": "2022-10-06T02:19:43.027Z", "content": "Tks for sharing！
", "rawMarkdown": "Tks for sharing！" }, { "id": 1952128, "postDate": "2022-09-23T13:14:25.613Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:48:01Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6781642, "solution_url": "/competitions/commonlitreadabilityprize/writeups/ofnt-6th-place-solution-gaussian-process-regressio" }