To begin with, We would like to thank Kaggle and Commonlit for hosting such an interesting competition! This is our first challenge in the field of NLP competition and very thanks to @rhtsingh, @maunish and @andretugan for sharing their wonderful kernels. Their notebooks were great help to making this competition a great one for sure.
\nWe chose the one with best Public LB and the one with the best CV as our final submissions. The following are our best submissions.
\n| \n | Public LB | \nPrivate LB | \n
|---|---|---|
| best Public LB | \n0.450 | \n0.451 | \n
| best CV | \n0.451 | \n0.448 | \n
We ensembled 10 models including two 25-fold CV models.
\nThe dataset size is one of the most important factor for machine learning. In this competition, train.csv contained about 3000 samples. Generally speaking, this number is very small for training deep learning models. To exhaustively utilize this small dataset, we carefully determined the number of folds (K) in K-fold cross validation. Our hypothesis was that if we increased K, models would perform better thanks to larger number of training samples (Of course, there would be an upper limit of K because the number of validation samples becomes too small). We tried 4 candidates(K=5,10,15,25) to test this hypothesis. As we expected, we observed small but stable improvement of LB score with larger K. To be more specific, increasing K from 5 to 15 improved public (and private) LB by 0.002. Increasing 15 to 25 did not change the displayed score but improved the score slightly (we confirmed that by sorting the submissions by the scores).We decided to employ 25 folds for some strong single models (deberta-large, roberta-large trained by cross entropy loss), paying attention to inference time. As a side note, we used normal k fold cross validation because stratified k fold using binned target did not seem to improve cv-lb correlation much.
\nLike most teams, we used pretrained models from huggingface implemented in pytorch. Through several experiments,and found that deberta-large performed significantly better. However,deberta-large required much longer training time than other models, so we attempted to obtain the optimal parameters for deberta-large in a short time by using deberta base. Several kernels and papers were very helpful in this experiments (e.g.: https://arxiv.org/abs/2006.04884) . After changing many parameters, including lerning rate, activation layer, the best results were obtained by changing loss function. Some models using cross-entropy loss have shown excellent results in previous NLP competitions(e.g. :https://www.kaggle.com/c/google-quest-challenge/discussion/129978). So we trained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label. Median(target) was added to the regressor during prediction.This idea significantly contributed to both CV and LB score, and also contributed significantly to the ensemble.
\n| Model | \nPublic | \nWeight | \n
|---|---|---|
| RoBERTa-Large (25 fold cross entropy) | \n- | \n0.4 | \n
| DeBERTa large (25fold) | \n0.458 | \n0.25 | \n
| DeBERTa large (15 fold cross entropy) | \n- | \n0.25 | \n
| DeBERTa large 5epochs 5fold | \n0.460 | \n0.1 | \n
I also could not find any improvement with data augmentation or SWA.
\nThanks for sharing, I did not know DeBERTa models. I will check them out
", "votes": 3, "replies": [] }, { "id": 1442407, "author_name": "Salim Khazem", "author_url": "", "post_date": "2021-08-03T22:39:44.573000", "content": "Congratulations! , Thanks for sharing your approach
", "votes": 3, "replies": [] }, { "id": 1436588, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-03T15:14:56.273000", "content": "Congratulations! Awesome idea to
\n\n", "votes": 4, "replies": [ { "id": 1437718, "author_name": "wakepon", "author_url": "", "post_date": "2021-08-03T15:42:22.230000", "content": "So we trained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label. Meidan(target) was added to the regressor during prediction.
\n
Thanks a lot!
\nWe never would have thought of your great ideas.
\n(https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257446)
\n\ntrained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label.
\nThanks to BCEWithLogitsLoss, we don't need to add another sigmoid layer. All we need is add Median(target) during prediction
\n
As I understand you pass the target thru' a sigmoid and then do a BCE. So say from -3.X to +1.Y we now have a target from 0-1. So we can apply BCE as a loss. So far I understood
\nMy doubt - to convert to actual predictions we would have to do the inverse of a sigmoid basically apply a logit function what Feng is saying. (and then add the mean or median that part is ok). Is that understanding correct? If so it is very elegant & effective method of converting a regression loss to entropy loss. Thanks for sharing
", "votes": 2, "replies": [ { "id": 1451803, "author_name": "K-NKSM", "author_url": "", "post_date": "2021-08-05T12:45:03.640000", "content": "Thank you for your comment😊
\n\n\nAs I understand you pass the target thru' a sigmoid and then do a BCE.
\n
BCEWithLogitsLoss internally applies sigmoid to the first argument. Therefore, we did not manually apply sigmoid to the model output. This is why logit function was unnecessary. What we did is conceptually equivalent to putting sigmoid layer after regressor, calculating binary cross entropy during training and inverting the sigmoid function during inference. However, BCEWithLogitsLoss is supposed to be numerically more stable than doing that. For details, please take a look at LitModel.training_step, LitModel.validation_step, LitModel.predict_step in this notebook we published.
", "votes": 1, "replies": [] }, { "id": 1451921, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-05T13:23:03.273000", "content": "Thanks a lot for your kind reply and notebook reference. This is really a neat technique!
", "votes": 1, "replies": [] } ] }, { "id": 1445501, "author_name": "Frank", "author_url": "", "post_date": "2021-08-04T09:08:00.483000", "content": "Congratulations! Thank you for sharing the great idea. I've got a few questions. @wakepon
\nSigmoid(target-Median(target)) using the target median of the training set to get the predicted target value? -Median(target) instead of sigmoid(target)? And why choosing median instead of mean?Thanks for your questions!
\n\n\n1 During prediction, did you reverse the calculation of Sigmoid(target-Median(target)) using the target median of the training set to get the predicted target value?
\n
Yes, that's right. Thanks to BCEWithLogitsLoss, we don't need to add another sigmoid layer. All we need is add Median(target) during prediction.
\n\n\n2 Why do you need -Median(target) instead of sigmoid(target)?
\n
Because we were concerned about the side effect of biased sigmoid(target).
\nSince target values are biased to the negative side by about -0.9, sigmoid(target) will be also biased to <0.5. This situation is close to class inbalance in ordinary classification problems. To avoid side effect of bias, we introduced-Median(target). However, we did not test whether the bias towards <0.5 really has any side effect.
\n\nAnd why choosing median instead of mean?
\n
Because median is known to be more robust against outliers than mean.
\nHowever, to be honest, we just wanted to get the center of the distribution and didn't have a strong reason.
Cool. Thanks for the reply!
", "votes": 1, "replies": [] } ] } ], "comment_count": 10, "messages": [ { "id": 1436358, "postDate": "2021-08-03T15:04:51.870Z", "content": "To begin with, We would like to thank Kaggle and Commonlit for hosting such an interesting competition! This is our first challenge in the field of NLP competition and very thanks to @rhtsingh, @maunish and @andretugan for sharing their wonderful kernels. Their notebooks were great help to making this competition a great one for sure.
\nWe chose the one with best Public LB and the one with the best CV as our final submissions. The following are our best submissions.
\n| \n | Public LB | \nPrivate LB | \n
|---|---|---|
| best Public LB | \n0.450 | \n0.451 | \n
| best CV | \n0.451 | \n0.448 | \n
We ensembled 10 models including two 25-fold CV models.
\nThe dataset size is one of the most important factor for machine learning. In this competition, train.csv contained about 3000 samples. Generally speaking, this number is very small for training deep learning models. To exhaustively utilize this small dataset, we carefully determined the number of folds (K) in K-fold cross validation. Our hypothesis was that if we increased K, models would perform better thanks to larger number of training samples (Of course, there would be an upper limit of K because the number of validation samples becomes too small). We tried 4 candidates(K=5,10,15,25) to test this hypothesis. As we expected, we observed small but stable improvement of LB score with larger K. To be more specific, increasing K from 5 to 15 improved public (and private) LB by 0.002. Increasing 15 to 25 did not change the displayed score but improved the score slightly (we confirmed that by sorting the submissions by the scores).We decided to employ 25 folds for some strong single models (deberta-large, roberta-large trained by cross entropy loss), paying attention to inference time. As a side note, we used normal k fold cross validation because stratified k fold using binned target did not seem to improve cv-lb correlation much.
\nLike most teams, we used pretrained models from huggingface implemented in pytorch. Through several experiments,and found that deberta-large performed significantly better. However,deberta-large required much longer training time than other models, so we attempted to obtain the optimal parameters for deberta-large in a short time by using deberta base. Several kernels and papers were very helpful in this experiments (e.g.: https://arxiv.org/abs/2006.04884) . After changing many parameters, including lerning rate, activation layer, the best results were obtained by changing loss function. Some models using cross-entropy loss have shown excellent results in previous NLP competitions(e.g. :https://www.kaggle.com/c/google-quest-challenge/discussion/129978). So we trained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label. Median(target) was added to the regressor during prediction.This idea significantly contributed to both CV and LB score, and also contributed significantly to the ensemble.
\n| Model | \nPublic | \nWeight | \n
|---|---|---|
| RoBERTa-Large (25 fold cross entropy) | \n- | \n0.4 | \n
| DeBERTa large (25fold) | \n0.458 | \n0.25 | \n
| DeBERTa large (15 fold cross entropy) | \n- | \n0.25 | \n
| DeBERTa large 5epochs 5fold | \n0.460 | \n0.1 | \n
I also could not find any improvement with data augmentation or SWA.
\nThanks for sharing, I did not know DeBERTa models. I will check them out
", "rawMarkdown": "I also could not find any improvement with data augmentation or SWA.\n\nThanks for sharing, I did not know DeBERTa models. I will check them out", "votes": 3 }, { "id": 1442407, "postDate": "2021-08-03T22:39:44.573Z", "content": "Congratulations! , Thanks for sharing your approach
", "rawMarkdown": "Congratulations! , Thanks for sharing your approach \n\n", "votes": 3 }, { "id": 1436588, "postDate": "2021-08-03T15:14:56.273Z", "content": "Congratulations! Awesome idea to
\n\n", "rawMarkdown": "Congratulations! Awesome idea to\n>So we trained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label. Meidan(target) was added to the regressor during prediction.\n", "votes": 4, "replies": [ { "id": 1437718, "postDate": "2021-08-03T15:42:22.230Z", "content": "So we trained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label. Meidan(target) was added to the regressor during prediction.
\n
Thanks a lot!
\nWe never would have thought of your great ideas.
\n(https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257446)
\n\ntrained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label.
\nThanks to BCEWithLogitsLoss, we don't need to add another sigmoid layer. All we need is add Median(target) during prediction
\n
As I understand you pass the target thru' a sigmoid and then do a BCE. So say from -3.X to +1.Y we now have a target from 0-1. So we can apply BCE as a loss. So far I understood
\nMy doubt - to convert to actual predictions we would have to do the inverse of a sigmoid basically apply a logit function what Feng is saying. (and then add the mean or median that part is ok). Is that understanding correct? If so it is very elegant & effective method of converting a regression loss to entropy loss. Thanks for sharing
", "rawMarkdown": "> *trained BCEWithLogitsLoss using Sigmoid(target-Median(target)) as the prediction label.*\n\n> *Thanks to BCEWithLogitsLoss, we don't need to add another sigmoid layer. All we need is add Median(target) during prediction*\n\nAs I understand you pass the target thru' a sigmoid and then do a BCE. So say from -3.X to +1.Y we now have a target from 0-1. So we can apply BCE as a loss. So far I understood\n\nMy doubt - to convert to actual predictions we would have to do the inverse of a sigmoid basically apply a logit function what Feng is saying. (and then add the mean or median that part is ok). Is that understanding correct? If so it is very elegant & effective method of converting a regression loss to entropy loss. Thanks for sharing", "votes": 2, "replies": [ { "id": 1451803, "postDate": "2021-08-05T12:45:03.640Z", "content": "Thank you for your comment😊
\n\n\nAs I understand you pass the target thru' a sigmoid and then do a BCE.
\n
BCEWithLogitsLoss internally applies sigmoid to the first argument. Therefore, we did not manually apply sigmoid to the model output. This is why logit function was unnecessary. What we did is conceptually equivalent to putting sigmoid layer after regressor, calculating binary cross entropy during training and inverting the sigmoid function during inference. However, BCEWithLogitsLoss is supposed to be numerically more stable than doing that. For details, please take a look at LitModel.training_step, LitModel.validation_step, LitModel.predict_step in this notebook we published.
", "rawMarkdown": "Thank you for your comment😊\n> As I understand you pass the target thru' a sigmoid and then do a BCE.\n\n[BCEWithLogitsLoss internally applies sigmoid to the first argument](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html). Therefore, we did not manually apply sigmoid to the model output. This is why logit function was unnecessary. What we did is conceptually equivalent to putting sigmoid layer after regressor, calculating binary cross entropy during training and inverting the sigmoid function during inference. However, BCEWithLogitsLoss is supposed to be numerically more stable than doing that. For details, please take a look at LitModel.training_step, LitModel.validation_step, LitModel.predict_step in [this notebook](https://www.kaggle.com/nmkt6e6d6b74/training-roberta-large-using-cross-entropy/log) we published.", "votes": 1 }, { "id": 1451921, "postDate": "2021-08-05T13:23:03.273Z", "content": "Thanks a lot for your kind reply and notebook reference. This is really a neat technique!
", "rawMarkdown": "Thanks a lot for your kind reply and notebook reference. This is really a neat technique!", "votes": 1 } ] }, { "id": 1445501, "postDate": "2021-08-04T09:08:00.483Z", "content": "Congratulations! Thank you for sharing the great idea. I've got a few questions. @wakepon
\nSigmoid(target-Median(target)) using the target median of the training set to get the predicted target value? -Median(target) instead of sigmoid(target)? And why choosing median instead of mean?Thanks for your questions!
\n\n\n1 During prediction, did you reverse the calculation of Sigmoid(target-Median(target)) using the target median of the training set to get the predicted target value?
\n
Yes, that's right. Thanks to BCEWithLogitsLoss, we don't need to add another sigmoid layer. All we need is add Median(target) during prediction.
\n\n\n2 Why do you need -Median(target) instead of sigmoid(target)?
\n
Because we were concerned about the side effect of biased sigmoid(target).
\nSince target values are biased to the negative side by about -0.9, sigmoid(target) will be also biased to <0.5. This situation is close to class inbalance in ordinary classification problems. To avoid side effect of bias, we introduced-Median(target). However, we did not test whether the bias towards <0.5 really has any side effect.
\n\nAnd why choosing median instead of mean?
\n
Because median is known to be more robust against outliers than mean.
\nHowever, to be honest, we just wanted to get the center of the distribution and didn't have a strong reason.
Cool. Thanks for the reply!
", "rawMarkdown": "Cool. Thanks for the reply!", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:48:45Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6784148, "solution_url": "/competitions/commonlitreadabilityprize/writeups/pondefrogisland-9th-place-solution-ensemble-of-fou" }