First of all, I would like to thank the competition organizers for this great competition. It was really interesting and I learned a lot. This was my first Kaggle competition and I enjoyed it very much.
\nThanks also to the community. You were friendly, shared interesting ideas in discussions and notebooks and I was able to learn so much from you.
\nSpecial thanks to the following users for providing helpful bits and pieces that I used in my submission:
\nThanks @teeyee314 for https://www.kaggle.com/teeyee314/readability-url-scrape . I used this data and it saved me a lot of time and effort because it was so well prepared.
Thanks to @maunish @rhtsingh @andretugan I used this notebook for pseudo-labeling https://www.kaggle.com/andretugan/commonlit-two-models and it greatly improved my score.
TLDR Solution
\nI made a large collection of external data. I used sentence bert (Reimers and Gurevych 2019 - https://github.com/UKPLab/sentence-transformers) to select for each excerpt in the train set the 5 most similar snippets in the external data collection. I then used a roberta-base (later on I used a roberta-base/-large ensemble) model trained on the original training data to label the selected data. I then filtered the data on standard error of each sample to keep roughly the same distribution as in the train set.
Then, I trained some different models on the pseudo-labeled data for 1-4 epochs. After training on the pseudo-labeled data, I trained each model on the original training set. Models used in my final submission were albert-xxlarge, deberta-large, roberta-large and electra-large. I used ridge regression to ensemble these models.
\nDetailed solution
\nBackground
\nThis is my first Kaggle competition and so far, my day job did not involve any practical work in data science or machine learning. I am working with data scientists in the field of NLP on a daily basis but in my role as Head of Product I do not code. When thinking about this competition, I knew from the start that I wouldn't be able to compete on sophisticated model design or extensive hyperparameter tuning. I decided to make smart use of external data instead.
\nI had read some papers which seemed useful in this competition and I decided to implement an approach similar to these papers.
\nThe excellent work by Nils Reimers and the UKPLab was certainly a foundation in my solution. Their work on sentence embeddings allowed me to select data that was relevant to this competition. You can read up on sentence transformers in:
\n\nOne of their papers that inspired my solution uses unlabeled external data and pseudo-labeling to improve sentence embedding performance. They use a small labeled dataset to train a cross encoder, retrieve pairs of unlabeled external data using BM-25 or Sentence BERT and label these pairs using the cross encoder trained on the gold dataset. They then proceed to train a Sentence BERT using the gold dataset and the pseudo-labeled data. See: Nandan Thakur, Nils Reimers, Johannes Daxenberger, Iryna Gurevych. 2020. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks
\nAnother paper that was very relevant to this competition is a paper published by Facebook AI and Stanford, where the authors use self-training to improve model performance on a wide range of NLU tasks. Here, a large external sentence bank is searched with sentence embeddings and the retrieved data is labeled by a model which was finetuned on the target data. They then use the pseudo-labeled data to train another model. See: Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, Alexis Conneau. 2020. Self-training Improves Pre-training for Natural Language Understanding
\nI also tried to transfer the results from Google's noisy student paper to NLP. In the original paper, the authors apply their strategy to image classification, but I wanted to see if it could also be applicable in this competition. They train a model on a set of labeled images and then use this model to label unlabeled images. They then use this data to train an equally-sized or larger student model. Contrary to the solutions above, they do not stop after one round of pseudo-labeling. Instead, they use the resulting student model as the new teacher and label more data using this model. They also introduce noise in the form of data augmentation and various dropout techniques for the student. In total, they find that 3 rounds of teacher/student labeling yield the best results. I tried to run multiple rounds with my approach and I also tried to introduce noise in the form of backtranslation and word replacements (predicting MASK tokens), I also added dropout for my student. However, while multiple rounds of pseudo-labeling continuously improved my CV, my LB score got worse. I think I was overfitting to my evaluation folds because I was evaluating so often. See: Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le. 2019. Self-training with Noisy Student improves ImageNet classification.
\nTraining process
\nBasics:
\nExternal data selection:
\nPseudo-labeling:
\nTraining:
\nThen:
\nEnsembling:
\nBonus
\nI also trained a completely different model using sentence transformers. For this model, I generated pairs of excerpts from the original data and the pseudo-labeled data. I then used the difference between the scores of each pair to generate labels for the difficulty distance between these pairs. I then trained a sentence embedding model using these pairs. For inference, I embedded both the train set and the test set. Then, I used the sentence embedding model to find for each test sample the 20 most similar (actually most similar in difficulty) samples from the train set. I then used the score of the selected train samples and the cosine similarity as features for another regressor (Ridge or BayesianRidge). My best model based on this sentence embedding approach scored 0.447 on public and private LB (sentence embeddings + other models). Using only sentence embeddings I got 0.456 on public LB and 0.463 on private LB.
\nEDIT
\nHere is the link to the code that I used for training: https://github.com/mathislucka/kaggle_clrp_1st_place_solution
I'm sure it saved you time - I spent a week working on compiling the scraped data. I'm glad it helped you. Congrats on solo gold! Well implemented solution!
", "votes": 21, "replies": [] }, { "id": 1424728, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-03T06:20:30.227000", "content": "Congrats @mathislucka !! It's so amazing that I actually approached this competition almost exactly the same as you xD. The mindset is nearly the same word to word in my post.
\n\n\nHyperparameters: I didn't tune much and use mostly original setup from torch's training script since I won't have much chance competing in what I'm far behind others.
\n
And yours:
\n\n", "votes": 5, "replies": [ { "id": 1425716, "author_name": "Mathis Lucka", "author_url": "", "post_date": "2021-08-03T07:01:30.503000", "content": "When thinking about this competition, I knew from the start that I wouldn't be able to compete on sophisticated model design or extensive hyperparameter tuning. I decided to make smart use of external data instead.
\n
Thanks @abcbum ! Also congrats to your 13th place. We did indeed apply the same reasoning :D
", "votes": 1, "replies": [] }, { "id": 1431803, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T12:49:24.380000", "content": "There is one more similarity. Yours and @abcbum are the only 2 solutions (at least as far as I remember…among the dozen odd which I read) which seemed to make pseudo labelling work and contribute to the score…
", "votes": 1, "replies": [] }, { "id": 1432197, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-03T13:05:18.887000", "content": "@allohvk, thank you so much but actually no, hxcc's 3rd place solution also used it
\n\n", "votes": 1, "replies": [] }, { "id": 1432653, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T13:28:58.427000", "content": "I then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.
\n
oh ok…got it. yes! I did read that but somehow didnt register in my memory..I think I quickly saw the last half of his comments (what worked) and didnt read in full detail. sorry
", "votes": 1, "replies": [] } ] }, { "id": 1456274, "author_name": "Gary Lai", "author_url": "", "post_date": "2021-08-06T20:52:57.850000", "content": "Thank you so much @mathislucka for the clear kernel and congrats on the gold! I came up with the exact idea about sentence embedding representing readability too haha.
\nI have two questions for you:
\n(1) Why didn't you use the sentence transformer model (cosine similarity) instead of the paraphrase model to retrieve the top 5 most similar sentences from the external dataset? Wouldn't using the paraphrase model retrieve the most similar sentences in terms of semantics as opposed to readability? Two sentences could have opposite meaning but same readability (e.g. \"I love this restaurant\" vs. \"I hate this restaurant\")
\n(2) Why did you drop the sentence transformer model from your ensemble in your final solution?
\nThank you so much and congrats again!
", "votes": 1, "replies": [ { "id": 1456376, "author_name": "Mathis Lucka", "author_url": "", "post_date": "2021-08-06T22:08:56.367000", "content": "(1) I didn't get a good \"difficulty sentence transformer\" up until close to the competition deadline. There was simply no time to do the data selection again using the difficulty model. My main motivation for choosing the paraphrase model was that I wanted data that was similar to the training data (similar in the sense of semantic similarity) because I expected that the test data would also be similar to the training data. It didn't really matter if the retrieved sentence had a different difficulty because I used a separate model to predict the difficulty of all retrieved sentences and then filtered these predictions on standard error of the original sample to keep roughly the same distribution. My score actually got worse quite a bit when I didn't do the filtering step. I also tried different top k values for the retrieval and 5 to 10 additional data snippets per train sample seemed to be the sweet spot. When selecting only data that was a bit further away from my training samples (cosine similarity wise) the score would also get worse.
\n(2) I actually submitted a model which included the sentence transformer model but it scored 0.447 both on the private and public leaderboard. I decided to also choose one model without the sentence transformer model because I did not want to take the risk of submitting 2 very similar models. Fun fact: my best performing model (which I didn't select) even scored 0.445 on the private test set but it was also not using the sentence transformer model. Also the sentence transformer model took quite a bit of time because each fold needed to encode both the train and the test set. With one submission I even hit the timeout. I think the model that timed out would have performed even better because it was combining the best of both worlds.
", "votes": 3, "replies": [] } ] }, { "id": 1451757, "author_name": "RB", "author_url": "", "post_date": "2021-08-05T12:29:31.390000", "content": "Congratulations and Thank you for sharing your approach. Why did you choose 6 fold CV vs lets say 5 fold CV ? Also 6 fold CV was random or stratified some way ?
", "votes": 1, "replies": [] }, { "id": 1445198, "author_name": "Shubham Singh", "author_url": "", "post_date": "2021-08-04T08:05:53.770000", "content": "Congrats @mathislucka and Thanks I was looking for some NLP resources.
", "votes": 1, "replies": [] }, { "id": 1428999, "author_name": "Daniel Guzman Olivares", "author_url": "", "post_date": "2021-08-03T10:15:25.973000", "content": "Congratulations @mathislucka! That was some clever use of external data, thanks a lot for sharing!
", "votes": 1, "replies": [] }, { "id": 1426438, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2021-08-03T07:33:47.900000", "content": "I bet your teachers are proud of you. PS - Pun intended
\nCongratulations on the solo win @mathislucka !
", "votes": 1, "replies": [] }, { "id": 1426239, "author_name": "Salim Khazem", "author_url": "", "post_date": "2021-08-03T07:24:09.300000", "content": "Great performance, well done
\nThanks sharing your approach
\nCongrats for your first place
Congrats! I am little confued about the ridge regression for ensemble. Is it similar to stacking or something else?
", "votes": 1, "replies": [ { "id": 1425825, "author_name": "Mathis Lucka", "author_url": "", "post_date": "2021-08-03T07:05:00.570000", "content": "Yes, it's stacking. Say I have trained 2 6-fold models: albert and deberta. This gives me a total of 12 models. For each model I predict the fold that was not seen during training, giving me 2 model predictions for each fold. I then merge all out of fold predictions and make a new 6-fold split. Then, I use these folds to train a RidgeRegression model. This results in 6 RidgeRegression models each using albert and deberta predictions as an input. I then averaged the 6 ridge models for submissions (simple average)
", "votes": 13, "replies": [] }, { "id": 1426203, "author_name": "Zacchaeus", "author_url": "", "post_date": "2021-08-03T07:22:16.533000", "content": "Thanks for the clear explanation!
", "votes": 0, "replies": [] } ] }, { "id": 1425063, "author_name": "spy1234567", "author_url": "", "post_date": "2021-08-03T06:34:27.643000", "content": "Congrats! Thanks for your sharing, by the way, have you considered to use dropout in the transformer structures?
", "votes": 1, "replies": [ { "id": 1425878, "author_name": "Mathis Lucka", "author_url": "", "post_date": "2021-08-03T07:07:17.707000", "content": "Thanks. I tried dropout but it didn't work very well for me.
", "votes": 1, "replies": [] } ] }, { "id": 1426231, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2021-08-03T07:23:39.433000", "content": "Great and simple idea! I always find it amazing when people arrive with fresh ideas and just win by approaching the problem from a very innovative angle! Big congrats and thanks for sharing
", "votes": 2, "replies": [] }, { "id": 1725149, "author_name": "Nancy Ma", "author_url": "", "post_date": "2022-03-16T21:08:41.627000", "content": "Hi, I'm running the 2nd notebook and encode_and_save() never finishes running. The df_to_csv() generates the csv very quickly but the encode_and_save() ran for 10hr before my colab notebook died.. Any suggestions?
", "votes": 0, "replies": [] }, { "id": 1578286, "author_name": "Marília Prata", "author_url": "", "post_date": "2021-11-10T20:20:08.440000", "content": "Es ist Wunderbar Mathis. Though 3 months delayed for congratulation you.
", "votes": 0, "replies": [] }, { "id": 1478650, "author_name": "Shishir Jain", "author_url": "", "post_date": "2021-08-18T05:09:21.577000", "content": "Congratulations on your performance and thanks for sharing the approach !!
", "votes": 0, "replies": [] }, { "id": 1470231, "author_name": "Artem", "author_url": "", "post_date": "2021-08-13T11:45:11.943000", "content": "Congrats @mathislucka
\nThanks for sharing!
Congratulations bro, you really made a good explanation 👌👌🙏
", "votes": 0, "replies": [] }, { "id": 1464133, "author_name": "Duc Tran", "author_url": "", "post_date": "2021-08-10T13:09:58.410000", "content": "Congrats!!!
", "votes": 0, "replies": [] }, { "id": 1461050, "author_name": "RIYADH HAITHAM", "author_url": "", "post_date": "2021-08-09T07:31:00", "content": "great work Congratulations
\ndid you remove all examples in external data that are outside minimum and maximum target value in original training set ??
No, I did not remove them.
", "votes": 1, "replies": [] } ] }, { "id": 1457523, "author_name": "Veerendra Jagadale", "author_url": "", "post_date": "2021-08-07T13:04:35.813000", "content": "Congratulations on Solo win and thank you for sharing your approach…!!!
", "votes": 0, "replies": [] }, { "id": 1456639, "author_name": "Rakesh Jarupula", "author_url": "", "post_date": "2021-08-07T03:03:26.433000", "content": "Congratulations @mathislucka on winning the competition.
", "votes": 0, "replies": [] }, { "id": 1456427, "author_name": "Diego Jiménez", "author_url": "", "post_date": "2021-08-06T22:46:08.560000", "content": "Congratulation for this huge achievement!!!
", "votes": 0, "replies": [] }, { "id": 1456370, "author_name": "Ashavi Ersad", "author_url": "", "post_date": "2021-08-06T22:03:49.183000", "content": "Thankyou for sharing this articles
", "votes": 0, "replies": [] }, { "id": 1455554, "author_name": "Arunkr", "author_url": "", "post_date": "2021-08-06T15:50:10.660000", "content": "congrats for the gold and thanks for sharing the process
", "votes": 0, "replies": [] }, { "id": 1452388, "author_name": "Arshak Mkhoyan", "author_url": "", "post_date": "2021-08-05T15:36:01.503000", "content": "Great use of external data!
\nCongrats!
Thanks for sharing your solution and congratulations for the first place!
", "votes": 0, "replies": [] }, { "id": 1451775, "author_name": "Shobana_360", "author_url": "", "post_date": "2021-08-05T12:34:25.050000", "content": "Congratulations on the solo win @mathislucka !!!
", "votes": 0, "replies": [] }, { "id": 1450413, "author_name": "ABINAYA.V", "author_url": "", "post_date": "2021-08-05T05:18:34.257000", "content": "Thanks for kaggle for providing free course for students.
", "votes": 0, "replies": [] }, { "id": 1449687, "author_name": "HinePo", "author_url": "", "post_date": "2021-08-05T00:00:33.407000", "content": "I like that you won with a smart approach to the data and 'few' models instead of ensembling dozens of models
\ncongrats!!
", "votes": 0, "replies": [ { "id": 1456387, "author_name": "Mathis Lucka", "author_url": "", "post_date": "2021-08-06T22:16:53.363000", "content": "Well :D Actually I did ensemble quite a few models. In total, I count 38 transformer models in my final submission.
", "votes": 1, "replies": [] }, { "id": 1456840, "author_name": "HinePo", "author_url": "", "post_date": "2021-08-07T05:38:57.237000", "content": "Wow, thanks for correction! I didn't get that from your explanation before. To me, what you did is quite magic. That's truly well deserved. Congrats again!
", "votes": 0, "replies": [] } ] }, { "id": 1440175, "author_name": "Vipin Kumar", "author_url": "", "post_date": "2021-08-03T17:58:12.027000", "content": "Congrats @mathislucka
\nThanks for sharing
First of all, I would like to thank the competition organizers for this great competition. It was really interesting and I learned a lot. This was my first Kaggle competition and I enjoyed it very much.
\nThanks also to the community. You were friendly, shared interesting ideas in discussions and notebooks and I was able to learn so much from you.
\nSpecial thanks to the following users for providing helpful bits and pieces that I used in my submission:
\nThanks @teeyee314 for https://www.kaggle.com/teeyee314/readability-url-scrape . I used this data and it saved me a lot of time and effort because it was so well prepared.
Thanks to @maunish @rhtsingh @andretugan I used this notebook for pseudo-labeling https://www.kaggle.com/andretugan/commonlit-two-models and it greatly improved my score.
TLDR Solution
\nI made a large collection of external data. I used sentence bert (Reimers and Gurevych 2019 - https://github.com/UKPLab/sentence-transformers) to select for each excerpt in the train set the 5 most similar snippets in the external data collection. I then used a roberta-base (later on I used a roberta-base/-large ensemble) model trained on the original training data to label the selected data. I then filtered the data on standard error of each sample to keep roughly the same distribution as in the train set.
Then, I trained some different models on the pseudo-labeled data for 1-4 epochs. After training on the pseudo-labeled data, I trained each model on the original training set. Models used in my final submission were albert-xxlarge, deberta-large, roberta-large and electra-large. I used ridge regression to ensemble these models.
\nDetailed solution
\nBackground
\nThis is my first Kaggle competition and so far, my day job did not involve any practical work in data science or machine learning. I am working with data scientists in the field of NLP on a daily basis but in my role as Head of Product I do not code. When thinking about this competition, I knew from the start that I wouldn't be able to compete on sophisticated model design or extensive hyperparameter tuning. I decided to make smart use of external data instead.
\nI had read some papers which seemed useful in this competition and I decided to implement an approach similar to these papers.
\nThe excellent work by Nils Reimers and the UKPLab was certainly a foundation in my solution. Their work on sentence embeddings allowed me to select data that was relevant to this competition. You can read up on sentence transformers in:
\n\nOne of their papers that inspired my solution uses unlabeled external data and pseudo-labeling to improve sentence embedding performance. They use a small labeled dataset to train a cross encoder, retrieve pairs of unlabeled external data using BM-25 or Sentence BERT and label these pairs using the cross encoder trained on the gold dataset. They then proceed to train a Sentence BERT using the gold dataset and the pseudo-labeled data. See: Nandan Thakur, Nils Reimers, Johannes Daxenberger, Iryna Gurevych. 2020. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks
\nAnother paper that was very relevant to this competition is a paper published by Facebook AI and Stanford, where the authors use self-training to improve model performance on a wide range of NLU tasks. Here, a large external sentence bank is searched with sentence embeddings and the retrieved data is labeled by a model which was finetuned on the target data. They then use the pseudo-labeled data to train another model. See: Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, Alexis Conneau. 2020. Self-training Improves Pre-training for Natural Language Understanding
\nI also tried to transfer the results from Google's noisy student paper to NLP. In the original paper, the authors apply their strategy to image classification, but I wanted to see if it could also be applicable in this competition. They train a model on a set of labeled images and then use this model to label unlabeled images. They then use this data to train an equally-sized or larger student model. Contrary to the solutions above, they do not stop after one round of pseudo-labeling. Instead, they use the resulting student model as the new teacher and label more data using this model. They also introduce noise in the form of data augmentation and various dropout techniques for the student. In total, they find that 3 rounds of teacher/student labeling yield the best results. I tried to run multiple rounds with my approach and I also tried to introduce noise in the form of backtranslation and word replacements (predicting MASK tokens), I also added dropout for my student. However, while multiple rounds of pseudo-labeling continuously improved my CV, my LB score got worse. I think I was overfitting to my evaluation folds because I was evaluating so often. See: Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le. 2019. Self-training with Noisy Student improves ImageNet classification.
\nTraining process
\nBasics:
\nExternal data selection:
\nPseudo-labeling:
\nTraining:
\nThen:
\nEnsembling:
\nBonus
\nI also trained a completely different model using sentence transformers. For this model, I generated pairs of excerpts from the original data and the pseudo-labeled data. I then used the difference between the scores of each pair to generate labels for the difficulty distance between these pairs. I then trained a sentence embedding model using these pairs. For inference, I embedded both the train set and the test set. Then, I used the sentence embedding model to find for each test sample the 20 most similar (actually most similar in difficulty) samples from the train set. I then used the score of the selected train samples and the cosine similarity as features for another regressor (Ridge or BayesianRidge). My best model based on this sentence embedding approach scored 0.447 on public and private LB (sentence embeddings + other models). Using only sentence embeddings I got 0.456 on public LB and 0.463 on private LB.
\nEDIT
\nHere is the link to the code that I used for training: https://github.com/mathislucka/kaggle_clrp_1st_place_solution
I'm sure it saved you time - I spent a week working on compiling the scraped data. I'm glad it helped you. Congrats on solo gold! Well implemented solution!
", "rawMarkdown": "I'm sure it saved you time - I spent a week working on compiling the scraped data. I'm glad it helped you. Congrats on solo gold! Well implemented solution!", "votes": 21 }, { "id": 1424728, "postDate": "2021-08-03T06:20:30.227Z", "content": "Congrats @mathislucka !! It's so amazing that I actually approached this competition almost exactly the same as you xD. The mindset is nearly the same word to word in my post.
\n\n\nHyperparameters: I didn't tune much and use mostly original setup from torch's training script since I won't have much chance competing in what I'm far behind others.
\n
And yours:
\n\n", "rawMarkdown": "Congrats @mathislucka !! It's so amazing that I actually approached this competition almost exactly the same as you xD. The mindset is nearly the same word to word in my [post](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257629).\n> Hyperparameters: I didn't tune much and use mostly original setup from torch's training script since I won't have much chance competing in what I'm far behind others.\n\nAnd yours:\n> When thinking about this competition, I knew from the start that I wouldn't be able to compete on sophisticated model design or extensive hyperparameter tuning. I decided to make smart use of external data instead.\n", "votes": 5, "replies": [ { "id": 1425716, "postDate": "2021-08-03T07:01:30.503Z", "content": "When thinking about this competition, I knew from the start that I wouldn't be able to compete on sophisticated model design or extensive hyperparameter tuning. I decided to make smart use of external data instead.
\n
Thanks @abcbum ! Also congrats to your 13th place. We did indeed apply the same reasoning :D
", "rawMarkdown": "Thanks @abcbum ! Also congrats to your 13th place. We did indeed apply the same reasoning :D ", "votes": 1 }, { "id": 1431803, "postDate": "2021-08-03T12:49:24.380Z", "content": "There is one more similarity. Yours and @abcbum are the only 2 solutions (at least as far as I remember…among the dozen odd which I read) which seemed to make pseudo labelling work and contribute to the score…
", "rawMarkdown": "There is one more similarity. Yours and @abcbum are the only 2 solutions (*at least as far as I remember...among the dozen odd which I read*) which seemed to make pseudo labelling work and contribute to the score...", "votes": 1 }, { "id": 1432197, "postDate": "2021-08-03T13:05:18.887Z", "content": "@allohvk, thank you so much but actually no, hxcc's 3rd place solution also used it
\n\n", "rawMarkdown": "@allohvk, thank you so much but actually no, [hxcc's 3rd place solution](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258095) also used it\n>I then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.\n", "votes": 1 }, { "id": 1432653, "postDate": "2021-08-03T13:28:58.427Z", "content": "I then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.
\n
oh ok…got it. yes! I did read that but somehow didnt register in my memory..I think I quickly saw the last half of his comments (what worked) and didnt read in full detail. sorry
", "rawMarkdown": "oh ok...got it. yes! I did read that but somehow didnt register in my memory..I think I quickly saw the last half of his comments (what worked) and didnt read in full detail. sorry", "votes": 1 } ] }, { "id": 1456274, "postDate": "2021-08-06T20:52:57.850Z", "content": "Thank you so much @mathislucka for the clear kernel and congrats on the gold! I came up with the exact idea about sentence embedding representing readability too haha.
\nI have two questions for you:
\n(1) Why didn't you use the sentence transformer model (cosine similarity) instead of the paraphrase model to retrieve the top 5 most similar sentences from the external dataset? Wouldn't using the paraphrase model retrieve the most similar sentences in terms of semantics as opposed to readability? Two sentences could have opposite meaning but same readability (e.g. \"I love this restaurant\" vs. \"I hate this restaurant\")
\n(2) Why did you drop the sentence transformer model from your ensemble in your final solution?
\nThank you so much and congrats again!
", "rawMarkdown": "Thank you so much @mathislucka for the clear kernel and congrats on the gold! I came up with the exact idea about sentence embedding representing readability too haha. \n\nI have two questions for you:\n\n(1) Why didn't you use the sentence transformer model (cosine similarity) instead of the paraphrase model to retrieve the top 5 most similar sentences from the external dataset? Wouldn't using the paraphrase model retrieve the most similar sentences in terms of semantics as opposed to readability? Two sentences could have opposite meaning but same readability (e.g. \"I love this restaurant\" vs. \"I hate this restaurant\")\n\n(2) Why did you drop the sentence transformer model from your ensemble in your final solution? \n\nThank you so much and congrats again!", "votes": 1, "replies": [ { "id": 1456376, "postDate": "2021-08-06T22:08:56.367Z", "content": "(1) I didn't get a good \"difficulty sentence transformer\" up until close to the competition deadline. There was simply no time to do the data selection again using the difficulty model. My main motivation for choosing the paraphrase model was that I wanted data that was similar to the training data (similar in the sense of semantic similarity) because I expected that the test data would also be similar to the training data. It didn't really matter if the retrieved sentence had a different difficulty because I used a separate model to predict the difficulty of all retrieved sentences and then filtered these predictions on standard error of the original sample to keep roughly the same distribution. My score actually got worse quite a bit when I didn't do the filtering step. I also tried different top k values for the retrieval and 5 to 10 additional data snippets per train sample seemed to be the sweet spot. When selecting only data that was a bit further away from my training samples (cosine similarity wise) the score would also get worse.
\n(2) I actually submitted a model which included the sentence transformer model but it scored 0.447 both on the private and public leaderboard. I decided to also choose one model without the sentence transformer model because I did not want to take the risk of submitting 2 very similar models. Fun fact: my best performing model (which I didn't select) even scored 0.445 on the private test set but it was also not using the sentence transformer model. Also the sentence transformer model took quite a bit of time because each fold needed to encode both the train and the test set. With one submission I even hit the timeout. I think the model that timed out would have performed even better because it was combining the best of both worlds.
", "rawMarkdown": "(1) I didn't get a good \"difficulty sentence transformer\" up until close to the competition deadline. There was simply no time to do the data selection again using the difficulty model. My main motivation for choosing the paraphrase model was that I wanted data that was similar to the training data (similar in the sense of semantic similarity) because I expected that the test data would also be similar to the training data. It didn't really matter if the retrieved sentence had a different difficulty because I used a separate model to predict the difficulty of all retrieved sentences and then filtered these predictions on standard error of the original sample to keep roughly the same distribution. My score actually got worse quite a bit when I didn't do the filtering step. I also tried different top k values for the retrieval and 5 to 10 additional data snippets per train sample seemed to be the sweet spot. When selecting only data that was a bit further away from my training samples (cosine similarity wise) the score would also get worse.\n\n(2) I actually submitted a model which included the sentence transformer model but it scored 0.447 both on the private and public leaderboard. I decided to also choose one model without the sentence transformer model because I did not want to take the risk of submitting 2 very similar models. Fun fact: my best performing model (which I didn't select) even scored 0.445 on the private test set but it was also not using the sentence transformer model. Also the sentence transformer model took quite a bit of time because each fold needed to encode both the train and the test set. With one submission I even hit the timeout. I think the model that timed out would have performed even better because it was combining the best of both worlds.", "votes": 3 } ] }, { "id": 1451757, "postDate": "2021-08-05T12:29:31.390Z", "content": "Congratulations and Thank you for sharing your approach. Why did you choose 6 fold CV vs lets say 5 fold CV ? Also 6 fold CV was random or stratified some way ?
", "rawMarkdown": "Congratulations and Thank you for sharing your approach. Why did you choose 6 fold CV vs lets say 5 fold CV ? Also 6 fold CV was random or stratified some way ? ", "votes": 1 }, { "id": 1445198, "postDate": "2021-08-04T08:05:53.770Z", "content": "Congrats @mathislucka and Thanks I was looking for some NLP resources.
", "rawMarkdown": "Congrats @mathislucka and Thanks I was looking for some NLP resources.", "votes": 1 }, { "id": 1428999, "postDate": "2021-08-03T10:15:25.973Z", "content": "Congratulations @mathislucka! That was some clever use of external data, thanks a lot for sharing!
", "rawMarkdown": "Congratulations @mathislucka! That was some clever use of external data, thanks a lot for sharing!", "votes": 1 }, { "id": 1426438, "postDate": "2021-08-03T07:33:47.900Z", "content": "I bet your teachers are proud of you. PS - Pun intended
\nCongratulations on the solo win @mathislucka !
", "rawMarkdown": "I bet your teachers are proud of you. PS - Pun intended \n\nCongratulations on the solo win @mathislucka !", "votes": 1 }, { "id": 1426239, "postDate": "2021-08-03T07:24:09.300Z", "content": "Great performance, well done
\nThanks sharing your approach
\nCongrats for your first place
Congrats! I am little confued about the ridge regression for ensemble. Is it similar to stacking or something else?
", "rawMarkdown": "Congrats! I am little confued about the ridge regression for ensemble. Is it similar to stacking or something else?", "votes": 1, "replies": [ { "id": 1425825, "postDate": "2021-08-03T07:05:00.570Z", "content": "Yes, it's stacking. Say I have trained 2 6-fold models: albert and deberta. This gives me a total of 12 models. For each model I predict the fold that was not seen during training, giving me 2 model predictions for each fold. I then merge all out of fold predictions and make a new 6-fold split. Then, I use these folds to train a RidgeRegression model. This results in 6 RidgeRegression models each using albert and deberta predictions as an input. I then averaged the 6 ridge models for submissions (simple average)
", "rawMarkdown": "Yes, it's stacking. Say I have trained 2 6-fold models: albert and deberta. This gives me a total of 12 models. For each model I predict the fold that was not seen during training, giving me 2 model predictions for each fold. I then merge all out of fold predictions and make a new 6-fold split. Then, I use these folds to train a RidgeRegression model. This results in 6 RidgeRegression models each using albert and deberta predictions as an input. I then averaged the 6 ridge models for submissions (simple average)", "votes": 13 }, { "id": 1426203, "postDate": "2021-08-03T07:22:16.533Z", "content": "Thanks for the clear explanation!
", "rawMarkdown": "Thanks for the clear explanation!" } ] }, { "id": 1425063, "postDate": "2021-08-03T06:34:27.643Z", "content": "Congrats! Thanks for your sharing, by the way, have you considered to use dropout in the transformer structures?
", "rawMarkdown": "Congrats! Thanks for your sharing, by the way, have you considered to use dropout in the transformer structures?", "votes": 1, "replies": [ { "id": 1425878, "postDate": "2021-08-03T07:07:17.707Z", "content": "Thanks. I tried dropout but it didn't work very well for me.
", "rawMarkdown": "Thanks. I tried dropout but it didn't work very well for me.", "votes": 1 } ] }, { "id": 1426231, "postDate": "2021-08-03T07:23:39.433Z", "content": "Great and simple idea! I always find it amazing when people arrive with fresh ideas and just win by approaching the problem from a very innovative angle! Big congrats and thanks for sharing
", "rawMarkdown": "Great and simple idea! I always find it amazing when people arrive with fresh ideas and just win by approaching the problem from a very innovative angle! Big congrats and thanks for sharing ", "votes": 2 }, { "id": 1725149, "postDate": "2022-03-16T21:08:41.627Z", "content": "Hi, I'm running the 2nd notebook and encode_and_save() never finishes running. The df_to_csv() generates the csv very quickly but the encode_and_save() ran for 10hr before my colab notebook died.. Any suggestions?
", "rawMarkdown": "Hi, I'm running the 2nd notebook and encode_and_save() never finishes running. The df_to_csv() generates the csv very quickly but the encode_and_save() ran for 10hr before my colab notebook died.. Any suggestions?\n" }, { "id": 1578286, "postDate": "2021-11-10T20:20:08.440Z", "content": "Es ist Wunderbar Mathis. Though 3 months delayed for congratulation you.
", "rawMarkdown": "Es ist Wunderbar Mathis. Though 3 months delayed for congratulation you." }, { "id": 1478650, "postDate": "2021-08-18T05:09:21.577Z", "content": "Congratulations on your performance and thanks for sharing the approach !!
", "rawMarkdown": "Congratulations on your performance and thanks for sharing the approach !!" }, { "id": 1470231, "postDate": "2021-08-13T11:45:11.943Z", "content": "Congrats @mathislucka
\nThanks for sharing!
Congratulations bro, you really made a good explanation 👌👌🙏
", "rawMarkdown": "Congratulations bro, you really made a good explanation 👌👌🙏" }, { "id": 1464133, "postDate": "2021-08-10T13:09:58.410Z", "content": "Congrats!!!
", "rawMarkdown": "Congrats!!!" }, { "id": 1461050, "postDate": "2021-08-09T07:31:00Z", "content": "great work Congratulations
\ndid you remove all examples in external data that are outside minimum and maximum target value in original training set ??
No, I did not remove them.
", "rawMarkdown": "No, I did not remove them.", "votes": 1 } ] }, { "id": 1457523, "postDate": "2021-08-07T13:04:35.813Z", "content": "Congratulations on Solo win and thank you for sharing your approach…!!!
", "rawMarkdown": "Congratulations on Solo win and thank you for sharing your approach...!!!" }, { "id": 1456639, "postDate": "2021-08-07T03:03:26.433Z", "content": "Congratulations @mathislucka on winning the competition.
", "rawMarkdown": "Congratulations @mathislucka on winning the competition." }, { "id": 1456427, "postDate": "2021-08-06T22:46:08.560Z", "content": "Congratulation for this huge achievement!!!
", "rawMarkdown": "Congratulation for this huge achievement!!!" }, { "id": 1456370, "postDate": "2021-08-06T22:03:49.183Z", "content": "Thankyou for sharing this articles
", "rawMarkdown": "Thankyou for sharing this articles" }, { "id": 1455554, "postDate": "2021-08-06T15:50:10.660Z", "content": "congrats for the gold and thanks for sharing the process
", "rawMarkdown": "congrats for the gold and thanks for sharing the process" }, { "id": 1452388, "postDate": "2021-08-05T15:36:01.503Z", "content": "Great use of external data!
\nCongrats!
Thanks for sharing your solution and congratulations for the first place!
", "rawMarkdown": "Thanks for sharing your solution and congratulations for the first place!" }, { "id": 1451775, "postDate": "2021-08-05T12:34:25.050Z", "content": "Congratulations on the solo win @mathislucka !!!
", "rawMarkdown": "Congratulations on the solo win @mathislucka !!!" }, { "id": 1450413, "postDate": "2021-08-05T05:18:34.257Z", "content": "Thanks for kaggle for providing free course for students.
", "rawMarkdown": "Thanks for kaggle for providing free course for students." }, { "id": 1449687, "postDate": "2021-08-05T00:00:33.407Z", "content": "I like that you won with a smart approach to the data and 'few' models instead of ensembling dozens of models
\ncongrats!!
", "rawMarkdown": "I like that you won with a smart approach to the data and 'few' models instead of ensembling dozens of models\n\ncongrats!!", "replies": [ { "id": 1456387, "postDate": "2021-08-06T22:16:53.363Z", "content": "Well :D Actually I did ensemble quite a few models. In total, I count 38 transformer models in my final submission.
", "rawMarkdown": "Well :D Actually I did ensemble quite a few models. In total, I count 38 transformer models in my final submission.", "votes": 1 }, { "id": 1456840, "postDate": "2021-08-07T05:38:57.237Z", "content": "Wow, thanks for correction! I didn't get that from your explanation before. To me, what you did is quite magic. That's truly well deserved. Congrats again!
", "rawMarkdown": "Wow, thanks for correction! I didn't get that from your explanation before. To me, what you did is quite magic. That's truly well deserved. Congrats again!" } ] }, { "id": 1440175, "postDate": "2021-08-03T17:58:12.027Z", "content": "Congrats @mathislucka
\nThanks for sharing
Congratulations and thanks to your sharING!
", "rawMarkdown": "Congratulations and thanks to your sharING!" }, { "id": 1429703, "postDate": "2021-08-03T10:54:12.960Z", "content": "Wow Congratulations for this incredible performance. The solution approach is too detailed and well written. Thanks!
", "rawMarkdown": "Wow Congratulations for this incredible performance. The solution approach is too detailed and well written. Thanks!" }, { "id": 1429555, "postDate": "2021-08-03T10:46:37.073Z", "content": "Congratulations! Well-deserved win, as everything seems so well thought out! Thanks for sharing
", "rawMarkdown": "Congratulations! Well-deserved win, as everything seems so well thought out! Thanks for sharing" }, { "id": 1429510, "postDate": "2021-08-03T10:44:48.437Z", "content": "Very well done @mathislucka a very simple and elegant solution , really inspiring , I was rather too busy after complex approaches I completely forgot about smart use of external data , new lessons learned
\nI had the idea of using sentence transformers as well , as I use it in my day job but I was just too lazy
\nThanks for sharing
Thanks for sharing the approach and Congratulations on the win.
", "rawMarkdown": "Thanks for sharing the approach and Congratulations on the win." }, { "id": 1428427, "postDate": "2021-08-03T09:27:48.977Z", "content": "Thanks sharing your approach
\nCongrats for your first place
Congratulations and thanks for sharing your approach!
", "rawMarkdown": "Congratulations and thanks for sharing your approach!" }, { "id": 1426786, "postDate": "2021-08-03T07:52:56.877Z", "content": "Congratulations @mathislucka !
", "rawMarkdown": "Congratulations @mathislucka !" }, { "id": 1426788, "postDate": "2021-08-03T07:52:56.877Z", "content": "Congratulations @mathislucka !
", "rawMarkdown": "Congratulations @mathislucka !" }, { "id": 1626330, "postDate": "2021-12-22T18:10:17.447Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1447827, "postDate": "2021-08-04T16:00:34.320Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1588210, "postDate": "2021-11-19T09:50:45.987Z", "content": "Thank you for the detailed breakdown
", "rawMarkdown": "Thank you for the detailed breakdown" }, { "id": 1468779, "postDate": "2021-08-12T15:09:51.113Z", "content": "Thank you for your sharing
", "rawMarkdown": "Thank you for your sharing" }, { "id": 1448438, "postDate": "2021-08-04T17:35:03.783Z", "content": "thank you for sharing.
", "rawMarkdown": "thank you for sharing." }, { "id": 1443170, "postDate": "2021-08-04T02:00:01.720Z", "content": "Thanks sharing your approach
", "rawMarkdown": "Thanks sharing your approach" }, { "id": 1430249, "postDate": "2021-08-03T11:27:12.253Z", "content": "Thanks for your sharing
", "rawMarkdown": "Thanks for your sharing" }, { "id": 1427313, "postDate": "2021-08-03T08:19:30.080Z", "content": "Thanks sharing your approach
", "rawMarkdown": "Thanks sharing your approach" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:46:04Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6896438, "solution_url": "/competitions/commonlitreadabilityprize/writeups/mathis-lucka-1st-place-solution-external-data-teac" }