# 3rd place solution (0.447 private - 3 models simple average)

update
Sept 10, 2021 - added link to submission notebook
Aug 16, 2021 - added code link
tl;dr at the bottom
First, I would like to thank Kaggle and CommonLit for hosting this competition and the community here at Kaggle for making this platform a wonderful place to learn. This is my second competition at Kaggle, with the first one being the Coleridge Initiative; I have to say the past three months or so has been a tremendous journey. If anyone told me three months ago that I would end up in 3rd place in a Kaggle competition, I would not believe it. None of this would be possible without Kaggle and its incredible community.
Special thanks to:
[@rhtsingh](https://www.kaggle.com/rhtsingh)
for sharing so many great notebooks with detailed and excellent explanations. Those notebooks guided my first few weeks in this competition.
[@andretugan](https://www.kaggle.com/andretugan)
for sharing a simple yet elegant solution. Some parts of my final model are from his approach.
Analysis of the objective
Before I go over my approach, I'd like to share my point of view on the competition objective. Please correct me if what I say is wrong or doesn't make sense to you.
This competition is characterized by small training data and noisy, non-intuitive to understand target values. The target is generated by the Bradley-Terry algorithm from a relatively small number of pair-wise comparison data. From my intuition, Bradley-Terry acts like an Elo rating system where match results are translated to raw numbers representing the "strength" of each player. (again, this could be wrong and is merely my point of view) In this competition, the texts are players, and the "strength" is "easiness to read," where an easier-to-read text would score higher.
Training data + public/private data nets to little less than 5k samples, and each sample is compared with others 46.47 times on average. That means each text was compared to only ~1% of the entire dataset on average. Imagine a chess player's Elo where the player has only played against 1% of the whole chess player population, and the opponents are selected at random. I believe most of the noise is due to the small number of comparisons.
The objective of the competition is to predict where an arbitrary text would land on that leaderboard generated from 5k texts.
Path to public LB 0.442:
I've played with Roberta base mostly at the beginning of the competition, and with a gru+multihead attention on the last hidden layer + learning rate decay, I was able to get
~0.473 on both public and private LB.
I then switched to large and with the same setup I was at around
0.465-0.467 on both public and private LB.
After which I learned from
[@andretugan](https://www.kaggle.com/andretugan)
's notebook and incorporated 3 group lr decay, simple attention head, and validation step policy into my solution
Roberta-large + simple attention head + learning rate decay:
0.462 on both LB
0.462 + weighted average of all layers and multisample dropout (from google quest #1 solution) + separate lr for head: **0.459 on public **
0.459 + a 3 stage lr schedule:
0.454 on public
![custom schedule](assets/84bf32502f92d5f17fdfbc4a.png)
0.454 + early stop at the beginning of 3rd epoch:
0.453 on public
However,
the private did not agree with the improvement to 0.453
, which I didn't know at the time.
Private LB stayed at ~0.461
An ensemble of 2 Roberta large and a Roberta base trained this way, however, got
0.454 on public
and
0.456 on private
.
I then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.
Using all folds to label lead to unrealistic low cv (due to leak) and no improvement to LB, so I used 5 fold pseudo label described in google quest #1 solution.
I first train the model on pseudo labeled data and then finish up with low lr 3 epoch and an lr scheduler with a decaying rate (1,0.5,0.25) for each epoch, no warmup.
2 cycles improved my public to 0.448:
0.453 -> 0.450 -> 0.448
yet private still stayed at ~0.461
I then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with
0.448 on public and 0.449 on private
ensemble of a Roberta-large and a Deberta-large:
0.442 on public, 0.448 on private
Another model trained with a further cycle labeled with 0.442 public was added to the blend and got
0.442/0.447 on public/private
. This is the best scoring out of the two I selected.
Best private was
0.446
, which I did not select because it didn't improve public LB. (I turned my eyes away from cv long ago)
The difference of this one to the 0.447 private is the third model. The new model is trained with training data mixed into pseudo labeled extra data. The train data however, was relabeled and mixed with the original train data with weight of 0.5. I tested 0.8 and 1 but 0.5 performed the best.
what I wish I had time to do:
ensembling techniques, In my solution I only used simple average as I didn't have time to explore ensembling techniques
better pseudo-label pipeline. I only tried a few variants and I believe some are always better than others
pairwise comparison model which simply outputs which text is easier to read to generate raw data for Bradley-Terry algorithm, convert it to target score using Bradley-Terry algorithm, and then maybe recalibrate to competition training data distribution. (This might be a wild-goose chase, but seems like a very promising one to me)
hardware:
1080ti at the beginning of the competition
bought a 3090 when switching to large models
tl;dr:
what worked:
3 group decaying lr for encoder
simple attention head
separate lr for head
multi-sample dropout
weighted average of hidden layers
early stop
fold specific pseudo-label
Deberta-large
code:
[https://github.com/Danielhuxc/CLRP-solution](https://github.com/Danielhuxc/CLRP-solution)
notebook:
[https://www.kaggle.com/potatoc5/final-sub](https://www.kaggle.com/potatoc5/final-sub)
PS: I am looking for ML/DL job (preferably within Canada), please DM me if you would like to get in touch.