# 61th place solution

61th place solution
First of all,  I'd like to thank the competition host for organizing such an intriguing contest and my team members.
Many of the ideas for this solution were contributed by my teammates.
Summary
The pipeline of our solution is as follows
![](assets/10711071a9b4d697b62b5b6f.png)
This solution was inspired by this
[notebook](https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect)
. Thanks for sharing this great notebook.
Strategy
The public leaderboard was derived from a single prompt, but based on
[discussion](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/443223)
, the private leaderboard encompasses over 100 prompts. This observation led me to place a heightened emphasis on cross-validation (CV). However, the CV too was based on merely four prompts, making it somewhat untrusted.
Given the uncertainty in both CV and the public leaderboard, I opted for a strategy that emphasizes robustness. Consequently, I integrated various backbones and input texts into the ensemble, optimizing to reduce the inference time. Furthermore, considering that the evaluation metric was RMSE, ensuring prediction accuracy was paramount. Therefore, even models with suboptimal individual scores were incorporated into the ensemble.
I couldn't effectively utilize the prompt_text, so I shifted my strategy to shorten the input length and reduce inference time.
BERT Model Training
Training used Nakama's excellent
[notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)
on Feedback3.
Common to all
epoch : 3
schduler : cosine
No dropout
backbone
text
maxlen
loss
freeze
CV
deberta-v3-base
title question summary
512
SmoothL1Loss
4
0.54756
deberta-v3-large
question summary
512
SmoothL1Loss
12
0.53988
deberta-xlarge
summary title question
512
SmoothL1Loss
2
0.54400
electra-large
question summary
1024
RMSE
2
0.55925
muppet-roberta-large
question summary
512
SmoothL1Loss
4
0.56337
GBDT
LightGBM, CatBoost, and XGBoost training were all parameter optimized with optuna.
The amount of features is based on this
[notebook](https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail)
.
did not work
awp
fgm
SVR
(feedback3 chirs notebook)[
[https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x](https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x)
]
pseudo (might have worked)
The CV of feedback3 and old commonlit single had a gain of more than 0.01 on any backbone, but it worsened when put in GBDT, so it was not used.
text cleaning
Could capitalization and STOPWORD have also been a feature of wording?
wording anomaly detection
By removing data with extremely poor wording, the RMSE deteriorated significantly, so an anomaly detection model was created specifically for this, but it was abandoned due to lack of progress in learning.
FlanT5 encoder
The encoder part was taken from FlanT5, an encoder-decoder model often used in actual summarization, and used in the backbone, but the CV was 0.8x, which was very poor, so it was not used.
Important Citations
[2stage pipeline](https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect)
[feedback3 train notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)
[no dropout](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729)
[get features](https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail)