# 114th Place Solution: standard_error Works as a Feature 

Thank you all !
At first, I would like to thank competition hosts, and Kaggle community.
I enjoyed, learned so many things, and was encouraged by you all.
Especially, I want to give great thanks to
[@andretugan](https://www.kaggle.com/andretugan)
,
[@maunish](https://www.kaggle.com/maunish)
, and
[@rhtsingh](https://www.kaggle.com/rhtsingh)
. Their notebooks and discussions inspired and led me a lot.
This is my first NLP competition. I'm very surprised to win silver medal, and so glad.
Key Points
In this solution, there are 3 important points.
Standard Error as a Feature
Model Diversity (BERT variants, custom head, LGBM……)
Stacking Strategy (weighted ensemble)
I will explain them one by one.
I did not use MLM pretraining nor external datasets. Just train dataset only.
Standard Error as a Feature (
MOST IMPORTANT
)
I noticed the standard_error is a key feature 3 days before the deadline.
The experiment notebook is here.
[https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature](https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature)
Main Result
Stacking Layer
CV Score
Boost
Linear (baseline)
0.4567
-
LightGBM (without standard_error)
0.4596
-0.0029
LightGBM (with standard_error)
0.3959
0.069
This experiment shows that standard_error is a key feature comparable to BERTs prediction.
The problem is how to get the standard_error, since it is not given in test set.
I deal with this problem by using LightGBM for standard_error prediction.
Here, feature engineering appeared.
Model Diversity
LB was very unstable for a single model prediction.  So, I decided to cover the instability with
model diversity.
I tried to train many diverse models.
architecture (Roberta, Electra, XLNet, Funnel, Albert, DeBerta, BART……LGBM)
size (base or large)
head (attention, conv1D, meanpooling and so on)
However, since I am not good at fine-turning or something unknown, only 8 models survived.
8 models are shown below.
Architecture
Size
Head
CV
Roberta
large
Attentions (using last 4 layers)
0.4757
Roberta
large
Attentions-Conv1D
0.4761
Roberta
base
Attention
0.4746
Electra
large
Attentions (using last 4 layers)
0.4760
XLNet
large
Attentions (using last 4 layers)
0.4850
Electra
large
Attentions-Conv1D
0.4818
Roberta
large
Meanpooling
0.4772
Funnel
large
Attention
0.4836
Stacking Strategy
Here is the stacking scheme figure.
It is very complicated, especially standard_error prediction part.
If not using standard_error, stacking scheme would be much simpler.
![stacking scheme](assets/d33fabf7afcc194fa103b298.png)
Results
My models overfitted somewhat.
Strategy
CV
Public LB
Private LB
Stacking BERTs
0.457
0.458
0.461
LGBM using BERTs Prediction + S.E.
0.454
0.458
0.459
Stacking BERTs + LGBM
0.452
0.456
0.458
What did not Work
MLM pretraining
Data augmentation (like
[Easy Data Augmentation](https://arxiv.org/abs/1901.11196)
)
I set
data_collator = DataCollatorForLanguageModeling
, then dynamically masked texts. I expect this trick works, but not.
As a result, I just set
data_collator = DataCollatorWithPadding
. Dynamically padding is realized.
Back translation
Layer reinitialization
ALBERT, DeBerta, BART (It is too difficult to finetune them……)
SmartBatchingSampler
The data order fixed to a extent would not be good in the small dataset competition.
[Multi-head Attention](https://www.kaggle.com/atsushiiwasaki/clrp-attention-head-variants)
I tried to divide the simple attention head into some blocks. This does not work.
Other Ideas
Stratify the dataset on prediction error
Stratified K Fold on target, s.e. and target & s.e. seems not to work enough. So, I decided to stratify the dataset on prediction error. There may be some important feature distributions to stratify.
[CLRP: Stratify on Target & S.E. Bins from Sturge's](https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-target-s-e-bins-from-sturge-s)
[CLRP: Stratify On Predictability](https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-predictability)
[Split based on Target & Standard Error](https://www.kaggle.com/atsushiiwasaki/split-based-on-target-standard-error)
However, this strategy would not work so well.
Larger depth in attention head in Funnel-large
In Roberta, Electra and XLNet, the depth of attention head is set 512~514 (I choose the depth based on the embedding size).
On the other hand, it is set 768 in Funnel with only last hidden layer, and this worked.
Maybe, this is due to Funnel architecture. We can extract the best encoded representation from last hidden layer in Funnel encoder using deep attention head having more representation ability.
Ideas that I did not come up with
[Psuedo Labeling](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257844)
[Gaussian Process Regression](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258554)
[Sentence Transformers](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260800)
[Bradley-Terry (BT) Loss](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258312)
[Binary Pre-training](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260939)
Notebooks Related to This Solution
Data
[Dataset Size Dependency](https://www.kaggle.com/atsushiiwasaki/how-many-samples-do-i-want)
Model Training
BERT Baseline (
[train](https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-train)
,
[infer](https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-infer)
)
BERT variants for target
Electra-large + Attentions + Conv1D (
[train](https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-attentions-conv1d-train)
,
[infer](https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-att-conv1d-infer)
)
[Roberta-large + Attentions](https://www.kaggle.com/atsushiiwasaki/clrp-roberta-large-att-mask-act-infer)
[Roberta-base + Attention](https://www.kaggle.com/atsushiiwasaki/clrp-roberta-base-oof-task-fineturning-infer)
[Funnel-large + Attention](https://www.kaggle.com/atsushiiwasaki/clrp-funnel-large-attentions-act)
LGBM for standard_error & target
[CLRP: Standard Error Feature EDA](https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-feature-eda?scriptVersionId=69709698)
This notebook is not EDA, but I have named……
Model Stacking
[CLRP: Model Stacking with Standard Error](https://www.kaggle.com/atsushiiwasaki/clrp-model-stacking-with-standard-error)