{
  "id": 257797,
  "title": "114th Place Solution: standard_error Works as a Feature ",
  "content": "<h1>Thank you all !</h1>\n<p>At first, I would like to thank competition hosts, and Kaggle community.<br>\nI enjoyed, learned so many things, and was encouraged by you all.<br>\nEspecially, I want to give great thanks to  <a href=\"https://www.kaggle.com/andretugan\" target=\"_blank\">@andretugan</a>, <a href=\"https://www.kaggle.com/maunish\" target=\"_blank\">@maunish</a>, and <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a>. Their notebooks and discussions inspired and led me a lot.</p>\n<p>This is my first NLP competition. I'm very surprised to win silver medal, and so glad.</p>\n<p><br></p>\n<h1>Key Points</h1>\n<p>In this solution, there are 3 important points.</p>\n<ul>\n<li>Standard Error as a Feature</li>\n<li>Model Diversity (BERT variants, custom head, LGBM……)</li>\n<li>Stacking Strategy (weighted ensemble)</li>\n</ul>\n<p>I will explain them one by one.<br>\nI did not use MLM pretraining nor external datasets. Just train dataset only.</p>\n<p><br></p>\n<h3>Standard Error as a Feature (<strong>MOST IMPORTANT</strong>)</h3>\n<p>I noticed the standard_error is a key feature 3 days before the deadline.<br>\nThe experiment notebook is here.<br>\n<a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature\" target=\"_blank\">https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature</a></p>\n<p><strong>Main Result</strong></p>\n<table>\n<thead>\n<tr>\n<th>Stacking Layer</th>\n<th>CV Score</th>\n<th>Boost</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Linear (baseline)</td>\n<td>0.4567</td>\n<td>-</td>\n</tr>\n<tr>\n<td>LightGBM (without standard_error)</td>\n<td>0.4596</td>\n<td>-0.0029</td>\n</tr>\n<tr>\n<td>LightGBM (with standard_error)</td>\n<td>0.3959</td>\n<td>0.069</td>\n</tr>\n</tbody>\n</table>\n<p>This experiment shows that standard_error is a key feature comparable to BERTs prediction.<br>\nThe problem is how to get the standard_error, since it is not given in test set.</p>\n<p>I deal with this problem by using LightGBM for standard_error prediction.<br>\nHere, feature engineering appeared.</p>\n<p><br></p>\n<h3>Model Diversity</h3>\n<p>LB was very unstable for a single model prediction.  So, I decided to cover the instability with <br>\n model diversity.<br>\nI tried to train many diverse models.</p>\n<ul>\n<li>architecture (Roberta, Electra, XLNet, Funnel, Albert, DeBerta, BART……LGBM)</li>\n<li>size (base or large)</li>\n<li>head (attention, conv1D, meanpooling and so on)</li>\n</ul>\n<p>However, since I am not good at fine-turning or something unknown, only 8 models survived.<br>\n8 models are shown below.</p>\n<table>\n<thead>\n<tr>\n<th>Architecture</th>\n<th>Size</th>\n<th>Head</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Roberta</td>\n<td>large</td>\n<td>Attentions (using last 4 layers)</td>\n<td>0.4757</td>\n</tr>\n<tr>\n<td>Roberta</td>\n<td>large</td>\n<td>Attentions-Conv1D</td>\n<td>0.4761</td>\n</tr>\n<tr>\n<td>Roberta</td>\n<td>base</td>\n<td>Attention</td>\n<td>0.4746</td>\n</tr>\n<tr>\n<td>Electra</td>\n<td>large</td>\n<td>Attentions (using last 4 layers)</td>\n<td>0.4760</td>\n</tr>\n<tr>\n<td>XLNet</td>\n<td>large</td>\n<td>Attentions (using last 4 layers)</td>\n<td>0.4850</td>\n</tr>\n<tr>\n<td>Electra</td>\n<td>large</td>\n<td>Attentions-Conv1D</td>\n<td>0.4818</td>\n</tr>\n<tr>\n<td>Roberta</td>\n<td>large</td>\n<td>Meanpooling</td>\n<td>0.4772</td>\n</tr>\n<tr>\n<td>Funnel</td>\n<td>large</td>\n<td>Attention</td>\n<td>0.4836</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<h3>Stacking Strategy</h3>\n<p>Here is the stacking scheme figure.<br>\nIt is very complicated, especially standard_error prediction part.</p>\n<p>If not using standard_error, stacking scheme would be much simpler.<br>\n<img src=\"https://i.postimg.cc/prVYT4mh/CLRP-Stacking-Scheme.png\" alt=\"stacking scheme\"><br>\n<br></p>\n<h1>Results</h1>\n<p>My models overfitted somewhat.</p>\n<table>\n<thead>\n<tr>\n<th>Strategy</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Stacking BERTs</td>\n<td>0.457</td>\n<td>0.458</td>\n<td>0.461</td>\n</tr>\n<tr>\n<td>LGBM using BERTs Prediction + S.E.</td>\n<td>0.454</td>\n<td>0.458</td>\n<td>0.459</td>\n</tr>\n<tr>\n<td>Stacking BERTs + LGBM</td>\n<td>0.452</td>\n<td>0.456</td>\n<td>0.458</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<h1>What did not Work</h1>\n<ul>\n<li>MLM pretraining</li>\n<li>Data augmentation (like <a href=\"https://arxiv.org/abs/1901.11196\" target=\"_blank\">Easy Data Augmentation</a>)<ul>\n<li>I set <code>data_collator = DataCollatorForLanguageModeling</code>, then dynamically masked texts. I expect this trick works, but not.</li>\n<li>As a result, I just set <code>data_collator = DataCollatorWithPadding</code>. Dynamically padding is realized.</li></ul></li>\n<li>Back translation</li>\n<li>Layer reinitialization</li>\n<li>ALBERT, DeBerta, BART (It is too difficult to finetune them……)</li>\n<li>SmartBatchingSampler<ul>\n<li>The data order fixed to a extent would not be good in the small dataset competition.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-attention-head-variants\" target=\"_blank\">Multi-head Attention</a><ul>\n<li>I tried to divide the simple attention head into some blocks. This does not work.<br>\n<br></li></ul></li>\n</ul>\n<h1>Other Ideas</h1>\n<ul>\n<li>Stratify the dataset on prediction error<ul>\n<li>Stratified K Fold on target, s.e. and target &amp; s.e. seems not to work enough. So, I decided to stratify the dataset on prediction error. There may be some important feature distributions to stratify.</li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-target-s-e-bins-from-sturge-s\" target=\"_blank\">CLRP: Stratify on Target &amp; S.E. Bins from Sturge's</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-predictability\" target=\"_blank\">CLRP: Stratify On Predictability</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/split-based-on-target-standard-error\" target=\"_blank\">Split based on Target &amp; Standard Error\n</a></li>\n<li>However, this strategy would not work so well.</li></ul></li>\n<li>Larger depth in attention head in Funnel-large<ul>\n<li>In Roberta, Electra and XLNet, the depth of attention head is set 512~514 (I choose the depth based on the embedding size).</li>\n<li>On the other hand, it is set 768 in Funnel with only last hidden layer, and this worked.</li>\n<li>Maybe, this is due to Funnel architecture. We can extract the best encoded representation from last hidden layer in Funnel encoder using deep attention head having more representation ability.</li></ul></li>\n</ul>\n<p><br></p>\n<h1>Ideas that I did not come up with</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257844\" target=\"_blank\">Psuedo Labeling</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258554\" target=\"_blank\">Gaussian Process Regression</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260800\" target=\"_blank\">Sentence Transformers</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258312\" target=\"_blank\">Bradley-Terry (BT) Loss</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260939\" target=\"_blank\">Binary Pre-training</a></li>\n</ul>\n<p><br></p>\n<h1>Notebooks Related to This Solution</h1>\n<ul>\n<li>Data<ul>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/how-many-samples-do-i-want\" target=\"_blank\">Dataset Size Dependency</a></li></ul></li>\n<li>Model Training<ul>\n<li>BERT Baseline (<a href=\"https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-train\" target=\"_blank\">train</a>, <a href=\"https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-infer\" target=\"_blank\">infer</a>)</li>\n<li>BERT variants for target<ul>\n<li>Electra-large + Attentions + Conv1D (<a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-attentions-conv1d-train\" target=\"_blank\">train</a>, <a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-att-conv1d-infer\" target=\"_blank\">infer</a>)</li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-roberta-large-att-mask-act-infer\" target=\"_blank\">Roberta-large + Attentions</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-roberta-base-oof-task-fineturning-infer\" target=\"_blank\">Roberta-base + Attention</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-funnel-large-attentions-act\" target=\"_blank\">Funnel-large + Attention</a></li></ul></li>\n<li>LGBM for standard_error &amp; target<ul>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-feature-eda?scriptVersionId=69709698\" target=\"_blank\">CLRP: Standard Error Feature EDA</a></li>\n<li>This notebook is not EDA, but I have named……</li></ul></li></ul></li>\n<li>Model Stacking<ul>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-model-stacking-with-standard-error\" target=\"_blank\">CLRP: Model Stacking with Standard Error</a></li></ul></li>\n</ul>",
  "comments": [
    {
      "id": 1430397,
      "author_name": "Nicholas Broad",
      "author_url": "",
      "post_date": "2021-08-03T11:36:34.360000",
      "content": "<p>If standard error is a feature, what did you use on the test set?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1430608,
          "author_name": "Atsushi Iwasaki",
          "author_url": "",
          "post_date": "2021-08-03T11:49:24.397000",
          "content": "<p><a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> Thank you for your comment. In addition to target prediction models (BERT variants), I created standard_error prediction model (LightGBM). The features for the model were engineered from the excerpts (for example,  number of sentences) and BERT variants predictions.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 1423435,
      "postDate": "2021-08-03T05:40:23.297Z",
      "content": "<h1>Thank you all !</h1>\n<p>At first, I would like to thank competition hosts, and Kaggle community.<br>\nI enjoyed, learned so many things, and was encouraged by you all.<br>\nEspecially, I want to give great thanks to  <a href=\"https://www.kaggle.com/andretugan\" target=\"_blank\">@andretugan</a>, <a href=\"https://www.kaggle.com/maunish\" target=\"_blank\">@maunish</a>, and <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a>. Their notebooks and discussions inspired and led me a lot.</p>\n<p>This is my first NLP competition. I'm very surprised to win silver medal, and so glad.</p>\n<p><br></p>\n<h1>Key Points</h1>\n<p>In this solution, there are 3 important points.</p>\n<ul>\n<li>Standard Error as a Feature</li>\n<li>Model Diversity (BERT variants, custom head, LGBM……)</li>\n<li>Stacking Strategy (weighted ensemble)</li>\n</ul>\n<p>I will explain them one by one.<br>\nI did not use MLM pretraining nor external datasets. Just train dataset only.</p>\n<p><br></p>\n<h3>Standard Error as a Feature (<strong>MOST IMPORTANT</strong>)</h3>\n<p>I noticed the standard_error is a key feature 3 days before the deadline.<br>\nThe experiment notebook is here.<br>\n<a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature\" target=\"_blank\">https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature</a></p>\n<p><strong>Main Result</strong></p>\n<table>\n<thead>\n<tr>\n<th>Stacking Layer</th>\n<th>CV Score</th>\n<th>Boost</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Linear (baseline)</td>\n<td>0.4567</td>\n<td>-</td>\n</tr>\n<tr>\n<td>LightGBM (without standard_error)</td>\n<td>0.4596</td>\n<td>-0.0029</td>\n</tr>\n<tr>\n<td>LightGBM (with standard_error)</td>\n<td>0.3959</td>\n<td>0.069</td>\n</tr>\n</tbody>\n</table>\n<p>This experiment shows that standard_error is a key feature comparable to BERTs prediction.<br>\nThe problem is how to get the standard_error, since it is not given in test set.</p>\n<p>I deal with this problem by using LightGBM for standard_error prediction.<br>\nHere, feature engineering appeared.</p>\n<p><br></p>\n<h3>Model Diversity</h3>\n<p>LB was very unstable for a single model prediction.  So, I decided to cover the instability with <br>\n model diversity.<br>\nI tried to train many diverse models.</p>\n<ul>\n<li>architecture (Roberta, Electra, XLNet, Funnel, Albert, DeBerta, BART……LGBM)</li>\n<li>size (base or large)</li>\n<li>head (attention, conv1D, meanpooling and so on)</li>\n</ul>\n<p>However, since I am not good at fine-turning or something unknown, only 8 models survived.<br>\n8 models are shown below.</p>\n<table>\n<thead>\n<tr>\n<th>Architecture</th>\n<th>Size</th>\n<th>Head</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Roberta</td>\n<td>large</td>\n<td>Attentions (using last 4 layers)</td>\n<td>0.4757</td>\n</tr>\n<tr>\n<td>Roberta</td>\n<td>large</td>\n<td>Attentions-Conv1D</td>\n<td>0.4761</td>\n</tr>\n<tr>\n<td>Roberta</td>\n<td>base</td>\n<td>Attention</td>\n<td>0.4746</td>\n</tr>\n<tr>\n<td>Electra</td>\n<td>large</td>\n<td>Attentions (using last 4 layers)</td>\n<td>0.4760</td>\n</tr>\n<tr>\n<td>XLNet</td>\n<td>large</td>\n<td>Attentions (using last 4 layers)</td>\n<td>0.4850</td>\n</tr>\n<tr>\n<td>Electra</td>\n<td>large</td>\n<td>Attentions-Conv1D</td>\n<td>0.4818</td>\n</tr>\n<tr>\n<td>Roberta</td>\n<td>large</td>\n<td>Meanpooling</td>\n<td>0.4772</td>\n</tr>\n<tr>\n<td>Funnel</td>\n<td>large</td>\n<td>Attention</td>\n<td>0.4836</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<h3>Stacking Strategy</h3>\n<p>Here is the stacking scheme figure.<br>\nIt is very complicated, especially standard_error prediction part.</p>\n<p>If not using standard_error, stacking scheme would be much simpler.<br>\n<img src=\"https://i.postimg.cc/prVYT4mh/CLRP-Stacking-Scheme.png\" alt=\"stacking scheme\"><br>\n<br></p>\n<h1>Results</h1>\n<p>My models overfitted somewhat.</p>\n<table>\n<thead>\n<tr>\n<th>Strategy</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Stacking BERTs</td>\n<td>0.457</td>\n<td>0.458</td>\n<td>0.461</td>\n</tr>\n<tr>\n<td>LGBM using BERTs Prediction + S.E.</td>\n<td>0.454</td>\n<td>0.458</td>\n<td>0.459</td>\n</tr>\n<tr>\n<td>Stacking BERTs + LGBM</td>\n<td>0.452</td>\n<td>0.456</td>\n<td>0.458</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<h1>What did not Work</h1>\n<ul>\n<li>MLM pretraining</li>\n<li>Data augmentation (like <a href=\"https://arxiv.org/abs/1901.11196\" target=\"_blank\">Easy Data Augmentation</a>)<ul>\n<li>I set <code>data_collator = DataCollatorForLanguageModeling</code>, then dynamically masked texts. I expect this trick works, but not.</li>\n<li>As a result, I just set <code>data_collator = DataCollatorWithPadding</code>. Dynamically padding is realized.</li></ul></li>\n<li>Back translation</li>\n<li>Layer reinitialization</li>\n<li>ALBERT, DeBerta, BART (It is too difficult to finetune them……)</li>\n<li>SmartBatchingSampler<ul>\n<li>The data order fixed to a extent would not be good in the small dataset competition.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-attention-head-variants\" target=\"_blank\">Multi-head Attention</a><ul>\n<li>I tried to divide the simple attention head into some blocks. This does not work.<br>\n<br></li></ul></li>\n</ul>\n<h1>Other Ideas</h1>\n<ul>\n<li>Stratify the dataset on prediction error<ul>\n<li>Stratified K Fold on target, s.e. and target &amp; s.e. seems not to work enough. So, I decided to stratify the dataset on prediction error. There may be some important feature distributions to stratify.</li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-target-s-e-bins-from-sturge-s\" target=\"_blank\">CLRP: Stratify on Target &amp; S.E. Bins from Sturge's</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-predictability\" target=\"_blank\">CLRP: Stratify On Predictability</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/split-based-on-target-standard-error\" target=\"_blank\">Split based on Target &amp; Standard Error\n</a></li>\n<li>However, this strategy would not work so well.</li></ul></li>\n<li>Larger depth in attention head in Funnel-large<ul>\n<li>In Roberta, Electra and XLNet, the depth of attention head is set 512~514 (I choose the depth based on the embedding size).</li>\n<li>On the other hand, it is set 768 in Funnel with only last hidden layer, and this worked.</li>\n<li>Maybe, this is due to Funnel architecture. We can extract the best encoded representation from last hidden layer in Funnel encoder using deep attention head having more representation ability.</li></ul></li>\n</ul>\n<p><br></p>\n<h1>Ideas that I did not come up with</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257844\" target=\"_blank\">Psuedo Labeling</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258554\" target=\"_blank\">Gaussian Process Regression</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260800\" target=\"_blank\">Sentence Transformers</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258312\" target=\"_blank\">Bradley-Terry (BT) Loss</a></li>\n<li><a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260939\" target=\"_blank\">Binary Pre-training</a></li>\n</ul>\n<p><br></p>\n<h1>Notebooks Related to This Solution</h1>\n<ul>\n<li>Data<ul>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/how-many-samples-do-i-want\" target=\"_blank\">Dataset Size Dependency</a></li></ul></li>\n<li>Model Training<ul>\n<li>BERT Baseline (<a href=\"https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-train\" target=\"_blank\">train</a>, <a href=\"https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-infer\" target=\"_blank\">infer</a>)</li>\n<li>BERT variants for target<ul>\n<li>Electra-large + Attentions + Conv1D (<a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-attentions-conv1d-train\" target=\"_blank\">train</a>, <a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-att-conv1d-infer\" target=\"_blank\">infer</a>)</li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-roberta-large-att-mask-act-infer\" target=\"_blank\">Roberta-large + Attentions</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-roberta-base-oof-task-fineturning-infer\" target=\"_blank\">Roberta-base + Attention</a></li>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-funnel-large-attentions-act\" target=\"_blank\">Funnel-large + Attention</a></li></ul></li>\n<li>LGBM for standard_error &amp; target<ul>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-feature-eda?scriptVersionId=69709698\" target=\"_blank\">CLRP: Standard Error Feature EDA</a></li>\n<li>This notebook is not EDA, but I have named……</li></ul></li></ul></li>\n<li>Model Stacking<ul>\n<li><a href=\"https://www.kaggle.com/atsushiiwasaki/clrp-model-stacking-with-standard-error\" target=\"_blank\">CLRP: Model Stacking with Standard Error</a></li></ul></li>\n</ul>",
      "rawMarkdown": "# Thank you all !\nAt first, I would like to thank competition hosts, and Kaggle community.\nI enjoyed, learned so many things, and was encouraged by you all.\nEspecially, I want to give great thanks to  @andretugan, @maunish, and @rhtsingh. Their notebooks and discussions inspired and led me a lot.\n\nThis is my first NLP competition. I'm very surprised to win silver medal, and so glad.\n\n<br>\n# Key Points\nIn this solution, there are 3 important points.\n* Standard Error as a Feature\n* Model Diversity (BERT variants, custom head, LGBM......)\n* Stacking Strategy (weighted ensemble)\n\nI will explain them one by one.\nI did not use MLM pretraining nor external datasets. Just train dataset only.\n\n<br>\n### Standard Error as a Feature (**MOST IMPORTANT**)\nI noticed the standard_error is a key feature 3 days before the deadline.\nThe experiment notebook is here.\nhttps://www.kaggle.com/atsushiiwasaki/clrp-standard-error-as-a-feature\n\n**Main Result**\n| Stacking Layer                | CV Score | Boost   | \n| ----------------------------- | -------- | ------- | \n| Linear (baseline)             | 0.4567   | -       | \n| LightGBM (without standard_error) | 0.4596   | -0.0029 | \n| LightGBM (with standard_error)    | 0.3959   | 0.069   | \n\nThis experiment shows that standard_error is a key feature comparable to BERTs prediction.\nThe problem is how to get the standard_error, since it is not given in test set.\n\nI deal with this problem by using LightGBM for standard_error prediction.\nHere, feature engineering appeared.\n\n<br>\n### Model Diversity\nLB was very unstable for a single model prediction.  So, I decided to cover the instability with \n model diversity.\nI tried to train many diverse models.\n* architecture (Roberta, Electra, XLNet, Funnel, Albert, DeBerta, BART......LGBM)\n* size (base or large)\n* head (attention, conv1D, meanpooling and so on)\n\nHowever, since I am not good at fine-turning or something unknown, only 8 models survived.\n8 models are shown below.\n| Architecture | Size  | Head                             | CV |\n| ------------ | ----- | -------------------------------- | ---------|\n| Roberta      | large | Attentions (using last 4 layers) | 0.4757 |\n| Roberta      | large | Attentions-Conv1D                | 0.4761 |\n| Roberta      | base  | Attention                        | 0.4746  |\n| Electra      | large | Attentions (using last 4 layers) | 0.4760 |\n| XLNet        | large | Attentions (using last 4 layers) | 0.4850 |\n| Electra      | large | Attentions-Conv1D                | 0.4818 |\n| Roberta      | large | Meanpooling                      | 0.4772 |\n| Funnel       | large | Attention                        | 0.4836 |\n\n<br>\n### Stacking Strategy\nHere is the stacking scheme figure.\nIt is very complicated, especially standard_error prediction part.\n\nIf not using standard_error, stacking scheme would be much simpler.\n![stacking scheme](https://i.postimg.cc/prVYT4mh/CLRP-Stacking-Scheme.png)\n<br>\n# Results\nMy models overfitted somewhat.\n\n| Strategy                           | CV    | Public LB | Private LB | \n| ---------------------------------- | ----- | --------- | ---------- | \n| Stacking BERTs                     | 0.457 | 0.458     | 0.461      | \n| LGBM using BERTs Prediction + S.E. | 0.454 | 0.458     | 0.459      | \n| Stacking BERTs + LGBM              | 0.452 | 0.456     | 0.458      | \n\n<br>\n# What did not Work\n- MLM pretraining\n- Data augmentation (like [Easy Data Augmentation](https://arxiv.org/abs/1901.11196))\n - I set `data_collator = DataCollatorForLanguageModeling`, then dynamically masked texts. I expect this trick works, but not.\n - As a result, I just set `data_collator = DataCollatorWithPadding`. Dynamically padding is realized.\n- Back translation\n- Layer reinitialization\n- ALBERT, DeBerta, BART (It is too difficult to finetune them......)\n- SmartBatchingSampler\n - The data order fixed to a extent would not be good in the small dataset competition.\n- [Multi-head Attention](https://www.kaggle.com/atsushiiwasaki/clrp-attention-head-variants)\n - I tried to divide the simple attention head into some blocks. This does not work.\n<br>\n# Other Ideas\n- Stratify the dataset on prediction error\n - Stratified K Fold on target, s.e. and target & s.e. seems not to work enough. So, I decided to stratify the dataset on prediction error. There may be some important feature distributions to stratify.\n - [CLRP: Stratify on Target & S.E. Bins from Sturge's](https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-target-s-e-bins-from-sturge-s)\n - [CLRP: Stratify On Predictability](https://www.kaggle.com/atsushiiwasaki/clrp-stratify-on-predictability)\n - [Split based on Target & Standard Error\n](https://www.kaggle.com/atsushiiwasaki/split-based-on-target-standard-error)\n - However, this strategy would not work so well.\n- Larger depth in attention head in Funnel-large\n - In Roberta, Electra and XLNet, the depth of attention head is set 512~514 (I choose the depth based on the embedding size).\n - On the other hand, it is set 768 in Funnel with only last hidden layer, and this worked.\n - Maybe, this is due to Funnel architecture. We can extract the best encoded representation from last hidden layer in Funnel encoder using deep attention head having more representation ability.\n\n<br>\n# Ideas that I did not come up with\n- [Psuedo Labeling](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257844)\n- [Gaussian Process Regression](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258554)\n- [Sentence Transformers](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260800)\n- [Bradley-Terry (BT) Loss](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258312)\n- [Binary Pre-training](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260939)\n\n<br>\n# Notebooks Related to This Solution\n- Data\n - [Dataset Size Dependency](https://www.kaggle.com/atsushiiwasaki/how-many-samples-do-i-want)\n- Model Training\n - BERT Baseline ([train](https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-train), [infer](https://www.kaggle.com/atsushiiwasaki/commonlit-bert-stratified-k-fold-baseline-infer))\n - BERT variants for target\n      - Electra-large + Attentions + Conv1D ([train](https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-attentions-conv1d-train), [infer](https://www.kaggle.com/atsushiiwasaki/clrp-electra-large-att-conv1d-infer))\n      - [Roberta-large + Attentions](https://www.kaggle.com/atsushiiwasaki/clrp-roberta-large-att-mask-act-infer)\n      - [Roberta-base + Attention](https://www.kaggle.com/atsushiiwasaki/clrp-roberta-base-oof-task-fineturning-infer)\n      - [Funnel-large + Attention](https://www.kaggle.com/atsushiiwasaki/clrp-funnel-large-attentions-act)\n - LGBM for standard_error & target\n     - [CLRP: Standard Error Feature EDA](https://www.kaggle.com/atsushiiwasaki/clrp-standard-error-feature-eda?scriptVersionId=69709698)\n     - This notebook is not EDA, but I have named......\n- Model Stacking\n - [CLRP: Model Stacking with Standard Error](https://www.kaggle.com/atsushiiwasaki/clrp-model-stacking-with-standard-error)",
      "votes": 3
    },
    {
      "id": 1430397,
      "postDate": "2021-08-03T11:36:34.360Z",
      "content": "<p>If standard error is a feature, what did you use on the test set?</p>",
      "rawMarkdown": "If standard error is a feature, what did you use on the test set?",
      "replies": [
        {
          "id": 1430608,
          "postDate": "2021-08-03T11:49:24.397Z",
          "content": "<p><a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> Thank you for your comment. In addition to target prediction models (BERT variants), I created standard_error prediction model (LightGBM). The features for the model were engineered from the excerpts (for example,  number of sentences) and BERT variants predictions.</p>",
          "rawMarkdown": "@nbroad Thank you for your comment. In addition to target prediction models (BERT variants), I created standard_error prediction model (LightGBM). The features for the model were engineered from the excerpts (for example,  number of sentences) and BERT variants predictions.",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:45:41Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6812306,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/atsushi-iwasaki-114th-place-solution-standard-erro"
}