{
  "id": 260325,
  "title": "12th (public 19th) place solution",
  "content": "<p>First of all, Thanks Kaggle and the host for organizing this interesting competition.<br>\nI also want to thank the kagglers who made this competition so much fun for me. Thank you so much!</p>\n<p>my final inference notebook: <a href=\"https://www.kaggle.com/kentaronakanishi/clrp-095-ensemble13-cv-kaggle93\" target=\"_blank\">https://www.kaggle.com/kentaronakanishi/clrp-095-ensemble13-cv-kaggle93</a></p>\n<h1>Summary</h1>\n<p>Final submissions are 13 and 11 models ensemble. The weights of ensembles are calculated by Nelder-mead with an objective using CV and LB.</p>\n<table>\n<thead>\n<tr>\n<th>Name</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ensemble 13</td>\n<td>0.4442</td>\n<td>0.450</td>\n<td>0.449</td>\n</tr>\n<tr>\n<td>ensemble 11</td>\n<td>0.4451</td>\n<td>0.449</td>\n<td>0.449</td>\n</tr>\n</tbody>\n</table>\n<p>I pre-trained huggingface base models by wikipedia (simple/normal) data with MLM and pseudo target, and finetuned them by competition data.<br>\nI didn't use other external data because I can't judge it is allowed to use scraped data.</p>\n<p>Works with big improvements are re-initialization, pre-training with pseudo target, and ensemble.</p>\n<h1>Models</h1>\n<p>Base models for 13 models ensemble are nine <code>microsoft/deberta-large</code>, two <code>albert-xxlarge-v2</code>, and one for <code>google/electra-large-discriminator</code> and <code>roberta-large</code>.</p>\n<table>\n<thead>\n<tr>\n<th>Name</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.474</td>\n<td>0.464</td>\n<td>0.468</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.467</td>\n<td>0.462</td>\n<td>0.467</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> attn head</td>\n<td>0.467</td>\n<td>0.461</td>\n<td>0.463</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.465</td>\n<td>0.457</td>\n<td>0.457</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.463</td>\n<td>0.456</td>\n<td>0.456</td>\n</tr>\n<tr>\n<td><code>roberta-large</code> multi custom head</td>\n<td>0.475</td>\n<td>0.465</td>\n<td>0.456</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.462</td>\n<td>0.456</td>\n<td>0.460</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.463</td>\n<td>0.455</td>\n<td>0.459</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.463</td>\n<td>0.455</td>\n<td>0.460</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> with small (not pre-trained) network</td>\n<td>0.467</td>\n<td>0.458</td>\n<td>0.458</td>\n</tr>\n<tr>\n<td><code>alberta-xxlarge-v2</code> cls head</td>\n<td>0.475</td>\n<td>0.461</td>\n<td>0.460</td>\n</tr>\n<tr>\n<td><code>alberta-xxlarge-v2</code> multi custom head</td>\n<td>0.467</td>\n<td>0.463</td>\n<td>0.454</td>\n</tr>\n<tr>\n<td><code>electra-large</code> multi custom head</td>\n<td>0.471</td>\n<td>0.464</td>\n<td>0.461</td>\n</tr>\n</tbody>\n</table>\n<p>Best public LB by single model is 0.453, but this is not used for ensemble.</p>\n<p>custom head is a combination of cls, avg pool, max pool, attn pool. Each head has output layer to target value, and then average (or weighted average) them.</p>\n<h1>Training</h1>\n<h2>Basic settings</h2>\n<pre><code>num_folds: 5\nnum_epochs: 6\nlr_scheduler_epochs: 10\nlearning_rate: 0.00002\nlearning_rate_output: 0.0001\nbatch_size: 16\nmax_length: 256\nvalidation_interval: 10\n</code></pre>\n<h2>Pre-training</h2>\n<p>I have pre-trained again for huggingface pre-trained models with wikipedia data. I used data by <a href=\"https://www.kaggle.com/markwijkhuizen\" target=\"_blank\">@markwijkhuizen</a> . Thanks a lot!</p>\n<p>Not only MLM, I use pseudo labeling by best ensemble model on that time for wikipedia data and train with it. The details is below:</p>\n<ol>\n<li>making good models and ensemble</li>\n<li>predicting target for wikipedia data (simple/normal) by best ensemble models. This is predicted by each fold to prevent target leakage.</li>\n<li>filtering wikipedia data by<ul>\n<li>has excerpt in both of simple and normal</li>\n<li>the pseudo-target by pseudo labeling looks correct: satisfying <code>simple - margin &gt; normal</code> where margin = 0.5</li></ul></li>\n<li>pre-training by wikipedia data with tasks: MLM and predict pseudo target.</li>\n</ol>\n<p>This pre-trained and fine-tuning framework improved scores a lot in public LB.</p>\n<h2>Others</h2>\n<ul>\n<li>re-initialization top N layers of pre-trained model. N=5<ul>\n<li>worked a lot</li>\n<li><a href=\"https://arxiv.org/abs/2006.05987\" target=\"_blank\">https://arxiv.org/abs/2006.05987</a></li></ul></li>\n<li>layerwise LR</li>\n<li>RDrop<ul>\n<li>maybe work a little</li>\n<li><a href=\"https://arxiv.org/abs/2106.14448\" target=\"_blank\">https://arxiv.org/abs/2106.14448</a></li></ul></li>\n<li>dropout output layer 0.2 ~ 0.3</li>\n<li>LAMB optimizer make training stable (but no improvement on score)</li>\n</ul>\n<h1>Not worked for me</h1>\n<ul>\n<li>classification approach<ul>\n<li>cross entropy</li>\n<li>rmse with softmax weighted sum by center value</li>\n<li>histgram loss<ul>\n<li>same or little worse performance as regression approach</li>\n<li><a href=\"https://arxiv.org/abs/1806.04613\" target=\"_blank\">https://arxiv.org/abs/1806.04613</a></li></ul></li></ul></li>\n<li>pairwise comparison and simulate bradley terry approach<ul>\n<li>make pairs by combination of all excerpts and predict which is easier to read.</li>\n<li>pairwise comparison is harder than regression for me</li></ul></li>\n<li>linguistic feature<ul>\n<li>little or not worked with output layer or with features of svm</li></ul></li>\n<li>hidden_dropout = 0.0<ul>\n<li>CV improves but LB didn't change or even worse</li></ul></li>\n<li>GaussianLoss, MAE, Huber, RankLoss have no improvements</li>\n<li>Target balanced loss (the bigger abs target, the bigger loss) not improvement</li>\n<li>pseudo labeling noisy training data<ul>\n<li>CV off-course improved but LB not.</li></ul></li>\n</ul>",
  "comments": [
    {
      "id": 1445348,
      "author_name": "Shibata",
      "author_url": "",
      "post_date": "2021-08-04T08:35:27.380000",
      "content": "<p><a href=\"https://www.kaggle.com/kentaronakanishi\" target=\"_blank\">@kentaronakanishi</a> <br>\nThank you for sharing this great solution🙏 I was very surprised to see you go up smoothly from silver zone in the last 2 weeks and finally got a gold medal🎉</p>\n<p>I have a question. </p>\n<blockquote>\n  <p>pre-training by wikipedia data with tasks: MLM and predict pseudo target.</p>\n</blockquote>\n<p>Does this mean you did following steps ?<br>\n①MLM on Pseudo labels　→ ② Normal training using only pseudo labels → ③Fine tuning using only competition data</p>\n<p>If so, please let me know the training setting of ② (learning rate, how many epochs)</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1445800,
          "author_name": "cfiken",
          "author_url": "",
          "post_date": "2021-08-04T10:09:07.687000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/yutoshibata\" target=\"_blank\">@yutoshibata</a> 😆 Maybe I'm slow starter ;)</p>\n<p>Training flow is:</p>\n<ol>\n<li>prepare wikipedia data with pseudo label</li>\n<li>pre-training by MLM and predict pseudo label at the same time, in order to grasp difference simple and normal wikipedia<ul>\n<li>This pre-training contains competition data only for MLM (not using target). I've omitted.</li></ul></li>\n<li>re-initialization some of language model layers and output layers</li>\n<li>finetuning on competition data target</li>\n</ol>\n<p>On pre-training, the settings are:</p>\n<pre><code>learning_rate: 5e-5  # no layer-wise lr\nbatch_size: 32\nnum_epochs: 2\n</code></pre>\n<p>It took so much time on pre-training (about 1~2 days on V100) since I've done this on 5 folds.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1442492,
      "author_name": "Sinan Calisir",
      "author_url": "",
      "post_date": "2021-08-03T23:03:27.823000",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/kentaronakanishi\" target=\"_blank\">@kentaronakanishi</a> </p>\n<p>Thank you for sharing the details of your solution! For the re-initialization part, do you completely initiate the weights of the layers (I guess the N parameter means last N - 5 in your case) with random numbers something like torch.randn()?</p>\n<p>Also, congrats on becoming Competitions Master!! 🎉🎉</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1443038,
          "author_name": "cfiken",
          "author_url": "",
          "post_date": "2021-08-04T01:30:42.820000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/snnclsr\" target=\"_blank\">@snnclsr</a> </p>\n<p>Yes, re-initialization N layers of pre-trained language models. I used <code>initializer_range</code> of huggingface original config.<br>\nHere is a sample code:</p>\n<pre><code>class Model(nn.Module):\n    ...\n    def initialize(self):\n        for i in range(self.num_reinit_layers):\n            self.model.encoder.layer[-(1 + i)].apply(self._init_weight)\n\n    def _init_weight(self, module):\n        if isinstance(module, nn.Linear):\n            module.weight.data.normal_(mean=0.0, std=self.model.config.initializer_range)\n            if module.bias is not None:\n                module.bias.data.zero_()\n</code></pre>",
          "votes": 3,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 1437216,
      "postDate": "2021-08-03T15:28:39.990Z",
      "content": "<p>First of all, Thanks Kaggle and the host for organizing this interesting competition.<br>\nI also want to thank the kagglers who made this competition so much fun for me. Thank you so much!</p>\n<p>my final inference notebook: <a href=\"https://www.kaggle.com/kentaronakanishi/clrp-095-ensemble13-cv-kaggle93\" target=\"_blank\">https://www.kaggle.com/kentaronakanishi/clrp-095-ensemble13-cv-kaggle93</a></p>\n<h1>Summary</h1>\n<p>Final submissions are 13 and 11 models ensemble. The weights of ensembles are calculated by Nelder-mead with an objective using CV and LB.</p>\n<table>\n<thead>\n<tr>\n<th>Name</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ensemble 13</td>\n<td>0.4442</td>\n<td>0.450</td>\n<td>0.449</td>\n</tr>\n<tr>\n<td>ensemble 11</td>\n<td>0.4451</td>\n<td>0.449</td>\n<td>0.449</td>\n</tr>\n</tbody>\n</table>\n<p>I pre-trained huggingface base models by wikipedia (simple/normal) data with MLM and pseudo target, and finetuned them by competition data.<br>\nI didn't use other external data because I can't judge it is allowed to use scraped data.</p>\n<p>Works with big improvements are re-initialization, pre-training with pseudo target, and ensemble.</p>\n<h1>Models</h1>\n<p>Base models for 13 models ensemble are nine <code>microsoft/deberta-large</code>, two <code>albert-xxlarge-v2</code>, and one for <code>google/electra-large-discriminator</code> and <code>roberta-large</code>.</p>\n<table>\n<thead>\n<tr>\n<th>Name</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.474</td>\n<td>0.464</td>\n<td>0.468</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.467</td>\n<td>0.462</td>\n<td>0.467</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> attn head</td>\n<td>0.467</td>\n<td>0.461</td>\n<td>0.463</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.465</td>\n<td>0.457</td>\n<td>0.457</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.463</td>\n<td>0.456</td>\n<td>0.456</td>\n</tr>\n<tr>\n<td><code>roberta-large</code> multi custom head</td>\n<td>0.475</td>\n<td>0.465</td>\n<td>0.456</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.462</td>\n<td>0.456</td>\n<td>0.460</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.463</td>\n<td>0.455</td>\n<td>0.459</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> multi custom head</td>\n<td>0.463</td>\n<td>0.455</td>\n<td>0.460</td>\n</tr>\n<tr>\n<td><code>deberta-large</code> with small (not pre-trained) network</td>\n<td>0.467</td>\n<td>0.458</td>\n<td>0.458</td>\n</tr>\n<tr>\n<td><code>alberta-xxlarge-v2</code> cls head</td>\n<td>0.475</td>\n<td>0.461</td>\n<td>0.460</td>\n</tr>\n<tr>\n<td><code>alberta-xxlarge-v2</code> multi custom head</td>\n<td>0.467</td>\n<td>0.463</td>\n<td>0.454</td>\n</tr>\n<tr>\n<td><code>electra-large</code> multi custom head</td>\n<td>0.471</td>\n<td>0.464</td>\n<td>0.461</td>\n</tr>\n</tbody>\n</table>\n<p>Best public LB by single model is 0.453, but this is not used for ensemble.</p>\n<p>custom head is a combination of cls, avg pool, max pool, attn pool. Each head has output layer to target value, and then average (or weighted average) them.</p>\n<h1>Training</h1>\n<h2>Basic settings</h2>\n<pre><code>num_folds: 5\nnum_epochs: 6\nlr_scheduler_epochs: 10\nlearning_rate: 0.00002\nlearning_rate_output: 0.0001\nbatch_size: 16\nmax_length: 256\nvalidation_interval: 10\n</code></pre>\n<h2>Pre-training</h2>\n<p>I have pre-trained again for huggingface pre-trained models with wikipedia data. I used data by <a href=\"https://www.kaggle.com/markwijkhuizen\" target=\"_blank\">@markwijkhuizen</a> . Thanks a lot!</p>\n<p>Not only MLM, I use pseudo labeling by best ensemble model on that time for wikipedia data and train with it. The details is below:</p>\n<ol>\n<li>making good models and ensemble</li>\n<li>predicting target for wikipedia data (simple/normal) by best ensemble models. This is predicted by each fold to prevent target leakage.</li>\n<li>filtering wikipedia data by<ul>\n<li>has excerpt in both of simple and normal</li>\n<li>the pseudo-target by pseudo labeling looks correct: satisfying <code>simple - margin &gt; normal</code> where margin = 0.5</li></ul></li>\n<li>pre-training by wikipedia data with tasks: MLM and predict pseudo target.</li>\n</ol>\n<p>This pre-trained and fine-tuning framework improved scores a lot in public LB.</p>\n<h2>Others</h2>\n<ul>\n<li>re-initialization top N layers of pre-trained model. N=5<ul>\n<li>worked a lot</li>\n<li><a href=\"https://arxiv.org/abs/2006.05987\" target=\"_blank\">https://arxiv.org/abs/2006.05987</a></li></ul></li>\n<li>layerwise LR</li>\n<li>RDrop<ul>\n<li>maybe work a little</li>\n<li><a href=\"https://arxiv.org/abs/2106.14448\" target=\"_blank\">https://arxiv.org/abs/2106.14448</a></li></ul></li>\n<li>dropout output layer 0.2 ~ 0.3</li>\n<li>LAMB optimizer make training stable (but no improvement on score)</li>\n</ul>\n<h1>Not worked for me</h1>\n<ul>\n<li>classification approach<ul>\n<li>cross entropy</li>\n<li>rmse with softmax weighted sum by center value</li>\n<li>histgram loss<ul>\n<li>same or little worse performance as regression approach</li>\n<li><a href=\"https://arxiv.org/abs/1806.04613\" target=\"_blank\">https://arxiv.org/abs/1806.04613</a></li></ul></li></ul></li>\n<li>pairwise comparison and simulate bradley terry approach<ul>\n<li>make pairs by combination of all excerpts and predict which is easier to read.</li>\n<li>pairwise comparison is harder than regression for me</li></ul></li>\n<li>linguistic feature<ul>\n<li>little or not worked with output layer or with features of svm</li></ul></li>\n<li>hidden_dropout = 0.0<ul>\n<li>CV improves but LB didn't change or even worse</li></ul></li>\n<li>GaussianLoss, MAE, Huber, RankLoss have no improvements</li>\n<li>Target balanced loss (the bigger abs target, the bigger loss) not improvement</li>\n<li>pseudo labeling noisy training data<ul>\n<li>CV off-course improved but LB not.</li></ul></li>\n</ul>",
      "rawMarkdown": "First of all, Thanks Kaggle and the host for organizing this interesting competition.\nI also want to thank the kagglers who made this competition so much fun for me. Thank you so much!\n\nmy final inference notebook: https://www.kaggle.com/kentaronakanishi/clrp-095-ensemble13-cv-kaggle93\n\n# Summary\n\nFinal submissions are 13 and 11 models ensemble. The weights of ensembles are calculated by Nelder-mead with an objective using CV and LB.\n \n| Name | CV | Public LB | Private LB |\n| --- | --- | --- | --- |\n| ensemble 13 | 0.4442 | 0.450 | 0.449 |\n| ensemble 11 | 0.4451 | 0.449 | 0.449 |\n\n\nI pre-trained huggingface base models by wikipedia (simple/normal) data with MLM and pseudo target, and finetuned them by competition data.\nI didn't use other external data because I can't judge it is allowed to use scraped data.\n\nWorks with big improvements are re-initialization, pre-training with pseudo target, and ensemble.\n\n# Models\n\nBase models for 13 models ensemble are nine `microsoft/deberta-large`, two `albert-xxlarge-v2`, and one for `google/electra-large-discriminator` and `roberta-large`.\n\n| Name | CV | Public LB | Private LB |\n| --- | --- | --- | --- |\n| `deberta-large` multi custom head | 0.474 | 0.464 | 0.468 |\n| `deberta-large` multi custom head | 0.467 | 0.462 | 0.467 |\n| `deberta-large` attn head | 0.467 | 0.461 | 0.463 |\n| `deberta-large` multi custom head | 0.465 | 0.457 | 0.457 |\n| `deberta-large` multi custom head | 0.463 | 0.456 | 0.456 |\n| `roberta-large` multi custom head | 0.475 | 0.465 | 0.456 |\n| `deberta-large` multi custom head | 0.462 | 0.456 | 0.460 |\n| `deberta-large` multi custom head | 0.463 | 0.455 | 0.459 |\n| `deberta-large` multi custom head | 0.463 | 0.455 | 0.460 |\n| `deberta-large` with small (not pre-trained) network | 0.467 | 0.458 | 0.458 |\n| `alberta-xxlarge-v2` cls head | 0.475 | 0.461 | 0.460 |\n| `alberta-xxlarge-v2` multi custom head | 0.467 | 0.463 | 0.454 |\n| `electra-large` multi custom head | 0.471 | 0.464 | 0.461 |\n\n\nBest public LB by single model is 0.453, but this is not used for ensemble.\n\ncustom head is a combination of cls, avg pool, max pool, attn pool. Each head has output layer to target value, and then average (or weighted average) them.\n\n# Training\n\n## Basic settings\n\n```yaml\nnum_folds: 5\nnum_epochs: 6\nlr_scheduler_epochs: 10\nlearning_rate: 0.00002\nlearning_rate_output: 0.0001\nbatch_size: 16\nmax_length: 256\nvalidation_interval: 10\n```\n\n## Pre-training\n\nI have pre-trained again for huggingface pre-trained models with wikipedia data. I used data by @markwijkhuizen . Thanks a lot!\n\nNot only MLM, I use pseudo labeling by best ensemble model on that time for wikipedia data and train with it. The details is below:\n\n1. making good models and ensemble\n2. predicting target for wikipedia data (simple/normal) by best ensemble models. This is predicted by each fold to prevent target leakage.\n3. filtering wikipedia data by\n    - has excerpt in both of simple and normal\n    - the pseudo-target by pseudo labeling looks correct: satisfying `simple - margin > normal` where margin = 0.5\n4. pre-training by wikipedia data with tasks: MLM and predict pseudo target.\n\nThis pre-trained and fine-tuning framework improved scores a lot in public LB.\n\n## Others\n\n- re-initialization top N layers of pre-trained model. N=5\n    - worked a lot\n    - [https://arxiv.org/abs/2006.05987](https://arxiv.org/abs/2006.05987)\n- layerwise LR\n- RDrop\n    - maybe work a little\n    - [https://arxiv.org/abs/2106.14448](https://arxiv.org/abs/2106.14448)\n- dropout output layer 0.2 ~ 0.3\n- LAMB optimizer make training stable (but no improvement on score)\n\n# Not worked for me\n\n- classification approach\n    - cross entropy\n    - rmse with softmax weighted sum by center value\n    - histgram loss\n        - same or little worse performance as regression approach\n        - [https://arxiv.org/abs/1806.04613](https://arxiv.org/abs/1806.04613)\n- pairwise comparison and simulate bradley terry approach\n    - make pairs by combination of all excerpts and predict which is easier to read.\n    - pairwise comparison is harder than regression for me\n- linguistic feature\n    - little or not worked with output layer or with features of svm\n- hidden_dropout = 0.0\n    - CV improves but LB didn't change or even worse\n- GaussianLoss, MAE, Huber, RankLoss have no improvements\n- Target balanced loss (the bigger abs target, the bigger loss) not improvement\n- pseudo labeling noisy training data\n    - CV off-course improved but LB not.\n",
      "votes": 28
    },
    {
      "id": 1445348,
      "postDate": "2021-08-04T08:35:27.380Z",
      "content": "<p><a href=\"https://www.kaggle.com/kentaronakanishi\" target=\"_blank\">@kentaronakanishi</a> <br>\nThank you for sharing this great solution🙏 I was very surprised to see you go up smoothly from silver zone in the last 2 weeks and finally got a gold medal🎉</p>\n<p>I have a question. </p>\n<blockquote>\n  <p>pre-training by wikipedia data with tasks: MLM and predict pseudo target.</p>\n</blockquote>\n<p>Does this mean you did following steps ?<br>\n①MLM on Pseudo labels　→ ② Normal training using only pseudo labels → ③Fine tuning using only competition data</p>\n<p>If so, please let me know the training setting of ② (learning rate, how many epochs)</p>",
      "rawMarkdown": "@kentaronakanishi \nThank you for sharing this great solution🙏 I was very surprised to see you go up smoothly from silver zone in the last 2 weeks and finally got a gold medal🎉\n\nI have a question. \n> pre-training by wikipedia data with tasks: MLM and predict pseudo target.\n\nDoes this mean you did following steps ?\n①MLM on Pseudo labels　→ ② Normal training using only pseudo labels → ③Fine tuning using only competition data\n\nIf so, please let me know the training setting of ② (learning rate, how many epochs)\n",
      "votes": 1,
      "replies": [
        {
          "id": 1445800,
          "postDate": "2021-08-04T10:09:07.687Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/yutoshibata\" target=\"_blank\">@yutoshibata</a> 😆 Maybe I'm slow starter ;)</p>\n<p>Training flow is:</p>\n<ol>\n<li>prepare wikipedia data with pseudo label</li>\n<li>pre-training by MLM and predict pseudo label at the same time, in order to grasp difference simple and normal wikipedia<ul>\n<li>This pre-training contains competition data only for MLM (not using target). I've omitted.</li></ul></li>\n<li>re-initialization some of language model layers and output layers</li>\n<li>finetuning on competition data target</li>\n</ol>\n<p>On pre-training, the settings are:</p>\n<pre><code>learning_rate: 5e-5  # no layer-wise lr\nbatch_size: 32\nnum_epochs: 2\n</code></pre>\n<p>It took so much time on pre-training (about 1~2 days on V100) since I've done this on 5 folds.</p>",
          "rawMarkdown": "Thanks @yutoshibata 😆 Maybe I'm slow starter ;)\n\nTraining flow is:\n\n0. prepare wikipedia data with pseudo label\n1. pre-training by MLM and predict pseudo label at the same time, in order to grasp difference simple and normal wikipedia\n    - This pre-training contains competition data only for MLM (not using target). I've omitted.\n2. re-initialization some of language model layers and output layers\n3. finetuning on competition data target\n\nOn pre-training, the settings are:\n\n```yaml\nlearning_rate: 5e-5  # no layer-wise lr\nbatch_size: 32\nnum_epochs: 2\n```\n\nIt took so much time on pre-training (about 1~2 days on V100) since I've done this on 5 folds."
        }
      ]
    },
    {
      "id": 1442492,
      "postDate": "2021-08-03T23:03:27.823Z",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/kentaronakanishi\" target=\"_blank\">@kentaronakanishi</a> </p>\n<p>Thank you for sharing the details of your solution! For the re-initialization part, do you completely initiate the weights of the layers (I guess the N parameter means last N - 5 in your case) with random numbers something like torch.randn()?</p>\n<p>Also, congrats on becoming Competitions Master!! 🎉🎉</p>",
      "rawMarkdown": "Hi @kentaronakanishi \n\nThank you for sharing the details of your solution! For the re-initialization part, do you completely initiate the weights of the layers (I guess the N parameter means last N - 5 in your case) with random numbers something like torch.randn()?\n\nAlso, congrats on becoming Competitions Master!! 🎉🎉",
      "votes": 2,
      "replies": [
        {
          "id": 1443038,
          "postDate": "2021-08-04T01:30:42.820Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/snnclsr\" target=\"_blank\">@snnclsr</a> </p>\n<p>Yes, re-initialization N layers of pre-trained language models. I used <code>initializer_range</code> of huggingface original config.<br>\nHere is a sample code:</p>\n<pre><code>class Model(nn.Module):\n    ...\n    def initialize(self):\n        for i in range(self.num_reinit_layers):\n            self.model.encoder.layer[-(1 + i)].apply(self._init_weight)\n\n    def _init_weight(self, module):\n        if isinstance(module, nn.Linear):\n            module.weight.data.normal_(mean=0.0, std=self.model.config.initializer_range)\n            if module.bias is not None:\n                module.bias.data.zero_()\n</code></pre>",
          "rawMarkdown": "Thanks @snnclsr \n\nYes, re-initialization N layers of pre-trained language models. I used `initializer_range` of huggingface original config.\nHere is a sample code:\n\n```\nclass Model(nn.Module):\n    ...\n    def initialize(self):\n        for i in range(self.num_reinit_layers):\n            self.model.encoder.layer[-(1 + i)].apply(self._init_weight)\n\n    def _init_weight(self, module):\n        if isinstance(module, nn.Linear):\n            module.weight.data.normal_(mean=0.0, std=self.model.config.initializer_range)\n            if module.bias is not None:\n                module.bias.data.zero_()\n```",
          "votes": 3
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:48:58Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6736659,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/cfiken-12th-public-19th-place-solution"
}