{
  "id": 257816,
  "title": "100th place : Ensemble public notebooks",
  "content": "<p>Since every attempts to pre-train Roberta failed for me, I used publicly pre-trained robertas from <a href=\"https://www.kaggle.com/maunish\" target=\"_blank\">@maunish</a> and <a href=\"https://www.kaggle.com/tutty1992\" target=\"_blank\">@tutty1992</a>. I ensembled roberta-base and roberta-large with different strategy, and here are the methods.</p>\n<h2>1.  Roberta-Base + SVR</h2>\n<p>Fine-tuning <a href=\"https://www.kaggle.com/maunish\" target=\"_blank\">@maunish</a> 's model and predicting with linear layer, I could get </p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.465</td>\n<td>0.466</td>\n</tr>\n</tbody>\n</table>\n<p>Then I tried to predict with SVR 5-folds wrt training data without using the linear layer. After fine-tuning the regularization parameters for each fold, I could get a slightly better score,</p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.464</td>\n<td>0.464</td>\n</tr>\n</tbody>\n</table>\n<h2>2.  Roberta-Large with different poolers and seeds</h2>\n<p>Fine-tuning <a href=\"https://www.kaggle.com/tutty1992\" target=\"_blank\">@tutty1992</a> 's model and referring to <a href=\"https://www.kaggle.com/jcesquiveld\" target=\"_blank\">@jcesquiveld</a> , I tried to ensemble Roberta-Large models with different poolers with different seeds. Although each model is around 0.468 on public/private leaderboard, ensembling 6 different models (2 different poolers and 3 different seeds) led to higher performances as I thought,</p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.463</td>\n<td>0.463</td>\n</tr>\n</tbody>\n</table>\n<p>Besides, I've also ensembled public notebooks and datasets from <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> , <a href=\"https://www.kaggle.com/chamecall\" target=\"_blank\">@chamecall</a> , <a href=\"https://www.kaggle.com/rajat95gupta\" target=\"_blank\">@rajat95gupta</a> and <a href=\"https://www.kaggle.com/nomlmean\" target=\"_blank\">@nomlmean</a> to get a score</p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.457</td>\n<td>0.457</td>\n</tr>\n</tbody>\n</table>\n<p>The codes will be made pubic after reference checkup. Hard work everyone, and thank you for everybody who shared great insights and their excellent works!!</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 1423625,
      "postDate": "2021-08-03T05:49:49.233Z",
      "content": "<p>Since every attempts to pre-train Roberta failed for me, I used publicly pre-trained robertas from <a href=\"https://www.kaggle.com/maunish\" target=\"_blank\">@maunish</a> and <a href=\"https://www.kaggle.com/tutty1992\" target=\"_blank\">@tutty1992</a>. I ensembled roberta-base and roberta-large with different strategy, and here are the methods.</p>\n<h2>1.  Roberta-Base + SVR</h2>\n<p>Fine-tuning <a href=\"https://www.kaggle.com/maunish\" target=\"_blank\">@maunish</a> 's model and predicting with linear layer, I could get </p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.465</td>\n<td>0.466</td>\n</tr>\n</tbody>\n</table>\n<p>Then I tried to predict with SVR 5-folds wrt training data without using the linear layer. After fine-tuning the regularization parameters for each fold, I could get a slightly better score,</p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.464</td>\n<td>0.464</td>\n</tr>\n</tbody>\n</table>\n<h2>2.  Roberta-Large with different poolers and seeds</h2>\n<p>Fine-tuning <a href=\"https://www.kaggle.com/tutty1992\" target=\"_blank\">@tutty1992</a> 's model and referring to <a href=\"https://www.kaggle.com/jcesquiveld\" target=\"_blank\">@jcesquiveld</a> , I tried to ensemble Roberta-Large models with different poolers with different seeds. Although each model is around 0.468 on public/private leaderboard, ensembling 6 different models (2 different poolers and 3 different seeds) led to higher performances as I thought,</p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.463</td>\n<td>0.463</td>\n</tr>\n</tbody>\n</table>\n<p>Besides, I've also ensembled public notebooks and datasets from <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> , <a href=\"https://www.kaggle.com/chamecall\" target=\"_blank\">@chamecall</a> , <a href=\"https://www.kaggle.com/rajat95gupta\" target=\"_blank\">@rajat95gupta</a> and <a href=\"https://www.kaggle.com/nomlmean\" target=\"_blank\">@nomlmean</a> to get a score</p>\n<table>\n<thead>\n<tr>\n<th>Private</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.457</td>\n<td>0.457</td>\n</tr>\n</tbody>\n</table>\n<p>The codes will be made pubic after reference checkup. Hard work everyone, and thank you for everybody who shared great insights and their excellent works!!</p>",
      "rawMarkdown": "Since every attempts to pre-train Roberta failed for me, I used publicly pre-trained robertas from @maunish and @tutty1992. I ensembled roberta-base and roberta-large with different strategy, and here are the methods.\n\n## 1.  Roberta-Base + SVR\nFine-tuning @maunish 's model and predicting with linear layer, I could get \n\n| Private | Public |\n| --- | --- |\n| 0.465 | 0.466 |.\n\nThen I tried to predict with SVR 5-folds wrt training data without using the linear layer. After fine-tuning the regularization parameters for each fold, I could get a slightly better score,\n\n| Private | Public |\n| --- | --- |\n| 0.464 | 0.464 |.\n\n## 2.  Roberta-Large with different poolers and seeds\nFine-tuning @tutty1992 's model and referring to @jcesquiveld , I tried to ensemble Roberta-Large models with different poolers with different seeds. Although each model is around 0.468 on public/private leaderboard, ensembling 6 different models (2 different poolers and 3 different seeds) led to higher performances as I thought,\n\n| Private | Public |\n| --- | --- |\n| 0.463 | 0.463 |.\n\nBesides, I've also ensembled public notebooks and datasets from @rhtsingh , @chamecall , @rajat95gupta and @nomlmean to get a score\n\n| Private | Public |\n| --- | --- |\n| 0.457 | 0.457 |.\n\nThe codes will be made pubic after reference checkup. Hard work everyone, and thank you for everybody who shared great insights and their excellent works!!",
      "votes": 3
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:45:53Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6951659,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/pndong-100th-place-ensemble-public-notebooks"
}