{
  "id": 258149,
  "title": "16th Solution Ensemble + Pseudo-labeling",
  "content": "<p>Thanks to the organizers for an interesting competition, and to <a href=\"https://www.kaggle.com/andretugan\" target=\"_blank\">@andretugan</a> and <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> for the baseline</p>\n<h1>Model</h1>\n<p>My solution is based on an ensemble of several architectures. roberta-large, deberta_large, electra_large. The others worked out worse.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>roberta-large-mnli</td>\n<td>455-458</td>\n<td>465-470</td>\n</tr>\n<tr>\n<td>roberta-large</td>\n<td>459</td>\n<td>462</td>\n</tr>\n<tr>\n<td>deberta_large</td>\n<td>464-466</td>\n<td>457-459</td>\n</tr>\n<tr>\n<td>electra_large</td>\n<td>466-467</td>\n<td>470-474</td>\n</tr>\n<tr>\n<td>ensemble</td>\n<td>446-450</td>\n<td>449-450</td>\n</tr>\n</tbody>\n</table>\n<h1>Train</h1>\n<p>The training went as follows:</p>\n<ol>\n<li>Training the roberta-large model.</li>\n<li>I predict targets on the texts of Wikipedia (20 000 lines, if more, the scor drops).</li>\n<li>I train different models Roberta, deberta, electra<br>\n3.1. First on wikipedia<br>\n3.2. Complete finetuning on a training sample</li>\n<li>Predict the test</li>\n</ol>\n<h1>Details</h1>\n<p>Using differential learning rates for BERT layers</p>\n<p>Tried different approaches, but they all work the same</p>\n<pre><code>     last_layer_hidden_states = roberta_output.hidden_states[-1]\n</code></pre>\n<p>or</p>\n<pre><code>     last_layer_hidden_states = roberta_output.hidden_states[-2]\n</code></pre>\n<p>or</p>\n<pre><code>    layer_hidden_states1 = roberta_output.hidden_states[-1]\n\n    layer_hidden_states2 = roberta_output.hidden_states[-2]\n\n    last_layer_hidden_states = torch.cat((layer_hidden_states1, layer_hidden_states2), 2)\n</code></pre>\n<p>And also</p>\n<pre><code>     weights = self.attention(last_layer_hidden_states)\n\n     y = torch.sum(weights * last_layer_hidden_states, dim=1)\n</code></pre>\n<p>or</p>\n<pre><code>      y = torch.mean(last_layer_hidden_states, 1)\n</code></pre>\n<p>etc.</p>\n<p>Feature engineering, Reinitializing did not work either</p>",
  "comments": [
    {
      "id": 1426091,
      "author_name": "Salim Khazem",
      "author_url": "",
      "post_date": "2021-08-03T07:17:52.507000",
      "content": "<p>Great work ,Thanks for sharing, do you plan to share your final notebook ?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1426232,
          "author_name": "Kramarenko Vladislav",
          "author_url": "",
          "post_date": "2021-08-03T07:23:44.413000",
          "content": "<p>Thank you. Ensemble inference or single model training?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1426271,
          "author_name": "Salim Khazem",
          "author_url": "",
          "post_date": "2021-08-03T07:25:48.220000",
          "content": "<p>Both if it's possible<br>\nbut i'm more interested about the single model training <br>\nThanks a lot </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1427339,
          "author_name": "Kramarenko Vladislav",
          "author_url": "",
          "post_date": "2021-08-03T08:20:43.710000",
          "content": "<p><a href=\"https://www.kaggle.com/vlomme/commonlit-deberta-large\" target=\"_blank\">https://www.kaggle.com/vlomme/commonlit-deberta-large</a></p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1430832,
          "author_name": "Salim Khazem",
          "author_url": "",
          "post_date": "2021-08-03T11:56:48.027000",
          "content": "<p>Thanks a lot for the notebook <a href=\"https://www.kaggle.com/vlomme\" target=\"_blank\">@vlomme</a> </p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 5,
  "messages": [
    {
      "id": 1425965,
      "postDate": "2021-08-03T07:11:32.177Z",
      "content": "<p>Thanks to the organizers for an interesting competition, and to <a href=\"https://www.kaggle.com/andretugan\" target=\"_blank\">@andretugan</a> and <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> for the baseline</p>\n<h1>Model</h1>\n<p>My solution is based on an ensemble of several architectures. roberta-large, deberta_large, electra_large. The others worked out worse.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>roberta-large-mnli</td>\n<td>455-458</td>\n<td>465-470</td>\n</tr>\n<tr>\n<td>roberta-large</td>\n<td>459</td>\n<td>462</td>\n</tr>\n<tr>\n<td>deberta_large</td>\n<td>464-466</td>\n<td>457-459</td>\n</tr>\n<tr>\n<td>electra_large</td>\n<td>466-467</td>\n<td>470-474</td>\n</tr>\n<tr>\n<td>ensemble</td>\n<td>446-450</td>\n<td>449-450</td>\n</tr>\n</tbody>\n</table>\n<h1>Train</h1>\n<p>The training went as follows:</p>\n<ol>\n<li>Training the roberta-large model.</li>\n<li>I predict targets on the texts of Wikipedia (20 000 lines, if more, the scor drops).</li>\n<li>I train different models Roberta, deberta, electra<br>\n3.1. First on wikipedia<br>\n3.2. Complete finetuning on a training sample</li>\n<li>Predict the test</li>\n</ol>\n<h1>Details</h1>\n<p>Using differential learning rates for BERT layers</p>\n<p>Tried different approaches, but they all work the same</p>\n<pre><code>     last_layer_hidden_states = roberta_output.hidden_states[-1]\n</code></pre>\n<p>or</p>\n<pre><code>     last_layer_hidden_states = roberta_output.hidden_states[-2]\n</code></pre>\n<p>or</p>\n<pre><code>    layer_hidden_states1 = roberta_output.hidden_states[-1]\n\n    layer_hidden_states2 = roberta_output.hidden_states[-2]\n\n    last_layer_hidden_states = torch.cat((layer_hidden_states1, layer_hidden_states2), 2)\n</code></pre>\n<p>And also</p>\n<pre><code>     weights = self.attention(last_layer_hidden_states)\n\n     y = torch.sum(weights * last_layer_hidden_states, dim=1)\n</code></pre>\n<p>or</p>\n<pre><code>      y = torch.mean(last_layer_hidden_states, 1)\n</code></pre>\n<p>etc.</p>\n<p>Feature engineering, Reinitializing did not work either</p>",
      "rawMarkdown": "Thanks to the organizers for an interesting competition, and to @andretugan and @rhtsingh for the baseline\n# Model\nMy solution is based on an ensemble of several architectures. roberta-large, deberta_large, electra_large. The others worked out worse.\n| model | public | private |\n| --- | --- | --- |\n| roberta-large-mnli | 455-458 | 465-470 |\n| roberta-large | 459 | 462 |\n| deberta_large | 464-466 | 457-459 |\n| electra_large  | 466-467  | 470-474 |\n| ensemble | 446-450 | 449-450 |\n\n# Train\nThe training went as follows:\n1. Training the roberta-large model.\n2. I predict targets on the texts of Wikipedia (20 000 lines, if more, the scor drops).\n3. I train different models Roberta, deberta, electra\n3.1. First on wikipedia\n3.2. Complete finetuning on a training sample\n4. Predict the test\n\n# Details\nUsing differential learning rates for BERT layers\n\nTried different approaches, but they all work the same\n\n         last_layer_hidden_states = roberta_output.hidden_states[-1]\n   \nor\n\n         last_layer_hidden_states = roberta_output.hidden_states[-2]\n \n or\n\n        layer_hidden_states1 = roberta_output.hidden_states[-1]\n\n        layer_hidden_states2 = roberta_output.hidden_states[-2]\n\n        last_layer_hidden_states = torch.cat((layer_hidden_states1, layer_hidden_states2), 2)\n\nAnd also\n\n         weights = self.attention(last_layer_hidden_states)\n\n         y = torch.sum(weights * last_layer_hidden_states, dim=1)\n\nor\n\n          y = torch.mean(last_layer_hidden_states, 1)\n\netc.\n\nFeature engineering, Reinitializing did not work either\n\n",
      "votes": 27
    },
    {
      "id": 1426091,
      "postDate": "2021-08-03T07:17:52.507Z",
      "content": "<p>Great work ,Thanks for sharing, do you plan to share your final notebook ?</p>",
      "rawMarkdown": "Great work ,Thanks for sharing, do you plan to share your final notebook ?",
      "votes": 1,
      "replies": [
        {
          "id": 1426232,
          "postDate": "2021-08-03T07:23:44.413Z",
          "content": "<p>Thank you. Ensemble inference or single model training?</p>",
          "rawMarkdown": "Thank you. Ensemble inference or single model training?",
          "votes": 1
        },
        {
          "id": 1426271,
          "postDate": "2021-08-03T07:25:48.220Z",
          "content": "<p>Both if it's possible<br>\nbut i'm more interested about the single model training <br>\nThanks a lot </p>",
          "rawMarkdown": "Both if it's possible\nbut i'm more interested about the single model training \nThanks a lot "
        },
        {
          "id": 1427339,
          "postDate": "2021-08-03T08:20:43.710Z",
          "content": "<p><a href=\"https://www.kaggle.com/vlomme/commonlit-deberta-large\" target=\"_blank\">https://www.kaggle.com/vlomme/commonlit-deberta-large</a></p>",
          "rawMarkdown": "https://www.kaggle.com/vlomme/commonlit-deberta-large",
          "votes": 1
        },
        {
          "id": 1430832,
          "postDate": "2021-08-03T11:56:48.027Z",
          "content": "<p>Thanks a lot for the notebook <a href=\"https://www.kaggle.com/vlomme\" target=\"_blank\">@vlomme</a> </p>",
          "rawMarkdown": "Thanks a lot for the notebook @vlomme "
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:46:40Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6737973,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/kramarenko-vladislav-16th-solution-ensemble-pseudo"
}