{
  "id": 257535,
  "title": "69th place solution",
  "content": "<p>First of all, thanks to kaggle host, and all participants.<br>\nI got a silver medal for the first time!</p>\n<h2>Models</h2>\n<p>I used one notebook as a baseline. Thank you very much, <a href=\"https://www.kaggle.com/andretugan\" target=\"_blank\">@andretugan</a>.<br>\n<a href=\"https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch/output\" target=\"_blank\">https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch/output</a></p>\n<ul>\n<li>4models ensemble<ul>\n<li>Roberta base</li>\n<li>Roberta Large LSTM*1 head</li>\n<li>Roberta Large LSTM*2 head</li>\n<li>Roberta Large MeanPooling head</li></ul></li>\n</ul>\n<h2>CV Strategy</h2>\n<p>5 Fold Stratified CV<br>\n<a href=\"https://www.kaggle.com/abhishek/step-1-create-folds\" target=\"_blank\">https://www.kaggle.com/abhishek/step-1-create-folds</a></p>\n<h2>Submissions</h2>\n<p>Public 0.458, Private 0.456</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Ratio</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Roberta base</td>\n<td>0.25</td>\n<td>0.477</td>\n<td>0.467</td>\n<td>0.470</td>\n</tr>\n<tr>\n<td>Roberta Large LSTM*1 head</td>\n<td>0.25</td>\n<td>0.480</td>\n<td>0.465</td>\n<td>0.464</td>\n</tr>\n<tr>\n<td>Roberta Large LSTM*2 head</td>\n<td>0.25</td>\n<td>0.480</td>\n<td>0.464</td>\n<td>0.462</td>\n</tr>\n<tr>\n<td>Roberta Large MeanPooling head</td>\n<td>0.25</td>\n<td>0.481</td>\n<td>0.468</td>\n<td>0.465</td>\n</tr>\n</tbody>\n</table>\n<p>See you at the next competition!</p>",
  "comments": [
    {
      "id": 1420760,
      "author_name": "over watch",
      "author_url": "",
      "post_date": "2021-08-03T03:31:48.197000",
      "content": "<p>Congrats to your first Silver medal! <br>\nDo you mind sharing the custom heads used in your models?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1421566,
          "author_name": "norikatamari",
          "author_url": "",
          "post_date": "2021-08-03T04:12:11.177000",
          "content": "<p>Thank you for your comment!<br>\nRoberta Large LSTM*2 head model is as shown below:</p>\n<pre><code>class LitModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n\n        config = AutoConfig.from_pretrained(ROBERTA_PATH)\n        config.update({\"output_hidden_states\": True,\n                       \"hidden_dropout_prob\": 0.0,\n                       \"layer_norm_eps\": 1e-7})\n\n        self.roberta = AutoModel.from_pretrained(ROBERTA_PATH, config=config)\n\n        self.attention = nn.Sequential(\n            nn.Linear(1024, 512),\n            nn.Tanh(),\n            nn.Linear(512, 1),\n            nn.Softmax(dim=1)\n        )\n\n        self.lstm1 = nn.Sequential(\n            nn.LSTM(1024, 768, batch_first=True, bidirectional=False)\n        )\n        self.lstm2 = nn.Sequential(\n            nn.LSTM(768, 512, batch_first=True, bidirectional=False)\n        )\n\n        self.regressor = nn.Sequential(\n            nn.Linear(512, 1)\n        )\n\n    def forward(self, input_ids, attention_mask):\n        roberta_output = self.roberta(input_ids=input_ids,\n                                      attention_mask=attention_mask)\n        last_layer_hidden_states = roberta_output.hidden_states[-1] # (8, 256, 1024)\n\n        weights = self.attention(last_layer_hidden_states)  # (8, 256, 1)\n\n        context_vector = weights * last_layer_hidden_states  # (8, 256, 1024)\n\n        # LSTM\n        lstm_output, (h, c) = self.lstm1(context_vector)  # (8, 256, 768)\n        lstm_output, (h, c) = self.lstm2(lstm_output)  # (8, 256, 512)\n\n        context_vector = torch.sum(lstm_output, dim=1)  # (8, 512)\n\n        return self.regressor(context_vector)  # [8, 1]\n</code></pre>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1421884,
          "author_name": "over watch",
          "author_url": "",
          "post_date": "2021-08-03T04:27:12.003000",
          "content": "<p>Wow, thanks a lot! </p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 3,
  "messages": [
    {
      "id": 1420696,
      "postDate": "2021-08-03T03:28:30.617Z",
      "content": "<p>First of all, thanks to kaggle host, and all participants.<br>\nI got a silver medal for the first time!</p>\n<h2>Models</h2>\n<p>I used one notebook as a baseline. Thank you very much, <a href=\"https://www.kaggle.com/andretugan\" target=\"_blank\">@andretugan</a>.<br>\n<a href=\"https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch/output\" target=\"_blank\">https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch/output</a></p>\n<ul>\n<li>4models ensemble<ul>\n<li>Roberta base</li>\n<li>Roberta Large LSTM*1 head</li>\n<li>Roberta Large LSTM*2 head</li>\n<li>Roberta Large MeanPooling head</li></ul></li>\n</ul>\n<h2>CV Strategy</h2>\n<p>5 Fold Stratified CV<br>\n<a href=\"https://www.kaggle.com/abhishek/step-1-create-folds\" target=\"_blank\">https://www.kaggle.com/abhishek/step-1-create-folds</a></p>\n<h2>Submissions</h2>\n<p>Public 0.458, Private 0.456</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Ratio</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Roberta base</td>\n<td>0.25</td>\n<td>0.477</td>\n<td>0.467</td>\n<td>0.470</td>\n</tr>\n<tr>\n<td>Roberta Large LSTM*1 head</td>\n<td>0.25</td>\n<td>0.480</td>\n<td>0.465</td>\n<td>0.464</td>\n</tr>\n<tr>\n<td>Roberta Large LSTM*2 head</td>\n<td>0.25</td>\n<td>0.480</td>\n<td>0.464</td>\n<td>0.462</td>\n</tr>\n<tr>\n<td>Roberta Large MeanPooling head</td>\n<td>0.25</td>\n<td>0.481</td>\n<td>0.468</td>\n<td>0.465</td>\n</tr>\n</tbody>\n</table>\n<p>See you at the next competition!</p>",
      "rawMarkdown": "First of all, thanks to kaggle host, and all participants.\nI got a silver medal for the first time!\n\n## Models\nI used one notebook as a baseline. Thank you very much, [@andretugan](https://www.kaggle.com/andretugan).\nhttps://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch/output\n\n- 4models ensemble\n    - Roberta base\n    - Roberta Large LSTM*1 head\n    - Roberta Large LSTM*2 head\n    - Roberta Large MeanPooling head\n\n## CV Strategy\n5 Fold Stratified CV\nhttps://www.kaggle.com/abhishek/step-1-create-folds\n\n## Submissions\nPublic 0.458, Private 0.456\n\n| Model |  Ratio | CV | Public LB | Private LB | \n| --- | --- | --- | --- | --- |\n| Roberta base |  0.25 | 0.477 | 0.467 | 0.470 | \n| Roberta Large LSTM*1 head | 0.25 | 0.480  | 0.465 | 0.464 | \n| Roberta Large LSTM*2 head | 0.25 | 0.480 | 0.464 | 0.462 | \n| Roberta Large MeanPooling head | 0.25 | 0.481 | 0.468 | 0.465 | \n\nSee you at the next competition!",
      "votes": 14
    },
    {
      "id": 1420760,
      "postDate": "2021-08-03T03:31:48.197Z",
      "content": "<p>Congrats to your first Silver medal! <br>\nDo you mind sharing the custom heads used in your models?</p>",
      "rawMarkdown": "Congrats to your first Silver medal! \nDo you mind sharing the custom heads used in your models?",
      "replies": [
        {
          "id": 1421566,
          "postDate": "2021-08-03T04:12:11.177Z",
          "content": "<p>Thank you for your comment!<br>\nRoberta Large LSTM*2 head model is as shown below:</p>\n<pre><code>class LitModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n\n        config = AutoConfig.from_pretrained(ROBERTA_PATH)\n        config.update({\"output_hidden_states\": True,\n                       \"hidden_dropout_prob\": 0.0,\n                       \"layer_norm_eps\": 1e-7})\n\n        self.roberta = AutoModel.from_pretrained(ROBERTA_PATH, config=config)\n\n        self.attention = nn.Sequential(\n            nn.Linear(1024, 512),\n            nn.Tanh(),\n            nn.Linear(512, 1),\n            nn.Softmax(dim=1)\n        )\n\n        self.lstm1 = nn.Sequential(\n            nn.LSTM(1024, 768, batch_first=True, bidirectional=False)\n        )\n        self.lstm2 = nn.Sequential(\n            nn.LSTM(768, 512, batch_first=True, bidirectional=False)\n        )\n\n        self.regressor = nn.Sequential(\n            nn.Linear(512, 1)\n        )\n\n    def forward(self, input_ids, attention_mask):\n        roberta_output = self.roberta(input_ids=input_ids,\n                                      attention_mask=attention_mask)\n        last_layer_hidden_states = roberta_output.hidden_states[-1] # (8, 256, 1024)\n\n        weights = self.attention(last_layer_hidden_states)  # (8, 256, 1)\n\n        context_vector = weights * last_layer_hidden_states  # (8, 256, 1024)\n\n        # LSTM\n        lstm_output, (h, c) = self.lstm1(context_vector)  # (8, 256, 768)\n        lstm_output, (h, c) = self.lstm2(lstm_output)  # (8, 256, 512)\n\n        context_vector = torch.sum(lstm_output, dim=1)  # (8, 512)\n\n        return self.regressor(context_vector)  # [8, 1]\n</code></pre>",
          "rawMarkdown": "Thank you for your comment!\nRoberta Large LSTM*2 head model is as shown below:\n```\nclass LitModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n\n        config = AutoConfig.from_pretrained(ROBERTA_PATH)\n        config.update({\"output_hidden_states\": True,\n                       \"hidden_dropout_prob\": 0.0,\n                       \"layer_norm_eps\": 1e-7})\n\n        self.roberta = AutoModel.from_pretrained(ROBERTA_PATH, config=config)\n\n        self.attention = nn.Sequential(\n            nn.Linear(1024, 512),\n            nn.Tanh(),\n            nn.Linear(512, 1),\n            nn.Softmax(dim=1)\n        )\n\n        self.lstm1 = nn.Sequential(\n            nn.LSTM(1024, 768, batch_first=True, bidirectional=False)\n        )\n        self.lstm2 = nn.Sequential(\n            nn.LSTM(768, 512, batch_first=True, bidirectional=False)\n        )\n\n        self.regressor = nn.Sequential(\n            nn.Linear(512, 1)\n        )\n\n    def forward(self, input_ids, attention_mask):\n        roberta_output = self.roberta(input_ids=input_ids,\n                                      attention_mask=attention_mask)\n        last_layer_hidden_states = roberta_output.hidden_states[-1] # (8, 256, 1024)\n\n        weights = self.attention(last_layer_hidden_states)  # (8, 256, 1)\n\n        context_vector = weights * last_layer_hidden_states  # (8, 256, 1024)\n\n        # LSTM\n        lstm_output, (h, c) = self.lstm1(context_vector)  # (8, 256, 768)\n        lstm_output, (h, c) = self.lstm2(lstm_output)  # (8, 256, 512)\n\n        context_vector = torch.sum(lstm_output, dim=1)  # (8, 512)\n\n        return self.regressor(context_vector)  # [8, 1]\n```\n\n",
          "votes": 3
        },
        {
          "id": 1421884,
          "postDate": "2021-08-03T04:27:12.003Z",
          "content": "<p>Wow, thanks a lot! </p>",
          "rawMarkdown": "Wow, thanks a lot! "
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:44:38Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6880284,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/norikatamari-69th-place-solution"
}