{
  "id": 369760,
  "title": "15th Place Solution : 6 kinds of AttentionPooling for each metric work well for us",
  "content": "<h3>First of all</h3>\n<p>We appreciate this fortunate luck, and are truly honored to receive the gold medal!!<br>\nFrom the results, Yuichi Tateno ( <a href=\"https://www.kaggle.com/hotchpotch\" target=\"_blank\">@hotchpotch</a> ) and Masaki Aota ( <a href=\"https://www.kaggle.com/masakiaota\" target=\"_blank\">@masakiaota</a> ) became the Kaggle Competitions Master.</p>\n<h3>Overview</h3>\n<p>Our final submission achieved Private Score: 0.4346, Public Score: 0.4351, Local CV: 0.4431. The inference is ensembling of 11 models. Our inference notebook is here. <a href=\"https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl\" target=\"_blank\">https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl</a></p>\n<p>The summary of 11 models is following. Abbreviations appearing in column names will be explained in <strong>What Worked</strong>.</p>\n<table>\n<thead>\n<tr>\n<th>exp num</th>\n<th>backbone model</th>\n<th>CV</th>\n<th>Pooling</th>\n<th>AWP</th>\n<th>LLRD</th>\n<th>ReInit</th>\n<th>SW</th>\n<th>max_len</th>\n<th>LayerNorm</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>072</td>\n<td>deberta-v3-large</td>\n<td>0.4491</td>\n<td>Mean</td>\n<td>Yes</td>\n<td>No</td>\n<td>No</td>\n<td>No</td>\n<td>Inf</td>\n<td>No</td>\n</tr>\n<tr>\n<td>089</td>\n<td>deberta-v3-large</td>\n<td>0.4487</td>\n<td>Mean</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>No</td>\n<td>No</td>\n<td>Inf</td>\n<td>No</td>\n</tr>\n<tr>\n<td>136</td>\n<td>deberta-v3-large</td>\n<td>0.4488</td>\n<td>6AP</td>\n<td>No</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Inf</td>\n<td>No</td>\n</tr>\n<tr>\n<td>182</td>\n<td>deberta-v3-base</td>\n<td>0.4496</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>No</td>\n</tr>\n<tr>\n<td>214</td>\n<td>deberta-v1-large</td>\n<td>0.4484</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>215</td>\n<td>deberta-v3-large</td>\n<td>0.4481</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>226</td>\n<td>deberta-v3-large</td>\n<td>0.4477</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>No</td>\n<td>Inf</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>231</td>\n<td>deberta-v3-base</td>\n<td>0.4487</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>232</td>\n<td>deberta-v3-base</td>\n<td>0.4508</td>\n<td>6AP</td>\n<td>No</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>233</td>\n<td>deberta-v1-large</td>\n<td>0.4486</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>No</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>234</td>\n<td>deberta-v1-large</td>\n<td>0.4495</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Inf</td>\n<td>Yes</td>\n</tr>\n</tbody>\n</table>\n<p>We use the following ensemble weights:</p>\n<table>\n<thead>\n<tr>\n<th>exp num</th>\n<th>cohesion</th>\n<th>syntax</th>\n<th>vocabulary</th>\n<th>phraseology</th>\n<th>grammar</th>\n<th>conventions</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>072</td>\n<td>0.0186</td>\n<td>0.1387</td>\n<td>0.1042</td>\n<td>0.1474</td>\n<td>0.1135</td>\n<td>0.1757</td>\n</tr>\n<tr>\n<td>089</td>\n<td>0.0022</td>\n<td>0.0123</td>\n<td>0.0001</td>\n<td>0.1071</td>\n<td>0.2395</td>\n<td>0.0512</td>\n</tr>\n<tr>\n<td>136</td>\n<td>0.1295</td>\n<td>0.1666</td>\n<td>0.0589</td>\n<td>0.0950</td>\n<td>0.1235</td>\n<td>0.0012</td>\n</tr>\n<tr>\n<td>182</td>\n<td>0.0011</td>\n<td>0.1051</td>\n<td>0.0749</td>\n<td>0.2280</td>\n<td>0.0794</td>\n<td>0.0577</td>\n</tr>\n<tr>\n<td>214</td>\n<td>0.2944</td>\n<td>0.0930</td>\n<td>0.1365</td>\n<td>0.0485</td>\n<td>0.0259</td>\n<td>0.0922</td>\n</tr>\n<tr>\n<td>215</td>\n<td>0.0965</td>\n<td>0.0036</td>\n<td>0.0244</td>\n<td>0.1390</td>\n<td>0.0772</td>\n<td>0.1055</td>\n</tr>\n<tr>\n<td>226</td>\n<td>0.1654</td>\n<td>0.1212</td>\n<td>0.1700</td>\n<td>0.0469</td>\n<td>0.0394</td>\n<td>0.0325</td>\n</tr>\n<tr>\n<td>231</td>\n<td>0.0562</td>\n<td>0.0801</td>\n<td>0.0157</td>\n<td>0.0216</td>\n<td>0.0958</td>\n<td>0.1178</td>\n</tr>\n<tr>\n<td>232</td>\n<td>0.0105</td>\n<td>0.0434</td>\n<td>0.1012</td>\n<td>0.0827</td>\n<td>0.1154</td>\n<td>0.0830</td>\n</tr>\n<tr>\n<td>233</td>\n<td>0.1543</td>\n<td>0.2058</td>\n<td>0.1807</td>\n<td>0.0194</td>\n<td>0.0317</td>\n<td>0.1106</td>\n</tr>\n<tr>\n<td>234</td>\n<td>0.0704</td>\n<td>0.0318</td>\n<td>0.1335</td>\n<td>0.0662</td>\n<td>0.0596</td>\n<td>0.1727</td>\n</tr>\n</tbody>\n</table>\n<h3>Our Code</h3>\n<p>Inference: <a href=\"https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl\" target=\"_blank\">https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl</a></p>\n<p>Training: <a href=\"https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook\" target=\"_blank\">https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook</a></p>\n<p>If the codes are helpful, don't forget to upvote not only mine but also Y.Nakama's one (<a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a>) .</p>\n<h3>What Worked</h3>\n<ul>\n<li>[High Impact] 6 kinds of AttentionPooling for each metric (6AP)<ul>\n<li>CV : -0.0015</li>\n<li>This allows our model to focus on different tokens for different metrics.</li>\n<li>The idea can be illustrated like this (right one).<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1655570%2F16fd4987360a19c9c775105b0164ac7d%2F6AP_M.jpg?generation=1670065314021872&amp;alt=media\" alt=\"\"></li></ul></li>\n<li>[High Impact] Adversarial Weights Perturbation (AWP)<ul>\n<li>CV : -0.002</li>\n<li>Original paper : <a href=\"https://proceedings.neurips.cc/paper/2020/file/1ef91c212e30e14bf125e9374262401f-Paper.pdf\" target=\"_blank\">https://proceedings.neurips.cc/paper/2020/file/1ef91c212e30e14bf125e9374262401f-Paper.pdf</a></li>\n<li>My slide (in Japanese) : <a href=\"https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization\" target=\"_blank\">https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization</a></li></ul></li>\n<li>[High Impact] Layer-wise Lerarning Rate Decay (LLRD)<ul>\n<li>CV : -0.0005</li>\n<li>Decay by *0.85 for large model, *0.7225 for base model.</li></ul></li>\n<li>[High Impact] Last Layer Re-initialization (ReInit)<ul>\n<li>CV: -0.0006</li></ul></li>\n<li>([High Impact] Of Cource, Ensembling)</li>\n<li>[Medium Impact] Max Token Length to 640 (max_len)<ul>\n<li>CV: - 0~0.001</li>\n<li>Only work for base model</li>\n<li>Not work for large model but training speed become fast</li></ul></li>\n<li>[Medium Impact] SlidingWindowTransformerModel (SW)<ul>\n<li>CV: - 0~0.001</li>\n<li>But bad combination with AWP.</li>\n<li>My implementation is similar with this <a href=\"https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322\" target=\"_blank\">https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322</a></li></ul></li>\n<li>[Medium Impact] Pseudo Labeling<ul>\n<li>CV: - 0.0001~0.0006</li>\n<li>Our submission which got highest private score did not contain the model learning with pseudo-label.</li></ul></li>\n<li>[Medium Impact] High Frequency Validation and Saving<ul>\n<li>CV: - 0.0008</li></ul></li>\n<li>[Medium Impact] Layer Normalization after Pooling Layer, before Fully Connected Layer (LayerNorm)<ul>\n<li>CV: - 0~0.001</li>\n<li>Only work for base model</li></ul></li>\n<li>[Medium Impact] Multi sample dropout<ul>\n<li>CV: +- 0.0</li>\n<li>The curve of validation loss becomes stable a little.</li></ul></li>\n<li>[Medium Impact] Optimize Ensembling Weights for Each Metrics Independently<ul>\n<li>CV: - 0.0001 ~ 0.0004</li></ul></li>\n</ul>\n<h3>What Didn’t Work</h3>\n<ul>\n<li>SiFT<ul>\n<li>May be bugs in our code…</li></ul></li>\n<li>SVR<ul>\n<li>No leakage but too low CV and bad public score. So, we didn't use SVR to ensemble.</li></ul></li>\n<li>Any Augmentations</li>\n<li>Any Losses except L1 Smooth</li>\n<li>Pretraining using Feedback1,2 Sentences<ul>\n<li>I’m not sure this is really not to work…</li></ul></li>\n<li>Dealing with “Generic_Name” and others</li>\n<li>Change the Way to Initialize Weights<ul>\n<li>Parameters sampled from normal distribution is the best for us</li></ul></li>\n</ul>\n<h3>Additional Context</h3>\n<ul>\n<li>Our Best Single model (in private and CV)<ul>\n<li>Private: 0.4361, Public: 0.4375, CV: 0.4450</li>\n<li>This model was <strong>Learning from Ensembled Pseudo-Labels.</strong></li>\n<li>But, our emsemble inference, which get highest private score, did not contains this model. If we leveraged this model to ensemble, we might get the more good standing.</li>\n<li>This idea may be useful especially for efficiency prize (in my opinion).</li></ul></li>\n</ul>\n<h3>Important Citations</h3>\n<ul>\n<li>Y.Nakama’s baseline <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></li>\n<li>AI Shift’s tech blog (Japanese) <a href=\"https://www.ai-shift.co.jp/techblog/2170\" target=\"_blank\">https://www.ai-shift.co.jp/techblog/2170</a></li>\n<li>Discussion about Attention Pooling <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678</a></li>\n<li>XGB+LGB Feedback Prize CV 0.7322 <a href=\"https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322\" target=\"_blank\">https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322</a></li>\n</ul>\n<h3>Thanks and Acknowledgements</h3>\n<p>We would like to thank the organizers of the competition for hosting this amazing event.<br>\nI personally would like to mention and thank Taro Masuda ( <a href=\"https://www.kaggle.com/taromasuda\" target=\"_blank\">@taromasuda</a> ), Raiii ( <a href=\"https://www.kaggle.com/skraiii\" target=\"_blank\">@skraiii</a> ) and MPEG ( <a href=\"https://www.kaggle.com/mpeg31\" target=\"_blank\">@mpeg31</a> ) who did not participate in this competition, as I have learnt so much from them in the past competitions. I certainly was able to utilize the knowledge this time!</p>\n<h3>Team Members</h3>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1655570%2Fb36ba8b5675df4b7e1b29ea114c0e8d6%2FTeamN.png?generation=1670066208813739&amp;alt=media\" alt=\"\"><br>\nFrom left in the above figure:</p>\n<ul>\n<li>Masaki Aota <a href=\"https://www.kaggle.com/masakiaota\" target=\"_blank\">@masakiaota</a></li>\n<li>Olivine-Ryo <a href=\"https://www.kaggle.com/olivineryo\" target=\"_blank\">@olivineryo</a> </li>\n<li>Yuichi Tateno <a href=\"https://www.kaggle.com/hotchpotch\" target=\"_blank\">@hotchpotch</a></li>\n</ul>",
  "comments": [
    {
      "id": 2904745,
      "author_name": "Adithya",
      "author_url": "",
      "post_date": "2024-07-04T14:45:43.550000",
      "content": "<p>How did u come up with exact values for learning rate decay</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 2051262,
      "postDate": "2022-12-01T10:01:32.223Z",
      "content": "<h3>First of all</h3>\n<p>We appreciate this fortunate luck, and are truly honored to receive the gold medal!!<br>\nFrom the results, Yuichi Tateno ( <a href=\"https://www.kaggle.com/hotchpotch\" target=\"_blank\">@hotchpotch</a> ) and Masaki Aota ( <a href=\"https://www.kaggle.com/masakiaota\" target=\"_blank\">@masakiaota</a> ) became the Kaggle Competitions Master.</p>\n<h3>Overview</h3>\n<p>Our final submission achieved Private Score: 0.4346, Public Score: 0.4351, Local CV: 0.4431. The inference is ensembling of 11 models. Our inference notebook is here. <a href=\"https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl\" target=\"_blank\">https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl</a></p>\n<p>The summary of 11 models is following. Abbreviations appearing in column names will be explained in <strong>What Worked</strong>.</p>\n<table>\n<thead>\n<tr>\n<th>exp num</th>\n<th>backbone model</th>\n<th>CV</th>\n<th>Pooling</th>\n<th>AWP</th>\n<th>LLRD</th>\n<th>ReInit</th>\n<th>SW</th>\n<th>max_len</th>\n<th>LayerNorm</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>072</td>\n<td>deberta-v3-large</td>\n<td>0.4491</td>\n<td>Mean</td>\n<td>Yes</td>\n<td>No</td>\n<td>No</td>\n<td>No</td>\n<td>Inf</td>\n<td>No</td>\n</tr>\n<tr>\n<td>089</td>\n<td>deberta-v3-large</td>\n<td>0.4487</td>\n<td>Mean</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>No</td>\n<td>No</td>\n<td>Inf</td>\n<td>No</td>\n</tr>\n<tr>\n<td>136</td>\n<td>deberta-v3-large</td>\n<td>0.4488</td>\n<td>6AP</td>\n<td>No</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Inf</td>\n<td>No</td>\n</tr>\n<tr>\n<td>182</td>\n<td>deberta-v3-base</td>\n<td>0.4496</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>No</td>\n</tr>\n<tr>\n<td>214</td>\n<td>deberta-v1-large</td>\n<td>0.4484</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>215</td>\n<td>deberta-v3-large</td>\n<td>0.4481</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>226</td>\n<td>deberta-v3-large</td>\n<td>0.4477</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>No</td>\n<td>Inf</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>231</td>\n<td>deberta-v3-base</td>\n<td>0.4487</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>232</td>\n<td>deberta-v3-base</td>\n<td>0.4508</td>\n<td>6AP</td>\n<td>No</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>233</td>\n<td>deberta-v1-large</td>\n<td>0.4486</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>No</td>\n<td>640</td>\n<td>Yes</td>\n</tr>\n<tr>\n<td>234</td>\n<td>deberta-v1-large</td>\n<td>0.4495</td>\n<td>6AP</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Yes</td>\n<td>Inf</td>\n<td>Yes</td>\n</tr>\n</tbody>\n</table>\n<p>We use the following ensemble weights:</p>\n<table>\n<thead>\n<tr>\n<th>exp num</th>\n<th>cohesion</th>\n<th>syntax</th>\n<th>vocabulary</th>\n<th>phraseology</th>\n<th>grammar</th>\n<th>conventions</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>072</td>\n<td>0.0186</td>\n<td>0.1387</td>\n<td>0.1042</td>\n<td>0.1474</td>\n<td>0.1135</td>\n<td>0.1757</td>\n</tr>\n<tr>\n<td>089</td>\n<td>0.0022</td>\n<td>0.0123</td>\n<td>0.0001</td>\n<td>0.1071</td>\n<td>0.2395</td>\n<td>0.0512</td>\n</tr>\n<tr>\n<td>136</td>\n<td>0.1295</td>\n<td>0.1666</td>\n<td>0.0589</td>\n<td>0.0950</td>\n<td>0.1235</td>\n<td>0.0012</td>\n</tr>\n<tr>\n<td>182</td>\n<td>0.0011</td>\n<td>0.1051</td>\n<td>0.0749</td>\n<td>0.2280</td>\n<td>0.0794</td>\n<td>0.0577</td>\n</tr>\n<tr>\n<td>214</td>\n<td>0.2944</td>\n<td>0.0930</td>\n<td>0.1365</td>\n<td>0.0485</td>\n<td>0.0259</td>\n<td>0.0922</td>\n</tr>\n<tr>\n<td>215</td>\n<td>0.0965</td>\n<td>0.0036</td>\n<td>0.0244</td>\n<td>0.1390</td>\n<td>0.0772</td>\n<td>0.1055</td>\n</tr>\n<tr>\n<td>226</td>\n<td>0.1654</td>\n<td>0.1212</td>\n<td>0.1700</td>\n<td>0.0469</td>\n<td>0.0394</td>\n<td>0.0325</td>\n</tr>\n<tr>\n<td>231</td>\n<td>0.0562</td>\n<td>0.0801</td>\n<td>0.0157</td>\n<td>0.0216</td>\n<td>0.0958</td>\n<td>0.1178</td>\n</tr>\n<tr>\n<td>232</td>\n<td>0.0105</td>\n<td>0.0434</td>\n<td>0.1012</td>\n<td>0.0827</td>\n<td>0.1154</td>\n<td>0.0830</td>\n</tr>\n<tr>\n<td>233</td>\n<td>0.1543</td>\n<td>0.2058</td>\n<td>0.1807</td>\n<td>0.0194</td>\n<td>0.0317</td>\n<td>0.1106</td>\n</tr>\n<tr>\n<td>234</td>\n<td>0.0704</td>\n<td>0.0318</td>\n<td>0.1335</td>\n<td>0.0662</td>\n<td>0.0596</td>\n<td>0.1727</td>\n</tr>\n</tbody>\n</table>\n<h3>Our Code</h3>\n<p>Inference: <a href=\"https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl\" target=\"_blank\">https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl</a></p>\n<p>Training: <a href=\"https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook\" target=\"_blank\">https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook</a></p>\n<p>If the codes are helpful, don't forget to upvote not only mine but also Y.Nakama's one (<a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a>) .</p>\n<h3>What Worked</h3>\n<ul>\n<li>[High Impact] 6 kinds of AttentionPooling for each metric (6AP)<ul>\n<li>CV : -0.0015</li>\n<li>This allows our model to focus on different tokens for different metrics.</li>\n<li>The idea can be illustrated like this (right one).<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1655570%2F16fd4987360a19c9c775105b0164ac7d%2F6AP_M.jpg?generation=1670065314021872&amp;alt=media\" alt=\"\"></li></ul></li>\n<li>[High Impact] Adversarial Weights Perturbation (AWP)<ul>\n<li>CV : -0.002</li>\n<li>Original paper : <a href=\"https://proceedings.neurips.cc/paper/2020/file/1ef91c212e30e14bf125e9374262401f-Paper.pdf\" target=\"_blank\">https://proceedings.neurips.cc/paper/2020/file/1ef91c212e30e14bf125e9374262401f-Paper.pdf</a></li>\n<li>My slide (in Japanese) : <a href=\"https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization\" target=\"_blank\">https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization</a></li></ul></li>\n<li>[High Impact] Layer-wise Lerarning Rate Decay (LLRD)<ul>\n<li>CV : -0.0005</li>\n<li>Decay by *0.85 for large model, *0.7225 for base model.</li></ul></li>\n<li>[High Impact] Last Layer Re-initialization (ReInit)<ul>\n<li>CV: -0.0006</li></ul></li>\n<li>([High Impact] Of Cource, Ensembling)</li>\n<li>[Medium Impact] Max Token Length to 640 (max_len)<ul>\n<li>CV: - 0~0.001</li>\n<li>Only work for base model</li>\n<li>Not work for large model but training speed become fast</li></ul></li>\n<li>[Medium Impact] SlidingWindowTransformerModel (SW)<ul>\n<li>CV: - 0~0.001</li>\n<li>But bad combination with AWP.</li>\n<li>My implementation is similar with this <a href=\"https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322\" target=\"_blank\">https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322</a></li></ul></li>\n<li>[Medium Impact] Pseudo Labeling<ul>\n<li>CV: - 0.0001~0.0006</li>\n<li>Our submission which got highest private score did not contain the model learning with pseudo-label.</li></ul></li>\n<li>[Medium Impact] High Frequency Validation and Saving<ul>\n<li>CV: - 0.0008</li></ul></li>\n<li>[Medium Impact] Layer Normalization after Pooling Layer, before Fully Connected Layer (LayerNorm)<ul>\n<li>CV: - 0~0.001</li>\n<li>Only work for base model</li></ul></li>\n<li>[Medium Impact] Multi sample dropout<ul>\n<li>CV: +- 0.0</li>\n<li>The curve of validation loss becomes stable a little.</li></ul></li>\n<li>[Medium Impact] Optimize Ensembling Weights for Each Metrics Independently<ul>\n<li>CV: - 0.0001 ~ 0.0004</li></ul></li>\n</ul>\n<h3>What Didn’t Work</h3>\n<ul>\n<li>SiFT<ul>\n<li>May be bugs in our code…</li></ul></li>\n<li>SVR<ul>\n<li>No leakage but too low CV and bad public score. So, we didn't use SVR to ensemble.</li></ul></li>\n<li>Any Augmentations</li>\n<li>Any Losses except L1 Smooth</li>\n<li>Pretraining using Feedback1,2 Sentences<ul>\n<li>I’m not sure this is really not to work…</li></ul></li>\n<li>Dealing with “Generic_Name” and others</li>\n<li>Change the Way to Initialize Weights<ul>\n<li>Parameters sampled from normal distribution is the best for us</li></ul></li>\n</ul>\n<h3>Additional Context</h3>\n<ul>\n<li>Our Best Single model (in private and CV)<ul>\n<li>Private: 0.4361, Public: 0.4375, CV: 0.4450</li>\n<li>This model was <strong>Learning from Ensembled Pseudo-Labels.</strong></li>\n<li>But, our emsemble inference, which get highest private score, did not contains this model. If we leveraged this model to ensemble, we might get the more good standing.</li>\n<li>This idea may be useful especially for efficiency prize (in my opinion).</li></ul></li>\n</ul>\n<h3>Important Citations</h3>\n<ul>\n<li>Y.Nakama’s baseline <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></li>\n<li>AI Shift’s tech blog (Japanese) <a href=\"https://www.ai-shift.co.jp/techblog/2170\" target=\"_blank\">https://www.ai-shift.co.jp/techblog/2170</a></li>\n<li>Discussion about Attention Pooling <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678</a></li>\n<li>XGB+LGB Feedback Prize CV 0.7322 <a href=\"https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322\" target=\"_blank\">https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322</a></li>\n</ul>\n<h3>Thanks and Acknowledgements</h3>\n<p>We would like to thank the organizers of the competition for hosting this amazing event.<br>\nI personally would like to mention and thank Taro Masuda ( <a href=\"https://www.kaggle.com/taromasuda\" target=\"_blank\">@taromasuda</a> ), Raiii ( <a href=\"https://www.kaggle.com/skraiii\" target=\"_blank\">@skraiii</a> ) and MPEG ( <a href=\"https://www.kaggle.com/mpeg31\" target=\"_blank\">@mpeg31</a> ) who did not participate in this competition, as I have learnt so much from them in the past competitions. I certainly was able to utilize the knowledge this time!</p>\n<h3>Team Members</h3>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1655570%2Fb36ba8b5675df4b7e1b29ea114c0e8d6%2FTeamN.png?generation=1670066208813739&amp;alt=media\" alt=\"\"><br>\nFrom left in the above figure:</p>\n<ul>\n<li>Masaki Aota <a href=\"https://www.kaggle.com/masakiaota\" target=\"_blank\">@masakiaota</a></li>\n<li>Olivine-Ryo <a href=\"https://www.kaggle.com/olivineryo\" target=\"_blank\">@olivineryo</a> </li>\n<li>Yuichi Tateno <a href=\"https://www.kaggle.com/hotchpotch\" target=\"_blank\">@hotchpotch</a></li>\n</ul>",
      "rawMarkdown": "### First of all\nWe appreciate this fortunate luck, and are truly honored to receive the gold medal!!\nFrom the results, Yuichi Tateno ( @hotchpotch ) and Masaki Aota ( @masakiaota ) became the Kaggle Competitions Master.\n\n### Overview\n\nOur final submission achieved Private Score: 0.4346, Public Score: 0.4351, Local CV: 0.4431. The inference is ensembling of 11 models. Our inference notebook is here. [https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl](https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl)\n\nThe summary of 11 models is following. Abbreviations appearing in column names will be explained in **What Worked**.\n\n| exp num | backbone model | CV | Pooling | AWP | LLRD | ReInit | SW | max_len | LayerNorm |\n| -------- | ------------------ | ------ | ----- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 072 | deberta-v3-large | 0.4491 | Mean | Yes | No | No | No | Inf | No |\n| 089 | deberta-v3-large | 0.4487 | Mean | Yes | Yes | No | No | Inf | No |\n| 136 | deberta-v3-large | 0.4488 | 6AP | No | Yes | Yes | Yes | Inf | No |\n| 182 | deberta-v3-base | 0.4496 | 6AP | Yes | Yes | Yes | Yes | 640 | No |\n| 214 | deberta-v1-large | 0.4484 | 6AP | Yes | Yes | Yes | Yes | 640 | Yes |\n| 215 | deberta-v3-large | 0.4481 | 6AP | Yes | Yes | Yes | Yes | 640 | Yes |\n| 226 | deberta-v3-large | 0.4477 | 6AP | Yes | Yes | Yes | No | Inf | Yes |\n| 231 | deberta-v3-base | 0.4487 | 6AP | Yes | Yes | Yes | Yes | 640 | Yes |\n| 232 | deberta-v3-base | 0.4508 | 6AP | No | Yes | Yes | Yes | 640 | Yes |\n| 233 | deberta-v1-large | 0.4486 | 6AP | Yes | Yes | Yes | No | 640 | Yes |\n| 234 | deberta-v1-large | 0.4495 | 6AP | Yes | Yes | Yes | Yes | Inf | Yes |\n\nWe use the following ensemble weights:\n\n| exp num | cohesion | syntax | vocabulary | phraseology | grammar | conventions |\n| --- | --- | --- | --- | --- | --- | --- |\n| 072 | 0.0186 | 0.1387 | 0.1042 | 0.1474 | 0.1135 | 0.1757 |\n| 089 | 0.0022 | 0.0123 | 0.0001 | 0.1071 | 0.2395 | 0.0512 |\n| 136 | 0.1295 | 0.1666 | 0.0589 | 0.0950 | 0.1235 | 0.0012 |\n| 182 | 0.0011 | 0.1051 | 0.0749 | 0.2280 | 0.0794 | 0.0577 |\n| 214 | 0.2944 | 0.0930 | 0.1365 | 0.0485 | 0.0259 | 0.0922 |\n| 215 | 0.0965 | 0.0036 | 0.0244 | 0.1390 | 0.0772 | 0.1055 |\n| 226 | 0.1654 | 0.1212 | 0.1700 | 0.0469 | 0.0394 | 0.0325 |\n| 231 | 0.0562 | 0.0801 | 0.0157 | 0.0216 | 0.0958 | 0.1178 |\n| 232 | 0.0105 | 0.0434 | 0.1012 | 0.0827 | 0.1154 | 0.0830 |\n| 233 | 0.1543 | 0.2058 | 0.1807 | 0.0194 | 0.0317 | 0.1106 |\n| 234 | 0.0704 | 0.0318 | 0.1335 | 0.0662 | 0.0596 | 0.1727 |\n\n\n\n### Our Code\n\nInference: [https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl](https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl)\n\nTraining: https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook\n\nIf the codes are helpful, don't forget to upvote not only mine but also Y.Nakama's one (https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) .\n\n### What Worked\n\n- [High Impact] 6 kinds of AttentionPooling for each metric (6AP)\n    - CV : -0.0015\n    - This allows our model to focus on different tokens for different metrics.\n    - The idea can be illustrated like this (right one).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1655570%2F16fd4987360a19c9c775105b0164ac7d%2F6AP_M.jpg?generation=1670065314021872&alt=media)\n- [High Impact] Adversarial Weights Perturbation (AWP)\n    - CV : -0.002\n    - Original paper : [https://proceedings.neurips.cc/paper/2020/file/1ef91c212e30e14bf125e9374262401f-Paper.pdf](https://proceedings.neurips.cc/paper/2020/file/1ef91c212e30e14bf125e9374262401f-Paper.pdf)\n    - My slide (in Japanese) : [https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization](https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization)\n- [High Impact] Layer-wise Lerarning Rate Decay (LLRD)\n    - CV : -0.0005\n    - Decay by *0.85 for large model, *0.7225 for base model.\n- [High Impact] Last Layer Re-initialization (ReInit)\n    - CV: -0.0006\n- ([High Impact] Of Cource, Ensembling)\n- [Medium Impact] Max Token Length to 640 (max_len)\n    - CV: - 0~0.001\n    - Only work for base model\n    - Not work for large model but training speed become fast\n- [Medium Impact] SlidingWindowTransformerModel (SW)\n    - CV: - 0~0.001\n    - But bad combination with AWP.\n    - My implementation is similar with this [https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322](https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322)\n- [Medium Impact] Pseudo Labeling\n    - CV: - 0.0001~0.0006\n    - Our submission which got highest private score did not contain the model learning with pseudo-label.\n- [Medium Impact] High Frequency Validation and Saving\n    - CV: - 0.0008\n- [Medium Impact] Layer Normalization after Pooling Layer, before Fully Connected Layer (LayerNorm)\n    - CV: - 0~0.001\n    - Only work for base model\n- [Medium Impact] Multi sample dropout\n    - CV: +- 0.0\n    - The curve of validation loss becomes stable a little.\n- [Medium Impact] Optimize Ensembling Weights for Each Metrics Independently\n    - CV: - 0.0001 ~ 0.0004\n\n### What Didn’t Work\n\n- SiFT\n    - May be bugs in our code…\n- SVR\n    - No leakage but too low CV and bad public score. So, we didn't use SVR to ensemble.\n- Any Augmentations\n- Any Losses except L1 Smooth\n- Pretraining using Feedback1,2 Sentences\n    - I’m not sure this is really not to work…\n- Dealing with “Generic_Name” and others\n- Change the Way to Initialize Weights\n    - Parameters sampled from normal distribution is the best for us\n\n### Additional Context\n\n- Our Best Single model (in private and CV)\n    - Private: 0.4361, Public: 0.4375, CV: 0.4450\n    - This model was **Learning from Ensembled Pseudo-Labels.**\n    - But, our emsemble inference, which get highest private score, did not contains this model. If we leveraged this model to ensemble, we might get the more good standing.\n    - This idea may be useful especially for efficiency prize (in my opinion).\n\n### Important Citations\n\n- Y.Nakama’s baseline [https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n- AI Shift’s tech blog (Japanese) [https://www.ai-shift.co.jp/techblog/2170](https://www.ai-shift.co.jp/techblog/2170)\n- Discussion about Attention Pooling [https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678)\n- XGB+LGB Feedback Prize CV 0.7322 [https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322](https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322)\n\n### Thanks and Acknowledgements\nWe would like to thank the organizers of the competition for hosting this amazing event.\nI personally would like to mention and thank Taro Masuda ( @taromasuda ), Raiii ( @skraiii ) and MPEG ( @mpeg31 ) who did not participate in this competition, as I have learnt so much from them in the past competitions. I certainly was able to utilize the knowledge this time!\n\n### Team Members\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1655570%2Fb36ba8b5675df4b7e1b29ea114c0e8d6%2FTeamN.png?generation=1670066208813739&alt=media)\nFrom left in the above figure:\n- Masaki Aota @masakiaota\n- Olivine-Ryo @olivineryo \n- Yuichi Tateno @hotchpotch",
      "votes": 21
    },
    {
      "id": 2904745,
      "postDate": "2024-07-04T14:45:43.550Z",
      "content": "<p>How did u come up with exact values for learning rate decay</p>",
      "rawMarkdown": "How did u come up with exact values for learning rate decay\n"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:03:41Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9193491,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/team-n-15th-place-solution-6-kinds-of-attentionpoo"
}