{
  "id": 347490,
  "title": "12th place solution",
  "content": "<p>I would like to thank all the participants and hosts of the competition.<br>\nAnd thank you to the teams for working hard together ( <a href=\"https://www.kaggle.com/tomoyayanagi\" target=\"_blank\">@tomoyayanagi</a>, <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a>,  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a>, <a href=\"https://www.kaggle.com/shairahama\" target=\"_blank\">@shairahama</a> )<br>\nWithout team I would not be in this position.</p>\n<h1>Model</h1>\n<ul>\n<li>Deberta-large</li>\n<li>Denerta-xlarge</li>\n<li>Deberta-v3-large</li>\n<li>bigbird-roberta-large</li>\n</ul>\n<h2>Token Classification</h2>\n<p>All models were built with token classification.<br>\nToken classification added a [CLS] token at the beginning of the discourse_text and a [SEP] token at the end.<br>\nThe method of pooling was different in members, cls token, average pooling was used<br>\nThe best input was to introduce special tokens (start, end) at the beginning and end (introduced by  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a>)</p>\n<pre><code>&lt;{discourse_type} start&gt; discourse text &lt;{discourse_type} end&gt;\n</code></pre>\n<h2>Adding meta description (Prompt)</h2>\n<p>We found a trend of labels for each essay topic. <br>\nBased on these results, we added topic text to the top of the input.<br>\nThis method gives us the best single model (build by  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a> )</p>\n<p>Topic information was taken from the public notebook. Thank you very much.<br>\n(<a href=\"https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts\" target=\"_blank\">https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts</a>)</p>\n<pre><code>topic prompt [SEP] &lt;{discourse_type} start&gt; discourse text &lt;{discourse_type} end&gt;, ...\n</code></pre>\n<p>I think there was a discussion that the host posted that the topic of the essay is common. Therefore, I think it worked.</p>\n<h1>Pseudo Labeling</h1>\n<p>At first, we used all of the 2021 data as pseudo-labels, but the training did not proceed well, probably because the distribution of the data was too different. Therefore, we adopted the method of sampling and using a portion of the data.</p>\n<p>so, the following methods are good for us (This was introduced by <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> )</p>\n<ul>\n<li>Discard essays with high Adequate predictions in all the discourse_texts that make up the essay</li>\n<li>Get everything that contains Ineffective's predictions</li>\n<li>Sampling the rest of the essay including Effective at 25~35%</li>\n</ul>\n<p>Pseudo labels matched folds trained base model to minimize leakage</p>\n<p>Another consideration was to sample the 2021 data according to topic, since the distribution of topic is different in 2022, but there was not enough time</p>\n<h1>Stacking</h1>\n<p>We have built a 2nd model with flattened 1st model predictions of the 3 labels as features.<br>\nDiscourse_type is added as a meta-feature.<br>\nThe model used logistic regression, lgbm and xgb.<br>\nBoosted cv and lb significantly more than weighted average using Nelder-Mead based ensemble</p>\n<h1>What else worked or tips</h1>\n<ul>\n<li>MLM<ul>\n<li>Smoothed loss and boosted cv/lb</li></ul></li>\n<li>AWP<ul>\n<li>It seems to be good to start at epoch 1 or 2</li></ul></li>\n<li>FGM</li>\n<li>Evaluate more validation in steps instead of epochs</li>\n<li>layer wise learning rate (decay rate 0.98)<ul>\n<li>Higher learning rate at transformer classification head</li></ul></li>\n</ul>",
  "comments": [
    {
      "id": 1913026,
      "author_name": "tomo",
      "author_url": "",
      "post_date": "2022-08-25T05:19:33.933000",
      "content": "<p>The stacking, which greatly improved our score, was done by ktr! Thank you!</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1912214,
      "author_name": "Zacchaeus",
      "author_url": "",
      "post_date": "2022-08-24T15:34:20.520000",
      "content": "<p>👏great team! Congrats on becoming a master </p>",
      "votes": 3,
      "replies": [
        {
          "id": 1912855,
          "author_name": "ktr",
          "author_url": "",
          "post_date": "2022-08-25T02:18:08.510000",
          "content": "<p>Thank you very much.😊<br>\nWe have learned so much from you.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 1912428,
      "author_name": "Sumir Sewlall",
      "author_url": "",
      "post_date": "2022-08-24T18:04:29.127000",
      "content": "<p>Well done!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1919430,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-08-30T11:59:32.703000",
      "content": "<p>I have tried FGM but it doesn't work online, I have tried AWP but I have NAN values in the late gradient after training. can you help me</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1919640,
          "author_name": "ktr",
          "author_url": "",
          "post_date": "2022-08-30T15:13:13.677000",
          "content": "<p>thanks for your comment.</p>\n<p>In my experiment, nan did not happen with awp.<br>\nI think they are sensitive to parameters, so it would be good to try several hyper parameters.<br>\ncode was referenced in this notebook (pppm 8th place).<br>\n<a href=\"https://www.kaggle.com/code/skraiii/pppm-tokenclassificationmodel-train-8th-place\" target=\"_blank\">https://www.kaggle.com/code/skraiii/pppm-tokenclassificationmodel-train-8th-place</a></p>\n<p>FGM was performed by teammates. I expect my teammates to respond.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1919653,
          "author_name": "tomo",
          "author_url": "",
          "post_date": "2022-08-30T15:22:14.630000",
          "content": "<p>Thank you for your comment.<br>\nI was using FGM, but the traning did not work on deberta-v3-large unless I used small eps. Therefore, I think it is a good idea to make the parameters a little smaller.<br>\n（Larger parameters worked for deberta-large）</p>\n<p>Code was based on this.<br>\n<a href=\"https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/143764\" target=\"_blank\">https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/143764</a></p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1920256,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-08-31T03:30:06.270000",
          "content": "<p>What parameters did you choose?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1920257,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-08-31T03:32:12.333000",
          "content": "<p>I used AWP on a text classification model and nan came up. Haven't tried it yet on the token classification model. So far I still don't know why</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 9,
  "messages": [
    {
      "id": 1911857,
      "postDate": "2022-08-24T11:05:52.037Z",
      "content": "<p>I would like to thank all the participants and hosts of the competition.<br>\nAnd thank you to the teams for working hard together ( <a href=\"https://www.kaggle.com/tomoyayanagi\" target=\"_blank\">@tomoyayanagi</a>, <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a>,  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a>, <a href=\"https://www.kaggle.com/shairahama\" target=\"_blank\">@shairahama</a> )<br>\nWithout team I would not be in this position.</p>\n<h1>Model</h1>\n<ul>\n<li>Deberta-large</li>\n<li>Denerta-xlarge</li>\n<li>Deberta-v3-large</li>\n<li>bigbird-roberta-large</li>\n</ul>\n<h2>Token Classification</h2>\n<p>All models were built with token classification.<br>\nToken classification added a [CLS] token at the beginning of the discourse_text and a [SEP] token at the end.<br>\nThe method of pooling was different in members, cls token, average pooling was used<br>\nThe best input was to introduce special tokens (start, end) at the beginning and end (introduced by  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a>)</p>\n<pre><code>&lt;{discourse_type} start&gt; discourse text &lt;{discourse_type} end&gt;\n</code></pre>\n<h2>Adding meta description (Prompt)</h2>\n<p>We found a trend of labels for each essay topic. <br>\nBased on these results, we added topic text to the top of the input.<br>\nThis method gives us the best single model (build by  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a> )</p>\n<p>Topic information was taken from the public notebook. Thank you very much.<br>\n(<a href=\"https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts\" target=\"_blank\">https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts</a>)</p>\n<pre><code>topic prompt [SEP] &lt;{discourse_type} start&gt; discourse text &lt;{discourse_type} end&gt;, ...\n</code></pre>\n<p>I think there was a discussion that the host posted that the topic of the essay is common. Therefore, I think it worked.</p>\n<h1>Pseudo Labeling</h1>\n<p>At first, we used all of the 2021 data as pseudo-labels, but the training did not proceed well, probably because the distribution of the data was too different. Therefore, we adopted the method of sampling and using a portion of the data.</p>\n<p>so, the following methods are good for us (This was introduced by <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> )</p>\n<ul>\n<li>Discard essays with high Adequate predictions in all the discourse_texts that make up the essay</li>\n<li>Get everything that contains Ineffective's predictions</li>\n<li>Sampling the rest of the essay including Effective at 25~35%</li>\n</ul>\n<p>Pseudo labels matched folds trained base model to minimize leakage</p>\n<p>Another consideration was to sample the 2021 data according to topic, since the distribution of topic is different in 2022, but there was not enough time</p>\n<h1>Stacking</h1>\n<p>We have built a 2nd model with flattened 1st model predictions of the 3 labels as features.<br>\nDiscourse_type is added as a meta-feature.<br>\nThe model used logistic regression, lgbm and xgb.<br>\nBoosted cv and lb significantly more than weighted average using Nelder-Mead based ensemble</p>\n<h1>What else worked or tips</h1>\n<ul>\n<li>MLM<ul>\n<li>Smoothed loss and boosted cv/lb</li></ul></li>\n<li>AWP<ul>\n<li>It seems to be good to start at epoch 1 or 2</li></ul></li>\n<li>FGM</li>\n<li>Evaluate more validation in steps instead of epochs</li>\n<li>layer wise learning rate (decay rate 0.98)<ul>\n<li>Higher learning rate at transformer classification head</li></ul></li>\n</ul>",
      "rawMarkdown": "I would like to thank all the participants and hosts of the competition.\nAnd thank you to the teams for working hard together ( @tomoyayanagi, @kashiwaba,  @zacchaeus, @shairahama )\nWithout team I would not be in this position.\n\n# Model\n- Deberta-large\n- Denerta-xlarge\n- Deberta-v3-large\n- bigbird-roberta-large\n\n## Token Classification\nAll models were built with token classification.\nToken classification added a [CLS] token at the beginning of the discourse_text and a [SEP] token at the end.\nThe method of pooling was different in members, cls token, average pooling was used\nThe best input was to introduce special tokens (start, end) at the beginning and end (introduced by  @zacchaeus)\n\n```\n<{discourse_type} start> discourse text <{discourse_type} end>\n```\n\n## Adding meta description (Prompt)\nWe found a trend of labels for each essay topic. \nBased on these results, we added topic text to the top of the input.\nThis method gives us the best single model (build by  @zacchaeus )\n\nTopic information was taken from the public notebook. Thank you very much.\n(https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts)\n\n```\ntopic prompt [SEP] <{discourse_type} start> discourse text <{discourse_type} end>, ...\n```\nI think there was a discussion that the host posted that the topic of the essay is common. Therefore, I think it worked.\n\n# Pseudo Labeling\nAt first, we used all of the 2021 data as pseudo-labels, but the training did not proceed well, probably because the distribution of the data was too different. Therefore, we adopted the method of sampling and using a portion of the data.\n\nso, the following methods are good for us (This was introduced by @kashiwaba )\n\n- Discard essays with high Adequate predictions in all the discourse_texts that make up the essay\n- Get everything that contains Ineffective's predictions\n- Sampling the rest of the essay including Effective at 25~35%\n\nPseudo labels matched folds trained base model to minimize leakage\n\nAnother consideration was to sample the 2021 data according to topic, since the distribution of topic is different in 2022, but there was not enough time\n\n# Stacking\nWe have built a 2nd model with flattened 1st model predictions of the 3 labels as features.\nDiscourse_type is added as a meta-feature.\nThe model used logistic regression, lgbm and xgb.\nBoosted cv and lb significantly more than weighted average using Nelder-Mead based ensemble\n\n# What else worked or tips\n- MLM\n - Smoothed loss and boosted cv/lb\n- AWP\n    - It seems to be good to start at epoch 1 or 2\n- FGM\n- Evaluate more validation in steps instead of epochs\n- layer wise learning rate (decay rate 0.98)\n - Higher learning rate at transformer classification head",
      "votes": 32
    },
    {
      "id": 1913026,
      "postDate": "2022-08-25T05:19:33.933Z",
      "content": "<p>The stacking, which greatly improved our score, was done by ktr! Thank you!</p>",
      "rawMarkdown": "The stacking, which greatly improved our score, was done by ktr! Thank you!",
      "votes": 3
    },
    {
      "id": 1912214,
      "postDate": "2022-08-24T15:34:20.520Z",
      "content": "<p>👏great team! Congrats on becoming a master </p>",
      "rawMarkdown": "👏great team! Congrats on becoming a master ",
      "votes": 3,
      "replies": [
        {
          "id": 1912855,
          "postDate": "2022-08-25T02:18:08.510Z",
          "content": "<p>Thank you very much.😊<br>\nWe have learned so much from you.</p>",
          "rawMarkdown": "Thank you very much.😊\nWe have learned so much from you.",
          "votes": 3
        }
      ]
    },
    {
      "id": 1912428,
      "postDate": "2022-08-24T18:04:29.127Z",
      "content": "<p>Well done!</p>",
      "rawMarkdown": "Well done!",
      "votes": 2
    },
    {
      "id": 1919430,
      "postDate": "2022-08-30T11:59:32.703Z",
      "content": "<p>I have tried FGM but it doesn't work online, I have tried AWP but I have NAN values in the late gradient after training. can you help me</p>",
      "rawMarkdown": "I have tried FGM but it doesn't work online, I have tried AWP but I have NAN values in the late gradient after training. can you help me",
      "replies": [
        {
          "id": 1919640,
          "postDate": "2022-08-30T15:13:13.677Z",
          "content": "<p>thanks for your comment.</p>\n<p>In my experiment, nan did not happen with awp.<br>\nI think they are sensitive to parameters, so it would be good to try several hyper parameters.<br>\ncode was referenced in this notebook (pppm 8th place).<br>\n<a href=\"https://www.kaggle.com/code/skraiii/pppm-tokenclassificationmodel-train-8th-place\" target=\"_blank\">https://www.kaggle.com/code/skraiii/pppm-tokenclassificationmodel-train-8th-place</a></p>\n<p>FGM was performed by teammates. I expect my teammates to respond.</p>",
          "rawMarkdown": "thanks for your comment.\n\nIn my experiment, nan did not happen with awp.\nI think they are sensitive to parameters, so it would be good to try several hyper parameters.\ncode was referenced in this notebook (pppm 8th place).\nhttps://www.kaggle.com/code/skraiii/pppm-tokenclassificationmodel-train-8th-place\n\nFGM was performed by teammates. I expect my teammates to respond.",
          "votes": 2
        },
        {
          "id": 1919653,
          "postDate": "2022-08-30T15:22:14.630Z",
          "content": "<p>Thank you for your comment.<br>\nI was using FGM, but the traning did not work on deberta-v3-large unless I used small eps. Therefore, I think it is a good idea to make the parameters a little smaller.<br>\n（Larger parameters worked for deberta-large）</p>\n<p>Code was based on this.<br>\n<a href=\"https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/143764\" target=\"_blank\">https://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/143764</a></p>",
          "rawMarkdown": "Thank you for your comment.\nI was using FGM, but the traning did not work on deberta-v3-large unless I used small eps. Therefore, I think it is a good idea to make the parameters a little smaller.\n（Larger parameters worked for deberta-large）\n\nCode was based on this.\nhttps://www.kaggle.com/competitions/tweet-sentiment-extraction/discussion/143764",
          "votes": 2
        },
        {
          "id": 1920256,
          "postDate": "2022-08-31T03:30:06.270Z",
          "content": "<p>What parameters did you choose?</p>",
          "rawMarkdown": "What parameters did you choose?"
        },
        {
          "id": 1920257,
          "postDate": "2022-08-31T03:32:12.333Z",
          "content": "<p>I used AWP on a text classification model and nan came up. Haven't tried it yet on the token classification model. So far I still don't know why</p>",
          "rawMarkdown": "I used AWP on a text classification model and nan came up. Haven't tried it yet on the token classification model. So far I still don't know why"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:53:12Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8770463,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/team-animals-12th-place-solution"
}