{
  "id": 347381,
  "title": "15th | How to train deberta & NLP tricks",
  "content": "<p>Congrats to every team that fought to the last day!  Thanks my great teammates <a href=\"https://www.kaggle.com/khyeh0719\" target=\"_blank\">@khyeh0719</a>  <a href=\"https://www.kaggle.com/xbustc\" target=\"_blank\">@xbustc</a> ,  We tried a lot and have no regrets.</p>\n<h1>Model</h1>\n<p><strong>Formulation</strong></p>\n<p>We transformed the competition task into <strong>Token Classification</strong>.</p>\n<ol>\n<li>First,  combine discourse text in essay like this:  [CLS] {discourse_type1}:  {discourse_text1} [SEP] {discourse_type2}:  {discourse_text2} [SEP] ….</li>\n<li>Use Deberta encode the text</li>\n<li>Gather all discourse first token (the discourse_type token) for classification</li>\n</ol>\n<p>Tried four ways to do classification, </p>\n<ol>\n<li>First token</li>\n<li>First and End token mean (not work)</li>\n<li>Span + avg pooling (not work)</li>\n<li>Span + attention pooling (not work)</li>\n</ol>\n<p><strong>Model Architecture</strong></p>\n<ul>\n<li>Deberta-large</li>\n<li>Denerta-xlarge</li>\n<li>Deberta-v2-large</li>\n<li>Deberta-v3-large</li>\n</ul>\n<h1>How to train deberta</h1>\n<p>It's hard to train deberta models,  the model weight often broken when training,  there are some strategy we proposal:</p>\n<ol>\n<li>Smaller LR,  like 5e-6</li>\n<li>LR decay when layer go deeper,  we split layers to 6 groups and use 0.8 decay ratio</li>\n<li>Warmup LR scheduler</li>\n<li>AWP (Adversarial Weight Perturbation)</li>\n<li>Set dropout to ZERO,  just usefull for mlm-pretrained weight</li>\n</ol>\n<h1>Some trick give us big improve</h1>\n<ol>\n<li>MLM pretrained on FB2021 &amp; FB2022 can improve both stability and accuracy</li>\n<li>Set dropout to ZERO,  just usefull for mlm-pretrained weight</li>\n<li>Pseudo Label on FB2021 dataset,  and use soft-cross-entropy</li>\n</ol>\n<h1>Tried but not work</h1>\n<ol>\n<li>Meta Pseudo</li>\n<li>back translation augmentation</li>\n<li>RNN ensemble</li>\n<li>SWA (Stochastic Weight Averaging)</li>\n</ol>",
  "comments": [
    {
      "id": 1911295,
      "author_name": "xbustc",
      "author_url": "",
      "post_date": "2022-08-24T02:27:14.483000",
      "content": "<p>Thank you， KKY， it is a great pleasure to work with you and <a href=\"https://www.kaggle.com/khyeh0719\" target=\"_blank\">@khyeh0719</a> .</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1911323,
          "author_name": "khyeh",
          "author_url": "",
          "post_date": "2022-08-24T02:58:27.087000",
          "content": "<p>Great pleasure working with you too!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911821,
          "author_name": "KKY",
          "author_url": "",
          "post_date": "2022-08-24T10:35:04.060000",
          "content": "<p>Me too!  ❤️</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911221,
      "author_name": "Zacchaeus",
      "author_url": "",
      "post_date": "2022-08-24T01:22:44.287000",
      "content": "<p>despite some others say MLM didn't work, i had same observation with you that MLM stabilized training a lot. Thanks for sharing</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1911227,
          "author_name": "KKY",
          "author_url": "",
          "post_date": "2022-08-24T01:28:16.963000",
          "content": "<p>you are welcome,  congrats on achieving gold zone!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1939857,
      "author_name": "TianZijing",
      "author_url": "",
      "post_date": "2022-09-15T04:48:12.357000",
      "content": "<p>Congratulations！May I ask what lr, lr_scheduler and train_epoch did you use? And how to judge a pretrained model ? By train loss or other metrics?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1913410,
      "author_name": "The Devastator",
      "author_url": "",
      "post_date": "2022-08-25T10:09:02.193000",
      "content": "<p>Thank you for this post! Really useful for the future! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1911654,
      "author_name": "tomo",
      "author_url": "",
      "post_date": "2022-08-24T08:11:18.363000",
      "content": "<p>Thanks for sharing the solution!<br>\nDid psuedo-labeling use all 2021 data for training?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911816,
          "author_name": "KKY",
          "author_url": "",
          "post_date": "2022-08-24T10:32:10.087000",
          "content": "<p>Yes,  concat the pl data and the train data for training with 2 epochs.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1912888,
          "author_name": "tomo",
          "author_url": "",
          "post_date": "2022-08-25T02:47:44.167000",
          "content": "<p>I understand. Thanks for letting me know!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1915251,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-08-26T19:47:43.137000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 11,
  "messages": [
    {
      "id": 1911212,
      "postDate": "2022-08-24T01:17:10.450Z",
      "content": "<p>Congrats to every team that fought to the last day!  Thanks my great teammates <a href=\"https://www.kaggle.com/khyeh0719\" target=\"_blank\">@khyeh0719</a>  <a href=\"https://www.kaggle.com/xbustc\" target=\"_blank\">@xbustc</a> ,  We tried a lot and have no regrets.</p>\n<h1>Model</h1>\n<p><strong>Formulation</strong></p>\n<p>We transformed the competition task into <strong>Token Classification</strong>.</p>\n<ol>\n<li>First,  combine discourse text in essay like this:  [CLS] {discourse_type1}:  {discourse_text1} [SEP] {discourse_type2}:  {discourse_text2} [SEP] ….</li>\n<li>Use Deberta encode the text</li>\n<li>Gather all discourse first token (the discourse_type token) for classification</li>\n</ol>\n<p>Tried four ways to do classification, </p>\n<ol>\n<li>First token</li>\n<li>First and End token mean (not work)</li>\n<li>Span + avg pooling (not work)</li>\n<li>Span + attention pooling (not work)</li>\n</ol>\n<p><strong>Model Architecture</strong></p>\n<ul>\n<li>Deberta-large</li>\n<li>Denerta-xlarge</li>\n<li>Deberta-v2-large</li>\n<li>Deberta-v3-large</li>\n</ul>\n<h1>How to train deberta</h1>\n<p>It's hard to train deberta models,  the model weight often broken when training,  there are some strategy we proposal:</p>\n<ol>\n<li>Smaller LR,  like 5e-6</li>\n<li>LR decay when layer go deeper,  we split layers to 6 groups and use 0.8 decay ratio</li>\n<li>Warmup LR scheduler</li>\n<li>AWP (Adversarial Weight Perturbation)</li>\n<li>Set dropout to ZERO,  just usefull for mlm-pretrained weight</li>\n</ol>\n<h1>Some trick give us big improve</h1>\n<ol>\n<li>MLM pretrained on FB2021 &amp; FB2022 can improve both stability and accuracy</li>\n<li>Set dropout to ZERO,  just usefull for mlm-pretrained weight</li>\n<li>Pseudo Label on FB2021 dataset,  and use soft-cross-entropy</li>\n</ol>\n<h1>Tried but not work</h1>\n<ol>\n<li>Meta Pseudo</li>\n<li>back translation augmentation</li>\n<li>RNN ensemble</li>\n<li>SWA (Stochastic Weight Averaging)</li>\n</ol>",
      "rawMarkdown": "Congrats to every team that fought to the last day!  Thanks my great teammates @khyeh0719  @xbustc ,  We tried a lot and have no regrets.\n\n# Model\n\n**Formulation**\n\nWe transformed the competition task into **Token Classification**.\n\n1. First,  combine discourse text in essay like this:  [CLS] {discourse_type1}:  {discourse_text1} [SEP] {discourse_type2}:  {discourse_text2} [SEP] ....\n2. Use Deberta encode the text\n3. Gather all discourse first token (the discourse_type token) for classification\n\nTried four ways to do classification, \n1. First token\n2. First and End token mean (not work)\n3. Span + avg pooling (not work)\n4. Span + attention pooling (not work)\n\n**Model Architecture**\n- Deberta-large\n- Denerta-xlarge\n- Deberta-v2-large\n- Deberta-v3-large\n\n# How to train deberta\n\nIt's hard to train deberta models,  the model weight often broken when training,  there are some strategy we proposal:\n1. Smaller LR,  like 5e-6\n2. LR decay when layer go deeper,  we split layers to 6 groups and use 0.8 decay ratio\n3. Warmup LR scheduler\n4. AWP (Adversarial Weight Perturbation)\n6. Set dropout to ZERO,  just usefull for mlm-pretrained weight\n\n# Some trick give us big improve\n\n1. MLM pretrained on FB2021 & FB2022 can improve both stability and accuracy\n2. Set dropout to ZERO,  just usefull for mlm-pretrained weight\n3. Pseudo Label on FB2021 dataset,  and use soft-cross-entropy\n\n# Tried but not work\n1. Meta Pseudo\n2. back translation augmentation\n3. RNN ensemble\n4. SWA (Stochastic Weight Averaging)",
      "votes": 42
    },
    {
      "id": 1911295,
      "postDate": "2022-08-24T02:27:14.483Z",
      "content": "<p>Thank you， KKY， it is a great pleasure to work with you and <a href=\"https://www.kaggle.com/khyeh0719\" target=\"_blank\">@khyeh0719</a> .</p>",
      "rawMarkdown": "Thank you， KKY， it is a great pleasure to work with you and @khyeh0719 .",
      "votes": 3,
      "replies": [
        {
          "id": 1911323,
          "postDate": "2022-08-24T02:58:27.087Z",
          "content": "<p>Great pleasure working with you too!</p>",
          "rawMarkdown": "Great pleasure working with you too!",
          "votes": 1
        },
        {
          "id": 1911821,
          "postDate": "2022-08-24T10:35:04.060Z",
          "content": "<p>Me too!  ❤️</p>",
          "rawMarkdown": "Me too!  ❤️",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911221,
      "postDate": "2022-08-24T01:22:44.287Z",
      "content": "<p>despite some others say MLM didn't work, i had same observation with you that MLM stabilized training a lot. Thanks for sharing</p>",
      "rawMarkdown": "despite some others say MLM didn't work, i had same observation with you that MLM stabilized training a lot. Thanks for sharing",
      "votes": 2,
      "replies": [
        {
          "id": 1911227,
          "postDate": "2022-08-24T01:28:16.963Z",
          "content": "<p>you are welcome,  congrats on achieving gold zone!</p>",
          "rawMarkdown": "you are welcome,  congrats on achieving gold zone!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1939857,
      "postDate": "2022-09-15T04:48:12.357Z",
      "content": "<p>Congratulations！May I ask what lr, lr_scheduler and train_epoch did you use? And how to judge a pretrained model ? By train loss or other metrics?</p>",
      "rawMarkdown": "Congratulations！May I ask what lr, lr_scheduler and train_epoch did you use? And how to judge a pretrained model ? By train loss or other metrics?"
    },
    {
      "id": 1913410,
      "postDate": "2022-08-25T10:09:02.193Z",
      "content": "<p>Thank you for this post! Really useful for the future! </p>",
      "rawMarkdown": "Thank you for this post! Really useful for the future! "
    },
    {
      "id": 1911654,
      "postDate": "2022-08-24T08:11:18.363Z",
      "content": "<p>Thanks for sharing the solution!<br>\nDid psuedo-labeling use all 2021 data for training?</p>",
      "rawMarkdown": "Thanks for sharing the solution!\nDid psuedo-labeling use all 2021 data for training?",
      "replies": [
        {
          "id": 1911816,
          "postDate": "2022-08-24T10:32:10.087Z",
          "content": "<p>Yes,  concat the pl data and the train data for training with 2 epochs.</p>",
          "rawMarkdown": "Yes,  concat the pl data and the train data for training with 2 epochs."
        },
        {
          "id": 1912888,
          "postDate": "2022-08-25T02:47:44.167Z",
          "content": "<p>I understand. Thanks for letting me know!</p>",
          "rawMarkdown": "I understand. Thanks for letting me know!"
        }
      ]
    },
    {
      "id": 1915251,
      "postDate": "2022-08-26T19:47:43.137Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:53:06Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8870113,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/bkk-15th-how-to-train-deberta-nlp-tricks"
}