{
  "id": 347386,
  "title": "[11th solution] Feature design and Knowledge distillation",
  "content": "<p>First of all, thanks to the organizers to host this interesting competition and thanks to community for those great ideas which inspired me a lot. </p>\n<h2>Methods that work</h2>\n<ul>\n<li>Pretraining</li>\n<li>Feature design</li>\n<li>Pseudo labeling</li>\n<li>Knowledge distillation</li>\n</ul>\n<h2>Transformer Model</h2>\n<ul>\n<li>Deberta-v3-large</li>\n<li>Deberta-large</li>\n<li>Deberta-xlarge</li>\n</ul>\n<h2>Feature Design</h2>\n<p>1.Concatenate different <code>discourse_id</code> in same essay and predict token probabilities of tags:</p>\n<blockquote>\n  <p>[CLS LEAD] lead discourse [END LEAD] [CLS POSITION] position discourse [END POSITION] …</p>\n</blockquote>\n<p>2.Predict token probabilities of <code>[CLS]</code> and <code>[END]</code> tags in essay directly:</p>\n<blockquote>\n  <p>essay….  [CLS CLAIM] I think that the face is a natural landform because there is no life on Mars that we have descovered yet  [END CLAIM] essay …</p>\n</blockquote>\n<p>In my case, training model with whole essays works better than w/o essays like feature <code>1</code>, but it add up some diversity.</p>\n<p>3.Add classifier tags:</p>\n<blockquote>\n  <p>essay….  [CLS CLAIM] [Ineffective] [Adequate] [Effective] I think that the face is a natural landform because there is no life on Mars that we have descovered yet  [END CLAIM] essay …</p>\n</blockquote>\n<p>This time, the predicted probabilities are not <code>[CLS]</code> and <code>[END]</code>, but <code>[Ineffective]</code>, <code>[Adequate]</code> and <code>[Effective]</code>. And the activation function is using <code>Sigmoid</code> but not <code>Softmax</code>. Then the prob of these 3 tags would be scaling to 1. This could add diversity also.</p>\n<p>4.Predict all tokens in a span and take a mean value for discourse probabilities.</p>\n<h2>Pretraining</h2>\n<p>I use <a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/data\" target=\"_blank\">feedback-2021-data</a> for model pretraining, it could improve model performance about ~0.003.</p>\n<h2>Pseudo labeling</h2>\n<p>I spend a lot of time for pseudo labeling, especially when training model with pseudo labels (extra more data). <br>\nIn early stage I tried pseudo with hard label, but it did not give any improvement. And finally, soft label of probabilities give some  improvement in model performance (The loss function is using <code>soft cross entropy</code>).</p>\n<h2>Knowledge Distillation (KD)</h2>\n<p>In my case, KD works better than pseudo labeling when model ensembling. I found that the higher of distillation loss weight,  the better cv but worse lb. So I take the high distillation loss in early stage of training, but lower weight in later stages, this could improve both cv and lb.</p>\n<h2>Post process</h2>\n<p>I didn't have extra time to explore post process method, so I just used weighted average.</p>\n<p>Anyway, solo in this game is a little bit hard. There left many things I have not tried yet.</p>",
  "comments": [
    {
      "id": 1920227,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-08-31T02:33:54.657000",
      "content": "<p>Will you open source your Knowledge distillation code? i want to learn</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1920469,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-31T07:13:40.577000",
          "content": "<p><a href=\"https://www.kaggle.com/yasso1\" target=\"_blank\">@yasso1</a> you can refer to this post <a href=\"https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832\" target=\"_blank\">here</a></p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1911523,
      "author_name": "Raja Biswas",
      "author_url": "",
      "post_date": "2022-08-24T06:37:16.097000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/fankaixie\" target=\"_blank\">@fankaixie</a> on solo gold! amazing job 😊 </p>\n<p>Really innovative idea on including the classifier tags!</p>\n<p>We also had great CV score with distillation (deberta-large beating deberta-xlarge) but LB score was worse. Not sure why this happed though. but you have managed to overcome this by changing the distillation loss weight on the fly. Thanks for sharing! </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911546,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-24T07:04:03.043000",
          "content": "<p><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> Thanks! I also learn a lot from your great work! Congrates for your 3rd place and prize!</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1911723,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T09:05:36.643000",
          "content": "<p>likewise! 😊</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911349,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-08-24T03:25:30.157000",
      "content": "<p>Congrats on your solo gold! </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911466,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-24T05:54:46.933000",
          "content": "<p>Thanks! Congrates for your gold too~</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911257,
      "author_name": "Zacchaeus",
      "author_url": "",
      "post_date": "2022-08-24T01:57:12.310000",
      "content": "<p>Congrats on your solo gold~~ Can you explain what \"distillation loss weight\" refers to?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911266,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-24T01:59:06.643000",
          "content": "<p>Thanks! And congrate for your gold! <br>\n<code>L_total = L_ce + w * L_distillation</code>, and <code>w</code> is distillation loss weight</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1911256,
      "author_name": "Tifo",
      "author_url": "",
      "post_date": "2022-08-24T01:55:55.070000",
      "content": "<p><code>the higher of distillation loss weight, the better cv but worse lb</code> <br>\nThe article I found related <a href=\"https://arxiv.org/pdf/2102.00650.pdf\" target=\"_blank\">here</a>. Are there any other possible explanations?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911291,
          "author_name": "Leon",
          "author_url": "",
          "post_date": "2022-08-24T02:24:42.757000",
          "content": "<p>This makes sense. Loss weight plays a role in adjusting the balance between bias and variance.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1911299,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-24T02:31:31.527000",
          "content": "<p>It's hard to set the weight empirically 😂</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911832,
      "author_name": "KKY",
      "author_url": "",
      "post_date": "2022-08-24T10:41:04.463000",
      "content": "<p>Congrates for your SOLO GOLD.    🎉🎉🎉</p>\n<p>BTW ,  i think it's time to add knowlegage distillation into my weapony.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1912042,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-24T13:27:13.010000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/evilpsycho42\" target=\"_blank\">@evilpsycho42</a> ! It's a powerful method worth to try it!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 14,
  "messages": [
    {
      "id": 1911232,
      "postDate": "2022-08-24T01:38:13.960Z",
      "content": "<p>First of all, thanks to the organizers to host this interesting competition and thanks to community for those great ideas which inspired me a lot. </p>\n<h2>Methods that work</h2>\n<ul>\n<li>Pretraining</li>\n<li>Feature design</li>\n<li>Pseudo labeling</li>\n<li>Knowledge distillation</li>\n</ul>\n<h2>Transformer Model</h2>\n<ul>\n<li>Deberta-v3-large</li>\n<li>Deberta-large</li>\n<li>Deberta-xlarge</li>\n</ul>\n<h2>Feature Design</h2>\n<p>1.Concatenate different <code>discourse_id</code> in same essay and predict token probabilities of tags:</p>\n<blockquote>\n  <p>[CLS LEAD] lead discourse [END LEAD] [CLS POSITION] position discourse [END POSITION] …</p>\n</blockquote>\n<p>2.Predict token probabilities of <code>[CLS]</code> and <code>[END]</code> tags in essay directly:</p>\n<blockquote>\n  <p>essay….  [CLS CLAIM] I think that the face is a natural landform because there is no life on Mars that we have descovered yet  [END CLAIM] essay …</p>\n</blockquote>\n<p>In my case, training model with whole essays works better than w/o essays like feature <code>1</code>, but it add up some diversity.</p>\n<p>3.Add classifier tags:</p>\n<blockquote>\n  <p>essay….  [CLS CLAIM] [Ineffective] [Adequate] [Effective] I think that the face is a natural landform because there is no life on Mars that we have descovered yet  [END CLAIM] essay …</p>\n</blockquote>\n<p>This time, the predicted probabilities are not <code>[CLS]</code> and <code>[END]</code>, but <code>[Ineffective]</code>, <code>[Adequate]</code> and <code>[Effective]</code>. And the activation function is using <code>Sigmoid</code> but not <code>Softmax</code>. Then the prob of these 3 tags would be scaling to 1. This could add diversity also.</p>\n<p>4.Predict all tokens in a span and take a mean value for discourse probabilities.</p>\n<h2>Pretraining</h2>\n<p>I use <a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/data\" target=\"_blank\">feedback-2021-data</a> for model pretraining, it could improve model performance about ~0.003.</p>\n<h2>Pseudo labeling</h2>\n<p>I spend a lot of time for pseudo labeling, especially when training model with pseudo labels (extra more data). <br>\nIn early stage I tried pseudo with hard label, but it did not give any improvement. And finally, soft label of probabilities give some  improvement in model performance (The loss function is using <code>soft cross entropy</code>).</p>\n<h2>Knowledge Distillation (KD)</h2>\n<p>In my case, KD works better than pseudo labeling when model ensembling. I found that the higher of distillation loss weight,  the better cv but worse lb. So I take the high distillation loss in early stage of training, but lower weight in later stages, this could improve both cv and lb.</p>\n<h2>Post process</h2>\n<p>I didn't have extra time to explore post process method, so I just used weighted average.</p>\n<p>Anyway, solo in this game is a little bit hard. There left many things I have not tried yet.</p>",
      "rawMarkdown": "First of all, thanks to the organizers to host this interesting competition and thanks to community for those great ideas which inspired me a lot. \n\n## Methods that work\n\n- Pretraining\n- Feature design\n- Pseudo labeling\n- Knowledge distillation\n\n## Transformer Model\n\n- Deberta-v3-large\n- Deberta-large\n- Deberta-xlarge\n\n## Feature Design\n\n1.Concatenate different `discourse_id` in same essay and predict token probabilities of tags:\n\n> [CLS LEAD] lead discourse [END LEAD] [CLS POSITION] position discourse [END POSITION] ...\n\n2.Predict token probabilities of `[CLS]` and `[END]` tags in essay directly:\n\n> essay....  [CLS CLAIM] I think that the face is a natural landform because there is no life on Mars that we have descovered yet  [END CLAIM] essay ...\n\nIn my case, training model with whole essays works better than w/o essays like feature `1`, but it add up some diversity.\n\n3.Add classifier tags:\n\n> essay....  [CLS CLAIM] [Ineffective] [Adequate] [Effective] I think that the face is a natural landform because there is no life on Mars that we have descovered yet  [END CLAIM] essay ...\n\nThis time, the predicted probabilities are not `[CLS]` and `[END]`, but `[Ineffective]`, `[Adequate]` and `[Effective]`. And the activation function is using `Sigmoid` but not `Softmax`. Then the prob of these 3 tags would be scaling to 1. This could add diversity also.\n\n 4.Predict all tokens in a span and take a mean value for discourse probabilities.\n\n## Pretraining\n\nI use [feedback-2021-data](https://www.kaggle.com/competitions/feedback-prize-2021/data) for model pretraining, it could improve model performance about ~0.003.\n\n## Pseudo labeling\n\nI spend a lot of time for pseudo labeling, especially when training model with pseudo labels (extra more data). \nIn early stage I tried pseudo with hard label, but it did not give any improvement. And finally, soft label of probabilities give some  improvement in model performance (The loss function is using `soft cross entropy`).\n\n## Knowledge Distillation (KD)\n\nIn my case, KD works better than pseudo labeling when model ensembling. I found that the higher of distillation loss weight,  the better cv but worse lb. So I take the high distillation loss in early stage of training, but lower weight in later stages, this could improve both cv and lb.\n\n## Post process\nI didn't have extra time to explore post process method, so I just used weighted average.\n\nAnyway, solo in this game is a little bit hard. There left many things I have not tried yet.",
      "votes": 37
    },
    {
      "id": 1920227,
      "postDate": "2022-08-31T02:33:54.657Z",
      "content": "<p>Will you open source your Knowledge distillation code? i want to learn</p>",
      "rawMarkdown": "Will you open source your Knowledge distillation code? i want to learn",
      "votes": 1,
      "replies": [
        {
          "id": 1920469,
          "postDate": "2022-08-31T07:13:40.577Z",
          "content": "<p><a href=\"https://www.kaggle.com/yasso1\" target=\"_blank\">@yasso1</a> you can refer to this post <a href=\"https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832\" target=\"_blank\">here</a></p>",
          "rawMarkdown": "@yasso1 you can refer to this post [here](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/322832)",
          "votes": 2
        }
      ]
    },
    {
      "id": 1911523,
      "postDate": "2022-08-24T06:37:16.097Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/fankaixie\" target=\"_blank\">@fankaixie</a> on solo gold! amazing job 😊 </p>\n<p>Really innovative idea on including the classifier tags!</p>\n<p>We also had great CV score with distillation (deberta-large beating deberta-xlarge) but LB score was worse. Not sure why this happed though. but you have managed to overcome this by changing the distillation loss weight on the fly. Thanks for sharing! </p>",
      "rawMarkdown": "Congratulations @fankaixie on solo gold! amazing job 😊 \n\nReally innovative idea on including the classifier tags!\n\nWe also had great CV score with distillation (deberta-large beating deberta-xlarge) but LB score was worse. Not sure why this happed though. but you have managed to overcome this by changing the distillation loss weight on the fly. Thanks for sharing! ",
      "votes": 1,
      "replies": [
        {
          "id": 1911546,
          "postDate": "2022-08-24T07:04:03.043Z",
          "content": "<p><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> Thanks! I also learn a lot from your great work! Congrates for your 3rd place and prize!</p>",
          "rawMarkdown": "@conjuring92 Thanks! I also learn a lot from your great work! Congrates for your 3rd place and prize!",
          "votes": 2
        },
        {
          "id": 1911723,
          "postDate": "2022-08-24T09:05:36.643Z",
          "content": "<p>likewise! 😊</p>",
          "rawMarkdown": "likewise! 😊"
        }
      ]
    },
    {
      "id": 1911349,
      "postDate": "2022-08-24T03:25:30.157Z",
      "content": "<p>Congrats on your solo gold! </p>",
      "rawMarkdown": "Congrats on your solo gold! ",
      "votes": 1,
      "replies": [
        {
          "id": 1911466,
          "postDate": "2022-08-24T05:54:46.933Z",
          "content": "<p>Thanks! Congrates for your gold too~</p>",
          "rawMarkdown": "Thanks! Congrates for your gold too~"
        }
      ]
    },
    {
      "id": 1911257,
      "postDate": "2022-08-24T01:57:12.310Z",
      "content": "<p>Congrats on your solo gold~~ Can you explain what \"distillation loss weight\" refers to?</p>",
      "rawMarkdown": "Congrats on your solo gold~~ Can you explain what \"distillation loss weight\" refers to?",
      "votes": 1,
      "replies": [
        {
          "id": 1911266,
          "postDate": "2022-08-24T01:59:06.643Z",
          "content": "<p>Thanks! And congrate for your gold! <br>\n<code>L_total = L_ce + w * L_distillation</code>, and <code>w</code> is distillation loss weight</p>",
          "rawMarkdown": "Thanks! And congrate for your gold! \n`L_total = L_ce + w * L_distillation`, and `w` is distillation loss weight",
          "votes": 2
        }
      ]
    },
    {
      "id": 1911256,
      "postDate": "2022-08-24T01:55:55.070Z",
      "content": "<p><code>the higher of distillation loss weight, the better cv but worse lb</code> <br>\nThe article I found related <a href=\"https://arxiv.org/pdf/2102.00650.pdf\" target=\"_blank\">here</a>. Are there any other possible explanations?</p>",
      "rawMarkdown": "`the higher of distillation loss weight, the better cv but worse lb` \nThe article I found related [here](https://arxiv.org/pdf/2102.00650.pdf). Are there any other possible explanations?",
      "votes": 1,
      "replies": [
        {
          "id": 1911291,
          "postDate": "2022-08-24T02:24:42.757Z",
          "content": "<p>This makes sense. Loss weight plays a role in adjusting the balance between bias and variance.</p>",
          "rawMarkdown": "This makes sense. Loss weight plays a role in adjusting the balance between bias and variance.",
          "votes": 3
        },
        {
          "id": 1911299,
          "postDate": "2022-08-24T02:31:31.527Z",
          "content": "<p>It's hard to set the weight empirically 😂</p>",
          "rawMarkdown": "It's hard to set the weight empirically 😂",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911832,
      "postDate": "2022-08-24T10:41:04.463Z",
      "content": "<p>Congrates for your SOLO GOLD.    🎉🎉🎉</p>\n<p>BTW ,  i think it's time to add knowlegage distillation into my weapony.</p>",
      "rawMarkdown": "Congrates for your SOLO GOLD.    🎉🎉🎉\n\nBTW ,  i think it's time to add knowlegage distillation into my weapony.",
      "votes": 2,
      "replies": [
        {
          "id": 1912042,
          "postDate": "2022-08-24T13:27:13.010Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/evilpsycho42\" target=\"_blank\">@evilpsycho42</a> ! It's a powerful method worth to try it!</p>",
          "rawMarkdown": "Thanks @evilpsycho42 ! It's a powerful method worth to try it!"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:52:44Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8673559,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/tifo-11th-solution-feature-design-and-knowledge-di"
}