{
  "id": 369903,
  "title": "Single 5 fold deberta v3 large to obtain silver medal",
  "content": "<p>Cv score of the model was 0.4447 and it scored 0.436405 in private leaderboard.<br>\nKey things were:</p>\n<ul>\n<li>Concat last 4 layers (it stabilized trained by a lot)</li>\n<li>Adding lstm on top</li>\n<li>AWP</li>\n</ul>\n<p>Here is training code:<br>\n<a href=\"https://www.kaggle.com/code/themadrambito/63th-place-deberta-v3-large-private-0-436405\" target=\"_blank\">https://www.kaggle.com/code/themadrambito/63th-place-deberta-v3-large-private-0-436405</a></p>",
  "comments": [
    {
      "id": 2052732,
      "author_name": "Ju7on9",
      "author_url": "",
      "post_date": "2022-12-02T12:59:57.030000",
      "content": "<p>Thank you for sharing.</p>\n<p>Is there any reason why did you not apply weight decay to [\"word_embeddings\", \"bias\", \"LayerNorm.weight\"]?</p>\n<p><code>no_wd = [\"word_embeddings\", \"bias\", \"LayerNorm.weight\"]</code></p>",
      "votes": 1,
      "replies": [
        {
          "id": 2052816,
          "author_name": "DarioAr",
          "author_url": "",
          "post_date": "2022-12-02T14:25:10.230000",
          "content": "<p>In theory it makes the model to converge faster, but it shouldnt matter at all. Weight decay only helps in parts where there are dot products.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2053198,
          "author_name": "Ju7on9",
          "author_url": "",
          "post_date": "2022-12-03T00:41:25.757000",
          "content": "<p>Thank you for answering.</p>\n<p>Would you elaborate more on why weight decay helps in parts where there are dot products? I am confused about the relationship between weight decay and dot products.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2054088,
          "author_name": "DarioAr",
          "author_url": "",
          "post_date": "2022-12-03T21:02:41.737000",
          "content": "<p>Overfitting may be caused by weights being too confident that some features of the input are more important than others. This can lead to weights ignoring most of the information in the input (weight decay pulls parameters towards 0 to reduce this).</p>\n<p>In the case of biases and embeddings you may want this feature or not. Maybe you want biases to grow very big, etc..<br>\nIn the case of layernorm weights, they are initiated to 1 to not affect the input, you dont want them to be 0.</p>\n<p>You can start reading about weight decay from here:<br>\n<a href=\"https://cs231n.github.io/\" target=\"_blank\">https://cs231n.github.io/</a></p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2052340,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-12-02T05:28:11.133000",
      "content": "<p>Thanks for sharing, your code worth to learn, I have tried lstm on top and concat last 4 layers but not achive improvment, amazing to see your model with such low cv and also pb.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2052591,
          "author_name": "DarioAr",
          "author_url": "",
          "post_date": "2022-12-02T10:21:49.513000",
          "content": "<p>Thank you!! Learning rate for deberta needed to be very low to make it work I think </p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 2052047,
      "postDate": "2022-12-01T21:10:48.107Z",
      "content": "<p>Cv score of the model was 0.4447 and it scored 0.436405 in private leaderboard.<br>\nKey things were:</p>\n<ul>\n<li>Concat last 4 layers (it stabilized trained by a lot)</li>\n<li>Adding lstm on top</li>\n<li>AWP</li>\n</ul>\n<p>Here is training code:<br>\n<a href=\"https://www.kaggle.com/code/themadrambito/63th-place-deberta-v3-large-private-0-436405\" target=\"_blank\">https://www.kaggle.com/code/themadrambito/63th-place-deberta-v3-large-private-0-436405</a></p>",
      "rawMarkdown": "Cv score of the model was 0.4447 and it scored 0.436405 in private leaderboard.\nKey things were:\n- Concat last 4 layers (it stabilized trained by a lot)\n- Adding lstm on top\n- AWP\n\nHere is training code:\nhttps://www.kaggle.com/code/themadrambito/63th-place-deberta-v3-large-private-0-436405",
      "votes": 16
    },
    {
      "id": 2052732,
      "postDate": "2022-12-02T12:59:57.030Z",
      "content": "<p>Thank you for sharing.</p>\n<p>Is there any reason why did you not apply weight decay to [\"word_embeddings\", \"bias\", \"LayerNorm.weight\"]?</p>\n<p><code>no_wd = [\"word_embeddings\", \"bias\", \"LayerNorm.weight\"]</code></p>",
      "rawMarkdown": "Thank you for sharing.\n\nIs there any reason why did you not apply weight decay to [\"word_embeddings\", \"bias\", \"LayerNorm.weight\"]?\n\n`no_wd = [\"word_embeddings\", \"bias\", \"LayerNorm.weight\"]`",
      "votes": 1,
      "replies": [
        {
          "id": 2052816,
          "postDate": "2022-12-02T14:25:10.230Z",
          "content": "<p>In theory it makes the model to converge faster, but it shouldnt matter at all. Weight decay only helps in parts where there are dot products.</p>",
          "rawMarkdown": "In theory it makes the model to converge faster, but it shouldnt matter at all. Weight decay only helps in parts where there are dot products.",
          "votes": 1
        },
        {
          "id": 2053198,
          "postDate": "2022-12-03T00:41:25.757Z",
          "content": "<p>Thank you for answering.</p>\n<p>Would you elaborate more on why weight decay helps in parts where there are dot products? I am confused about the relationship between weight decay and dot products.</p>",
          "rawMarkdown": "Thank you for answering.\n\nWould you elaborate more on why weight decay helps in parts where there are dot products? I am confused about the relationship between weight decay and dot products.",
          "votes": 1
        },
        {
          "id": 2054088,
          "postDate": "2022-12-03T21:02:41.737Z",
          "content": "<p>Overfitting may be caused by weights being too confident that some features of the input are more important than others. This can lead to weights ignoring most of the information in the input (weight decay pulls parameters towards 0 to reduce this).</p>\n<p>In the case of biases and embeddings you may want this feature or not. Maybe you want biases to grow very big, etc..<br>\nIn the case of layernorm weights, they are initiated to 1 to not affect the input, you dont want them to be 0.</p>\n<p>You can start reading about weight decay from here:<br>\n<a href=\"https://cs231n.github.io/\" target=\"_blank\">https://cs231n.github.io/</a></p>",
          "rawMarkdown": "Overfitting may be caused by weights being too confident that some features of the input are more important than others. This can lead to weights ignoring most of the information in the input (weight decay pulls parameters towards 0 to reduce this).\n\nIn the case of biases and embeddings you may want this feature or not. Maybe you want biases to grow very big, etc..\nIn the case of layernorm weights, they are initiated to 1 to not affect the input, you dont want them to be 0.\n\nYou can start reading about weight decay from here:\nhttps://cs231n.github.io/\n",
          "votes": 2
        }
      ]
    },
    {
      "id": 2052340,
      "postDate": "2022-12-02T05:28:11.133Z",
      "content": "<p>Thanks for sharing, your code worth to learn, I have tried lstm on top and concat last 4 layers but not achive improvment, amazing to see your model with such low cv and also pb.</p>",
      "rawMarkdown": "Thanks for sharing, your code worth to learn, I have tried lstm on top and concat last 4 layers but not achive improvment, amazing to see your model with such low cv and also pb.",
      "votes": 1,
      "replies": [
        {
          "id": 2052591,
          "postDate": "2022-12-02T10:21:49.513Z",
          "content": "<p>Thank you!! Learning rate for deberta needed to be very low to make it work I think </p>",
          "rawMarkdown": "Thank you!! Learning rate for deberta needed to be very low to make it work I think ",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:04:16Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9095897,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/darioar-single-5-fold-deberta-v3-large-to-obtain-s"
}