{
  "id": 369796,
  "title": "Lower value for LLRD (67th → 207th solution)",
  "content": "<p>Dear Kagglers.</p>\n<p>Thank you to the competition organizers for hosting this competition.<br>\nCongrats to all who won this game!</p>\n<p>We had a big shakedown from silver to bronze. (67th → 207th) :(<br>\nHowever, we learned a lot about NLP from the shared notebooks, discussions and top solutions. We had a very good experience.</p>\n<h2>Summary</h2>\n<p>Our solution is a simple ensemble of finetuned models and SVR with pre-trained embeddings(with several poolings and max_length patterns).</p>\n<ul>\n<li><p>finetuned models</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>roberta-large</li>\n<li>longformer-base-4096</li></ul></li>\n<li><p>pretrained models for SVR</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>deberta-v2-xlarge</li>\n<li>deberta-xlarge</li>\n<li>deberta-large-mnli</li>\n<li>roberta-large</li>\n<li>muppet-roberta-large</li>\n<li>longformer-large-4096</li>\n<li>bart-large</li>\n<li>electra-large-discriminator</li></ul></li>\n</ul>\n<h2>fine-tuning</h2>\n<p>While fine-tuning the single model, we noticed that a combination of very low LLRD <br>\n and high LR improves both CV and Public LB. For example, the following parameters got the scores.</p>\n<ul>\n<li>Model: deberta-v3-base</li>\n<li><b>LR: 16.0e-05<b></b></b></li>\n<li><b>LLRD: 0.1<b></b></b></li>\n<li>batch_size: 16</li>\n<li>other: AWP</li>\n<li>CV: 0.44912</li>\n<li>Public LB: 0.436664</li>\n<li>Private LB: 0.440466</li>\n</ul>\n<p>Has anyone tried parameters like these? Why would scores improve with these parameters? <br>\nIt is inexplicable to us. We will study and think about it some more.</p>\n<p>Thank you for your attention!</p>",
  "comments": [
    {
      "id": 2387224,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-08-12T14:04:04.720000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 2051514,
      "postDate": "2022-12-01T13:31:08.530Z",
      "content": "<p>Dear Kagglers.</p>\n<p>Thank you to the competition organizers for hosting this competition.<br>\nCongrats to all who won this game!</p>\n<p>We had a big shakedown from silver to bronze. (67th → 207th) :(<br>\nHowever, we learned a lot about NLP from the shared notebooks, discussions and top solutions. We had a very good experience.</p>\n<h2>Summary</h2>\n<p>Our solution is a simple ensemble of finetuned models and SVR with pre-trained embeddings(with several poolings and max_length patterns).</p>\n<ul>\n<li><p>finetuned models</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>roberta-large</li>\n<li>longformer-base-4096</li></ul></li>\n<li><p>pretrained models for SVR</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>deberta-v2-xlarge</li>\n<li>deberta-xlarge</li>\n<li>deberta-large-mnli</li>\n<li>roberta-large</li>\n<li>muppet-roberta-large</li>\n<li>longformer-large-4096</li>\n<li>bart-large</li>\n<li>electra-large-discriminator</li></ul></li>\n</ul>\n<h2>fine-tuning</h2>\n<p>While fine-tuning the single model, we noticed that a combination of very low LLRD <br>\n and high LR improves both CV and Public LB. For example, the following parameters got the scores.</p>\n<ul>\n<li>Model: deberta-v3-base</li>\n<li><b>LR: 16.0e-05<b></b></b></li>\n<li><b>LLRD: 0.1<b></b></b></li>\n<li>batch_size: 16</li>\n<li>other: AWP</li>\n<li>CV: 0.44912</li>\n<li>Public LB: 0.436664</li>\n<li>Private LB: 0.440466</li>\n</ul>\n<p>Has anyone tried parameters like these? Why would scores improve with these parameters? <br>\nIt is inexplicable to us. We will study and think about it some more.</p>\n<p>Thank you for your attention!</p>",
      "rawMarkdown": "Dear Kagglers.\n\nThank you to the competition organizers for hosting this competition.\nCongrats to all who won this game!\n\nWe had a big shakedown from silver to bronze. (67th → 207th) :(\nHowever, we learned a lot about NLP from the shared notebooks, discussions and top solutions. We had a very good experience.\n\n## Summary\nOur solution is a simple ensemble of finetuned models and SVR with pre-trained embeddings(with several poolings and max_length patterns).\n\n- finetuned models\n - deberta-v3-base\n - deberta-v3-large\n - roberta-large\n - longformer-base-4096\n\n- pretrained models for SVR\n - deberta-v3-base\n - deberta-v3-large\n - deberta-v2-xlarge\n - deberta-xlarge\n - deberta-large-mnli\n - roberta-large\n - muppet-roberta-large\n - longformer-large-4096\n - bart-large\n - electra-large-discriminator\n\n## fine-tuning\nWhile fine-tuning the single model, we noticed that a combination of very low LLRD \n and high LR improves both CV and Public LB. For example, the following parameters got the scores.\n- Model: deberta-v3-base\n- <b>LR: 16.0e-05<b>\n- <b>LLRD: 0.1<b>\n- batch_size: 16\n- other: AWP\n- CV: 0.44912\n- Public LB: 0.436664\n- Private LB: 0.440466\n\nHas anyone tried parameters like these? Why would scores improve with these parameters? \nIt is inexplicable to us. We will study and think about it some more.\n\nThank you for your attention!",
      "votes": 10
    },
    {
      "id": 2387224,
      "postDate": "2023-08-12T14:04:04.720Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:04:04Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9106660,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/taichicchi-lower-value-for-llrd-67th-207th-solutio"
}