{
  "id": 446560,
  "title": "69th Place Solution ",
  "content": "<p>Firstly, thanks to Kaggle and the hosts for this interesting competition, this is the first competition that I ever commit to, and I'm glad that it paid off.</p>\n<p>As always, I knew that diversity wins, so I focused on building a diverse set of models. My final submission consisted of two LGBM models on top of Deberta-Base predictions and raw Deberta-Large predictions, with weights optimized on oofs. </p>\n<p><strong>LGBM Models</strong><br>\nThese two models are based on Feature Engineering ideas and model from <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> (thanks!). Each model was trained with a different set of features and hyperparameters, each scoring around 0.505 cv, 0.447 public and 0.480 private. A simple average between these two models scores 0.493 cv , 0.442 lb and a bronze zone finish. </p>\n<p><strong>Stabilizing Deberta-Large Training</strong><br>\nI knew that this would be key for diversity. However, at first, the training seemed unstable and the predictions not trustworthy, the model would converge at different epochs for different folds. So I spent the last week trying to stabilize the training pipeline and these are the key things that helped:</p>\n<ul>\n<li>Re-Initializing  last layers</li>\n<li>Attention Pooling seemed to work best</li>\n<li>Layer-wise Learning Rate Decay</li>\n<li>I've also experimented with AWP but with no success</li>\n</ul>\n<p>With this in mind, I managed to somewhat stabilize the training, I trained the final model for 4 epochs with 1 epoch for warmup and always picking the last epoch as the final checkpoint.</p>\n<p><strong>Final Ensemble</strong><br>\nThe final weights of the 3 models were optimized on oofs on each of the targets, I managed to get a small boost for the cv (0.489) but a slightly worse lb (0.445). But that performed the best on private and got me a 0.473 ( Trust your cv!). </p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2478763,
      "postDate": "2023-10-12T07:41:18.553Z",
      "content": "<p>Firstly, thanks to Kaggle and the hosts for this interesting competition, this is the first competition that I ever commit to, and I'm glad that it paid off.</p>\n<p>As always, I knew that diversity wins, so I focused on building a diverse set of models. My final submission consisted of two LGBM models on top of Deberta-Base predictions and raw Deberta-Large predictions, with weights optimized on oofs. </p>\n<p><strong>LGBM Models</strong><br>\nThese two models are based on Feature Engineering ideas and model from <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> (thanks!). Each model was trained with a different set of features and hyperparameters, each scoring around 0.505 cv, 0.447 public and 0.480 private. A simple average between these two models scores 0.493 cv , 0.442 lb and a bronze zone finish. </p>\n<p><strong>Stabilizing Deberta-Large Training</strong><br>\nI knew that this would be key for diversity. However, at first, the training seemed unstable and the predictions not trustworthy, the model would converge at different epochs for different folds. So I spent the last week trying to stabilize the training pipeline and these are the key things that helped:</p>\n<ul>\n<li>Re-Initializing  last layers</li>\n<li>Attention Pooling seemed to work best</li>\n<li>Layer-wise Learning Rate Decay</li>\n<li>I've also experimented with AWP but with no success</li>\n</ul>\n<p>With this in mind, I managed to somewhat stabilize the training, I trained the final model for 4 epochs with 1 epoch for warmup and always picking the last epoch as the final checkpoint.</p>\n<p><strong>Final Ensemble</strong><br>\nThe final weights of the 3 models were optimized on oofs on each of the targets, I managed to get a small boost for the cv (0.489) but a slightly worse lb (0.445). But that performed the best on private and got me a 0.473 ( Trust your cv!). </p>",
      "rawMarkdown": "Firstly, thanks to Kaggle and the hosts for this interesting competition, this is the first competition that I ever commit to, and I'm glad that it paid off.\n\nAs always, I knew that diversity wins, so I focused on building a diverse set of models. My final submission consisted of two LGBM models on top of Deberta-Base predictions and raw Deberta-Large predictions, with weights optimized on oofs. \n\n**LGBM Models**\nThese two models are based on Feature Engineering ideas and model from @tsunotsuno (thanks!). Each model was trained with a different set of features and hyperparameters, each scoring around 0.505 cv, 0.447 public and 0.480 private. A simple average between these two models scores 0.493 cv , 0.442 lb and a bronze zone finish. \n\n**Stabilizing Deberta-Large Training**\nI knew that this would be key for diversity. However, at first, the training seemed unstable and the predictions not trustworthy, the model would converge at different epochs for different folds. So I spent the last week trying to stabilize the training pipeline and these are the key things that helped:\n- Re-Initializing  last layers\n- Attention Pooling seemed to work best\n- Layer-wise Learning Rate Decay\n- I've also experimented with AWP but with no success\n\nWith this in mind, I managed to somewhat stabilize the training, I trained the final model for 4 epochs with 1 epoch for warmup and always picking the last epoch as the final checkpoint.\n\n**Final Ensemble**\nThe final weights of the 3 models were optimized on oofs on each of the targets, I managed to get a small boost for the cv (0.489) but a slightly worse lb (0.445). But that performed the best on private and got me a 0.473 ( Trust your cv!). ",
      "votes": 16
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:02:12Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10682037,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/catharis-69th-place-solution"
}