{
  "id": 347373,
  "title": "80th Place Solution (Single Model)",
  "content": "<p><strong>First of all, thanks to the competition organizers for a great competition, and all the people who published great notebooks, which everyone was able to learn from. Additionally, <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> <a href=\"https://www.kaggle.com/code/nbroad/token-classification-approach-fpe/notebook\" target=\"_blank\">notebook</a> was a gamechanger, because it not only improved results drastically, but also cut down runtime ten fold.</strong></p>\n<p>My final solution only used a single model, as I didn't have enough time to ensemble multiple models. The model I used was of course, deberta large, as it consistently produced the best results (cv and lb). </p>\n<h1>Training Process:</h1>\n<p>As mentioned above, I used the token classification approach proposed by <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> as it gave a major boost in cv, and lb. Additionally, I changed the pooler to a weighted layer pooler, which seems to give the best results across many competitions/tasks. These changes helped me break 0.600 on the public leaderboard. To get the model down to 0.589 (0.590 private lb), I added an auxiliary task which stabilized training, and also of course improved cv and lb. The auxiliary task was another classification task, where the model predicted the type of discourse, in addition to it's effectiveness. For example, given an essay, the model would predict what each discourse was, and it's effectiveness. </p>\n<h2>Things that worked:</h2>\n<p>token classification<br>\nmulti-task learning<br>\nweighted pooler<br>\nincreased epochs for scheduler</p>\n<h2>Things that didn't work:</h2>\n<p>mlm <br>\npsuedo labeling (I had a feeling that the winners would be those who could figure out how to get this to work, but I never got convincing results in my experiments)<br>\ncontrastive learning<br>\nusing the last competitions task as an auxiliary task</p>\n<h3>Conclusion</h3>\n<p>I was very happy to medal in this competition, as this is my first competition medal. I am very curious to understand how the top solutions utilized the extra unlabeled data.<br>\nTraining code: <a href=\"https://www.kaggle.com/code/ryanbarretto/debertalargeauxtraining\" target=\"_blank\">Training Notebook</a> </p>",
  "comments": [
    {
      "id": 1911468,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-08-24T05:55:38.530000",
      "content": "<p>multi-task learning</p>\n<p>How much benefit did multi-task learning give you online? I found that adding multi-task learning to my model seems to have no effect on the final score.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911922,
          "author_name": "Ryan Barretto",
          "author_url": "",
          "post_date": "2022-08-24T11:48:03.297000",
          "content": "<p>For my private and public lb, it gave me a boost of around 0.01, which was pretty significant. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 1911184,
      "postDate": "2022-08-24T00:54:55.147Z",
      "content": "<p><strong>First of all, thanks to the competition organizers for a great competition, and all the people who published great notebooks, which everyone was able to learn from. Additionally, <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> <a href=\"https://www.kaggle.com/code/nbroad/token-classification-approach-fpe/notebook\" target=\"_blank\">notebook</a> was a gamechanger, because it not only improved results drastically, but also cut down runtime ten fold.</strong></p>\n<p>My final solution only used a single model, as I didn't have enough time to ensemble multiple models. The model I used was of course, deberta large, as it consistently produced the best results (cv and lb). </p>\n<h1>Training Process:</h1>\n<p>As mentioned above, I used the token classification approach proposed by <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> as it gave a major boost in cv, and lb. Additionally, I changed the pooler to a weighted layer pooler, which seems to give the best results across many competitions/tasks. These changes helped me break 0.600 on the public leaderboard. To get the model down to 0.589 (0.590 private lb), I added an auxiliary task which stabilized training, and also of course improved cv and lb. The auxiliary task was another classification task, where the model predicted the type of discourse, in addition to it's effectiveness. For example, given an essay, the model would predict what each discourse was, and it's effectiveness. </p>\n<h2>Things that worked:</h2>\n<p>token classification<br>\nmulti-task learning<br>\nweighted pooler<br>\nincreased epochs for scheduler</p>\n<h2>Things that didn't work:</h2>\n<p>mlm <br>\npsuedo labeling (I had a feeling that the winners would be those who could figure out how to get this to work, but I never got convincing results in my experiments)<br>\ncontrastive learning<br>\nusing the last competitions task as an auxiliary task</p>\n<h3>Conclusion</h3>\n<p>I was very happy to medal in this competition, as this is my first competition medal. I am very curious to understand how the top solutions utilized the extra unlabeled data.<br>\nTraining code: <a href=\"https://www.kaggle.com/code/ryanbarretto/debertalargeauxtraining\" target=\"_blank\">Training Notebook</a> </p>",
      "rawMarkdown": "**First of all, thanks to the competition organizers for a great competition, and all the people who published great notebooks, which everyone was able to learn from. Additionally, @nbroad [notebook](https://www.kaggle.com/code/nbroad/token-classification-approach-fpe/notebook) was a gamechanger, because it not only improved results drastically, but also cut down runtime ten fold.**\n\nMy final solution only used a single model, as I didn't have enough time to ensemble multiple models. The model I used was of course, deberta large, as it consistently produced the best results (cv and lb). \n\n\n\n# Training Process:\nAs mentioned above, I used the token classification approach proposed by @nbroad as it gave a major boost in cv, and lb. Additionally, I changed the pooler to a weighted layer pooler, which seems to give the best results across many competitions/tasks. These changes helped me break 0.600 on the public leaderboard. To get the model down to 0.589 (0.590 private lb), I added an auxiliary task which stabilized training, and also of course improved cv and lb. The auxiliary task was another classification task, where the model predicted the type of discourse, in addition to it's effectiveness. For example, given an essay, the model would predict what each discourse was, and it's effectiveness. \n\n## Things that worked:\n\ntoken classification\nmulti-task learning\nweighted pooler\nincreased epochs for scheduler\n\n## Things that didn't work:\n\nmlm \npsuedo labeling (I had a feeling that the winners would be those who could figure out how to get this to work, but I never got convincing results in my experiments)\ncontrastive learning\nusing the last competitions task as an auxiliary task\n\n### Conclusion\n\nI was very happy to medal in this competition, as this is my first competition medal. I am very curious to understand how the top solutions utilized the extra unlabeled data.\nTraining code: [Training Notebook](https://www.kaggle.com/code/ryanbarretto/debertalargeauxtraining) ",
      "votes": 13
    },
    {
      "id": 1911468,
      "postDate": "2022-08-24T05:55:38.530Z",
      "content": "<p>multi-task learning</p>\n<p>How much benefit did multi-task learning give you online? I found that adding multi-task learning to my model seems to have no effect on the final score.</p>",
      "rawMarkdown": "multi-task learning\n\nHow much benefit did multi-task learning give you online? I found that adding multi-task learning to my model seems to have no effect on the final score.",
      "replies": [
        {
          "id": 1911922,
          "postDate": "2022-08-24T11:48:03.297Z",
          "content": "<p>For my private and public lb, it gave me a boost of around 0.01, which was pretty significant. </p>",
          "rawMarkdown": "For my private and public lb, it gave me a boost of around 0.01, which was pretty significant. ",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:55:01Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8677070,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/ryan-barretto-80th-place-solution-single-model"
}