{
  "id": 369565,
  "title": "12th Place Solution",
  "content": "<p>First of all I would like to thank the competition organizers and Kaggle and congratulate all the winners.<br>\nI would also like to thank all the kagglers who provided useful information in the various discussions.<br>\nI have been working hard to achieve Solo Gold and am happy to finally win it.</p>\n<h1>Overview</h1>\n<p>My solution is an ensemble of various deberta models and RAPIDS SVR.<br>\nI have been referring to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> excellent notebook on RAPIDS SVR. Thank you very much.<br>\n<a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a></p>\n<h1>Preprocessing</h1>\n<p>Two types of text data were prepared, one with \\n converted to [BR] and the other without.<br>\nThe model trained on the text data converted to [BR] scored higher for both Public and Private.</p>\n<h1>Cross validation</h1>\n<p>I used 5folds in MultiLabelStratifiedKFold by target.</p>\n<h1>Model</h1>\n<p>backborn</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>deberta-large</li>\n<li>deberta-xlarge</li>\n<li>deberta-xlarge-mnli</li>\n</ul>\n<p>pooling</p>\n<ul>\n<li>meanpolling&amp;layernorm</li>\n</ul>\n<h1>RAPIDS SVR</h1>\n<p>Features were extracted from the following models and SVR trained.</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-large</li>\n<li>deberta-v3-large</li>\n<li>deberta-xlarge</li>\n<li>deberta-base-mnli</li>\n<li>deberta-large-mnli</li>\n<li>deberta-xlarge-mnli</li>\n<li>deberta-v2-xlarge</li>\n<li>deberta-v2-xlarge-mnli</li>\n</ul>\n<h1>ensemble</h1>\n<p>Weights were adjusted by target using the Nelder-Mead method.</p>\n<h1>What worked</h1>\n<ul>\n<li>Soft Labeling</li>\n<li>Pseudo Labeling</li>\n<li>Small batch size (batchsize=2)</li>\n</ul>\n<h1>What didn't work</h1>\n<ul>\n<li>MLM</li>\n<li>Non-deberta models</li>\n</ul>\n<h1>What worked in Private</h1>\n<p>The ensemble with GPR (Gaussian process regression) had the highest private score, but was not selected because the public score was not high.(public:0.437852/private:0.433646)</p>\n<h1>Efficiency Prize(4th place)</h1>\n<ul>\n<li>deberta-v3-xsmall</li>\n<li>Soft labeling</li>\n<li>Pseudo Labeling</li>\n<li>Use the model with the highest CV score out of 5folds</li>\n</ul>",
  "comments": [
    {
      "id": 2050261,
      "author_name": "Mark Baushenko",
      "author_url": "",
      "post_date": "2022-11-30T16:03:20.850000",
      "content": "<p>Good job! </p>",
      "votes": 1,
      "replies": [
        {
          "id": 2050727,
          "author_name": "hiro",
          "author_url": "",
          "post_date": "2022-12-01T00:00:29.713000",
          "content": "<p>Thank you!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2050251,
      "author_name": "Noir.s",
      "author_url": "",
      "post_date": "2022-11-30T15:57:24.530000",
      "content": "<p>Thanks for sharing! Congratulations on your gold medal finish 😄</p>\n<p>I have a question, can you explain what soft labeling and pseudo labeling means?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2050743,
          "author_name": "hiro",
          "author_url": "",
          "post_date": "2022-12-01T00:30:06.020000",
          "content": "<p>Thank you!</p>\n<p>Soft labeling was learned by adding a certain percentage of oof predictions to the labels in the training data.<br>\nFor the train:oof ratio, 0.5:0.5 was well balanced and 0.2:0.8 showed a strong tendency toward overfitting.</p>\n<p>Pseudo-labeling was trained by labeling the fb1 data.<br>\nWe also always used the same CV to avoid leakage.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3207762,
      "author_name": "Julien Mouchnino",
      "author_url": "",
      "post_date": "2025-05-23T07:40:24.267000",
      "content": "<p>How did you combine the prompt and the answer? Did you simply paste the prompt and answer?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2051396,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-12-01T11:40:31.450000",
      "content": "",
      "votes": 1,
      "replies": [
        {
          "id": 2051462,
          "author_name": "hiro",
          "author_url": "",
          "post_date": "2022-12-01T12:40:50.533000",
          "content": "<p>[BR] is converted in preprocessing as follows.</p>\n<p><code>test[\"full_text\"] = [text.replace(\"\\n\", \"[BR]\") for text in test[\"full_text\"]]</code></p>\n<p>Add it to the tokenizer as a special token.</p>\n<p><code>tokenizer.add_special_tokens(\n    {\"additional_special_tokens\": ['[BR]']}\n)</code></p>\n<p>The following may be of interest regarding Nelder-Mead.</p>\n<p><a href=\"url\" target=\"_blank\">https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328</a>.</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2872716,
              "author_name": "HappyOldMan",
              "author_url": "",
              "post_date": "2024-06-15T02:55:50.927000",
              "content": "<p>hello, may I ask that what the purpose of translating <code>\"\\n\"</code> to <code>['BR']</code>? Will this bring any differences?<br>\nBest Regards!</p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 2051644,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-12-01T14:40:44.433000",
          "content": "",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 9,
  "messages": [
    {
      "id": 2050065,
      "postDate": "2022-11-30T14:01:34.287Z",
      "content": "<p>First of all I would like to thank the competition organizers and Kaggle and congratulate all the winners.<br>\nI would also like to thank all the kagglers who provided useful information in the various discussions.<br>\nI have been working hard to achieve Solo Gold and am happy to finally win it.</p>\n<h1>Overview</h1>\n<p>My solution is an ensemble of various deberta models and RAPIDS SVR.<br>\nI have been referring to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> excellent notebook on RAPIDS SVR. Thank you very much.<br>\n<a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a></p>\n<h1>Preprocessing</h1>\n<p>Two types of text data were prepared, one with \\n converted to [BR] and the other without.<br>\nThe model trained on the text data converted to [BR] scored higher for both Public and Private.</p>\n<h1>Cross validation</h1>\n<p>I used 5folds in MultiLabelStratifiedKFold by target.</p>\n<h1>Model</h1>\n<p>backborn</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>deberta-large</li>\n<li>deberta-xlarge</li>\n<li>deberta-xlarge-mnli</li>\n</ul>\n<p>pooling</p>\n<ul>\n<li>meanpolling&amp;layernorm</li>\n</ul>\n<h1>RAPIDS SVR</h1>\n<p>Features were extracted from the following models and SVR trained.</p>\n<ul>\n<li>deberta-v3-base</li>\n<li>deberta-large</li>\n<li>deberta-v3-large</li>\n<li>deberta-xlarge</li>\n<li>deberta-base-mnli</li>\n<li>deberta-large-mnli</li>\n<li>deberta-xlarge-mnli</li>\n<li>deberta-v2-xlarge</li>\n<li>deberta-v2-xlarge-mnli</li>\n</ul>\n<h1>ensemble</h1>\n<p>Weights were adjusted by target using the Nelder-Mead method.</p>\n<h1>What worked</h1>\n<ul>\n<li>Soft Labeling</li>\n<li>Pseudo Labeling</li>\n<li>Small batch size (batchsize=2)</li>\n</ul>\n<h1>What didn't work</h1>\n<ul>\n<li>MLM</li>\n<li>Non-deberta models</li>\n</ul>\n<h1>What worked in Private</h1>\n<p>The ensemble with GPR (Gaussian process regression) had the highest private score, but was not selected because the public score was not high.(public:0.437852/private:0.433646)</p>\n<h1>Efficiency Prize(4th place)</h1>\n<ul>\n<li>deberta-v3-xsmall</li>\n<li>Soft labeling</li>\n<li>Pseudo Labeling</li>\n<li>Use the model with the highest CV score out of 5folds</li>\n</ul>",
      "rawMarkdown": "First of all I would like to thank the competition organizers and Kaggle and congratulate all the winners.\nI would also like to thank all the kagglers who provided useful information in the various discussions.\nI have been working hard to achieve Solo Gold and am happy to finally win it.\n\n# Overview\nMy solution is an ensemble of various deberta models and RAPIDS SVR.\nI have been referring to @cdeotte excellent notebook on RAPIDS SVR. Thank you very much.\n[https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x](url)\n\n# Preprocessing\nTwo types of text data were prepared, one with \\n converted to [BR] and the other without.\nThe model trained on the text data converted to [BR] scored higher for both Public and Private.\n\n# Cross validation\nI used 5folds in MultiLabelStratifiedKFold by target.\n\n# Model\nbackborn\n- deberta-v3-base\n- deberta-v3-large\n- deberta-large\n- deberta-xlarge\n- deberta-xlarge-mnli\n\npooling\n- meanpolling&layernorm\n\n# RAPIDS SVR\nFeatures were extracted from the following models and SVR trained.\n- deberta-v3-base\n- deberta-large\n- deberta-v3-large\n- deberta-xlarge\n- deberta-base-mnli\n- deberta-large-mnli\n- deberta-xlarge-mnli\n- deberta-v2-xlarge\n- deberta-v2-xlarge-mnli\n\n# ensemble\nWeights were adjusted by target using the Nelder-Mead method.\n\n# What worked\n- Soft Labeling\n- Pseudo Labeling\n- Small batch size (batchsize=2)\n\n# What didn't work\n- MLM\n- Non-deberta models\n\n# What worked in Private\nThe ensemble with GPR (Gaussian process regression) had the highest private score, but was not selected because the public score was not high.(public:0.437852/private:0.433646)\n\n# Efficiency Prize(4th place)\n- deberta-v3-xsmall\n- Soft labeling\n- Pseudo Labeling\n- Use the model with the highest CV score out of 5folds",
      "votes": 20
    },
    {
      "id": 2050261,
      "postDate": "2022-11-30T16:03:20.850Z",
      "content": "<p>Good job! </p>",
      "rawMarkdown": "Good job! ",
      "votes": 1,
      "replies": [
        {
          "id": 2050727,
          "postDate": "2022-12-01T00:00:29.713Z",
          "content": "<p>Thank you!</p>",
          "rawMarkdown": "Thank you!"
        }
      ]
    },
    {
      "id": 2050251,
      "postDate": "2022-11-30T15:57:24.530Z",
      "content": "<p>Thanks for sharing! Congratulations on your gold medal finish 😄</p>\n<p>I have a question, can you explain what soft labeling and pseudo labeling means?</p>",
      "rawMarkdown": "Thanks for sharing! Congratulations on your gold medal finish 😄\n\nI have a question, can you explain what soft labeling and pseudo labeling means?",
      "votes": 1,
      "replies": [
        {
          "id": 2050743,
          "postDate": "2022-12-01T00:30:06.020Z",
          "content": "<p>Thank you!</p>\n<p>Soft labeling was learned by adding a certain percentage of oof predictions to the labels in the training data.<br>\nFor the train:oof ratio, 0.5:0.5 was well balanced and 0.2:0.8 showed a strong tendency toward overfitting.</p>\n<p>Pseudo-labeling was trained by labeling the fb1 data.<br>\nWe also always used the same CV to avoid leakage.</p>",
          "rawMarkdown": "Thank you!\n\nSoft labeling was learned by adding a certain percentage of oof predictions to the labels in the training data.\nFor the train:oof ratio, 0.5:0.5 was well balanced and 0.2:0.8 showed a strong tendency toward overfitting.\n\nPseudo-labeling was trained by labeling the fb1 data.\nWe also always used the same CV to avoid leakage.",
          "votes": 1
        }
      ]
    },
    {
      "id": 3207762,
      "postDate": "2025-05-23T07:40:24.267Z",
      "content": "<p>How did you combine the prompt and the answer? Did you simply paste the prompt and answer?</p>",
      "rawMarkdown": "How did you combine the prompt and the answer? Did you simply paste the prompt and answer?"
    },
    {
      "id": 2051396,
      "postDate": "2022-12-01T11:40:31.450Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true,
      "replies": [
        {
          "id": 2051462,
          "postDate": "2022-12-01T12:40:50.533Z",
          "content": "<p>[BR] is converted in preprocessing as follows.</p>\n<p><code>test[\"full_text\"] = [text.replace(\"\\n\", \"[BR]\") for text in test[\"full_text\"]]</code></p>\n<p>Add it to the tokenizer as a special token.</p>\n<p><code>tokenizer.add_special_tokens(\n    {\"additional_special_tokens\": ['[BR]']}\n)</code></p>\n<p>The following may be of interest regarding Nelder-Mead.</p>\n<p><a href=\"url\" target=\"_blank\">https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328</a>.</p>",
          "rawMarkdown": "[BR] is converted in preprocessing as follows.\n\n`test[\"full_text\"] = [text.replace(\"\\n\", \"[BR]\") for text in test[\"full_text\"]]`\n\nAdd it to the tokenizer as a special token.\n\n`tokenizer.add_special_tokens(\n    {\"additional_special_tokens\": ['[BR]']}\n)`\n\nThe following may be of interest regarding Nelder-Mead.\n\n[https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328](url).",
          "replies": [
            {
              "id": 2872716,
              "postDate": "2024-06-15T02:55:50.927Z",
              "content": "<p>hello, may I ask that what the purpose of translating <code>\"\\n\"</code> to <code>['BR']</code>? Will this bring any differences?<br>\nBest Regards!</p>",
              "rawMarkdown": "hello, may I ask that what the purpose of translating `\"\\n\"` to `['BR']`? Will this bring any differences?\nBest Regards!"
            }
          ]
        },
        {
          "id": 2051644,
          "postDate": "2022-12-01T14:40:44.433Z",
          "rawMarkdown": "",
          "votes": 1,
          "isDeleted": true
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:00:55Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9130506,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/ho-12th-place-solution"
}