{
  "id": 347356,
  "title": "Private 23rd solution",
  "content": "<p>First of all, I would like to thank the competition organizers for this competition. Also, I would like to say thank you for all the people for providing great notebooks and discussions, especially <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a>. As a matter of fact, my best single model is heavily based on <a href=\"https://www.kaggle.com/code/nbroad/token-classification-approach-fpe\" target=\"_blank\">his kernel</a>.</p>\n<h1>stage1: bert prediction</h1>\n<p>Because simple text classification approach is not very good at score and took too much inference time, we abandoned this approach and adopt a token classification approach. We trained models different ways in the point of post process, averaging the prediction of all tokens in the discourse_text like US PPPM 8th solution or just adopting the prediction of the first sep token of the discourse_text. In the former type of models, we used weighted cross entropy loss and this helped a lot.</p>\n<h1>stage2: lgb and xgb stacking and blending</h1>\n<p>Our best public sub adopt blending by Nelder-Mead and stacking using LightGBM and XGBoost by the ratio of 1: 1: 1. In stacking, some features such as the length of text, counts of each discourse type appearing each essay, and mean and std. of each discourse label in the essay_text. The details of some models used in ensemble is like below.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>token</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-large</td>\n<td>sep only</td>\n<td>0.5892</td>\n<td>0.577</td>\n<td>0.580</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>text token mean</td>\n<td>0.5907</td>\n<td>0.579</td>\n<td>0.587</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>sep only</td>\n<td>0.5921</td>\n<td>Not submitted</td>\n<td>Not submitted</td>\n</tr>\n</tbody>\n</table>\n<h1>summary</h1>\n<h2>useful attempt</h2>\n<ul>\n<li>Pseudo labeling using Feedback 2021 dataset<ul>\n<li>Both CV and Public LB decreased about 0.01.</li></ul></li>\n<li>AWP (eps: 1e-4, lr: 1.0, CV decreased about 0.003.)</li>\n<li>LightGBM and XGBoost stacking and some feature engineering</li>\n<li>segmented prediction and splicing like Feedback 2021 1st place solution</li>\n<li>ensembling different models that used different inference ways (averaging predictions of all tokens in the texts or just using prediction of the first sep token of the texts)</li>\n</ul>",
  "comments": [
    {
      "id": 1911154,
      "author_name": "Nicholas Broad",
      "author_url": "",
      "post_date": "2022-08-24T00:33:53.843000",
      "content": "<p>I'm happy to know my notebook was helpful 😊</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1911168,
          "author_name": "Vadim Irtlach",
          "author_url": "",
          "post_date": "2022-08-24T00:45:20.680000",
          "content": "<p>It was very helpful!!!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1920220,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-08-31T02:19:54.160000",
      "content": "<p><code>Our best public sub adopt blending by Nelder-Mead</code>What does this mean?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1920222,
          "author_name": "SUNFADED",
          "author_url": "",
          "post_date": "2022-08-31T02:25:20.773000",
          "content": "<p>In blending, we decided weights of each prediction by this code.</p>\n<pre><code>weights_list = []\nfor i in range(LOOP):\n    for fold in range(NUM_FOLDS):\n        train_idx = (oof[\"fold\"] != fold) &amp; (oof[\"fold\"].isin(CFG.trn_fold))\n        valid_idx = (oof[\"fold\"] == fold) &amp; (oof[\"fold\"].isin(CFG.trn_fold))\n\n        res = sp.optimize.minimize(\n            get_weighted_score, \n            weights,\n            args=(train_idx),\n            method=\"Nelder-Mead\",\n            bounds=tuple((0, np.inf) for _ in range(NUM_MODELS)),\n            tol=1e-6\n        )\n\n        print(\"score:\", res.fun)\n        print(\"weights:\", res.x)\n        weights_list.append(res.x)\n\nweights = np.array(weights_list).mean(0)\nprint(weights)\n</code></pre>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1920253,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-08-31T03:23:32.200000",
          "content": "<p>I got it, thanks</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 5,
  "messages": [
    {
      "id": 1911114,
      "postDate": "2022-08-24T00:10:38.083Z",
      "content": "<p>First of all, I would like to thank the competition organizers for this competition. Also, I would like to say thank you for all the people for providing great notebooks and discussions, especially <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a>. As a matter of fact, my best single model is heavily based on <a href=\"https://www.kaggle.com/code/nbroad/token-classification-approach-fpe\" target=\"_blank\">his kernel</a>.</p>\n<h1>stage1: bert prediction</h1>\n<p>Because simple text classification approach is not very good at score and took too much inference time, we abandoned this approach and adopt a token classification approach. We trained models different ways in the point of post process, averaging the prediction of all tokens in the discourse_text like US PPPM 8th solution or just adopting the prediction of the first sep token of the discourse_text. In the former type of models, we used weighted cross entropy loss and this helped a lot.</p>\n<h1>stage2: lgb and xgb stacking and blending</h1>\n<p>Our best public sub adopt blending by Nelder-Mead and stacking using LightGBM and XGBoost by the ratio of 1: 1: 1. In stacking, some features such as the length of text, counts of each discourse type appearing each essay, and mean and std. of each discourse label in the essay_text. The details of some models used in ensemble is like below.</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>token</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-large</td>\n<td>sep only</td>\n<td>0.5892</td>\n<td>0.577</td>\n<td>0.580</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>text token mean</td>\n<td>0.5907</td>\n<td>0.579</td>\n<td>0.587</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>sep only</td>\n<td>0.5921</td>\n<td>Not submitted</td>\n<td>Not submitted</td>\n</tr>\n</tbody>\n</table>\n<h1>summary</h1>\n<h2>useful attempt</h2>\n<ul>\n<li>Pseudo labeling using Feedback 2021 dataset<ul>\n<li>Both CV and Public LB decreased about 0.01.</li></ul></li>\n<li>AWP (eps: 1e-4, lr: 1.0, CV decreased about 0.003.)</li>\n<li>LightGBM and XGBoost stacking and some feature engineering</li>\n<li>segmented prediction and splicing like Feedback 2021 1st place solution</li>\n<li>ensembling different models that used different inference ways (averaging predictions of all tokens in the texts or just using prediction of the first sep token of the texts)</li>\n</ul>",
      "rawMarkdown": "First of all, I would like to thank the competition organizers for this competition. Also, I would like to say thank you for all the people for providing great notebooks and discussions, especially @nbroad. As a matter of fact, my best single model is heavily based on [his kernel](https://www.kaggle.com/code/nbroad/token-classification-approach-fpe).\n\n# stage1: bert prediction\nBecause simple text classification approach is not very good at score and took too much inference time, we abandoned this approach and adopt a token classification approach. We trained models different ways in the point of post process, averaging the prediction of all tokens in the discourse_text like US PPPM 8th solution or just adopting the prediction of the first sep token of the discourse_text. In the former type of models, we used weighted cross entropy loss and this helped a lot.\n\n# stage2: lgb and xgb stacking and blending\nOur best public sub adopt blending by Nelder-Mead and stacking using LightGBM and XGBoost by the ratio of 1: 1: 1. In stacking, some features such as the length of text, counts of each discourse type appearing each essay, and mean and std. of each discourse label in the essay_text. The details of some models used in ensemble is like below.\n| model | token | CV | Public | Private |\n| --- | --- | --- | --- | --- |\n| deberta-v3-large | sep only | 0.5892 | 0.577 | 0.580 |\n| deberta-v3-large | text token mean | 0.5907 | 0.579 | 0.587 |\n| deberta-large | sep only | 0.5921 | Not submitted | Not submitted |\n\n\n# summary\n## useful attempt\n-\tPseudo labeling using Feedback 2021 dataset\n    -\t    Both CV and Public LB decreased about 0.01.\n-\tAWP (eps: 1e-4, lr: 1.0, CV decreased about 0.003.)\n-\tLightGBM and XGBoost stacking and some feature engineering\n-\tsegmented prediction and splicing like Feedback 2021 1st place solution\n-\tensembling different models that used different inference ways (averaging predictions of all tokens in the texts or just using prediction of the first sep token of the texts)\n",
      "votes": 20
    },
    {
      "id": 1911154,
      "postDate": "2022-08-24T00:33:53.843Z",
      "content": "<p>I'm happy to know my notebook was helpful 😊</p>",
      "rawMarkdown": "I'm happy to know my notebook was helpful 😊",
      "votes": 2,
      "replies": [
        {
          "id": 1911168,
          "postDate": "2022-08-24T00:45:20.680Z",
          "content": "<p>It was very helpful!!!</p>",
          "rawMarkdown": "It was very helpful!!!"
        }
      ]
    },
    {
      "id": 1920220,
      "postDate": "2022-08-31T02:19:54.160Z",
      "content": "<p><code>Our best public sub adopt blending by Nelder-Mead</code>What does this mean?</p>",
      "rawMarkdown": "`Our best public sub adopt blending by Nelder-Mead `What does this mean?",
      "replies": [
        {
          "id": 1920222,
          "postDate": "2022-08-31T02:25:20.773Z",
          "content": "<p>In blending, we decided weights of each prediction by this code.</p>\n<pre><code>weights_list = []\nfor i in range(LOOP):\n    for fold in range(NUM_FOLDS):\n        train_idx = (oof[\"fold\"] != fold) &amp; (oof[\"fold\"].isin(CFG.trn_fold))\n        valid_idx = (oof[\"fold\"] == fold) &amp; (oof[\"fold\"].isin(CFG.trn_fold))\n\n        res = sp.optimize.minimize(\n            get_weighted_score, \n            weights,\n            args=(train_idx),\n            method=\"Nelder-Mead\",\n            bounds=tuple((0, np.inf) for _ in range(NUM_MODELS)),\n            tol=1e-6\n        )\n\n        print(\"score:\", res.fun)\n        print(\"weights:\", res.x)\n        weights_list.append(res.x)\n\nweights = np.array(weights_list).mean(0)\nprint(weights)\n</code></pre>",
          "rawMarkdown": "In blending, we decided weights of each prediction by this code.\n```\nweights_list = []\nfor i in range(LOOP):\n    for fold in range(NUM_FOLDS):\n        train_idx = (oof[\"fold\"] != fold) & (oof[\"fold\"].isin(CFG.trn_fold))\n        valid_idx = (oof[\"fold\"] == fold) & (oof[\"fold\"].isin(CFG.trn_fold))\n\n        res = sp.optimize.minimize(\n            get_weighted_score, \n            weights,\n            args=(train_idx),\n            method=\"Nelder-Mead\",\n            bounds=tuple((0, np.inf) for _ in range(NUM_MODELS)),\n            tol=1e-6\n        )\n\n        print(\"score:\", res.fun)\n        print(\"weights:\", res.x)\n        weights_list.append(res.x)\n\nweights = np.array(weights_list).mean(0)\nprint(weights)\n```",
          "votes": 1
        },
        {
          "id": 1920253,
          "postDate": "2022-08-31T03:23:32.200Z",
          "content": "<p>I got it, thanks</p>",
          "rawMarkdown": "I got it, thanks"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:53:53Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8798028,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/matslab-private-23rd-solution"
}