{
  "id": 313331,
  "title": "33rd Solution",
  "content": "<p>First of all, thanks for competition host and competitor. In this competition I learned a lot about many insights and knowledge about NLP.<br>\nEspecially I experienced NLP competition on Kaggle and token classification at the first time.</p>\n<p>I modified my following baseline. Mainly I describe different and important points here</p>\n<ul>\n<li>training<br>\n<a href=\"https://www.kaggle.com/ytakayama/train-pytorch-longformer-5fold-forgooglecolab\" target=\"_blank\">https://www.kaggle.com/ytakayama/train-pytorch-longformer-5fold-forgooglecolab</a></li>\n<li>inference<br>\n<a href=\"https://www.kaggle.com/ytakayama/infer-pytorch-longformer-5fold\" target=\"_blank\">https://www.kaggle.com/ytakayama/infer-pytorch-longformer-5fold</a></li>\n</ul>\n<h2>training</h2>\n<ul>\n<li><p>preprocess: replace LF to sep token.<br>\nThis time line feed seems to be important so that we judge discourse type, because it might be the point discourse type changes.</p></li>\n<li><p>augmentation: replace 10% of tokens of each documents except special tokens to mask tokens<br>\n<a href=\"https://www.kaggle.com/spidermandance/masking-feedback-prize\" target=\"_blank\">https://www.kaggle.com/spidermandance/masking-feedback-prize</a></p></li>\n<li><p>multi sample dropout: the same as the baseline</p></li>\n<li><p>model: BERT backbone + 15 class output(NER) the same as baseline</p></li>\n<li><p>optimizer: AdamW,</p></li>\n<li><p>weight_decay =0.01(except LayerNorm and bias)</p></li>\n<li><p>lr scheduler: linear warmup(warmup rate:0.05)</p></li>\n<li><p>epoch: 6</p></li>\n<li><p>loss function: Cross Entropy Loss(label smoothing=0.1)<br>\nlabel smoothing is effective.</p></li>\n<li><p>max_length(inference): 1536</p></li>\n<li><p>cv: KFold(n_splits=5)</p></li>\n<li><p>batch size * gradient accumulation steps: fixed to 4.<br>\nI trained all models on single GPU V100/A100(Google Colab Pro+), to prevent from OOM I used gradient accumulation. I got A100 only when training deberta-large. Most of all batch size is 1 and accumulation step is 4</p></li>\n</ul>\n<h1>detail about models</h1>\n<p>Including post process described at next section</p>\n<table>\n<thead>\n<tr>\n<th>No</th>\n<th>model</th>\n<th>max length※</th>\n<th>max lr</th>\n<th>cv</th>\n<th>public LB</th>\n<th>private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>allenai/longformer-large-4096</td>\n<td>1536</td>\n<td>1e-5</td>\n<td>0.6814</td>\n<td>0.683</td>\n<td>0.694</td>\n</tr>\n<tr>\n<td>2</td>\n<td>funnel-transformer/large</td>\n<td>1536</td>\n<td>8e-6</td>\n<td>0.6926</td>\n<td>0.698</td>\n<td>0.709</td>\n</tr>\n<tr>\n<td>3</td>\n<td>microsoft/deberta-large</td>\n<td>1536</td>\n<td>1e-5</td>\n<td>0.6956</td>\n<td>0.700</td>\n<td>0.712</td>\n</tr>\n<tr>\n<td>4</td>\n<td>microsoft/deberta-v3-large</td>\n<td>1024</td>\n<td>1e-5</td>\n<td>0.699</td>\n<td>0.703</td>\n<td>0.715</td>\n</tr>\n<tr>\n<td>5</td>\n<td>1+2+3</td>\n<td>-</td>\n<td>-</td>\n<td>0.7059</td>\n<td>0.705</td>\n<td>0.717</td>\n</tr>\n<tr>\n<td>6</td>\n<td>1+2+3+4 simple average</td>\n<td>-</td>\n<td>-</td>\n<td>0.7098</td>\n<td>0.709</td>\n<td>0.719</td>\n</tr>\n<tr>\n<td>7</td>\n<td>1+2+3+4 weighted average</td>\n<td>-</td>\n<td>-</td>\n<td>0.7111</td>\n<td>0.710</td>\n<td>0.720</td>\n</tr>\n</tbody>\n</table>\n<p>※at training</p>\n<h2>post process</h2>\n<ul>\n<li>fixed minimum length of words about each discourse type to 3</li>\n<li>Calculate minimum probability of each discourse type by using OOF so that f1 is higher and chose reasonable thresholds.</li>\n<li>fixed threshold of probability and tuned minimum length of words about each discourse type</li>\n<li>average probability of some models<br>\nNo.7: No.1 * 0.1 + No.2* 0.25 + No.3 * 0.3 + No.4 * 0.35</li>\n<li>link evidence<br>\n<a href=\"https://www.kaggle.com/abhishek/two-longformers-are-better-than-1\" target=\"_blank\">https://www.kaggle.com/abhishek/two-longformers-are-better-than-1</a></li>\n</ul>\n<h2>submission</h2>\n<p>I chose No.6 and No.7, best on both cv and lb <br>\nAggregated documents which are similar length to minibatch, I accelerated inference and colud ensemble 5 folds of 4 models.<br>\nTo aggregate documents to minibatch, I calculate length of words instead of that of characters.<br>\nwhich is different from the following notebook.<br>\n<a href=\"https://www.kaggle.com/librauee/infer-fast-ensemble-models\" target=\"_blank\">https://www.kaggle.com/librauee/infer-fast-ensemble-models</a></p>\n<h2>what did not work for me ( I cannot utilize better)</h2>\n<ul>\n<li>symmmetric cross entropy loss<br>\ntuned alpha but convergence became worse </li>\n<li>CRF</li>\n<li>use last some layers in hidden states</li>\n<li>add activation functions / arrange head of model</li>\n<li>calculate not first logits but average logits in each word</li>\n</ul>\n<p>I wrote overview of my solution in a hurry. I may update it later.<br>\nIn this competition I became competition expert and notebooks expert. Thanks for upvoting my notebooks and sharing great knowledge.</p>",
  "comments": [
    {
      "id": 1729283,
      "author_name": "chumajin",
      "author_url": "",
      "post_date": "2022-03-19T22:16:09.990000",
      "content": "<p>Thank you for sharing.  It is very helpful because it has detailed conditions. I'll try.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1729620,
          "author_name": "y.takayama",
          "author_url": "",
          "post_date": "2022-03-20T09:44:37.600000",
          "content": "<p><a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> You're welcome. I learned a lot from your notebooks, thanks.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1729540,
      "author_name": "Ishan Mehta115",
      "author_url": "",
      "post_date": "2022-03-20T07:36:23.533000",
      "content": "<p>Really helpful</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1724936,
      "author_name": "Nicholas Broad",
      "author_url": "",
      "post_date": "2022-03-16T16:31:21.750000",
      "content": "<p>よく出来ました!</p>\n<p>A100 is available in Google Colab Pro+?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1725226,
          "author_name": "y.takayama",
          "author_url": "",
          "post_date": "2022-03-16T23:58:59.357000",
          "content": "<p>Sometimes.  I don't know how runtime is provided.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1725261,
          "author_name": "Nicholas Broad",
          "author_url": "",
          "post_date": "2022-03-17T01:06:37.533000",
          "content": "<p>What fraction of the time did you get an A100?</p>\n<p>And was V100 the only other GPU you used through Pro+?</p>\n<p>I'm deciding if I should try Pro+.  I'll start a discussion in the general section to get other people's opinion</p>",
          "votes": 0,
          "replies": [
            {
              "id": 1725455,
              "author_name": "Udbhav Bamba",
              "author_url": "",
              "post_date": "2022-03-17T05:41:34.423000",
              "content": "<p>I tried Pro+ at the start of competition always got P100 and after about 1 week of usage they declined any background execution as well to me.</p>",
              "votes": 1,
              "replies": []
            }
          ]
        },
        {
          "id": 1725457,
          "author_name": "Atharva Ingle",
          "author_url": "",
          "post_date": "2022-03-17T05:45:09.647000",
          "content": "<blockquote>\n  <p>I tried Pro+ at the start of competition always got P100 and after about 1 week of usage they declined any background execution as well to me.</p>\n</blockquote>\n<p>I second this, normal colab pro is far better than Pro+ and a great value for money, I always get 2 P100s at a time.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 8,
  "messages": [
    {
      "id": 1724800,
      "postDate": "2022-03-16T14:18:17.543Z",
      "content": "<p>First of all, thanks for competition host and competitor. In this competition I learned a lot about many insights and knowledge about NLP.<br>\nEspecially I experienced NLP competition on Kaggle and token classification at the first time.</p>\n<p>I modified my following baseline. Mainly I describe different and important points here</p>\n<ul>\n<li>training<br>\n<a href=\"https://www.kaggle.com/ytakayama/train-pytorch-longformer-5fold-forgooglecolab\" target=\"_blank\">https://www.kaggle.com/ytakayama/train-pytorch-longformer-5fold-forgooglecolab</a></li>\n<li>inference<br>\n<a href=\"https://www.kaggle.com/ytakayama/infer-pytorch-longformer-5fold\" target=\"_blank\">https://www.kaggle.com/ytakayama/infer-pytorch-longformer-5fold</a></li>\n</ul>\n<h2>training</h2>\n<ul>\n<li><p>preprocess: replace LF to sep token.<br>\nThis time line feed seems to be important so that we judge discourse type, because it might be the point discourse type changes.</p></li>\n<li><p>augmentation: replace 10% of tokens of each documents except special tokens to mask tokens<br>\n<a href=\"https://www.kaggle.com/spidermandance/masking-feedback-prize\" target=\"_blank\">https://www.kaggle.com/spidermandance/masking-feedback-prize</a></p></li>\n<li><p>multi sample dropout: the same as the baseline</p></li>\n<li><p>model: BERT backbone + 15 class output(NER) the same as baseline</p></li>\n<li><p>optimizer: AdamW,</p></li>\n<li><p>weight_decay =0.01(except LayerNorm and bias)</p></li>\n<li><p>lr scheduler: linear warmup(warmup rate:0.05)</p></li>\n<li><p>epoch: 6</p></li>\n<li><p>loss function: Cross Entropy Loss(label smoothing=0.1)<br>\nlabel smoothing is effective.</p></li>\n<li><p>max_length(inference): 1536</p></li>\n<li><p>cv: KFold(n_splits=5)</p></li>\n<li><p>batch size * gradient accumulation steps: fixed to 4.<br>\nI trained all models on single GPU V100/A100(Google Colab Pro+), to prevent from OOM I used gradient accumulation. I got A100 only when training deberta-large. Most of all batch size is 1 and accumulation step is 4</p></li>\n</ul>\n<h1>detail about models</h1>\n<p>Including post process described at next section</p>\n<table>\n<thead>\n<tr>\n<th>No</th>\n<th>model</th>\n<th>max length※</th>\n<th>max lr</th>\n<th>cv</th>\n<th>public LB</th>\n<th>private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>allenai/longformer-large-4096</td>\n<td>1536</td>\n<td>1e-5</td>\n<td>0.6814</td>\n<td>0.683</td>\n<td>0.694</td>\n</tr>\n<tr>\n<td>2</td>\n<td>funnel-transformer/large</td>\n<td>1536</td>\n<td>8e-6</td>\n<td>0.6926</td>\n<td>0.698</td>\n<td>0.709</td>\n</tr>\n<tr>\n<td>3</td>\n<td>microsoft/deberta-large</td>\n<td>1536</td>\n<td>1e-5</td>\n<td>0.6956</td>\n<td>0.700</td>\n<td>0.712</td>\n</tr>\n<tr>\n<td>4</td>\n<td>microsoft/deberta-v3-large</td>\n<td>1024</td>\n<td>1e-5</td>\n<td>0.699</td>\n<td>0.703</td>\n<td>0.715</td>\n</tr>\n<tr>\n<td>5</td>\n<td>1+2+3</td>\n<td>-</td>\n<td>-</td>\n<td>0.7059</td>\n<td>0.705</td>\n<td>0.717</td>\n</tr>\n<tr>\n<td>6</td>\n<td>1+2+3+4 simple average</td>\n<td>-</td>\n<td>-</td>\n<td>0.7098</td>\n<td>0.709</td>\n<td>0.719</td>\n</tr>\n<tr>\n<td>7</td>\n<td>1+2+3+4 weighted average</td>\n<td>-</td>\n<td>-</td>\n<td>0.7111</td>\n<td>0.710</td>\n<td>0.720</td>\n</tr>\n</tbody>\n</table>\n<p>※at training</p>\n<h2>post process</h2>\n<ul>\n<li>fixed minimum length of words about each discourse type to 3</li>\n<li>Calculate minimum probability of each discourse type by using OOF so that f1 is higher and chose reasonable thresholds.</li>\n<li>fixed threshold of probability and tuned minimum length of words about each discourse type</li>\n<li>average probability of some models<br>\nNo.7: No.1 * 0.1 + No.2* 0.25 + No.3 * 0.3 + No.4 * 0.35</li>\n<li>link evidence<br>\n<a href=\"https://www.kaggle.com/abhishek/two-longformers-are-better-than-1\" target=\"_blank\">https://www.kaggle.com/abhishek/two-longformers-are-better-than-1</a></li>\n</ul>\n<h2>submission</h2>\n<p>I chose No.6 and No.7, best on both cv and lb <br>\nAggregated documents which are similar length to minibatch, I accelerated inference and colud ensemble 5 folds of 4 models.<br>\nTo aggregate documents to minibatch, I calculate length of words instead of that of characters.<br>\nwhich is different from the following notebook.<br>\n<a href=\"https://www.kaggle.com/librauee/infer-fast-ensemble-models\" target=\"_blank\">https://www.kaggle.com/librauee/infer-fast-ensemble-models</a></p>\n<h2>what did not work for me ( I cannot utilize better)</h2>\n<ul>\n<li>symmmetric cross entropy loss<br>\ntuned alpha but convergence became worse </li>\n<li>CRF</li>\n<li>use last some layers in hidden states</li>\n<li>add activation functions / arrange head of model</li>\n<li>calculate not first logits but average logits in each word</li>\n</ul>\n<p>I wrote overview of my solution in a hurry. I may update it later.<br>\nIn this competition I became competition expert and notebooks expert. Thanks for upvoting my notebooks and sharing great knowledge.</p>",
      "rawMarkdown": "First of all, thanks for competition host and competitor. In this competition I learned a lot about many insights and knowledge about NLP.\nEspecially I experienced NLP competition on Kaggle and token classification at the first time.\n\nI modified my following baseline. Mainly I describe different and important points here\n- training\nhttps://www.kaggle.com/ytakayama/train-pytorch-longformer-5fold-forgooglecolab\n- inference\nhttps://www.kaggle.com/ytakayama/infer-pytorch-longformer-5fold\n\n## training\n- preprocess: replace LF to sep token.\nThis time line feed seems to be important so that we judge discourse type, because it might be the point discourse type changes.\n- augmentation: replace 10% of tokens of each documents except special tokens to mask tokens\nhttps://www.kaggle.com/spidermandance/masking-feedback-prize\n- multi sample dropout: the same as the baseline\n- model: BERT backbone + 15 class output(NER) the same as baseline\n\n- optimizer: AdamW,\n- weight_decay =0.01(except LayerNorm and bias)\n-  lr scheduler: linear warmup(warmup rate:0.05)\n- epoch: 6\n- loss function: Cross Entropy Loss(label smoothing=0.1)\nlabel smoothing is effective.\n- max_length(inference): 1536\n- cv: KFold(n_splits=5)\n- batch size * gradient accumulation steps: fixed to 4.\nI trained all models on single GPU V100/A100(Google Colab Pro+), to prevent from OOM I used gradient accumulation. I got A100 only when training deberta-large. Most of all batch size is 1 and accumulation step is 4\n\n# detail about models\nIncluding post process described at next section\n|No |model |max length※ | max lr |cv|public LB| private LB|\n| ---|--- | --- | --- |--- |--- |--- |\n| 1 |allenai/longformer-large-4096  | 1536 | 1e-5| 0.6814| 0.683| 0.694| \n| 2 |funnel-transformer/large  | 1536 | 8e-6| 0.6926| 0.698| 0.709| \n| 3 |microsoft/deberta-large  | 1536 | 1e-5| 0.6956| 0.700| 0.712| \n| 4 |microsoft/deberta-v3-large  | 1024| 1e-5| 0.699| 0.703| 0.715| \n| 5 | 1+2+3   | -| -| 0.7059| 0.705| 0.717| \n| 6 | 1+2+3+4 simple average   | -| - | 0.7098| 0.709| 0.719| \n| 7 | 1+2+3+4 weighted average   | -| - | 0.7111| 0.710| 0.720| \n\n\n※at training\n## post process\n- fixed minimum length of words about each discourse type to 3\n- Calculate minimum probability of each discourse type by using OOF so that f1 is higher and chose reasonable thresholds.\n- fixed threshold of probability and tuned minimum length of words about each discourse type\n- average probability of some models\nNo.7: No.1 * 0.1 + No.2* 0.25 + No.3 * 0.3 + No.4 * 0.35\n- link evidence\nhttps://www.kaggle.com/abhishek/two-longformers-are-better-than-1\n\n\n## submission\nI chose No.6 and No.7, best on both cv and lb \nAggregated documents which are similar length to minibatch, I accelerated inference and colud ensemble 5 folds of 4 models.\nTo aggregate documents to minibatch, I calculate length of words instead of that of characters.\nwhich is different from the following notebook.\nhttps://www.kaggle.com/librauee/infer-fast-ensemble-models\n\n## what did not work for me ( I cannot utilize better)\n- symmmetric cross entropy loss\ntuned alpha but convergence became worse \n- CRF\n- use last some layers in hidden states\n- add activation functions / arrange head of model\n- calculate not first logits but average logits in each word\n\n\nI wrote overview of my solution in a hurry. I may update it later.\nIn this competition I became competition expert and notebooks expert. Thanks for upvoting my notebooks and sharing great knowledge.",
      "votes": 29
    },
    {
      "id": 1729283,
      "postDate": "2022-03-19T22:16:09.990Z",
      "content": "<p>Thank you for sharing.  It is very helpful because it has detailed conditions. I'll try.</p>",
      "rawMarkdown": "Thank you for sharing.  It is very helpful because it has detailed conditions. I'll try.",
      "votes": 2,
      "replies": [
        {
          "id": 1729620,
          "postDate": "2022-03-20T09:44:37.600Z",
          "content": "<p><a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> You're welcome. I learned a lot from your notebooks, thanks.</p>",
          "rawMarkdown": "@chumajin You're welcome. I learned a lot from your notebooks, thanks.",
          "votes": 2
        }
      ]
    },
    {
      "id": 1729540,
      "postDate": "2022-03-20T07:36:23.533Z",
      "content": "<p>Really helpful</p>",
      "rawMarkdown": "Really helpful"
    },
    {
      "id": 1724936,
      "postDate": "2022-03-16T16:31:21.750Z",
      "content": "<p>よく出来ました!</p>\n<p>A100 is available in Google Colab Pro+?</p>",
      "rawMarkdown": "よく出来ました!\n\nA100 is available in Google Colab Pro+?",
      "replies": [
        {
          "id": 1725226,
          "postDate": "2022-03-16T23:58:59.357Z",
          "content": "<p>Sometimes.  I don't know how runtime is provided.</p>",
          "rawMarkdown": "Sometimes.  I don't know how runtime is provided."
        },
        {
          "id": 1725261,
          "postDate": "2022-03-17T01:06:37.533Z",
          "content": "<p>What fraction of the time did you get an A100?</p>\n<p>And was V100 the only other GPU you used through Pro+?</p>\n<p>I'm deciding if I should try Pro+.  I'll start a discussion in the general section to get other people's opinion</p>",
          "rawMarkdown": "What fraction of the time did you get an A100?\n\nAnd was V100 the only other GPU you used through Pro+?\n\nI'm deciding if I should try Pro+.  I'll start a discussion in the general section to get other people's opinion",
          "replies": [
            {
              "id": 1725455,
              "postDate": "2022-03-17T05:41:34.423Z",
              "content": "<p>I tried Pro+ at the start of competition always got P100 and after about 1 week of usage they declined any background execution as well to me.</p>",
              "rawMarkdown": "I tried Pro+ at the start of competition always got P100 and after about 1 week of usage they declined any background execution as well to me.",
              "votes": 1
            }
          ]
        },
        {
          "id": 1725457,
          "postDate": "2022-03-17T05:45:09.647Z",
          "content": "<blockquote>\n  <p>I tried Pro+ at the start of competition always got P100 and after about 1 week of usage they declined any background execution as well to me.</p>\n</blockquote>\n<p>I second this, normal colab pro is far better than Pro+ and a great value for money, I always get 2 P100s at a time.</p>",
          "rawMarkdown": "> I tried Pro+ at the start of competition always got P100 and after about 1 week of usage they declined any background execution as well to me.\n\nI second this, normal colab pro is far better than Pro+ and a great value for money, I always get 2 P100s at a time.",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T12:35:38Z",
  "competition_id": 31779,
  "competition_slug": "feedback-prize-2021",
  "team_id": 7973873,
  "solution_url": "/competitions/feedback-prize-2021/writeups/y-takayama-33rd-solution"
}