{
  "id": 313330,
  "title": "4th place solution - 🎖️ my first gold medal 🎖️ (+source code available!)",
  "content": "<p><a href=\"https://github.com/affjljoo3581/Feedback-Prize-Competition\" target=\"_blank\">Here is a source code in github!</a></p>\n<p>Hi kagglers! Congrats to the winners and everyone who enjoyed this competition! I saw many people were tired, and me as well… But I've got my very first solo gold medal 🎖️ and I'm now really happy to become a Kaggle master!</p>\n<p>To be honest, I participated this competition about 3 weeks ago. Thanks to the community, I can accelerate my experiments and improve the score quickly. If I had more time, maybe I could get more chances… 🤔🤔</p>\n<p>I think some tricks and techniques are important in this competition. <a href=\"https://www.kaggle.com/c/feedback-prize-2021/discussion/313177\" target=\"_blank\">The lightgbm postprocessing</a> is really cool and I've never thought about that idea. I focused on some technical processing and I managed to get a high score.</p>\n<h2>Resolve Encoding Error</h2>\n<p>As you can see, there are some abnormal characters in the documents. Let's see <code>0D1493FDAAD3</code>:</p>\n<pre><code>...\nAnother reason to talk to multiple people when making a purchase is to learn from their successes. ÃÅf you saw a group of people successfully do what you are trying now, learn what they did to overcome the obstacle.\n...\n</code></pre>\n<p>There are nonrecognizable characters <code>ÃÅ</code> and you may see this frequently in other documents. After some tries, I found this is a problem about <code>cp1252</code> and <code>utf8</code> encodings, and the below code can clean the documents.<br>\n<img src=\"https://i.ibb.co/Q9Wzrzr/image.png\" alt=\"\"><br>\nThe result is as follows:</p>\n<pre><code>...\nAnother reason to talk to multiple people when making a purchase is to learn from their successes. If you saw a group of people successfully do what you are trying now, learn what they did to overcome the obstacle.\n...\n</code></pre>\n<p>Actually, It doesn't seem to improve the final scores significantly, but I just applied this to make sure.<br>\nAnd because it reduces the characters in the documents, we have to adjust the <code>discourse_start</code> and <code>discourse_end</code> offsets. To correct the offsets, I use <code>difflib.SequenceMatcher</code> to compare the differences of the characters. The details are in my code.</p>\n<h2><code>word_ids</code> vs <code>offset_mapping</code>?</h2>\n<p>Because many NER examples use <code>word_ids</code> with <code>is_split_into_words=True</code> parameter, I also tried to use this for labeling BIO-named subword tags. However, I got 0.595 public LB with a single-fold bigbird base model. So I tried another way from the community, <code>offset_mapping</code> with <code>return_offsets_mapping=True</code> parameter to make subword NER tags, and I could get 0.630 public LB score.</p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>bigbird-roberta-base (single fold, <code>word_ids</code>)</td>\n<td>0.595</td>\n<td>0.609</td>\n</tr>\n<tr>\n<td>bigbird-roberta-base (single fold)</td>\n<td>0.630</td>\n<td>0.644</td>\n</tr>\n<tr>\n<td>bigbird-roberta-base (5 folds)</td>\n<td>0.659</td>\n<td>0.677</td>\n</tr>\n</tbody>\n</table>\n<p>Why is this happened? What is the difference between <code>word_ids</code> and <code>offset_mapping</code> approach? After trying some experiences, I found that <code>word_ids</code> needs to split the text into words using <code>.split()</code>, and it prunes the line-break information <code>\\n</code>. Since this task is for recognizing the structure of the document, <strong>it would be necessary to use the line-break characters.</strong> Remember this because I will mention this fact in the later section.</p>\n<h2>Beam-Search Decoding</h2>\n<p>BIO-naming rule is quite complex, so the greedy decoding (i.e. argmax from the token predictions) cannot perform well. I observed many cases like:</p>\n<table>\n<thead>\n<tr>\n<th>sorted candidate tags</th>\n<th>sorted corresponding probs</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>B-Claim</code> …</td>\n<td>1.0 …</td>\n</tr>\n<tr>\n<td><code>I-Claim</code> …</td>\n<td>0.99  …</td>\n</tr>\n<tr>\n<td><code>I-Claim</code> …</td>\n<td>0.99 …</td>\n</tr>\n<tr>\n<td><code>I-Claim</code> <code>B-Evidence</code> …</td>\n<td>0.49 0.43 …</td>\n</tr>\n<tr>\n<td><code>I-Evidence</code> <code>I-Claim</code> …</td>\n<td>0.99 0.01 …</td>\n</tr>\n<tr>\n<td><code>I-Evidence</code> <code>I-Claim</code> …</td>\n<td>0.99 0.01 …</td>\n</tr>\n</tbody>\n</table>\n<p>According to the greedy decoding, the prediction result would be <code>B-Claim I-Claim I-Claim I-Claim I-Evidence I-Evidence</code>. The entities should be started with B-tags, so the Evidence entity will be dropped even it has high confidence. Therefore I implement beam search decoding algorithm for NER tagging:<br>\n<img src=\"https://i.ibb.co/RjhFk1P/image.png\" alt=\"\"><br>\nI wrote this code on PyTorch and GPU-capable for parallelization. I use <code>beam_size=4</code> for all both evaluation and prediction.</p>\n<h2>Entity-Level Post-Ensemble</h2>\n<p>The above multi-fold predictions are from averaged token probabilities. However, if the architectures are different and the subword tokenizers are different as well, it is impossible to combine the subword token predictions (probabilities). Some people tried to ensemble with character-level probabilities, but it did not work for me. Hence I created entity-level post ensemble algorithm.</p>\n<p>The main idea is from the metric of this competition. This compeition treats more than 50% matched entities as the same group. So I group at least 50% overlapped entities which have same class. After that, I average the ranges in each group. I tested the longest, shortest, union and intersection ways, but they were all worse than the average method.<br>\n<img src=\"https://i.ibb.co/5cCqwYb/image.png\" alt=\"\"><br>\nIt indeed showed significant improvements by ensembling several models! It can be even applied to the folds (I mean, it is useful for the same tokenizer and same subwords as well) and I can get about +0.002 lb score.</p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>cv</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-large (5folds)</td>\n<td>0.6938</td>\n<td>0.705</td>\n<td>0.713</td>\n</tr>\n<tr>\n<td>deberta-large (5folds, entity-level ensemble)</td>\n<td>0.6938</td>\n<td>0.707</td>\n<td>0.714</td>\n</tr>\n<tr>\n<td>deberta-large + deberta-xlarge</td>\n<td>0.718</td>\n<td>0.712</td>\n<td>0.722</td>\n</tr>\n<tr>\n<td>deberta-v3-large-v2 + deberta-v2-xlarge-v2</td>\n<td>0.7251</td>\n<td>0.719</td>\n<td>0.731</td>\n</tr>\n</tbody>\n</table>\n<h2><code>DebertaV2TokenizerFast</code>?</h2>\n<p>Unfortunately, deberta-v2 does not have fast-version of its tokenizer. The slow-version does not support <code>offset_mapping</code>, so we need the fast one. Fortunately, <a href=\"https://github.com/huggingface/transformers/pull/14928\" target=\"_blank\">the code</a> is already written and I could train deberta-v2 and deberta-v3 without tokenization errors. But the problem was the performance of models. CV scores were around ~0.68, even worse than longformers. After some experiments, I observed that <strong>the deberta-v2 tokenizer removes line-break <code>\\n</code> characters.</strong> As I mentioned above, including <code>\\n</code> characters is necessary, so I changed the code as below:<br>\n<img src=\"https://i.ibb.co/JFz2mNK/image.png\" alt=\"\"><br>\nFinally I can get the correct scores from deberta-v2 and deberta-v3 models.</p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>cv</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v2-xlarge</td>\n<td>0.7019</td>\n<td>0.705</td>\n<td>0.714</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>0.7038</td>\n<td>0.707</td>\n<td>0.719</td>\n</tr>\n</tbody>\n</table>\n<p>I also tried larger models (e.g. deberta-v2-xxlarge) but they are all worse than the above models. Thus I only used up to deberta-xlarge scale.</p>\n<h2>Hyperparameters</h2>\n<p>I know many people used small batch size like 2 or 4, but I use 8 to activate tensor-cores on A100 😁 I had to train the model faster because I had no time. All models are trained for 5k/7k steps with 500/700 warmup steps. The learning rate is basically 3e-5, but depends on the scale. The detailed configurations are in my code as well. The learning rate is decayed linearly. I applied gradient clipping with <code>1.0</code> and gradient checkpointing to reduce the memory usage. AdamW optimizer is used, and I evaluate 20 times per epoch to save the best-scored model. Note that I use exactly same evaluation metric (overlap-based f1 score) to avoid the score mismatch from validation to lb score.</p>\n<h2>Conclusion</h2>\n<p>I tried many combinations of the models to submit because there is a time limit. The belows are my last three submissions🥲<br>\n<img src=\"https://i.ibb.co/9pmKZbb/image.png\" alt=\"\"><br>\nAnd these are my final selections:<br>\n<img src=\"https://i.ibb.co/Dp7DGNn/image.png\" alt=\"\"></p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-large (10 folds) + deberta-xlarge (3/5) + deberta-v2-xlarge (4/5)</td>\n<td>0.721</td>\n<td>0.735</td>\n</tr>\n<tr>\n<td>deberta-large + deberta-v3-large + deberta-v3-large + deberta-xlarge (3/5) + deberta-v2-xlarge (3/5)</td>\n<td>0.724</td>\n<td>0.735</td>\n</tr>\n</tbody>\n</table>\n<p>Since deberta-xlarge and deberta-v2-xlarge are too heavy to run all, I only use some of them.</p>\n<p><br>\n<a href=\"https://github.com/affjljoo3581/Feedback-Prize-Competition\" target=\"_blank\">Here is a source code in github!</a></p>\n<p>P.S. I also tried pseudo-labeling for wikipedia talk corpus (150k), it did not work. Maybe more data is required…?</p>",
  "comments": [
    {
      "id": 1905721,
      "author_name": "Chouchou",
      "author_url": "",
      "post_date": "2022-08-19T08:49:28.123000",
      "content": "<p>Really like your work ! Rich analysis !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1837632,
      "author_name": "Lei Wang",
      "author_url": "",
      "post_date": "2022-06-29T18:30:13.283000",
      "content": "<p>nice work, congratulations for the solo gold medal 🎖️ and thank you very much for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1849880,
      "author_name": "Marília Prata",
      "author_url": "",
      "post_date": "2022-07-09T23:00:10.400000",
      "content": "<p>Congrats for your Solo Gold. Thanks for the write-up.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1729831,
      "author_name": "assaf",
      "author_url": "",
      "post_date": "2022-03-20T15:17:50.920000",
      "content": "<p>very nice !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1729797,
      "author_name": "Dmitry Lakhov",
      "author_url": "",
      "post_date": "2022-03-20T14:27:27.357000",
      "content": "<p>nice work , keep it up🔥</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1729465,
      "author_name": "SOYOUNG PARK",
      "author_url": "",
      "post_date": "2022-03-20T05:45:43.003000",
      "content": "<p>Congrats on solo winner !!!!! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1729282,
      "author_name": "Mahmoud Fahmy",
      "author_url": "",
      "post_date": "2022-03-19T22:13:48.193000",
      "content": "<p>Congratulations!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1728814,
      "author_name": "umarchi",
      "author_url": "",
      "post_date": "2022-03-19T10:04:01.803000",
      "content": "<p>Congrats on your first solo gold. I'm so impressed.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1727819,
      "author_name": "xbustc",
      "author_url": "",
      "post_date": "2022-03-18T10:57:19.680000",
      "content": "<p>Congrats! The beam searching method is very amazing, how high is the improvement from it?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1728086,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-18T15:17:44.153000",
          "content": "<p>Well… actually I didn't strictly compare the performance between greedy and beam search</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1727816,
      "author_name": "iamcalculating",
      "author_url": "",
      "post_date": "2022-03-18T10:49:11.957000",
      "content": "<p>Really nice on the analysis of \"\\n\" problem. Now I know why my notebook is outperformed by public kernals for all models…Thanks indeed.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1727643,
      "author_name": "Niek van der Zwaag",
      "author_url": "",
      "post_date": "2022-03-18T06:24:49.163000",
      "content": "<p>Congrats on solo gold 🎉 Thanks for your explanation and source code 🙌</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1726233,
      "author_name": "Francis Brempong",
      "author_url": "",
      "post_date": "2022-03-17T19:30:25.347000",
      "content": "<p>A solo gold medal! 💪 That's impressive  <a href=\"https://www.kaggle.com/affjljoo3581\" target=\"_blank\">@affjljoo3581</a> Congrats. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1724973,
      "author_name": "Atharva Ingle",
      "author_url": "",
      "post_date": "2022-03-16T17:16:49.653000",
      "content": "<p>A tremendous amount of work and congrats, great to see you get a solo gold medal!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1725040,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T18:48:59.243000",
          "content": "<p>thank you 👍👍👍👍</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1724869,
      "author_name": "TianyuZhu",
      "author_url": "",
      "post_date": "2022-03-16T15:31:06.063000",
      "content": "<p>Thanks for the detailed summary and the investigation on the inferior performance of deberta-v2/v3 (with default fast tokenizer) compared with v1. <br>\nCongrats on finishing with solo gold. 🎉<br>\n(btw, the code snippets you shared are really clean and tidy.👍</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1724859,
      "author_name": "chumajin",
      "author_url": "",
      "post_date": "2022-03-16T15:17:29.287000",
      "content": "<p>Congratulations on Solo Gold. This is a great strategy and commentary. Thank you for sharing !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1724841,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-03-16T15:05:24.557000",
      "content": "<p>\"P.S. I also tried pseudo-labeling for wikipedia talk corpus (150k), it did not work. Maybe more data is required…?\" I think pl not work due to lower precision of this data/model. If it works, Chunming would have used them😋</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1724852,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T15:14:42.160000",
          "content": "<p>Oh that's right :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1724832,
      "author_name": "kkiller",
      "author_url": "",
      "post_date": "2022-03-16T14:48:53.110000",
      "content": "<p>We were desperately trying to pass you on the LB, but after reading all your hard works, I think the ranking is Ok as it is :). Congratulations on your first and solo gold medal !</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1724854,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T15:15:58.267000",
          "content": "<p>Thanks! You made me really nervous 😣</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1724876,
          "author_name": "kkiller",
          "author_url": "",
          "post_date": "2022-03-16T15:41:40.910000",
          "content": "<p>You're welcome mate 😄</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1724816,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-03-16T14:34:52.783000",
      "content": "<p>I also notices you ensemble all in deberta but you ensemble with many folds. How much gain will many folds model ensemble get, comparing to submit using only models train on all 15k training data?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1724825,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T14:42:23.957000",
          "content": "<p>I cannot understand what many folds mean, but I think kfolds and train-val splits are not pretty different if the number of models are same</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1724828,
          "author_name": "gezi",
          "author_url": "",
          "post_date": "2022-03-16T14:46:32.357000",
          "content": "<p>I mean, did you test for deberta-v3 for example, submitting kfold like 5 deberta-v3 models, does it work much better then submitting 1 deberta-v3 model training on all 15k training data.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1724851,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T15:14:09.560000",
          "content": "<p>I didn't train with all 15k training data because I needed a validation set to evaluate the model. If you mean just single-fold model, yes it indeed performs much better. Or not, I didn't try full training dataset. I always prepare the validation set by splitting from the train set.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1724864,
          "author_name": "gezi",
          "author_url": "",
          "post_date": "2022-03-16T15:21:57.857000",
          "content": "<p>Got it thanks!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1724812,
      "author_name": "TelmaZzzz",
      "author_url": "",
      "post_date": "2022-03-16T14:30:59.230000",
      "content": "<p>Congratulations. And thanks for your sharing, I find the reason why my deberta-v3-large is much worse than deberta-large.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1724817,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T14:35:32.350000",
          "content": "<p>Yes… I was also surprised when v2 and v3 models worked worse</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1724801,
      "author_name": "ds.wook",
      "author_url": "",
      "post_date": "2022-03-16T14:21:45.747000",
      "content": "<p>Congratulations on winning the gold medal!! I think your solution is very nice</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1725956,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2022-03-17T14:34:02.163000",
      "content": "<p>Congrats on solo gold and prize money. A lot of works and thanks for sharing solution <a href=\"https://www.kaggle.com/affjljoo3581\" target=\"_blank\">@affjljoo3581</a> </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1724809,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-03-16T14:26:49.123000",
      "content": "<p>Congrats for solo gold! Really nice sharing！<br>\nGreat idea of using beam search. How much does it increase score on LB/PB?<br>\nFor deberta-v3-large and deberta-v2-xlarge did you train using the same parameter? I got similar result as you using deberta-v3-large, however I could not make deberta-v2-xlarge model convergent.<br>\nAlso agree with you, if someone say deberta-v3/v2 perform worse then deberta-v1 then it is due to tokenizer which remove \\n.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1724814,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-16T14:33:58.010000",
          "content": "<p>I have no idea how much does the beam search increase the score because I already applied it before using deberta models… I didn't compare the scores strictly.<br>\nAnd here is a configuration of the deberta-v2-xlarge:</p>\n<pre><code>optim:\n  optimizer:\n    lr: 7e-6\n    betas: [0.9, 0.999]\n    eps: 1e-6\n    weight_decay: 0.01\n  scheduler:\n    name: linear\n    num_warmup_steps: 700\n    num_training_steps: 7000\n\ntrain:\n  name: deberta-v2-xlarge\n  batch_size: 8\n  accumulate_grads: 1\n  max_grad_norm: 1.0\n  gradient_checkpointing: true\n  validation_interval: 0.05\n  logging_interval: 10\n  evaluate_after_steps: 4000\n  save_best_checkpoint: true\n  precision: 16\n  gpus: 1\n</code></pre>\n<p>But it also worked with similar scores:</p>\n<pre><code>optim:\n  optimizer:\n    lr: 1e-5\n    betas: [0.9, 0.999]\n    eps: 1e-6\n    weight_decay: 0.01\n  scheduler:\n    name: linear\n    num_warmup_steps: 700\n    num_training_steps: 7000\n</code></pre>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 1724818,
          "author_name": "gezi",
          "author_url": "",
          "post_date": "2022-03-16T14:35:39.090000",
          "content": "<p>Thanks will investigate your params:)</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1727767,
          "author_name": "patriot",
          "author_url": "",
          "post_date": "2022-03-18T09:08:08.837000",
          "content": "<p>In my case, applying beam search for deberta-L-5fold changed private/public:708/702→709/700.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1871073,
      "author_name": "Trunway",
      "author_url": "",
      "post_date": "2022-07-26T04:11:49.613000",
      "content": "<p>Thank you for your sharing. But I do not understand the <code>resolve_encodings_and_normalize</code> function, mainly two problems.</p>\n<ol>\n<li><p>I used part of the sentence \"successes. ÃÅf you\" to test the function, but the result is \"successes. AAf you\", and my code is as follows.<br>\n<code>unidecode(\"successes. ÃÅf you\".encode(\"raw_unicode_escape\")</code><br>\n                                                  <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")</code><br>\n                                                 <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code></p></li>\n<li><p>What's the purpose for the repeated <code>decode</code> and <code>encode</code> in the code? The result I got from running this part of the code seems to have no difference from the input \"successes. ÃÅf you\" and when I tried to repeat them more times like what is shown below, I still couldn't see any changes. <br>\n<code>\"successes. ÃÅf you\".encode(\"raw_unicode_escape\")</code><br>\n                               <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")</code><br>\n                                                  <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                 <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code><br>\n                                                <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code><br>\n                                                <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code></p></li>\n</ol>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1829916,
      "author_name": "Alvin Chen",
      "author_url": "",
      "post_date": "2022-06-23T03:38:32.987000",
      "content": "<blockquote>\n  <p>deberta-v3-large (10 folds) + deberta-xlarge (3/5) + deberta-v2-xlarge (4/5)</p>\n</blockquote>\n<p>Hi guys, what does this (3/5, 4/5) mean ?</p>\n<p>Does that mean model trained 5 fold and best 3 or 4 out of 5 were choosen? And plus 10 fold of deberta-v3-large?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1725648,
      "author_name": "SchenbergZ",
      "author_url": "",
      "post_date": "2022-03-17T09:34:07.630000",
      "content": "<p>Congrats! I wan to know more about your ensemble methods: what is your averaging.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1725664,
          "author_name": "Jungwoo Park",
          "author_url": "",
          "post_date": "2022-03-17T09:52:29.857000",
          "content": "<p>The detailed code is <a href=\"https://github.com/affjljoo3581/Feedback-Prize-Competition/blob/master/src/ensemble.py\" target=\"_blank\">here</a>. Basically I grouped entities from the submissions of the models. That means the similarly located and same-class entities will be grouped. And then I averaged the lengths of the entities in each group. This is what I did:</p>\n<pre><code>if args.group_strategy == \"union\":\n    start, end = min(x[1] for x in group), max(x[2] for x in group)\nelif args.group_strategy == \"intersect\":\n    start, end = max(x[1] for x in group), min(x[2] for x in group)\nelif args.group_strategy == \"longest\":\n    start, end, _ = max(\n        ((x[1], x[2], x[2] - x[1] + 1) for x in group), key=lambda i: i[2]\n    )\nelif args.group_strategy == \"shortest\":\n    start, end, _ = min(\n        ((x[1], x[2], x[2] - x[1] + 1) for x in group), key=lambda i: i[2]\n    )\nelif args.group_strategy == \"mean\":\n    start = int(sum(x[1] for x in group) / len(group))\n    end = int(sum(x[2] for x in group) / len(group))\nelse:\n    raise NotImplementedError(f\"{args.group_strategy} is not supported.\")\n</code></pre>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1724909,
      "author_name": "Gianetan",
      "author_url": "",
      "post_date": "2022-03-16T16:10:37.980000",
      "content": "<p><a href=\"https://www.kaggle.com/affjljoo3581\" target=\"_blank\">@affjljoo3581</a> Congrats on your first gold medal!!! It is the victory of cool code with great documentation and hard work. Cheers!!!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1724890,
      "author_name": "Ishan Mehta115",
      "author_url": "",
      "post_date": "2022-03-16T15:48:24.737000",
      "content": "<p>Well done </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 41,
  "messages": [
    {
      "id": 1724796,
      "postDate": "2022-03-16T14:17:08.613Z",
      "content": "<p><a href=\"https://github.com/affjljoo3581/Feedback-Prize-Competition\" target=\"_blank\">Here is a source code in github!</a></p>\n<p>Hi kagglers! Congrats to the winners and everyone who enjoyed this competition! I saw many people were tired, and me as well… But I've got my very first solo gold medal 🎖️ and I'm now really happy to become a Kaggle master!</p>\n<p>To be honest, I participated this competition about 3 weeks ago. Thanks to the community, I can accelerate my experiments and improve the score quickly. If I had more time, maybe I could get more chances… 🤔🤔</p>\n<p>I think some tricks and techniques are important in this competition. <a href=\"https://www.kaggle.com/c/feedback-prize-2021/discussion/313177\" target=\"_blank\">The lightgbm postprocessing</a> is really cool and I've never thought about that idea. I focused on some technical processing and I managed to get a high score.</p>\n<h2>Resolve Encoding Error</h2>\n<p>As you can see, there are some abnormal characters in the documents. Let's see <code>0D1493FDAAD3</code>:</p>\n<pre><code>...\nAnother reason to talk to multiple people when making a purchase is to learn from their successes. ÃÅf you saw a group of people successfully do what you are trying now, learn what they did to overcome the obstacle.\n...\n</code></pre>\n<p>There are nonrecognizable characters <code>ÃÅ</code> and you may see this frequently in other documents. After some tries, I found this is a problem about <code>cp1252</code> and <code>utf8</code> encodings, and the below code can clean the documents.<br>\n<img src=\"https://i.ibb.co/Q9Wzrzr/image.png\" alt=\"\"><br>\nThe result is as follows:</p>\n<pre><code>...\nAnother reason to talk to multiple people when making a purchase is to learn from their successes. If you saw a group of people successfully do what you are trying now, learn what they did to overcome the obstacle.\n...\n</code></pre>\n<p>Actually, It doesn't seem to improve the final scores significantly, but I just applied this to make sure.<br>\nAnd because it reduces the characters in the documents, we have to adjust the <code>discourse_start</code> and <code>discourse_end</code> offsets. To correct the offsets, I use <code>difflib.SequenceMatcher</code> to compare the differences of the characters. The details are in my code.</p>\n<h2><code>word_ids</code> vs <code>offset_mapping</code>?</h2>\n<p>Because many NER examples use <code>word_ids</code> with <code>is_split_into_words=True</code> parameter, I also tried to use this for labeling BIO-named subword tags. However, I got 0.595 public LB with a single-fold bigbird base model. So I tried another way from the community, <code>offset_mapping</code> with <code>return_offsets_mapping=True</code> parameter to make subword NER tags, and I could get 0.630 public LB score.</p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>bigbird-roberta-base (single fold, <code>word_ids</code>)</td>\n<td>0.595</td>\n<td>0.609</td>\n</tr>\n<tr>\n<td>bigbird-roberta-base (single fold)</td>\n<td>0.630</td>\n<td>0.644</td>\n</tr>\n<tr>\n<td>bigbird-roberta-base (5 folds)</td>\n<td>0.659</td>\n<td>0.677</td>\n</tr>\n</tbody>\n</table>\n<p>Why is this happened? What is the difference between <code>word_ids</code> and <code>offset_mapping</code> approach? After trying some experiences, I found that <code>word_ids</code> needs to split the text into words using <code>.split()</code>, and it prunes the line-break information <code>\\n</code>. Since this task is for recognizing the structure of the document, <strong>it would be necessary to use the line-break characters.</strong> Remember this because I will mention this fact in the later section.</p>\n<h2>Beam-Search Decoding</h2>\n<p>BIO-naming rule is quite complex, so the greedy decoding (i.e. argmax from the token predictions) cannot perform well. I observed many cases like:</p>\n<table>\n<thead>\n<tr>\n<th>sorted candidate tags</th>\n<th>sorted corresponding probs</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><code>B-Claim</code> …</td>\n<td>1.0 …</td>\n</tr>\n<tr>\n<td><code>I-Claim</code> …</td>\n<td>0.99  …</td>\n</tr>\n<tr>\n<td><code>I-Claim</code> …</td>\n<td>0.99 …</td>\n</tr>\n<tr>\n<td><code>I-Claim</code> <code>B-Evidence</code> …</td>\n<td>0.49 0.43 …</td>\n</tr>\n<tr>\n<td><code>I-Evidence</code> <code>I-Claim</code> …</td>\n<td>0.99 0.01 …</td>\n</tr>\n<tr>\n<td><code>I-Evidence</code> <code>I-Claim</code> …</td>\n<td>0.99 0.01 …</td>\n</tr>\n</tbody>\n</table>\n<p>According to the greedy decoding, the prediction result would be <code>B-Claim I-Claim I-Claim I-Claim I-Evidence I-Evidence</code>. The entities should be started with B-tags, so the Evidence entity will be dropped even it has high confidence. Therefore I implement beam search decoding algorithm for NER tagging:<br>\n<img src=\"https://i.ibb.co/RjhFk1P/image.png\" alt=\"\"><br>\nI wrote this code on PyTorch and GPU-capable for parallelization. I use <code>beam_size=4</code> for all both evaluation and prediction.</p>\n<h2>Entity-Level Post-Ensemble</h2>\n<p>The above multi-fold predictions are from averaged token probabilities. However, if the architectures are different and the subword tokenizers are different as well, it is impossible to combine the subword token predictions (probabilities). Some people tried to ensemble with character-level probabilities, but it did not work for me. Hence I created entity-level post ensemble algorithm.</p>\n<p>The main idea is from the metric of this competition. This compeition treats more than 50% matched entities as the same group. So I group at least 50% overlapped entities which have same class. After that, I average the ranges in each group. I tested the longest, shortest, union and intersection ways, but they were all worse than the average method.<br>\n<img src=\"https://i.ibb.co/5cCqwYb/image.png\" alt=\"\"><br>\nIt indeed showed significant improvements by ensembling several models! It can be even applied to the folds (I mean, it is useful for the same tokenizer and same subwords as well) and I can get about +0.002 lb score.</p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>cv</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-large (5folds)</td>\n<td>0.6938</td>\n<td>0.705</td>\n<td>0.713</td>\n</tr>\n<tr>\n<td>deberta-large (5folds, entity-level ensemble)</td>\n<td>0.6938</td>\n<td>0.707</td>\n<td>0.714</td>\n</tr>\n<tr>\n<td>deberta-large + deberta-xlarge</td>\n<td>0.718</td>\n<td>0.712</td>\n<td>0.722</td>\n</tr>\n<tr>\n<td>deberta-v3-large-v2 + deberta-v2-xlarge-v2</td>\n<td>0.7251</td>\n<td>0.719</td>\n<td>0.731</td>\n</tr>\n</tbody>\n</table>\n<h2><code>DebertaV2TokenizerFast</code>?</h2>\n<p>Unfortunately, deberta-v2 does not have fast-version of its tokenizer. The slow-version does not support <code>offset_mapping</code>, so we need the fast one. Fortunately, <a href=\"https://github.com/huggingface/transformers/pull/14928\" target=\"_blank\">the code</a> is already written and I could train deberta-v2 and deberta-v3 without tokenization errors. But the problem was the performance of models. CV scores were around ~0.68, even worse than longformers. After some experiments, I observed that <strong>the deberta-v2 tokenizer removes line-break <code>\\n</code> characters.</strong> As I mentioned above, including <code>\\n</code> characters is necessary, so I changed the code as below:<br>\n<img src=\"https://i.ibb.co/JFz2mNK/image.png\" alt=\"\"><br>\nFinally I can get the correct scores from deberta-v2 and deberta-v3 models.</p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>cv</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v2-xlarge</td>\n<td>0.7019</td>\n<td>0.705</td>\n<td>0.714</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>0.7038</td>\n<td>0.707</td>\n<td>0.719</td>\n</tr>\n</tbody>\n</table>\n<p>I also tried larger models (e.g. deberta-v2-xxlarge) but they are all worse than the above models. Thus I only used up to deberta-xlarge scale.</p>\n<h2>Hyperparameters</h2>\n<p>I know many people used small batch size like 2 or 4, but I use 8 to activate tensor-cores on A100 😁 I had to train the model faster because I had no time. All models are trained for 5k/7k steps with 500/700 warmup steps. The learning rate is basically 3e-5, but depends on the scale. The detailed configurations are in my code as well. The learning rate is decayed linearly. I applied gradient clipping with <code>1.0</code> and gradient checkpointing to reduce the memory usage. AdamW optimizer is used, and I evaluate 20 times per epoch to save the best-scored model. Note that I use exactly same evaluation metric (overlap-based f1 score) to avoid the score mismatch from validation to lb score.</p>\n<h2>Conclusion</h2>\n<p>I tried many combinations of the models to submit because there is a time limit. The belows are my last three submissions🥲<br>\n<img src=\"https://i.ibb.co/9pmKZbb/image.png\" alt=\"\"><br>\nAnd these are my final selections:<br>\n<img src=\"https://i.ibb.co/Dp7DGNn/image.png\" alt=\"\"></p>\n<table>\n<thead>\n<tr>\n<th>model name</th>\n<th>public lb score</th>\n<th>private lb score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-large (10 folds) + deberta-xlarge (3/5) + deberta-v2-xlarge (4/5)</td>\n<td>0.721</td>\n<td>0.735</td>\n</tr>\n<tr>\n<td>deberta-large + deberta-v3-large + deberta-v3-large + deberta-xlarge (3/5) + deberta-v2-xlarge (3/5)</td>\n<td>0.724</td>\n<td>0.735</td>\n</tr>\n</tbody>\n</table>\n<p>Since deberta-xlarge and deberta-v2-xlarge are too heavy to run all, I only use some of them.</p>\n<p><br>\n<a href=\"https://github.com/affjljoo3581/Feedback-Prize-Competition\" target=\"_blank\">Here is a source code in github!</a></p>\n<p>P.S. I also tried pseudo-labeling for wikipedia talk corpus (150k), it did not work. Maybe more data is required…?</p>",
      "rawMarkdown": "[Here is a source code in github!](https://github.com/affjljoo3581/Feedback-Prize-Competition)\n\nHi kagglers! Congrats to the winners and everyone who enjoyed this competition! I saw many people were tired, and me as well... But I've got my very first solo gold medal 🎖️ and I'm now really happy to become a Kaggle master!\n\nTo be honest, I participated this competition about 3 weeks ago. Thanks to the community, I can accelerate my experiments and improve the score quickly. If I had more time, maybe I could get more chances... 🤔🤔\n\nI think some tricks and techniques are important in this competition. [The lightgbm postprocessing](https://www.kaggle.com/c/feedback-prize-2021/discussion/313177) is really cool and I've never thought about that idea. I focused on some technical processing and I managed to get a high score.\n\n\n## Resolve Encoding Error\nAs you can see, there are some abnormal characters in the documents. Let's see `0D1493FDAAD3`:\n```\n...\nAnother reason to talk to multiple people when making a purchase is to learn from their successes. ÃÅf you saw a group of people successfully do what you are trying now, learn what they did to overcome the obstacle.\n...\n```\n\nThere are nonrecognizable characters `ÃÅ` and you may see this frequently in other documents. After some tries, I found this is a problem about `cp1252` and `utf8` encodings, and the below code can clean the documents.\n![](https://i.ibb.co/Q9Wzrzr/image.png)\nThe result is as follows:\n```\n...\nAnother reason to talk to multiple people when making a purchase is to learn from their successes. If you saw a group of people successfully do what you are trying now, learn what they did to overcome the obstacle.\n...\n```\nActually, It doesn't seem to improve the final scores significantly, but I just applied this to make sure.\nAnd because it reduces the characters in the documents, we have to adjust the `discourse_start` and `discourse_end` offsets. To correct the offsets, I use `difflib.SequenceMatcher` to compare the differences of the characters. The details are in my code.\n\n\n## `word_ids` vs `offset_mapping`?\nBecause many NER examples use `word_ids` with `is_split_into_words=True` parameter, I also tried to use this for labeling BIO-named subword tags. However, I got 0.595 public LB with a single-fold bigbird base model. So I tried another way from the community, `offset_mapping` with `return_offsets_mapping=True` parameter to make subword NER tags, and I could get 0.630 public LB score.\n| model name | public lb score | private lb score |\n|:--:|:--:|:--:|\n| bigbird-roberta-base (single fold, `word_ids`) | 0.595 | 0.609 |\n| bigbird-roberta-base (single fold) | 0.630 | 0.644 |\n| bigbird-roberta-base (5 folds) | 0.659| 0.677 |\n\nWhy is this happened? What is the difference between `word_ids` and `offset_mapping` approach? After trying some experiences, I found that `word_ids` needs to split the text into words using `.split()`, and it prunes the line-break information `\\n`. Since this task is for recognizing the structure of the document, **it would be necessary to use the line-break characters.** Remember this because I will mention this fact in the later section.\n\n\n## Beam-Search Decoding\nBIO-naming rule is quite complex, so the greedy decoding (i.e. argmax from the token predictions) cannot perform well. I observed many cases like:\n| sorted candidate tags | sorted corresponding probs |\n|:--:|:--:|\n| `B-Claim` ... | 1.0 ... |\n| `I-Claim` ... | 0.99  ... |\n| `I-Claim` ... | 0.99 ... |\n| `I-Claim` `B-Evidence` ... | 0.49 0.43 ... |\n| `I-Evidence` `I-Claim` ... | 0.99 0.01 ... |\n| `I-Evidence` `I-Claim` ... | 0.99 0.01 ... |\n\nAccording to the greedy decoding, the prediction result would be `B-Claim I-Claim I-Claim I-Claim I-Evidence I-Evidence`. The entities should be started with B-tags, so the Evidence entity will be dropped even it has high confidence. Therefore I implement beam search decoding algorithm for NER tagging:\n![](https://i.ibb.co/RjhFk1P/image.png)\nI wrote this code on PyTorch and GPU-capable for parallelization. I use `beam_size=4` for all both evaluation and prediction.\n\n\n## Entity-Level Post-Ensemble\nThe above multi-fold predictions are from averaged token probabilities. However, if the architectures are different and the subword tokenizers are different as well, it is impossible to combine the subword token predictions (probabilities). Some people tried to ensemble with character-level probabilities, but it did not work for me. Hence I created entity-level post ensemble algorithm.\n\nThe main idea is from the metric of this competition. This compeition treats more than 50% matched entities as the same group. So I group at least 50% overlapped entities which have same class. After that, I average the ranges in each group. I tested the longest, shortest, union and intersection ways, but they were all worse than the average method.\n![](https://i.ibb.co/5cCqwYb/image.png)\nIt indeed showed significant improvements by ensembling several models! It can be even applied to the folds (I mean, it is useful for the same tokenizer and same subwords as well) and I can get about +0.002 lb score.\n\n| model name | cv | public lb score | private lb score |\n|:--:|:--:|:--:|:--:|\n| deberta-large (5folds) | 0.6938 | 0.705 | 0.713 |\n| deberta-large (5folds, entity-level ensemble) | 0.6938 | 0.707 | 0.714 |\n| deberta-large + deberta-xlarge | 0.718 | 0.712 | 0.722 |\n| deberta-v3-large-v2 + deberta-v2-xlarge-v2 | 0.7251 | 0.719 | 0.731 |\n\n\n## `DebertaV2TokenizerFast`?\nUnfortunately, deberta-v2 does not have fast-version of its tokenizer. The slow-version does not support `offset_mapping`, so we need the fast one. Fortunately, [the code](https://github.com/huggingface/transformers/pull/14928) is already written and I could train deberta-v2 and deberta-v3 without tokenization errors. But the problem was the performance of models. CV scores were around ~0.68, even worse than longformers. After some experiments, I observed that **the deberta-v2 tokenizer removes line-break `\\n` characters.** As I mentioned above, including `\\n` characters is necessary, so I changed the code as below:\n![](https://i.ibb.co/JFz2mNK/image.png)\nFinally I can get the correct scores from deberta-v2 and deberta-v3 models.\n| model name | cv | public lb score | private lb score|\n|:--:|:--:|:--:|:--:|\n| deberta-v2-xlarge | 0.7019 | 0.705 | 0.714 |\n| deberta-v3-large | 0.7038 | 0.707 | 0.719 |\n\nI also tried larger models (e.g. deberta-v2-xxlarge) but they are all worse than the above models. Thus I only used up to deberta-xlarge scale.\n\n\n## Hyperparameters\nI know many people used small batch size like 2 or 4, but I use 8 to activate tensor-cores on A100 😁 I had to train the model faster because I had no time. All models are trained for 5k/7k steps with 500/700 warmup steps. The learning rate is basically 3e-5, but depends on the scale. The detailed configurations are in my code as well. The learning rate is decayed linearly. I applied gradient clipping with `1.0` and gradient checkpointing to reduce the memory usage. AdamW optimizer is used, and I evaluate 20 times per epoch to save the best-scored model. Note that I use exactly same evaluation metric (overlap-based f1 score) to avoid the score mismatch from validation to lb score.\n\n\n## Conclusion\nI tried many combinations of the models to submit because there is a time limit. The belows are my last three submissions🥲\n![](https://i.ibb.co/9pmKZbb/image.png)\nAnd these are my final selections:\n![](https://i.ibb.co/Dp7DGNn/image.png)\n| model name | public lb score | private lb score |\n|:--:|:--:|:--:|\n| deberta-v3-large (10 folds) + deberta-xlarge (3/5) + deberta-v2-xlarge (4/5) | 0.721 | 0.735 |\n| deberta-large + deberta-v3-large + deberta-v3-large + deberta-xlarge (3/5) + deberta-v2-xlarge (3/5) | 0.724 | 0.735 |\n\nSince deberta-xlarge and deberta-v2-xlarge are too heavy to run all, I only use some of them.\n\n~~I am currently cleaning up my code and preparing to upload the repository to my github. If all works are done, I'll add a link for the code.~~\n[Here is a source code in github!](https://github.com/affjljoo3581/Feedback-Prize-Competition)\n\nP.S. I also tried pseudo-labeling for wikipedia talk corpus (150k), it did not work. Maybe more data is required...?",
      "votes": 149
    },
    {
      "id": 1905721,
      "postDate": "2022-08-19T08:49:28.123Z",
      "content": "<p>Really like your work ! Rich analysis !</p>",
      "rawMarkdown": "Really like your work ! Rich analysis !",
      "votes": 1
    },
    {
      "id": 1837632,
      "postDate": "2022-06-29T18:30:13.283Z",
      "content": "<p>nice work, congratulations for the solo gold medal 🎖️ and thank you very much for sharing!</p>",
      "rawMarkdown": "nice work, congratulations for the solo gold medal 🎖️ and thank you very much for sharing!\n",
      "votes": 1
    },
    {
      "id": 1849880,
      "postDate": "2022-07-09T23:00:10.400Z",
      "content": "<p>Congrats for your Solo Gold. Thanks for the write-up.</p>",
      "rawMarkdown": "Congrats for your Solo Gold. Thanks for the write-up."
    },
    {
      "id": 1729831,
      "postDate": "2022-03-20T15:17:50.920Z",
      "content": "<p>very nice !</p>",
      "rawMarkdown": "very nice !",
      "votes": 1
    },
    {
      "id": 1729797,
      "postDate": "2022-03-20T14:27:27.357Z",
      "content": "<p>nice work , keep it up🔥</p>",
      "rawMarkdown": "nice work , keep it up🔥",
      "votes": 1
    },
    {
      "id": 1729465,
      "postDate": "2022-03-20T05:45:43.003Z",
      "content": "<p>Congrats on solo winner !!!!! </p>",
      "rawMarkdown": "Congrats on solo winner !!!!! ",
      "votes": 1
    },
    {
      "id": 1729282,
      "postDate": "2022-03-19T22:13:48.193Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": 1
    },
    {
      "id": 1728814,
      "postDate": "2022-03-19T10:04:01.803Z",
      "content": "<p>Congrats on your first solo gold. I'm so impressed.</p>",
      "rawMarkdown": "Congrats on your first solo gold. I'm so impressed.",
      "votes": 1
    },
    {
      "id": 1727819,
      "postDate": "2022-03-18T10:57:19.680Z",
      "content": "<p>Congrats! The beam searching method is very amazing, how high is the improvement from it?</p>",
      "rawMarkdown": "Congrats! The beam searching method is very amazing, how high is the improvement from it?",
      "votes": 1,
      "replies": [
        {
          "id": 1728086,
          "postDate": "2022-03-18T15:17:44.153Z",
          "content": "<p>Well… actually I didn't strictly compare the performance between greedy and beam search</p>",
          "rawMarkdown": "Well... actually I didn't strictly compare the performance between greedy and beam search"
        }
      ]
    },
    {
      "id": 1727816,
      "postDate": "2022-03-18T10:49:11.957Z",
      "content": "<p>Really nice on the analysis of \"\\n\" problem. Now I know why my notebook is outperformed by public kernals for all models…Thanks indeed.</p>",
      "rawMarkdown": "Really nice on the analysis of \"\\n\" problem. Now I know why my notebook is outperformed by public kernals for all models...Thanks indeed.",
      "votes": 1
    },
    {
      "id": 1727643,
      "postDate": "2022-03-18T06:24:49.163Z",
      "content": "<p>Congrats on solo gold 🎉 Thanks for your explanation and source code 🙌</p>",
      "rawMarkdown": "Congrats on solo gold 🎉 Thanks for your explanation and source code 🙌",
      "votes": 1
    },
    {
      "id": 1726233,
      "postDate": "2022-03-17T19:30:25.347Z",
      "content": "<p>A solo gold medal! 💪 That's impressive  <a href=\"https://www.kaggle.com/affjljoo3581\" target=\"_blank\">@affjljoo3581</a> Congrats. </p>",
      "rawMarkdown": "A solo gold medal! 💪 That's impressive  @affjljoo3581 Congrats. ",
      "votes": 1
    },
    {
      "id": 1724973,
      "postDate": "2022-03-16T17:16:49.653Z",
      "content": "<p>A tremendous amount of work and congrats, great to see you get a solo gold medal!</p>",
      "rawMarkdown": "A tremendous amount of work and congrats, great to see you get a solo gold medal!",
      "votes": 1,
      "replies": [
        {
          "id": 1725040,
          "postDate": "2022-03-16T18:48:59.243Z",
          "content": "<p>thank you 👍👍👍👍</p>",
          "rawMarkdown": "thank you 👍👍👍👍",
          "votes": 1
        }
      ]
    },
    {
      "id": 1724869,
      "postDate": "2022-03-16T15:31:06.063Z",
      "content": "<p>Thanks for the detailed summary and the investigation on the inferior performance of deberta-v2/v3 (with default fast tokenizer) compared with v1. <br>\nCongrats on finishing with solo gold. 🎉<br>\n(btw, the code snippets you shared are really clean and tidy.👍</p>",
      "rawMarkdown": "Thanks for the detailed summary and the investigation on the inferior performance of deberta-v2/v3 (with default fast tokenizer) compared with v1. \nCongrats on finishing with solo gold. 🎉\n(btw, the code snippets you shared are really clean and tidy.👍",
      "votes": 1
    },
    {
      "id": 1724859,
      "postDate": "2022-03-16T15:17:29.287Z",
      "content": "<p>Congratulations on Solo Gold. This is a great strategy and commentary. Thank you for sharing !</p>",
      "rawMarkdown": "Congratulations on Solo Gold. This is a great strategy and commentary. Thank you for sharing !",
      "votes": 1
    },
    {
      "id": 1724841,
      "postDate": "2022-03-16T15:05:24.557Z",
      "content": "<p>\"P.S. I also tried pseudo-labeling for wikipedia talk corpus (150k), it did not work. Maybe more data is required…?\" I think pl not work due to lower precision of this data/model. If it works, Chunming would have used them😋</p>",
      "rawMarkdown": "\"P.S. I also tried pseudo-labeling for wikipedia talk corpus (150k), it did not work. Maybe more data is required…?\" I think pl not work due to lower precision of this data/model. If it works, Chunming would have used them😋",
      "votes": 1,
      "replies": [
        {
          "id": 1724852,
          "postDate": "2022-03-16T15:14:42.160Z",
          "content": "<p>Oh that's right :)</p>",
          "rawMarkdown": "Oh that's right :)"
        }
      ]
    },
    {
      "id": 1724832,
      "postDate": "2022-03-16T14:48:53.110Z",
      "content": "<p>We were desperately trying to pass you on the LB, but after reading all your hard works, I think the ranking is Ok as it is :). Congratulations on your first and solo gold medal !</p>",
      "rawMarkdown": "We were desperately trying to pass you on the LB, but after reading all your hard works, I think the ranking is Ok as it is :). Congratulations on your first and solo gold medal !",
      "votes": 1,
      "replies": [
        {
          "id": 1724854,
          "postDate": "2022-03-16T15:15:58.267Z",
          "content": "<p>Thanks! You made me really nervous 😣</p>",
          "rawMarkdown": "Thanks! You made me really nervous 😣",
          "votes": 1
        },
        {
          "id": 1724876,
          "postDate": "2022-03-16T15:41:40.910Z",
          "content": "<p>You're welcome mate 😄</p>",
          "rawMarkdown": "You're welcome mate 😄"
        }
      ]
    },
    {
      "id": 1724816,
      "postDate": "2022-03-16T14:34:52.783Z",
      "content": "<p>I also notices you ensemble all in deberta but you ensemble with many folds. How much gain will many folds model ensemble get, comparing to submit using only models train on all 15k training data?</p>",
      "rawMarkdown": "I also notices you ensemble all in deberta but you ensemble with many folds. How much gain will many folds model ensemble get, comparing to submit using only models train on all 15k training data?",
      "votes": 1,
      "replies": [
        {
          "id": 1724825,
          "postDate": "2022-03-16T14:42:23.957Z",
          "content": "<p>I cannot understand what many folds mean, but I think kfolds and train-val splits are not pretty different if the number of models are same</p>",
          "rawMarkdown": "I cannot understand what many folds mean, but I think kfolds and train-val splits are not pretty different if the number of models are same",
          "votes": 1
        },
        {
          "id": 1724828,
          "postDate": "2022-03-16T14:46:32.357Z",
          "content": "<p>I mean, did you test for deberta-v3 for example, submitting kfold like 5 deberta-v3 models, does it work much better then submitting 1 deberta-v3 model training on all 15k training data.</p>",
          "rawMarkdown": "I mean, did you test for deberta-v3 for example, submitting kfold like 5 deberta-v3 models, does it work much better then submitting 1 deberta-v3 model training on all 15k training data.",
          "votes": 1
        },
        {
          "id": 1724851,
          "postDate": "2022-03-16T15:14:09.560Z",
          "content": "<p>I didn't train with all 15k training data because I needed a validation set to evaluate the model. If you mean just single-fold model, yes it indeed performs much better. Or not, I didn't try full training dataset. I always prepare the validation set by splitting from the train set.</p>",
          "rawMarkdown": "I didn't train with all 15k training data because I needed a validation set to evaluate the model. If you mean just single-fold model, yes it indeed performs much better. Or not, I didn't try full training dataset. I always prepare the validation set by splitting from the train set.",
          "votes": 2
        },
        {
          "id": 1724864,
          "postDate": "2022-03-16T15:21:57.857Z",
          "content": "<p>Got it thanks!</p>",
          "rawMarkdown": "Got it thanks!",
          "votes": 2
        }
      ]
    },
    {
      "id": 1724812,
      "postDate": "2022-03-16T14:30:59.230Z",
      "content": "<p>Congratulations. And thanks for your sharing, I find the reason why my deberta-v3-large is much worse than deberta-large.</p>",
      "rawMarkdown": "Congratulations. And thanks for your sharing, I find the reason why my deberta-v3-large is much worse than deberta-large.",
      "votes": 1,
      "replies": [
        {
          "id": 1724817,
          "postDate": "2022-03-16T14:35:32.350Z",
          "content": "<p>Yes… I was also surprised when v2 and v3 models worked worse</p>",
          "rawMarkdown": "Yes... I was also surprised when v2 and v3 models worked worse",
          "votes": 2
        }
      ]
    },
    {
      "id": 1724801,
      "postDate": "2022-03-16T14:21:45.747Z",
      "content": "<p>Congratulations on winning the gold medal!! I think your solution is very nice</p>",
      "rawMarkdown": "Congratulations on winning the gold medal!! I think your solution is very nice",
      "votes": 1
    },
    {
      "id": 1725956,
      "postDate": "2022-03-17T14:34:02.163Z",
      "content": "<p>Congrats on solo gold and prize money. A lot of works and thanks for sharing solution <a href=\"https://www.kaggle.com/affjljoo3581\" target=\"_blank\">@affjljoo3581</a> </p>",
      "rawMarkdown": "Congrats on solo gold and prize money. A lot of works and thanks for sharing solution @affjljoo3581 ",
      "votes": 2
    },
    {
      "id": 1724809,
      "postDate": "2022-03-16T14:26:49.123Z",
      "content": "<p>Congrats for solo gold! Really nice sharing！<br>\nGreat idea of using beam search. How much does it increase score on LB/PB?<br>\nFor deberta-v3-large and deberta-v2-xlarge did you train using the same parameter? I got similar result as you using deberta-v3-large, however I could not make deberta-v2-xlarge model convergent.<br>\nAlso agree with you, if someone say deberta-v3/v2 perform worse then deberta-v1 then it is due to tokenizer which remove \\n.</p>",
      "rawMarkdown": "Congrats for solo gold! Really nice sharing！\nGreat idea of using beam search. How much does it increase score on LB/PB?\nFor deberta-v3-large and deberta-v2-xlarge did you train using the same parameter? I got similar result as you using deberta-v3-large, however I could not make deberta-v2-xlarge model convergent.\nAlso agree with you, if someone say deberta-v3/v2 perform worse then deberta-v1 then it is due to tokenizer which remove \\n.",
      "votes": 2,
      "replies": [
        {
          "id": 1724814,
          "postDate": "2022-03-16T14:33:58.010Z",
          "content": "<p>I have no idea how much does the beam search increase the score because I already applied it before using deberta models… I didn't compare the scores strictly.<br>\nAnd here is a configuration of the deberta-v2-xlarge:</p>\n<pre><code>optim:\n  optimizer:\n    lr: 7e-6\n    betas: [0.9, 0.999]\n    eps: 1e-6\n    weight_decay: 0.01\n  scheduler:\n    name: linear\n    num_warmup_steps: 700\n    num_training_steps: 7000\n\ntrain:\n  name: deberta-v2-xlarge\n  batch_size: 8\n  accumulate_grads: 1\n  max_grad_norm: 1.0\n  gradient_checkpointing: true\n  validation_interval: 0.05\n  logging_interval: 10\n  evaluate_after_steps: 4000\n  save_best_checkpoint: true\n  precision: 16\n  gpus: 1\n</code></pre>\n<p>But it also worked with similar scores:</p>\n<pre><code>optim:\n  optimizer:\n    lr: 1e-5\n    betas: [0.9, 0.999]\n    eps: 1e-6\n    weight_decay: 0.01\n  scheduler:\n    name: linear\n    num_warmup_steps: 700\n    num_training_steps: 7000\n</code></pre>",
          "rawMarkdown": "I have no idea how much does the beam search increase the score because I already applied it before using deberta models... I didn't compare the scores strictly.\nAnd here is a configuration of the deberta-v2-xlarge:\n```yaml\noptim:\n  optimizer:\n    lr: 7e-6\n    betas: [0.9, 0.999]\n    eps: 1e-6\n    weight_decay: 0.01\n  scheduler:\n    name: linear\n    num_warmup_steps: 700\n    num_training_steps: 7000\n\ntrain:\n  name: deberta-v2-xlarge\n  batch_size: 8\n  accumulate_grads: 1\n  max_grad_norm: 1.0\n  gradient_checkpointing: true\n  validation_interval: 0.05\n  logging_interval: 10\n  evaluate_after_steps: 4000\n  save_best_checkpoint: true\n  precision: 16\n  gpus: 1\n```\nBut it also worked with similar scores:\n```yaml\noptim:\n  optimizer:\n    lr: 1e-5\n    betas: [0.9, 0.999]\n    eps: 1e-6\n    weight_decay: 0.01\n  scheduler:\n    name: linear\n    num_warmup_steps: 700\n    num_training_steps: 7000\n```",
          "votes": 4
        },
        {
          "id": 1724818,
          "postDate": "2022-03-16T14:35:39.090Z",
          "content": "<p>Thanks will investigate your params:)</p>",
          "rawMarkdown": "Thanks will investigate your params:)",
          "votes": 1
        },
        {
          "id": 1727767,
          "postDate": "2022-03-18T09:08:08.837Z",
          "content": "<p>In my case, applying beam search for deberta-L-5fold changed private/public:708/702→709/700.</p>",
          "rawMarkdown": "In my case, applying beam search for deberta-L-5fold changed private/public:708/702→709/700.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1871073,
      "postDate": "2022-07-26T04:11:49.613Z",
      "content": "<p>Thank you for your sharing. But I do not understand the <code>resolve_encodings_and_normalize</code> function, mainly two problems.</p>\n<ol>\n<li><p>I used part of the sentence \"successes. ÃÅf you\" to test the function, but the result is \"successes. AAf you\", and my code is as follows.<br>\n<code>unidecode(\"successes. ÃÅf you\".encode(\"raw_unicode_escape\")</code><br>\n                                                  <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")</code><br>\n                                                 <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code></p></li>\n<li><p>What's the purpose for the repeated <code>decode</code> and <code>encode</code> in the code? The result I got from running this part of the code seems to have no difference from the input \"successes. ÃÅf you\" and when I tried to repeat them more times like what is shown below, I still couldn't see any changes. <br>\n<code>\"successes. ÃÅf you\".encode(\"raw_unicode_escape\")</code><br>\n                               <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")</code><br>\n                                                  <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                 <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code><br>\n                                                <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code><br>\n                                                <code>.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")</code><br>\n                                                <code>.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))</code></p></li>\n</ol>",
      "rawMarkdown": "Thank you for your sharing. But I do not understand the `resolve_encodings_and_normalize` function, mainly two problems.\n1. I used part of the sentence \"successes. ÃÅf you\" to test the function, but the result is \"successes. AAf you\", and my code is as follows.\n`unidecode(\"successes. ÃÅf you\".encode(\"raw_unicode_escape\")`\n                                                      `.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")`\n                                                     `.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")`\n                                                    `.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))`\n\n2. What's the purpose for the repeated `decode` and `encode` in the code? The result I got from running this part of the code seems to have no difference from the input \"successes. ÃÅf you\" and when I tried to repeat them more times like what is shown below, I still couldn't see any changes. \n`\"successes. ÃÅf you\".encode(\"raw_unicode_escape\")`\n                                   `.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")`\n                                                      `.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")`\n                                                     `.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))`\n                                                    `.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")`\n                                                    `.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))`\n                                                    `.encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")`\n                                                    `.decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\"))`\n"
    },
    {
      "id": 1829916,
      "postDate": "2022-06-23T03:38:32.987Z",
      "content": "<blockquote>\n  <p>deberta-v3-large (10 folds) + deberta-xlarge (3/5) + deberta-v2-xlarge (4/5)</p>\n</blockquote>\n<p>Hi guys, what does this (3/5, 4/5) mean ?</p>\n<p>Does that mean model trained 5 fold and best 3 or 4 out of 5 were choosen? And plus 10 fold of deberta-v3-large?</p>",
      "rawMarkdown": "> deberta-v3-large (10 folds) + deberta-xlarge (3/5) + deberta-v2-xlarge (4/5)\n\nHi guys, what does this (3/5, 4/5) mean ?\n\nDoes that mean model trained 5 fold and best 3 or 4 out of 5 were choosen? And plus 10 fold of deberta-v3-large?"
    },
    {
      "id": 1725648,
      "postDate": "2022-03-17T09:34:07.630Z",
      "content": "<p>Congrats! I wan to know more about your ensemble methods: what is your averaging.</p>",
      "rawMarkdown": "Congrats! I wan to know more about your ensemble methods: what is your averaging.",
      "replies": [
        {
          "id": 1725664,
          "postDate": "2022-03-17T09:52:29.857Z",
          "content": "<p>The detailed code is <a href=\"https://github.com/affjljoo3581/Feedback-Prize-Competition/blob/master/src/ensemble.py\" target=\"_blank\">here</a>. Basically I grouped entities from the submissions of the models. That means the similarly located and same-class entities will be grouped. And then I averaged the lengths of the entities in each group. This is what I did:</p>\n<pre><code>if args.group_strategy == \"union\":\n    start, end = min(x[1] for x in group), max(x[2] for x in group)\nelif args.group_strategy == \"intersect\":\n    start, end = max(x[1] for x in group), min(x[2] for x in group)\nelif args.group_strategy == \"longest\":\n    start, end, _ = max(\n        ((x[1], x[2], x[2] - x[1] + 1) for x in group), key=lambda i: i[2]\n    )\nelif args.group_strategy == \"shortest\":\n    start, end, _ = min(\n        ((x[1], x[2], x[2] - x[1] + 1) for x in group), key=lambda i: i[2]\n    )\nelif args.group_strategy == \"mean\":\n    start = int(sum(x[1] for x in group) / len(group))\n    end = int(sum(x[2] for x in group) / len(group))\nelse:\n    raise NotImplementedError(f\"{args.group_strategy} is not supported.\")\n</code></pre>",
          "rawMarkdown": "The detailed code is [here](https://github.com/affjljoo3581/Feedback-Prize-Competition/blob/master/src/ensemble.py). Basically I grouped entities from the submissions of the models. That means the similarly located and same-class entities will be grouped. And then I averaged the lengths of the entities in each group. This is what I did:\n```python\nif args.group_strategy == \"union\":\n    start, end = min(x[1] for x in group), max(x[2] for x in group)\nelif args.group_strategy == \"intersect\":\n    start, end = max(x[1] for x in group), min(x[2] for x in group)\nelif args.group_strategy == \"longest\":\n    start, end, _ = max(\n        ((x[1], x[2], x[2] - x[1] + 1) for x in group), key=lambda i: i[2]\n    )\nelif args.group_strategy == \"shortest\":\n    start, end, _ = min(\n        ((x[1], x[2], x[2] - x[1] + 1) for x in group), key=lambda i: i[2]\n    )\nelif args.group_strategy == \"mean\":\n    start = int(sum(x[1] for x in group) / len(group))\n    end = int(sum(x[2] for x in group) / len(group))\nelse:\n    raise NotImplementedError(f\"{args.group_strategy} is not supported.\")\n```",
          "votes": 1
        }
      ]
    },
    {
      "id": 1724909,
      "postDate": "2022-03-16T16:10:37.980Z",
      "content": "<p><a href=\"https://www.kaggle.com/affjljoo3581\" target=\"_blank\">@affjljoo3581</a> Congrats on your first gold medal!!! It is the victory of cool code with great documentation and hard work. Cheers!!!</p>",
      "rawMarkdown": "@affjljoo3581 Congrats on your first gold medal!!! It is the victory of cool code with great documentation and hard work. Cheers!!!"
    },
    {
      "id": 1724890,
      "postDate": "2022-03-16T15:48:24.737Z",
      "content": "<p>Well done </p>",
      "rawMarkdown": "Well done "
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T12:37:52Z",
  "competition_id": 31779,
  "competition_slug": "feedback-prize-2021",
  "team_id": 7898311,
  "solution_url": "/competitions/feedback-prize-2021/writeups/jungwoo-park-4th-place-solution-my-first-gold-meda"
}