{
  "id": 347433,
  "title": "3rd Place Solution - Span MLM + T5 Augmentations",
  "content": "<p>A big thanks to Kaggle &amp; the completion hosts for introducing the <code>Efficiency Track</code>, which is an amazing addition and will surely lead to more creative solutions. This is a detailed version of our solution for the <code>Feedback Prize - Predicting Effective Arguments</code> competition. </p>\n<h1>Links</h1>\n<ul>\n<li>TL;DR: <a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347371\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347371</a></li>\n<li>Code: <a href=\"https://github.com/rbiswasfc/kaggle-feedback-effectiveness-3rd-place-solution\" target=\"_blank\">https://github.com/rbiswasfc/kaggle-feedback-effectiveness-3rd-place-solution</a></li>\n<li>Inference Notebook: <a href=\"https://www.kaggle.com/code/conjuring92/ens58-lstm-lgb-all/notebook?scriptVersionId=104014668\" target=\"_blank\">https://www.kaggle.com/code/conjuring92/ens58-lstm-lgb-all/notebook?scriptVersionId=104014668</a></li>\n</ul>\n<h1>Model Architecture</h1>\n<p>Overall, we followed the span classification approach with model architecture as below<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F1b6b3d2fbb7551da6de6fe45f3f00f94%2FScreenshot%202022-08-24%20at%2010.40.29%20AM.png?generation=1661308851093353&amp;alt=media\" alt=\"\"></p>\n<h1>Pre-Processing</h1>\n<p>We pre-processed each essay to insert newly added span start and span end tokens for each discourse types. To provide more context to our models, we added a topic segment in this format <code>[TOPIC] &lt;prompt_text&gt; [TOPIC END] &lt;essay&gt;</code>. As a minor detail, we inserted <code>[SOE]</code> and <code>[EOE]</code> tokens to indicate essay start and end. As an illustration here is an example:</p>\n<pre><code>[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE] Lead [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  Position [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] \n\n...\n\nConcluding Statement [CONCLUDING_STATEMENT] In conclusion I think this is not a very good idea, but not in the way students should be watched. The way teachers are able to read students' facial expressions tells them how they feel. I don't believe it's important to look at students faces when they can fake their emotions. But, if the teacher is watching you then they're gonna get angry. This is how I feel on this topic. [CONCLUDING_STATEMENT_END] [EOE]\n</code></pre>\n<h1>Span MLM</h1>\n<p>To train the newly added tokens (e.g. <code>[LEAD]</code>, <code>[POSITION]</code>) and adapting to the specific task domain (i.e. student essays from grade 6-12) we continued the pre-training phase of each backbone (e.g. <code>deberta-v3-large</code>) with masked language modelling (MLM) objective. While standard MLM works alright, we found big boost with the following modifications:</p>\n<ul>\n<li>Changing masking probability to 40-50% instead of 15% used typically. For a detained analysis on masking rate, please refer to this : <strong>Should You Mask 15% in Masked Language Modeling?</strong> <a href=\"https://arxiv.org/abs/2202.08005\" target=\"_blank\">https://arxiv.org/abs/2202.08005</a></li>\n<li>Masking contiguous tokens of length 3-15 instead of regular random masking approach. Out motivation came from this paper: <strong>SpanBERT: Improving Pre-training by Representing and Predicting Spans</strong> <a href=\"https://arxiv.org/abs/1907.10529\" target=\"_blank\">https://arxiv.org/abs/1907.10529</a></li>\n<li>Changing chuck size / max length to 720 to match average essay length in the fine-tuning stage</li>\n</ul>\n<p>Here we used the ~15k essays from the 2021 feedback competition. In addition, we added ~25k synthetically generated essays by leveraging T5-large model. More on this in the next section.</p>\n<h1>T5 Augmentations</h1>\n<p>Since we couldn't find external data sources that worked for us, we explored ways to create our own augmented / synthetic data / essays. To this end, we adopted two approaches</p>\n<p><strong>Label Preserving Training Data Augmentation with T5</strong> Here we used 2022 labelled data to generate synthetic examples, which are directly added to train dataset during fine-tuning step. For this purpose, we first train the T5-large model with a seq2seq (text-to-text) text generation task. To be specific, this was the task input-output format:</p>\n<ul>\n<li>Input Template: <code>Generate text for &lt;discourse effectiveness&gt; &lt;discourse type&gt; | Prompt: &lt;prompt text&gt; | Left Context: &lt;left context&gt; | Right Context: &lt;right context&gt;</code>, where left context is essay up to the discourse element we try to generate and right context is anything after the discourse element</li>\n<li>Output Template: <code>&lt;discourse text&gt;</code> </li>\n</ul>\n<p>Here is one example:</p>\n<p><code>Generate text for Ineffective Evidence | Prompt: Should the electoral college be abolished in favor of popular vote? | Left Context:Dear, State Senator\\n\\nWith the elctoral college vote most of people are not getting what they prefer. For the the electoral college vote, voters vote fro not the president, but for a not slate of electors. &lt;....&gt; majority of the people would rather have the popular vote.| Right Context: the electoral college election consist of the 538 electors. All you have to do is win most of those votes and you win the election. &lt;.....&gt; The electoral college is unfair, outdated, and irrational to all the people</code><br>\n--&gt;<br>\n<code>It does not make any sense to me how the president with the votes does not win the presidential election. The electoral college is unfair for all voters that vote. When you think of a vote and how it works you would think by the most one with the most votes wins cause that pretty much means that most of the people would rather have the most wins over the win that had less votes but more electoral votes.</code></p>\n<p>The trained T5 model was pretty amazing, we couldn't distinguish which one was generated text and which one was original. The augmentations copied the student writing styles, identity masks e.g.  and included effects such as that mentioned by <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> in the previous competition discussion:</p>\n<blockquote>\n  <p>just imagine you are asked to write the essay for homework. you are a bad student and want to copy other people work. you want to do so, such that that your teacher will not know that you are copying other people's work.<br>\n  you can:<br>\n  1) choose an essay of the train data. replace the claim from another essay. (if the other essay has the same position as the first essay)</p>\n</blockquote>\n<p>Reference: <strong>Data Augmentation using Pre-trained Transformer Models</strong> <a href=\"https://arxiv.org/abs/2003.02245\" target=\"_blank\">https://arxiv.org/abs/2003.02245</a></p>\n<p><strong>T5: Non-Label Preserving Essay Augmentation</strong><br>\nWe used this to mainly generate essays for MLM. It's quite similar to previous augmentation task with minor changes</p>\n<ul>\n<li>uses all ~15k essays from feedback 2021 dataset</li>\n<li>model only sees left context </li>\n<li>no discourse effectiveness label is given in input prompt</li>\n</ul>\n<h1>Fine-tuning</h1>\n<p>In the fine-tuning stage, we mixed varying degree of T5 label-preserved augmented data (0-50% of original essays) with actual data. This was pretty much standard with following key highlights</p>\n<ul>\n<li>Inclusion of prompts in essay: This helped models to boost performance on Lead and Position discourse types</li>\n<li>Adversarial Training with AWP</li>\n<li>Mask Augmentations (10-25%)</li>\n<li>Multi-sample dropouts</li>\n<li>Multi-head attention over span representations</li>\n<li>LSTM over transformer encoder representations</li>\n<li>Loss = CE Loss + Focal loss</li>\n<li>Cosine scheduler</li>\n<li>Layer wise learning rate decay</li>\n<li>Mean pooling to extract span representations as opposed to [CLS] token based ones</li>\n</ul>\n<h1>Model Backbones</h1>\n<ul>\n<li>DeBERTa-Large</li>\n<li>DeBERTa-XLarge</li>\n<li>DeBERTa-V3-Large</li>\n<li>Longformer</li>\n<li>LUKE</li>\n</ul>\n<h1>Ensembling</h1>\n<p>We built two meta models based on OOF model probabilities + additional features</p>\n<p><strong>LSTM based</strong></p>\n<ul>\n<li>Used only stage-1 model probabilities as features</li>\n<li>Had better correlations with CV and LB</li>\n</ul>\n<p><strong>LGB based</strong></p>\n<ul>\n<li>Used stage-1 model probabilities as features + additional features</li>\n<li>Count of next line per essay text</li>\n<li>Count of discourse_id per essay_id</li>\n<li>Pos tags features (NN, VB etc)</li>\n</ul>\n<h1>Things that had most impact</h1>\n<ul>\n<li>Span MLM: 0.02 to 0.03</li>\n<li>AWP: 0.005 to 0.01</li>\n<li>Prompts: 0.002 to 0.005</li>\n<li>Direct use of T5 Augmentations: -0.002 to 0.005</li>\n<li>Mask Augmentation + MSD: 0.002 - 0.005</li>\n<li>LSTM + LGB ensemble: 0.002-0.004 </li>\n</ul>\n<h1>Things that added for diversity</h1>\n<ul>\n<li>LUKE model</li>\n<li>Multitask objectives e.g. predicting discourse type, ordered encoding of labels e.g. ineffective -&gt; [0, 0], adequate -&gt; [1, 0], effective -&gt; [1, 1]</li>\n<li>Different sampling ratio of augmented data</li>\n<li>Impact moderation of focal loss with its gamma parameter</li>\n<li>Knowledge distillation</li>\n</ul>\n<h1>How to make training stable?</h1>\n<p>We could train our models up to 6-7 epoch with best results obtained in penultimate/last epoch. The following helped in prolonging and stabilizing the training</p>\n<ul>\n<li>Careful selection of key params e.g. batch size, learning rate, weight decay</li>\n<li>Cosine learning rate scheduler</li>\n<li>Task adaptation with MLM</li>\n<li>AWP</li>\n<li>Mask augmentations + Multi-sample dropouts</li>\n<li>Layer wise learning rate decay</li>\n<li><strong>Full precision training</strong></li>\n</ul>\n<h1>Things that didn't work for us</h1>\n<ul>\n<li>Pseudo labelling / meta pseudo labelling</li>\n<li>on hindsight, this was a big miss for us</li>\n<li>Random augmentation </li>\n<li>UDA - Unsupervised Data Augmentations</li>\n<li>Contrastive Learning</li>\n<li>SWA</li>\n<li>Mixout</li>\n</ul>\n<p>Thanks for reading this far! Hope this helps you in your future NLP competitions! Looking forward to your comments and feedbacks.</p>\n<p>Our team ( <a href=\"https://www.kaggle.com/trushk\" target=\"_blank\">@trushk</a> , <a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a>) dynamics was brilliant throughout the competition. We had a lot of fun and learned so much together. Looking forward to future teaming up! </p>\n<p>Thanks 😊</p>",
  "comments": [
    {
      "id": 1911993,
      "author_name": "Psi",
      "author_url": "",
      "post_date": "2022-08-24T12:42:55.947000",
      "content": "<p>Thanks for the writeup and congrats on the great result! It was a fun back and forth on LB :)</p>\n<p>I assume your synthetic data generation approach is replacing the pseudo labeling approach which is why you did not gain much more out from it. </p>",
      "votes": 3,
      "replies": [
        {
          "id": 1912034,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T13:23:59.467000",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a>, Congratulations on winning the competition on both efficiency and conventional track 🎉🎉🎉 We really enjoyed the fight and believed that pushed each other. On synthetic data, yes i also think it compensated for pseudo labelling partially. However, we only went there as we couldn't make any sort of pseudo labelling work. Soft targets seem to have worked better, it was on our list of things to try but never got to it. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912084,
          "author_name": "Psi",
          "author_url": "",
          "post_date": "2022-08-24T13:50:31.760000",
          "content": "<p>Oh did you only try hard labels? Hard labels did not work AT ALL, you had to use soft labels. One core problem with hard labels is that Adequate is way too prominent there. But even on soft labels it took us quite some time to make use of them properly…</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1912126,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T14:25:19.713000",
          "content": "<p>sadly yes, we only tried many different ways with hard labels. Thanks for the explanation, that must be it! We got distracted in finding an explanation why it doesn't work, instead of making it work. Was partially influenced by prior experience where I mostly found soft and hard targets to perform equally. But every dataset + task have subtle differences.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912263,
          "author_name": "Trushant Kalyanpur",
          "author_url": "",
          "post_date": "2022-08-24T15:58:52.633000",
          "content": "<p>Yes unfortunately we didnt spend enough time to get pseudo labeling to work. Seems like that was the missing piece that would have given us an edge. Looking back I think the 3rd place without pseudo labels is pretty amazing 😄</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912347,
          "author_name": "Psi",
          "author_url": "",
          "post_date": "2022-08-24T16:53:59.077000",
          "content": "<p>Yeah it is amazing! We were so sure that you are using pseudos heavily, specifically meta pseudo labels as you used that in last competition. </p>\n<p>But overall, after a competition ends and you feel like you missed out on something, you have to always consider that there is only limited time and that many ideas. If you would have spent time on pseudos---and it really took us quite some effort to use them properly---you would not have found time to try other ideas that also gave you an edge. So it might balance itself out in the end. </p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 1932053,
      "author_name": "TouTie",
      "author_url": "",
      "post_date": "2022-09-09T08:27:02.420000",
      "content": "<p>Congrats on your amazing work! I learn a lot!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1916576,
      "author_name": "🐳鲲(China)",
      "author_url": "",
      "post_date": "2022-08-28T02:44:40.017000",
      "content": "<p>For the span mlm part, can you recommand some code snippet? thanks!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1918004,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-29T08:41:22.423000",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/nickycan\" target=\"_blank\">@nickycan</a>, for the span mlm part we just modified the <code>DataCollatorForLanguageModeling</code> from HF library. For the modification, we referred to SpanBert git repo, specifically here: <a href=\"https://github.com/facebookresearch/SpanBERT/blob/main/pretraining/fairseq/data/masking.py#L137\" target=\"_blank\">https://github.com/facebookresearch/SpanBERT/blob/main/pretraining/fairseq/data/masking.py#L137</a></p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1911839,
      "author_name": "RDizzl3",
      "author_url": "",
      "post_date": "2022-08-24T10:49:28.387000",
      "content": "<p><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> will you be sharing your training code? Looks like there is a lot to learn here!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1912057,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T13:32:07.740000",
          "content": "<p>sorry, we don't have any immediate plan to make the code public. Thanks for your understanding.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911683,
      "author_name": "Arvind Devarkonda",
      "author_url": "",
      "post_date": "2022-08-24T08:40:13.213000",
      "content": "<p>Congratulation for 3rd place.</p>\n<p>1. What is Contrastive Learning?</p>\n<p>2.what do you mean \"Careful selection of key params e.g. batch size, learning rate, weight decay\" ?,  I can understand careful selection of batch size and lr but what about weight decay. Till now i thought weight decay is always 0.01.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911694,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T08:48:27.893000",
          "content": "<p>Thanks! </p>\n<p>For contrastive learning please refer to this paper:</p>\n<ul>\n<li>SUPERVISED CONTRASTIVE LEARNING FOR PRE-TRAINED LANGUAGE MODEL FINE-TUNING (<a href=\"https://openreview.net/pdf?id=cu7IUiOhujH\" target=\"_blank\">https://openreview.net/pdf?id=cu7IUiOhujH</a>)</li>\n</ul>\n<p>Yes, weight decay of 1e-2 works in most cases but few models are quite specific e.g. electra recommends weight decay to be set to 0. I normally decrease it when i'm training longer…</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911707,
          "author_name": "Arvind Devarkonda",
          "author_url": "",
          "post_date": "2022-08-24T08:54:19.853000",
          "content": "<ol>\n<li><p>Did you used essay as it is  for mlm  or you used this way</p>\n<p>[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE] Lead            [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  Position [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] </p>\n<p>…</p>\n<p>Concluding Statement [CONCLUDING_STATEMENT] In conclusion I think this is not a very good idea, but not in the way students should be watched. The way teachers are able to read students' facial expressions tells them how they feel. I don't believe it's important to look at students faces when they can fake their emotions. But, if the teacher is watching you then they're gonna get angry. This is how I feel on this topic.   [CONCLUDING_STATEMENT_END] [EOE]</p></li>\n<li><p>We can see in model config in hugging face that model knows [CLS] and [SEP] tokens already can we create tokens like this because i thought of using token classification too but i used [CLS_[lead]] this token to indicate start of lead and [[lead]<em>END] for end of lead if my discourse_type is lead and joining other discourse text -&gt;\n[position]  [position] [claim]  [claim]. But tokenizer was breaking my [CLS</em>[lead]] in this way -&gt; \"[\"  ,    \"CL\"  ,    \"S\"  ,    \"_\"  ,    \"[\"  ,    \"lead\"  , '\"]\"  ,  \"]\" . </p></li>\n</ol>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1911715,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T08:59:18.513000",
          "content": "<p>In MLM it was mostly same, but without the discourse type texts e.g. </p>\n<p><code>[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE]  [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END]</code></p>\n<p>i.e. the words <code>Lead</code> and <code>Position</code> are removed. We did this because we wanted models to learn meaning of discourse type tokens.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911650,
      "author_name": "hosuke",
      "author_url": "",
      "post_date": "2022-08-24T08:08:40.500000",
      "content": "<p>Congrats on 3rd place! At first I thought the structure of your model was similar to mine, but you succeeded in doing many things we never thought of, which explains the difference in scores!<br>\nOne thing I would like to know, may I know what model you are using for the summarizer that generates <code>&lt;prompt_text&gt;</code>?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911685,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T08:42:45.433000",
          "content": "<p>Congratulations <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a> for becoming Competition Master 🎉  and gold position in this competition!</p>\n<p>For prompts we followed this great notebook: <a href=\"https://www.kaggle.com/code/lextoumbourou/feedback-topics-identification-with-bertopic\" target=\"_blank\">https://www.kaggle.com/code/lextoumbourou/feedback-topics-identification-with-bertopic</a> for topic identification with BERTopic model, then we had a mapping from topic to prompt e.g. <code>'seagoing luke animals cowboys': 'Should you join the Seagoing Cowboys program?'</code></p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911713,
          "author_name": "hosuke",
          "author_url": "",
          "post_date": "2022-08-24T08:57:40.317000",
          "content": "<p>I see, thank you! <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> <br>\nElegant augmentation with T5 is something I'd like to use myself someday…😃</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911450,
      "author_name": "Nicholas Broad",
      "author_url": "",
      "post_date": "2022-08-24T05:32:43.467000",
      "content": "<p>Congrats on the 3rd place finish! 🎉</p>\n<p>I have two questions: </p>\n<p>I tried a higher masking rate (per Should You Mask 15% in Masked Language Modeling? ) for NBME but I didn't see an improvement. Did you see it helping without doing the long span masking?</p>\n<p>What compute did you use? AWP + Deberta with long sequences in full precision sounds like it would take quite a long time.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911452,
          "author_name": "Zacchaeus",
          "author_url": "",
          "post_date": "2022-08-24T05:37:14.663000",
          "content": "<p><a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> I didn't try long span masking, but for me 0.3 is better than 0.15. Also thanks for your great public baseline, one of our team's model heavily relied on it, which added up much diversity to the ensemble. Also I tried half precision with 8-bit Adam you shared. It worked well.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1911464,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T05:53:48.920000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> </p>\n<blockquote>\n  <p>What compute did you use? AWP + Deberta with long sequences in full precision sounds like it would take quite a long time.</p>\n</blockquote>\n<p>We mainly used A100 and A6000s. Training deberta-v3-large took around 7-8 hours for each fold, it was almost on limit for GPU memory for our training setting.</p>\n<blockquote>\n  <p>I tried a higher masking rate (per Should You Mask 15% in Masked Language Modeling? ) for NBME but I didn't see an improvement. Did you see it helping without doing the long span masking?</p>\n</blockquote>\n<p>We don't have results on higher masking for regular mlm for this competition. In NBME, i used 20% which was better than 15%.  Here, we switch to span mlm quite early as the boost was amazing. Initially, we tried 20% then went 30% and then up to 50%, each time with noticeable boost in CV and LB. </p>\n<p>In addition, at later stage of the competition we found backbone gets better and better with longer training mlm. Eventually we were using mlm training up to 50 epochs (up from ~15 epochs earlier) not sure how far could we have pushed it. </p>\n<p>For some models (e.g. deberta-v2-xlarge) span mlm didn't work, model broke down couldn't cross 4% accuracy… </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912148,
          "author_name": "Nicholas Broad",
          "author_url": "",
          "post_date": "2022-08-24T14:40:19.977000",
          "content": "<p>Interesting. I'll have to keep this in mind for the next span-related competition 😄</p>\n<p>Did you use pure masking or did you do random replacements/no changes too?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912151,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T14:42:49.847000",
          "content": "<p>10% random replacement and 10% no change, same as in <code>DataCollatorForLanguageModeling</code></p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912160,
          "author_name": "Nicholas Broad",
          "author_url": "",
          "post_date": "2022-08-24T14:46:41.360000",
          "content": "<p>How high could your eval accuracy go for MLM?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912178,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T15:04:34.063000",
          "content": "<p>each model had different accuracy on the span mlm, deberta-v3-larger was around 45% accuracy, 16.x perplexity. LUKE on the other hand had ~25% accuracy</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912265,
          "author_name": "Nicholas Broad",
          "author_url": "",
          "post_date": "2022-08-24T16:01:16.580000",
          "content": "<p>Thanks for the details.</p>\n<p>Did you use the default AutoModelForMaskedLM for deberta?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912282,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T16:12:59.383000",
          "content": "<p>yes, we used this to be specific</p>\n<pre><code># model\nbase_config = AutoConfig.from_pretrained(config[\"model_checkpoint\"])\nbase_config.update(\n    {\n        # \"vocab_size\": len(tokenizer),\n        \"max_position_embeddings\": 1024,\n\n    }\n)\nmodel = AutoModelForMaskedLM.from_pretrained(config[\"model_checkpoint\"], config=base_config)\nmodel.deberta.resize_token_embeddings(len(tokenizer))\nmodel.cls = DebertaV2OnlyMLMHead(base_config)  # TODO: check impact\n</code></pre>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911447,
      "author_name": "Zacchaeus",
      "author_url": "",
      "post_date": "2022-08-24T05:28:46.667000",
      "content": "<p>Congrats! I learned a lot from your writeup. One question, the synthetic data used for MLM is fraction of essay, not the whole passage right?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911469,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T05:56:24.860000",
          "content": "<p>yes, up to 750 token length. We used this from huggingface code for splitting the essays</p>\n<pre><code>    def group_texts(examples):\n        # Concatenate all texts\n        concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n\n        # Compute length of concatenated texts\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n\n        # We drop the last chunk if it's smaller than chunk_size\n        total_length = (total_length // chunk_size) * chunk_size\n\n        # Split by chunks of max_len\n        result = {\n            k: [t[i: i + chunk_size] for i in range(0, total_length, chunk_size)]\n            for k, t in concatenated_examples.items()\n        }\n\n        # Create a new labels column\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n</code></pre>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911421,
      "author_name": "leemop",
      "author_url": "",
      "post_date": "2022-08-24T05:01:59.847000",
      "content": "<p>Congraturation! Thansk for your sharing :)</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911472,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T05:57:46.313000",
          "content": "<p>thanks 😊 congratulations to you too!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1917283,
      "author_name": "Sanyam Bhutani",
      "author_url": "",
      "post_date": "2022-08-28T15:50:47.867000",
      "content": "<p>Congrats on the great finish! <br>\nPS: great team name :D </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1918008,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-29T08:45:24.633000",
          "content": "<p>Thanks Sanyam! 😊 </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1913399,
      "author_name": "The Devastator",
      "author_url": "",
      "post_date": "2022-08-25T10:05:12.003000",
      "content": "<p>There is so much on this post.. it is just amazing! <br>\nTotally deserve your strong finish! </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1913638,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-25T12:28:06.397000",
          "content": "<p>thanks! 😊</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911711,
      "author_name": "Patrick Chan",
      "author_url": "",
      "post_date": "2022-08-24T08:55:48.490000",
      "content": "<p>Brilliant usage of T5! Thanks for the sharing and congratulations on the gold.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1911719,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T09:04:06.127000",
          "content": "<p>thanks <a href=\"https://www.kaggle.com/drpatrickchan\" target=\"_blank\">@drpatrickchan</a>!! congratulations to you too!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912876,
          "author_name": "Patrick Chan",
          "author_url": "",
          "post_date": "2022-08-25T02:39:56.193000",
          "content": "<p>Thanks Raja, any chance you can throw the T5 model in a kernel for us to have a test?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1912880,
          "author_name": "Patrick Chan",
          "author_url": "",
          "post_date": "2022-08-25T02:41:28.497000",
          "content": "<p>Do you face any issues training the T5 large model? I have done several at colab before.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1913051,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-25T05:32:37.807000",
          "content": "<p>haven't decided on releasing the model yet…</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1913054,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-25T05:33:36.837000",
          "content": "<p>no didn't face much issues, training was stable.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911535,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2022-08-24T06:56:26.147000",
      "content": "<p>Very nice solution! Congrats and thanks for sharing the writeup <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1911614,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T07:44:56.127000",
          "content": "<p>thanks <a href=\"https://www.kaggle.com/duykhanh99\" target=\"_blank\">@duykhanh99</a>! </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911499,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-08-24T06:18:31.610000",
      "content": "",
      "votes": 1,
      "replies": [
        {
          "id": 1911504,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-08-24T06:21:44.473000",
          "content": "<p>thanks <a href=\"https://www.kaggle.com/utsavsavaliya\" target=\"_blank\">@utsavsavaliya</a>!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 45,
  "messages": [
    {
      "id": 1911419,
      "postDate": "2022-08-24T04:59:25.277Z",
      "content": "<p>A big thanks to Kaggle &amp; the completion hosts for introducing the <code>Efficiency Track</code>, which is an amazing addition and will surely lead to more creative solutions. This is a detailed version of our solution for the <code>Feedback Prize - Predicting Effective Arguments</code> competition. </p>\n<h1>Links</h1>\n<ul>\n<li>TL;DR: <a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347371\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347371</a></li>\n<li>Code: <a href=\"https://github.com/rbiswasfc/kaggle-feedback-effectiveness-3rd-place-solution\" target=\"_blank\">https://github.com/rbiswasfc/kaggle-feedback-effectiveness-3rd-place-solution</a></li>\n<li>Inference Notebook: <a href=\"https://www.kaggle.com/code/conjuring92/ens58-lstm-lgb-all/notebook?scriptVersionId=104014668\" target=\"_blank\">https://www.kaggle.com/code/conjuring92/ens58-lstm-lgb-all/notebook?scriptVersionId=104014668</a></li>\n</ul>\n<h1>Model Architecture</h1>\n<p>Overall, we followed the span classification approach with model architecture as below<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F1b6b3d2fbb7551da6de6fe45f3f00f94%2FScreenshot%202022-08-24%20at%2010.40.29%20AM.png?generation=1661308851093353&amp;alt=media\" alt=\"\"></p>\n<h1>Pre-Processing</h1>\n<p>We pre-processed each essay to insert newly added span start and span end tokens for each discourse types. To provide more context to our models, we added a topic segment in this format <code>[TOPIC] &lt;prompt_text&gt; [TOPIC END] &lt;essay&gt;</code>. As a minor detail, we inserted <code>[SOE]</code> and <code>[EOE]</code> tokens to indicate essay start and end. As an illustration here is an example:</p>\n<pre><code>[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE] Lead [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  Position [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] \n\n...\n\nConcluding Statement [CONCLUDING_STATEMENT] In conclusion I think this is not a very good idea, but not in the way students should be watched. The way teachers are able to read students' facial expressions tells them how they feel. I don't believe it's important to look at students faces when they can fake their emotions. But, if the teacher is watching you then they're gonna get angry. This is how I feel on this topic. [CONCLUDING_STATEMENT_END] [EOE]\n</code></pre>\n<h1>Span MLM</h1>\n<p>To train the newly added tokens (e.g. <code>[LEAD]</code>, <code>[POSITION]</code>) and adapting to the specific task domain (i.e. student essays from grade 6-12) we continued the pre-training phase of each backbone (e.g. <code>deberta-v3-large</code>) with masked language modelling (MLM) objective. While standard MLM works alright, we found big boost with the following modifications:</p>\n<ul>\n<li>Changing masking probability to 40-50% instead of 15% used typically. For a detained analysis on masking rate, please refer to this : <strong>Should You Mask 15% in Masked Language Modeling?</strong> <a href=\"https://arxiv.org/abs/2202.08005\" target=\"_blank\">https://arxiv.org/abs/2202.08005</a></li>\n<li>Masking contiguous tokens of length 3-15 instead of regular random masking approach. Out motivation came from this paper: <strong>SpanBERT: Improving Pre-training by Representing and Predicting Spans</strong> <a href=\"https://arxiv.org/abs/1907.10529\" target=\"_blank\">https://arxiv.org/abs/1907.10529</a></li>\n<li>Changing chuck size / max length to 720 to match average essay length in the fine-tuning stage</li>\n</ul>\n<p>Here we used the ~15k essays from the 2021 feedback competition. In addition, we added ~25k synthetically generated essays by leveraging T5-large model. More on this in the next section.</p>\n<h1>T5 Augmentations</h1>\n<p>Since we couldn't find external data sources that worked for us, we explored ways to create our own augmented / synthetic data / essays. To this end, we adopted two approaches</p>\n<p><strong>Label Preserving Training Data Augmentation with T5</strong> Here we used 2022 labelled data to generate synthetic examples, which are directly added to train dataset during fine-tuning step. For this purpose, we first train the T5-large model with a seq2seq (text-to-text) text generation task. To be specific, this was the task input-output format:</p>\n<ul>\n<li>Input Template: <code>Generate text for &lt;discourse effectiveness&gt; &lt;discourse type&gt; | Prompt: &lt;prompt text&gt; | Left Context: &lt;left context&gt; | Right Context: &lt;right context&gt;</code>, where left context is essay up to the discourse element we try to generate and right context is anything after the discourse element</li>\n<li>Output Template: <code>&lt;discourse text&gt;</code> </li>\n</ul>\n<p>Here is one example:</p>\n<p><code>Generate text for Ineffective Evidence | Prompt: Should the electoral college be abolished in favor of popular vote? | Left Context:Dear, State Senator\\n\\nWith the elctoral college vote most of people are not getting what they prefer. For the the electoral college vote, voters vote fro not the president, but for a not slate of electors. &lt;....&gt; majority of the people would rather have the popular vote.| Right Context: the electoral college election consist of the 538 electors. All you have to do is win most of those votes and you win the election. &lt;.....&gt; The electoral college is unfair, outdated, and irrational to all the people</code><br>\n--&gt;<br>\n<code>It does not make any sense to me how the president with the votes does not win the presidential election. The electoral college is unfair for all voters that vote. When you think of a vote and how it works you would think by the most one with the most votes wins cause that pretty much means that most of the people would rather have the most wins over the win that had less votes but more electoral votes.</code></p>\n<p>The trained T5 model was pretty amazing, we couldn't distinguish which one was generated text and which one was original. The augmentations copied the student writing styles, identity masks e.g.  and included effects such as that mentioned by <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> in the previous competition discussion:</p>\n<blockquote>\n  <p>just imagine you are asked to write the essay for homework. you are a bad student and want to copy other people work. you want to do so, such that that your teacher will not know that you are copying other people's work.<br>\n  you can:<br>\n  1) choose an essay of the train data. replace the claim from another essay. (if the other essay has the same position as the first essay)</p>\n</blockquote>\n<p>Reference: <strong>Data Augmentation using Pre-trained Transformer Models</strong> <a href=\"https://arxiv.org/abs/2003.02245\" target=\"_blank\">https://arxiv.org/abs/2003.02245</a></p>\n<p><strong>T5: Non-Label Preserving Essay Augmentation</strong><br>\nWe used this to mainly generate essays for MLM. It's quite similar to previous augmentation task with minor changes</p>\n<ul>\n<li>uses all ~15k essays from feedback 2021 dataset</li>\n<li>model only sees left context </li>\n<li>no discourse effectiveness label is given in input prompt</li>\n</ul>\n<h1>Fine-tuning</h1>\n<p>In the fine-tuning stage, we mixed varying degree of T5 label-preserved augmented data (0-50% of original essays) with actual data. This was pretty much standard with following key highlights</p>\n<ul>\n<li>Inclusion of prompts in essay: This helped models to boost performance on Lead and Position discourse types</li>\n<li>Adversarial Training with AWP</li>\n<li>Mask Augmentations (10-25%)</li>\n<li>Multi-sample dropouts</li>\n<li>Multi-head attention over span representations</li>\n<li>LSTM over transformer encoder representations</li>\n<li>Loss = CE Loss + Focal loss</li>\n<li>Cosine scheduler</li>\n<li>Layer wise learning rate decay</li>\n<li>Mean pooling to extract span representations as opposed to [CLS] token based ones</li>\n</ul>\n<h1>Model Backbones</h1>\n<ul>\n<li>DeBERTa-Large</li>\n<li>DeBERTa-XLarge</li>\n<li>DeBERTa-V3-Large</li>\n<li>Longformer</li>\n<li>LUKE</li>\n</ul>\n<h1>Ensembling</h1>\n<p>We built two meta models based on OOF model probabilities + additional features</p>\n<p><strong>LSTM based</strong></p>\n<ul>\n<li>Used only stage-1 model probabilities as features</li>\n<li>Had better correlations with CV and LB</li>\n</ul>\n<p><strong>LGB based</strong></p>\n<ul>\n<li>Used stage-1 model probabilities as features + additional features</li>\n<li>Count of next line per essay text</li>\n<li>Count of discourse_id per essay_id</li>\n<li>Pos tags features (NN, VB etc)</li>\n</ul>\n<h1>Things that had most impact</h1>\n<ul>\n<li>Span MLM: 0.02 to 0.03</li>\n<li>AWP: 0.005 to 0.01</li>\n<li>Prompts: 0.002 to 0.005</li>\n<li>Direct use of T5 Augmentations: -0.002 to 0.005</li>\n<li>Mask Augmentation + MSD: 0.002 - 0.005</li>\n<li>LSTM + LGB ensemble: 0.002-0.004 </li>\n</ul>\n<h1>Things that added for diversity</h1>\n<ul>\n<li>LUKE model</li>\n<li>Multitask objectives e.g. predicting discourse type, ordered encoding of labels e.g. ineffective -&gt; [0, 0], adequate -&gt; [1, 0], effective -&gt; [1, 1]</li>\n<li>Different sampling ratio of augmented data</li>\n<li>Impact moderation of focal loss with its gamma parameter</li>\n<li>Knowledge distillation</li>\n</ul>\n<h1>How to make training stable?</h1>\n<p>We could train our models up to 6-7 epoch with best results obtained in penultimate/last epoch. The following helped in prolonging and stabilizing the training</p>\n<ul>\n<li>Careful selection of key params e.g. batch size, learning rate, weight decay</li>\n<li>Cosine learning rate scheduler</li>\n<li>Task adaptation with MLM</li>\n<li>AWP</li>\n<li>Mask augmentations + Multi-sample dropouts</li>\n<li>Layer wise learning rate decay</li>\n<li><strong>Full precision training</strong></li>\n</ul>\n<h1>Things that didn't work for us</h1>\n<ul>\n<li>Pseudo labelling / meta pseudo labelling</li>\n<li>on hindsight, this was a big miss for us</li>\n<li>Random augmentation </li>\n<li>UDA - Unsupervised Data Augmentations</li>\n<li>Contrastive Learning</li>\n<li>SWA</li>\n<li>Mixout</li>\n</ul>\n<p>Thanks for reading this far! Hope this helps you in your future NLP competitions! Looking forward to your comments and feedbacks.</p>\n<p>Our team ( <a href=\"https://www.kaggle.com/trushk\" target=\"_blank\">@trushk</a> , <a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a>) dynamics was brilliant throughout the competition. We had a lot of fun and learned so much together. Looking forward to future teaming up! </p>\n<p>Thanks 😊</p>",
      "rawMarkdown": "A big thanks to Kaggle & the completion hosts for introducing the `Efficiency Track`, which is an amazing addition and will surely lead to more creative solutions. This is a detailed version of our solution for the `Feedback Prize - Predicting Effective Arguments` competition. \n\n# Links\n* TL;DR: https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347371\n* Code: https://github.com/rbiswasfc/kaggle-feedback-effectiveness-3rd-place-solution\n* Inference Notebook: https://www.kaggle.com/code/conjuring92/ens58-lstm-lgb-all/notebook?scriptVersionId=104014668\n\n# Model Architecture \nOverall, we followed the span classification approach with model architecture as below\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F1b6b3d2fbb7551da6de6fe45f3f00f94%2FScreenshot%202022-08-24%20at%2010.40.29%20AM.png?generation=1661308851093353&alt=media)\n\n# Pre-Processing\nWe pre-processed each essay to insert newly added span start and span end tokens for each discourse types. To provide more context to our models, we added a topic segment in this format `[TOPIC] <prompt_text> [TOPIC END] <essay>`. As a minor detail, we inserted `[SOE]` and `[EOE]` tokens to indicate essay start and end. As an illustration here is an example:\n\n\n```\n[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE] Lead [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  Position [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] \n\n...\n\nConcluding Statement [CONCLUDING_STATEMENT] In conclusion I think this is not a very good idea, but not in the way students should be watched. The way teachers are able to read students' facial expressions tells them how they feel. I don't believe it's important to look at students faces when they can fake their emotions. But, if the teacher is watching you then they're gonna get angry. This is how I feel on this topic. [CONCLUDING_STATEMENT_END] [EOE]\n```\n\n# Span MLM\nTo train the newly added tokens (e.g. `[LEAD]`, `[POSITION]`) and adapting to the specific task domain (i.e. student essays from grade 6-12) we continued the pre-training phase of each backbone (e.g. `deberta-v3-large`) with masked language modelling (MLM) objective. While standard MLM works alright, we found big boost with the following modifications:\n* Changing masking probability to 40-50% instead of 15% used typically. For a detained analysis on masking rate, please refer to this : **Should You Mask 15% in Masked Language Modeling?** https://arxiv.org/abs/2202.08005\n* Masking contiguous tokens of length 3-15 instead of regular random masking approach. Out motivation came from this paper: **SpanBERT: Improving Pre-training by Representing and Predicting Spans** https://arxiv.org/abs/1907.10529\n* Changing chuck size / max length to 720 to match average essay length in the fine-tuning stage\n\nHere we used the ~15k essays from the 2021 feedback competition. In addition, we added ~25k synthetically generated essays by leveraging T5-large model. More on this in the next section.\n\n# T5 Augmentations\nSince we couldn't find external data sources that worked for us, we explored ways to create our own augmented / synthetic data / essays. To this end, we adopted two approaches\n\n **Label Preserving Training Data Augmentation with T5** Here we used 2022 labelled data to generate synthetic examples, which are directly added to train dataset during fine-tuning step. For this purpose, we first train the T5-large model with a seq2seq (text-to-text) text generation task. To be specific, this was the task input-output format:\n\n- Input Template: `Generate text for <discourse effectiveness> <discourse type> | Prompt: <prompt text> | Left Context: <left context> | Right Context: <right context>`, where left context is essay up to the discourse element we try to generate and right context is anything after the discourse element\n- Output Template: `<discourse text>` \n\nHere is one example:\n\n`Generate text for Ineffective Evidence | Prompt: Should the electoral college be abolished in favor of popular vote? | Left Context:Dear, State Senator\\n\\nWith the elctoral college vote most of people are not getting what they prefer. For the the electoral college vote, voters vote fro not the president, but for a not slate of electors. <....> majority of the people would rather have the popular vote.| Right Context: the electoral college election consist of the 538 electors. All you have to do is win most of those votes and you win the election. <.....> The electoral college is unfair, outdated, and irrational to all the people`\n-->\n`It does not make any sense to me how the president with the votes does not win the presidential election. The electoral college is unfair for all voters that vote. When you think of a vote and how it works you would think by the most one with the most votes wins cause that pretty much means that most of the people would rather have the most wins over the win that had less votes but more electoral votes.`\n\nThe trained T5 model was pretty amazing, we couldn't distinguish which one was generated text and which one was original. The augmentations copied the student writing styles, identity masks e.g. <LOCATION_NAME> and included effects such as that mentioned by @hengck23 in the previous competition discussion:\n\n> just imagine you are asked to write the essay for homework. you are a bad student and want to copy other people work. you want to do so, such that that your teacher will not know that you are copying other people's work.\nyou can:\n1) choose an essay of the train data. replace the claim from another essay. (if the other essay has the same position as the first essay)\n\nReference: **Data Augmentation using Pre-trained Transformer Models** https://arxiv.org/abs/2003.02245\n\n **T5: Non-Label Preserving Essay Augmentation**\nWe used this to mainly generate essays for MLM. It's quite similar to previous augmentation task with minor changes\n* uses all ~15k essays from feedback 2021 dataset\n* model only sees left context \n* no discourse effectiveness label is given in input prompt\n\n\n# Fine-tuning\nIn the fine-tuning stage, we mixed varying degree of T5 label-preserved augmented data (0-50% of original essays) with actual data. This was pretty much standard with following key highlights\n\n* Inclusion of prompts in essay: This helped models to boost performance on Lead and Position discourse types\n* Adversarial Training with AWP\n* Mask Augmentations (10-25%)\n* Multi-sample dropouts\n* Multi-head attention over span representations\n* LSTM over transformer encoder representations\n* Loss = CE Loss + Focal loss\n* Cosine scheduler\n* Layer wise learning rate decay\n* Mean pooling to extract span representations as opposed to [CLS] token based ones\n\n# Model Backbones\n* DeBERTa-Large\n* DeBERTa-XLarge\n* DeBERTa-V3-Large\n* Longformer\n* LUKE\n\n# Ensembling\nWe built two meta models based on OOF model probabilities + additional features\n\n**LSTM based**\n* Used only stage-1 model probabilities as features\n* Had better correlations with CV and LB\n\n**LGB based**\n* Used stage-1 model probabilities as features + additional features\n- Count of next line per essay text\n- Count of discourse_id per essay_id\n- Pos tags features (NN, VB etc)\n\n# Things that had most impact\n* Span MLM: 0.02 to 0.03\n* AWP: 0.005 to 0.01\n* Prompts: 0.002 to 0.005\n* Direct use of T5 Augmentations: -0.002 to 0.005\n* Mask Augmentation + MSD: 0.002 - 0.005\n* LSTM + LGB ensemble: 0.002-0.004 \n\n# Things that added for diversity\n* LUKE model\n* Multitask objectives e.g. predicting discourse type, ordered encoding of labels e.g. ineffective -> [0, 0], adequate -> [1, 0], effective -> [1, 1]\n* Different sampling ratio of augmented data\n* Impact moderation of focal loss with its gamma parameter\n* Knowledge distillation\n\n# How to make training stable?\nWe could train our models up to 6-7 epoch with best results obtained in penultimate/last epoch. The following helped in prolonging and stabilizing the training\n* Careful selection of key params e.g. batch size, learning rate, weight decay\n* Cosine learning rate scheduler\n* Task adaptation with MLM\n* AWP\n* Mask augmentations + Multi-sample dropouts\n* Layer wise learning rate decay\n* **Full precision training**\n\n# Things that didn't work for us \n* Pseudo labelling / meta pseudo labelling\n- on hindsight, this was a big miss for us\n* Random augmentation \n* UDA - Unsupervised Data Augmentations\n* Contrastive Learning\n* SWA\n* Mixout\n\n\nThanks for reading this far! Hope this helps you in your future NLP competitions! Looking forward to your comments and feedbacks.\n\nOur team ( @trushk , @harshit92) dynamics was brilliant throughout the competition. We had a lot of fun and learned so much together. Looking forward to future teaming up! \n\nThanks 😊",
      "votes": 77
    },
    {
      "id": 1911993,
      "postDate": "2022-08-24T12:42:55.947Z",
      "content": "<p>Thanks for the writeup and congrats on the great result! It was a fun back and forth on LB :)</p>\n<p>I assume your synthetic data generation approach is replacing the pseudo labeling approach which is why you did not gain much more out from it. </p>",
      "rawMarkdown": "Thanks for the writeup and congrats on the great result! It was a fun back and forth on LB :)\n\nI assume your synthetic data generation approach is replacing the pseudo labeling approach which is why you did not gain much more out from it. ",
      "votes": 3,
      "replies": [
        {
          "id": 1912034,
          "postDate": "2022-08-24T13:23:59.467Z",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a>, Congratulations on winning the competition on both efficiency and conventional track 🎉🎉🎉 We really enjoyed the fight and believed that pushed each other. On synthetic data, yes i also think it compensated for pseudo labelling partially. However, we only went there as we couldn't make any sort of pseudo labelling work. Soft targets seem to have worked better, it was on our list of things to try but never got to it. </p>",
          "rawMarkdown": "Hi @philippsinger, Congratulations on winning the competition on both efficiency and conventional track 🎉🎉🎉 We really enjoyed the fight and believed that pushed each other. On synthetic data, yes i also think it compensated for pseudo labelling partially. However, we only went there as we couldn't make any sort of pseudo labelling work. Soft targets seem to have worked better, it was on our list of things to try but never got to it. ",
          "votes": 1
        },
        {
          "id": 1912084,
          "postDate": "2022-08-24T13:50:31.760Z",
          "content": "<p>Oh did you only try hard labels? Hard labels did not work AT ALL, you had to use soft labels. One core problem with hard labels is that Adequate is way too prominent there. But even on soft labels it took us quite some time to make use of them properly…</p>",
          "rawMarkdown": "Oh did you only try hard labels? Hard labels did not work AT ALL, you had to use soft labels. One core problem with hard labels is that Adequate is way too prominent there. But even on soft labels it took us quite some time to make use of them properly...",
          "votes": 3
        },
        {
          "id": 1912126,
          "postDate": "2022-08-24T14:25:19.713Z",
          "content": "<p>sadly yes, we only tried many different ways with hard labels. Thanks for the explanation, that must be it! We got distracted in finding an explanation why it doesn't work, instead of making it work. Was partially influenced by prior experience where I mostly found soft and hard targets to perform equally. But every dataset + task have subtle differences.</p>",
          "rawMarkdown": "sadly yes, we only tried many different ways with hard labels. Thanks for the explanation, that must be it! We got distracted in finding an explanation why it doesn't work, instead of making it work. Was partially influenced by prior experience where I mostly found soft and hard targets to perform equally. But every dataset + task have subtle differences.",
          "votes": 1
        },
        {
          "id": 1912263,
          "postDate": "2022-08-24T15:58:52.633Z",
          "content": "<p>Yes unfortunately we didnt spend enough time to get pseudo labeling to work. Seems like that was the missing piece that would have given us an edge. Looking back I think the 3rd place without pseudo labels is pretty amazing 😄</p>",
          "rawMarkdown": "Yes unfortunately we didnt spend enough time to get pseudo labeling to work. Seems like that was the missing piece that would have given us an edge. Looking back I think the 3rd place without pseudo labels is pretty amazing 😄",
          "votes": 1
        },
        {
          "id": 1912347,
          "postDate": "2022-08-24T16:53:59.077Z",
          "content": "<p>Yeah it is amazing! We were so sure that you are using pseudos heavily, specifically meta pseudo labels as you used that in last competition. </p>\n<p>But overall, after a competition ends and you feel like you missed out on something, you have to always consider that there is only limited time and that many ideas. If you would have spent time on pseudos---and it really took us quite some effort to use them properly---you would not have found time to try other ideas that also gave you an edge. So it might balance itself out in the end. </p>",
          "rawMarkdown": "Yeah it is amazing! We were so sure that you are using pseudos heavily, specifically meta pseudo labels as you used that in last competition. \n\nBut overall, after a competition ends and you feel like you missed out on something, you have to always consider that there is only limited time and that many ideas. If you would have spent time on pseudos---and it really took us quite some effort to use them properly---you would not have found time to try other ideas that also gave you an edge. So it might balance itself out in the end. ",
          "votes": 3
        }
      ]
    },
    {
      "id": 1932053,
      "postDate": "2022-09-09T08:27:02.420Z",
      "content": "<p>Congrats on your amazing work! I learn a lot!</p>",
      "rawMarkdown": "Congrats on your amazing work! I learn a lot!",
      "votes": 1
    },
    {
      "id": 1916576,
      "postDate": "2022-08-28T02:44:40.017Z",
      "content": "<p>For the span mlm part, can you recommand some code snippet? thanks!</p>",
      "rawMarkdown": "For the span mlm part, can you recommand some code snippet? thanks!",
      "votes": 1,
      "replies": [
        {
          "id": 1918004,
          "postDate": "2022-08-29T08:41:22.423Z",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/nickycan\" target=\"_blank\">@nickycan</a>, for the span mlm part we just modified the <code>DataCollatorForLanguageModeling</code> from HF library. For the modification, we referred to SpanBert git repo, specifically here: <a href=\"https://github.com/facebookresearch/SpanBERT/blob/main/pretraining/fairseq/data/masking.py#L137\" target=\"_blank\">https://github.com/facebookresearch/SpanBERT/blob/main/pretraining/fairseq/data/masking.py#L137</a></p>",
          "rawMarkdown": "Hi @nickycan, for the span mlm part we just modified the `DataCollatorForLanguageModeling` from HF library. For the modification, we referred to SpanBert git repo, specifically here: https://github.com/facebookresearch/SpanBERT/blob/main/pretraining/fairseq/data/masking.py#L137",
          "votes": 2
        }
      ]
    },
    {
      "id": 1911839,
      "postDate": "2022-08-24T10:49:28.387Z",
      "content": "<p><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> will you be sharing your training code? Looks like there is a lot to learn here!</p>",
      "rawMarkdown": "@conjuring92 will you be sharing your training code? Looks like there is a lot to learn here!",
      "votes": 1,
      "replies": [
        {
          "id": 1912057,
          "postDate": "2022-08-24T13:32:07.740Z",
          "content": "<p>sorry, we don't have any immediate plan to make the code public. Thanks for your understanding.</p>",
          "rawMarkdown": "sorry, we don't have any immediate plan to make the code public. Thanks for your understanding."
        }
      ]
    },
    {
      "id": 1911683,
      "postDate": "2022-08-24T08:40:13.213Z",
      "content": "<p>Congratulation for 3rd place.</p>\n<p>1. What is Contrastive Learning?</p>\n<p>2.what do you mean \"Careful selection of key params e.g. batch size, learning rate, weight decay\" ?,  I can understand careful selection of batch size and lr but what about weight decay. Till now i thought weight decay is always 0.01.</p>",
      "rawMarkdown": "<p>Congratulation for 3rd place.</p>\n<p>1. What is Contrastive Learning?</p>\n<p>2.what do you mean \"Careful selection of key params e.g. batch size, learning rate, weight decay\" ?,  I can understand careful selection of batch size and lr but what about weight decay. Till now i thought weight decay is always 0.01.</p>\n",
      "votes": 1,
      "replies": [
        {
          "id": 1911694,
          "postDate": "2022-08-24T08:48:27.893Z",
          "content": "<p>Thanks! </p>\n<p>For contrastive learning please refer to this paper:</p>\n<ul>\n<li>SUPERVISED CONTRASTIVE LEARNING FOR PRE-TRAINED LANGUAGE MODEL FINE-TUNING (<a href=\"https://openreview.net/pdf?id=cu7IUiOhujH\" target=\"_blank\">https://openreview.net/pdf?id=cu7IUiOhujH</a>)</li>\n</ul>\n<p>Yes, weight decay of 1e-2 works in most cases but few models are quite specific e.g. electra recommends weight decay to be set to 0. I normally decrease it when i'm training longer…</p>",
          "rawMarkdown": "Thanks! \n\nFor contrastive learning please refer to this paper:\n* SUPERVISED CONTRASTIVE LEARNING FOR PRE-TRAINED LANGUAGE MODEL FINE-TUNING (https://openreview.net/pdf?id=cu7IUiOhujH)\n\nYes, weight decay of 1e-2 works in most cases but few models are quite specific e.g. electra recommends weight decay to be set to 0. I normally decrease it when i'm training longer...",
          "votes": 1
        },
        {
          "id": 1911707,
          "postDate": "2022-08-24T08:54:19.853Z",
          "content": "<ol>\n<li><p>Did you used essay as it is  for mlm  or you used this way</p>\n<p>[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE] Lead            [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  Position [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] </p>\n<p>…</p>\n<p>Concluding Statement [CONCLUDING_STATEMENT] In conclusion I think this is not a very good idea, but not in the way students should be watched. The way teachers are able to read students' facial expressions tells them how they feel. I don't believe it's important to look at students faces when they can fake their emotions. But, if the teacher is watching you then they're gonna get angry. This is how I feel on this topic.   [CONCLUDING_STATEMENT_END] [EOE]</p></li>\n<li><p>We can see in model config in hugging face that model knows [CLS] and [SEP] tokens already can we create tokens like this because i thought of using token classification too but i used [CLS_[lead]] this token to indicate start of lead and [[lead]<em>END] for end of lead if my discourse_type is lead and joining other discourse text -&gt;\n[position]  [position] [claim]  [claim]. But tokenizer was breaking my [CLS</em>[lead]] in this way -&gt; \"[\"  ,    \"CL\"  ,    \"S\"  ,    \"_\"  ,    \"[\"  ,    \"lead\"  , '\"]\"  ,  \"]\" . </p></li>\n</ol>",
          "rawMarkdown": "1. Did you used essay as it is  for mlm  or you used this way\n\n    [TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE] Lead            [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  Position [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] \n\n    ...\n\n    Concluding Statement [CONCLUDING_STATEMENT] In conclusion I think this is not a very good idea, but not in the way students should be watched. The way teachers are able to read students' facial expressions tells them how they feel. I don't believe it's important to look at students faces when they can fake their emotions. But, if the teacher is watching you then they're gonna get angry. This is how I feel on this topic.   [CONCLUDING_STATEMENT_END] [EOE]\n\n\n2. We can see in model config in hugging face that model knows [CLS] and [SEP] tokens already can we create tokens like this because i thought of using token classification too but i used [CLS_[lead]] this token to indicate start of lead and [[lead]_END] for end of lead if my discourse_type is lead and joining other discourse text ->\n[position] <discourse_text> [position] [claim] <discourse_text> [claim]. But tokenizer was breaking my [CLS_[lead]] in this way -> \"[\"  ,    \"CL\"  ,    \"S\"  ,    \"_\"  ,    \"[\"  ,    \"lead\"  , '\"]\"  ,  \"]\" . "
        },
        {
          "id": 1911715,
          "postDate": "2022-08-24T08:59:18.513Z",
          "content": "<p>In MLM it was mostly same, but without the discourse type texts e.g. </p>\n<p><code>[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE]  [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END]</code></p>\n<p>i.e. the words <code>Lead</code> and <code>Position</code> are removed. We did this because we wanted models to learn meaning of discourse type tokens.</p>",
          "rawMarkdown": "In MLM it was mostly same, but without the discourse type texts e.g. \n\n`[TOPIC] Should computers read the emotional expressions of students in a classroom? [TOPIC_END] [SOE]  [LEAD] Imagine being told about a new new technology that reads emotions. Now imagine this, imagine being able to tell how someone truly feels about their emotional expressions because they don't really know how they really feel. [LEAD_END]  In this essay I am going to tell you about my opinions on why  [POSITION] I believe that the value of using technology to read student's emotional expressions is over the top. [POSITION_END] `\n\ni.e. the words `Lead` and `Position` are removed. We did this because we wanted models to learn meaning of discourse type tokens.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911650,
      "postDate": "2022-08-24T08:08:40.500Z",
      "content": "<p>Congrats on 3rd place! At first I thought the structure of your model was similar to mine, but you succeeded in doing many things we never thought of, which explains the difference in scores!<br>\nOne thing I would like to know, may I know what model you are using for the summarizer that generates <code>&lt;prompt_text&gt;</code>?</p>",
      "rawMarkdown": "Congrats on 3rd place! At first I thought the structure of your model was similar to mine, but you succeeded in doing many things we never thought of, which explains the difference in scores!\nOne thing I would like to know, may I know what model you are using for the summarizer that generates `<prompt_text>`?",
      "votes": 1,
      "replies": [
        {
          "id": 1911685,
          "postDate": "2022-08-24T08:42:45.433Z",
          "content": "<p>Congratulations <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a> for becoming Competition Master 🎉  and gold position in this competition!</p>\n<p>For prompts we followed this great notebook: <a href=\"https://www.kaggle.com/code/lextoumbourou/feedback-topics-identification-with-bertopic\" target=\"_blank\">https://www.kaggle.com/code/lextoumbourou/feedback-topics-identification-with-bertopic</a> for topic identification with BERTopic model, then we had a mapping from topic to prompt e.g. <code>'seagoing luke animals cowboys': 'Should you join the Seagoing Cowboys program?'</code></p>",
          "rawMarkdown": "Congratulations @horsek for becoming Competition Master 🎉  and gold position in this competition!\n \nFor prompts we followed this great notebook: https://www.kaggle.com/code/lextoumbourou/feedback-topics-identification-with-bertopic for topic identification with BERTopic model, then we had a mapping from topic to prompt e.g. `'seagoing luke animals cowboys': 'Should you join the Seagoing Cowboys program?'`",
          "votes": 1
        },
        {
          "id": 1911713,
          "postDate": "2022-08-24T08:57:40.317Z",
          "content": "<p>I see, thank you! <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> <br>\nElegant augmentation with T5 is something I'd like to use myself someday…😃</p>",
          "rawMarkdown": "I see, thank you! @conjuring92 \nElegant augmentation with T5 is something I'd like to use myself someday...😃",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911450,
      "postDate": "2022-08-24T05:32:43.467Z",
      "content": "<p>Congrats on the 3rd place finish! 🎉</p>\n<p>I have two questions: </p>\n<p>I tried a higher masking rate (per Should You Mask 15% in Masked Language Modeling? ) for NBME but I didn't see an improvement. Did you see it helping without doing the long span masking?</p>\n<p>What compute did you use? AWP + Deberta with long sequences in full precision sounds like it would take quite a long time.</p>",
      "rawMarkdown": "Congrats on the 3rd place finish! 🎉\n\nI have two questions: \n\nI tried a higher masking rate (per Should You Mask 15% in Masked Language Modeling? ) for NBME but I didn't see an improvement. Did you see it helping without doing the long span masking?\n\nWhat compute did you use? AWP + Deberta with long sequences in full precision sounds like it would take quite a long time.",
      "votes": 1,
      "replies": [
        {
          "id": 1911452,
          "postDate": "2022-08-24T05:37:14.663Z",
          "content": "<p><a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> I didn't try long span masking, but for me 0.3 is better than 0.15. Also thanks for your great public baseline, one of our team's model heavily relied on it, which added up much diversity to the ensemble. Also I tried half precision with 8-bit Adam you shared. It worked well.</p>",
          "rawMarkdown": "@nbroad I didn't try long span masking, but for me 0.3 is better than 0.15. Also thanks for your great public baseline, one of our team's model heavily relied on it, which added up much diversity to the ensemble. Also I tried half precision with 8-bit Adam you shared. It worked well.",
          "votes": 2
        },
        {
          "id": 1911464,
          "postDate": "2022-08-24T05:53:48.920Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> </p>\n<blockquote>\n  <p>What compute did you use? AWP + Deberta with long sequences in full precision sounds like it would take quite a long time.</p>\n</blockquote>\n<p>We mainly used A100 and A6000s. Training deberta-v3-large took around 7-8 hours for each fold, it was almost on limit for GPU memory for our training setting.</p>\n<blockquote>\n  <p>I tried a higher masking rate (per Should You Mask 15% in Masked Language Modeling? ) for NBME but I didn't see an improvement. Did you see it helping without doing the long span masking?</p>\n</blockquote>\n<p>We don't have results on higher masking for regular mlm for this competition. In NBME, i used 20% which was better than 15%.  Here, we switch to span mlm quite early as the boost was amazing. Initially, we tried 20% then went 30% and then up to 50%, each time with noticeable boost in CV and LB. </p>\n<p>In addition, at later stage of the competition we found backbone gets better and better with longer training mlm. Eventually we were using mlm training up to 50 epochs (up from ~15 epochs earlier) not sure how far could we have pushed it. </p>\n<p>For some models (e.g. deberta-v2-xlarge) span mlm didn't work, model broke down couldn't cross 4% accuracy… </p>",
          "rawMarkdown": "Thanks @nbroad \n\n>What compute did you use? AWP + Deberta with long sequences in full precision sounds like it would take quite a long time.\n\nWe mainly used A100 and A6000s. Training deberta-v3-large took around 7-8 hours for each fold, it was almost on limit for GPU memory for our training setting.\n\n> I tried a higher masking rate (per Should You Mask 15% in Masked Language Modeling? ) for NBME but I didn't see an improvement. Did you see it helping without doing the long span masking?\n\nWe don't have results on higher masking for regular mlm for this competition. In NBME, i used 20% which was better than 15%.  Here, we switch to span mlm quite early as the boost was amazing. Initially, we tried 20% then went 30% and then up to 50%, each time with noticeable boost in CV and LB. \n\nIn addition, at later stage of the competition we found backbone gets better and better with longer training mlm. Eventually we were using mlm training up to 50 epochs (up from ~15 epochs earlier) not sure how far could we have pushed it. \n\nFor some models (e.g. deberta-v2-xlarge) span mlm didn't work, model broke down couldn't cross 4% accuracy... ",
          "votes": 1
        },
        {
          "id": 1912148,
          "postDate": "2022-08-24T14:40:19.977Z",
          "content": "<p>Interesting. I'll have to keep this in mind for the next span-related competition 😄</p>\n<p>Did you use pure masking or did you do random replacements/no changes too?</p>",
          "rawMarkdown": "Interesting. I'll have to keep this in mind for the next span-related competition 😄\n\nDid you use pure masking or did you do random replacements/no changes too?",
          "votes": 1
        },
        {
          "id": 1912151,
          "postDate": "2022-08-24T14:42:49.847Z",
          "content": "<p>10% random replacement and 10% no change, same as in <code>DataCollatorForLanguageModeling</code></p>",
          "rawMarkdown": "10% random replacement and 10% no change, same as in `DataCollatorForLanguageModeling`",
          "votes": 1
        },
        {
          "id": 1912160,
          "postDate": "2022-08-24T14:46:41.360Z",
          "content": "<p>How high could your eval accuracy go for MLM?</p>",
          "rawMarkdown": "How high could your eval accuracy go for MLM?",
          "votes": 1
        },
        {
          "id": 1912178,
          "postDate": "2022-08-24T15:04:34.063Z",
          "content": "<p>each model had different accuracy on the span mlm, deberta-v3-larger was around 45% accuracy, 16.x perplexity. LUKE on the other hand had ~25% accuracy</p>",
          "rawMarkdown": "each model had different accuracy on the span mlm, deberta-v3-larger was around 45% accuracy, 16.x perplexity. LUKE on the other hand had ~25% accuracy",
          "votes": 1
        },
        {
          "id": 1912265,
          "postDate": "2022-08-24T16:01:16.580Z",
          "content": "<p>Thanks for the details.</p>\n<p>Did you use the default AutoModelForMaskedLM for deberta?</p>",
          "rawMarkdown": "Thanks for the details.\n\nDid you use the default AutoModelForMaskedLM for deberta?",
          "votes": 1
        },
        {
          "id": 1912282,
          "postDate": "2022-08-24T16:12:59.383Z",
          "content": "<p>yes, we used this to be specific</p>\n<pre><code># model\nbase_config = AutoConfig.from_pretrained(config[\"model_checkpoint\"])\nbase_config.update(\n    {\n        # \"vocab_size\": len(tokenizer),\n        \"max_position_embeddings\": 1024,\n\n    }\n)\nmodel = AutoModelForMaskedLM.from_pretrained(config[\"model_checkpoint\"], config=base_config)\nmodel.deberta.resize_token_embeddings(len(tokenizer))\nmodel.cls = DebertaV2OnlyMLMHead(base_config)  # TODO: check impact\n</code></pre>",
          "rawMarkdown": "yes, we used this to be specific\n```\n# model\nbase_config = AutoConfig.from_pretrained(config[\"model_checkpoint\"])\nbase_config.update(\n    {\n        # \"vocab_size\": len(tokenizer),\n        \"max_position_embeddings\": 1024,\n\n    }\n)\nmodel = AutoModelForMaskedLM.from_pretrained(config[\"model_checkpoint\"], config=base_config)\nmodel.deberta.resize_token_embeddings(len(tokenizer))\nmodel.cls = DebertaV2OnlyMLMHead(base_config)  # TODO: check impact\n```",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911447,
      "postDate": "2022-08-24T05:28:46.667Z",
      "content": "<p>Congrats! I learned a lot from your writeup. One question, the synthetic data used for MLM is fraction of essay, not the whole passage right?</p>",
      "rawMarkdown": "Congrats! I learned a lot from your writeup. One question, the synthetic data used for MLM is fraction of essay, not the whole passage right?",
      "votes": 1,
      "replies": [
        {
          "id": 1911469,
          "postDate": "2022-08-24T05:56:24.860Z",
          "content": "<p>yes, up to 750 token length. We used this from huggingface code for splitting the essays</p>\n<pre><code>    def group_texts(examples):\n        # Concatenate all texts\n        concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n\n        # Compute length of concatenated texts\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n\n        # We drop the last chunk if it's smaller than chunk_size\n        total_length = (total_length // chunk_size) * chunk_size\n\n        # Split by chunks of max_len\n        result = {\n            k: [t[i: i + chunk_size] for i in range(0, total_length, chunk_size)]\n            for k, t in concatenated_examples.items()\n        }\n\n        # Create a new labels column\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n</code></pre>",
          "rawMarkdown": "yes, up to 750 token length. We used this from huggingface code for splitting the essays\n\n```\n    def group_texts(examples):\n        # Concatenate all texts\n        concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n\n        # Compute length of concatenated texts\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n\n        # We drop the last chunk if it's smaller than chunk_size\n        total_length = (total_length // chunk_size) * chunk_size\n\n        # Split by chunks of max_len\n        result = {\n            k: [t[i: i + chunk_size] for i in range(0, total_length, chunk_size)]\n            for k, t in concatenated_examples.items()\n        }\n\n        # Create a new labels column\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n```",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911421,
      "postDate": "2022-08-24T05:01:59.847Z",
      "content": "<p>Congraturation! Thansk for your sharing :)</p>",
      "rawMarkdown": "Congraturation! Thansk for your sharing :)",
      "votes": 1,
      "replies": [
        {
          "id": 1911472,
          "postDate": "2022-08-24T05:57:46.313Z",
          "content": "<p>thanks 😊 congratulations to you too!</p>",
          "rawMarkdown": "thanks 😊 congratulations to you too!"
        }
      ]
    },
    {
      "id": 1917283,
      "postDate": "2022-08-28T15:50:47.867Z",
      "content": "<p>Congrats on the great finish! <br>\nPS: great team name :D </p>",
      "rawMarkdown": "Congrats on the great finish! \nPS: great team name :D ",
      "votes": 2,
      "replies": [
        {
          "id": 1918008,
          "postDate": "2022-08-29T08:45:24.633Z",
          "content": "<p>Thanks Sanyam! 😊 </p>",
          "rawMarkdown": "Thanks Sanyam! 😊 ",
          "votes": 1
        }
      ]
    },
    {
      "id": 1913399,
      "postDate": "2022-08-25T10:05:12.003Z",
      "content": "<p>There is so much on this post.. it is just amazing! <br>\nTotally deserve your strong finish! </p>",
      "rawMarkdown": "There is so much on this post.. it is just amazing! \nTotally deserve your strong finish! ",
      "votes": 2,
      "replies": [
        {
          "id": 1913638,
          "postDate": "2022-08-25T12:28:06.397Z",
          "content": "<p>thanks! 😊</p>",
          "rawMarkdown": "thanks! 😊"
        }
      ]
    },
    {
      "id": 1911711,
      "postDate": "2022-08-24T08:55:48.490Z",
      "content": "<p>Brilliant usage of T5! Thanks for the sharing and congratulations on the gold.</p>",
      "rawMarkdown": "Brilliant usage of T5! Thanks for the sharing and congratulations on the gold.",
      "votes": 2,
      "replies": [
        {
          "id": 1911719,
          "postDate": "2022-08-24T09:04:06.127Z",
          "content": "<p>thanks <a href=\"https://www.kaggle.com/drpatrickchan\" target=\"_blank\">@drpatrickchan</a>!! congratulations to you too!</p>",
          "rawMarkdown": "thanks @drpatrickchan!! congratulations to you too!",
          "votes": 1
        },
        {
          "id": 1912876,
          "postDate": "2022-08-25T02:39:56.193Z",
          "content": "<p>Thanks Raja, any chance you can throw the T5 model in a kernel for us to have a test?</p>",
          "rawMarkdown": "Thanks Raja, any chance you can throw the T5 model in a kernel for us to have a test?",
          "votes": 1
        },
        {
          "id": 1912880,
          "postDate": "2022-08-25T02:41:28.497Z",
          "content": "<p>Do you face any issues training the T5 large model? I have done several at colab before.</p>",
          "rawMarkdown": "Do you face any issues training the T5 large model? I have done several at colab before."
        },
        {
          "id": 1913051,
          "postDate": "2022-08-25T05:32:37.807Z",
          "content": "<p>haven't decided on releasing the model yet…</p>",
          "rawMarkdown": "haven't decided on releasing the model yet..."
        },
        {
          "id": 1913054,
          "postDate": "2022-08-25T05:33:36.837Z",
          "content": "<p>no didn't face much issues, training was stable.</p>",
          "rawMarkdown": "no didn't face much issues, training was stable."
        }
      ]
    },
    {
      "id": 1911535,
      "postDate": "2022-08-24T06:56:26.147Z",
      "content": "<p>Very nice solution! Congrats and thanks for sharing the writeup <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> </p>",
      "rawMarkdown": "Very nice solution! Congrats and thanks for sharing the writeup @conjuring92 ",
      "votes": 2,
      "replies": [
        {
          "id": 1911614,
          "postDate": "2022-08-24T07:44:56.127Z",
          "content": "<p>thanks <a href=\"https://www.kaggle.com/duykhanh99\" target=\"_blank\">@duykhanh99</a>! </p>",
          "rawMarkdown": "thanks @duykhanh99! ",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911499,
      "postDate": "2022-08-24T06:18:31.610Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true,
      "replies": [
        {
          "id": 1911504,
          "postDate": "2022-08-24T06:21:44.473Z",
          "content": "<p>thanks <a href=\"https://www.kaggle.com/utsavsavaliya\" target=\"_blank\">@utsavsavaliya</a>!</p>",
          "rawMarkdown": "thanks @utsavsavaliya!"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:51:50Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8801764,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/darjeeling-tea-3rd-place-solution-span-mlm-t5-augm"
}