{
  "id": 369956,
  "title": "Team Turing: 9th Place Solution",
  "content": "<p>A big thanks to Kaggle and the competition hosts for organizing the amazing feedback series! We learned a lot during the competitions and especially enjoyed the addition of efficiency track in Feedback 2 &amp; 3. We look forward to participating in future competitions!</p>\n<h2>Special Mention</h2>\n<p>Congratulations to our teammate <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> for becoming a Kaggle Competitions Master! Very well deserved and this was special after many narrow gold misses! </p>\n<h2>Overview</h2>\n<p>Our solution focused on building a set of diverse models to facilitate better ensemble performance. To this end, we trained the following models which were finally used in our ensemble. The actual experiments trained were a lot more :) </p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Backbone</th>\n<th>Training Approach</th>\n<th>PL</th>\n<th>CV &nbsp;&nbsp; &nbsp;</th>\n<th>PB &nbsp;</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>exp009a</td>\n<td>deberta-v3-large</td>\n<td>PET</td>\n<td>Yes, 10k Random Samples</td>\n<td>0.4464</td>\n<td>0.4374</td>\n</tr>\n<tr>\n<td>exp022b</td>\n<td>deberta-v3-large</td>\n<td>Keywords</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4444</td>\n<td>0.4382</td>\n</tr>\n<tr>\n<td>exp024</td>\n<td>deberta-v3-large</td>\n<td>Multiscale</td>\n<td>No</td>\n<td>0.4493</td>\n<td>0.4399</td>\n</tr>\n<tr>\n<td>exp024a</td>\n<td>deberta-v3-large</td>\n<td>Multiscale, MSE + Ranking Loss</td>\n<td>No</td>\n<td>0.4484</td>\n<td>0.4424</td>\n</tr>\n<tr>\n<td>exp026b</td>\n<td>deberta-v3-large</td>\n<td>Freeze + Reinit</td>\n<td>Yes, 4k Random Samples</td>\n<td>0.4447</td>\n<td>0.4373</td>\n</tr>\n<tr>\n<td>exp027</td>\n<td>deberta-v3-large</td>\n<td>SETFIT</td>\n<td>No</td>\n<td>0.4470</td>\n<td>0.4371</td>\n</tr>\n<tr>\n<td>exp030</td>\n<td>deberta-v3-small</td>\n<td>MSE + Ranking Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4490</td>\n<td>0.4396</td>\n</tr>\n<tr>\n<td>exp120</td>\n<td>deberta-v3-small</td>\n<td>2 Stage PL + Loss Function</td>\n<td>Yes</td>\n<td>0.4586</td>\n<td>0.4467</td>\n</tr>\n<tr>\n<td>exp121</td>\n<td>luke-large</td>\n<td>Same as above</td>\n<td>Yes</td>\n<td>0.4567</td>\n<td>0.4462</td>\n</tr>\n<tr>\n<td>exp132</td>\n<td>roberta-large</td>\n<td>Same as above</td>\n<td>Yes</td>\n<td>0.4492</td>\n<td>0.4438</td>\n</tr>\n<tr>\n<td>exp203b</td>\n<td>deberta-large</td>\n<td>Prompt + Smooth L1 Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.448</td>\n<td>0.4380</td>\n</tr>\n<tr>\n<td>exp207a</td>\n<td>deberta-v3-large</td>\n<td>PET</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4468</td>\n<td>0.4381</td>\n</tr>\n<tr>\n<td>exp208</td>\n<td>deberta-large</td>\n<td>Prompt + Smooth L1 Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4501</td>\n<td></td>\n</tr>\n<tr>\n<td>exp300</td>\n<td>deberta-v3-base</td>\n<td>Clean Text MCRMSE Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4553</td>\n<td>0.4460</td>\n</tr>\n<tr>\n<td>exp302</td>\n<td>deberta-large</td>\n<td>GRU Head. <br> Mean Max Pool.<br>  Smooth L1 Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4549</td>\n<td>0.4460</td>\n</tr>\n<tr>\n<td>exp303d</td>\n<td>deberta-v3-large</td>\n<td>LSTM Head. Mean Max Pool.</td>\n<td>2 Stage PL</td>\n<td>0.4464</td>\n<td>0.4366</td>\n</tr>\n<tr>\n<td>exp310</td>\n<td>deberta-v3-large</td>\n<td></td>\n<td>2 Stage PL</td>\n<td>0.4508</td>\n<td>0.4387</td>\n</tr>\n<tr>\n<td>exp320</td>\n<td>deberta-v3-large</td>\n<td></td>\n<td>No</td>\n<td>0.4512</td>\n<td>0.4387</td>\n</tr>\n<tr>\n<td>SVR</td>\n<td>SVR Models trained on embeddings</td>\n<td></td>\n<td>No</td>\n<td>0.45</td>\n<td>0.4431</td>\n</tr>\n</tbody>\n</table>\n<h2>What Worked</h2>\n<ul>\n<li><p><strong>[High Impact] Model Diversity</strong></p>\n<ul>\n<li><strong>PET Based Approach</strong><ul>\n<li>Use the Pattern-Exploiting Training (PET) that reformulates input examples as cloze-style phrases to help language models understand a given task.</li>\n<li>Pattern:</li></ul></li></ul>\n<pre><code>   Student Essay Evaluation:\n\n   Performance on cohesion (essay organization; transition; logical sequencing)? [MASK]. Performance on syntax (sentence structure and formation; word order)? [MASK]. Performance on vocabulary (word diversity; topic related terms)? [MASK]. Performance on phraseology (phrases; idioms; collocations)? [MASK]. Performance on grammar? [MASK]. Performance on conventions (spelling; capitalization; punctuation; contractions)?[MASK]. [SEP]\n\n   Essay: {full_essay}\n</code></pre>\n<ul>\n<li>Verbalizer<ul>\n<li>Positive Tokens List: <code>['Excellent', 'Good', 'A', '5', '10', '100', 'excellent', 'good', 'High', 'high', 'Strong', 'strong']</code></li>\n<li>Negative Tokens List: <code>['1', 'C', 'D', 'F', 'Bad', 'Limited', 'limited', 'poor', 'Poor', '0', 'Low', 'low']</code></li></ul></li>\n<li>Additional Pattern/Verbalizer<ul>\n<li>Pattern: Example for cohesion here. </li></ul></li></ul>\n<pre><code>    Rate the following essay for cohesion (good text organization, transitional, overlap) with a score between 1 (bad) and 5 (great). The cohesion score for the following essay is 1 or 2 or 3 or 4 or 5? The score is [MASK]. {full_essay}\n</code></pre>\n<pre><code>- Verbalizer\n   - Positive Tokens List: `['3', '4', '5']`\n   - Negative Tokens List: `['1', '2']`\n</code></pre>\n<ul>\n<li><strong>Keywords Approach</strong><ul>\n<li>Create a set of keywords (words / phrases) by studying Feedback 3 that carries important signal regarding the different targets</li>\n<li>Use the <strong>**<code>Finding Dataset Shortcuts with Grammar Induction</code></strong> to identify the keywords</li>\n<li>Pre-process the essay by prefixing the identified keywords to the essay. For example:</li></ul></li></ul>\n<pre><code>   thomas jefferson, much we, always doing, this statement, it is always, a sense, it makes, makes you, more than, what you, you can also, the way, the first reason, thing is, it gives you, you worked, your school, real world, do more, i di, with my, my life, could make you, want to do, such a, give you, opportunity to, you may be, able to, how to, finishing school, you will take, that you learn, our life, in life, also a, trying to accomplish, you become, better person, in conclusion, it helps you, can alway [SEP] [SOE] Thomas Jefferson once states that &lt;……&gt; makes you feel accomplished in life, and shows that you can always strive to do more in you life [EOE]\n</code></pre>\n<ul>\n<li>Multi-scale approach<ul>\n<li>Predict target using different granularities of the essay<ul>\n<li>Scale 1: mean pooling over entire essay (standard)</li>\n<li>Scale 2: mean pooling over paragraphs, followed by LSTM and mean pooling transformation</li>\n<li>Scale 3: mean pooling over sentences, followed by LSTM and mean pooling transformation</li></ul></li>\n<li>Mean of 3-scale predictions to get the final score</li></ul></li>\n<li>SETFIT approach<ul>\n<li>Two step process<ul>\n<li>Step 1: Fine-tuning of a sentence-transformer in contrastive manner</li>\n<li>Step 2: Training a regressor using embeddings from step-1 fine-tuned sentence transformer</li></ul></li>\n<li>Reference: <a href=\"https://huggingface.co/blog/setfit\" target=\"_blank\">https://huggingface.co/blog/setfit</a></li></ul></li></ul></li>\n<li><p><strong>[High Impact] Ensemble</strong></p>\n<ul>\n<li>Result : Filter N models down to a manageable list keeping diversity in mind. </li>\n<li>Reasoning:  Given the large number of diverse experiments trained by our team, it was important to find a subset of diverse models that helped the overall CV. Hill climbing was the perfect approach for this. </li></ul></li>\n<li><p><strong>[High Impact] Pseudo Labelling</strong></p>\n<ul>\n<li>We carefully curated an unlabelled essay corpus and subsequently sampled essays from that corpus such that PL distribution matches the training distribution as closely as possible. For details,  please refer to our 1st place efficiency solution here: <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646</a></li>\n<li>Result: CV improved by around 0.003</li>\n<li>Reasoning: PL helped to distill knowledge from our best ensemble into the low capacity model used for efficiency. PL also has regularizing effect and often leads to better generalization.</li></ul></li>\n<li><p><strong>[High Impact] 2 Stage Pseudo Labeling</strong></p>\n<ul>\n<li>Result: CV improved 0.002 to 0.005</li>\n<li>Reasoning and Context:  There were 2 sets of pseudo labeling(PL) for the team</li>\n<li>Based on FB1 + FB2 - FB3 unlabeled data - Two-stage training works best. <br>\n[stage 1] Train a model with PL only, validate 5 times an epoch and save the checkpoint. <br>\n[stage 2] Load the checkpoint from stage 1 and fine-tuning on the train set with relatively smaller lr. </li>\n<li>Min-dist PL - The only difference is save the last checkpoint of stage 1</li></ul></li>\n<li><p><strong>High Impact] Freezing &amp; Re-initializing</strong></p>\n<ul>\n<li>Freeze bottom 6 layers of the model and re-initialized the top layer of the transformer backbone.</li>\n<li>Result: CV around 0.001 better</li>\n<li>For this task, it was helpful to freeze bottom transformer layers to keep intact lower level syntactic patterns learned by the backbone during its LM pre-training. On the other hand, re-initializing a top layer gave more flexibility to learn task specific signals.</li></ul></li>\n<li><p><strong>[High Impact] AWP &amp; EMA</strong></p>\n<ul>\n<li>Adversarial training via AWP helps the model to become more robust by inducing a flatter weight loss landscape. CV also improved by around 0.001</li>\n<li>EMA helps with generalization by keeping track of running average of model weights (similar to SWA)</li></ul></li>\n<li><p><strong>[High Impact] LSTM + MeanMax Pooling</strong></p>\n<ul>\n<li>Reasoning and Context: Tried different combinations of pooling and heads but LSTM + MeanMax pooling worked.</li></ul></li>\n<li><p><strong>[High Impact] 2 Stage Loss and PL Training</strong></p>\n<ul>\n<li>This methodology was used by some models to increase diversity.</li>\n<li>Stage 1: Use loss function X with Pseudo Labels and train for one epoch</li>\n<li>Stage 2: Use stage 1 checkpoint. Use another loss function Y for original train data and train for more epochs. Same loss function across stages gave worse CV vs different loss functions. </li></ul></li>\n<li><p><strong>[Medium Impact] Grouped differential learning rate</strong></p>\n<ul>\n<li>Grouped differential learning rate: This is very similar to standard LLRD. Split backbone's encoder layers into 3 groups, set lr per group. [e.g. sSet 5e-5/2e-5 as base_lr for high level group, base_lr / 2.6 for middle level group, base_lr / 2.6 ** 2 for low level group; for head and LSTM + MeanMax pooling, set 5e-3 or 1e-3]<br>\n​</li></ul></li>\n<li><p><strong>[Low Impact] MLM</strong></p>\n<ul>\n<li>Result: -~0.001</li>\n<li>Reasoning and Context: MLM seemed to only work with deberta-large and FB2 dataset. </li></ul></li>\n<li><p><strong>[Low Impact] Adding a new token for <code>\\n\\n</code></strong></p></li>\n<li><p><strong>[Low Impact] Ranking Loss</strong></p>\n<ul>\n<li>Adding ranking loss to standard MSE loss made aware of relative ranking of training examples within a batch</li></ul></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<ul>\n<li>Mask Augmentation</li>\n<li>Random Augmentation</li>\n<li>Auxiliary Target Training  -  add POS tagging, Statistical feature, … as auxiliary target </li>\n<li>Stacking - Add statistical and readability features to apply stacking with OOFs </li>\n<li>Paragraph Random Shuffle </li>\n</ul>\n<h2>Important Citations</h2>\n<ul>\n<li>Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (<a href=\"https://arxiv.org/abs/2001.07676\" target=\"_blank\">https://arxiv.org/abs/2001.07676</a>)</li>\n<li>Improving and Simplifying Pattern Exploiting Training (<a href=\"https://arxiv.org/abs/2103.11955\" target=\"_blank\">https://arxiv.org/abs/2103.11955</a>)</li>\n<li>SetFit: Efficient Few-Shot Learning Without Prompts (<a href=\"https://huggingface.co/blog/setfit\" target=\"_blank\">https://huggingface.co/blog/setfit</a>)</li>\n<li>On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation (<a href=\"https://aclanthology.org/2022.naacl-main.249/\" target=\"_blank\">https://aclanthology.org/2022.naacl-main.249/</a>)</li>\n<li>Finding Dataset Shortcuts with Grammar Induction (<a href=\"https://github.com/princeton-nlp/ShortcutGrammar\" target=\"_blank\">https://github.com/princeton-nlp/ShortcutGrammar</a>)</li>\n<li>SVR <a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a></li>\n</ul>\n<h2>Thanks and Acknowledgements</h2>\n<p>Thanks to the organizers and Kaggle for a very exciting competition!<br>\nMany thanks to my teammates (@conjuring92, <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a>, <a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a>) for such an amazing collaboration and perfect teamwork! I am sure we will be teaming up again for future competitions! </p>\n<h2>Team Members</h2>\n<p>Raja Biswas <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a><br>\nYao He <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a><br>\nHarshit Mehta <a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a><br>\nTrushant Kalyanpur <a href=\"https://www.kaggle.com/trushk\" target=\"_blank\">@trushk</a> </p>",
  "comments": [
    {
      "id": 2052428,
      "author_name": "yao",
      "author_url": "",
      "post_date": "2022-12-02T07:40:15.103000",
      "content": "<p>Thanks teammates thanks Trushant! Very well team work and unforgettable experience! Looking forward to the team up in the next competition! </p>",
      "votes": 5,
      "replies": [
        {
          "id": 2052435,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-12-02T07:46:37.317000",
          "content": "<p>Congratulations <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> 🎉🎉🎉 fully deserved!! </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2056694,
          "author_name": "Minh Tri Phan",
          "author_url": "",
          "post_date": "2022-12-06T11:17:33.517000",
          "content": "<p>Congrats <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> for becomming a Master!</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2056935,
          "author_name": "yao",
          "author_url": "",
          "post_date": "2022-12-06T15:37:48.380000",
          "content": "<p>Thank you Tri! <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a> </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2057866,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-07T11:55:00.607000",
          "content": "<p><a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> Congrats! I've seen your efforts on the leaderboards! well deserved !!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2058763,
          "author_name": "yao",
          "author_url": "",
          "post_date": "2022-12-08T07:33:58.517000",
          "content": "<p>Thanks! <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> Congrats to your strong silver!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2055295,
      "author_name": "chumajin",
      "author_url": "",
      "post_date": "2022-12-05T00:55:45.813000",
      "content": "<p><a href=\"https://www.kaggle.com/trushk\" target=\"_blank\">@trushk</a> Congrats!! They are very creative in many ways. I learned from it. That's one more gold medal to go before you become a GM. Good luck!</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2055452,
          "author_name": "Trushant Kalyanpur",
          "author_url": "",
          "post_date": "2022-12-05T05:00:16.693000",
          "content": "<p>Thank you my friend and congratulations on a strong solo 18th performance in this competition! Great work! Yes, one more gold to go for KGM! This solution describes our Team Turings collective team effort!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2052463,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2022-12-02T08:10:56.313000",
      "content": "<p>Great writeup. Congrats on the result.  What corpus did you use for this:</p>\n<blockquote>\n  <p>We carefully curated an unlabelled essay corpus </p>\n</blockquote>",
      "votes": 3,
      "replies": [
        {
          "id": 2052471,
          "author_name": "Trushant Kalyanpur",
          "author_url": "",
          "post_date": "2022-12-02T08:16:34.407000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/cpmpml\" target=\"_blank\">@cpmpml</a> ! The details of that are actually posted in our efficiency solution here: <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646</a>. I can also add a link to this post. </p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2055693,
      "author_name": "SchopenHacker75",
      "author_url": "",
      "post_date": "2022-12-05T10:18:59.570000",
      "content": "<p>Congratulations🎉🎉 and thanks a loot for sharing, <br>\nit's very clear and concise! <br>\nI learned many new techniques from your explanation💕</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2052422,
      "author_name": "nju_xuetianci",
      "author_url": "",
      "post_date": "2022-12-02T07:26:22.013000",
      "content": "<p>I want to ask if you are going to open source code? Because I think your method is so good and I have learned a lot of knowledge from it. I really want to learn more. If you are willing to share, I am really grateful.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2052436,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-12-02T07:47:26.267000",
          "content": "<p>yes, we will release our code soon!</p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 2052439,
          "author_name": "nju_xuetianci",
          "author_url": "",
          "post_date": "2022-12-02T07:50:17.490000",
          "content": "<p>Thank you very much, I can't wait to watch and learn！</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2059002,
          "author_name": "nju_xuetianci",
          "author_url": "",
          "post_date": "2022-12-08T11:48:04.013000",
          "content": "<p>I have been waiting for the code for a long time, I would like to ask, when will the code be released? I really want to learn from code</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2059015,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2022-12-08T11:59:22.577000",
          "content": "<p>yes, we are working on it, we will make the repo public latest by Dec 20th.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 16,
  "messages": [
    {
      "id": 2052369,
      "postDate": "2022-12-02T06:15:33.720Z",
      "content": "<p>A big thanks to Kaggle and the competition hosts for organizing the amazing feedback series! We learned a lot during the competitions and especially enjoyed the addition of efficiency track in Feedback 2 &amp; 3. We look forward to participating in future competitions!</p>\n<h2>Special Mention</h2>\n<p>Congratulations to our teammate <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> for becoming a Kaggle Competitions Master! Very well deserved and this was special after many narrow gold misses! </p>\n<h2>Overview</h2>\n<p>Our solution focused on building a set of diverse models to facilitate better ensemble performance. To this end, we trained the following models which were finally used in our ensemble. The actual experiments trained were a lot more :) </p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Backbone</th>\n<th>Training Approach</th>\n<th>PL</th>\n<th>CV &nbsp;&nbsp; &nbsp;</th>\n<th>PB &nbsp;</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>exp009a</td>\n<td>deberta-v3-large</td>\n<td>PET</td>\n<td>Yes, 10k Random Samples</td>\n<td>0.4464</td>\n<td>0.4374</td>\n</tr>\n<tr>\n<td>exp022b</td>\n<td>deberta-v3-large</td>\n<td>Keywords</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4444</td>\n<td>0.4382</td>\n</tr>\n<tr>\n<td>exp024</td>\n<td>deberta-v3-large</td>\n<td>Multiscale</td>\n<td>No</td>\n<td>0.4493</td>\n<td>0.4399</td>\n</tr>\n<tr>\n<td>exp024a</td>\n<td>deberta-v3-large</td>\n<td>Multiscale, MSE + Ranking Loss</td>\n<td>No</td>\n<td>0.4484</td>\n<td>0.4424</td>\n</tr>\n<tr>\n<td>exp026b</td>\n<td>deberta-v3-large</td>\n<td>Freeze + Reinit</td>\n<td>Yes, 4k Random Samples</td>\n<td>0.4447</td>\n<td>0.4373</td>\n</tr>\n<tr>\n<td>exp027</td>\n<td>deberta-v3-large</td>\n<td>SETFIT</td>\n<td>No</td>\n<td>0.4470</td>\n<td>0.4371</td>\n</tr>\n<tr>\n<td>exp030</td>\n<td>deberta-v3-small</td>\n<td>MSE + Ranking Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4490</td>\n<td>0.4396</td>\n</tr>\n<tr>\n<td>exp120</td>\n<td>deberta-v3-small</td>\n<td>2 Stage PL + Loss Function</td>\n<td>Yes</td>\n<td>0.4586</td>\n<td>0.4467</td>\n</tr>\n<tr>\n<td>exp121</td>\n<td>luke-large</td>\n<td>Same as above</td>\n<td>Yes</td>\n<td>0.4567</td>\n<td>0.4462</td>\n</tr>\n<tr>\n<td>exp132</td>\n<td>roberta-large</td>\n<td>Same as above</td>\n<td>Yes</td>\n<td>0.4492</td>\n<td>0.4438</td>\n</tr>\n<tr>\n<td>exp203b</td>\n<td>deberta-large</td>\n<td>Prompt + Smooth L1 Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.448</td>\n<td>0.4380</td>\n</tr>\n<tr>\n<td>exp207a</td>\n<td>deberta-v3-large</td>\n<td>PET</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4468</td>\n<td>0.4381</td>\n</tr>\n<tr>\n<td>exp208</td>\n<td>deberta-large</td>\n<td>Prompt + Smooth L1 Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4501</td>\n<td></td>\n</tr>\n<tr>\n<td>exp300</td>\n<td>deberta-v3-base</td>\n<td>Clean Text MCRMSE Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4553</td>\n<td>0.4460</td>\n</tr>\n<tr>\n<td>exp302</td>\n<td>deberta-large</td>\n<td>GRU Head. <br> Mean Max Pool.<br>  Smooth L1 Loss</td>\n<td>Yes, 4k Min-distance Samples</td>\n<td>0.4549</td>\n<td>0.4460</td>\n</tr>\n<tr>\n<td>exp303d</td>\n<td>deberta-v3-large</td>\n<td>LSTM Head. Mean Max Pool.</td>\n<td>2 Stage PL</td>\n<td>0.4464</td>\n<td>0.4366</td>\n</tr>\n<tr>\n<td>exp310</td>\n<td>deberta-v3-large</td>\n<td></td>\n<td>2 Stage PL</td>\n<td>0.4508</td>\n<td>0.4387</td>\n</tr>\n<tr>\n<td>exp320</td>\n<td>deberta-v3-large</td>\n<td></td>\n<td>No</td>\n<td>0.4512</td>\n<td>0.4387</td>\n</tr>\n<tr>\n<td>SVR</td>\n<td>SVR Models trained on embeddings</td>\n<td></td>\n<td>No</td>\n<td>0.45</td>\n<td>0.4431</td>\n</tr>\n</tbody>\n</table>\n<h2>What Worked</h2>\n<ul>\n<li><p><strong>[High Impact] Model Diversity</strong></p>\n<ul>\n<li><strong>PET Based Approach</strong><ul>\n<li>Use the Pattern-Exploiting Training (PET) that reformulates input examples as cloze-style phrases to help language models understand a given task.</li>\n<li>Pattern:</li></ul></li></ul>\n<pre><code>   Student Essay Evaluation:\n\n   Performance on cohesion (essay organization; transition; logical sequencing)? [MASK]. Performance on syntax (sentence structure and formation; word order)? [MASK]. Performance on vocabulary (word diversity; topic related terms)? [MASK]. Performance on phraseology (phrases; idioms; collocations)? [MASK]. Performance on grammar? [MASK]. Performance on conventions (spelling; capitalization; punctuation; contractions)?[MASK]. [SEP]\n\n   Essay: {full_essay}\n</code></pre>\n<ul>\n<li>Verbalizer<ul>\n<li>Positive Tokens List: <code>['Excellent', 'Good', 'A', '5', '10', '100', 'excellent', 'good', 'High', 'high', 'Strong', 'strong']</code></li>\n<li>Negative Tokens List: <code>['1', 'C', 'D', 'F', 'Bad', 'Limited', 'limited', 'poor', 'Poor', '0', 'Low', 'low']</code></li></ul></li>\n<li>Additional Pattern/Verbalizer<ul>\n<li>Pattern: Example for cohesion here. </li></ul></li></ul>\n<pre><code>    Rate the following essay for cohesion (good text organization, transitional, overlap) with a score between 1 (bad) and 5 (great). The cohesion score for the following essay is 1 or 2 or 3 or 4 or 5? The score is [MASK]. {full_essay}\n</code></pre>\n<pre><code>- Verbalizer\n   - Positive Tokens List: `['3', '4', '5']`\n   - Negative Tokens List: `['1', '2']`\n</code></pre>\n<ul>\n<li><strong>Keywords Approach</strong><ul>\n<li>Create a set of keywords (words / phrases) by studying Feedback 3 that carries important signal regarding the different targets</li>\n<li>Use the <strong>**<code>Finding Dataset Shortcuts with Grammar Induction</code></strong> to identify the keywords</li>\n<li>Pre-process the essay by prefixing the identified keywords to the essay. For example:</li></ul></li></ul>\n<pre><code>   thomas jefferson, much we, always doing, this statement, it is always, a sense, it makes, makes you, more than, what you, you can also, the way, the first reason, thing is, it gives you, you worked, your school, real world, do more, i di, with my, my life, could make you, want to do, such a, give you, opportunity to, you may be, able to, how to, finishing school, you will take, that you learn, our life, in life, also a, trying to accomplish, you become, better person, in conclusion, it helps you, can alway [SEP] [SOE] Thomas Jefferson once states that &lt;……&gt; makes you feel accomplished in life, and shows that you can always strive to do more in you life [EOE]\n</code></pre>\n<ul>\n<li>Multi-scale approach<ul>\n<li>Predict target using different granularities of the essay<ul>\n<li>Scale 1: mean pooling over entire essay (standard)</li>\n<li>Scale 2: mean pooling over paragraphs, followed by LSTM and mean pooling transformation</li>\n<li>Scale 3: mean pooling over sentences, followed by LSTM and mean pooling transformation</li></ul></li>\n<li>Mean of 3-scale predictions to get the final score</li></ul></li>\n<li>SETFIT approach<ul>\n<li>Two step process<ul>\n<li>Step 1: Fine-tuning of a sentence-transformer in contrastive manner</li>\n<li>Step 2: Training a regressor using embeddings from step-1 fine-tuned sentence transformer</li></ul></li>\n<li>Reference: <a href=\"https://huggingface.co/blog/setfit\" target=\"_blank\">https://huggingface.co/blog/setfit</a></li></ul></li></ul></li>\n<li><p><strong>[High Impact] Ensemble</strong></p>\n<ul>\n<li>Result : Filter N models down to a manageable list keeping diversity in mind. </li>\n<li>Reasoning:  Given the large number of diverse experiments trained by our team, it was important to find a subset of diverse models that helped the overall CV. Hill climbing was the perfect approach for this. </li></ul></li>\n<li><p><strong>[High Impact] Pseudo Labelling</strong></p>\n<ul>\n<li>We carefully curated an unlabelled essay corpus and subsequently sampled essays from that corpus such that PL distribution matches the training distribution as closely as possible. For details,  please refer to our 1st place efficiency solution here: <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646</a></li>\n<li>Result: CV improved by around 0.003</li>\n<li>Reasoning: PL helped to distill knowledge from our best ensemble into the low capacity model used for efficiency. PL also has regularizing effect and often leads to better generalization.</li></ul></li>\n<li><p><strong>[High Impact] 2 Stage Pseudo Labeling</strong></p>\n<ul>\n<li>Result: CV improved 0.002 to 0.005</li>\n<li>Reasoning and Context:  There were 2 sets of pseudo labeling(PL) for the team</li>\n<li>Based on FB1 + FB2 - FB3 unlabeled data - Two-stage training works best. <br>\n[stage 1] Train a model with PL only, validate 5 times an epoch and save the checkpoint. <br>\n[stage 2] Load the checkpoint from stage 1 and fine-tuning on the train set with relatively smaller lr. </li>\n<li>Min-dist PL - The only difference is save the last checkpoint of stage 1</li></ul></li>\n<li><p><strong>High Impact] Freezing &amp; Re-initializing</strong></p>\n<ul>\n<li>Freeze bottom 6 layers of the model and re-initialized the top layer of the transformer backbone.</li>\n<li>Result: CV around 0.001 better</li>\n<li>For this task, it was helpful to freeze bottom transformer layers to keep intact lower level syntactic patterns learned by the backbone during its LM pre-training. On the other hand, re-initializing a top layer gave more flexibility to learn task specific signals.</li></ul></li>\n<li><p><strong>[High Impact] AWP &amp; EMA</strong></p>\n<ul>\n<li>Adversarial training via AWP helps the model to become more robust by inducing a flatter weight loss landscape. CV also improved by around 0.001</li>\n<li>EMA helps with generalization by keeping track of running average of model weights (similar to SWA)</li></ul></li>\n<li><p><strong>[High Impact] LSTM + MeanMax Pooling</strong></p>\n<ul>\n<li>Reasoning and Context: Tried different combinations of pooling and heads but LSTM + MeanMax pooling worked.</li></ul></li>\n<li><p><strong>[High Impact] 2 Stage Loss and PL Training</strong></p>\n<ul>\n<li>This methodology was used by some models to increase diversity.</li>\n<li>Stage 1: Use loss function X with Pseudo Labels and train for one epoch</li>\n<li>Stage 2: Use stage 1 checkpoint. Use another loss function Y for original train data and train for more epochs. Same loss function across stages gave worse CV vs different loss functions. </li></ul></li>\n<li><p><strong>[Medium Impact] Grouped differential learning rate</strong></p>\n<ul>\n<li>Grouped differential learning rate: This is very similar to standard LLRD. Split backbone's encoder layers into 3 groups, set lr per group. [e.g. sSet 5e-5/2e-5 as base_lr for high level group, base_lr / 2.6 for middle level group, base_lr / 2.6 ** 2 for low level group; for head and LSTM + MeanMax pooling, set 5e-3 or 1e-3]<br>\n​</li></ul></li>\n<li><p><strong>[Low Impact] MLM</strong></p>\n<ul>\n<li>Result: -~0.001</li>\n<li>Reasoning and Context: MLM seemed to only work with deberta-large and FB2 dataset. </li></ul></li>\n<li><p><strong>[Low Impact] Adding a new token for <code>\\n\\n</code></strong></p></li>\n<li><p><strong>[Low Impact] Ranking Loss</strong></p>\n<ul>\n<li>Adding ranking loss to standard MSE loss made aware of relative ranking of training examples within a batch</li></ul></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<ul>\n<li>Mask Augmentation</li>\n<li>Random Augmentation</li>\n<li>Auxiliary Target Training  -  add POS tagging, Statistical feature, … as auxiliary target </li>\n<li>Stacking - Add statistical and readability features to apply stacking with OOFs </li>\n<li>Paragraph Random Shuffle </li>\n</ul>\n<h2>Important Citations</h2>\n<ul>\n<li>Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (<a href=\"https://arxiv.org/abs/2001.07676\" target=\"_blank\">https://arxiv.org/abs/2001.07676</a>)</li>\n<li>Improving and Simplifying Pattern Exploiting Training (<a href=\"https://arxiv.org/abs/2103.11955\" target=\"_blank\">https://arxiv.org/abs/2103.11955</a>)</li>\n<li>SetFit: Efficient Few-Shot Learning Without Prompts (<a href=\"https://huggingface.co/blog/setfit\" target=\"_blank\">https://huggingface.co/blog/setfit</a>)</li>\n<li>On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation (<a href=\"https://aclanthology.org/2022.naacl-main.249/\" target=\"_blank\">https://aclanthology.org/2022.naacl-main.249/</a>)</li>\n<li>Finding Dataset Shortcuts with Grammar Induction (<a href=\"https://github.com/princeton-nlp/ShortcutGrammar\" target=\"_blank\">https://github.com/princeton-nlp/ShortcutGrammar</a>)</li>\n<li>SVR <a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a></li>\n</ul>\n<h2>Thanks and Acknowledgements</h2>\n<p>Thanks to the organizers and Kaggle for a very exciting competition!<br>\nMany thanks to my teammates (@conjuring92, <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a>, <a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a>) for such an amazing collaboration and perfect teamwork! I am sure we will be teaming up again for future competitions! </p>\n<h2>Team Members</h2>\n<p>Raja Biswas <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a><br>\nYao He <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a><br>\nHarshit Mehta <a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a><br>\nTrushant Kalyanpur <a href=\"https://www.kaggle.com/trushk\" target=\"_blank\">@trushk</a> </p>",
      "rawMarkdown": "\nA big thanks to Kaggle and the competition hosts for organizing the amazing feedback series! We learned a lot during the competitions and especially enjoyed the addition of efficiency track in Feedback 2 & 3. We look forward to participating in future competitions!\n\n## Special Mention\nCongratulations to our teammate @syhens for becoming a Kaggle Competitions Master! Very well deserved and this was special after many narrow gold misses! \n\n## Overview\n\nOur solution focused on building a set of diverse models to facilitate better ensemble performance. To this end, we trained the following models which were finally used in our ensemble. The actual experiments trained were a lot more :) \n\n| Model | Backbone | Training Approach | PL | CV &nbsp;&nbsp; &nbsp;| PB &nbsp;|\n| --- | --- | --- | --- | --- | --- |\n| exp009a | deberta-v3-large | PET | Yes, 10k Random Samples | 0.4464 | 0.4374 |\n| exp022b | deberta-v3-large | Keywords | Yes, 4k Min-distance Samples | 0.4444 | 0.4382 |\n| exp024 | deberta-v3-large | Multiscale | No | 0.4493 | 0.4399 |\n| exp024a | deberta-v3-large | Multiscale, MSE + Ranking Loss | No | 0.4484 | 0.4424 |\n| exp026b | deberta-v3-large | Freeze + Reinit | Yes, 4k Random Samples | 0.4447 | 0.4373 |\n| exp027 | deberta-v3-large | SETFIT | No | 0.4470 | 0.4371 |\n| exp030 | deberta-v3-small | MSE + Ranking Loss | Yes, 4k Min-distance Samples | 0.4490 | 0.4396 |\n| exp120 | deberta-v3-small |2 Stage PL + Loss Function | Yes | 0.4586 | 0.4467 | \n| exp121 | luke-large | Same as above | Yes | 0.4567 | 0.4462 |\n| exp132 | roberta-large | Same as above  | Yes | 0.4492 | 0.4438 |\n| exp203b | deberta-large | Prompt + Smooth L1 Loss | Yes, 4k Min-distance Samples | 0.448 | 0.4380 |\n| exp207a | deberta-v3-large | PET | Yes, 4k Min-distance Samples | 0.4468 | 0.4381 |\n| exp208 | deberta-large |Prompt + Smooth L1 Loss | Yes, 4k Min-distance Samples | 0.4501 |  |\n| exp300 | deberta-v3-base | Clean Text MCRMSE Loss | Yes, 4k Min-distance Samples | 0.4553 | 0.4460 |\n| exp302 | deberta-large | GRU Head. <br> Mean Max Pool.<br>  Smooth L1 Loss| Yes, 4k Min-distance Samples | 0.4549 |0.4460 |\n| exp303d | deberta-v3-large | LSTM Head. Mean Max Pool. | 2 Stage PL | 0.4464 | 0.4366 |\n| exp310 | deberta-v3-large |  | 2 Stage PL | 0.4508 | 0.4387 |\n| exp320 | deberta-v3-large |  | No| 0.4512 | 0.4387 |\n| SVR | SVR Models trained on embeddings| | No | 0.45 | 0.4431 |\n\n\n\n## What Worked\n\n- **[High Impact] Model Diversity**\n    - **PET Based Approach**\n        - Use the Pattern-Exploiting Training (PET) that reformulates input examples as cloze-style phrases to help language models understand a given task.\n        - Pattern:\n        \n        ```\n        Student Essay Evaluation:\n        \n        Performance on cohesion (essay organization; transition; logical sequencing)? [MASK]. Performance on syntax (sentence structure and formation; word order)? [MASK]. Performance on vocabulary (word diversity; topic related terms)? [MASK]. Performance on phraseology (phrases; idioms; collocations)? [MASK]. Performance on grammar? [MASK]. Performance on conventions (spelling; capitalization; punctuation; contractions)?[MASK]. [SEP]\n        \n        Essay: {full_essay}\n        ```\n        \n        - Verbalizer\n            - Positive Tokens List: `['Excellent', 'Good', 'A', '5', '10', '100', 'excellent', 'good', 'High', 'high', 'Strong', 'strong']`\n            - Negative Tokens List: `['1', 'C', 'D', 'F', 'Bad', 'Limited', 'limited', 'poor', 'Poor', '0', 'Low', 'low']`\n\n        - Additional Pattern/Verbalizer\n         - Pattern: Example for cohesion here. \n        \n        ```\n         Rate the following essay for cohesion (good text organization, transitional, overlap) with a score between 1 (bad) and 5 (great). The cohesion score for the following essay is 1 or 2 or 3 or 4 or 5? The score is [MASK]. {full_essay}\n        ```\n        \n         - Verbalizer\n            - Positive Tokens List: `['3', '4', '5']`\n            - Negative Tokens List: `['1', '2']`\n\n\n    - **Keywords Approach**\n        - Create a set of keywords (words / phrases) by studying Feedback 3 that carries important signal regarding the different targets\n        - Use the ****`Finding Dataset Shortcuts with Grammar Induction`** to identify the keywords\n        - Pre-process the essay by prefixing the identified keywords to the essay. For example:\n        \n        ```\n        thomas jefferson, much we, always doing, this statement, it is always, a sense, it makes, makes you, more than, what you, you can also, the way, the first reason, thing is, it gives you, you worked, your school, real world, do more, i di, with my, my life, could make you, want to do, such a, give you, opportunity to, you may be, able to, how to, finishing school, you will take, that you learn, our life, in life, also a, trying to accomplish, you become, better person, in conclusion, it helps you, can alway [SEP] [SOE] Thomas Jefferson once states that <……> makes you feel accomplished in life, and shows that you can always strive to do more in you life [EOE]\n        ```\n        \n    - Multi-scale approach\n        - Predict target using different granularities of the essay\n            - Scale 1: mean pooling over entire essay (standard)\n            - Scale 2: mean pooling over paragraphs, followed by LSTM and mean pooling transformation\n            - Scale 3: mean pooling over sentences, followed by LSTM and mean pooling transformation\n        - Mean of 3-scale predictions to get the final score\n    - SETFIT approach\n        - Two step process\n            - Step 1: Fine-tuning of a sentence-transformer in contrastive manner\n            - Step 2: Training a regressor using embeddings from step-1 fine-tuned sentence transformer\n        - Reference: [https://huggingface.co/blog/setfit](https://huggingface.co/blog/setfit)\n\n- **[High Impact] Ensemble**\n    - Result : Filter N models down to a manageable list keeping diversity in mind. \n    - Reasoning:  Given the large number of diverse experiments trained by our team, it was important to find a subset of diverse models that helped the overall CV. Hill climbing was the perfect approach for this. \n\n- **[High Impact] Pseudo Labelling**\n    - We carefully curated an unlabelled essay corpus and subsequently sampled essays from that corpus such that PL distribution matches the training distribution as closely as possible. For details,  please refer to our 1st place efficiency solution here: https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646\n    - Result: CV improved by around 0.003\n    - Reasoning: PL helped to distill knowledge from our best ensemble into the low capacity model used for efficiency. PL also has regularizing effect and often leads to better generalization.\n\n- **[High Impact] 2 Stage Pseudo Labeling**\n  - Result: CV improved 0.002 to 0.005\n  - Reasoning and Context:  There were 2 sets of pseudo labeling(PL) for the team\n    * Based on FB1 + FB2 - FB3 unlabeled data - Two-stage training works best. \n    [stage 1] Train a model with PL only, validate 5 times an epoch and save the checkpoint. \n    [stage 2] Load the checkpoint from stage 1 and fine-tuning on the train set with relatively smaller lr. \n    *  Min-dist PL - The only difference is save the last checkpoint of stage 1\n\n- **High Impact] Freezing & Re-initializing**\n    - Freeze bottom 6 layers of the model and re-initialized the top layer of the transformer backbone.\n    - Result: CV around 0.001 better\n    - For this task, it was helpful to freeze bottom transformer layers to keep intact lower level syntactic patterns learned by the backbone during its LM pre-training. On the other hand, re-initializing a top layer gave more flexibility to learn task specific signals.\n\n- **[High Impact] AWP & EMA**\n    - Adversarial training via AWP helps the model to become more robust by inducing a flatter weight loss landscape. CV also improved by around 0.001\n    - EMA helps with generalization by keeping track of running average of model weights (similar to SWA)\n    \n- **[High Impact] LSTM + MeanMax Pooling**\n    - Reasoning and Context: Tried different combinations of pooling and heads but LSTM + MeanMax pooling worked.\n\n- **[High Impact] 2 Stage Loss and PL Training**\n  - This methodology was used by some models to increase diversity.\n  - Stage 1: Use loss function X with Pseudo Labels and train for one epoch\n  - Stage 2: Use stage 1 checkpoint. Use another loss function Y for original train data and train for more epochs. Same loss function across stages gave worse CV vs different loss functions. \n\n- **[Medium Impact] Grouped differential learning rate**\n  - Grouped differential learning rate: This is very similar to standard LLRD. Split backbone's encoder layers into 3 groups, set lr per group. [e.g. sSet 5e-5/2e-5 as base_lr for high level group, base_lr / 2.6 for middle level group, base_lr / 2.6 ** 2 for low level group; for head and LSTM + MeanMax pooling, set 5e-3 or 1e-3]\n​\n- **[Low Impact] MLM**\n  - Result: -~0.001\n  - Reasoning and Context: MLM seemed to only work with deberta-large and FB2 dataset. \n- **[Low Impact] Adding a new token for `\\n\\n`**\n- **[Low Impact] Ranking Loss**\n    - Adding ranking loss to standard MSE loss made aware of relative ranking of training examples within a batch\n\n## What Didn’t Work\n\n- Mask Augmentation\n- Random Augmentation\n- Auxiliary Target Training  -  add POS tagging, Statistical feature, ... as auxiliary target \n- Stacking - Add statistical and readability features to apply stacking with OOFs \n- Paragraph Random Shuffle \n\n## Important Citations\n\n- Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference ([https://arxiv.org/abs/2001.07676](https://arxiv.org/abs/2001.07676))\n- Improving and Simplifying Pattern Exploiting Training ([https://arxiv.org/abs/2103.11955](https://arxiv.org/abs/2103.11955))\n- SetFit: Efficient Few-Shot Learning Without Prompts ([https://huggingface.co/blog/setfit](https://huggingface.co/blog/setfit))\n- On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation ([https://aclanthology.org/2022.naacl-main.249/](https://aclanthology.org/2022.naacl-main.249/))\n- Finding Dataset Shortcuts with Grammar Induction ([https://github.com/princeton-nlp/ShortcutGrammar](https://github.com/princeton-nlp/ShortcutGrammar))\n- SVR https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\n\n## Thanks and Acknowledgements\nThanks to the organizers and Kaggle for a very exciting competition!\nMany thanks to my teammates (@conjuring92, @syhens, @harshit92) for such an amazing collaboration and perfect teamwork! I am sure we will be teaming up again for future competitions! \n\n## Team Members\n\nRaja Biswas @conjuring92\nYao He @syhens\nHarshit Mehta @harshit92\nTrushant Kalyanpur @trushk ",
      "votes": 34
    },
    {
      "id": 2052428,
      "postDate": "2022-12-02T07:40:15.103Z",
      "content": "<p>Thanks teammates thanks Trushant! Very well team work and unforgettable experience! Looking forward to the team up in the next competition! </p>",
      "rawMarkdown": "Thanks teammates thanks Trushant! Very well team work and unforgettable experience! Looking forward to the team up in the next competition! ",
      "votes": 5,
      "replies": [
        {
          "id": 2052435,
          "postDate": "2022-12-02T07:46:37.317Z",
          "content": "<p>Congratulations <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> 🎉🎉🎉 fully deserved!! </p>",
          "rawMarkdown": "Congratulations @syhens 🎉🎉🎉 fully deserved!! ",
          "votes": 2
        },
        {
          "id": 2056694,
          "postDate": "2022-12-06T11:17:33.517Z",
          "content": "<p>Congrats <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> for becomming a Master!</p>",
          "rawMarkdown": "Congrats @syhens for becomming a Master!",
          "votes": 2
        },
        {
          "id": 2056935,
          "postDate": "2022-12-06T15:37:48.380Z",
          "content": "<p>Thank you Tri! <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a> </p>",
          "rawMarkdown": "Thank you Tri! @shinomoriaoshi ",
          "votes": 1
        },
        {
          "id": 2057866,
          "postDate": "2022-12-07T11:55:00.607Z",
          "content": "<p><a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> Congrats! I've seen your efforts on the leaderboards! well deserved !!</p>",
          "rawMarkdown": "@syhens Congrats! I've seen your efforts on the leaderboards! well deserved !!",
          "votes": 1
        },
        {
          "id": 2058763,
          "postDate": "2022-12-08T07:33:58.517Z",
          "content": "<p>Thanks! <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> Congrats to your strong silver!</p>",
          "rawMarkdown": "Thanks! @chumajin Congrats to your strong silver!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2055295,
      "postDate": "2022-12-05T00:55:45.813Z",
      "content": "<p><a href=\"https://www.kaggle.com/trushk\" target=\"_blank\">@trushk</a> Congrats!! They are very creative in many ways. I learned from it. That's one more gold medal to go before you become a GM. Good luck!</p>",
      "rawMarkdown": "@trushk Congrats!! They are very creative in many ways. I learned from it. That's one more gold medal to go before you become a GM. Good luck!",
      "votes": 3,
      "replies": [
        {
          "id": 2055452,
          "postDate": "2022-12-05T05:00:16.693Z",
          "content": "<p>Thank you my friend and congratulations on a strong solo 18th performance in this competition! Great work! Yes, one more gold to go for KGM! This solution describes our Team Turings collective team effort!</p>",
          "rawMarkdown": "Thank you my friend and congratulations on a strong solo 18th performance in this competition! Great work! Yes, one more gold to go for KGM! This solution describes our Team Turings collective team effort!",
          "votes": 2
        }
      ]
    },
    {
      "id": 2052463,
      "postDate": "2022-12-02T08:10:56.313Z",
      "content": "<p>Great writeup. Congrats on the result.  What corpus did you use for this:</p>\n<blockquote>\n  <p>We carefully curated an unlabelled essay corpus </p>\n</blockquote>",
      "rawMarkdown": "Great writeup. Congrats on the result.  What corpus did you use for this:\n> We carefully curated an unlabelled essay corpus ",
      "votes": 3,
      "replies": [
        {
          "id": 2052471,
          "postDate": "2022-12-02T08:16:34.407Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/cpmpml\" target=\"_blank\">@cpmpml</a> ! The details of that are actually posted in our efficiency solution here: <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646</a>. I can also add a link to this post. </p>",
          "rawMarkdown": "Thanks @cpmpml ! The details of that are actually posted in our efficiency solution here: https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369646. I can also add a link to this post. \n",
          "votes": 2
        }
      ]
    },
    {
      "id": 2055693,
      "postDate": "2022-12-05T10:18:59.570Z",
      "content": "<p>Congratulations🎉🎉 and thanks a loot for sharing, <br>\nit's very clear and concise! <br>\nI learned many new techniques from your explanation💕</p>",
      "rawMarkdown": "Congratulations🎉🎉 and thanks a loot for sharing, \nit's very clear and concise! \nI learned many new techniques from your explanation💕",
      "votes": 1
    },
    {
      "id": 2052422,
      "postDate": "2022-12-02T07:26:22.013Z",
      "content": "<p>I want to ask if you are going to open source code? Because I think your method is so good and I have learned a lot of knowledge from it. I really want to learn more. If you are willing to share, I am really grateful.</p>",
      "rawMarkdown": "I want to ask if you are going to open source code? Because I think your method is so good and I have learned a lot of knowledge from it. I really want to learn more. If you are willing to share, I am really grateful.",
      "votes": 2,
      "replies": [
        {
          "id": 2052436,
          "postDate": "2022-12-02T07:47:26.267Z",
          "content": "<p>yes, we will release our code soon!</p>",
          "rawMarkdown": "yes, we will release our code soon!",
          "votes": 4
        },
        {
          "id": 2052439,
          "postDate": "2022-12-02T07:50:17.490Z",
          "content": "<p>Thank you very much, I can't wait to watch and learn！</p>",
          "rawMarkdown": "Thank you very much, I can't wait to watch and learn！",
          "votes": 2
        },
        {
          "id": 2059002,
          "postDate": "2022-12-08T11:48:04.013Z",
          "content": "<p>I have been waiting for the code for a long time, I would like to ask, when will the code be released? I really want to learn from code</p>",
          "rawMarkdown": "I have been waiting for the code for a long time, I would like to ask, when will the code be released? I really want to learn from code"
        },
        {
          "id": 2059015,
          "postDate": "2022-12-08T11:59:22.577Z",
          "content": "<p>yes, we are working on it, we will make the repo public latest by Dec 20th.</p>",
          "rawMarkdown": "yes, we are working on it, we will make the repo public latest by Dec 20th."
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:04:22Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9090383,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/team-turing-team-turing-9th-place-solution"
}