{
  "id": 446524,
  "title": "4th place solution",
  "content": "<p>Thank you very much for organizing such an interesting competition. I am greatly thankful to the hosts and the Kaggle staff.</p>\n<p>This competition turned out to be exactly what we had imagined and we are really happy to have won 4th place.</p>\n<p>And thank you so much <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> for teaming up with me and for the daily discussions and experiments. Congratulations on your 4th GOLD MEDAL.<br>\n(And I will become competition Grand Master!!!).<br>\nWe are a great team!</p>\n<h1>1. Summary</h1>\n<p>Our best private solution is 1 fulltrain x 7 models ensemble.</p>\n<p>This competition had only 4 prompts in the training data and with the information that 122 prompts were in the test data, one could imagine a big shake. Also, the public lb was very unstable. These things made us think that robustness is important, not only for cv.</p>\n<p>I used all 4kfold at first. Especially the models with prompt text, because I had to increase the maxlen to increase the score, I had to increase the inference time and could not ensemble other models (only 2-3 models could be included).</p>\n<p>Just then, I and kuro have team merged. And we mixed the fulltrain idea he had been working on. By replacing 4kfold with 1fulltrain, we arrived at the idea of compressing the information of 4kfold to 1fulltrain and ensemble more models.</p>\n<p>We imagined that by doing so, we would prevent the variation and reduction of scores on each prompt in the private test data and, in total, get better score.</p>\n<p>As a result, we believe we were able to prevent a shake down and finish in this position!</p>\n<table>\n<thead>\n<tr>\n<th>sub</th>\n<th>type</th>\n<th>models</th>\n<th>cv</th>\n<th>public lb</th>\n<th>private lb</th>\n<th>rank</th>\n<th>comment</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>sub1</td>\n<td>best lb</td>\n<td>1 fulltrain × 9 models</td>\n<td>0.4679</td>\n<td>0.41991</td>\n<td>0.45785</td>\n<td>11</td>\n<td>using gbdt and expand of inference-maxlen</td>\n</tr>\n<tr>\n<td>sub2</td>\n<td>best cv</td>\n<td>1 fulltrain × 7 models</td>\n<td>0.4639</td>\n<td>0.42979</td>\n<td>0.45515</td>\n<td>4</td>\n<td>just NLP result(expand of inference-maxlen)</td>\n</tr>\n<tr>\n<td>sub3</td>\n<td>insurance</td>\n<td>1 fulltrain × 8 models</td>\n<td>0.4693</td>\n<td>0.43855</td>\n<td>0.45597</td>\n<td>6</td>\n<td>just NLP result</td>\n</tr>\n</tbody>\n</table>\n<p>The best cv in sub2 is our best private, and that is the main explanation below.</p>\n<h1>2. training</h1>\n<p>Each of our models in sub2 is shown below.</p>\n<table>\n<thead>\n<tr>\n<th>modelno</th>\n<th>InfNo</th>\n<th>model</th>\n<th>training<br>maxlen</th>\n<th>inference <br>maxlen</th>\n<th>freezing</th>\n<th>layerwise</th>\n<th>input</th>\n<th>pooling</th>\n<th></th>\n<th></th>\n<th>2nd loss</th>\n<th>preprocess</th>\n<th>cv of 4kfold<br> earlystop</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>original prompt</td>\n<td>cls</td>\n<td>attention of <br>original prompt</td>\n<td>mean of <br>text part only</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>1</td>\n<td>91</td>\n<td>deberta-v3-large</td>\n<td>768</td>\n<td>1500</td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4818</td>\n</tr>\n<tr>\n<td>2</td>\n<td>22</td>\n<td>deberta-v3-large</td>\n<td>1050</td>\n<td>950</td>\n<td></td>\n<td>✓</td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4855</td>\n</tr>\n<tr>\n<td>3</td>\n<td>63</td>\n<td>deberta-v3-large</td>\n<td>850</td>\n<td>1500</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>0.4984</td>\n</tr>\n<tr>\n<td>4</td>\n<td>72</td>\n<td>deberta-v3-large</td>\n<td>868</td>\n<td>1024</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>Arcface</td>\n<td>✓</td>\n<td>0.4919</td>\n</tr>\n<tr>\n<td>5</td>\n<td>2,3</td>\n<td>deberta-v3-large</td>\n<td>868</td>\n<td>868</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4880</td>\n</tr>\n<tr>\n<td>6</td>\n<td>259</td>\n<td>deberta-v3-large-squad2</td>\n<td>768</td>\n<td>1500</td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4952</td>\n</tr>\n<tr>\n<td>7</td>\n<td>331</td>\n<td>deberta-v3-large-squad2</td>\n<td>1050</td>\n<td>950</td>\n<td></td>\n<td>✓</td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4993</td>\n</tr>\n</tbody>\n</table>\n<p>The details are described below.</p>\n<h2>2.1 model no1 : basemodel</h2>\n<p>This model is a typical base for our model.</p>\n<p>First, two inputs are prepared and put into the tokenizer as a pair (an example of dataset). The same result can be obtained by connecting them with [SEP] even if you do not put them as a pair.</p>\n<pre><code>self = self.\nself = self +  + self +  + self\n\ntokens = tokenizer(\n            self, \n            self,.\n            ...\n        )\n</code></pre>\n<p>Now all we have to do is put this in the model, increase the maxlen, and output the cls in the fc layer.</p>\n<h2>2.2 model no2 : original prompt</h2>\n<p>In this model, an original prompt was created and used for input.</p>\n<pre><code>self =  + self\nself = prompt_title +  + prompt_question +  + prompt_text\n\ntokens = tokenizer(\n            self, \n            self,\n            ・・・\n        )\n</code></pre>\n<p>Then, only the part of the original prompt (Evaluating the summarized…) is attentional pooled (an example of a model).</p>\n<pre><code>## init\n.pool = AttentionPooling(..hidden_size)\n...\n\n## forward\n = .model(ids, mask, token_type_ids)\n = [][:,:,:]\n = .pool(,mask[:,:])\n = .fc()\n</code></pre>\n<h2>2.3 model no4 : using 2nd loss and preprocess</h2>\n<p>As discussed in the discussion and notebook, the train data can be classified into 38 types. I thought I could make good use of this, so I included arcface as an auxiliary loss.<br>\nI tried to make good use of embeddings at the end, but it was not available. However, it did contribute to the diversity of the ensemble.</p>\n<p>Also, as a preprocss, I made sure to include a space after the period and comma; after doing EDA, I noticed that if a sentence comes after a period or comma without a space, it is divided differently in the tokenizer. However, I don't think this affected the score significantly.</p>\n<h2>2.4 Extending the inference max length</h2>\n<p>Scores have been greatly improved by increasing the inference length over that of training. An example is shown below. The following was trained with maxlen 850, but by setting maxlen to 1500 during inference, cv and public lb improved (maybe not so much contribution to private).</p>\n<table>\n<thead>\n<tr>\n<th>traing<br>maxlen</th>\n<th>inference<br>maxlen</th>\n<th>39c16e<br>fold0</th>\n<th>814d6b<br>fold1</th>\n<th>ebad26<br>fold2</th>\n<th>3b9047<br>fold3</th>\n<th>cv</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>850</td>\n<td>850</td>\n<td>0.4505</td>\n<td>0.5595</td>\n<td>0.5051</td>\n<td>0.5024</td>\n<td>0.4984</td>\n</tr>\n<tr>\n<td>850</td>\n<td>1024</td>\n<td>0.4524</td>\n<td>0.5590</td>\n<td>0.4836</td>\n<td>0.5018</td>\n<td>0.4927</td>\n</tr>\n<tr>\n<td>850</td>\n<td>1500</td>\n<td>0.4527</td>\n<td>0.5588</td>\n<td>0.4614</td>\n<td>0.5013</td>\n<td>0.4867</td>\n</tr>\n</tbody>\n</table>\n<p>However, some models may degrade. insurance submission(sub3) consisted only of models that did not expand this inference max length. As a result, cv and public were affected, but private had almost no effect.</p>\n<h1>3. inference (ensemble)</h1>\n<p>For best cv sub, we initially post-processed using GBDT (LGBM + Catboost + XGboost), but in the end, ensemble without GBDT resulted in a better CV, The submission that produced the best private did not use post-processing with GBDT. We also considered nealder-mead and optuna, but we did not use them because of the concern that they would definitely overfit. So we use the simple mean ensemble using a hill climb technique (we also added a full train with different seed to further increase robustness for the extra inference time).</p>\n<p>final cv(4kfold earlystopping result) : 0.46394 , public : 0.42979, private 0.45515 4th</p>\n<p>※ Futhermore, we took as insurance(sub3) that the inference max lengths were the same as that in training. This result was as follows.</p>\n<p>final cv 0.4693 , public : 0.43855, private 0.45597 6th.</p>\n<p>In this result, cv and lb were bad, but private was good, which shows how important the robustness of model was in this competition.</p>\n<h1>4. Not working for us</h1>\n<ul>\n<li>mlm</li>\n<li>awp</li>\n<li>svr</li>\n<li>other model</li>\n<li>regression from classification model</li>\n<li>other many many things…</li>\n</ul>\n<h1>5. Acknowledgements</h1>\n<p>We could not have achieved these results on our own. We were greatly influenced by those who we have collaborated with in the past, and we are grateful for their contributions. We would also like to express our sincere gratitude to those who have shared their knowledge and insights through previous competitions. Thank you very much.</p>\n<p>We would especially like to thank the following people for their help in this  competition! Thank you very much.</p>\n<ul>\n<li><p>Fulltrain : Team Hydrogen <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369578\" target=\"_blank\">ref1</a> <a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536\" target=\"_blank\">ref2</a>, Raja Biswas <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/424330#2354068\" target=\"_blank\">ref3</a></p></li>\n<li><p>Freezing : Takamichi Toda <a href=\"https://www.kaggle.com/takamichitoda\" target=\"_blank\">@takamichitoda</a>  <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/433754#2405543\" target=\"_blank\">ref4</a></p></li>\n<li><p>37-38 classification : MOTH <a href=\"https://www.kaggle.com/alejopaullier\" target=\"_blank\">@alejopaullier</a> <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/430705\" target=\"_blank\">ref5</a>, <br>\nAlexander Chervov <a href=\"https://www.kaggle.com/alexandervc\" target=\"_blank\">@alexandervc</a> <a href=\"https://www.kaggle.com/code/alexandervc/commonlit-levenshtein-distances?scriptVersionId=141542492&amp;cellId=23\" target=\"_blank\">ref6</a></p></li>\n<li><p>Postprocess : nogawanogawa <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> <a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">ref7</a></p></li>\n<li><p>Model selection(feedback3) : CHRIS DEOTTE <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369609\" target=\"_blank\">ref8</a>, TOM <a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369567\" target=\"_blank\">ref9</a></p></li>\n</ul>\n<h1>6. team member</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a></li>\n<li><a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> </li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fa311809edebd32836455da5b13e1dc1e%2Fkuromajin.jpg?generation=1697084418755389&amp;alt=media\" alt=\"\"></p>\n<h1>7. code</h1>\n<p>inference code : <a href=\"https://www.kaggle.com/code/chumajin/commonlit2-4th-place-inference\" target=\"_blank\">https://www.kaggle.com/code/chumajin/commonlit2-4th-place-inference</a> (This is the cleaned code. Same score as the submission.)</p>\n<p>training code(chumajin part) : <a href=\"https://github.com/chumajin/kaggle-commonlit\" target=\"_blank\">https://github.com/chumajin/kaggle-commonlit</a><br>\ntraining code(kuro_B part) : <a href=\"https://github.com/kurokuroB/kaggle-commonlit2-4th_place_solution-my_training_code\" target=\"_blank\">https://github.com/kurokuroB/kaggle-commonlit2-4th_place_solution-my_training_code</a></p>",
  "comments": [
    {
      "id": 2479284,
      "author_name": "moth",
      "author_url": "",
      "post_date": "2023-10-12T14:24:34.847000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> for the win and <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> for becoming competition GM, impressive achievement!</p>\n<p>I found the <code>max_len</code> change for inference fascinating. Could you share details on your training configuration such as batch size, learning rate, epochs? </p>\n<p>To me the most difficult thing was getting a stable Train Loss vs. Valid Loss. My models would not only overfit quickly (usually after 2 epochs) but there would also be large variances in performances across different seeds.</p>\n<p>Thanks in advance</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2479473,
          "author_name": "kuro_B",
          "author_url": "",
          "post_date": "2023-10-12T16:28:49.030000",
          "content": "<p>Thanks for your comment!<br>\nWe used different configuration each other.<br>\nMy main settting is batch size 8, learning rate 3e-5, cosine warmup 0.2, epoch 2. (model no 1, 6)</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2478571,
      "author_name": "HarshitMehta",
      "author_url": "",
      "post_date": "2023-10-12T04:58:49.003000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> and <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> (also for becoming competition GM!!). </p>",
      "votes": 3,
      "replies": [
        {
          "id": 2478660,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T06:22:29.067000",
          "content": "<p><a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a> I'm most happy to receive congratulations from you. You are the key person in my kaggle life.<br>\nLet's do it together again someday.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2480347,
      "author_name": "HideBu",
      "author_url": "",
      "post_date": "2023-10-13T09:19:38.283000",
      "content": "<p>Congratuations <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> for 4th place in this competition. </p>\n<p>I have a question about your approach. Please let me know.</p>\n<p>[Question]<br>\nWhat does fulltrain strategy mean? </p>\n<p>I assume that you use all train data for training and no validation, is it right?<br>\nIf it is right, how do you define training condition such as n-epoch, lr, and so on.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2480418,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-13T09:58:10.620000",
          "content": "<p><a href=\"https://www.kaggle.com/hidebu\" target=\"_blank\">@hidebu</a> <br>\nThank you for question.<br>\nI show the process as follows:<br>\n1) We trained 4kfold. We can get 4 model weights and calculate cv.<br>\n2) If the cv result of 1) is good, we trained using all train data. In this train, we use the condition which is the same as 1) including the n-epoch, lr and so on.<br>\n3) for inference, we used the one model which was made in 2). We did not use the model weights which was made in 1).</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2480474,
              "author_name": "RB",
              "author_url": "",
              "post_date": "2023-10-13T10:33:26.860000",
              "content": "<p>Thank you for sharing your approach. </p>\n<p>When you train 4 Kfold, I am guessing you have last epoch = best epoch ?   (And not save model when best validation loss is achieved in any epoch) </p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2480532,
              "author_name": "chumajin",
              "author_url": "",
              "post_date": "2023-10-13T11:32:24.220000",
              "content": "<p><a href=\"https://www.kaggle.com/rashmibanthia\" target=\"_blank\">@rashmibanthia</a> Thank you for comment !! This is a good question. You are right. To be precise, we did several validation evaluations per epoch at 4kfold training, at which point we calculated the oof after the 4kfold training and stopped learning the fulltrain at the point where the oof was the highest. However, many of the oofs were stable at the very end of the last epoch, and we often let them run to the end.</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2480325,
      "author_name": "Takamichi Toda",
      "author_url": "",
      "post_date": "2023-10-13T08:49:47.777000",
      "content": "<p>Congratulations becoming competition GM🎉 🎉 </p>\n<p>This competition didn't go well for me at all, so I'm planning to review various things, including your approach, moving forward.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2480414,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-13T09:53:53.387000",
          "content": "<p><a href=\"https://www.kaggle.com/takamichitoda\" target=\"_blank\">@takamichitoda</a> <br>\nThank you very much! Your posts on discussion were very helpful. Again, I have looked at your blog many times! Thank you every time!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2479666,
      "author_name": "Hussein Ah95",
      "author_url": "",
      "post_date": "2023-10-12T19:11:51.940000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> for Win and becoming competition GM </p>",
      "votes": 1,
      "replies": [
        {
          "id": 2479889,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-13T01:20:01.473000",
          "content": "<p><a href=\"https://www.kaggle.com/husseinah95\" target=\"_blank\">@husseinah95</a> Thank you very much!! </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2478895,
      "author_name": "Ivan Aerlic",
      "author_url": "",
      "post_date": "2023-10-12T09:01:13.453000",
      "content": "<p><a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> Thanks for the detailed write-up. I think you did a good job on not over-fitting the CV.</p>\n<p>Also, congratulations on becoming a competition grandmaster. A difficult accomplishment and I'm happy that your hard work has paid off.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478918,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T09:24:44.903000",
          "content": "<p><a href=\"https://www.kaggle.com/ivanaerlic\" target=\"_blank\">@ivanaerlic</a> Thank you very much!! I was really afraid to just shake down! Competition GM has been a tough road, but it feels great to have accomplished it. See you again at one of the competitions!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2478890,
      "author_name": "Muhammad Ahmed",
      "author_url": "",
      "post_date": "2023-10-12T08:57:06.913000",
      "content": "<p>Congratulations on winning the gold and becoming competition GM! 🎉</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478892,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T08:58:49.703000",
          "content": "<p><a href=\"https://www.kaggle.com/muhammad4hmed\" target=\"_blank\">@muhammad4hmed</a> Thank you so much!!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2478575,
      "author_name": "takai380",
      "author_url": "",
      "post_date": "2023-10-12T05:04:59.697000",
      "content": "<p>First, thank you for sharing the solution. And congratulations on becoming the competition GrandMaster!!<br>\nAmong the teams we observed attempting to shake down, I believe your team was able to achieve stable results without overfitting to the CV by creating a robust model. Congratulations!</p>\n<p>The idea of utilizing a full train model is also being used by our team, and its usefulness in reducing inference time was confirmed here. While we had the idea of solving the sub task of the classification problem, I felt there might be differences here compared to the gold zone.<br>\nUltimately, I strongly felt that the most important aspect is not to overfit to the CV due to the limited amount of data.</p>\n<p>Once again, congratulations on becoming the competition GrandMaster!!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478667,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T06:26:57.963000",
          "content": "<p><a href=\"https://www.kaggle.com/takai380\" target=\"_blank\">@takai380</a> Thank you very much. This is my personal opinion, but maybe one of the reason why we did not shake down is because we did not use the model without prompt text.  In our case, the model without prompt text did raise the cv by gbdt, but it did not raise the cv like prompt id 3b9047 compared to that with prompt text. If there were more prompts like 3b9047 in the private test, we would have expected the score to go down.</p>",
          "votes": 3,
          "replies": [
            {
              "id": 2478684,
              "author_name": "takai380",
              "author_url": "",
              "post_date": "2023-10-12T06:34:28.390000",
              "content": "<p>You may have a point. Thanks for your response! Congratulations!</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2480485,
              "author_name": "RB",
              "author_url": "",
              "post_date": "2023-10-13T10:40:52.500000",
              "content": "<blockquote>\n  <p>This is my personal opinion, but maybe one of the reason why we did not shake down is because we did not use the model without prompt text.</p>\n</blockquote>\n<p>This 💯 - I noticed this in your solution and <a href=\"https://www.kaggle.com/ivanaerlic\" target=\"_blank\">@ivanaerlic</a> solution as well. <br>\nOur very last submission was with 3 models with prompt text but it was not the best cv (but best lb) and too late to refine it by adding more models with prompt_text. </p>",
              "votes": 2,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2478545,
      "author_name": "A cat hidden in the cloud",
      "author_url": "",
      "post_date": "2023-10-12T04:43:01.750000",
      "content": "<p>Thanks for sharing, you have done a nice job! May I ask how you choose best model with fulltrain idea?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478657,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T06:21:22.437000",
          "content": "<p><a href=\"https://www.kaggle.com/nlchen\" target=\"_blank\">@nlchen</a> Thank you for comment ! we used the cv when we created the same model with 4kfold, and hill climb method.</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2478817,
              "author_name": "A cat hidden in the cloud",
              "author_url": "",
              "post_date": "2023-10-12T08:17:31.920000",
              "content": "<p>Thanks a lot!</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2478535,
      "author_name": "Shiner",
      "author_url": "",
      "post_date": "2023-10-12T04:35:16.403000",
      "content": "<p>Great solution, congrats! Since you team used fulltrain strategy, may I ask how do you do cv? Just random sample a few samples from train dataset? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478656,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T06:20:19.983000",
          "content": "<p><a href=\"https://www.kaggle.com/judith007\" target=\"_blank\">@judith007</a> Thank you for your comment! Instead, we used the cv when we created the same model with 4kfold! (4kfold model is not used for inference)</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2478686,
              "author_name": "Shiner",
              "author_url": "",
              "post_date": "2023-10-12T06:35:44.187000",
              "content": "<p>Thank you. Do you mean that you did cv using 4 kfold, but the models from 4 kfold cv were not used in inference stage, and you just retrained a model on full dataset and used this model in inference stage?</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2478704,
              "author_name": "chumajin",
              "author_url": "",
              "post_date": "2023-10-12T06:59:39.637000",
              "content": "<p><a href=\"https://www.kaggle.com/judith007\" target=\"_blank\">@judith007</a> yes. you're right!</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2480835,
      "author_name": "Raja Biswas",
      "author_url": "",
      "post_date": "2023-10-13T15:17:23.113000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> &amp; <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> on 4th position finish🎉 🎉 <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> special congratulations on becoming competition GM, very well deserved!! Thanks for the great writeup!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2480962,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-13T16:59:43.473000",
          "content": "<p><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> I am specially glad to be congratulated by you. Your post about fulltrain has made me more aware of fulltrain and this strategy. Thank you very much! And congrats on your 6th place at LLM competition! I think it must have been tough days for you. See you again at one of the competitions! Looking forward to it!</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2481268,
              "author_name": "Raja Biswas",
              "author_url": "",
              "post_date": "2023-10-14T01:37:19.697000",
              "content": "<p>Thanks 🙂 Glad that full-train strategy was helpful! Yes, it was exhausting to compete in 2 comps together, will try to avoid in future 🙈 All the best for the next one!</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2478978,
      "author_name": "Oluwatobi Betiku",
      "author_url": "",
      "post_date": "2023-10-12T10:09:24.063000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> and <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> this is an amazing feat, realizing there is a lot to learn and also motivated at the same time, Thanks a lot for sharing this solution.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2479065,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2023-10-12T11:37:28.553000",
          "content": "<p><a href=\"https://www.kaggle.com/oluwatobibetiku\" target=\"_blank\">@oluwatobibetiku</a> Thank you for comment!!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 32,
  "messages": [
    {
      "id": 2478528,
      "postDate": "2023-10-12T04:22:33.280Z",
      "content": "<p>Thank you very much for organizing such an interesting competition. I am greatly thankful to the hosts and the Kaggle staff.</p>\n<p>This competition turned out to be exactly what we had imagined and we are really happy to have won 4th place.</p>\n<p>And thank you so much <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> for teaming up with me and for the daily discussions and experiments. Congratulations on your 4th GOLD MEDAL.<br>\n(And I will become competition Grand Master!!!).<br>\nWe are a great team!</p>\n<h1>1. Summary</h1>\n<p>Our best private solution is 1 fulltrain x 7 models ensemble.</p>\n<p>This competition had only 4 prompts in the training data and with the information that 122 prompts were in the test data, one could imagine a big shake. Also, the public lb was very unstable. These things made us think that robustness is important, not only for cv.</p>\n<p>I used all 4kfold at first. Especially the models with prompt text, because I had to increase the maxlen to increase the score, I had to increase the inference time and could not ensemble other models (only 2-3 models could be included).</p>\n<p>Just then, I and kuro have team merged. And we mixed the fulltrain idea he had been working on. By replacing 4kfold with 1fulltrain, we arrived at the idea of compressing the information of 4kfold to 1fulltrain and ensemble more models.</p>\n<p>We imagined that by doing so, we would prevent the variation and reduction of scores on each prompt in the private test data and, in total, get better score.</p>\n<p>As a result, we believe we were able to prevent a shake down and finish in this position!</p>\n<table>\n<thead>\n<tr>\n<th>sub</th>\n<th>type</th>\n<th>models</th>\n<th>cv</th>\n<th>public lb</th>\n<th>private lb</th>\n<th>rank</th>\n<th>comment</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>sub1</td>\n<td>best lb</td>\n<td>1 fulltrain × 9 models</td>\n<td>0.4679</td>\n<td>0.41991</td>\n<td>0.45785</td>\n<td>11</td>\n<td>using gbdt and expand of inference-maxlen</td>\n</tr>\n<tr>\n<td>sub2</td>\n<td>best cv</td>\n<td>1 fulltrain × 7 models</td>\n<td>0.4639</td>\n<td>0.42979</td>\n<td>0.45515</td>\n<td>4</td>\n<td>just NLP result(expand of inference-maxlen)</td>\n</tr>\n<tr>\n<td>sub3</td>\n<td>insurance</td>\n<td>1 fulltrain × 8 models</td>\n<td>0.4693</td>\n<td>0.43855</td>\n<td>0.45597</td>\n<td>6</td>\n<td>just NLP result</td>\n</tr>\n</tbody>\n</table>\n<p>The best cv in sub2 is our best private, and that is the main explanation below.</p>\n<h1>2. training</h1>\n<p>Each of our models in sub2 is shown below.</p>\n<table>\n<thead>\n<tr>\n<th>modelno</th>\n<th>InfNo</th>\n<th>model</th>\n<th>training<br>maxlen</th>\n<th>inference <br>maxlen</th>\n<th>freezing</th>\n<th>layerwise</th>\n<th>input</th>\n<th>pooling</th>\n<th></th>\n<th></th>\n<th>2nd loss</th>\n<th>preprocess</th>\n<th>cv of 4kfold<br> earlystop</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>original prompt</td>\n<td>cls</td>\n<td>attention of <br>original prompt</td>\n<td>mean of <br>text part only</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>1</td>\n<td>91</td>\n<td>deberta-v3-large</td>\n<td>768</td>\n<td>1500</td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4818</td>\n</tr>\n<tr>\n<td>2</td>\n<td>22</td>\n<td>deberta-v3-large</td>\n<td>1050</td>\n<td>950</td>\n<td></td>\n<td>✓</td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4855</td>\n</tr>\n<tr>\n<td>3</td>\n<td>63</td>\n<td>deberta-v3-large</td>\n<td>850</td>\n<td>1500</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>0.4984</td>\n</tr>\n<tr>\n<td>4</td>\n<td>72</td>\n<td>deberta-v3-large</td>\n<td>868</td>\n<td>1024</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>Arcface</td>\n<td>✓</td>\n<td>0.4919</td>\n</tr>\n<tr>\n<td>5</td>\n<td>2,3</td>\n<td>deberta-v3-large</td>\n<td>868</td>\n<td>868</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4880</td>\n</tr>\n<tr>\n<td>6</td>\n<td>259</td>\n<td>deberta-v3-large-squad2</td>\n<td>768</td>\n<td>1500</td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4952</td>\n</tr>\n<tr>\n<td>7</td>\n<td>331</td>\n<td>deberta-v3-large-squad2</td>\n<td>1050</td>\n<td>950</td>\n<td></td>\n<td>✓</td>\n<td>✓</td>\n<td></td>\n<td>✓</td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.4993</td>\n</tr>\n</tbody>\n</table>\n<p>The details are described below.</p>\n<h2>2.1 model no1 : basemodel</h2>\n<p>This model is a typical base for our model.</p>\n<p>First, two inputs are prepared and put into the tokenizer as a pair (an example of dataset). The same result can be obtained by connecting them with [SEP] even if you do not put them as a pair.</p>\n<pre><code>self = self.\nself = self +  + self +  + self\n\ntokens = tokenizer(\n            self, \n            self,.\n            ...\n        )\n</code></pre>\n<p>Now all we have to do is put this in the model, increase the maxlen, and output the cls in the fc layer.</p>\n<h2>2.2 model no2 : original prompt</h2>\n<p>In this model, an original prompt was created and used for input.</p>\n<pre><code>self =  + self\nself = prompt_title +  + prompt_question +  + prompt_text\n\ntokens = tokenizer(\n            self, \n            self,\n            ・・・\n        )\n</code></pre>\n<p>Then, only the part of the original prompt (Evaluating the summarized…) is attentional pooled (an example of a model).</p>\n<pre><code>## init\n.pool = AttentionPooling(..hidden_size)\n...\n\n## forward\n = .model(ids, mask, token_type_ids)\n = [][:,:,:]\n = .pool(,mask[:,:])\n = .fc()\n</code></pre>\n<h2>2.3 model no4 : using 2nd loss and preprocess</h2>\n<p>As discussed in the discussion and notebook, the train data can be classified into 38 types. I thought I could make good use of this, so I included arcface as an auxiliary loss.<br>\nI tried to make good use of embeddings at the end, but it was not available. However, it did contribute to the diversity of the ensemble.</p>\n<p>Also, as a preprocss, I made sure to include a space after the period and comma; after doing EDA, I noticed that if a sentence comes after a period or comma without a space, it is divided differently in the tokenizer. However, I don't think this affected the score significantly.</p>\n<h2>2.4 Extending the inference max length</h2>\n<p>Scores have been greatly improved by increasing the inference length over that of training. An example is shown below. The following was trained with maxlen 850, but by setting maxlen to 1500 during inference, cv and public lb improved (maybe not so much contribution to private).</p>\n<table>\n<thead>\n<tr>\n<th>traing<br>maxlen</th>\n<th>inference<br>maxlen</th>\n<th>39c16e<br>fold0</th>\n<th>814d6b<br>fold1</th>\n<th>ebad26<br>fold2</th>\n<th>3b9047<br>fold3</th>\n<th>cv</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>850</td>\n<td>850</td>\n<td>0.4505</td>\n<td>0.5595</td>\n<td>0.5051</td>\n<td>0.5024</td>\n<td>0.4984</td>\n</tr>\n<tr>\n<td>850</td>\n<td>1024</td>\n<td>0.4524</td>\n<td>0.5590</td>\n<td>0.4836</td>\n<td>0.5018</td>\n<td>0.4927</td>\n</tr>\n<tr>\n<td>850</td>\n<td>1500</td>\n<td>0.4527</td>\n<td>0.5588</td>\n<td>0.4614</td>\n<td>0.5013</td>\n<td>0.4867</td>\n</tr>\n</tbody>\n</table>\n<p>However, some models may degrade. insurance submission(sub3) consisted only of models that did not expand this inference max length. As a result, cv and public were affected, but private had almost no effect.</p>\n<h1>3. inference (ensemble)</h1>\n<p>For best cv sub, we initially post-processed using GBDT (LGBM + Catboost + XGboost), but in the end, ensemble without GBDT resulted in a better CV, The submission that produced the best private did not use post-processing with GBDT. We also considered nealder-mead and optuna, but we did not use them because of the concern that they would definitely overfit. So we use the simple mean ensemble using a hill climb technique (we also added a full train with different seed to further increase robustness for the extra inference time).</p>\n<p>final cv(4kfold earlystopping result) : 0.46394 , public : 0.42979, private 0.45515 4th</p>\n<p>※ Futhermore, we took as insurance(sub3) that the inference max lengths were the same as that in training. This result was as follows.</p>\n<p>final cv 0.4693 , public : 0.43855, private 0.45597 6th.</p>\n<p>In this result, cv and lb were bad, but private was good, which shows how important the robustness of model was in this competition.</p>\n<h1>4. Not working for us</h1>\n<ul>\n<li>mlm</li>\n<li>awp</li>\n<li>svr</li>\n<li>other model</li>\n<li>regression from classification model</li>\n<li>other many many things…</li>\n</ul>\n<h1>5. Acknowledgements</h1>\n<p>We could not have achieved these results on our own. We were greatly influenced by those who we have collaborated with in the past, and we are grateful for their contributions. We would also like to express our sincere gratitude to those who have shared their knowledge and insights through previous competitions. Thank you very much.</p>\n<p>We would especially like to thank the following people for their help in this  competition! Thank you very much.</p>\n<ul>\n<li><p>Fulltrain : Team Hydrogen <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369578\" target=\"_blank\">ref1</a> <a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536\" target=\"_blank\">ref2</a>, Raja Biswas <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/424330#2354068\" target=\"_blank\">ref3</a></p></li>\n<li><p>Freezing : Takamichi Toda <a href=\"https://www.kaggle.com/takamichitoda\" target=\"_blank\">@takamichitoda</a>  <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/433754#2405543\" target=\"_blank\">ref4</a></p></li>\n<li><p>37-38 classification : MOTH <a href=\"https://www.kaggle.com/alejopaullier\" target=\"_blank\">@alejopaullier</a> <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/430705\" target=\"_blank\">ref5</a>, <br>\nAlexander Chervov <a href=\"https://www.kaggle.com/alexandervc\" target=\"_blank\">@alexandervc</a> <a href=\"https://www.kaggle.com/code/alexandervc/commonlit-levenshtein-distances?scriptVersionId=141542492&amp;cellId=23\" target=\"_blank\">ref6</a></p></li>\n<li><p>Postprocess : nogawanogawa <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> <a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">ref7</a></p></li>\n<li><p>Model selection(feedback3) : CHRIS DEOTTE <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369609\" target=\"_blank\">ref8</a>, TOM <a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369567\" target=\"_blank\">ref9</a></p></li>\n</ul>\n<h1>6. team member</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a></li>\n<li><a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> </li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fa311809edebd32836455da5b13e1dc1e%2Fkuromajin.jpg?generation=1697084418755389&amp;alt=media\" alt=\"\"></p>\n<h1>7. code</h1>\n<p>inference code : <a href=\"https://www.kaggle.com/code/chumajin/commonlit2-4th-place-inference\" target=\"_blank\">https://www.kaggle.com/code/chumajin/commonlit2-4th-place-inference</a> (This is the cleaned code. Same score as the submission.)</p>\n<p>training code(chumajin part) : <a href=\"https://github.com/chumajin/kaggle-commonlit\" target=\"_blank\">https://github.com/chumajin/kaggle-commonlit</a><br>\ntraining code(kuro_B part) : <a href=\"https://github.com/kurokuroB/kaggle-commonlit2-4th_place_solution-my_training_code\" target=\"_blank\">https://github.com/kurokuroB/kaggle-commonlit2-4th_place_solution-my_training_code</a></p>",
      "rawMarkdown": "Thank you very much for organizing such an interesting competition. I am greatly thankful to the hosts and the Kaggle staff.\n\nThis competition turned out to be exactly what we had imagined and we are really happy to have won 4th place.\n\nAnd thank you so much @kurokurob for teaming up with me and for the daily discussions and experiments. Congratulations on your 4th GOLD MEDAL.\n(And I will become competition Grand Master!!!).\nWe are a great team!\n\n\n# 1. Summary\n\nOur best private solution is 1 fulltrain x 7 models ensemble.\n\nThis competition had only 4 prompts in the training data and with the information that 122 prompts were in the test data, one could imagine a big shake. Also, the public lb was very unstable. These things made us think that robustness is important, not only for cv.\n\nI used all 4kfold at first. Especially the models with prompt text, because I had to increase the maxlen to increase the score, I had to increase the inference time and could not ensemble other models (only 2-3 models could be included).\n\nJust then, I and kuro have team merged. And we mixed the fulltrain idea he had been working on. By replacing 4kfold with 1fulltrain, we arrived at the idea of compressing the information of 4kfold to 1fulltrain and ensemble more models.\n\nWe imagined that by doing so, we would prevent the variation and reduction of scores on each prompt in the private test data and, in total, get better score.\n\nAs a result, we believe we were able to prevent a shake down and finish in this position!\n| sub  | type      | models                 | cv      | public lb | private lb | rank | comment                                     |\n|------|-----------|------------------------|---------|-----------|------------|------|---------------------------------------------|\n| sub1 | best lb   | 1 fulltrain × 9 models | 0.4679 | 0.41991     | 0.45785      | 11 | using gbdt and expand of inference-maxlen   |\n| sub2 | best cv   | 1 fulltrain × 7 models | 0.4639 | 0.42979     | 0.45515      | 4    | just NLP result(expand of inference-maxlen) |\n| sub3 | insurance | 1 fulltrain × 8 models | 0.4693  | 0.43855     | 0.45597      | 6  | just NLP result                             |\n\nThe best cv in sub2 is our best private, and that is the main explanation below.\n\n\n# 2. training\n\n\nEach of our models in sub2 is shown below.\n\n| modelno | InfNo | model                   | training<br/>maxlen | inference <br/>maxlen | freezing | layerwise | input           | pooling |                                   |                             | 2nd loss | preprocess | cv of 4kfold<br/> earlystop |\n|---------|-------|-------------------------|---------------------|-----------------------|----------|-----------|-----------------|---------|-----------------------------------|-----------------------------|----------|------------|-----------------------------|\n|         |       |                         |                     |                       |          |           | original prompt | cls     | attention of <br/>original prompt | mean of <br/>text part only |          |            |                             |\n| 1       | 91    | deberta-v3-large        | 768                 | 1500                  | ✓        |           |                 | ✓       |                                   |                             |          |            | 0.4818                      |\n| 2       | 22    | deberta-v3-large        | 1050                | 950                   |          | ✓         | ✓               |         | ✓                                 |                             |          |            | 0.4855                      |\n| 3       | 63    | deberta-v3-large        | 850                 | 1500                  |          | ✓         |                 |         |                                   | ✓                           |          |            | 0.4984                      |\n| 4       | 72    | deberta-v3-large        | 868                 | 1024                  |          | ✓         |                 | ✓       |                                   |                             | Arcface  | ✓          | 0.4919                      |\n| 5       | 2,3   | deberta-v3-large        | 868                 | 868                   |          | ✓         |                 | ✓       |                                   |                             |          |            | 0.4880                       |\n| 6       | 259   | deberta-v3-large-squad2 | 768                 | 1500                  | ✓        |           |                 | ✓       |                                   |                             |          |            | 0.4952                      |\n| 7       | 331   | deberta-v3-large-squad2 | 1050                | 950                   |          | ✓         | ✓               |         | ✓                                 |                             |          |            | 0.4993                      |\n\nThe details are described below.\n\n## 2.1 model no1 : basemodel\n\nThis model is a typical base for our model.\n\nFirst, two inputs are prepared and put into the tokenizer as a pair (an example of dataset). The same result can be obtained by connecting them with [SEP] even if you do not put them as a pair.\n~~~~\nself.text = self.df[\"text\"].\nself.prompt = self.df[\"prompt_title\"] + [SEP] + self.df[\"prompt_question\"] + [SEP] + self.df[\"prompt_text\"]\n\ntokens = tokenizer.encode_plus(\n            self.text, \n            self.prompt,.\n            ...\n        )\n~~~~\nNow all we have to do is put this in the model, increase the maxlen, and output the cls in the fc layer.\n\n## 2.2 model no2 : original prompt\n\nIn this model, an original prompt was created and used for input.\n\n~~~\nself.text = \"Evaluating the summarized text and calculating content and wording score : \" + self.df[\"text\"].values\nself.prompt = prompt_title + [SEP] + prompt_question + [SEP] + prompt_text\n\ntokens = tokenizer.encode_plus(\n            self.text, \n            self.prompt,\n            ・・・\n        )\n~~~\n\nThen, only the part of the original prompt (Evaluating the summarized...) is attentional pooled (an example of a model).\n\n~~~\n## init\nself.pool = AttentionPooling(self.config.hidden_size)\n...\n\n## forward\noutput = self.model(ids, mask, token_type_ids)\noutput = output[0][:,:12,:]\noutput = self.pool(output,mask[:,:12])\noutput = self.fc(output)\n~~~\n\n## 2.3 model no4 : using 2nd loss and preprocess\n\nAs discussed in the discussion and notebook, the train data can be classified into 38 types. I thought I could make good use of this, so I included arcface as an auxiliary loss.\nI tried to make good use of embeddings at the end, but it was not available. However, it did contribute to the diversity of the ensemble.\n\nAlso, as a preprocss, I made sure to include a space after the period and comma; after doing EDA, I noticed that if a sentence comes after a period or comma without a space, it is divided differently in the tokenizer. However, I don't think this affected the score significantly.\n\n## 2.4 Extending the inference max length\n\nScores have been greatly improved by increasing the inference length over that of training. An example is shown below. The following was trained with maxlen 850, but by setting maxlen to 1500 during inference, cv and public lb improved (maybe not so much contribution to private).\n\n\n| traing<br/>maxlen | inference<br/>maxlen | 39c16e<br/>fold0 | 814d6b<br/>fold1 | ebad26<br/>fold2 | 3b9047<br/>fold3 | cv      |\n|-------------------|----------------------|------------------|------------------|------------------|------------------|---------|\n| 850               | 850                  | 0.4505           | 0.5595           | 0.5051           | 0.5024           | 0.4984  |\n| 850               | 1024                 | 0.4524           | 0.5590           | 0.4836           | 0.5018           | 0.4927  |\n| 850               | 1500                 | 0.4527           | 0.5588           | 0.4614           | 0.5013           | 0.4867  |\n\n\nHowever, some models may degrade. insurance submission(sub3) consisted only of models that did not expand this inference max length. As a result, cv and public were affected, but private had almost no effect.\n\n# 3. inference (ensemble)\n\nFor best cv sub, we initially post-processed using GBDT (LGBM + Catboost + XGboost), but in the end, ensemble without GBDT resulted in a better CV, The submission that produced the best private did not use post-processing with GBDT. We also considered nealder-mead and optuna, but we did not use them because of the concern that they would definitely overfit. So we use the simple mean ensemble using a hill climb technique (we also added a full train with different seed to further increase robustness for the extra inference time).\n\nfinal cv(4kfold earlystopping result) : 0.46394 , public : 0.42979, private 0.45515 4th\n\n※ Futhermore, we took as insurance(sub3) that the inference max lengths were the same as that in training. This result was as follows.\n\nfinal cv 0.4693 , public : 0.43855, private 0.45597 6th.\n\nIn this result, cv and lb were bad, but private was good, which shows how important the robustness of model was in this competition.\n\n# 4. Not working for us\n\n- mlm\n- awp\n- svr\n- other model\n- regression from classification model\n- other many many things...\n\n# 5. Acknowledgements\n\nWe could not have achieved these results on our own. We were greatly influenced by those who we have collaborated with in the past, and we are grateful for their contributions. We would also like to express our sincere gratitude to those who have shared their knowledge and insights through previous competitions. Thank you very much.\n\nWe would especially like to thank the following people for their help in this  competition! Thank you very much.\n\n* Fulltrain : Team Hydrogen [ref1](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369578) [ref2](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536), Raja Biswas @conjuring92 [ref3](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/424330#2354068)\n\n* Freezing : Takamichi Toda @takamichitoda  [ref4](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/433754#2405543)\n\n* 37-38 classification : MOTH @alejopaullier [ref5](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/430705), \nAlexander Chervov @alexandervc [ref6](https://www.kaggle.com/code/alexandervc/commonlit-levenshtein-distances?scriptVersionId=141542492&cellId=23)\n\n* Postprocess : nogawanogawa @tsunotsuno [ref7](https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect)\n\n* Model selection(feedback3) : CHRIS DEOTTE @cdeotte [ref8](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369609), TOM @tikutiku [ref9](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369567)\n\n# 6. team member\n* @chumajin\n* @kurokurob \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fa311809edebd32836455da5b13e1dc1e%2Fkuromajin.jpg?generation=1697084418755389&alt=media)\n\n# 7. code\ninference code : https://www.kaggle.com/code/chumajin/commonlit2-4th-place-inference (This is the cleaned code. Same score as the submission.)\n\ntraining code(chumajin part) : https://github.com/chumajin/kaggle-commonlit\ntraining code(kuro_B part) : https://github.com/kurokuroB/kaggle-commonlit2-4th_place_solution-my_training_code\n",
      "votes": 80
    },
    {
      "id": 2479284,
      "postDate": "2023-10-12T14:24:34.847Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> for the win and <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> for becoming competition GM, impressive achievement!</p>\n<p>I found the <code>max_len</code> change for inference fascinating. Could you share details on your training configuration such as batch size, learning rate, epochs? </p>\n<p>To me the most difficult thing was getting a stable Train Loss vs. Valid Loss. My models would not only overfit quickly (usually after 2 epochs) but there would also be large variances in performances across different seeds.</p>\n<p>Thanks in advance</p>",
      "rawMarkdown": "Congrats @chumajin @kurokurob for the win and @chumajin for becoming competition GM, impressive achievement!\n\nI found the `max_len` change for inference fascinating. Could you share details on your training configuration such as batch size, learning rate, epochs? \n\nTo me the most difficult thing was getting a stable Train Loss vs. Valid Loss. My models would not only overfit quickly (usually after 2 epochs) but there would also be large variances in performances across different seeds.\n\nThanks in advance",
      "votes": 3,
      "replies": [
        {
          "id": 2479473,
          "postDate": "2023-10-12T16:28:49.030Z",
          "content": "<p>Thanks for your comment!<br>\nWe used different configuration each other.<br>\nMy main settting is batch size 8, learning rate 3e-5, cosine warmup 0.2, epoch 2. (model no 1, 6)</p>",
          "rawMarkdown": "Thanks for your comment!\nWe used different configuration each other.\nMy main settting is batch size 8, learning rate 3e-5, cosine warmup 0.2, epoch 2. (model no 1, 6)",
          "votes": 3
        }
      ]
    },
    {
      "id": 2478571,
      "postDate": "2023-10-12T04:58:49.003Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> and <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> (also for becoming competition GM!!). </p>",
      "rawMarkdown": "Congratulations @kurokurob and @chumajin (also for becoming competition GM!!). ",
      "votes": 3,
      "replies": [
        {
          "id": 2478660,
          "postDate": "2023-10-12T06:22:29.067Z",
          "content": "<p><a href=\"https://www.kaggle.com/harshit92\" target=\"_blank\">@harshit92</a> I'm most happy to receive congratulations from you. You are the key person in my kaggle life.<br>\nLet's do it together again someday.</p>",
          "rawMarkdown": "@harshit92 I'm most happy to receive congratulations from you. You are the key person in my kaggle life.\nLet's do it together again someday.",
          "votes": 2
        }
      ]
    },
    {
      "id": 2480347,
      "postDate": "2023-10-13T09:19:38.283Z",
      "content": "<p>Congratuations <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> for 4th place in this competition. </p>\n<p>I have a question about your approach. Please let me know.</p>\n<p>[Question]<br>\nWhat does fulltrain strategy mean? </p>\n<p>I assume that you use all train data for training and no validation, is it right?<br>\nIf it is right, how do you define training condition such as n-epoch, lr, and so on.</p>",
      "rawMarkdown": "Congratuations @chumajin for 4th place in this competition. \n\nI have a question about your approach. Please let me know.\n\n[Question]\nWhat does fulltrain strategy mean? \n\nI assume that you use all train data for training and no validation, is it right?\nIf it is right, how do you define training condition such as n-epoch, lr, and so on.",
      "votes": 1,
      "replies": [
        {
          "id": 2480418,
          "postDate": "2023-10-13T09:58:10.620Z",
          "content": "<p><a href=\"https://www.kaggle.com/hidebu\" target=\"_blank\">@hidebu</a> <br>\nThank you for question.<br>\nI show the process as follows:<br>\n1) We trained 4kfold. We can get 4 model weights and calculate cv.<br>\n2) If the cv result of 1) is good, we trained using all train data. In this train, we use the condition which is the same as 1) including the n-epoch, lr and so on.<br>\n3) for inference, we used the one model which was made in 2). We did not use the model weights which was made in 1).</p>",
          "rawMarkdown": "@hidebu \nThank you for question.\nI show the process as follows:\n1) We trained 4kfold. We can get 4 model weights and calculate cv.\n2) If the cv result of 1) is good, we trained using all train data. In this train, we use the condition which is the same as 1) including the n-epoch, lr and so on.\n3) for inference, we used the one model which was made in 2). We did not use the model weights which was made in 1).\n",
          "votes": 1,
          "replies": [
            {
              "id": 2480474,
              "postDate": "2023-10-13T10:33:26.860Z",
              "content": "<p>Thank you for sharing your approach. </p>\n<p>When you train 4 Kfold, I am guessing you have last epoch = best epoch ?   (And not save model when best validation loss is achieved in any epoch) </p>",
              "rawMarkdown": "Thank you for sharing your approach. \n\nWhen you train 4 Kfold, I am guessing you have last epoch = best epoch ?   (And not save model when best validation loss is achieved in any epoch) ",
              "votes": 1
            },
            {
              "id": 2480532,
              "postDate": "2023-10-13T11:32:24.220Z",
              "content": "<p><a href=\"https://www.kaggle.com/rashmibanthia\" target=\"_blank\">@rashmibanthia</a> Thank you for comment !! This is a good question. You are right. To be precise, we did several validation evaluations per epoch at 4kfold training, at which point we calculated the oof after the 4kfold training and stopped learning the fulltrain at the point where the oof was the highest. However, many of the oofs were stable at the very end of the last epoch, and we often let them run to the end.</p>",
              "rawMarkdown": "@rashmibanthia Thank you for comment !! This is a good question. You are right. To be precise, we did several validation evaluations per epoch at 4kfold training, at which point we calculated the oof after the 4kfold training and stopped learning the fulltrain at the point where the oof was the highest. However, many of the oofs were stable at the very end of the last epoch, and we often let them run to the end.",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2480325,
      "postDate": "2023-10-13T08:49:47.777Z",
      "content": "<p>Congratulations becoming competition GM🎉 🎉 </p>\n<p>This competition didn't go well for me at all, so I'm planning to review various things, including your approach, moving forward.</p>",
      "rawMarkdown": "Congratulations becoming competition GM🎉 🎉 \n\nThis competition didn't go well for me at all, so I'm planning to review various things, including your approach, moving forward.\n",
      "votes": 1,
      "replies": [
        {
          "id": 2480414,
          "postDate": "2023-10-13T09:53:53.387Z",
          "content": "<p><a href=\"https://www.kaggle.com/takamichitoda\" target=\"_blank\">@takamichitoda</a> <br>\nThank you very much! Your posts on discussion were very helpful. Again, I have looked at your blog many times! Thank you every time!</p>",
          "rawMarkdown": "@takamichitoda \nThank you very much! Your posts on discussion were very helpful. Again, I have looked at your blog many times! Thank you every time!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2479666,
      "postDate": "2023-10-12T19:11:51.940Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> for Win and becoming competition GM </p>",
      "rawMarkdown": "Congratulations @chumajin for Win and becoming competition GM ",
      "votes": 1,
      "replies": [
        {
          "id": 2479889,
          "postDate": "2023-10-13T01:20:01.473Z",
          "content": "<p><a href=\"https://www.kaggle.com/husseinah95\" target=\"_blank\">@husseinah95</a> Thank you very much!! </p>",
          "rawMarkdown": "@husseinah95 Thank you very much!! "
        }
      ]
    },
    {
      "id": 2478895,
      "postDate": "2023-10-12T09:01:13.453Z",
      "content": "<p><a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> Thanks for the detailed write-up. I think you did a good job on not over-fitting the CV.</p>\n<p>Also, congratulations on becoming a competition grandmaster. A difficult accomplishment and I'm happy that your hard work has paid off.</p>",
      "rawMarkdown": "@chumajin Thanks for the detailed write-up. I think you did a good job on not over-fitting the CV.\n\nAlso, congratulations on becoming a competition grandmaster. A difficult accomplishment and I'm happy that your hard work has paid off.",
      "votes": 1,
      "replies": [
        {
          "id": 2478918,
          "postDate": "2023-10-12T09:24:44.903Z",
          "content": "<p><a href=\"https://www.kaggle.com/ivanaerlic\" target=\"_blank\">@ivanaerlic</a> Thank you very much!! I was really afraid to just shake down! Competition GM has been a tough road, but it feels great to have accomplished it. See you again at one of the competitions!</p>",
          "rawMarkdown": "@ivanaerlic Thank you very much!! I was really afraid to just shake down! Competition GM has been a tough road, but it feels great to have accomplished it. See you again at one of the competitions!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2478890,
      "postDate": "2023-10-12T08:57:06.913Z",
      "content": "<p>Congratulations on winning the gold and becoming competition GM! 🎉</p>",
      "rawMarkdown": "Congratulations on winning the gold and becoming competition GM! 🎉",
      "votes": 1,
      "replies": [
        {
          "id": 2478892,
          "postDate": "2023-10-12T08:58:49.703Z",
          "content": "<p><a href=\"https://www.kaggle.com/muhammad4hmed\" target=\"_blank\">@muhammad4hmed</a> Thank you so much!!</p>",
          "rawMarkdown": "@muhammad4hmed Thank you so much!!"
        }
      ]
    },
    {
      "id": 2478575,
      "postDate": "2023-10-12T05:04:59.697Z",
      "content": "<p>First, thank you for sharing the solution. And congratulations on becoming the competition GrandMaster!!<br>\nAmong the teams we observed attempting to shake down, I believe your team was able to achieve stable results without overfitting to the CV by creating a robust model. Congratulations!</p>\n<p>The idea of utilizing a full train model is also being used by our team, and its usefulness in reducing inference time was confirmed here. While we had the idea of solving the sub task of the classification problem, I felt there might be differences here compared to the gold zone.<br>\nUltimately, I strongly felt that the most important aspect is not to overfit to the CV due to the limited amount of data.</p>\n<p>Once again, congratulations on becoming the competition GrandMaster!!</p>",
      "rawMarkdown": "First, thank you for sharing the solution. And congratulations on becoming the competition GrandMaster!!\nAmong the teams we observed attempting to shake down, I believe your team was able to achieve stable results without overfitting to the CV by creating a robust model. Congratulations!\n\nThe idea of utilizing a full train model is also being used by our team, and its usefulness in reducing inference time was confirmed here. While we had the idea of solving the sub task of the classification problem, I felt there might be differences here compared to the gold zone.\nUltimately, I strongly felt that the most important aspect is not to overfit to the CV due to the limited amount of data.\n\nOnce again, congratulations on becoming the competition GrandMaster!!",
      "votes": 1,
      "replies": [
        {
          "id": 2478667,
          "postDate": "2023-10-12T06:26:57.963Z",
          "content": "<p><a href=\"https://www.kaggle.com/takai380\" target=\"_blank\">@takai380</a> Thank you very much. This is my personal opinion, but maybe one of the reason why we did not shake down is because we did not use the model without prompt text.  In our case, the model without prompt text did raise the cv by gbdt, but it did not raise the cv like prompt id 3b9047 compared to that with prompt text. If there were more prompts like 3b9047 in the private test, we would have expected the score to go down.</p>",
          "rawMarkdown": "@takai380 Thank you very much. This is my personal opinion, but maybe one of the reason why we did not shake down is because we did not use the model without prompt text.  In our case, the model without prompt text did raise the cv by gbdt, but it did not raise the cv like prompt id 3b9047 compared to that with prompt text. If there were more prompts like 3b9047 in the private test, we would have expected the score to go down.",
          "votes": 3,
          "replies": [
            {
              "id": 2478684,
              "postDate": "2023-10-12T06:34:28.390Z",
              "content": "<p>You may have a point. Thanks for your response! Congratulations!</p>",
              "rawMarkdown": "You may have a point. Thanks for your response! Congratulations!",
              "votes": 1
            },
            {
              "id": 2480485,
              "postDate": "2023-10-13T10:40:52.500Z",
              "content": "<blockquote>\n  <p>This is my personal opinion, but maybe one of the reason why we did not shake down is because we did not use the model without prompt text.</p>\n</blockquote>\n<p>This 💯 - I noticed this in your solution and <a href=\"https://www.kaggle.com/ivanaerlic\" target=\"_blank\">@ivanaerlic</a> solution as well. <br>\nOur very last submission was with 3 models with prompt text but it was not the best cv (but best lb) and too late to refine it by adding more models with prompt_text. </p>",
              "rawMarkdown": ">This is my personal opinion, but maybe one of the reason why we did not shake down is because we did not use the model without prompt text.\n\nThis 💯 - I noticed this in your solution and @ivanaerlic solution as well. \nOur very last submission was with 3 models with prompt text but it was not the best cv (but best lb) and too late to refine it by adding more models with prompt_text. ",
              "votes": 2
            }
          ]
        }
      ]
    },
    {
      "id": 2478545,
      "postDate": "2023-10-12T04:43:01.750Z",
      "content": "<p>Thanks for sharing, you have done a nice job! May I ask how you choose best model with fulltrain idea?</p>",
      "rawMarkdown": "Thanks for sharing, you have done a nice job! May I ask how you choose best model with fulltrain idea?",
      "votes": 1,
      "replies": [
        {
          "id": 2478657,
          "postDate": "2023-10-12T06:21:22.437Z",
          "content": "<p><a href=\"https://www.kaggle.com/nlchen\" target=\"_blank\">@nlchen</a> Thank you for comment ! we used the cv when we created the same model with 4kfold, and hill climb method.</p>",
          "rawMarkdown": "@nlchen Thank you for comment ! we used the cv when we created the same model with 4kfold, and hill climb method.",
          "replies": [
            {
              "id": 2478817,
              "postDate": "2023-10-12T08:17:31.920Z",
              "content": "<p>Thanks a lot!</p>",
              "rawMarkdown": "Thanks a lot!",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2478535,
      "postDate": "2023-10-12T04:35:16.403Z",
      "content": "<p>Great solution, congrats! Since you team used fulltrain strategy, may I ask how do you do cv? Just random sample a few samples from train dataset? </p>",
      "rawMarkdown": "Great solution, congrats! Since you team used fulltrain strategy, may I ask how do you do cv? Just random sample a few samples from train dataset? ",
      "votes": 1,
      "replies": [
        {
          "id": 2478656,
          "postDate": "2023-10-12T06:20:19.983Z",
          "content": "<p><a href=\"https://www.kaggle.com/judith007\" target=\"_blank\">@judith007</a> Thank you for your comment! Instead, we used the cv when we created the same model with 4kfold! (4kfold model is not used for inference)</p>",
          "rawMarkdown": "@judith007 Thank you for your comment! Instead, we used the cv when we created the same model with 4kfold! (4kfold model is not used for inference)",
          "votes": 1,
          "replies": [
            {
              "id": 2478686,
              "postDate": "2023-10-12T06:35:44.187Z",
              "content": "<p>Thank you. Do you mean that you did cv using 4 kfold, but the models from 4 kfold cv were not used in inference stage, and you just retrained a model on full dataset and used this model in inference stage?</p>",
              "rawMarkdown": "Thank you. Do you mean that you did cv using 4 kfold, but the models from 4 kfold cv were not used in inference stage, and you just retrained a model on full dataset and used this model in inference stage?",
              "votes": 1
            },
            {
              "id": 2478704,
              "postDate": "2023-10-12T06:59:39.637Z",
              "content": "<p><a href=\"https://www.kaggle.com/judith007\" target=\"_blank\">@judith007</a> yes. you're right!</p>",
              "rawMarkdown": "@judith007 yes. you're right!",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2480835,
      "postDate": "2023-10-13T15:17:23.113Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> &amp; <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> on 4th position finish🎉 🎉 <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> special congratulations on becoming competition GM, very well deserved!! Thanks for the great writeup!</p>",
      "rawMarkdown": "Congratulations @chumajin & @kurokurob on 4th position finish🎉 🎉 @chumajin special congratulations on becoming competition GM, very well deserved!! Thanks for the great writeup!",
      "votes": 2,
      "replies": [
        {
          "id": 2480962,
          "postDate": "2023-10-13T16:59:43.473Z",
          "content": "<p><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> I am specially glad to be congratulated by you. Your post about fulltrain has made me more aware of fulltrain and this strategy. Thank you very much! And congrats on your 6th place at LLM competition! I think it must have been tough days for you. See you again at one of the competitions! Looking forward to it!</p>",
          "rawMarkdown": "@conjuring92 I am specially glad to be congratulated by you. Your post about fulltrain has made me more aware of fulltrain and this strategy. Thank you very much! And congrats on your 6th place at LLM competition! I think it must have been tough days for you. See you again at one of the competitions! Looking forward to it!",
          "votes": 1,
          "replies": [
            {
              "id": 2481268,
              "postDate": "2023-10-14T01:37:19.697Z",
              "content": "<p>Thanks 🙂 Glad that full-train strategy was helpful! Yes, it was exhausting to compete in 2 comps together, will try to avoid in future 🙈 All the best for the next one!</p>",
              "rawMarkdown": "Thanks 🙂 Glad that full-train strategy was helpful! Yes, it was exhausting to compete in 2 comps together, will try to avoid in future 🙈 All the best for the next one!",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2478978,
      "postDate": "2023-10-12T10:09:24.063Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurokurob\" target=\"_blank\">@kurokurob</a> and <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> this is an amazing feat, realizing there is a lot to learn and also motivated at the same time, Thanks a lot for sharing this solution.</p>",
      "rawMarkdown": "Congratulations @kurokurob and @chumajin this is an amazing feat, realizing there is a lot to learn and also motivated at the same time, Thanks a lot for sharing this solution.",
      "votes": 2,
      "replies": [
        {
          "id": 2479065,
          "postDate": "2023-10-12T11:37:28.553Z",
          "content": "<p><a href=\"https://www.kaggle.com/oluwatobibetiku\" target=\"_blank\">@oluwatobibetiku</a> Thank you for comment!!</p>",
          "rawMarkdown": "@oluwatobibetiku Thank you for comment!!",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:01:32Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10726220,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/kuromajin-4th-place-solution"
}