{
  "id": 369457,
  "title": "1st Place Solution",
  "content": "<p>Thanks a lot to the hosts and Kaggle for hosting this interesting competition, we had great fun working on this one. Also congratulations to all other competitors for the great solutions and results. Special thanks to my teammates <a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a> and <a href=\"https://www.kaggle.com/poteman\" target=\"_blank\">@poteman</a> for such a perfect teamwork. </p>\n<h2>Summary:</h2>\n<p>Our solution is based on training lots of different models with different pooling techniques and max lengths and finally ensembling. We also used 3 models to extract embeddings and add them to ensemble. </p>\n<h2>Cross validation</h2>\n<p>Throughout this competition we had near perfect correlation between ensemble CV and LB. Whenever we saw some improvement on CV, we saw it reflected in a similar manner on the LB with very small random range. For splitting the folds, we just used MultilabelStratifiedKFold startegy. </p>\n<h2>Modeling:</h2>\n<p>Our final solution is a combination of different modeling approaches. </p>\n<p>Train/PL data + Different Pooling techniques + model + max_len(768/1462 for deberta models, 512 for others) + freezing top n layers + re_init/no re_init  top layer + differential LR + AWP. </p>\n<h3>Poolings Used:</h3>\n<ol>\n<li>MeanPooling</li>\n<li>ConcatPooling</li>\n<li>WeightedLayerPooling</li>\n<li>GemPooling</li>\n<li>LSTMPooling</li>\n</ol>\n<h3>Models Used:</h3>\n<ol>\n<li>microsoft-deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>deberta-v2-xlarge</li>\n<li>roberta-large</li>\n<li>distilbert-base-uncased</li>\n</ol>\n<p>Model Used for Embeddings and SVR</p>\n<ol>\n<li>\"facebook/bart-large\"</li>\n<li>\"flax-sentence-embeddings/all_datasets_v3_roberta-large\"</li>\n<li>\"facebook/bart-large-mnli\"</li>\n</ol>\n<p>Public Models: </p>\n<h3>Weight Tuning:</h3>\n<p></p><ul><br>\n<li>We used optuna to tune the model weights, since we used different folds for training, we used final oofs to tune the weights. Also weights are tuned target wise.</li><br>\n<li>We only add a model to our ensemble if it improves both our Ensemble CV/LB.</li><br>\n<li>The best CV we get using this approach was 0.44073 (We didn't select this sub fearing overfitting but this has best private score.) <br><br>\nOur Second Best CV was 0.44096 (It has good CV/LB correlation).<br><p></p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n<th><br></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-base with PL (Roh)</td>\n<td>0.4464</td>\n<td>0.437219</td>\n<td>0.437982<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (Yev)</td>\n<td>0.4460</td>\n<td>0.436758</td>\n<td>0.434625<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (pub)</td>\n<td>0.4548</td>\n<td>0.439502</td>\n<td>0.437965<br></td>\n<td></td>\n</tr>\n<tr>\n<td>nischay (pub)</td>\n<td>4588</td>\n<td>0.442645</td>\n<td>0.439982<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v2-xlarge (pub)</td>\n<td>0.4675</td>\n<td>0.442497</td>\n<td>0.443604<br></td>\n<td></td>\n</tr>\n<tr>\n<td>roberta-large (Roh)</td>\n<td>0.4596</td>\n<td>0.443616</td>\n<td>0.444081<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (pub)</td>\n<td>0.4552</td>\n<td>0.440045</td>\n<td>0.437234<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (Roh)</td>\n<td>0.4556</td>\n<td>0.444469</td>\n<td>0.440331<br></td>\n<td></td>\n</tr>\n<tr>\n<td>distilbert-base-uncased (Roh)</td>\n<td>0.4794</td>\n<td>0.458219</td>\n<td>0.459658<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4440</td>\n<td>0.435268</td>\n<td>0.434334<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-base with AVG(trainPl + train) (Roh)</td>\n<td>0.4482</td>\n<td>0.43856</td>\n<td>0.441147<br></td>\n<td></td>\n</tr>\n<tr>\n<td>roberta-large with AVG(trainPl + train) finetune train (Roh)</td>\n<td>0.4649</td>\n<td>0.449538</td>\n<td>0.447643<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4538</td>\n<td>0.441582</td>\n<td>0.439262<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4617</td>\n<td>0.448342</td>\n<td>0.442106<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4447</td>\n<td>0.436268</td>\n<td>0.435978<br></td>\n<td></td>\n</tr>\n<tr>\n<td>fb_bart_large_svr  (Roh)</td>\n<td>0.4635</td>\n<td>0.455645</td>\n<td>0.453132<br></td>\n<td></td>\n</tr>\n<tr>\n<td>ad_v3_roberta_large_svr (Roh)</td>\n<td>0.5113</td>\n<td>0.501901</td>\n<td>0.500084<br></td>\n<td></td>\n</tr>\n<tr>\n<td>fb_bart_large_mnli_svr (Roh)</td>\n<td>0.4708</td>\n<td>0.456652</td>\n<td>0.459615<br></td>\n<td></td>\n</tr>\n<tr>\n<td>Final Ensemble CV</td>\n<td>0.44096</td>\n<td>0.433821</td>\n<td>0.433356</td></tr></tbody></table></li>\n\n\n\n</ul>\n\n\n\n\n\n\nWe also noticed that generating Soft/Pseudo labels on train data leads to huge overfitting, CV around 0.437.\n\n\n\n\n\n\n\n<h2>Pseudo labels:</h2>\n<ol>\n<li>FB1 data</li>\n<li>Train data</li>\n<li>Mean of Train Pl / Actual Labels  [This gives nice boost in both CV/LB]</li>\n</ol>\n<h2>What Worked:</h2>\n<ol>\n<li>Different Pooling techniques</li>\n<li>Different max_len</li>\n<li>freezing top n layers</li>\n<li>re_init top n layers.</li>\n<li>Training with Differential learning rate</li>\n<li>Pseudo labels</li>\n</ol>\n<h2>What didn't worked</h2>\n<ol>\n<li>Augmentations</li>\n<li>Post processing</li>\n</ol>\n<h2>Important Citations:</h2>\n<p><a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a><br>\n<a href=\"https://www.kaggle.com/code/nischaydnk/fb3-pytorch-lightning-training-baseline\" target=\"_blank\">https://www.kaggle.com/code/nischaydnk/fb3-pytorch-lightning-training-baseline</a><br>\n<a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a><br>\n<a href=\"https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated</a><br>\n<a href=\"https://github.com/ybabakhin/kaggle-feedback-effectiveness-1st-place-solution\" target=\"_blank\">https://github.com/ybabakhin/kaggle-feedback-effectiveness-1st-place-solution</a></p>\n<h2>Extra Links:</h2>\n<p>Link to our best submission  <a href=\"https://www.kaggle.com/code/rohitsingh9990/merged-submission-01/notebook?scriptVersionId=111953356\" target=\"_blank\">https://www.kaggle.com/code/rohitsingh9990/merged-submission-01/notebook?scriptVersionId=111953356</a></p>\n<h2>Thanks and Acknowledgements:</h2>\n<p>Special thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>, <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a>, <a href=\"https://www.kaggle.com/nischay\" target=\"_blank\">@nischay</a>, <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a>, <a href=\"https://www.kaggle.com/e0xextazy\" target=\"_blank\">@e0xextazy</a> and <a href=\"https://www.kaggle.com/kojimar\" target=\"_blank\">@kojimar</a> for sharing their notebooks/datasets/ideas publicly. </p>\n<h2>Team Members</h2>\n<p><a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a><br>\n<a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a><br>\n<a href=\"https://www.kaggle.com/poteman\" target=\"_blank\">@poteman</a></p>\n<p>Training code can be accessed from <a href=\"https://github.com/rohitsingh02/kaggle-feedback-english-language-learning-1st-place-solution\" target=\"_blank\">https://github.com/rohitsingh02/kaggle-feedback-english-language-learning-1st-place-solution</a></p>",
  "comments": [
    {
      "id": 2049875,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2022-11-30T11:57:13.023000",
      "content": "<p>Congrats on the the win!</p>\n<p>Can you expand a bit what you mean by \"Mean of Train Pl / Actual Labels\" ? Thanks.</p>",
      "votes": 5,
      "replies": [
        {
          "id": 2049888,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T12:03:45.223000",
          "content": "<p>We generated soft labels / pseudo labels for train data using our earlier best ensemble. Then i took average of newly generated train labels and actual train labels. That's what <code>Mean of Train Pl / Actual Labels</code> means.</p>",
          "votes": 7,
          "replies": []
        },
        {
          "id": 2049969,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2022-11-30T13:10:29.880000",
          "content": "<p>Thanks.  I never did that, will try it.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2052549,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-12-02T09:42:41.817000",
          "content": "<p>I still don't quite understand, can you explain with a simple example? thank you so much</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2050246,
      "author_name": "pineapple",
      "author_url": "",
      "post_date": "2022-11-30T15:54:59.387000",
      "content": "<p>Congrats on your first place and thank you for the great explanation! Also, how did you get your CV and LB to line up so well? Really impressive stuff</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 2050215,
      "author_name": "Muhammad Hilmi Asyrofi",
      "author_url": "",
      "post_date": "2022-11-30T15:34:00.177000",
      "content": "<p>Congrats for your team victory <a href=\"https://www.kaggle.com/poteman\" target=\"_blank\">@poteman</a> <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> <a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a> <br>\nCongrats Rohit and Yevhenii for becoming the Kaggle Master! well deserved! </p>\n<p>I wanna ask, what's your strategy to use AWP? </p>\n<ul>\n<li>how do you choose the start epoch for AWP?</li>\n<li>how  do you choose the learning rate for the AWP? must it be lower then the model learning rate?</li>\n</ul>\n<p>From my experiment, the model becomes harder to converge after applying AWP.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2050228,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T15:41:14.103000",
          "content": "<p><a href=\"https://www.kaggle.com/mhilmiasyrofi\" target=\"_blank\">@mhilmiasyrofi</a> Thanks, the idea to use AWP came from past NLP competitions, most of our models we trained for 4 epochs and start epoch of AWP was 2, and the learning rate we used is 2e-5 for all layers except head for which it was 5e-4.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2050243,
          "author_name": "Muhammad Hilmi Asyrofi",
          "author_url": "",
          "post_date": "2022-11-30T15:51:46.373000",
          "content": "<p>Thank you for sharing!  <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a><br>\ncan't wait to see your training notebook so that I can deep dive the things behind!<br>\nKeep the good work up 😀</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2049777,
      "author_name": "Nischay Dhankhar",
      "author_url": "",
      "post_date": "2022-11-30T10:32:20.153000",
      "content": "<p>Thank you for the mention. Congratulations <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> and team on the first place victory 💥💥💥 </p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 2049537,
      "author_name": "Dracarys",
      "author_url": "",
      "post_date": "2022-11-30T07:52:37.193000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a> for becoming Competitions Master.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2049570,
          "author_name": "Yevhenii Maslov",
          "author_url": "",
          "post_date": "2022-11-30T08:08:11.677000",
          "content": "<p>Thank you) I also want to congratulate you on becoming the competitions Master)</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2049805,
      "author_name": "Ioannis M",
      "author_url": "",
      "post_date": "2022-11-30T10:58:00.257000",
      "content": "<p>congrats <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> and team, well deserved victory 👍</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2049791,
      "author_name": "Dylan Liu",
      "author_url": "",
      "post_date": "2022-11-30T10:44:38.323000",
      "content": "<p>Congratulations!<br>\nI'm curious that you said \"used different folds for training\", dose that mean the numbers of folds are also different? If so, do models trained with more folds have better cv? It seems unfair for models with fewer folds.</p>",
      "votes": 4,
      "replies": [
        {
          "id": 2049866,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T11:46:58.273000",
          "content": "<p>In ensemble we used 18 models in total, out of which 15 were trained using 5 folds and remaining 3 models which we used for Embedding extraction and SVR are trained using 25 folds. </p>\n<p>We used more folds only train Embedding model and their CV was already weak comparable to others. But you are right ensembling using different folds will lead to leakage and we need to be very careful while doing that. We can't blindly rely on CV score, we need some validation from Lb score as well. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2049543,
      "author_name": "Psi",
      "author_url": "",
      "post_date": "2022-11-30T07:54:52.037000",
      "content": "<p>Congrats amazing job!</p>\n<p>I also had near perfect CB/LB correlation and my best CV was also best LB (unselected for same reasons as you). </p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2049595,
      "author_name": "Ravi Ramakrishnan",
      "author_url": "",
      "post_date": "2022-11-30T08:24:55.580000",
      "content": "<p>Amazing result <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> and team! Hearty congratulations and thanks for the detailed approach note!<br>\nAll the best and regards!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2081983,
      "author_name": "Zhaoji Xu",
      "author_url": "",
      "post_date": "2022-12-31T23:47:59.463000",
      "content": "<p>Such a great solution. Will try my best to learn from this.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2059028,
      "author_name": "fafinozzi",
      "author_url": "",
      "post_date": "2022-12-08T12:10:38.730000",
      "content": "<p>Well done and thanks for sharing all the material!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2053381,
      "author_name": "ZHENYI SHEN",
      "author_url": "",
      "post_date": "2022-12-03T07:53:55.883000",
      "content": "<p>Congrats!</p>\n<p>I find you mentioned using generated Soft/Pseudo labels on train data and found there is huge overfitting. I also tried that and got the same CV, 0.437, as you did.</p>\n<p>Do you know why the overfitting occurs? I am very confused on this point. As we predict the values on different folds, there should be no leakage.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2053104,
      "author_name": "Mohsen Mahmoud",
      "author_url": "",
      "post_date": "2022-12-02T20:38:19.727000",
      "content": "<p>Keep going</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2050770,
      "author_name": "JimmyLiao",
      "author_url": "",
      "post_date": "2022-12-01T01:41:11.850000",
      "content": "<p>Congratulations!  Looking forward to reading your code.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2050131,
      "author_name": "Feng Qilong",
      "author_url": "",
      "post_date": "2022-11-30T14:42:11.363000",
      "content": "<p>Congrats! Can you explain a bit on the rationale of using the \"Mean of Train Pl / Actual Labels\" method?<br>\nThx :)</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2050145,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T15:00:08.470000",
          "content": "<p>What we tried first is generating soft labels (pl) for train data, training on this was hugely overfitted as we were getting CV around 0.437, then i thought i should try average of train pl and train data in an effort to reduce overfitting and this experiment worked, as it gave some boost in both ensemble CV/LB. </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2050189,
          "author_name": "Feng Qilong",
          "author_url": "",
          "post_date": "2022-11-30T15:20:46.940000",
          "content": "<p>Thanks for the info! I was suspicious that training with pl is overfitting as I was getting really good cv but not improving lb, but I could not find any form of leakage so I decided to stick with it…Glad that you found some rly cool ways to address this problem. Congrats again!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2049891,
      "author_name": "IOC META",
      "author_url": "",
      "post_date": "2022-11-30T12:05:23.547000",
      "content": "<p>can i ask you how to use freezing top n layers</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2049896,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T12:10:19.853000",
          "content": "<p>Here is a code snippet. <br>\n<code>if 'deberta-v3-base' in cfg.architecture.model_name or 'mdeberta-v3-base' in cfg.architecture.model_name:\n            self.model.embeddings.requires_grad_(False)\n            self.model.encoder.layer[:9].requires_grad_(False)</code><br>\nFull training code will be released soon. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2049689,
      "author_name": "william.wu",
      "author_url": "",
      "post_date": "2022-11-30T09:23:02.997000",
      "content": "<blockquote>\n  <p>We used optuna to tune the model weights, since we used different folds for training, we used final oofs to tune the weights. Also weights are tuned target wise</p>\n</blockquote>\n<p><code>since we used different folds for training</code>, do you mean you were training on different splits? Will that cause data leakage? Actually, I raised this question in this <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363448\" target=\"_blank\">discussion</a>. They convinced me that it would cause data leakage. Could you explain why it won't cause data leakage?</p>\n<p>My previous understanding was, for every single sample, it's predicted by models trained on data without it, so it wouldn't cause data leakage.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2049695,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T09:26:40.920000",
          "content": "<p>Since we used 2-3 public models and embedding models [they were surely trained on different splits]. We were aware that this might cause leakage thats why we only added those models to ensemble that gave boost in both ensemble CV/LB. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2049703,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-11-30T09:32:15.820000",
          "content": "<p>Got it, so it might cause leakage, and you prevent this by checking the CV &amp; LB score</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049707,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T09:33:55.157000",
          "content": "<p>yes, and since we had 3 final subs to choose. we took this risk with one sub.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2050778,
      "author_name": "Santhoshkumar",
      "author_url": "",
      "post_date": "2022-12-01T01:51:17.503000",
      "content": "<p>Congratulations 👏, way2go, thanks for sharing👍🤗</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2050332,
      "author_name": "Vaidehi Savaliya",
      "author_url": "",
      "post_date": "2022-11-30T16:58:48.410000",
      "content": "<p>Congratulations and good job…</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2049919,
      "author_name": "Jackson You",
      "author_url": "",
      "post_date": "2022-11-30T12:25:13.133000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> . Could you explain or find the reason that consistently keeps a good CV/LB correlation? That's attractive stuff. </p>",
      "votes": 2,
      "replies": [
        {
          "id": 2050148,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T15:01:44.707000",
          "content": "<p>First we trained a base model and tested its Lb score, after that every model we train we check its ensemble CV/LB, if it improve both, we add that model to our ensemble else we move to next experiment.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2050804,
          "author_name": "Jackson You",
          "author_url": "",
          "post_date": "2022-12-01T02:24:01.013000",
          "content": "<p>So the first base model maybe important than others. I thought it would be best to choose a model with a CV score close to the LB score as the first model.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2049877,
      "author_name": "Mark Baushenko",
      "author_url": "",
      "post_date": "2022-11-30T11:59:15.730000",
      "content": "<p>Congratulations on your victory, it's very nice to see that you used my model in your final submission!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2049601,
      "author_name": "william.wu",
      "author_url": "",
      "post_date": "2022-11-30T08:30:31.007000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a>! For using the <code>MultilabelStratifiedKFold</code>, did you create dummies for <code>y</code> before feeding in the <code>Fold.split</code>? As the following shows:</p>\n<pre><code>df = train.copy()\ny = pd.get_dummies(data=df[CFG.target_cols], columns=CFG.target_cols)\n n, (train_index, val_index)  (Fold.split(X=train, y=y)):\n        train.loc[val_index, ] = (n)\n</code></pre>\n<p>My CV and LB weren't correlated. I think it's mainly due to the splitting.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2049604,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T08:32:08.407000",
          "content": "<p>No, it was too late to do this change as most of the models were already trained and we didn't wanted to retrain them.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049609,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-11-30T08:34:30.437000",
          "content": "<p>Got it, thanks</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049611,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-11-30T08:35:26.487000",
          "content": "<p>How many splits did you use?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049614,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T08:38:09.553000",
          "content": "<p>5 for training all models and 25 for training embeddings using SVR </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049621,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-11-30T08:41:56.780000",
          "content": "<p>Got it, thanks. What is the seed for data splitting? 42? </p>\n<p>Sorry, I asked too many questions😂 I did ~ 1000 offline experiments. Just wanna learning more from this competition. Will definitely deep dive into your training code. Thanks in advance!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049628,
          "author_name": "Psi",
          "author_url": "",
          "post_date": "2022-11-30T08:44:16.530000",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> did you do early stopping? it can be very dangerous<br>\nI also just split it without the fix, I think even random kfold is fine.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049630,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-11-30T08:44:54.887000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2049631,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T08:45:38.377000",
          "content": "<p>yes we used only one seed which was 42</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2049642,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-11-30T08:53:12.743000",
          "content": "<p><a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a> No, but I saved the best model. The best model is always at the final epoch, so I think it's not an issue.<br>\nI was struggling with the converge issue when using the learning rate scheduler. Sometimes, I increased the epochs and used the same learning rate, but it converged to a sub-optimal result.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3563032%2Fec24a30736ee58058133dff5bafedd6b%2FScreenshot%20at%20Nov%2030%2016-49-49.png?generation=1669798221412297&amp;alt=media\" alt=\"training log\"></p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2049564,
      "author_name": "AK",
      "author_url": "",
      "post_date": "2022-11-30T08:05:36.117000",
      "content": "<p>Nicely done and congrats <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a>  for becomng competition Master &lt;3</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2049579,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-11-30T08:13:38.083000",
          "content": "<p>Thanks and congratulations to you and <a href=\"https://www.kaggle.com/rashmibanthia\" target=\"_blank\">@rashmibanthia</a> for becoming competition Master as well.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2049730,
          "author_name": "RB",
          "author_url": "",
          "post_date": "2022-11-30T09:46:38.747000",
          "content": "<p>Thank you, Congratulations <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a>, well done and well deserved 🎉🎉  <br>\nThank you for sharing your solution. </p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2052385,
      "author_name": "Jaideep",
      "author_url": "",
      "post_date": "2022-12-02T06:38:45.893000",
      "content": "<p><a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> &amp;team many congrats ,shows the real hardwork from you solution and same reflected in leaderboard also <br>\nKeep going .. <br>\nHope i join kaggle back soon </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2650271,
      "author_name": "Julien Mouchnino",
      "author_url": "",
      "post_date": "2024-02-13T11:00:12.110000",
      "content": "<p>What is FB1 data?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2058300,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-12-07T19:28:15.963000",
      "content": "<p>nice work!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2050315,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-11-30T16:41:11.127000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 2052849,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2022-12-02T15:01:41.373000",
          "content": "<p><a href=\"https://www.kaggle.com/rohitsingh9990/weight-tuning-optuna\" target=\"_blank\">https://www.kaggle.com/rohitsingh9990/weight-tuning-optuna</a> here i shared code to optimize weights using optuna.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2127761,
      "author_name": "Maomao050315",
      "author_url": "",
      "post_date": "2023-02-03T06:16:50.460000",
      "content": "<p>Nice work and thanks for sharing.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2057925,
      "author_name": "Ahmed Nasser",
      "author_url": "",
      "post_date": "2022-12-07T13:30:58.670000",
      "content": "<p><strong><em>Thanks for sharing, Keep going</em></strong></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2056127,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-12-05T18:57:04.683000",
      "content": "",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2056044,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-12-05T17:14:15.313000",
      "content": "",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2456551,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-09-26T09:14:09.183000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 59,
  "messages": [
    {
      "id": 2049531,
      "postDate": "2022-11-30T07:50:38.453Z",
      "content": "<p>Thanks a lot to the hosts and Kaggle for hosting this interesting competition, we had great fun working on this one. Also congratulations to all other competitors for the great solutions and results. Special thanks to my teammates <a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a> and <a href=\"https://www.kaggle.com/poteman\" target=\"_blank\">@poteman</a> for such a perfect teamwork. </p>\n<h2>Summary:</h2>\n<p>Our solution is based on training lots of different models with different pooling techniques and max lengths and finally ensembling. We also used 3 models to extract embeddings and add them to ensemble. </p>\n<h2>Cross validation</h2>\n<p>Throughout this competition we had near perfect correlation between ensemble CV and LB. Whenever we saw some improvement on CV, we saw it reflected in a similar manner on the LB with very small random range. For splitting the folds, we just used MultilabelStratifiedKFold startegy. </p>\n<h2>Modeling:</h2>\n<p>Our final solution is a combination of different modeling approaches. </p>\n<p>Train/PL data + Different Pooling techniques + model + max_len(768/1462 for deberta models, 512 for others) + freezing top n layers + re_init/no re_init  top layer + differential LR + AWP. </p>\n<h3>Poolings Used:</h3>\n<ol>\n<li>MeanPooling</li>\n<li>ConcatPooling</li>\n<li>WeightedLayerPooling</li>\n<li>GemPooling</li>\n<li>LSTMPooling</li>\n</ol>\n<h3>Models Used:</h3>\n<ol>\n<li>microsoft-deberta-v3-base</li>\n<li>deberta-v3-large</li>\n<li>deberta-v2-xlarge</li>\n<li>roberta-large</li>\n<li>distilbert-base-uncased</li>\n</ol>\n<p>Model Used for Embeddings and SVR</p>\n<ol>\n<li>\"facebook/bart-large\"</li>\n<li>\"flax-sentence-embeddings/all_datasets_v3_roberta-large\"</li>\n<li>\"facebook/bart-large-mnli\"</li>\n</ol>\n<p>Public Models: </p>\n<h3>Weight Tuning:</h3>\n<p></p><ul><br>\n<li>We used optuna to tune the model weights, since we used different folds for training, we used final oofs to tune the weights. Also weights are tuned target wise.</li><br>\n<li>We only add a model to our ensemble if it improves both our Ensemble CV/LB.</li><br>\n<li>The best CV we get using this approach was 0.44073 (We didn't select this sub fearing overfitting but this has best private score.) <br><br>\nOur Second Best CV was 0.44096 (It has good CV/LB correlation).<br><p></p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n<th><br></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-base with PL (Roh)</td>\n<td>0.4464</td>\n<td>0.437219</td>\n<td>0.437982<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (Yev)</td>\n<td>0.4460</td>\n<td>0.436758</td>\n<td>0.434625<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (pub)</td>\n<td>0.4548</td>\n<td>0.439502</td>\n<td>0.437965<br></td>\n<td></td>\n</tr>\n<tr>\n<td>nischay (pub)</td>\n<td>4588</td>\n<td>0.442645</td>\n<td>0.439982<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v2-xlarge (pub)</td>\n<td>0.4675</td>\n<td>0.442497</td>\n<td>0.443604<br></td>\n<td></td>\n</tr>\n<tr>\n<td>roberta-large (Roh)</td>\n<td>0.4596</td>\n<td>0.443616</td>\n<td>0.444081<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (pub)</td>\n<td>0.4552</td>\n<td>0.440045</td>\n<td>0.437234<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (Roh)</td>\n<td>0.4556</td>\n<td>0.444469</td>\n<td>0.440331<br></td>\n<td></td>\n</tr>\n<tr>\n<td>distilbert-base-uncased (Roh)</td>\n<td>0.4794</td>\n<td>0.458219</td>\n<td>0.459658<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4440</td>\n<td>0.435268</td>\n<td>0.434334<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-base with AVG(trainPl + train) (Roh)</td>\n<td>0.4482</td>\n<td>0.43856</td>\n<td>0.441147<br></td>\n<td></td>\n</tr>\n<tr>\n<td>roberta-large with AVG(trainPl + train) finetune train (Roh)</td>\n<td>0.4649</td>\n<td>0.449538</td>\n<td>0.447643<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4538</td>\n<td>0.441582</td>\n<td>0.439262<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4617</td>\n<td>0.448342</td>\n<td>0.442106<br></td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-large (yev)</td>\n<td>0.4447</td>\n<td>0.436268</td>\n<td>0.435978<br></td>\n<td></td>\n</tr>\n<tr>\n<td>fb_bart_large_svr  (Roh)</td>\n<td>0.4635</td>\n<td>0.455645</td>\n<td>0.453132<br></td>\n<td></td>\n</tr>\n<tr>\n<td>ad_v3_roberta_large_svr (Roh)</td>\n<td>0.5113</td>\n<td>0.501901</td>\n<td>0.500084<br></td>\n<td></td>\n</tr>\n<tr>\n<td>fb_bart_large_mnli_svr (Roh)</td>\n<td>0.4708</td>\n<td>0.456652</td>\n<td>0.459615<br></td>\n<td></td>\n</tr>\n<tr>\n<td>Final Ensemble CV</td>\n<td>0.44096</td>\n<td>0.433821</td>\n<td>0.433356</td></tr></tbody></table></li>\n\n\n\n</ul>\n\n\n\n\n\n\nWe also noticed that generating Soft/Pseudo labels on train data leads to huge overfitting, CV around 0.437.\n\n\n\n\n\n\n\n<h2>Pseudo labels:</h2>\n<ol>\n<li>FB1 data</li>\n<li>Train data</li>\n<li>Mean of Train Pl / Actual Labels  [This gives nice boost in both CV/LB]</li>\n</ol>\n<h2>What Worked:</h2>\n<ol>\n<li>Different Pooling techniques</li>\n<li>Different max_len</li>\n<li>freezing top n layers</li>\n<li>re_init top n layers.</li>\n<li>Training with Differential learning rate</li>\n<li>Pseudo labels</li>\n</ol>\n<h2>What didn't worked</h2>\n<ol>\n<li>Augmentations</li>\n<li>Post processing</li>\n</ol>\n<h2>Important Citations:</h2>\n<p><a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a><br>\n<a href=\"https://www.kaggle.com/code/nischaydnk/fb3-pytorch-lightning-training-baseline\" target=\"_blank\">https://www.kaggle.com/code/nischaydnk/fb3-pytorch-lightning-training-baseline</a><br>\n<a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a><br>\n<a href=\"https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated</a><br>\n<a href=\"https://github.com/ybabakhin/kaggle-feedback-effectiveness-1st-place-solution\" target=\"_blank\">https://github.com/ybabakhin/kaggle-feedback-effectiveness-1st-place-solution</a></p>\n<h2>Extra Links:</h2>\n<p>Link to our best submission  <a href=\"https://www.kaggle.com/code/rohitsingh9990/merged-submission-01/notebook?scriptVersionId=111953356\" target=\"_blank\">https://www.kaggle.com/code/rohitsingh9990/merged-submission-01/notebook?scriptVersionId=111953356</a></p>\n<h2>Thanks and Acknowledgements:</h2>\n<p>Special thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>, <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a>, <a href=\"https://www.kaggle.com/nischay\" target=\"_blank\">@nischay</a>, <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a>, <a href=\"https://www.kaggle.com/e0xextazy\" target=\"_blank\">@e0xextazy</a> and <a href=\"https://www.kaggle.com/kojimar\" target=\"_blank\">@kojimar</a> for sharing their notebooks/datasets/ideas publicly. </p>\n<h2>Team Members</h2>\n<p><a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a><br>\n<a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a><br>\n<a href=\"https://www.kaggle.com/poteman\" target=\"_blank\">@poteman</a></p>\n<p>Training code can be accessed from <a href=\"https://github.com/rohitsingh02/kaggle-feedback-english-language-learning-1st-place-solution\" target=\"_blank\">https://github.com/rohitsingh02/kaggle-feedback-english-language-learning-1st-place-solution</a></p>",
      "rawMarkdown": "Thanks a lot to the hosts and Kaggle for hosting this interesting competition, we had great fun working on this one. Also congratulations to all other competitors for the great solutions and results. Special thanks to my teammates @evgeniimaslov2 and @poteman for such a perfect teamwork. \n\n## Summary: \nOur solution is based on training lots of different models with different pooling techniques and max lengths and finally ensembling. We also used 3 models to extract embeddings and add them to ensemble. \n\n## Cross validation\n\nThroughout this competition we had near perfect correlation between ensemble CV and LB. Whenever we saw some improvement on CV, we saw it reflected in a similar manner on the LB with very small random range. For splitting the folds, we just used MultilabelStratifiedKFold startegy. \n\n## Modeling: \nOur final solution is a combination of different modeling approaches. \n\nTrain/PL data + Different Pooling techniques + model + max_len(768/1462 for deberta models, 512 for others) + freezing top n layers + re_init/no re_init  top layer + differential LR + AWP. \n\n### Poolings Used: \n1. MeanPooling\n2. ConcatPooling\n3. WeightedLayerPooling\n4. GemPooling\n5. LSTMPooling\n\n### Models Used: \n1. microsoft-deberta-v3-base\n2. deberta-v3-large\n3. deberta-v2-xlarge\n4. roberta-large\n5. distilbert-base-uncased\n\nModel Used for Embeddings and SVR\n1. \"facebook/bart-large\"\n2. \"flax-sentence-embeddings/all_datasets_v3_roberta-large\"\n3. \"facebook/bart-large-mnli\"\n\nPublic Models: \n\n\n### Weight Tuning: \n\n* We used optuna to tune the model weights, since we used different folds for training, we used final oofs to tune the weights. Also weights are tuned target wise.\n* We only add a model to our ensemble if it improves both our Ensemble CV/LB.\n* The best CV we get using this approach was 0.44073 (We didn't select this sub fearing overfitting but this has best private score.) \nOur Second Best CV was 0.44096 (It has good CV/LB correlation).\nModel | CV | Public LB | Private LB |\n| --- | --- |--- |--- \n| deberta-v3-base with PL (Roh) | 0.4464 | 0.437219 | 0.437982\n| deberta-v3-large (Yev) | 0.4460 | 0.436758 | 0.434625\n| deberta-v3-large (pub) |0.4548 | 0.439502 | 0.437965\n| nischay (pub) |4588| 0.442645 | 0.439982\n| deberta-v2-xlarge (pub) | 0.4675 | 0.442497 | 0.443604\n| roberta-large (Roh) |0.4596 | 0.443616 | 0.444081\n| deberta-v3-large (pub) | 0.4552 | 0.440045 | 0.437234\n| deberta-v3-large (Roh) | 0.4556 | 0.444469 | 0.440331\n| distilbert-base-uncased (Roh) | 0.4794| 0.458219 | 0.459658\n| deberta-v3-large (yev) | 0.4440 |  0.435268 | 0.434334\n| deberta-v3-base with AVG(trainPl + train) (Roh) | 0.4482 | 0.43856 | 0.441147\n| roberta-large with AVG(trainPl + train) finetune train (Roh) | 0.4649 | 0.449538 | 0.447643\n| deberta-v3-large (yev) | 0.4538 | 0.441582 | 0.439262\n| deberta-v3-large (yev) | 0.4617 | 0.448342 | 0.442106\n| deberta-v3-large (yev) | 0.4447 | 0.436268 | 0.435978\n| fb_bart_large_svr  (Roh) | 0.4635 | 0.455645 | 0.453132\n| ad_v3_roberta_large_svr (Roh) | 0.5113 | 0.501901 | 0.500084\n| fb_bart_large_mnli_svr (Roh) | 0.4708 | 0.456652 | 0.459615\n|Final Ensemble CV | 0.44096 | 0.433821 | 0.433356\n\nWe also noticed that generating Soft/Pseudo labels on train data leads to huge overfitting, CV around 0.437. \n\n## Pseudo labels: \n1. FB1 data\n2. Train data\n3. Mean of Train Pl / Actual Labels  [This gives nice boost in both CV/LB]\n\n\n## What Worked: \n1. Different Pooling techniques\n2. Different max_len\n3. freezing top n layers\n4. re_init top n layers.\n5. Training with Differential learning rate\n6. Pseudo labels\n\n## What didn't worked\n1. Augmentations\n2. Post processing\n\n## Important Citations:\nhttps://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\nhttps://www.kaggle.com/code/nischaydnk/fb3-pytorch-lightning-training-baseline\nhttps://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\nhttps://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\nhttps://github.com/ybabakhin/kaggle-feedback-effectiveness-1st-place-solution\n\n## Extra Links: \nLink to our best submission  https://www.kaggle.com/code/rohitsingh9990/merged-submission-01/notebook?scriptVersionId=111953356\n\n## Thanks and Acknowledgements:\nSpecial thanks to @cdeotte, @yasufuminakama, @nischay, @nischaydnk, @e0xextazy and @kojimar for sharing their notebooks/datasets/ideas publicly. \n\n## Team Members\n@rohitsingh9990\n@evgeniimaslov2\n@poteman\n\nTraining code can be accessed from https://github.com/rohitsingh02/kaggle-feedback-english-language-learning-1st-place-solution",
      "votes": 129
    },
    {
      "id": 2049875,
      "postDate": "2022-11-30T11:57:13.023Z",
      "content": "<p>Congrats on the the win!</p>\n<p>Can you expand a bit what you mean by \"Mean of Train Pl / Actual Labels\" ? Thanks.</p>",
      "rawMarkdown": "Congrats on the the win!\n\nCan you expand a bit what you mean by \"Mean of Train Pl / Actual Labels\" ? Thanks.",
      "votes": 5,
      "replies": [
        {
          "id": 2049888,
          "postDate": "2022-11-30T12:03:45.223Z",
          "content": "<p>We generated soft labels / pseudo labels for train data using our earlier best ensemble. Then i took average of newly generated train labels and actual train labels. That's what <code>Mean of Train Pl / Actual Labels</code> means.</p>",
          "rawMarkdown": "We generated soft labels / pseudo labels for train data using our earlier best ensemble. Then i took average of newly generated train labels and actual train labels. That's what `Mean of Train Pl / Actual Labels` means.",
          "votes": 7
        },
        {
          "id": 2049969,
          "postDate": "2022-11-30T13:10:29.880Z",
          "content": "<p>Thanks.  I never did that, will try it.</p>",
          "rawMarkdown": "Thanks.  I never did that, will try it.",
          "votes": 1
        },
        {
          "id": 2052549,
          "postDate": "2022-12-02T09:42:41.817Z",
          "content": "<p>I still don't quite understand, can you explain with a simple example? thank you so much</p>",
          "rawMarkdown": "I still don't quite understand, can you explain with a simple example? thank you so much",
          "votes": 3
        }
      ]
    },
    {
      "id": 2050246,
      "postDate": "2022-11-30T15:54:59.387Z",
      "content": "<p>Congrats on your first place and thank you for the great explanation! Also, how did you get your CV and LB to line up so well? Really impressive stuff</p>",
      "rawMarkdown": "Congrats on your first place and thank you for the great explanation! Also, how did you get your CV and LB to line up so well? Really impressive stuff",
      "votes": 3
    },
    {
      "id": 2050215,
      "postDate": "2022-11-30T15:34:00.177Z",
      "content": "<p>Congrats for your team victory <a href=\"https://www.kaggle.com/poteman\" target=\"_blank\">@poteman</a> <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> <a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a> <br>\nCongrats Rohit and Yevhenii for becoming the Kaggle Master! well deserved! </p>\n<p>I wanna ask, what's your strategy to use AWP? </p>\n<ul>\n<li>how do you choose the start epoch for AWP?</li>\n<li>how  do you choose the learning rate for the AWP? must it be lower then the model learning rate?</li>\n</ul>\n<p>From my experiment, the model becomes harder to converge after applying AWP.</p>",
      "rawMarkdown": "Congrats for your team victory @poteman @rohitsingh9990 @evgeniimaslov2 \nCongrats Rohit and Yevhenii for becoming the Kaggle Master! well deserved! \n\nI wanna ask, what's your strategy to use AWP? \n- how do you choose the start epoch for AWP?\n- how  do you choose the learning rate for the AWP? must it be lower then the model learning rate?\n\nFrom my experiment, the model becomes harder to converge after applying AWP.",
      "votes": 3,
      "replies": [
        {
          "id": 2050228,
          "postDate": "2022-11-30T15:41:14.103Z",
          "content": "<p><a href=\"https://www.kaggle.com/mhilmiasyrofi\" target=\"_blank\">@mhilmiasyrofi</a> Thanks, the idea to use AWP came from past NLP competitions, most of our models we trained for 4 epochs and start epoch of AWP was 2, and the learning rate we used is 2e-5 for all layers except head for which it was 5e-4.</p>",
          "rawMarkdown": "@mhilmiasyrofi Thanks, the idea to use AWP came from past NLP competitions, most of our models we trained for 4 epochs and start epoch of AWP was 2, and the learning rate we used is 2e-5 for all layers except head for which it was 5e-4.",
          "votes": 2
        },
        {
          "id": 2050243,
          "postDate": "2022-11-30T15:51:46.373Z",
          "content": "<p>Thank you for sharing!  <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a><br>\ncan't wait to see your training notebook so that I can deep dive the things behind!<br>\nKeep the good work up 😀</p>",
          "rawMarkdown": "Thank you for sharing!  @rohitsingh9990\ncan't wait to see your training notebook so that I can deep dive the things behind!\nKeep the good work up 😀",
          "votes": 1
        }
      ]
    },
    {
      "id": 2049777,
      "postDate": "2022-11-30T10:32:20.153Z",
      "content": "<p>Thank you for the mention. Congratulations <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> and team on the first place victory 💥💥💥 </p>",
      "rawMarkdown": "Thank you for the mention. Congratulations @rohitsingh9990 and team on the first place victory 💥💥💥 ",
      "votes": 3
    },
    {
      "id": 2049537,
      "postDate": "2022-11-30T07:52:37.193Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/evgeniimaslov2\" target=\"_blank\">@evgeniimaslov2</a> for becoming Competitions Master.</p>",
      "rawMarkdown": "Congratulations @evgeniimaslov2 for becoming Competitions Master.",
      "votes": 3,
      "replies": [
        {
          "id": 2049570,
          "postDate": "2022-11-30T08:08:11.677Z",
          "content": "<p>Thank you) I also want to congratulate you on becoming the competitions Master)</p>",
          "rawMarkdown": "Thank you) I also want to congratulate you on becoming the competitions Master)",
          "votes": 2
        }
      ]
    },
    {
      "id": 2049805,
      "postDate": "2022-11-30T10:58:00.257Z",
      "content": "<p>congrats <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> and team, well deserved victory 👍</p>",
      "rawMarkdown": "congrats @rohitsingh9990 and team, well deserved victory 👍",
      "votes": 4
    },
    {
      "id": 2049791,
      "postDate": "2022-11-30T10:44:38.323Z",
      "content": "<p>Congratulations!<br>\nI'm curious that you said \"used different folds for training\", dose that mean the numbers of folds are also different? If so, do models trained with more folds have better cv? It seems unfair for models with fewer folds.</p>",
      "rawMarkdown": "Congratulations!\nI'm curious that you said \"used different folds for training\", dose that mean the numbers of folds are also different? If so, do models trained with more folds have better cv? It seems unfair for models with fewer folds.",
      "votes": 4,
      "replies": [
        {
          "id": 2049866,
          "postDate": "2022-11-30T11:46:58.273Z",
          "content": "<p>In ensemble we used 18 models in total, out of which 15 were trained using 5 folds and remaining 3 models which we used for Embedding extraction and SVR are trained using 25 folds. </p>\n<p>We used more folds only train Embedding model and their CV was already weak comparable to others. But you are right ensembling using different folds will lead to leakage and we need to be very careful while doing that. We can't blindly rely on CV score, we need some validation from Lb score as well. </p>",
          "rawMarkdown": "In ensemble we used 18 models in total, out of which 15 were trained using 5 folds and remaining 3 models which we used for Embedding extraction and SVR are trained using 25 folds. \n\nWe used more folds only train Embedding model and their CV was already weak comparable to others. But you are right ensembling using different folds will lead to leakage and we need to be very careful while doing that. We can't blindly rely on CV score, we need some validation from Lb score as well. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 2049543,
      "postDate": "2022-11-30T07:54:52.037Z",
      "content": "<p>Congrats amazing job!</p>\n<p>I also had near perfect CB/LB correlation and my best CV was also best LB (unselected for same reasons as you). </p>",
      "rawMarkdown": "Congrats amazing job!\n\nI also had near perfect CB/LB correlation and my best CV was also best LB (unselected for same reasons as you). ",
      "votes": 4
    },
    {
      "id": 2049595,
      "postDate": "2022-11-30T08:24:55.580Z",
      "content": "<p>Amazing result <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> and team! Hearty congratulations and thanks for the detailed approach note!<br>\nAll the best and regards!</p>",
      "rawMarkdown": "Amazing result @rohitsingh9990 and team! Hearty congratulations and thanks for the detailed approach note!\nAll the best and regards!",
      "votes": 2
    },
    {
      "id": 2081983,
      "postDate": "2022-12-31T23:47:59.463Z",
      "content": "<p>Such a great solution. Will try my best to learn from this.</p>",
      "rawMarkdown": "Such a great solution. Will try my best to learn from this.",
      "votes": 1
    },
    {
      "id": 2059028,
      "postDate": "2022-12-08T12:10:38.730Z",
      "content": "<p>Well done and thanks for sharing all the material!</p>",
      "rawMarkdown": "Well done and thanks for sharing all the material!",
      "votes": 1
    },
    {
      "id": 2053381,
      "postDate": "2022-12-03T07:53:55.883Z",
      "content": "<p>Congrats!</p>\n<p>I find you mentioned using generated Soft/Pseudo labels on train data and found there is huge overfitting. I also tried that and got the same CV, 0.437, as you did.</p>\n<p>Do you know why the overfitting occurs? I am very confused on this point. As we predict the values on different folds, there should be no leakage.</p>",
      "rawMarkdown": "Congrats!\n\nI find you mentioned using generated Soft/Pseudo labels on train data and found there is huge overfitting. I also tried that and got the same CV, 0.437, as you did.\n\nDo you know why the overfitting occurs? I am very confused on this point. As we predict the values on different folds, there should be no leakage.",
      "votes": 1
    },
    {
      "id": 2053104,
      "postDate": "2022-12-02T20:38:19.727Z",
      "content": "<p>Keep going</p>",
      "rawMarkdown": "Keep going",
      "votes": 1
    },
    {
      "id": 2050770,
      "postDate": "2022-12-01T01:41:11.850Z",
      "content": "<p>Congratulations!  Looking forward to reading your code.</p>",
      "rawMarkdown": "Congratulations!  Looking forward to reading your code.",
      "votes": 1
    },
    {
      "id": 2050131,
      "postDate": "2022-11-30T14:42:11.363Z",
      "content": "<p>Congrats! Can you explain a bit on the rationale of using the \"Mean of Train Pl / Actual Labels\" method?<br>\nThx :)</p>",
      "rawMarkdown": "Congrats! Can you explain a bit on the rationale of using the \"Mean of Train Pl / Actual Labels\" method?\nThx :)",
      "votes": 1,
      "replies": [
        {
          "id": 2050145,
          "postDate": "2022-11-30T15:00:08.470Z",
          "content": "<p>What we tried first is generating soft labels (pl) for train data, training on this was hugely overfitted as we were getting CV around 0.437, then i thought i should try average of train pl and train data in an effort to reduce overfitting and this experiment worked, as it gave some boost in both ensemble CV/LB. </p>",
          "rawMarkdown": "What we tried first is generating soft labels (pl) for train data, training on this was hugely overfitted as we were getting CV around 0.437, then i thought i should try average of train pl and train data in an effort to reduce overfitting and this experiment worked, as it gave some boost in both ensemble CV/LB. ",
          "votes": 1
        },
        {
          "id": 2050189,
          "postDate": "2022-11-30T15:20:46.940Z",
          "content": "<p>Thanks for the info! I was suspicious that training with pl is overfitting as I was getting really good cv but not improving lb, but I could not find any form of leakage so I decided to stick with it…Glad that you found some rly cool ways to address this problem. Congrats again!</p>",
          "rawMarkdown": "Thanks for the info! I was suspicious that training with pl is overfitting as I was getting really good cv but not improving lb, but I could not find any form of leakage so I decided to stick with it...Glad that you found some rly cool ways to address this problem. Congrats again!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2049891,
      "postDate": "2022-11-30T12:05:23.547Z",
      "content": "<p>can i ask you how to use freezing top n layers</p>",
      "rawMarkdown": "can i ask you how to use freezing top n layers",
      "votes": 1,
      "replies": [
        {
          "id": 2049896,
          "postDate": "2022-11-30T12:10:19.853Z",
          "content": "<p>Here is a code snippet. <br>\n<code>if 'deberta-v3-base' in cfg.architecture.model_name or 'mdeberta-v3-base' in cfg.architecture.model_name:\n            self.model.embeddings.requires_grad_(False)\n            self.model.encoder.layer[:9].requires_grad_(False)</code><br>\nFull training code will be released soon. </p>",
          "rawMarkdown": "Here is a code snippet. \n`        if 'deberta-v3-base' in cfg.architecture.model_name or 'mdeberta-v3-base' in cfg.architecture.model_name:\n            self.model.embeddings.requires_grad_(False)\n            self.model.encoder.layer[:9].requires_grad_(False) `\nFull training code will be released soon. "
        }
      ]
    },
    {
      "id": 2049689,
      "postDate": "2022-11-30T09:23:02.997Z",
      "content": "<blockquote>\n  <p>We used optuna to tune the model weights, since we used different folds for training, we used final oofs to tune the weights. Also weights are tuned target wise</p>\n</blockquote>\n<p><code>since we used different folds for training</code>, do you mean you were training on different splits? Will that cause data leakage? Actually, I raised this question in this <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363448\" target=\"_blank\">discussion</a>. They convinced me that it would cause data leakage. Could you explain why it won't cause data leakage?</p>\n<p>My previous understanding was, for every single sample, it's predicted by models trained on data without it, so it wouldn't cause data leakage.</p>",
      "rawMarkdown": "> We used optuna to tune the model weights, since we used different folds for training, we used final oofs to tune the weights. Also weights are tuned target wise\n\n`since we used different folds for training`, do you mean you were training on different splits? Will that cause data leakage? Actually, I raised this question in this [discussion](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363448). They convinced me that it would cause data leakage. Could you explain why it won't cause data leakage?\n\nMy previous understanding was, for every single sample, it's predicted by models trained on data without it, so it wouldn't cause data leakage.\n\n\n\n",
      "votes": 1,
      "replies": [
        {
          "id": 2049695,
          "postDate": "2022-11-30T09:26:40.920Z",
          "content": "<p>Since we used 2-3 public models and embedding models [they were surely trained on different splits]. We were aware that this might cause leakage thats why we only added those models to ensemble that gave boost in both ensemble CV/LB. </p>",
          "rawMarkdown": "Since we used 2-3 public models and embedding models [they were surely trained on different splits]. We were aware that this might cause leakage thats why we only added those models to ensemble that gave boost in both ensemble CV/LB. \n"
        },
        {
          "id": 2049703,
          "postDate": "2022-11-30T09:32:15.820Z",
          "content": "<p>Got it, so it might cause leakage, and you prevent this by checking the CV &amp; LB score</p>",
          "rawMarkdown": "Got it, so it might cause leakage, and you prevent this by checking the CV & LB score",
          "votes": 1
        },
        {
          "id": 2049707,
          "postDate": "2022-11-30T09:33:55.157Z",
          "content": "<p>yes, and since we had 3 final subs to choose. we took this risk with one sub.</p>",
          "rawMarkdown": "yes, and since we had 3 final subs to choose. we took this risk with one sub.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2050778,
      "postDate": "2022-12-01T01:51:17.503Z",
      "content": "<p>Congratulations 👏, way2go, thanks for sharing👍🤗</p>",
      "rawMarkdown": "Congratulations 👏, way2go, thanks for sharing👍🤗",
      "votes": 2
    },
    {
      "id": 2050332,
      "postDate": "2022-11-30T16:58:48.410Z",
      "content": "<p>Congratulations and good job…</p>",
      "rawMarkdown": "Congratulations and good job...",
      "votes": 2
    },
    {
      "id": 2049919,
      "postDate": "2022-11-30T12:25:13.133Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> . Could you explain or find the reason that consistently keeps a good CV/LB correlation? That's attractive stuff. </p>",
      "rawMarkdown": "Congrats @rohitsingh9990 . Could you explain or find the reason that consistently keeps a good CV/LB correlation? That's attractive stuff. ",
      "votes": 2,
      "replies": [
        {
          "id": 2050148,
          "postDate": "2022-11-30T15:01:44.707Z",
          "content": "<p>First we trained a base model and tested its Lb score, after that every model we train we check its ensemble CV/LB, if it improve both, we add that model to our ensemble else we move to next experiment.</p>",
          "rawMarkdown": "First we trained a base model and tested its Lb score, after that every model we train we check its ensemble CV/LB, if it improve both, we add that model to our ensemble else we move to next experiment."
        },
        {
          "id": 2050804,
          "postDate": "2022-12-01T02:24:01.013Z",
          "content": "<p>So the first base model maybe important than others. I thought it would be best to choose a model with a CV score close to the LB score as the first model.</p>",
          "rawMarkdown": "So the first base model maybe important than others. I thought it would be best to choose a model with a CV score close to the LB score as the first model."
        }
      ]
    },
    {
      "id": 2049877,
      "postDate": "2022-11-30T11:59:15.730Z",
      "content": "<p>Congratulations on your victory, it's very nice to see that you used my model in your final submission!</p>",
      "rawMarkdown": "Congratulations on your victory, it's very nice to see that you used my model in your final submission!",
      "votes": 2
    },
    {
      "id": 2049601,
      "postDate": "2022-11-30T08:30:31.007Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a>! For using the <code>MultilabelStratifiedKFold</code>, did you create dummies for <code>y</code> before feeding in the <code>Fold.split</code>? As the following shows:</p>\n<pre><code>df = train.copy()\ny = pd.get_dummies(data=df[CFG.target_cols], columns=CFG.target_cols)\n n, (train_index, val_index)  (Fold.split(X=train, y=y)):\n        train.loc[val_index, ] = (n)\n</code></pre>\n<p>My CV and LB weren't correlated. I think it's mainly due to the splitting.</p>",
      "rawMarkdown": "Congratulations @rohitsingh9990! For using the `MultilabelStratifiedKFold`, did you create dummies for `y` before feeding in the `Fold.split`? As the following shows:\n\n```Python\ndf = train.copy()\ny = pd.get_dummies(data=df[CFG.target_cols], columns=CFG.target_cols)\nfor n, (train_index, val_index) in enumerate(Fold.split(X=train, y=y)):\n        train.loc[val_index, 'fold'] = int(n)\n```\n\nMy CV and LB weren't correlated. I think it's mainly due to the splitting.",
      "votes": 2,
      "replies": [
        {
          "id": 2049604,
          "postDate": "2022-11-30T08:32:08.407Z",
          "content": "<p>No, it was too late to do this change as most of the models were already trained and we didn't wanted to retrain them.</p>",
          "rawMarkdown": "No, it was too late to do this change as most of the models were already trained and we didn't wanted to retrain them.\n",
          "votes": 1
        },
        {
          "id": 2049609,
          "postDate": "2022-11-30T08:34:30.437Z",
          "content": "<p>Got it, thanks</p>",
          "rawMarkdown": "Got it, thanks",
          "votes": 1
        },
        {
          "id": 2049611,
          "postDate": "2022-11-30T08:35:26.487Z",
          "content": "<p>How many splits did you use?</p>",
          "rawMarkdown": "How many splits did you use?",
          "votes": 1
        },
        {
          "id": 2049614,
          "postDate": "2022-11-30T08:38:09.553Z",
          "content": "<p>5 for training all models and 25 for training embeddings using SVR </p>",
          "rawMarkdown": "5 for training all models and 25 for training embeddings using SVR ",
          "votes": 1
        },
        {
          "id": 2049621,
          "postDate": "2022-11-30T08:41:56.780Z",
          "content": "<p>Got it, thanks. What is the seed for data splitting? 42? </p>\n<p>Sorry, I asked too many questions😂 I did ~ 1000 offline experiments. Just wanna learning more from this competition. Will definitely deep dive into your training code. Thanks in advance!</p>",
          "rawMarkdown": "Got it, thanks. What is the seed for data splitting? 42? \n\nSorry, I asked too many questions😂 I did ~ 1000 offline experiments. Just wanna learning more from this competition. Will definitely deep dive into your training code. Thanks in advance!",
          "votes": 1
        },
        {
          "id": 2049628,
          "postDate": "2022-11-30T08:44:16.530Z",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> did you do early stopping? it can be very dangerous<br>\nI also just split it without the fix, I think even random kfold is fine.</p>",
          "rawMarkdown": "@wuwenmin did you do early stopping? it can be very dangerous\nI also just split it without the fix, I think even random kfold is fine.",
          "votes": 1
        },
        {
          "id": 2049630,
          "postDate": "2022-11-30T08:44:54.887Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 2049631,
          "postDate": "2022-11-30T08:45:38.377Z",
          "content": "<p>yes we used only one seed which was 42</p>",
          "rawMarkdown": "yes we used only one seed which was 42\n",
          "votes": 1
        },
        {
          "id": 2049642,
          "postDate": "2022-11-30T08:53:12.743Z",
          "content": "<p><a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a> No, but I saved the best model. The best model is always at the final epoch, so I think it's not an issue.<br>\nI was struggling with the converge issue when using the learning rate scheduler. Sometimes, I increased the epochs and used the same learning rate, but it converged to a sub-optimal result.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3563032%2Fec24a30736ee58058133dff5bafedd6b%2FScreenshot%20at%20Nov%2030%2016-49-49.png?generation=1669798221412297&amp;alt=media\" alt=\"training log\"></p>",
          "rawMarkdown": "@philippsinger No, but I saved the best model. The best model is always at the final epoch, so I think it's not an issue.\nI was struggling with the converge issue when using the learning rate scheduler. Sometimes, I increased the epochs and used the same learning rate, but it converged to a sub-optimal result.\n![training log](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3563032%2Fec24a30736ee58058133dff5bafedd6b%2FScreenshot%20at%20Nov%2030%2016-49-49.png?generation=1669798221412297&alt=media)",
          "votes": 2
        }
      ]
    },
    {
      "id": 2049564,
      "postDate": "2022-11-30T08:05:36.117Z",
      "content": "<p>Nicely done and congrats <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a>  for becomng competition Master &lt;3</p>",
      "rawMarkdown": "Nicely done and congrats @rohitsingh9990  for becomng competition Master <3",
      "votes": 2,
      "replies": [
        {
          "id": 2049579,
          "postDate": "2022-11-30T08:13:38.083Z",
          "content": "<p>Thanks and congratulations to you and <a href=\"https://www.kaggle.com/rashmibanthia\" target=\"_blank\">@rashmibanthia</a> for becoming competition Master as well.</p>",
          "rawMarkdown": "Thanks and congratulations to you and @rashmibanthia for becoming competition Master as well.",
          "votes": 2
        },
        {
          "id": 2049730,
          "postDate": "2022-11-30T09:46:38.747Z",
          "content": "<p>Thank you, Congratulations <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a>, well done and well deserved 🎉🎉  <br>\nThank you for sharing your solution. </p>",
          "rawMarkdown": "Thank you, Congratulations @rohitsingh9990, well done and well deserved 🎉🎉  \nThank you for sharing your solution. ",
          "votes": 2
        }
      ]
    },
    {
      "id": 2052385,
      "postDate": "2022-12-02T06:38:45.893Z",
      "content": "<p><a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> &amp;team many congrats ,shows the real hardwork from you solution and same reflected in leaderboard also <br>\nKeep going .. <br>\nHope i join kaggle back soon </p>",
      "rawMarkdown": "@rohitsingh9990 &team many congrats ,shows the real hardwork from you solution and same reflected in leaderboard also \nKeep going .. \nHope i join kaggle back soon ",
      "votes": 1
    },
    {
      "id": 2650271,
      "postDate": "2024-02-13T11:00:12.110Z",
      "content": "<p>What is FB1 data?</p>",
      "rawMarkdown": "What is FB1 data?"
    },
    {
      "id": 2058300,
      "postDate": "2022-12-07T19:28:15.963Z",
      "content": "<p>nice work!!</p>",
      "rawMarkdown": "nice work!!",
      "votes": 1,
      "isDeleted": true
    },
    {
      "id": 2050315,
      "postDate": "2022-11-30T16:41:11.127Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true,
      "replies": [
        {
          "id": 2052849,
          "postDate": "2022-12-02T15:01:41.373Z",
          "content": "<p><a href=\"https://www.kaggle.com/rohitsingh9990/weight-tuning-optuna\" target=\"_blank\">https://www.kaggle.com/rohitsingh9990/weight-tuning-optuna</a> here i shared code to optimize weights using optuna.</p>",
          "rawMarkdown": "https://www.kaggle.com/rohitsingh9990/weight-tuning-optuna here i shared code to optimize weights using optuna.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2127761,
      "postDate": "2023-02-03T06:16:50.460Z",
      "content": "<p>Nice work and thanks for sharing.</p>",
      "rawMarkdown": "Nice work and thanks for sharing.",
      "votes": 1
    },
    {
      "id": 2057925,
      "postDate": "2022-12-07T13:30:58.670Z",
      "content": "<p><strong><em>Thanks for sharing, Keep going</em></strong></p>",
      "rawMarkdown": "***Thanks for sharing, Keep going***",
      "votes": 1
    },
    {
      "id": 2056127,
      "postDate": "2022-12-05T18:57:04.683Z",
      "content": "<p>thanks for sharing the great idea</p>",
      "rawMarkdown": "thanks for sharing the great idea",
      "votes": 1
    },
    {
      "id": 2056044,
      "postDate": "2022-12-05T17:14:15.313Z",
      "content": "<p>thanks for sharing the great info </p>",
      "rawMarkdown": "thanks for sharing the great info ",
      "votes": 1
    },
    {
      "id": 2456551,
      "postDate": "2023-09-26T09:14:09.183Z",
      "content": "<p>Thanks for sharing, it helped a lot.</p>",
      "rawMarkdown": "Thanks for sharing, it helped a lot."
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:58:30Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9103062,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/autox-rohit-yevhenii-1st-place-solution"
}