{
  "id": 347359,
  "title": "2nd place solution (updated with code/notebooks)",
  "content": "<p>First, thanks to all the organizers and kaggle staff and congrats to all the winners and thanks to my amazing teammates <a href=\"https://www.kaggle.com/kneroma\" target=\"_blank\">@kneroma</a> and <a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> !  With this gold medal, <a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> and I finally become competition GMs. We have released our code/notebooks:</p>\n<p>code: <a href=\"https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution\" target=\"_blank\">https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution</a></p>\n<p>2nd logloss notebook: <a href=\"https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824\" target=\"_blank\">https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824</a></p>\n<p>3rd place efficiency notebook: <a href=\"https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605\" target=\"_blank\">https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605</a></p>\n<p>Below is a summary of our solution. Our best private is 0.553 and our best selected private is 0.554. Please feel free to ask if you have any questions.</p>\n<h1>Transformer modeling</h1>\n<p>Each of our team members has their own training pipeline for transformer models. On a high level, our transformer models look at the entirety of each essay and output predictions of effectiveness for each discourse either via pooling of discourse tokens or a classification token added to the front of the each discourse. Importantly, directly inputting essays results in a situation where the model is not 100% sure about where it needs to make predictions, so to circumvent this, we use either a prompt (i.e. concat <code>f'({discourse_type} start)'</code> and <code>f'({discourse_type} end)'</code> to the beginning and end of each discourse to signal where predictions need to be made) or simply concat special tokens added to the tokenizer instead (i.e. <code>f'&lt;{discourse_type}&gt;'</code> and <code>f'&lt;{discourse_type}\\&gt;'</code>). You can find an example below with a highlighted segment.</p>\n<p><img src=\"https://raw.githubusercontent.com/Shujun-He/Shujun-He-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution/main/graphics/example_segments.png\" alt=\"example\"></p>\n<h2>Encoders</h2>\n<p>Deberta worked the best since (IMO) it supports unlimited input length and uses disentangled attention with relative positional embedding; in fact, our ensemble consists entirely of deberta variants. For me, it was also helpful to add a GRU/LSTM after pooling on the pooled discourse representations. Tom used my SlidingWindowTransformerModel (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235</a>) from the last Feedback competiion, which stablized training for him.</p>\n<h2>Pretraining</h2>\n<p>Kkiller used pretrained weights from his solution in the last competiion (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478</a>) and Tom and I found pretrained tascj0 models to be good starting points. We used some of the weights that tascj0 released after the last Feedback competition and Tom also pretrained some new ones on his own. Please checkout out tascj0's solution post if you'd like to learn more (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424)\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424)</a>.<br>\n In addition, Tom used MLM for some of his models.  Further, some of our models simply used huggingface weights.</p>\n<h2>Max sequence length</h2>\n<p>I used a max sequence length of 1280 in both training and inference, since I found that 99.9% of discourses fall within that range, whereas my teammates used up to around 1800 during inference and as low as 648 in training.</p>\n<h1>Pseudo labeling</h1>\n<p>Pseudo labeling is an integral part of all our solution. We use essays from the training set of last Feedback competition that are also not present in the training set of this competition. Our procedure is as follows:</p>\n<ol>\n<li>Train model with gt labels</li>\n<li>Make predictions for old data (around 11000 essays) with each fold model</li>\n<li>Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only</li>\n<li>Repeat from step 2</li>\n</ol>\n<p>For my pipeline, I saw improvement until 5 rounds of the above procedure. For Tom, it was only helpful for one round and kkiller did not have enough time to try multi-round pl.</p>\n<h1>Stacking</h1>\n<p>Stacking provides significantly improvement in both cv/lb (around 0.004). Our stacking framework is primarily inspired by my team's solution (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235</a>) in the previous feedback competition. In addition to predicted probabilities outputted by the transformer models, we also utilized the token probabilities for each discourse, which we call prob_sequences. Compared to the previous Feedback competition, stacking is much faster since we don't have to deal with a huge amount of candidate sequences. Our features are as follows:</p>\n<pre><code> #make features\ndef get_xgb_features(train_df,prob_sequences):\n    features2calculate=[f\"instability_{i}\" for i in range(4)]+\\\n    [f\"begin_{i}\" for i in range(3)]+\\\n    [f\"end_{i}\" for i in range(3)]#+\\\n    #[\"entropy\"]\n\n    calculated_features=[]\n    for i,prob_seq in tqdm(enumerate(prob_sequences)):\n\n        tmp=[]\n        #quants = np.linspace(0,1,n_quan)\n        prob_seq=np.array(prob_seq)\n        instability = []\n        #all_quants=[]\n        tmp.append(np.diff(prob_seq[:,:],0).mean(0))\n        tmp.append([(np.diff(prob_seq[:,[1,2]].sum(1))**2).mean()])\n\n        tmp.append(prob_seq[:5,:].mean(0))\n        tmp.append(prob_seq[-5:,:].mean(0))\n\n        calculated_features.append(np.concatenate(tmp))\n\n\n    train_df[features2calculate]=calculated_features\n    train_df['len']=[len(s) for s in prob_sequences]\n\n    calculated_features=np.array(calculated_features)\n    calculated_features.shape\n\n    p_features=[]\n    n_features=[]\n    neighbor_features=['Ineffective','Adequate','Effective','discourse_type']\n    neighbor_features_values=train_df[neighbor_features].values\n    for i in tqdm(range(len(train_df))):\n        if i&gt;1 and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i-1]:\n            p_features.append(neighbor_features_values[i-1])\n        else:\n            p_features.append(neighbor_features_values[i])\n\n        if i&lt;(len(train_df)-1) and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i+1]:\n            n_features.append(neighbor_features_values[i+1])\n        else:\n            n_features.append(neighbor_features_values[i])\n\n    train_df[[f+\"_previous\" for f in neighbor_features]]=p_features\n    train_df[[f+\"_next\" for f in neighbor_features]]=n_features\n\n    train_df['mean_Ineffective']=train_df.groupby(\"essay_id\")[\"Ineffective\"].transform(\"mean\")\n    train_df['mean_Adequate']=train_df.groupby(\"essay_id\")[\"Adequate\"].transform(\"mean\")\n    train_df['mean_Effective']=train_df.groupby(\"essay_id\")[\"Effective\"].transform(\"mean\")\n\n    train_df['std_Ineffective']=train_df.groupby(\"essay_id\")[\"Ineffective\"].transform(\"std\")\n    train_df['std_Adequate']=train_df.groupby(\"essay_id\")[\"Adequate\"].transform(\"std\")\n    train_df['std_Effective']=train_df.groupby(\"essay_id\")[\"Effective\"].transform(\"std\")\n\n    train_df['discourse_count']=train_df.groupby(\"essay_id\")['discourse_type'].transform(\"count\")\n\n    cnts=train_df.groupby('essay_id')['discourse_type'].apply(lambda x: x.value_counts())\n\n    #new_df=[]\n    discourse_types=['Claim','Evidence','Concluding Statement','Lead','Position','Counterclaim','Rebuttal']\n    value_count_hash={}\n    for t in discourse_types:\n        value_count_hash[t]={}\n    for key in cnts.keys():\n        value_count_hash[key[1]][key[0]]=cnts[key]\n\n    discourse_cnts=[]    \n    for essay_id in train_df['essay_id'].unique():\n        row=[essay_id]\n        for d in discourse_types:\n            try:\n                row.append(value_count_hash[d][essay_id])\n            except:\n                row.append(0)\n        discourse_cnts.append(row)\n\n    discourse_cnts=pd.DataFrame(discourse_cnts,columns=['essay_id']+[f'{d}_count' for d in discourse_types])    \n    #discourse_cnts\n\n    train_df=train_df.merge(discourse_cnts,how='left',on='essay_id')\n    train_df\n\n    #train_df\n\n    return train_df\n</code></pre>\n<pre><code>\n</code></pre>\n<p>Since stacking is fast, it works best when we use each fold predictions with xgb separately and then avg. For instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way I have 6x6=36 preds to avg for each single model.</p>\n<h1>Best single models</h1>\n<p>All our best single models were deberta-large/deberta-v3-large variants. For Tom and Kkiller, their best single models came from 1st round PL, whereas for me it came from 4th round PL.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Shujun</th>\n<th>Tom</th>\n<th>Kkiller</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Public LB</td>\n<td>0.560</td>\n<td>0.566</td>\n<td>0.562</td>\n</tr>\n<tr>\n<td>Private LB</td>\n<td>0.558</td>\n<td>0.571</td>\n<td>0.562</td>\n</tr>\n<tr>\n<td>Local CV</td>\n<td>0.571</td>\n<td>N/A</td>\n<td>0.572</td>\n</tr>\n</tbody>\n</table>\n<h1>Some more tips/tricks</h1>\n<p>For Tom AWP was useful, and he reported around 0.003 cv improvement with AWP (eps=1e-3 and lr=1e-2 for large models, 1e-4 for xlarge models).</p>\n<p>It was also important to split the data with <code>StratifiedGroupKFold</code> instead of <code>GroupKFold</code>. For me I started out with <code>GroupKFold</code> but found better correlation between cv/lb after switching to <code>StratifiedGroupKFold</code>.</p>\n<p>For ensembling models with the same cv split, we used GP_minimize to find optimal weights and otherwise weights were determined arbitrarily.</p>\n<p>Gradient accumulation was useful since we had to deal with very long sequences.</p>",
  "comments": [
    {
      "id": 1911190,
      "author_name": "kkiller",
      "author_url": "",
      "post_date": "2022-08-24T00:58:19.240000",
      "content": "<p>Sample pre-processed essay from my pipeline</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1498852%2F68da2ecd156457e223a19e9b452a7d9b%2Fessay_with_special_tokens.png?generation=1661302642758307&amp;alt=media\" alt=\"essay_with_special_tokens\"></p>",
      "votes": 6,
      "replies": [
        {
          "id": 1911599,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-08-24T07:36:44.333000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1936456,
      "author_name": "MPWARE",
      "author_url": "",
      "post_date": "2022-09-12T18:57:28.800000",
      "content": "<p><a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> Congratulations and welcome to the club! I'm really happy for you.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1936613,
          "author_name": "Tom",
          "author_url": "",
          "post_date": "2022-09-12T23:01:25.263000",
          "content": "<p>Thank you <a href=\"https://www.kaggle.com/mpware\" target=\"_blank\">@mpware</a> ! It took me a while to get the last gold, but finally I did it :) </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1936944,
          "author_name": "MPWARE",
          "author_url": "",
          "post_date": "2022-09-13T06:43:24.340000",
          "content": "<p>Yes, I will be happy to team up again with you in a further competition.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1936992,
          "author_name": "Tom",
          "author_url": "",
          "post_date": "2022-09-13T07:15:12.077000",
          "content": "<p>Sure, I look forward to that!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911987,
      "author_name": "Muhammad Anas Mahmood",
      "author_url": "",
      "post_date": "2022-08-24T12:37:57.770000",
      "content": "<p>Congratulations Great Work <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1911669,
      "author_name": "Patrick Chan",
      "author_url": "",
      "post_date": "2022-08-24T08:24:24.220000",
      "content": "<p>Great share and congratulations on becoming KGM! Cheers</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1911455,
      "author_name": "Seonil Kwon",
      "author_url": "",
      "post_date": "2022-08-24T05:39:26.720000",
      "content": "<p>Supa thx for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1911406,
      "author_name": "yao",
      "author_url": "",
      "post_date": "2022-08-24T04:44:40.490000",
      "content": "<p>Congratulations on in the money and becoming KGM! I have a question:</p>\n<blockquote>\n  <p>3 epochs on pl labels only first and then 3 more epochs on gt labels only</p>\n</blockquote>\n<p>Did training on gt labels use the same config as training with pl labels? Thanks!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1911744,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T09:23:46.190000",
          "content": "<p><a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> Thanks and yes I use the same config for both first round training and multi-round pl training. The only difference is the number of epochs. Since I use a linear scheduler, I change the number of total steps to total pl label steps + total gt label steps. My teammates also do something similar: the training schedule changes a bit but nothing else</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911759,
          "author_name": "yao",
          "author_url": "",
          "post_date": "2022-08-24T09:42:53.837000",
          "content": "<p>Get it! Thanks for reply!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1920251,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-08-31T03:17:44.163000",
          "content": "<p><a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> Retrain the original model in step 2. Is the data set used gt+pl? I found that this approach is similar to the pseudo-label method of the first place😂. Is there a related paper for this gt+pl retraining method?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1920494,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-31T07:28:00.593000",
          "content": "<p><a href=\"https://www.kaggle.com/yasso1\" target=\"_blank\">@yasso1</a> It is indeed similar to the first place solution. Only pl data ( from feedback 2021) is used in step 2. For me, the pseudo labeling procedure is mainly based on my experience with previous competitions, and the original inspiration comes from <a href=\"https://arxiv.org/pdf/1905.00546.pdf\" target=\"_blank\">https://arxiv.org/pdf/1905.00546.pdf</a></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1920504,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-08-31T07:36:24.877000",
          "content": "<p>Sorry, I made a typo. I mean step 2. Is the data set used gt+pl?  I'm also curious why this technique works? Can I hear your interpretation?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1977876,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-10-08T10:16:59.990000",
          "content": "<blockquote>\n  <p>Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only</p>\n</blockquote>\n<p>I also have a question for this step, if I'm not wrong for this step, the model of each fold ar trained on 2 datasets, i.e. previous competition dataset with pl labels (probabilities) and this competition dataset with gt (discretized). <br>\nThe 1st 3 epochs are trained on the 1st dataset, and the 2nd 3 epochs is trained on the 2 are trained on the 2nd dataset. Am I right?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1978469,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-10-08T18:10:10.483000",
          "content": "<p>Yes you're correct</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1978794,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-10-09T02:40:30.973000",
          "content": "<p>Does this method improve compared to \"mixing pseudo-labels and the current training set for 5 epochs\"?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1987108,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-10-14T14:03:54.280000",
          "content": "<blockquote>\n  <ol>\n  <li>Train model with gt labels</li>\n  <li>Make predictions for old data (around 11000 essays) with each fold model</li>\n  <li>Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only</li>\n  </ol>\n</blockquote>\n<p>I just found that the loss of step 3 would be 0, because the model would output the same predictions as step 2. How can this work?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1989403,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-10-15T23:26:51.973000",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> You need to retrain the model from the original checkpoint you used (e.g. huggingface checkpoint)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1989404,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-10-15T23:27:43.757000",
          "content": "<p><a href=\"https://www.kaggle.com/yasso1\" target=\"_blank\">@yasso1</a> I didnt experiment too much with mixing pseudo labels except for initially, and the results were worse when mixing</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1989479,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-10-16T02:42:12.867000",
          "content": "<p>Thanks, <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> ! I figured it out after reading the paper. I'm participating in the FB3 competition. Seems the pseudo-labeling <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/360277\" target=\"_blank\">doesn't work well for this competition</a>. Maybe because the data of these 2 competitions are from different distributions. Any suggestion for utilizing unlabeled data from different distributions when using pseudo-labeling?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1989481,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-10-16T02:45:29.157000",
          "content": "<p>I think another benefit of training on PL first followed by training on the train set is that we're able to monitor the actual <code>train_loss</code> and <code>valid_loss</code>. It's super important when the datasets are from different distributions. Otherwise, you may unaware of overfitting on the training set.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1993093,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-10-18T05:39:35.870000",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> Np. Not too sure about how useful pl is for FB3 yet or how to use unlabeled data. Usually I'm not concerned about overfitting as long as val loss gets better, but indeed training loss becomes somewhat useless if you mix PL and GT labels</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1993373,
          "author_name": "william.wu",
          "author_url": "",
          "post_date": "2022-10-18T10:19:20.900000",
          "content": "<blockquote>\n  <p>Usually I'm not concerned about overfitting as long as val loss gets better</p>\n</blockquote>\n<p>Thanks <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> could you please share more about the insights for this? <br>\nWith traditional ML, if the tran_loss is much less than the val_loss, we'll say the model is overfitting and the error would be amplified on the test set.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1999378,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-10-22T09:30:40.890000",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> while what you said may be true, in many cases, difficult examples cannot be learned until train loss is much smaller than val loss. If you then assume you're splitting the data properly and your val loss is well correlated with test loss, it makes sense to keep training until val stops improving</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911348,
      "author_name": "Trushant Kalyanpur",
      "author_url": "",
      "post_date": "2022-08-24T03:24:28.020000",
      "content": "<p>Great work <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> ! Congratulations on becoming KGM! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1911166,
      "author_name": "Tifo",
      "author_url": "",
      "post_date": "2022-08-24T00:45:07.667000",
      "content": "<p>Congratulations! Learn a lot from your stacking method!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1911215,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-08-24T01:18:36.207000",
      "content": "<p>congratulations! Your team have a good job!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1911153,
      "author_name": "kkiller",
      "author_url": "",
      "post_date": "2022-08-24T00:31:53.500000",
      "content": "<p>Well done for us <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> !</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2691908,
      "author_name": "uestcoffergetwyq",
      "author_url": "",
      "post_date": "2024-03-11T14:27:50.277000",
      "content": "<p>Congratulations! Learn a lot from your stacking method!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2479786,
      "author_name": "kazisabahat",
      "author_url": "",
      "post_date": "2023-10-12T21:34:30.233000",
      "content": "<p>Congratulations. <br>\nThanks for sharing. 👍</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1956062,
      "author_name": "Kin",
      "author_url": "",
      "post_date": "2022-09-26T09:04:38.677000",
      "content": "<p>Can't find any code in Shujun_solution/src/Feedback_Prize  😂<br>\n<a href=\"url\" target=\"_blank\">https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution/tree/main/Shujun_solution/src</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 1956356,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-09-26T12:06:42.370000",
          "content": "<p>Fixed it! Sorry about that</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1932024,
      "author_name": "TouTie",
      "author_url": "",
      "post_date": "2022-09-09T07:56:23.933000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a>! But what do you mean by saying \"gt labels\" and \"pl labels“? Sry, i am new here.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1920254,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-08-31T03:26:56.663000",
      "content": "<p>I don't quite understand the SlidingWindowTransformerMode model, can you explain it?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1913403,
      "author_name": "The Devastator",
      "author_url": "",
      "post_date": "2022-08-25T10:06:31.623000",
      "content": "<p>When reading the stacking code it makes so much sense and looks so easy it makes you think \"why didn't I think about this?\"<br>\nAmazing work and congrats! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1912488,
      "author_name": "cyPress0208",
      "author_url": "",
      "post_date": "2022-08-24T18:57:01.963000",
      "content": "<p>Thx for sharing!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1911720,
      "author_name": "Tifo",
      "author_url": "",
      "post_date": "2022-08-24T09:04:14.067000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> ! Cloud you please give more details about <code>For instance, because I have 6 folds of nn and xgb, this way i had 6x6=36 preds to avg for each single model.</code>? </p>\n<p>I have some trouble understanding <code>nn</code> here and why <code>36 preds</code></p>\n<p>Thanks in advance!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911737,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T09:19:38.260000",
          "content": "<p><a href=\"https://www.kaggle.com/fankaixie\" target=\"_blank\">@fankaixie</a> Thansk and congrats on getting gold! Sorry about the confusion. I changed the text in question to</p>\n<pre><code>For instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way i have 6x6=36 preds to avg for each single model.\n</code></pre>\n<p>What I did in the beginning was average the neural network predictions and then use those predictions to generate features for the stacking models (1x6); however, later I found that it was better to get features for each fold model and use all folds of xgb models to make predictions for that fold (thus 6x6).</p>\n<p>LMK if this clears it up! </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1911778,
          "author_name": "Tifo",
          "author_url": "",
          "post_date": "2022-08-24T09:57:52.270000",
          "content": "<p>Got it! Very clear~<br>\nThanks for your reply!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911359,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-08-24T03:47:24.297000",
      "content": "<p>Congratulations and thanks for this great sharing, for StartifiedGroupKFold, which field did you Stratify on?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911364,
          "author_name": "Zacchaeus",
          "author_url": "",
          "post_date": "2022-08-24T03:49:33.413000",
          "content": "<p>Effectiveness I guess? Like this: <a href=\"https://www.kaggle.com/code/tanlikesmath/here-have-some-folds\" target=\"_blank\">https://www.kaggle.com/code/tanlikesmath/here-have-some-folds</a></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1911377,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T04:05:11.537000",
          "content": "<p><a href=\"https://www.kaggle.com/goldenlock\" target=\"_blank\">@goldenlock</a> Thanks for the kind words and congrats on getting gold to you as well. My split is similar to what Zacchaues mentioned, but I simply stratefied on effectiveness instead:</p>\n<pre><code>cv = StratifiedGroupKFold(n_splits=args.nfolds,shuffle=True,random_state=2022)\nfor fold, (train_idxs, test_idxs) in enumerate(cv.split(train, train['label'], train['essay_id'])):\n    train['fold'][test_idxs]=fold\n</code></pre>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1911381,
          "author_name": "gezi",
          "author_url": "",
          "post_date": "2022-08-24T04:11:48.683000",
          "content": "<p>Thanks！ <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a>  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a> </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1911178,
      "author_name": "RDizzl3",
      "author_url": "",
      "post_date": "2022-08-24T00:52:08.227000",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a>! Congrats on the awesome result. You mention that you trained the essays in entirety with special tokens. Could you provide a data example for an essay?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911204,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T01:09:21.197000",
          "content": "<p>Added an example in the beginning of the post. LMK if it makes sense</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911210,
          "author_name": "RDizzl3",
          "author_url": "",
          "post_date": "2022-08-24T01:14:01.910000",
          "content": "<p>Yes this is great! So after this treatment for the model the original data set was condensed to the ~4K essays and ~15k with PL? Can’t recall the data shapes exactly for training? </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1911214,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T01:18:28.433000",
          "content": "<p>4K essays and 11K PL since we exlcude those in the training set</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911784,
          "author_name": "RDizzl3",
          "author_url": "",
          "post_date": "2022-08-24T10:06:06.393000",
          "content": "<p><a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> will you be sharing your training code?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1912645,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T22:14:49.007000",
          "content": "<p>Yes very soon!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911148,
      "author_name": "Raja Biswas",
      "author_url": "",
      "post_date": "2022-08-24T00:29:47.393000",
      "content": "<p>congratulations! amazing work :-)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911205,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T01:09:41.407000",
          "content": "<p>Thanks! Congrats on getting 3rd!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1911134,
      "author_name": "Zacchaeus",
      "author_url": "",
      "post_date": "2022-08-24T00:21:18.873000",
      "content": "<p>congrats new GM!🎉 May I ask how was your best single model?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1911137,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T00:24:09.923000",
          "content": "<p>Thanks! Congrats on your gold medal as well!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911147,
          "author_name": "Shujun",
          "author_url": "",
          "post_date": "2022-08-24T00:29:14.950000",
          "content": "<p>Best single model was my 5th round pl deberta-v3-large with public 0.56 and private 0.558</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911151,
          "author_name": "Zacchaeus",
          "author_url": "",
          "post_date": "2022-08-24T00:31:15.117000",
          "content": "<p>wow that's impressive! Our best was public 0.583 </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1911172,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-08-24T00:47:28.393000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1912652,
      "author_name": "OscarAbeleda",
      "author_url": "",
      "post_date": "2022-08-24T22:25:03.367000",
      "content": "<p>thanks for sharing</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 59,
  "messages": [
    {
      "id": 1911130,
      "postDate": "2022-08-24T00:19:11.443Z",
      "content": "<p>First, thanks to all the organizers and kaggle staff and congrats to all the winners and thanks to my amazing teammates <a href=\"https://www.kaggle.com/kneroma\" target=\"_blank\">@kneroma</a> and <a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> !  With this gold medal, <a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> and I finally become competition GMs. We have released our code/notebooks:</p>\n<p>code: <a href=\"https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution\" target=\"_blank\">https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution</a></p>\n<p>2nd logloss notebook: <a href=\"https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824\" target=\"_blank\">https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824</a></p>\n<p>3rd place efficiency notebook: <a href=\"https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605\" target=\"_blank\">https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605</a></p>\n<p>Below is a summary of our solution. Our best private is 0.553 and our best selected private is 0.554. Please feel free to ask if you have any questions.</p>\n<h1>Transformer modeling</h1>\n<p>Each of our team members has their own training pipeline for transformer models. On a high level, our transformer models look at the entirety of each essay and output predictions of effectiveness for each discourse either via pooling of discourse tokens or a classification token added to the front of the each discourse. Importantly, directly inputting essays results in a situation where the model is not 100% sure about where it needs to make predictions, so to circumvent this, we use either a prompt (i.e. concat <code>f'({discourse_type} start)'</code> and <code>f'({discourse_type} end)'</code> to the beginning and end of each discourse to signal where predictions need to be made) or simply concat special tokens added to the tokenizer instead (i.e. <code>f'&lt;{discourse_type}&gt;'</code> and <code>f'&lt;{discourse_type}\\&gt;'</code>). You can find an example below with a highlighted segment.</p>\n<p><img src=\"https://raw.githubusercontent.com/Shujun-He/Shujun-He-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution/main/graphics/example_segments.png\" alt=\"example\"></p>\n<h2>Encoders</h2>\n<p>Deberta worked the best since (IMO) it supports unlimited input length and uses disentangled attention with relative positional embedding; in fact, our ensemble consists entirely of deberta variants. For me, it was also helpful to add a GRU/LSTM after pooling on the pooled discourse representations. Tom used my SlidingWindowTransformerModel (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235</a>) from the last Feedback competiion, which stablized training for him.</p>\n<h2>Pretraining</h2>\n<p>Kkiller used pretrained weights from his solution in the last competiion (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478</a>) and Tom and I found pretrained tascj0 models to be good starting points. We used some of the weights that tascj0 released after the last Feedback competition and Tom also pretrained some new ones on his own. Please checkout out tascj0's solution post if you'd like to learn more (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424)\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424)</a>.<br>\n In addition, Tom used MLM for some of his models.  Further, some of our models simply used huggingface weights.</p>\n<h2>Max sequence length</h2>\n<p>I used a max sequence length of 1280 in both training and inference, since I found that 99.9% of discourses fall within that range, whereas my teammates used up to around 1800 during inference and as low as 648 in training.</p>\n<h1>Pseudo labeling</h1>\n<p>Pseudo labeling is an integral part of all our solution. We use essays from the training set of last Feedback competition that are also not present in the training set of this competition. Our procedure is as follows:</p>\n<ol>\n<li>Train model with gt labels</li>\n<li>Make predictions for old data (around 11000 essays) with each fold model</li>\n<li>Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only</li>\n<li>Repeat from step 2</li>\n</ol>\n<p>For my pipeline, I saw improvement until 5 rounds of the above procedure. For Tom, it was only helpful for one round and kkiller did not have enough time to try multi-round pl.</p>\n<h1>Stacking</h1>\n<p>Stacking provides significantly improvement in both cv/lb (around 0.004). Our stacking framework is primarily inspired by my team's solution (<a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235</a>) in the previous feedback competition. In addition to predicted probabilities outputted by the transformer models, we also utilized the token probabilities for each discourse, which we call prob_sequences. Compared to the previous Feedback competition, stacking is much faster since we don't have to deal with a huge amount of candidate sequences. Our features are as follows:</p>\n<pre><code> #make features\ndef get_xgb_features(train_df,prob_sequences):\n    features2calculate=[f\"instability_{i}\" for i in range(4)]+\\\n    [f\"begin_{i}\" for i in range(3)]+\\\n    [f\"end_{i}\" for i in range(3)]#+\\\n    #[\"entropy\"]\n\n    calculated_features=[]\n    for i,prob_seq in tqdm(enumerate(prob_sequences)):\n\n        tmp=[]\n        #quants = np.linspace(0,1,n_quan)\n        prob_seq=np.array(prob_seq)\n        instability = []\n        #all_quants=[]\n        tmp.append(np.diff(prob_seq[:,:],0).mean(0))\n        tmp.append([(np.diff(prob_seq[:,[1,2]].sum(1))**2).mean()])\n\n        tmp.append(prob_seq[:5,:].mean(0))\n        tmp.append(prob_seq[-5:,:].mean(0))\n\n        calculated_features.append(np.concatenate(tmp))\n\n\n    train_df[features2calculate]=calculated_features\n    train_df['len']=[len(s) for s in prob_sequences]\n\n    calculated_features=np.array(calculated_features)\n    calculated_features.shape\n\n    p_features=[]\n    n_features=[]\n    neighbor_features=['Ineffective','Adequate','Effective','discourse_type']\n    neighbor_features_values=train_df[neighbor_features].values\n    for i in tqdm(range(len(train_df))):\n        if i&gt;1 and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i-1]:\n            p_features.append(neighbor_features_values[i-1])\n        else:\n            p_features.append(neighbor_features_values[i])\n\n        if i&lt;(len(train_df)-1) and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i+1]:\n            n_features.append(neighbor_features_values[i+1])\n        else:\n            n_features.append(neighbor_features_values[i])\n\n    train_df[[f+\"_previous\" for f in neighbor_features]]=p_features\n    train_df[[f+\"_next\" for f in neighbor_features]]=n_features\n\n    train_df['mean_Ineffective']=train_df.groupby(\"essay_id\")[\"Ineffective\"].transform(\"mean\")\n    train_df['mean_Adequate']=train_df.groupby(\"essay_id\")[\"Adequate\"].transform(\"mean\")\n    train_df['mean_Effective']=train_df.groupby(\"essay_id\")[\"Effective\"].transform(\"mean\")\n\n    train_df['std_Ineffective']=train_df.groupby(\"essay_id\")[\"Ineffective\"].transform(\"std\")\n    train_df['std_Adequate']=train_df.groupby(\"essay_id\")[\"Adequate\"].transform(\"std\")\n    train_df['std_Effective']=train_df.groupby(\"essay_id\")[\"Effective\"].transform(\"std\")\n\n    train_df['discourse_count']=train_df.groupby(\"essay_id\")['discourse_type'].transform(\"count\")\n\n    cnts=train_df.groupby('essay_id')['discourse_type'].apply(lambda x: x.value_counts())\n\n    #new_df=[]\n    discourse_types=['Claim','Evidence','Concluding Statement','Lead','Position','Counterclaim','Rebuttal']\n    value_count_hash={}\n    for t in discourse_types:\n        value_count_hash[t]={}\n    for key in cnts.keys():\n        value_count_hash[key[1]][key[0]]=cnts[key]\n\n    discourse_cnts=[]    \n    for essay_id in train_df['essay_id'].unique():\n        row=[essay_id]\n        for d in discourse_types:\n            try:\n                row.append(value_count_hash[d][essay_id])\n            except:\n                row.append(0)\n        discourse_cnts.append(row)\n\n    discourse_cnts=pd.DataFrame(discourse_cnts,columns=['essay_id']+[f'{d}_count' for d in discourse_types])    \n    #discourse_cnts\n\n    train_df=train_df.merge(discourse_cnts,how='left',on='essay_id')\n    train_df\n\n    #train_df\n\n    return train_df\n</code></pre>\n<pre><code>\n</code></pre>\n<p>Since stacking is fast, it works best when we use each fold predictions with xgb separately and then avg. For instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way I have 6x6=36 preds to avg for each single model.</p>\n<h1>Best single models</h1>\n<p>All our best single models were deberta-large/deberta-v3-large variants. For Tom and Kkiller, their best single models came from 1st round PL, whereas for me it came from 4th round PL.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Shujun</th>\n<th>Tom</th>\n<th>Kkiller</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Public LB</td>\n<td>0.560</td>\n<td>0.566</td>\n<td>0.562</td>\n</tr>\n<tr>\n<td>Private LB</td>\n<td>0.558</td>\n<td>0.571</td>\n<td>0.562</td>\n</tr>\n<tr>\n<td>Local CV</td>\n<td>0.571</td>\n<td>N/A</td>\n<td>0.572</td>\n</tr>\n</tbody>\n</table>\n<h1>Some more tips/tricks</h1>\n<p>For Tom AWP was useful, and he reported around 0.003 cv improvement with AWP (eps=1e-3 and lr=1e-2 for large models, 1e-4 for xlarge models).</p>\n<p>It was also important to split the data with <code>StratifiedGroupKFold</code> instead of <code>GroupKFold</code>. For me I started out with <code>GroupKFold</code> but found better correlation between cv/lb after switching to <code>StratifiedGroupKFold</code>.</p>\n<p>For ensembling models with the same cv split, we used GP_minimize to find optimal weights and otherwise weights were determined arbitrarily.</p>\n<p>Gradient accumulation was useful since we had to deal with very long sequences.</p>",
      "rawMarkdown": "First, thanks to all the organizers and kaggle staff and congrats to all the winners and thanks to my amazing teammates @kneroma and @tikutiku !  With this gold medal, @tikutiku and I finally become competition GMs. We have released our code/notebooks:\n\ncode: https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution\n\n2nd logloss notebook: https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824\n\n3rd place efficiency notebook: https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605\n\n\n\nBelow is a summary of our solution. Our best private is 0.553 and our best selected private is 0.554. Please feel free to ask if you have any questions.\n\n\n# Transformer modeling\n\nEach of our team members has their own training pipeline for transformer models. On a high level, our transformer models look at the entirety of each essay and output predictions of effectiveness for each discourse either via pooling of discourse tokens or a classification token added to the front of the each discourse. Importantly, directly inputting essays results in a situation where the model is not 100% sure about where it needs to make predictions, so to circumvent this, we use either a prompt (i.e. concat ```f'({discourse_type} start)'``` and ```f'({discourse_type} end)'``` to the beginning and end of each discourse to signal where predictions need to be made) or simply concat special tokens added to the tokenizer instead (i.e. ```f'<{discourse_type}>'``` and ```f'<{discourse_type}\\>'```). You can find an example below with a highlighted segment.\n\n![example](https://raw.githubusercontent.com/Shujun-He/Shujun-He-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution/main/graphics/example_segments.png)\n\n## Encoders\n\nDeberta worked the best since (IMO) it supports unlimited input length and uses disentangled attention with relative positional embedding; in fact, our ensemble consists entirely of deberta variants. For me, it was also helpful to add a GRU/LSTM after pooling on the pooled discourse representations. Tom used my SlidingWindowTransformerModel (https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235) from the last Feedback competiion, which stablized training for him.\n\n## Pretraining\n\nKkiller used pretrained weights from his solution in the last competiion (https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478) and Tom and I found pretrained tascj0 models to be good starting points. We used some of the weights that tascj0 released after the last Feedback competition and Tom also pretrained some new ones on his own. Please checkout out tascj0's solution post if you'd like to learn more (https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424).\n In addition, Tom used MLM for some of his models.  Further, some of our models simply used huggingface weights.\n## Max sequence length\n\nI used a max sequence length of 1280 in both training and inference, since I found that 99.9% of discourses fall within that range, whereas my teammates used up to around 1800 during inference and as low as 648 in training.\n\n# Pseudo labeling\nPseudo labeling is an integral part of all our solution. We use essays from the training set of last Feedback competition that are also not present in the training set of this competition. Our procedure is as follows:\n\n1. Train model with gt labels\n2. Make predictions for old data (around 11000 essays) with each fold model\n3. Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only\n4. Repeat from step 2\n\nFor my pipeline, I saw improvement until 5 rounds of the above procedure. For Tom, it was only helpful for one round and kkiller did not have enough time to try multi-round pl.\n\n# Stacking\n\nStacking provides significantly improvement in both cv/lb (around 0.004). Our stacking framework is primarily inspired by my team's solution (https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235) in the previous feedback competition. In addition to predicted probabilities outputted by the transformer models, we also utilized the token probabilities for each discourse, which we call prob_sequences. Compared to the previous Feedback competition, stacking is much faster since we don't have to deal with a huge amount of candidate sequences. Our features are as follows:\n\n ```\n #make features\ndef get_xgb_features(train_df,prob_sequences):\n    features2calculate=[f\"instability_{i}\" for i in range(4)]+\\\n    [f\"begin_{i}\" for i in range(3)]+\\\n    [f\"end_{i}\" for i in range(3)]#+\\\n    #[\"entropy\"]\n\n    calculated_features=[]\n    for i,prob_seq in tqdm(enumerate(prob_sequences)):\n\n        tmp=[]\n        #quants = np.linspace(0,1,n_quan)\n        prob_seq=np.array(prob_seq)\n        instability = []\n        #all_quants=[]\n        tmp.append(np.diff(prob_seq[:,:],0).mean(0))\n        tmp.append([(np.diff(prob_seq[:,[1,2]].sum(1))**2).mean()])\n\n        tmp.append(prob_seq[:5,:].mean(0))\n        tmp.append(prob_seq[-5:,:].mean(0))\n\n        calculated_features.append(np.concatenate(tmp))\n\n\n    train_df[features2calculate]=calculated_features\n    train_df['len']=[len(s) for s in prob_sequences]\n\n    calculated_features=np.array(calculated_features)\n    calculated_features.shape\n\n    p_features=[]\n    n_features=[]\n    neighbor_features=['Ineffective','Adequate','Effective','discourse_type']\n    neighbor_features_values=train_df[neighbor_features].values\n    for i in tqdm(range(len(train_df))):\n        if i>1 and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i-1]:\n            p_features.append(neighbor_features_values[i-1])\n        else:\n            p_features.append(neighbor_features_values[i])\n\n        if i<(len(train_df)-1) and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i+1]:\n            n_features.append(neighbor_features_values[i+1])\n        else:\n            n_features.append(neighbor_features_values[i])\n\n    train_df[[f+\"_previous\" for f in neighbor_features]]=p_features\n    train_df[[f+\"_next\" for f in neighbor_features]]=n_features\n\n    train_df['mean_Ineffective']=train_df.groupby(\"essay_id\")[\"Ineffective\"].transform(\"mean\")\n    train_df['mean_Adequate']=train_df.groupby(\"essay_id\")[\"Adequate\"].transform(\"mean\")\n    train_df['mean_Effective']=train_df.groupby(\"essay_id\")[\"Effective\"].transform(\"mean\")\n\n    train_df['std_Ineffective']=train_df.groupby(\"essay_id\")[\"Ineffective\"].transform(\"std\")\n    train_df['std_Adequate']=train_df.groupby(\"essay_id\")[\"Adequate\"].transform(\"std\")\n    train_df['std_Effective']=train_df.groupby(\"essay_id\")[\"Effective\"].transform(\"std\")\n\n    train_df['discourse_count']=train_df.groupby(\"essay_id\")['discourse_type'].transform(\"count\")\n\n    cnts=train_df.groupby('essay_id')['discourse_type'].apply(lambda x: x.value_counts())\n\n    #new_df=[]\n    discourse_types=['Claim','Evidence','Concluding Statement','Lead','Position','Counterclaim','Rebuttal']\n    value_count_hash={}\n    for t in discourse_types:\n        value_count_hash[t]={}\n    for key in cnts.keys():\n        value_count_hash[key[1]][key[0]]=cnts[key]\n\n    discourse_cnts=[]    \n    for essay_id in train_df['essay_id'].unique():\n        row=[essay_id]\n        for d in discourse_types:\n            try:\n                row.append(value_count_hash[d][essay_id])\n            except:\n                row.append(0)\n        discourse_cnts.append(row)\n\n    discourse_cnts=pd.DataFrame(discourse_cnts,columns=['essay_id']+[f'{d}_count' for d in discourse_types])    \n    #discourse_cnts\n\n    train_df=train_df.merge(discourse_cnts,how='left',on='essay_id')\n    train_df\n\n    #train_df\n\n    return train_df\n```    \n\nSince stacking is fast, it works best when we use each fold predictions with xgb separately and then avg. For instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way I have 6x6=36 preds to avg for each single model.\n\n# Best single models\n\nAll our best single models were deberta-large/deberta-v3-large variants. For Tom and Kkiller, their best single models came from 1st round PL, whereas for me it came from 4th round PL.\n\n|            | Shujun |  Tom  | Kkiller |\n|:----------:|:------:|:-----:|:-------:|\n|  Public LB |  0.560 | 0.566 |  0.562  |\n| Private LB |  0.558 | 0.571 |  0.562  |\n|  Local CV  |  0.571 |  N/A  |  0.572  |\n\n\n\n# Some more tips/tricks\n\nFor Tom AWP was useful, and he reported around 0.003 cv improvement with AWP (eps=1e-3 and lr=1e-2 for large models, 1e-4 for xlarge models).\n\nIt was also important to split the data with ```StratifiedGroupKFold``` instead of ```GroupKFold```. For me I started out with ```GroupKFold``` but found better correlation between cv/lb after switching to ```StratifiedGroupKFold```.\n\nFor ensembling models with the same cv split, we used GP_minimize to find optimal weights and otherwise weights were determined arbitrarily.\n\nGradient accumulation was useful since we had to deal with very long sequences.\n\n\n\n",
      "votes": 94
    },
    {
      "id": 1911190,
      "postDate": "2022-08-24T00:58:19.240Z",
      "content": "<p>Sample pre-processed essay from my pipeline</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1498852%2F68da2ecd156457e223a19e9b452a7d9b%2Fessay_with_special_tokens.png?generation=1661302642758307&amp;alt=media\" alt=\"essay_with_special_tokens\"></p>",
      "rawMarkdown": "Sample pre-processed essay from my pipeline\n\n\n![essay_with_special_tokens](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1498852%2F68da2ecd156457e223a19e9b452a7d9b%2Fessay_with_special_tokens.png?generation=1661302642758307&alt=media)",
      "votes": 6,
      "replies": [
        {
          "id": 1911599,
          "postDate": "2022-08-24T07:36:44.333Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1936456,
      "postDate": "2022-09-12T18:57:28.800Z",
      "content": "<p><a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> Congratulations and welcome to the club! I'm really happy for you.</p>",
      "rawMarkdown": "@tikutiku Congratulations and welcome to the club! I'm really happy for you.",
      "votes": 1,
      "replies": [
        {
          "id": 1936613,
          "postDate": "2022-09-12T23:01:25.263Z",
          "content": "<p>Thank you <a href=\"https://www.kaggle.com/mpware\" target=\"_blank\">@mpware</a> ! It took me a while to get the last gold, but finally I did it :) </p>",
          "rawMarkdown": "Thank you @mpware ! It took me a while to get the last gold, but finally I did it :) ",
          "votes": 1
        },
        {
          "id": 1936944,
          "postDate": "2022-09-13T06:43:24.340Z",
          "content": "<p>Yes, I will be happy to team up again with you in a further competition.</p>",
          "rawMarkdown": "Yes, I will be happy to team up again with you in a further competition.",
          "votes": 1
        },
        {
          "id": 1936992,
          "postDate": "2022-09-13T07:15:12.077Z",
          "content": "<p>Sure, I look forward to that!</p>",
          "rawMarkdown": "Sure, I look forward to that!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911987,
      "postDate": "2022-08-24T12:37:57.770Z",
      "content": "<p>Congratulations Great Work <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> </p>",
      "rawMarkdown": "Congratulations Great Work @shujun717 ",
      "votes": 1
    },
    {
      "id": 1911669,
      "postDate": "2022-08-24T08:24:24.220Z",
      "content": "<p>Great share and congratulations on becoming KGM! Cheers</p>",
      "rawMarkdown": "Great share and congratulations on becoming KGM! Cheers",
      "votes": 1
    },
    {
      "id": 1911455,
      "postDate": "2022-08-24T05:39:26.720Z",
      "content": "<p>Supa thx for sharing!</p>",
      "rawMarkdown": "Supa thx for sharing!",
      "votes": 1
    },
    {
      "id": 1911406,
      "postDate": "2022-08-24T04:44:40.490Z",
      "content": "<p>Congratulations on in the money and becoming KGM! I have a question:</p>\n<blockquote>\n  <p>3 epochs on pl labels only first and then 3 more epochs on gt labels only</p>\n</blockquote>\n<p>Did training on gt labels use the same config as training with pl labels? Thanks!</p>",
      "rawMarkdown": "Congratulations on in the money and becoming KGM! I have a question:\n> 3 epochs on pl labels only first and then 3 more epochs on gt labels only\n\nDid training on gt labels use the same config as training with pl labels? Thanks!",
      "votes": 1,
      "replies": [
        {
          "id": 1911744,
          "postDate": "2022-08-24T09:23:46.190Z",
          "content": "<p><a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> Thanks and yes I use the same config for both first round training and multi-round pl training. The only difference is the number of epochs. Since I use a linear scheduler, I change the number of total steps to total pl label steps + total gt label steps. My teammates also do something similar: the training schedule changes a bit but nothing else</p>",
          "rawMarkdown": "@syhens Thanks and yes I use the same config for both first round training and multi-round pl training. The only difference is the number of epochs. Since I use a linear scheduler, I change the number of total steps to total pl label steps + total gt label steps. My teammates also do something similar: the training schedule changes a bit but nothing else",
          "votes": 1
        },
        {
          "id": 1911759,
          "postDate": "2022-08-24T09:42:53.837Z",
          "content": "<p>Get it! Thanks for reply!</p>",
          "rawMarkdown": "Get it! Thanks for reply!"
        },
        {
          "id": 1920251,
          "postDate": "2022-08-31T03:17:44.163Z",
          "content": "<p><a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> Retrain the original model in step 2. Is the data set used gt+pl? I found that this approach is similar to the pseudo-label method of the first place😂. Is there a related paper for this gt+pl retraining method?</p>",
          "rawMarkdown": "@shujun717 Retrain the original model in step 2. Is the data set used gt+pl? I found that this approach is similar to the pseudo-label method of the first place😂. Is there a related paper for this gt+pl retraining method?"
        },
        {
          "id": 1920494,
          "postDate": "2022-08-31T07:28:00.593Z",
          "content": "<p><a href=\"https://www.kaggle.com/yasso1\" target=\"_blank\">@yasso1</a> It is indeed similar to the first place solution. Only pl data ( from feedback 2021) is used in step 2. For me, the pseudo labeling procedure is mainly based on my experience with previous competitions, and the original inspiration comes from <a href=\"https://arxiv.org/pdf/1905.00546.pdf\" target=\"_blank\">https://arxiv.org/pdf/1905.00546.pdf</a></p>",
          "rawMarkdown": "@yasso1 It is indeed similar to the first place solution. Only pl data ( from feedback 2021) is used in step 2. For me, the pseudo labeling procedure is mainly based on my experience with previous competitions, and the original inspiration comes from https://arxiv.org/pdf/1905.00546.pdf"
        },
        {
          "id": 1920504,
          "postDate": "2022-08-31T07:36:24.877Z",
          "content": "<p>Sorry, I made a typo. I mean step 2. Is the data set used gt+pl?  I'm also curious why this technique works? Can I hear your interpretation?</p>",
          "rawMarkdown": "Sorry, I made a typo. I mean step 2. Is the data set used gt+pl?  I'm also curious why this technique works? Can I hear your interpretation?"
        },
        {
          "id": 1977876,
          "postDate": "2022-10-08T10:16:59.990Z",
          "content": "<blockquote>\n  <p>Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only</p>\n</blockquote>\n<p>I also have a question for this step, if I'm not wrong for this step, the model of each fold ar trained on 2 datasets, i.e. previous competition dataset with pl labels (probabilities) and this competition dataset with gt (discretized). <br>\nThe 1st 3 epochs are trained on the 1st dataset, and the 2nd 3 epochs is trained on the 2 are trained on the 2nd dataset. Am I right?</p>",
          "rawMarkdown": ">Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only\n\nI also have a question for this step, if I'm not wrong for this step, the model of each fold ar trained on 2 datasets, i.e. previous competition dataset with pl labels (probabilities) and this competition dataset with gt (discretized). \nThe 1st 3 epochs are trained on the 1st dataset, and the 2nd 3 epochs is trained on the 2 are trained on the 2nd dataset. Am I right?"
        },
        {
          "id": 1978469,
          "postDate": "2022-10-08T18:10:10.483Z",
          "content": "<p>Yes you're correct</p>",
          "rawMarkdown": "Yes you're correct"
        },
        {
          "id": 1978794,
          "postDate": "2022-10-09T02:40:30.973Z",
          "content": "<p>Does this method improve compared to \"mixing pseudo-labels and the current training set for 5 epochs\"?</p>",
          "rawMarkdown": "Does this method improve compared to \"mixing pseudo-labels and the current training set for 5 epochs\"?"
        },
        {
          "id": 1987108,
          "postDate": "2022-10-14T14:03:54.280Z",
          "content": "<blockquote>\n  <ol>\n  <li>Train model with gt labels</li>\n  <li>Make predictions for old data (around 11000 essays) with each fold model</li>\n  <li>Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only</li>\n  </ol>\n</blockquote>\n<p>I just found that the loss of step 3 would be 0, because the model would output the same predictions as step 2. How can this work?</p>",
          "rawMarkdown": "> 1. Train model with gt labels\n> 2. Make predictions for old data (around 11000 essays) with each fold model\n> 3. Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only\n\nI just found that the loss of step 3 would be 0, because the model would output the same predictions as step 2. How can this work?"
        },
        {
          "id": 1989403,
          "postDate": "2022-10-15T23:26:51.973Z",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> You need to retrain the model from the original checkpoint you used (e.g. huggingface checkpoint)</p>",
          "rawMarkdown": "@wuwenmin You need to retrain the model from the original checkpoint you used (e.g. huggingface checkpoint)"
        },
        {
          "id": 1989404,
          "postDate": "2022-10-15T23:27:43.757Z",
          "content": "<p><a href=\"https://www.kaggle.com/yasso1\" target=\"_blank\">@yasso1</a> I didnt experiment too much with mixing pseudo labels except for initially, and the results were worse when mixing</p>",
          "rawMarkdown": "@yasso1 I didnt experiment too much with mixing pseudo labels except for initially, and the results were worse when mixing"
        },
        {
          "id": 1989479,
          "postDate": "2022-10-16T02:42:12.867Z",
          "content": "<p>Thanks, <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> ! I figured it out after reading the paper. I'm participating in the FB3 competition. Seems the pseudo-labeling <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/360277\" target=\"_blank\">doesn't work well for this competition</a>. Maybe because the data of these 2 competitions are from different distributions. Any suggestion for utilizing unlabeled data from different distributions when using pseudo-labeling?</p>",
          "rawMarkdown": "Thanks, @shujun717 ! I figured it out after reading the paper. I'm participating in the FB3 competition. Seems the pseudo-labeling [doesn't work well for this competition](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/360277). Maybe because the data of these 2 competitions are from different distributions. Any suggestion for utilizing unlabeled data from different distributions when using pseudo-labeling?"
        },
        {
          "id": 1989481,
          "postDate": "2022-10-16T02:45:29.157Z",
          "content": "<p>I think another benefit of training on PL first followed by training on the train set is that we're able to monitor the actual <code>train_loss</code> and <code>valid_loss</code>. It's super important when the datasets are from different distributions. Otherwise, you may unaware of overfitting on the training set.</p>",
          "rawMarkdown": "I think another benefit of training on PL first followed by training on the train set is that we're able to monitor the actual `train_loss` and `valid_loss`. It's super important when the datasets are from different distributions. Otherwise, you may unaware of overfitting on the training set."
        },
        {
          "id": 1993093,
          "postDate": "2022-10-18T05:39:35.870Z",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> Np. Not too sure about how useful pl is for FB3 yet or how to use unlabeled data. Usually I'm not concerned about overfitting as long as val loss gets better, but indeed training loss becomes somewhat useless if you mix PL and GT labels</p>",
          "rawMarkdown": "@wuwenmin Np. Not too sure about how useful pl is for FB3 yet or how to use unlabeled data. Usually I'm not concerned about overfitting as long as val loss gets better, but indeed training loss becomes somewhat useless if you mix PL and GT labels"
        },
        {
          "id": 1993373,
          "postDate": "2022-10-18T10:19:20.900Z",
          "content": "<blockquote>\n  <p>Usually I'm not concerned about overfitting as long as val loss gets better</p>\n</blockquote>\n<p>Thanks <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> could you please share more about the insights for this? <br>\nWith traditional ML, if the tran_loss is much less than the val_loss, we'll say the model is overfitting and the error would be amplified on the test set.</p>",
          "rawMarkdown": ">Usually I'm not concerned about overfitting as long as val loss gets better\n\nThanks @shujun717 could you please share more about the insights for this? \nWith traditional ML, if the tran_loss is much less than the val_loss, we'll say the model is overfitting and the error would be amplified on the test set."
        },
        {
          "id": 1999378,
          "postDate": "2022-10-22T09:30:40.890Z",
          "content": "<p><a href=\"https://www.kaggle.com/wuwenmin\" target=\"_blank\">@wuwenmin</a> while what you said may be true, in many cases, difficult examples cannot be learned until train loss is much smaller than val loss. If you then assume you're splitting the data properly and your val loss is well correlated with test loss, it makes sense to keep training until val stops improving</p>",
          "rawMarkdown": "@wuwenmin while what you said may be true, in many cases, difficult examples cannot be learned until train loss is much smaller than val loss. If you then assume you're splitting the data properly and your val loss is well correlated with test loss, it makes sense to keep training until val stops improving"
        }
      ]
    },
    {
      "id": 1911348,
      "postDate": "2022-08-24T03:24:28.020Z",
      "content": "<p>Great work <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> ! Congratulations on becoming KGM! </p>",
      "rawMarkdown": "Great work @shujun717 ! Congratulations on becoming KGM! ",
      "votes": 1
    },
    {
      "id": 1911166,
      "postDate": "2022-08-24T00:45:07.667Z",
      "content": "<p>Congratulations! Learn a lot from your stacking method!</p>",
      "rawMarkdown": "Congratulations! Learn a lot from your stacking method!",
      "votes": 1
    },
    {
      "id": 1911215,
      "postDate": "2022-08-24T01:18:36.207Z",
      "content": "<p>congratulations! Your team have a good job!</p>",
      "rawMarkdown": "congratulations! Your team have a good job!",
      "votes": 2
    },
    {
      "id": 1911153,
      "postDate": "2022-08-24T00:31:53.500Z",
      "content": "<p>Well done for us <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> !</p>",
      "rawMarkdown": "Well done for us @shujun717 !",
      "votes": 2
    },
    {
      "id": 2691908,
      "postDate": "2024-03-11T14:27:50.277Z",
      "content": "<p>Congratulations! Learn a lot from your stacking method!</p>",
      "rawMarkdown": "Congratulations! Learn a lot from your stacking method!"
    },
    {
      "id": 2479786,
      "postDate": "2023-10-12T21:34:30.233Z",
      "content": "<p>Congratulations. <br>\nThanks for sharing. 👍</p>",
      "rawMarkdown": "Congratulations. \nThanks for sharing. 👍"
    },
    {
      "id": 1956062,
      "postDate": "2022-09-26T09:04:38.677Z",
      "content": "<p>Can't find any code in Shujun_solution/src/Feedback_Prize  😂<br>\n<a href=\"url\" target=\"_blank\">https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution/tree/main/Shujun_solution/src</a></p>",
      "rawMarkdown": "Can't find any code in Shujun_solution/src/Feedback_Prize  😂\n[https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution/tree/main/Shujun_solution/src](url)",
      "replies": [
        {
          "id": 1956356,
          "postDate": "2022-09-26T12:06:42.370Z",
          "content": "<p>Fixed it! Sorry about that</p>",
          "rawMarkdown": "Fixed it! Sorry about that"
        }
      ]
    },
    {
      "id": 1932024,
      "postDate": "2022-09-09T07:56:23.933Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a>! But what do you mean by saying \"gt labels\" and \"pl labels“? Sry, i am new here.</p>",
      "rawMarkdown": "Congratulations @shujun717! But what do you mean by saying \"gt labels\" and \"pl labels“? Sry, i am new here."
    },
    {
      "id": 1920254,
      "postDate": "2022-08-31T03:26:56.663Z",
      "content": "<p>I don't quite understand the SlidingWindowTransformerMode model, can you explain it?</p>",
      "rawMarkdown": "I don't quite understand the SlidingWindowTransformerMode model, can you explain it?"
    },
    {
      "id": 1913403,
      "postDate": "2022-08-25T10:06:31.623Z",
      "content": "<p>When reading the stacking code it makes so much sense and looks so easy it makes you think \"why didn't I think about this?\"<br>\nAmazing work and congrats! </p>",
      "rawMarkdown": "When reading the stacking code it makes so much sense and looks so easy it makes you think \"why didn't I think about this?\"\nAmazing work and congrats! "
    },
    {
      "id": 1912488,
      "postDate": "2022-08-24T18:57:01.963Z",
      "content": "<p>Thx for sharing!</p>",
      "rawMarkdown": "Thx for sharing!"
    },
    {
      "id": 1911720,
      "postDate": "2022-08-24T09:04:14.067Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> ! Cloud you please give more details about <code>For instance, because I have 6 folds of nn and xgb, this way i had 6x6=36 preds to avg for each single model.</code>? </p>\n<p>I have some trouble understanding <code>nn</code> here and why <code>36 preds</code></p>\n<p>Thanks in advance!</p>",
      "rawMarkdown": "Congratulations @shujun717 ! Cloud you please give more details about `For instance, because I have 6 folds of nn and xgb, this way i had 6x6=36 preds to avg for each single model.`? \n\nI have some trouble understanding `nn` here and why `36 preds`\n\nThanks in advance!",
      "replies": [
        {
          "id": 1911737,
          "postDate": "2022-08-24T09:19:38.260Z",
          "content": "<p><a href=\"https://www.kaggle.com/fankaixie\" target=\"_blank\">@fankaixie</a> Thansk and congrats on getting gold! Sorry about the confusion. I changed the text in question to</p>\n<pre><code>For instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way i have 6x6=36 preds to avg for each single model.\n</code></pre>\n<p>What I did in the beginning was average the neural network predictions and then use those predictions to generate features for the stacking models (1x6); however, later I found that it was better to get features for each fold model and use all folds of xgb models to make predictions for that fold (thus 6x6).</p>\n<p>LMK if this clears it up! </p>",
          "rawMarkdown": "@fankaixie Thansk and congrats on getting gold! Sorry about the confusion. I changed the text in question to\n```\nFor instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way i have 6x6=36 preds to avg for each single model.\n```\nWhat I did in the beginning was average the neural network predictions and then use those predictions to generate features for the stacking models (1x6); however, later I found that it was better to get features for each fold model and use all folds of xgb models to make predictions for that fold (thus 6x6).\n\nLMK if this clears it up! \n"
        },
        {
          "id": 1911778,
          "postDate": "2022-08-24T09:57:52.270Z",
          "content": "<p>Got it! Very clear~<br>\nThanks for your reply!</p>",
          "rawMarkdown": "Got it! Very clear~\nThanks for your reply!"
        }
      ]
    },
    {
      "id": 1911359,
      "postDate": "2022-08-24T03:47:24.297Z",
      "content": "<p>Congratulations and thanks for this great sharing, for StartifiedGroupKFold, which field did you Stratify on?</p>",
      "rawMarkdown": "Congratulations and thanks for this great sharing, for StartifiedGroupKFold, which field did you Stratify on?",
      "replies": [
        {
          "id": 1911364,
          "postDate": "2022-08-24T03:49:33.413Z",
          "content": "<p>Effectiveness I guess? Like this: <a href=\"https://www.kaggle.com/code/tanlikesmath/here-have-some-folds\" target=\"_blank\">https://www.kaggle.com/code/tanlikesmath/here-have-some-folds</a></p>",
          "rawMarkdown": "Effectiveness I guess? Like this: https://www.kaggle.com/code/tanlikesmath/here-have-some-folds"
        },
        {
          "id": 1911377,
          "postDate": "2022-08-24T04:05:11.537Z",
          "content": "<p><a href=\"https://www.kaggle.com/goldenlock\" target=\"_blank\">@goldenlock</a> Thanks for the kind words and congrats on getting gold to you as well. My split is similar to what Zacchaues mentioned, but I simply stratefied on effectiveness instead:</p>\n<pre><code>cv = StratifiedGroupKFold(n_splits=args.nfolds,shuffle=True,random_state=2022)\nfor fold, (train_idxs, test_idxs) in enumerate(cv.split(train, train['label'], train['essay_id'])):\n    train['fold'][test_idxs]=fold\n</code></pre>",
          "rawMarkdown": "@goldenlock Thanks for the kind words and congrats on getting gold to you as well. My split is similar to what Zacchaues mentioned, but I simply stratefied on effectiveness instead:\n\n```\ncv = StratifiedGroupKFold(n_splits=args.nfolds,shuffle=True,random_state=2022)\nfor fold, (train_idxs, test_idxs) in enumerate(cv.split(train, train['label'], train['essay_id'])):\n    train['fold'][test_idxs]=fold\n``` "
        },
        {
          "id": 1911381,
          "postDate": "2022-08-24T04:11:48.683Z",
          "content": "<p>Thanks！ <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a>  <a href=\"https://www.kaggle.com/zacchaeus\" target=\"_blank\">@zacchaeus</a> </p>",
          "rawMarkdown": "Thanks！ @shujun717  @zacchaeus "
        }
      ]
    },
    {
      "id": 1911178,
      "postDate": "2022-08-24T00:52:08.227Z",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a>! Congrats on the awesome result. You mention that you trained the essays in entirety with special tokens. Could you provide a data example for an essay?</p>",
      "rawMarkdown": "Hey @shujun717! Congrats on the awesome result. You mention that you trained the essays in entirety with special tokens. Could you provide a data example for an essay?",
      "replies": [
        {
          "id": 1911204,
          "postDate": "2022-08-24T01:09:21.197Z",
          "content": "<p>Added an example in the beginning of the post. LMK if it makes sense</p>",
          "rawMarkdown": "Added an example in the beginning of the post. LMK if it makes sense",
          "votes": 1
        },
        {
          "id": 1911210,
          "postDate": "2022-08-24T01:14:01.910Z",
          "content": "<p>Yes this is great! So after this treatment for the model the original data set was condensed to the ~4K essays and ~15k with PL? Can’t recall the data shapes exactly for training? </p>",
          "rawMarkdown": "Yes this is great! So after this treatment for the model the original data set was condensed to the ~4K essays and ~15k with PL? Can’t recall the data shapes exactly for training? "
        },
        {
          "id": 1911214,
          "postDate": "2022-08-24T01:18:28.433Z",
          "content": "<p>4K essays and 11K PL since we exlcude those in the training set</p>",
          "rawMarkdown": "4K essays and 11K PL since we exlcude those in the training set",
          "votes": 1
        },
        {
          "id": 1911784,
          "postDate": "2022-08-24T10:06:06.393Z",
          "content": "<p><a href=\"https://www.kaggle.com/shujun717\" target=\"_blank\">@shujun717</a> will you be sharing your training code?</p>",
          "rawMarkdown": "@shujun717 will you be sharing your training code?"
        },
        {
          "id": 1912645,
          "postDate": "2022-08-24T22:14:49.007Z",
          "content": "<p>Yes very soon!</p>",
          "rawMarkdown": "Yes very soon!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911148,
      "postDate": "2022-08-24T00:29:47.393Z",
      "content": "<p>congratulations! amazing work :-)</p>",
      "rawMarkdown": "congratulations! amazing work :-)",
      "replies": [
        {
          "id": 1911205,
          "postDate": "2022-08-24T01:09:41.407Z",
          "content": "<p>Thanks! Congrats on getting 3rd!</p>",
          "rawMarkdown": "Thanks! Congrats on getting 3rd!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1911134,
      "postDate": "2022-08-24T00:21:18.873Z",
      "content": "<p>congrats new GM!🎉 May I ask how was your best single model?</p>",
      "rawMarkdown": "congrats new GM!🎉 May I ask how was your best single model?",
      "replies": [
        {
          "id": 1911137,
          "postDate": "2022-08-24T00:24:09.923Z",
          "content": "<p>Thanks! Congrats on your gold medal as well!</p>",
          "rawMarkdown": "Thanks! Congrats on your gold medal as well!",
          "votes": 1
        },
        {
          "id": 1911147,
          "postDate": "2022-08-24T00:29:14.950Z",
          "content": "<p>Best single model was my 5th round pl deberta-v3-large with public 0.56 and private 0.558</p>",
          "rawMarkdown": "Best single model was my 5th round pl deberta-v3-large with public 0.56 and private 0.558",
          "votes": 1
        },
        {
          "id": 1911151,
          "postDate": "2022-08-24T00:31:15.117Z",
          "content": "<p>wow that's impressive! Our best was public 0.583 </p>",
          "rawMarkdown": "wow that's impressive! Our best was public 0.583 ",
          "votes": 1
        },
        {
          "id": 1911172,
          "postDate": "2022-08-24T00:47:28.393Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1912652,
      "postDate": "2022-08-24T22:25:03.367Z",
      "content": "<p>thanks for sharing</p>",
      "rawMarkdown": "thanks for sharing"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:52:03Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8703342,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/skt-2nd-place-solution-updated-with-code-notebooks"
}