{ "id": 446573, "title": "2nd Place Solution", "content": "

Inputs

\n

The data was input to the model as follows :

\n

'Think through this step by step : ' + prompt_question + [SEP] + 'Pay attention to the content and wording : ' + text + [SEP] + prompt_text

\n

Pooling Method [High Impact]

\n

Input : [TOKEN] [TOKEN] [SEP] [TOKEN] [TOKEN] [SEP] [TOKEN] [TOKEN]
\nHead Mask : [0] [0] [1] [1] [1] [0] [0] [0]

\n

Instead of using the normal attention mask created by the model tokenizer. I used a head mask that only had ones for the students' answer (text) portion of the input and zeros for all other tokens. I used the normal attention mask for the attention mask that the model consumed but I used the head mask for the mean pooling.

\n

This had the biggest impact out of all the tricks I used. It increased the CV by a huge margin in all folds, but especially for the difficult prompts : 3b9047 and 814d6b. In my opinion this was the โ€œmagicโ€ for this competition.

\n

Prompt Question Augmentation [Moderate Impact]

\n

I created 10 extra prompt questions per a prompt. I used an LLM. I asked the LLM to give me 10 variations of the prompt question. I then used this as augmentation during training. In inference, I used the default prompt question. In total I had 44 different prompt questions across all folds.

\n

Auxiliary Classes [Moderate Impact]

\n

I used auxiliary classes during the competition. These auxiliary classes were the target classes from Feedback 3.0 -
\n['cohesion','syntax','vocabulary','phraseology','grammar','conventions'].

\n

To create these labels I used models that were trained on the Feedback 3.0 data and ran the data from this competition through those models. I used only the โ€˜textโ€™ column from this competition. In doing this I produced pseudo labels to use for this competition.

\n

I used the auxiliary classes in the following way : (loss * .5) + (aux_loss * .5)

\n

The auxiliary classes were used every second step.

\n

The Feedback 3.0 competition was hosted by The Learning Agency Lab and to the best of my knowledge this is a legal technique.

\n

Max Length

\n

Models were trained on a maximum length ranging from 896-1280 during initial training. During the pseudo labelling rounds they were trained with a maximum length ranging from 1280-2048. Pseudo labels allowed the models to learn at a higher maximum length.

\n

During inference the models used 1792 for large and 2048 for base.

\n

Pseudo Labels [Moderate Impact]

\n

Once a CV of .4581 was reached across the grouped kfold I started creating pseudo labels.

\n

The pseudo labels allowed me to train deberta-v3-base effectively. Before PL, I was not able to train the base model. They also allowed me to increase the maximum length during training.

\n

PL increased the CV from .4581 to .4476

\n

The models were trained using a concatenation of the original labels and pseudo labels.

\n

Final Ensemble (PL)

\n


\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model NameTraining Max LengthInference Max LengthHeadModel CV
microsoft/deberta-v3-large20481792Mean Pooling + LSTM Layer Pooling.460
microsoft/deberta-v3-base20482048Mean Pooling + LSTM Sequence Pooling.468
OpenAssistant/reward-model-deberta-v3-large-v220481792Mean Pooling + LSTM Layer Pooling.464
microsoft/deberta-large20481792Mean Pooling + Linear.466
microsoft/deberta-v3-large12801792Mean Pooling + LSTM Sequence Pooling.461
\n



\n

\n

Did work:

\n\n

Did not work:

\n\n

Inference script : https://www.kaggle.com/code/ivanaerlic/2nd-place-solution/notebook

\n

Note : The inference script and input datasets will contain extra tricks that I haven't mentioned here, such as the inclusion of additional head for the model. Only the main points are outlined in this write-up.

\n

Also, you can find the training script for each model inside of the dataset that contains the weights.

", "comments": [ { "id": 2478947, "author_name": "CPMP", "author_url": "", "post_date": "2023-10-12T09:48:53.770000", "content": "

Well done! The selective mask is a great idea.

", "votes": 8, "replies": [] }, { "id": 2480576, "author_name": "Ranamalla Nithin Reddy", "author_url": "", "post_date": "2023-10-13T12:21:41.950000", "content": "

Congratulations @ivanaerlic on becoming GrandMaster, Great win!! Well deserved!! ๐Ÿค—

", "votes": 3, "replies": [] }, { "id": 2480326, "author_name": "Adam Montgomerie", "author_url": "", "post_date": "2023-10-13T08:50:43.127000", "content": "

Setting the mask to ignore anything other than the student summary is a really nice idea! Definitely something to try in similar future competitions.

\n

Congratulations!

", "votes": 3, "replies": [] }, { "id": 2480079, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2023-10-13T05:40:11", "content": "

Congratulations @ivanaerlic on becoming GrandMaster, Great win!! Well deserved!! ๐Ÿค—

", "votes": 3, "replies": [ { "id": 2480144, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-13T06:16:06.427000", "content": "

@nischaydnk Good seeing you here! And good luck with RSNA ๐Ÿฅ‡

", "votes": 2, "replies": [] } ] }, { "id": 2483384, "author_name": "Darek Kล‚eczek", "author_url": "", "post_date": "2023-10-15T16:49:02.090000", "content": "

Fantastic solution, worthy of the GM title :) Congrats Ivan, the solution shows your deep understanding of transformers/NLP and proves your ability to avoid shake-up, a critical skill both on Kaggle and in the real world.

", "votes": 4, "replies": [] }, { "id": 2478862, "author_name": "chumajin", "author_url": "", "post_date": "2023-10-12T08:37:35.323000", "content": "

Congratulations on your strong second place finish and also on your promotion to competition grand master.

\n

I also tried to create several variations of the original question in the prompt using LLM. However, I did not think to use it as an AUG (As a result, I did not use it).
\nAnd I also tried to use the past commonlit data or the past feedback3 data, but I could not do it as well as you did.
\nI think this is a great use of the data.

\n

Anyway, cv is too awesome. I have nothing but amazement and respect for your method. Thank you for sharing it with us!

", "votes": 4, "replies": [ { "id": 2478869, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T08:41:40.773000", "content": "

Thanks for the kind words @chumajin. It was a great competition ๐Ÿ˜€

", "votes": 2, "replies": [] } ] }, { "id": 2478878, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T08:46:24.887000", "content": "

The write-up may change from time to time. I am fixing mistakes and adding content as I go.

", "votes": 2, "replies": [] }, { "id": 2489729, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-10-20T07:05:43.673000", "content": "

Congratulations on 2nd place. Thanks for sharing details of your notebook.

", "votes": 1, "replies": [] }, { "id": 2484322, "author_name": "Yusuke Nakayama", "author_url": "", "post_date": "2023-10-16T12:00:07.037000", "content": "

Congratulations on your results, and thank you for sharing your great solution!

\n
\n

The models were trained using a concatenation of the original labels and pseudo labels.

\n
\n

Does this mean that you concatenated all the training data with original labels and the training data with pseudo labels? In other words, were there exactly the same texts with both original labels and pseudo labels in the training data?

", "votes": 2, "replies": [ { "id": 2484399, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-16T13:11:06", "content": "

Correct. 7165 x 2.

", "votes": 1, "replies": [ { "id": 2484451, "author_name": "Yusuke Nakayama", "author_url": "", "post_date": "2023-10-16T13:57:28.057000", "content": "

Thank you for your reply!

", "votes": 0, "replies": [] } ] } ] }, { "id": 2480716, "author_name": "ls", "author_url": "", "post_date": "2023-10-13T14:05:17.420000", "content": "

Congrats on win! Great solution.

", "votes": 2, "replies": [] }, { "id": 2479375, "author_name": "Reacher", "author_url": "", "post_date": "2023-10-12T15:31:54.900000", "content": "

Congrats on your promotion to competition grand master.
\nPooling method had a significant impact for us also.

", "votes": 2, "replies": [] }, { "id": 2479319, "author_name": "moth", "author_url": "", "post_date": "2023-10-12T14:56:00.127000", "content": "

Congrats @ivanaerlic for your win and for becoming competition GM.

\n

If I understood correctly your mask idea filters out (in the last layer of the transformer model) all values associated to positions which do not belong to the text summary, is this correct? For example, if the text summary goes from token position 50 to 250, then the last layer embedding of the transformer is zeroed on all tokens except for these and then the custom head is applied?

", "votes": 2, "replies": [ { "id": 2479964, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-13T03:17:09.503000", "content": "

Yes, that's correct. You can find the implementation in the inference notebook if you want to see the code.

", "votes": 1, "replies": [] } ] }, { "id": 2478955, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-10-12T09:55:36.767000", "content": "

Thanks for the clear write-up, and congrats on getting competition GM. I hope you'll be releasing the code, I need to update my deberta knowledge.

\n

And I'm gonna take a (not so) wild bet and congratulate you on 1st place as well =)

", "votes": 2, "replies": [ { "id": 2478962, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T09:59:15.477000", "content": "

Haha, thank you @theoviel. Do you want training or inference. Training would be a bit difficult for me. Inference I can publish quickly.

", "votes": 1, "replies": [ { "id": 2478969, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-10-12T10:02:12.737000", "content": "

Ideally training but inference is fine as well. Thanks ! :D

", "votes": 0, "replies": [] } ] } ] }, { "id": 2478886, "author_name": "Shreyansh Murathia", "author_url": "", "post_date": "2023-10-12T08:53:01.393000", "content": "

First of all, brilliant work! ๐Ÿ™Œ
\nCould you please explain how you determined the layer wise rate decay and which layers to freeze.

", "votes": 2, "replies": [ { "id": 2478907, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T09:08:08.070000", "content": "

I can't really tell you how to determine it, because it is based off of past experience and it changes per a competition. There aren't any hard rules for it.

\n

I can give you a few tips, though. In general I find it best to distribute the learning rate across the model into four sections (sometimes 5). For this competition I chose to freeze the first 8, 2e-6 for 8 to 16, 1e-5 for 16 to 20, 2e-5 from 20 to 22 and 5e-5 from 22 to 24. This is with LSTM pooling across model layers and mean pooling across sequences, if you are pooling across model sequences with an LSTM then I tend not to go past 2e-5.

\n

These are the deberta-v3-large numbers. It is different per a model.

", "votes": 2, "replies": [ { "id": 2479074, "author_name": "BUUMOO", "author_url": "", "post_date": "2023-10-12T11:43:22.463000", "content": "

Very valuable experience! Thanks for sharing.
\nIt seems we need to maintain the model's original low-level feature extraction capability while adjust the high-level feature extractions according to the specific downstream task.

", "votes": 1, "replies": [] } ] }, { "id": 2478911, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T09:12:18.273000", "content": "

Also it will change based on the batch size and max length.

\n

As max length goes up - decrease the learning rate.
\nAs batch size goes up - increase the learning rate

", "votes": 2, "replies": [ { "id": 2478946, "author_name": "Shreyansh Murathia", "author_url": "", "post_date": "2023-10-12T09:48:20.790000", "content": "

Thanks for the information. It's always interesting to learn how professionals approach such tasks.

", "votes": 0, "replies": [] }, { "id": 2478952, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T09:53:38.720000", "content": "

You are welcome. I hope to become a professional soon.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2478833, "author_name": "AK", "author_url": "", "post_date": "2023-10-12T08:26:38.840000", "content": "

Great finish @ivanaerlic and congrats on achieving GM๐Ÿ˜€
\nCan you provide the snippet on the custom head mask code you used if possible

", "votes": 2, "replies": [ { "id": 2478845, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T08:29:30.110000", "content": "
 :\n    def :\n        super.\n        self.hidden_size = hidden_size\n        self.cat_size = hidden_size*\n\n        self.sequence_pooler = utilities. \n        self.rnn = utilities.   \n\n        self.head = nn.) \n        self.aux_head = nn.)) \n\n\n    def forward(self, x, mask): \n\n        x = torch.cat(x.hidden_states, dim=-)  \n\n        hidden_mask = mask.unsqueeze.expand(x.size).\n        x = xhidden_mask \n\n        x = self.rnn(x) \n        x = self.sequence \n\n        x_preds = self.head(x)  \n        aux = self.aux  \n\n        return x_preds, aux\n
", "votes": 6, "replies": [ { "id": 2478846, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T08:30:34.247000", "content": "

Here is how the head mask is created in the dataset class :

\n
        head_mask = \n        use_full = False\n         token  tokenized_dict:\n\n             token == CFG:\n                use_full = not use_full  \n\n            head_mask(  use_full  .) \n
", "votes": 2, "replies": [] }, { "id": 2478851, "author_name": "AK", "author_url": "", "post_date": "2023-10-12T08:32:05.880000", "content": "

Thanks @ivanaerlic ๐Ÿ˜€

", "votes": 2, "replies": [] } ] } ] }, { "id": 2590473, "author_name": "BJUT4Science", "author_url": "", "post_date": "2024-01-07T08:09:36.540000", "content": "

Nice work. Congratulations!

", "votes": 0, "replies": [] }, { "id": 2484103, "author_name": "Fayรงal Baki", "author_url": "", "post_date": "2023-10-16T08:19:34.387000", "content": "

Thank you for the very concise and precise explanations ! I learned many things, and got many ideas that I will try to reuse for sure !

\n

Congratulations !

", "votes": 0, "replies": [] }, { "id": 2480651, "author_name": "Earth", "author_url": "", "post_date": "2023-10-13T13:06:44.267000", "content": "

What do AWP and SWA means and how to implement them? Sorry, I'm new here

", "votes": 0, "replies": [ { "id": 2482479, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-15T04:37:18.570000", "content": "

AWP : https://www.kaggle.com/code/itsuki9180/introducing-adversarial-weight-perturbation-awp
\nSWA : https://www.kaggle.com/competitions/google-quest-challenge/discussion/119371

\n

Here are some links. It's difficult to explain these two techniques in one comment. You will have to read through the code.

", "votes": 1, "replies": [ { "id": 2484367, "author_name": "Earth", "author_url": "", "post_date": "2023-10-16T12:41:19.787000", "content": "

Thank you so much

", "votes": 0, "replies": [] } ] } ] }, { "id": 2480364, "author_name": "Xiang Huang", "author_url": "", "post_date": "2023-10-13T09:36:07.333000", "content": "

Congratulations and thanks for your detail writeup!

", "votes": 0, "replies": [] }, { "id": 2479134, "author_name": "Igor Litvin", "author_url": "", "post_date": "2023-10-12T12:36:31.950000", "content": "

Thanks for the information. Your maximum length (2048) is the before tokenizer (letters) or after (encoded text)? I used 412 after tokenizer. I mean:
\ntokenizer(โ€ฆโ€ฆ.. max_length=412)

\n

Is it too short? What do you thing? I am trying to figure out where is my problem.

\n

Thanks.

", "votes": 0, "replies": [ { "id": 2479958, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-13T03:09:03.843000", "content": "

Yeah that's probably too short. As long as you are using the right pooling method, which is zeroing out the prompt_text and prompt_question portions you should be good to do higher max lengths of 1024+.

", "votes": 0, "replies": [ { "id": 2480389, "author_name": "Igor Litvin", "author_url": "", "post_date": "2023-10-13T09:44:15.493000", "content": "

Thanks. It was diffenetly my huge mistake because of my models and trainig approaches were quite close to winning solutions.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2479031, "author_name": "RB", "author_url": "", "post_date": "2023-10-12T11:01:47.290000", "content": "

Congratulations on solo gold and becoming GM and thank you for sharing your code and writeup.

\n

It seems your models are averaged , no stacking .. right . Also you didn't have any CV / public LB correlation problem - could this be because 5 full text models (large+base) made your ensemble stable enough for public LB ?

", "votes": 0, "replies": [ { "id": 2479037, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T11:11:52.150000", "content": "

I think the reason why my CV had good correlation with the LB was because of using the prompt text and the head mask, while not using feature engineering at all. I only started using the base model towards the end when I implemented PL, so that would not have been the reason. I had a stable CV and LB correlation before that.

\n

There were still things which didn't correlate well, one of them was increasing the max length. This increased LB drastically while not doing much for CV.

\n

One of the reasons why people were experiencing an unstable LB may have been due to feature engineering. These are hard number values and are less generalizable than a proper understanding of language.

", "votes": 1, "replies": [ { "id": 2479073, "author_name": "RB", "author_url": "", "post_date": "2023-10-12T11:42:55.223000", "content": "

Our experience was that stack seemed more stable than individual models but then we only had 10 features in stack.
\nUp until last week our inference was on p100 so we only had single model with prompt_text and many models without prompt_text.

", "votes": 1, "replies": [] }, { "id": 2479128, "author_name": "Ivan Aerlic", "author_url": "", "post_date": "2023-10-12T12:34:44.273000", "content": "

Thanks for sharing, this give me something to try out next time.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2479111, "author_name": "", "author_url": "", "post_date": "2023-10-12T12:14:25.053000", "content": "", "votes": -2, "replies": [] } ], "comment_count": 44, "messages": [ { "id": 2478816, "postDate": "2023-10-12T08:17:05.680Z", "content": "

Inputs

\n

The data was input to the model as follows :

\n

'Think through this step by step : ' + prompt_question + [SEP] + 'Pay attention to the content and wording : ' + text + [SEP] + prompt_text

\n

Pooling Method [High Impact]

\n

Input : [TOKEN] [TOKEN] [SEP] [TOKEN] [TOKEN] [SEP] [TOKEN] [TOKEN]
\nHead Mask : [0] [0] [1] [1] [1] [0] [0] [0]

\n

Instead of using the normal attention mask created by the model tokenizer. I used a head mask that only had ones for the students' answer (text) portion of the input and zeros for all other tokens. I used the normal attention mask for the attention mask that the model consumed but I used the head mask for the mean pooling.

\n

This had the biggest impact out of all the tricks I used. It increased the CV by a huge margin in all folds, but especially for the difficult prompts : 3b9047 and 814d6b. In my opinion this was the โ€œmagicโ€ for this competition.

\n

Prompt Question Augmentation [Moderate Impact]

\n

I created 10 extra prompt questions per a prompt. I used an LLM. I asked the LLM to give me 10 variations of the prompt question. I then used this as augmentation during training. In inference, I used the default prompt question. In total I had 44 different prompt questions across all folds.

\n

Auxiliary Classes [Moderate Impact]

\n

I used auxiliary classes during the competition. These auxiliary classes were the target classes from Feedback 3.0 -
\n['cohesion','syntax','vocabulary','phraseology','grammar','conventions'].

\n

To create these labels I used models that were trained on the Feedback 3.0 data and ran the data from this competition through those models. I used only the โ€˜textโ€™ column from this competition. In doing this I produced pseudo labels to use for this competition.

\n

I used the auxiliary classes in the following way : (loss * .5) + (aux_loss * .5)

\n

The auxiliary classes were used every second step.

\n

The Feedback 3.0 competition was hosted by The Learning Agency Lab and to the best of my knowledge this is a legal technique.

\n

Max Length

\n

Models were trained on a maximum length ranging from 896-1280 during initial training. During the pseudo labelling rounds they were trained with a maximum length ranging from 1280-2048. Pseudo labels allowed the models to learn at a higher maximum length.

\n

During inference the models used 1792 for large and 2048 for base.

\n

Pseudo Labels [Moderate Impact]

\n

Once a CV of .4581 was reached across the grouped kfold I started creating pseudo labels.

\n

The pseudo labels allowed me to train deberta-v3-base effectively. Before PL, I was not able to train the base model. They also allowed me to increase the maximum length during training.

\n

PL increased the CV from .4581 to .4476

\n

The models were trained using a concatenation of the original labels and pseudo labels.

\n

Final Ensemble (PL)

\n


\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model NameTraining Max LengthInference Max LengthHeadModel CV
microsoft/deberta-v3-large20481792Mean Pooling + LSTM Layer Pooling.460
microsoft/deberta-v3-base20482048Mean Pooling + LSTM Sequence Pooling.468
OpenAssistant/reward-model-deberta-v3-large-v220481792Mean Pooling + LSTM Layer Pooling.464
microsoft/deberta-large20481792Mean Pooling + Linear.466
microsoft/deberta-v3-large12801792Mean Pooling + LSTM Sequence Pooling.461
\n



\n

\n

Did work:

\n\n

Did not work:

\n\n

Inference script : https://www.kaggle.com/code/ivanaerlic/2nd-place-solution/notebook

\n

Note : The inference script and input datasets will contain extra tricks that I haven't mentioned here, such as the inclusion of additional head for the model. Only the main points are outlined in this write-up.

\n

Also, you can find the training script for each model inside of the dataset that contains the weights.

", "rawMarkdown": "**Inputs**\n\nThe data was input to the model as follows : \n\n'Think through this step by step : ' + prompt_question + [SEP] + 'Pay attention to the content and wording : ' + text + [SEP] + prompt_text\n\n**Pooling Method [High Impact]**\n\n**Input :** [TOKEN] [TOKEN] [SEP] [TOKEN] [TOKEN] [SEP] [TOKEN] [TOKEN]\n**Head Mask :** [0] [0] [1] [1] [1] [0] [0] [0]\n\nInstead of using the normal attention mask created by the model tokenizer. I used a head mask that only had ones for the students' answer (text) portion of the input and zeros for all other tokens. I used the normal attention mask for the attention mask that the model consumed but I used the head mask for the mean pooling.\n\nThis had the biggest impact out of all the tricks I used. It increased the CV by a huge margin in all folds, but especially for the difficult prompts : 3b9047 and 814d6b. In my opinion this was the โ€œmagicโ€ for this competition.\n\n**Prompt Question Augmentation [Moderate Impact]**\n\nI created 10 extra prompt questions per a prompt. I used an LLM. I asked the LLM to give me 10 variations of the prompt question. I then used this as augmentation during training. In inference, I used the default prompt question. In total I had 44 different prompt questions across all folds.\n\n**Auxiliary Classes [Moderate Impact]**\n\nI used auxiliary classes during the competition. These auxiliary classes were the target classes from Feedback 3.0 - \n['cohesion','syntax','vocabulary','phraseology','grammar','conventions'].\n\nTo create these labels I used models that were trained on the Feedback 3.0 data and ran the data from this competition through those models. I used only the โ€˜textโ€™ column from this competition. In doing this I produced pseudo labels to use for this competition.\n\nI used the auxiliary classes in the following way : (loss * .5) + (aux_loss * .5)\n\nThe auxiliary classes were used every second step.\n\nThe Feedback 3.0 competition was hosted by The Learning Agency Lab and to the best of my knowledge this is a legal technique.\n\n**Max Length**\n\nModels were trained on a maximum length ranging from 896-1280 during initial training. During the pseudo labelling rounds they were trained with a maximum length ranging from 1280-2048. Pseudo labels allowed the models to learn at a higher maximum length.\n\nDuring inference the models used 1792 for large and 2048 for base.\n\n**Pseudo Labels [Moderate Impact]**\n\nOnce a CV of .4581 was reached across the grouped kfold I started creating pseudo labels. \n\nThe pseudo labels allowed me to train deberta-v3-base effectively. Before PL, I was not able to train the base model. They also allowed me to increase the maximum length during training.\n\nPL increased the CV from .4581 to .4476\n\nThe models were trained using a concatenation of the original labels and pseudo labels.\n\n**Final Ensemble (PL)**\n\n
\n\n| Model Name | Training Max Length | Inference Max Length | Head | Model CV |\n| --- | --- |\n| microsoft/deberta-v3-large | 2048 | 1792 | Mean Pooling + LSTM Layer Pooling | .460 |\n| microsoft/deberta-v3-base | 2048 | 2048 | Mean Pooling + LSTM Sequence Pooling | .468 |\n| OpenAssistant/reward-model-deberta-v3-large-v2 | 2048 | 1792 | Mean Pooling + LSTM Layer Pooling | .464 |\n| microsoft/deberta-large | 2048 | 1792 | Mean Pooling + Linear | .466 |\n| microsoft/deberta-v3-large | 1280 | 1792 | Mean Pooling + LSTM Sequence Pooling | .461 |\n\n
\n
\n\n**Did work:**\n\n- Layer wise learning rate decay\n- Freezing layers (bottom 8)\n- LSTM layer pooling\n- LSTM Sequence pooling\n- Turn off dropout in transformer backbone\n- Multisample dropout in head\n\n**Did not work:**\n\n- AWP\n- SWA\n\n**Inference script :** https://www.kaggle.com/code/ivanaerlic/2nd-place-solution/notebook\n\n**Note :** The inference script and input datasets will contain extra tricks that I haven't mentioned here, such as the inclusion of additional head for the model. Only the main points are outlined in this write-up.\n\nAlso, you can find the training script for each model inside of the dataset that contains the weights.", "votes": 141 }, { "id": 2478947, "postDate": "2023-10-12T09:48:53.770Z", "content": "

Well done! The selective mask is a great idea.

", "rawMarkdown": "Well done! The selective mask is a great idea. ", "votes": 8 }, { "id": 2480576, "postDate": "2023-10-13T12:21:41.950Z", "content": "

Congratulations @ivanaerlic on becoming GrandMaster, Great win!! Well deserved!! ๐Ÿค—

", "rawMarkdown": "Congratulations @ivanaerlic on becoming GrandMaster, Great win!! Well deserved!! ๐Ÿค—", "votes": 3 }, { "id": 2480326, "postDate": "2023-10-13T08:50:43.127Z", "content": "

Setting the mask to ignore anything other than the student summary is a really nice idea! Definitely something to try in similar future competitions.

\n

Congratulations!

", "rawMarkdown": "Setting the mask to ignore anything other than the student summary is a really nice idea! Definitely something to try in similar future competitions.\n\nCongratulations!", "votes": 3 }, { "id": 2480079, "postDate": "2023-10-13T05:40:11Z", "content": "

Congratulations @ivanaerlic on becoming GrandMaster, Great win!! Well deserved!! ๐Ÿค—

", "rawMarkdown": "Congratulations @ivanaerlic on becoming GrandMaster, Great win!! Well deserved!! ๐Ÿค—", "votes": 3, "replies": [ { "id": 2480144, "postDate": "2023-10-13T06:16:06.427Z", "content": "

@nischaydnk Good seeing you here! And good luck with RSNA ๐Ÿฅ‡

", "rawMarkdown": "@nischaydnk Good seeing you here! And good luck with RSNA ๐Ÿฅ‡", "votes": 2 } ] }, { "id": 2483384, "postDate": "2023-10-15T16:49:02.090Z", "content": "

Fantastic solution, worthy of the GM title :) Congrats Ivan, the solution shows your deep understanding of transformers/NLP and proves your ability to avoid shake-up, a critical skill both on Kaggle and in the real world.

", "rawMarkdown": "Fantastic solution, worthy of the GM title :) Congrats Ivan, the solution shows your deep understanding of transformers/NLP and proves your ability to avoid shake-up, a critical skill both on Kaggle and in the real world. ", "votes": 4 }, { "id": 2478862, "postDate": "2023-10-12T08:37:35.323Z", "content": "

Congratulations on your strong second place finish and also on your promotion to competition grand master.

\n

I also tried to create several variations of the original question in the prompt using LLM. However, I did not think to use it as an AUG (As a result, I did not use it).
\nAnd I also tried to use the past commonlit data or the past feedback3 data, but I could not do it as well as you did.
\nI think this is a great use of the data.

\n

Anyway, cv is too awesome. I have nothing but amazement and respect for your method. Thank you for sharing it with us!

", "rawMarkdown": "Congratulations on your strong second place finish and also on your promotion to competition grand master.\n\nI also tried to create several variations of the original question in the prompt using LLM. However, I did not think to use it as an AUG (As a result, I did not use it). \nAnd I also tried to use the past commonlit data or the past feedback3 data, but I could not do it as well as you did.\nI think this is a great use of the data.\n\nAnyway, cv is too awesome. I have nothing but amazement and respect for your method. Thank you for sharing it with us!", "votes": 4, "replies": [ { "id": 2478869, "postDate": "2023-10-12T08:41:40.773Z", "content": "

Thanks for the kind words @chumajin. It was a great competition ๐Ÿ˜€

", "rawMarkdown": "Thanks for the kind words @chumajin. It was a great competition ๐Ÿ˜€", "votes": 2 } ] }, { "id": 2478878, "postDate": "2023-10-12T08:46:24.887Z", "content": "

The write-up may change from time to time. I am fixing mistakes and adding content as I go.

", "rawMarkdown": "The write-up may change from time to time. I am fixing mistakes and adding content as I go.", "votes": 2 }, { "id": 2489729, "postDate": "2023-10-20T07:05:43.673Z", "content": "

Congratulations on 2nd place. Thanks for sharing details of your notebook.

", "rawMarkdown": "Congratulations on 2nd place. Thanks for sharing details of your notebook. ", "votes": 1 }, { "id": 2484322, "postDate": "2023-10-16T12:00:07.037Z", "content": "

Congratulations on your results, and thank you for sharing your great solution!

\n
\n

The models were trained using a concatenation of the original labels and pseudo labels.

\n
\n

Does this mean that you concatenated all the training data with original labels and the training data with pseudo labels? In other words, were there exactly the same texts with both original labels and pseudo labels in the training data?

", "rawMarkdown": "Congratulations on your results, and thank you for sharing your great solution!\n\n>The models were trained using a concatenation of the original labels and pseudo labels. \n\nDoes this mean that you concatenated all the training data with original labels and the training data with pseudo labels? In other words, were there exactly the same texts with both original labels and pseudo labels in the training data? ", "votes": 2, "replies": [ { "id": 2484399, "postDate": "2023-10-16T13:11:06Z", "content": "

Correct. 7165 x 2.

", "rawMarkdown": "Correct. 7165 x 2.", "votes": 1, "replies": [ { "id": 2484451, "postDate": "2023-10-16T13:57:28.057Z", "content": "

Thank you for your reply!

", "rawMarkdown": "Thank you for your reply!" } ] } ] }, { "id": 2480716, "postDate": "2023-10-13T14:05:17.420Z", "content": "

Congrats on win! Great solution.

", "rawMarkdown": "Congrats on win! Great solution.", "votes": 2 }, { "id": 2479375, "postDate": "2023-10-12T15:31:54.900Z", "content": "

Congrats on your promotion to competition grand master.
\nPooling method had a significant impact for us also.

", "rawMarkdown": "Congrats on your promotion to competition grand master.\nPooling method had a significant impact for us also.", "votes": 2 }, { "id": 2479319, "postDate": "2023-10-12T14:56:00.127Z", "content": "

Congrats @ivanaerlic for your win and for becoming competition GM.

\n

If I understood correctly your mask idea filters out (in the last layer of the transformer model) all values associated to positions which do not belong to the text summary, is this correct? For example, if the text summary goes from token position 50 to 250, then the last layer embedding of the transformer is zeroed on all tokens except for these and then the custom head is applied?

", "rawMarkdown": "Congrats @ivanaerlic for your win and for becoming competition GM.\n\nIf I understood correctly your mask idea filters out (in the last layer of the transformer model) all values associated to positions which do not belong to the text summary, is this correct? For example, if the text summary goes from token position 50 to 250, then the last layer embedding of the transformer is zeroed on all tokens except for these and then the custom head is applied? ", "votes": 2, "replies": [ { "id": 2479964, "postDate": "2023-10-13T03:17:09.503Z", "content": "

Yes, that's correct. You can find the implementation in the inference notebook if you want to see the code.

", "rawMarkdown": "Yes, that's correct. You can find the implementation in the inference notebook if you want to see the code.", "votes": 1 } ] }, { "id": 2478955, "postDate": "2023-10-12T09:55:36.767Z", "content": "

Thanks for the clear write-up, and congrats on getting competition GM. I hope you'll be releasing the code, I need to update my deberta knowledge.

\n

And I'm gonna take a (not so) wild bet and congratulate you on 1st place as well =)

", "rawMarkdown": "Thanks for the clear write-up, and congrats on getting competition GM. I hope you'll be releasing the code, I need to update my deberta knowledge.\n\nAnd I'm gonna take a (not so) wild bet and congratulate you on 1st place as well =)", "votes": 2, "replies": [ { "id": 2478962, "postDate": "2023-10-12T09:59:15.477Z", "content": "

Haha, thank you @theoviel. Do you want training or inference. Training would be a bit difficult for me. Inference I can publish quickly.

", "rawMarkdown": "Haha, thank you @theoviel. Do you want training or inference. Training would be a bit difficult for me. Inference I can publish quickly.", "votes": 1, "replies": [ { "id": 2478969, "postDate": "2023-10-12T10:02:12.737Z", "content": "

Ideally training but inference is fine as well. Thanks ! :D

", "rawMarkdown": "Ideally training but inference is fine as well. Thanks ! :D " } ] } ] }, { "id": 2478886, "postDate": "2023-10-12T08:53:01.393Z", "content": "

First of all, brilliant work! ๐Ÿ™Œ
\nCould you please explain how you determined the layer wise rate decay and which layers to freeze.

", "rawMarkdown": "First of all, brilliant work! ๐Ÿ™Œ \nCould you please explain how you determined the layer wise rate decay and which layers to freeze.", "votes": 2, "replies": [ { "id": 2478907, "postDate": "2023-10-12T09:08:08.070Z", "content": "

I can't really tell you how to determine it, because it is based off of past experience and it changes per a competition. There aren't any hard rules for it.

\n

I can give you a few tips, though. In general I find it best to distribute the learning rate across the model into four sections (sometimes 5). For this competition I chose to freeze the first 8, 2e-6 for 8 to 16, 1e-5 for 16 to 20, 2e-5 from 20 to 22 and 5e-5 from 22 to 24. This is with LSTM pooling across model layers and mean pooling across sequences, if you are pooling across model sequences with an LSTM then I tend not to go past 2e-5.

\n

These are the deberta-v3-large numbers. It is different per a model.

", "rawMarkdown": "I can't really tell you how to determine it, because it is based off of past experience and it changes per a competition. There aren't any hard rules for it.\n\nI can give you a few tips, though. In general I find it best to distribute the learning rate across the model into four sections (sometimes 5). For this competition I chose to freeze the first 8, 2e-6 for 8 to 16, 1e-5 for 16 to 20, 2e-5 from 20 to 22 and 5e-5 from 22 to 24. This is with LSTM pooling across model layers and mean pooling across sequences, if you are pooling across model sequences with an LSTM then I tend not to go past 2e-5.\n\nThese are the deberta-v3-large numbers. It is different per a model.\n", "votes": 2, "replies": [ { "id": 2479074, "postDate": "2023-10-12T11:43:22.463Z", "content": "

Very valuable experience! Thanks for sharing.
\nIt seems we need to maintain the model's original low-level feature extraction capability while adjust the high-level feature extractions according to the specific downstream task.

", "rawMarkdown": "Very valuable experience! Thanks for sharing.\nIt seems we need to maintain the model's original low-level feature extraction capability while adjust the high-level feature extractions according to the specific downstream task.", "votes": 1 } ] }, { "id": 2478911, "postDate": "2023-10-12T09:12:18.273Z", "content": "

Also it will change based on the batch size and max length.

\n

As max length goes up - decrease the learning rate.
\nAs batch size goes up - increase the learning rate

", "rawMarkdown": "Also it will change based on the batch size and max length.\n\nAs max length goes up - decrease the learning rate.\nAs batch size goes up - increase the learning rate", "votes": 2, "replies": [ { "id": 2478946, "postDate": "2023-10-12T09:48:20.790Z", "content": "

Thanks for the information. It's always interesting to learn how professionals approach such tasks.

", "rawMarkdown": "Thanks for the information. It's always interesting to learn how professionals approach such tasks. " }, { "id": 2478952, "postDate": "2023-10-12T09:53:38.720Z", "content": "

You are welcome. I hope to become a professional soon.

", "rawMarkdown": "You are welcome. I hope to become a professional soon. ", "votes": 1 } ] } ] }, { "id": 2478833, "postDate": "2023-10-12T08:26:38.840Z", "content": "

Great finish @ivanaerlic and congrats on achieving GM๐Ÿ˜€
\nCan you provide the snippet on the custom head mask code you used if possible

", "rawMarkdown": "Great finish @ivanaerlic and congrats on achieving GM๐Ÿ˜€\nCan you provide the snippet on the custom head mask code you used if possible", "votes": 2, "replies": [ { "id": 2478845, "postDate": "2023-10-12T08:29:30.110Z", "content": "
 :\n    def :\n        super.\n        self.hidden_size = hidden_size\n        self.cat_size = hidden_size*\n\n        self.sequence_pooler = utilities. \n        self.rnn = utilities.   \n\n        self.head = nn.) \n        self.aux_head = nn.)) \n\n\n    def forward(self, x, mask): \n\n        x = torch.cat(x.hidden_states, dim=-)  \n\n        hidden_mask = mask.unsqueeze.expand(x.size).\n        x = xhidden_mask \n\n        x = self.rnn(x) \n        x = self.sequence \n\n        x_preds = self.head(x)  \n        aux = self.aux  \n\n        return x_preds, aux\n
", "rawMarkdown": "```\nclass Head(nn.Module):\n def __init__(self, hidden_size):\n super().__init__()\n self.hidden_size = hidden_size\n self.cat_size = hidden_size*2\n\n self.sequence_pooler = utilities.MeanPooling(.0) \n self.rnn = utilities.Bi_RNN_FOUT(self.cat_size, self.cat_size//2) \n \n self.head = nn.Sequential(nn.Linear(self.cat_size, CFG.n_classes)) \n self.aux_head = nn.Sequential(nn.Linear(self.cat_size, len(CFG.aux_classes))) \n\n \n def forward(self, x, mask): \n \n x = torch.cat(x.hidden_states[-2:], dim=-1) \n \n hidden_mask = mask.unsqueeze(-1).expand(x.size()).float()\n x = x * hidden_mask \n \n x = self.rnn(x) \n x = self.sequence_pooler(x, mask) \n\n x_preds = self.head(x) \n aux = self.aux_head(x) \n\n return x_preds, aux\n```", "votes": 6, "replies": [ { "id": 2478846, "postDate": "2023-10-12T08:30:34.247Z", "content": "

Here is how the head mask is created in the dataset class :

\n
        head_mask = \n        use_full = False\n         token  tokenized_dict:\n\n             token == CFG:\n                use_full = not use_full  \n\n            head_mask(  use_full  .) \n
", "rawMarkdown": "Here is how the head mask is created in the dataset class :\n\n```\n head_mask = []\n use_full = False\n for token in tokenized_dict.input_ids:\n \n if token == CFG.tokenizer.sep_token_id:\n use_full = not use_full \n\n head_mask.append(1 if use_full else .0) \n```", "votes": 2 }, { "id": 2478851, "postDate": "2023-10-12T08:32:05.880Z", "content": "

Thanks @ivanaerlic ๐Ÿ˜€

", "rawMarkdown": "Thanks @ivanaerlic ๐Ÿ˜€", "votes": 2 } ] } ] }, { "id": 2590473, "postDate": "2024-01-07T08:09:36.540Z", "content": "

Nice work. Congratulations!

", "rawMarkdown": "Nice work. Congratulations!" }, { "id": 2484103, "postDate": "2023-10-16T08:19:34.387Z", "content": "

Thank you for the very concise and precise explanations ! I learned many things, and got many ideas that I will try to reuse for sure !

\n

Congratulations !

", "rawMarkdown": "Thank you for the very concise and precise explanations ! I learned many things, and got many ideas that I will try to reuse for sure !\n\nCongratulations !" }, { "id": 2480651, "postDate": "2023-10-13T13:06:44.267Z", "content": "

What do AWP and SWA means and how to implement them? Sorry, I'm new here

", "rawMarkdown": "What do AWP and SWA means and how to implement them? Sorry, I'm new here", "replies": [ { "id": 2482479, "postDate": "2023-10-15T04:37:18.570Z", "content": "

AWP : https://www.kaggle.com/code/itsuki9180/introducing-adversarial-weight-perturbation-awp
\nSWA : https://www.kaggle.com/competitions/google-quest-challenge/discussion/119371

\n

Here are some links. It's difficult to explain these two techniques in one comment. You will have to read through the code.

", "rawMarkdown": "AWP : https://www.kaggle.com/code/itsuki9180/introducing-adversarial-weight-perturbation-awp\nSWA : https://www.kaggle.com/competitions/google-quest-challenge/discussion/119371\n\nHere are some links. It's difficult to explain these two techniques in one comment. You will have to read through the code. ", "votes": 1, "replies": [ { "id": 2484367, "postDate": "2023-10-16T12:41:19.787Z", "content": "

Thank you so much

", "rawMarkdown": "Thank you so much" } ] } ] }, { "id": 2480364, "postDate": "2023-10-13T09:36:07.333Z", "content": "

Congratulations and thanks for your detail writeup!

", "rawMarkdown": "Congratulations and thanks for your detail writeup!" }, { "id": 2479134, "postDate": "2023-10-12T12:36:31.950Z", "content": "

Thanks for the information. Your maximum length (2048) is the before tokenizer (letters) or after (encoded text)? I used 412 after tokenizer. I mean:
\ntokenizer(โ€ฆโ€ฆ.. max_length=412)

\n

Is it too short? What do you thing? I am trying to figure out where is my problem.

\n

Thanks.

", "rawMarkdown": "Thanks for the information. Your maximum length (2048) is the before tokenizer (letters) or after (encoded text)? I used 412 after tokenizer. I mean:\ntokenizer(........ max_length=412)\n\nIs it too short? What do you thing? I am trying to figure out where is my problem.\n\nThanks.", "replies": [ { "id": 2479958, "postDate": "2023-10-13T03:09:03.843Z", "content": "

Yeah that's probably too short. As long as you are using the right pooling method, which is zeroing out the prompt_text and prompt_question portions you should be good to do higher max lengths of 1024+.

", "rawMarkdown": "Yeah that's probably too short. As long as you are using the right pooling method, which is zeroing out the prompt_text and prompt_question portions you should be good to do higher max lengths of 1024+. ", "replies": [ { "id": 2480389, "postDate": "2023-10-13T09:44:15.493Z", "content": "

Thanks. It was diffenetly my huge mistake because of my models and trainig approaches were quite close to winning solutions.

", "rawMarkdown": "Thanks. It was diffenetly my huge mistake because of my models and trainig approaches were quite close to winning solutions. " } ] } ] }, { "id": 2479031, "postDate": "2023-10-12T11:01:47.290Z", "content": "

Congratulations on solo gold and becoming GM and thank you for sharing your code and writeup.

\n

It seems your models are averaged , no stacking .. right . Also you didn't have any CV / public LB correlation problem - could this be because 5 full text models (large+base) made your ensemble stable enough for public LB ?

", "rawMarkdown": "Congratulations on solo gold and becoming GM and thank you for sharing your code and writeup. \n\nIt seems your models are averaged , no stacking .. right . Also you didn't have any CV / public LB correlation problem - could this be because 5 full text models (large+base) made your ensemble stable enough for public LB ?", "replies": [ { "id": 2479037, "postDate": "2023-10-12T11:11:52.150Z", "content": "

I think the reason why my CV had good correlation with the LB was because of using the prompt text and the head mask, while not using feature engineering at all. I only started using the base model towards the end when I implemented PL, so that would not have been the reason. I had a stable CV and LB correlation before that.

\n

There were still things which didn't correlate well, one of them was increasing the max length. This increased LB drastically while not doing much for CV.

\n

One of the reasons why people were experiencing an unstable LB may have been due to feature engineering. These are hard number values and are less generalizable than a proper understanding of language.

", "rawMarkdown": "I think the reason why my CV had good correlation with the LB was because of using the prompt text and the head mask, while not using feature engineering at all. I only started using the base model towards the end when I implemented PL, so that would not have been the reason. I had a stable CV and LB correlation before that.\n\nThere were still things which didn't correlate well, one of them was increasing the max length. This increased LB drastically while not doing much for CV.\n\nOne of the reasons why people were experiencing an unstable LB may have been due to feature engineering. These are hard number values and are less generalizable than a proper understanding of language.", "votes": 1, "replies": [ { "id": 2479073, "postDate": "2023-10-12T11:42:55.223Z", "content": "

Our experience was that stack seemed more stable than individual models but then we only had 10 features in stack.
\nUp until last week our inference was on p100 so we only had single model with prompt_text and many models without prompt_text.

", "rawMarkdown": "Our experience was that stack seemed more stable than individual models but then we only had 10 features in stack. \nUp until last week our inference was on p100 so we only had single model with `prompt_text` and many models without `prompt_text`.", "votes": 1 }, { "id": 2479128, "postDate": "2023-10-12T12:34:44.273Z", "content": "

Thanks for sharing, this give me something to try out next time.

", "rawMarkdown": "Thanks for sharing, this give me something to try out next time." } ] } ] }, { "id": 2479111, "postDate": "2023-10-12T12:14:25.053Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:02:18Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10675000, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/ivan-aerlic-2nd-place-solution" }