![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F530572%2Fc4af5af50522e5116d8ff8a626d594ef%2FScreenshot%202023-10-12%20at%2017.42.56.png?generation=1697129165509557&alt=media\")
Congratulations to the winners and everyone who put lots of efforts on this competition, and thank Kaggle & host for organising this competition.
\nEven though my team shakedown on the private learderboard, we don't think it's truly 'shakedown'. Because our CVs are very close to the LBs (public + private) and we have already selected almost our best results. Our final solutions are purely based on best CVs.
● Various inputs using common words, phrases, etc. between “prompt_text” and “text” as inputs.
\n● Custom heads combining various pooling.
\n● Pseudo labeling with back translated texts.
\n● Weighted loss.
\n● Ensemble weights based on ridge regression with large alpha
CV LB LB\n①best public LB 0.4596 0.420 0.463
\n②best CV 0.4566 0.423 0.463
\n③other 0.4579 0.422 0.464
\n(best private LB) 0.4601 0.420 0.462
\ncv roughly euqals to LB: 0.457 = 0.420x0.13(public) + 0.463x0.87(private)
We added common words, phrases, etc. between “prompt_text” and “text” to the end of the input texts.(While 2 and 3 grams overlap were added to the input, 2nd Stage LGB models are not working anymore, at least this is the case on Jie’s models)
\n● overlap words
\n● overlap N-grams (N=2,3)
\n● overlap phrases
\n● overlap key phrases by Rake method
We created many custom heads that cat the output values of different poolings.
\n● CLS output
\n● Mean pooling
\n● Max pooling
\n● GeM pooling
\n● Attention pooling
\n● Mean-Max-Attention Pooling (MMA)
\n● Mean-Max-Gem Pooling (MMG)
\n● Gem-Attention Pooling (GA)
\nExample: GA Pooling
\ngemtext_out = self.gempooler(lasthiddenstate, attention_mask)
\nattpool_out = self.attpooler(lasthiddenstate, attention_mask)
\ncontext_vector = torch.cat((gemtext_out, attpool_out), dim=-1)
Data Augmentation and Pseudo Labelling: Translate summary text to Chinese, then translate back to English, then use a good model to Pseudo label.
\nWe created a weighted loss for wording and content as following, ratios are decided by the RMSE ratio between content and wording.
\nclass WeightedCRMSE(nn.Module):
\n def init(self):
\n super(WeightedCRMSE, self).init()
\n self.weights = torch.tensor([0.85, 1.15]).to(device)
():\n squared_errors = torch.square(y_pred - y_true)\n weighted_root_squared_errors = torch.mean(squared_errors, dim=) ** * .weights\n loss = torch.mean(weighted_root_squared_errors)\n loss\nWe trained ridge regression out-of-fold and use the mean of the normalized coefficients as the ensemble weights. Both CV and LB scores were better when using ridge regression with a large alpha than using linear regression and other methods.
\n● Backbone
\n○ deberta-v3-large
\n○ electra-large
\n○ roberta-large
\n○ roberta-large-squad2
\n● Model for ensemble
\n○ LightGBM
\n○ Ridge regression
\n● Max length
\n○ 300
\n○ 512
\n○ 768
● LSTM head
\n● Freezing the embbedings and some layers.
\n● AWP
\n● MLM
\n● 30° rotated targets
\n● Different max length between training and inference.
● Name convention backbon_maxlength_2&3gramsoverlap_dataaugmentionflag_pooling, e.g.
\ndel_768_23_aug_mmg: Debertal-V3-Large, maxlength=768, 2and3 grams overlap were added to input, data augmention is used for model training, use Mean-Max-Gem pooling.
\n● Table below: 1st row: model name, 2nd row: CV(average of 4 folds), 3rd row: CV (oof), 4th row: LB, 5th row: Inference time.
\n● Ensemble of LB 0.437, 0.432, 0.439, 0.436, 0.433 and 0.436 gives CV: 0.45909, Pubic LB 0.424, Private LB: 0.464
\n● Note: Deberta Large models inference time are 80mins not 125-140min, all other models are <=36mins\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F530572%2F231869326fa934ba8ccf68f282a8a442%2FJies%20Model%20Performance.png?generation=1697129193727475&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F530572%2F692c701395194fb917ea8a5be76ef559%2Fyyykrk.png?generation=1697129202595643&alt=media\")
Congratulations to the winners and everyone who put lots of efforts on this competition, and thank Kaggle & host for organising this competition.
\nEven though my team shakedown on the private learderboard, we don't think it's truly 'shakedown'. Because our CVs are very close to the LBs (public + private) and we have already selected almost our best results. Our final solutions are purely based on best CVs.
● Various inputs using common words, phrases, etc. between “prompt_text” and “text” as inputs.
\n● Custom heads combining various pooling.
\n● Pseudo labeling with back translated texts.
\n● Weighted loss.
\n● Ensemble weights based on ridge regression with large alpha
CV LB LB\n①best public LB 0.4596 0.420 0.463
\n②best CV 0.4566 0.423 0.463
\n③other 0.4579 0.422 0.464
\n(best private LB) 0.4601 0.420 0.462
\ncv roughly euqals to LB: 0.457 = 0.420x0.13(public) + 0.463x0.87(private)
We added common words, phrases, etc. between “prompt_text” and “text” to the end of the input texts.(While 2 and 3 grams overlap were added to the input, 2nd Stage LGB models are not working anymore, at least this is the case on Jie’s models)
\n● overlap words
\n● overlap N-grams (N=2,3)
\n● overlap phrases
\n● overlap key phrases by Rake method
We created many custom heads that cat the output values of different poolings.
\n● CLS output
\n● Mean pooling
\n● Max pooling
\n● GeM pooling
\n● Attention pooling
\n● Mean-Max-Attention Pooling (MMA)
\n● Mean-Max-Gem Pooling (MMG)
\n● Gem-Attention Pooling (GA)
\nExample: GA Pooling
\ngemtext_out = self.gempooler(lasthiddenstate, attention_mask)
\nattpool_out = self.attpooler(lasthiddenstate, attention_mask)
\ncontext_vector = torch.cat((gemtext_out, attpool_out), dim=-1)
Data Augmentation and Pseudo Labelling: Translate summary text to Chinese, then translate back to English, then use a good model to Pseudo label.
\nWe created a weighted loss for wording and content as following, ratios are decided by the RMSE ratio between content and wording.
\nclass WeightedCRMSE(nn.Module):
\n def init(self):
\n super(WeightedCRMSE, self).init()
\n self.weights = torch.tensor([0.85, 1.15]).to(device)
():\n squared_errors = torch.square(y_pred - y_true)\n weighted_root_squared_errors = torch.mean(squared_errors, dim=) ** * .weights\n loss = torch.mean(weighted_root_squared_errors)\n loss\nWe trained ridge regression out-of-fold and use the mean of the normalized coefficients as the ensemble weights. Both CV and LB scores were better when using ridge regression with a large alpha than using linear regression and other methods.
\n● Backbone
\n○ deberta-v3-large
\n○ electra-large
\n○ roberta-large
\n○ roberta-large-squad2
\n● Model for ensemble
\n○ LightGBM
\n○ Ridge regression
\n● Max length
\n○ 300
\n○ 512
\n○ 768
● LSTM head
\n● Freezing the embbedings and some layers.
\n● AWP
\n● MLM
\n● 30° rotated targets
\n● Different max length between training and inference.
● Name convention backbon_maxlength_2&3gramsoverlap_dataaugmentionflag_pooling, e.g.
\ndel_768_23_aug_mmg: Debertal-V3-Large, maxlength=768, 2and3 grams overlap were added to input, data augmention is used for model training, use Mean-Max-Gem pooling.
\n● Table below: 1st row: model name, 2nd row: CV(average of 4 folds), 3rd row: CV (oof), 4th row: LB, 5th row: Inference time.
\n● Ensemble of LB 0.437, 0.432, 0.439, 0.436, 0.433 and 0.436 gives CV: 0.45909, Pubic LB 0.424, Private LB: 0.464
\n● Note: Deberta Large models inference time are 80mins not 125-140min, all other models are <=36mins\n