# 15th Place Solution : 6 kinds of AttentionPooling for each metric work well for us First of all We appreciate this fortunate luck, and are truly honored to receive the gold medal!! From the results, Yuichi Tateno ( [@hotchpotch](https://www.kaggle.com/hotchpotch) ) and Masaki Aota ( [@masakiaota](https://www.kaggle.com/masakiaota) ) became the Kaggle Competitions Master. Overview Our final submission achieved Private Score: 0.4346, Public Score: 0.4351, Local CV: 0.4431. The inference is ensembling of 11 models. Our inference notebook is here. [https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl](https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl) The summary of 11 models is following. Abbreviations appearing in column names will be explained in What Worked . exp num backbone model CV Pooling AWP LLRD ReInit SW max_len LayerNorm 072 deberta-v3-large 0.4491 Mean Yes No No No Inf No 089 deberta-v3-large 0.4487 Mean Yes Yes No No Inf No 136 deberta-v3-large 0.4488 6AP No Yes Yes Yes Inf No 182 deberta-v3-base 0.4496 6AP Yes Yes Yes Yes 640 No 214 deberta-v1-large 0.4484 6AP Yes Yes Yes Yes 640 Yes 215 deberta-v3-large 0.4481 6AP Yes Yes Yes Yes 640 Yes 226 deberta-v3-large 0.4477 6AP Yes Yes Yes No Inf Yes 231 deberta-v3-base 0.4487 6AP Yes Yes Yes Yes 640 Yes 232 deberta-v3-base 0.4508 6AP No Yes Yes Yes 640 Yes 233 deberta-v1-large 0.4486 6AP Yes Yes Yes No 640 Yes 234 deberta-v1-large 0.4495 6AP Yes Yes Yes Yes Inf Yes We use the following ensemble weights: exp num cohesion syntax vocabulary phraseology grammar conventions 072 0.0186 0.1387 0.1042 0.1474 0.1135 0.1757 089 0.0022 0.0123 0.0001 0.1071 0.2395 0.0512 136 0.1295 0.1666 0.0589 0.0950 0.1235 0.0012 182 0.0011 0.1051 0.0749 0.2280 0.0794 0.0577 214 0.2944 0.0930 0.1365 0.0485 0.0259 0.0922 215 0.0965 0.0036 0.0244 0.1390 0.0772 0.1055 226 0.1654 0.1212 0.1700 0.0469 0.0394 0.0325 231 0.0562 0.0801 0.0157 0.0216 0.0958 0.1178 232 0.0105 0.0434 0.1012 0.0827 0.1154 0.0830 233 0.1543 0.2058 0.1807 0.0194 0.0317 0.1106 234 0.0704 0.0318 0.1335 0.0662 0.0596 0.1727 Our Code Inference: [https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl](https://www.kaggle.com/code/hotchpotch/fb3-infer-ens043-no-pl) Training: [https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook](https://www.kaggle.com/code/masakiaota/fb3-15th-place-solution-train-exp231/notebook) If the codes are helpful, don't forget to upvote not only mine but also Y.Nakama's one ( [https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) ) . What Worked [High Impact] 6 kinds of AttentionPooling for each metric (6AP) CV : -0.0015 This allows our model to focus on different tokens for different metrics. The idea can be illustrated like this (right one). ![](assets/af5c25d8c928747498f71460.jpg) [High Impact] Adversarial Weights Perturbation (AWP) CV : -0.002 Original paper : [assets/122c99b56afc5a669d68f0cb.pdf](assets/122c99b56afc5a669d68f0cb.pdf) My slide (in Japanese) : [https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization](https://speakerdeck.com/masakiaota/kaggledeshi-yong-sarerudi-dui-xue-xi-fang-fa-awpnolun-wen-jie-shuo-toshi-zhuang-jie-shuo-adversarial-weight-perturbation-helps-robust-generalization) [High Impact] Layer-wise Lerarning Rate Decay (LLRD) CV : -0.0005 Decay by *0.85 for large model, *0.7225 for base model. [High Impact] Last Layer Re-initialization (ReInit) CV: -0.0006 ([High Impact] Of Cource, Ensembling) [Medium Impact] Max Token Length to 640 (max_len) CV: - 0~0.001 Only work for base model Not work for large model but training speed become fast [Medium Impact] SlidingWindowTransformerModel (SW) CV: - 0~0.001 But bad combination with AWP. My implementation is similar with this [https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322](https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322) [Medium Impact] Pseudo Labeling CV: - 0.0001~0.0006 Our submission which got highest private score did not contain the model learning with pseudo-label. [Medium Impact] High Frequency Validation and Saving CV: - 0.0008 [Medium Impact] Layer Normalization after Pooling Layer, before Fully Connected Layer (LayerNorm) CV: - 0~0.001 Only work for base model [Medium Impact] Multi sample dropout CV: +- 0.0 The curve of validation loss becomes stable a little. [Medium Impact] Optimize Ensembling Weights for Each Metrics Independently CV: - 0.0001 ~ 0.0004 What Didn’t Work SiFT May be bugs in our code… SVR No leakage but too low CV and bad public score. So, we didn't use SVR to ensemble. Any Augmentations Any Losses except L1 Smooth Pretraining using Feedback1,2 Sentences I’m not sure this is really not to work… Dealing with “Generic_Name” and others Change the Way to Initialize Weights Parameters sampled from normal distribution is the best for us Additional Context Our Best Single model (in private and CV) Private: 0.4361, Public: 0.4375, CV: 0.4450 This model was Learning from Ensembled Pseudo-Labels. But, our emsemble inference, which get highest private score, did not contains this model. If we leveraged this model to ensemble, we might get the more good standing. This idea may be useful especially for efficiency prize (in my opinion). Important Citations Y.Nakama’s baseline [https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) AI Shift’s tech blog (Japanese) [https://www.ai-shift.co.jp/techblog/2170](https://www.ai-shift.co.jp/techblog/2170) Discussion about Attention Pooling [https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/361678) XGB+LGB Feedback Prize CV 0.7322 [https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322](https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322) Thanks and Acknowledgements We would like to thank the organizers of the competition for hosting this amazing event. I personally would like to mention and thank Taro Masuda ( [@taromasuda](https://www.kaggle.com/taromasuda) ), Raiii ( [@skraiii](https://www.kaggle.com/skraiii) ) and MPEG ( [@mpeg31](https://www.kaggle.com/mpeg31) ) who did not participate in this competition, as I have learnt so much from them in the past competitions. I certainly was able to utilize the knowledge this time! Team Members ![](assets/e83c427cebcbc1f5f6d24675.png) From left in the above figure: Masaki Aota [@masakiaota](https://www.kaggle.com/masakiaota) Olivine-Ryo [@olivineryo](https://www.kaggle.com/olivineryo) Yuichi Tateno [@hotchpotch](https://www.kaggle.com/hotchpotch)