I would like to express my gratitude to the Kaggle staff and competition hosts for organizing this event. This document outlines the tactics that were successful, and those that weren't as effective in my experiments. Without the remarkable notebook at https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect and various insights from numerous discussions, my work wouldn't have been as comprehensive. Credit goes to all those contributors who I, unfortunately, do not recall by name.
\nMy solution involved an ensemble of 17 models; 11 deberta-v3-large models and 6 deberta-v3-base models, each with different learning rates. All of this work was conducted on a workstation with an i9-12900k and a 3090 24G system.
\nSetting hidden_dropout_prob=0 [High Impact]
\nGiven that this is a regression problem, using dropout creates a discrepancy between the output scales during inference and training.
Using a fusion model for wording and content predictions [Decreases training and inference time without impacting performance]
\nThe original notebook used separate models for these tasks, which extends training time without any significant improvement in inference performance.
Leveraging the LightGBM model with enhanced feature engineering [High Impact]
\nI put more emphasis on feature engineering, as illustrated in this notebook: https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail.
Using an ensemble of deberta-v3-large and deberta-v3-base models with varying learning rates [High Impact]
\nA lower learning rate (~ 8e10-6) works better for deberta-v3-large models, while a higher one (~ 2e10-5) is more effective for deberta-v3-base models.
Maintaining uniform ensemble weights [High Impact]
\nOptimizing the model weights using Optuna tends to overfit the ensemble model.
Using different weights for different models with Optuna
\nWhile this improves CV scores, it leads to overfitting in the LB and PB tests.
Cleaning the text
\nI attempted this late in the process, and it led to overfitting in both the PB and LB tests. I'm unsure why this happened and appreciate anyone willing to share their successful experiences with text cleaning.
Setting attention_probs_dropout_prob = 0
\nThis ended up deteriorating the model's scores.
I am very lucky that my best notebook for LB is best notebook for PB.
\n| Model | \nLB | \nPB | \n
|---|---|---|
| deberta-v3-base 4-fold ( only change hidden dropout =0) | \n0.44565 | \n0.48406 | \n
| Ensemble deberta-v3-large 4-fold with 2 base | \n0.44073 | \n0.48301 | \n
| Ensemble model ( 17 model find tune weight with optuna) | \n0.43694 | \n0.47308 | \n
| Ensemble model ( 17 model with more feature engineer) | \n0.43551 | \n0.47284 | \n
I would like to express my gratitude to the Kaggle staff and competition hosts for organizing this event. This document outlines the tactics that were successful, and those that weren't as effective in my experiments. Without the remarkable notebook at https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect and various insights from numerous discussions, my work wouldn't have been as comprehensive. Credit goes to all those contributors who I, unfortunately, do not recall by name.
\nMy solution involved an ensemble of 17 models; 11 deberta-v3-large models and 6 deberta-v3-base models, each with different learning rates. All of this work was conducted on a workstation with an i9-12900k and a 3090 24G system.
\nSetting hidden_dropout_prob=0 [High Impact]
\nGiven that this is a regression problem, using dropout creates a discrepancy between the output scales during inference and training.
Using a fusion model for wording and content predictions [Decreases training and inference time without impacting performance]
\nThe original notebook used separate models for these tasks, which extends training time without any significant improvement in inference performance.
Leveraging the LightGBM model with enhanced feature engineering [High Impact]
\nI put more emphasis on feature engineering, as illustrated in this notebook: https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail.
Using an ensemble of deberta-v3-large and deberta-v3-base models with varying learning rates [High Impact]
\nA lower learning rate (~ 8e10-6) works better for deberta-v3-large models, while a higher one (~ 2e10-5) is more effective for deberta-v3-base models.
Maintaining uniform ensemble weights [High Impact]
\nOptimizing the model weights using Optuna tends to overfit the ensemble model.
Using different weights for different models with Optuna
\nWhile this improves CV scores, it leads to overfitting in the LB and PB tests.
Cleaning the text
\nI attempted this late in the process, and it led to overfitting in both the PB and LB tests. I'm unsure why this happened and appreciate anyone willing to share their successful experiences with text cleaning.
Setting attention_probs_dropout_prob = 0
\nThis ended up deteriorating the model's scores.
I am very lucky that my best notebook for LB is best notebook for PB.
\n| Model | \nLB | \nPB | \n
|---|---|---|
| deberta-v3-base 4-fold ( only change hidden dropout =0) | \n0.44565 | \n0.48406 | \n
| Ensemble deberta-v3-large 4-fold with 2 base | \n0.44073 | \n0.48301 | \n
| Ensemble model ( 17 model find tune weight with optuna) | \n0.43694 | \n0.47308 | \n
| Ensemble model ( 17 model with more feature engineer) | \n0.43551 | \n0.47284 | \n