First of all, congratulate to all winners and thank you for the organizers.
\nAlso, thank you for all competitors especially @abhishek and @kashiwaba who publish the kernels that I refer the most.
For the classification, I use token prediction and span prediction. In the essay text, there are multiple target texts. In my solution, the model is trained to predict the effectiveness using specific token or average of text span. Furthermore, I find that adding CLS and SEP token at the beginning of target text affects the CV scores. Using the deberta-v3-large and deberta-v2-xlarge, I obtained the following results.
\n| Model | \nApproach | \nCV | \n
|---|---|---|
| deberta-v3-large | \nspan prediction | \n0.6182 | \n
| deberta-v3-large | \nCLS/SEP, token prediction | \n0.6165 | \n
| deberta-v3-large | \nCLS/SEP, span prediction | \n0.6181 | \n
| deberta-v2-xlarge | \nspan prediction | \n0.6290 | \n
| deberta-v2-xlarge | \nspan prediction, overfitting | \n0.7193 | \n
| deberta-v2-xlarge | \nCLS/SEP, token prediction | \n0.6308 | \n
| deberta-v2-xlarge | \nCLS/SEP, token prediction, overfitting | \n0.7404 | \n
Interestingly, deberta-v2-xlarge shows lower CV score, but it boosts the CV after the ensemble.
\nAfter simple ensemble (manual weight tuning, CV 0.589, 0.583).
To improve the score, I first ensemble the models where the weights are optimized to minimize OOF CV using Bayesian optimization (CV 0.579).
\nThen, I use LGBM to improve the score further, I add effectiveness of prev/next text and location informations of the text as the additional information (CV 0.578, LB 0.575, Private LB 0.573).
The solutions of other competitors are always astonishing and attractive. I regret that MLM pretraining using previous competition and pseudo labeling could improve my score. Appreciate other competitors for sharing great solutions. My solution is in https://www.kaggle.com/learnitanyway/24th-inference-deberta-ensemble.
", "comments": [], "comment_count": 0, "messages": [ { "id": 1911352, "postDate": "2022-08-24T03:30:48.857Z", "content": "First of all, congratulate to all winners and thank you for the organizers.
\nAlso, thank you for all competitors especially @abhishek and @kashiwaba who publish the kernels that I refer the most.
For the classification, I use token prediction and span prediction. In the essay text, there are multiple target texts. In my solution, the model is trained to predict the effectiveness using specific token or average of text span. Furthermore, I find that adding CLS and SEP token at the beginning of target text affects the CV scores. Using the deberta-v3-large and deberta-v2-xlarge, I obtained the following results.
\n| Model | \nApproach | \nCV | \n
|---|---|---|
| deberta-v3-large | \nspan prediction | \n0.6182 | \n
| deberta-v3-large | \nCLS/SEP, token prediction | \n0.6165 | \n
| deberta-v3-large | \nCLS/SEP, span prediction | \n0.6181 | \n
| deberta-v2-xlarge | \nspan prediction | \n0.6290 | \n
| deberta-v2-xlarge | \nspan prediction, overfitting | \n0.7193 | \n
| deberta-v2-xlarge | \nCLS/SEP, token prediction | \n0.6308 | \n
| deberta-v2-xlarge | \nCLS/SEP, token prediction, overfitting | \n0.7404 | \n
Interestingly, deberta-v2-xlarge shows lower CV score, but it boosts the CV after the ensemble.
\nAfter simple ensemble (manual weight tuning, CV 0.589, 0.583).
To improve the score, I first ensemble the models where the weights are optimized to minimize OOF CV using Bayesian optimization (CV 0.579).
\nThen, I use LGBM to improve the score further, I add effectiveness of prev/next text and location informations of the text as the additional information (CV 0.578, LB 0.575, Private LB 0.573).
The solutions of other competitors are always astonishing and attractive. I regret that MLM pretraining using previous competition and pseudo labeling could improve my score. Appreciate other competitors for sharing great solutions. My solution is in https://www.kaggle.com/learnitanyway/24th-inference-deberta-ensemble.
", "rawMarkdown": "First of all, congratulate to all winners and thank you for the organizers. \nAlso, thank you for all competitors especially [@abhishek](https://www.kaggle.com/code/abhishek/tez-for-feedback-v2-0) and [@kashiwaba](https://www.kaggle.com/code/kashiwaba/train-deberta-v3-large-with-optimization-approach) who publish the kernels that I refer the most.\n\n#### Phase 1 Token prediction and span prediction\n\nFor the classification, I use token prediction and span prediction. In the essay text, there are multiple target texts. In my solution, the model is trained to predict the effectiveness using specific token or average of text span. Furthermore, I find that adding CLS and SEP token at the beginning of target text affects the CV scores. Using the deberta-v3-large and deberta-v2-xlarge, I obtained the following results.\n| Model | Approach | CV |\n| --- | --- | --- | \n| deberta-v3-large | span prediction | 0.6182 | \n| deberta-v3-large | CLS/SEP, token prediction| 0.6165 | \n| deberta-v3-large | CLS/SEP, span prediction | 0.6181 | \n| deberta-v2-xlarge | span prediction| 0.6290| \n| deberta-v2-xlarge | span prediction, overfitting| 0.7193 | \n| deberta-v2-xlarge | CLS/SEP, token prediction| 0.6308 | \n| deberta-v2-xlarge | CLS/SEP, token prediction, overfitting | 0.7404 | \n\nInterestingly, deberta-v2-xlarge shows lower CV score, but it boosts the CV after the ensemble. \nAfter simple ensemble (manual weight tuning, CV 0.589, 0.583).\n\n#### Phase 2 Bayesian optimization for ensemble weight and LGBM\n\nTo improve the score, I first ensemble the models where the weights are optimized to minimize OOF CV using Bayesian optimization (CV 0.579). \nThen, I use LGBM to improve the score further, I add effectiveness of prev/next text and location informations of the text as the additional information (CV 0.578, LB 0.575, Private LB 0.573).\n\n#### Summary - Useful attempt\n-\tInference effectiveness using whole essay text \n-\tEnsemble span and token classification, use CLS/SEP token\n-\tWeight optimization and LGBM to deal with additional features\n\nThe solutions of other competitors are always astonishing and attractive. I regret that MLM pretraining using previous competition and pseudo labeling could improve my score. Appreciate other competitors for sharing great solutions. My solution is in https://www.kaggle.com/learnitanyway/24th-inference-deberta-ensemble.\n", "votes": 17 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:54:24Z", "competition_id": 35308, "competition_slug": "feedback-prize-effectiveness", "team_id": 8831953, "solution_url": "/competitions/feedback-prize-effectiveness/writeups/lia-24th-short-solution" }