Before writing our solution, we really thank to Georgia State University and Kaggle for hosting this competition and congrats to all. Also, I really enjoyed doing this competition with my UPSTAGE team (sergei, ducky, and dlrgy22).
\nThe training code and inference notebook are here:
\n\nThe pipeline is here.
\n\n\nValidation strategy -> Text pre-processing -> Model -> Ensemble -> Post-processing on logits
\n
✔️ Things that worked.
\n\n\n\n
\n- Initial learning rate 1e-5
\n- Max gradient norm to small (about 1.0)
\n- Plateau or Linearly Reduced LR With Warmup
\n- SWA (stabilize valid performance, at least +0.01 boost)
\n- Mean teacher with noise label filtering by exponential moving average
\n
❌ Not worked.
\n\n\n\n
\n- Initial learning rate 3e-5
\n- Max gradient norm to large (abot 10)
\n- SAM Optimizer
\n- Dice Loss / Focal Loss with gamma 2.0
\n- Position Bucket Expanding at DeBERTaV3
\n
\n\nentity before pre-p: ,Some quotation here.
\n
\n entity after pre-p: Some quotation here.
\n\nEx) discourse_id -> 1621804837671
\n
\n given text: t would allow students to …
\n modified text: it would allow students to …
We used 4 DeBERTa models: DeBERTa version 1 with large and xlarge model, DeBERTa version 2 with xlarge model and DeBERTa version 3 with large model.
\nHere are the training code for each model and how to run it.
\n\nFor more details with examples, see the solution documents above.
\nWe used First Token Ensemble.
\nFor this competition we conducted an ensemble targeting logits of tokens. However, since the vocab defined for each tokenizer used in each model is different, the tokenized result may be different even for the same text. Therefore, it is not possible to simply sum the results of the two models because the results from tokenizer are different, and one idea is applied here: First Token Ensemble that will group overlapping tokens and add log probabilities for the first token of each group.
\nFor more details with examples, see the solution documents above.
\nWe applied Start With I-tag, Look a Head, First Appearance and Extending end span pp.
\n\n\nNot really a B tag but it starts with an I tag.
\n
\n\nThere were cases where tags appeared in succession and only one was empty.
\n
\n\nHandling of classes that appear once per essay.
\n
\n\nExtending end span to some extent.
\n
For more details with examples, see the solution documents above.
", "comments": [ { "id": 1740930, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2022-03-31T11:45:15.477000", "content": "Thanks for sharing and congratulation on your gold! May I ask a bit about SWA, it seems to gain a big improvement. May I ask how much the score is before and after using it?
", "votes": 3, "replies": [ { "id": 1741681, "author_name": "Ducky", "author_url": "", "post_date": "2022-04-01T04:12:43.713000", "content": "We used SWA for the Pytorch version and also customed sliding window version(that uses 25 weights for the recent models collected by specific steps period).
Just for the single model the performance
But didn't check the performance on the ensemble model. maybe other teammates could know.
", "votes": 3, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 1740640, "postDate": "2022-03-31T06:24:23.177Z", "content": "Before writing our solution, we really thank to Georgia State University and Kaggle for hosting this competition and congrats to all. Also, I really enjoyed doing this competition with my UPSTAGE team (sergei, ducky, and dlrgy22).
\nThe training code and inference notebook are here:
\n\nThe pipeline is here.
\n\n\nValidation strategy -> Text pre-processing -> Model -> Ensemble -> Post-processing on logits
\n
✔️ Things that worked.
\n\n\n\n
\n- Initial learning rate 1e-5
\n- Max gradient norm to small (about 1.0)
\n- Plateau or Linearly Reduced LR With Warmup
\n- SWA (stabilize valid performance, at least +0.01 boost)
\n- Mean teacher with noise label filtering by exponential moving average
\n
❌ Not worked.
\n\n\n\n
\n- Initial learning rate 3e-5
\n- Max gradient norm to large (abot 10)
\n- SAM Optimizer
\n- Dice Loss / Focal Loss with gamma 2.0
\n- Position Bucket Expanding at DeBERTaV3
\n
\n\nentity before pre-p: ,Some quotation here.
\n
\n entity after pre-p: Some quotation here.
\n\nEx) discourse_id -> 1621804837671
\n
\n given text: t would allow students to …
\n modified text: it would allow students to …
We used 4 DeBERTa models: DeBERTa version 1 with large and xlarge model, DeBERTa version 2 with xlarge model and DeBERTa version 3 with large model.
\nHere are the training code for each model and how to run it.
\n\nFor more details with examples, see the solution documents above.
\nWe used First Token Ensemble.
\nFor this competition we conducted an ensemble targeting logits of tokens. However, since the vocab defined for each tokenizer used in each model is different, the tokenized result may be different even for the same text. Therefore, it is not possible to simply sum the results of the two models because the results from tokenizer are different, and one idea is applied here: First Token Ensemble that will group overlapping tokens and add log probabilities for the first token of each group.
\nFor more details with examples, see the solution documents above.
\nWe applied Start With I-tag, Look a Head, First Appearance and Extending end span pp.
\n\n\nNot really a B tag but it starts with an I tag.
\n
\n\nThere were cases where tags appeared in succession and only one was empty.
\n
\n\nHandling of classes that appear once per essay.
\n
\n\nExtending end span to some extent.
\n
For more details with examples, see the solution documents above.
", "rawMarkdown": "# Overview\nBefore writing our solution, we really thank to Georgia State University and Kaggle for hosting this competition and congrats to all. Also, I really enjoyed doing this competition with my UPSTAGE team (sergei, ducky, and dlrgy22).\n\nThe training code and inference notebook are here:\n* [Training code](https://github.com/anotherchudov/feedback)\n* [Inference notebook](https://www.kaggle.com/code/sergeichudov/8th-place-inference-notebook)\n* [model weights](https://www.kaggle.com/code/sergeichudov/8th-place-inference-notebook/data)\n* [solution documents](https://docs.google.com/document/d/1GTBAplv6uWjq-jlLk4_toIYLp6mnCntAzwOb6DlPZ3Q/edit?usp=sharing)\n\nThe pipeline is here.\n> Validation strategy -> Text pre-processing -> Model -> Ensemble -> Post-processing on logits\n\n✔️ Things that worked.\n> - Initial learning rate 1e-5\n- Max gradient norm to small (about 1.0)\n- Plateau or Linearly Reduced LR With Warmup\n- SWA (stabilize valid performance, at least +0.01 boost)\n- Mean teacher with noise label filtering by exponential moving average\n\n❌ Not worked.\n> - Initial learning rate 3e-5\n- Max gradient norm to large (abot 10)\n- SAM Optimizer\n- Dice Loss / Focal Loss with gamma 2.0\n- Position Bucket Expanding at DeBERTaV3\n\n## Validation strategy\n* Cross validation with topic from [CHRIS DEOTTE notebook](https://www.kaggle.com/code/cdeotte/rapids-umap-tfidf-kmeans-discovers-15-topics).\n* Use half of all oof data as a test to find post-processing hyper-parameters.\n\n## Text pre-processing\n* We use mask tokens to reflect newline information.\n* adjusting start of each entity to nearest alphanumeric character to the right \n> entity before pre-p: ,Some quotation here.\nentity after pre-p: Some quotation here.\n\n* some samples with a word split by the start or end of entity text\n > Ex) discourse_id -> 1621804837671\ngiven text: t would allow students to ...\nmodified text: it would allow students to ...\n\n## Model\nWe used 4 DeBERTa models: DeBERTa version 1 with large and xlarge model, DeBERTa version 2 with xlarge model and DeBERTa version 3 with large model. \n\nHere are the training code for each model and how to run it.\n* [DeBERTa v1 large and DeBERTa v2 xlarge](https://github.com/anotherchudov/feedback/tree/sergei)\n* [DeBERTa v1 xlarge](https://github.com/anotherchudov/feedback/tree/ynot)\n* [DeBERTa v3 large](https://github.com/anotherchudov/feedback/tree/dlrgy22)\n\nFor more details with examples, see the solution documents above.\n\n## Ensemble\nWe used **First Token Ensemble**.\n\nFor this competition we conducted an ensemble targeting logits of tokens. However, since the vocab defined for each tokenizer used in each model is different, the tokenized result may be different even for the same text. Therefore, it is not possible to simply sum the results of the two models because the results from tokenizer are different, and one idea is applied here: **First Token Ensemble** that will group overlapping tokens and add log probabilities for the first token of each group.\n\nFor more details with examples, see the solution documents above.\n\n## Post-processing\nWe applied **Start With I-tag**, **Look a Head**, **First Appearance** and **Extending end span** pp.\n\n* Start With I-tag pp\n> Not really a B tag but it starts with an I tag.\n\n* Look a Head pp\n> There were cases where tags appeared in succession and only one was empty.\n\n* First Appearance pp\n> Handling of classes that appear once per essay.\n\n* Extended extracted entities to the right\n> Extending end span to some extent.\n\nFor more details with examples, see the solution documents above.\n\n\n\n", "votes": 30 }, { "id": 1740930, "postDate": "2022-03-31T11:45:15.477Z", "content": "Thanks for sharing and congratulation on your gold! May I ask a bit about SWA, it seems to gain a big improvement. May I ask how much the score is before and after using it?
", "rawMarkdown": "Thanks for sharing and congratulation on your gold! May I ask a bit about SWA, it seems to gain a big improvement. May I ask how much the score is before and after using it?", "votes": 3, "replies": [ { "id": 1741681, "postDate": "2022-04-01T04:12:43.713Z", "content": "We used SWA for the Pytorch version and also customed sliding window version(that uses 25 weights for the recent models collected by specific steps period).
Just for the single model the performance
But didn't check the performance on the ensemble model. maybe other teammates could know.
", "rawMarkdown": "We used SWA for the `Pytorch version` and also customed `sliding window version`(that uses 25 weights for the recent models collected by specific steps period). \n\nJust for the `single model` the performance\n- DeBERTaV3 was around 0.67\n- SWA(Stochastic Weight Average) helped to improve this by over 0.68\n- SWA(25 Sliding Window Average) helped to improve this by over 0.68 also.\n- mean teacher was even better. It helped to get over 0.69\n\nBut didn't check the performance on the ensemble model. maybe other teammates could know.", "votes": 3 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:36:36Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 8014367, "solution_url": "/competitions/feedback-prize-2021/writeups/upstage-8th-place-solution" }