{ "id": 369793, "title": "49th place (Weighted Loss etc.)", "content": "

I'm new to the NLP and this is my first NLP competition.
\nSo I learned a lot of things from the competition.
\nThanks for hosting the competition.

\n

OverView

\n

I ensemble 15 models (with 4 folds).
\nI used orthogonal initialization, reinitialize the last 1 layer and layer wise learning rate decay for my all model.
\nThese methods also worked well for me.

\n

And Weighted Loss slightly improved CV and LB for some models.
\nThe Weighted loss is maybe my unique point.
\nI shared my code for Weighted Loss.
\nThe Weighted loss tend to improve the score of targets which can be easily predictable.
\nFor model diversity, I wanted to make two model (One is with Weighted Loss, other is without Weigted Loss) for each models, but There was no time to make all these model.

\n

Then I tried applying some method (AWP, SiFT, MIXout, and so on..) into the Deberta-v3-base model. But all of them didn't work.
\nHowever at the all most end of this competition, I found SiFT and MIXout worked well for Deberta-v3-small (Without Weighted Loss).
\nI wanted to apply these method into other models (large, xsmall, Roberta and so on), but there was no time to train…
\nSo these methods could be applied into only Deberta-v3-small model.

\n

Models used for the my best submission is below.
\n\"\"

\n

For deberta-v3-XXX set the max_len=1462, others set the max_len=512

\n

What Worked

\n\n

What Didn't Work:

\n\n

Ensemble strategy:

\n
    \n
  1. Optimize ensemble weights (target-wise) using optuna. (Trust CV)
    \nIf I use optuna for my OOF. the optuna set some Deberta-v3-base weights ZERO.
    \nI was afraid to submit, but I believed in my CV and this is the best in the PB!
    2. \n
  2. Manually optimize weights considering LB rank. (Trust LB)
    3. \n
  3. Just averaging 1. + 2. (Trust CV and LB))
    4. \n
\n

CV strategy

\n\n

Thanks.

", "comments": [ { "id": 2057359, "author_name": "Apa", "author_url": "", "post_date": "2022-12-07T02:56:47.403000", "content": "

I'm curious about how you used SiFT technology in this competition, can you share the code for this part?

", "votes": 1, "replies": [ { "id": 2060658, "author_name": "taruto", "author_url": "", "post_date": "2022-12-10T08:40:26.053000", "content": "

Sorry for my delay.

\n

There are no big change from the source code.
\nI changed the KL loss into the MSE loss in the line 161.

", "votes": 0, "replies": [] }, { "id": 2060662, "author_name": "taruto", "author_url": "", "post_date": "2022-12-10T08:50:43.930000", "content": "

I changed this class like below.
\nThanks.

\n
 :\n   ():\n     adv_modules  :\n      self.adv_modules = [m  m  model.modules()  (m, PerturbationLayer)]\n    :\n      self.adv_modules = adv_modules\n    self.parameters = [p  p  model.parameters()]\n    self.model = model\n    self.rmse = RMSELoss(reduction=)\n   (): \n    self.prepare()\n     (loss_fn, ):\n      loss_fn = perturbation_loss_fns[loss_fn]\n    pert_logits = self.model(inputs) \n    pert_loss = loss_fn(pert_logits, target.detach())\n     scaler != :\n      scaler.scale(pert_loss).backward()\n    :\n      pert_loss.backward()\n     m  self.adv_modules:\n      ok = m.update_delta()\n     r,p  (self.prev, self.parameters):\n      p.requires_grad_(r)\n    pert_logits = self.model(inputs) \n    \n    pert_loss = self.rmse(pert_logits, target)\n    self.cleanup()\n    \n     pert_loss\n   ():\n    self.prev = [p.requires_grad  p  self.parameters]\n     p  self.parameters:\n      p.requires_grad_()\n     m  self.adv_modules:\n      m.adversarial_()\n   ():\n     r,p  (self.prev, self.parameters):\n      p.requires_grad_(r)\n     m  self.adv_modules:\n      m.adversarial_()\n
", "votes": 0, "replies": [] } ] }, { "id": 2206853, "author_name": "qcqced", "author_url": "", "post_date": "2023-04-03T00:20:25.433000", "content": "

thank you for detailed explain… your hard work so much helpful for me

", "votes": 0, "replies": [] }, { "id": 2054661, "author_name": "", "author_url": "", "post_date": "2022-12-04T10:26:44.147000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 2051493, "postDate": "2022-12-01T13:11:33.347Z", "content": "

I'm new to the NLP and this is my first NLP competition.
\nSo I learned a lot of things from the competition.
\nThanks for hosting the competition.

\n

OverView

\n

I ensemble 15 models (with 4 folds).
\nI used orthogonal initialization, reinitialize the last 1 layer and layer wise learning rate decay for my all model.
\nThese methods also worked well for me.

\n

And Weighted Loss slightly improved CV and LB for some models.
\nThe Weighted loss is maybe my unique point.
\nI shared my code for Weighted Loss.
\nThe Weighted loss tend to improve the score of targets which can be easily predictable.
\nFor model diversity, I wanted to make two model (One is with Weighted Loss, other is without Weigted Loss) for each models, but There was no time to make all these model.

\n

Then I tried applying some method (AWP, SiFT, MIXout, and so on..) into the Deberta-v3-base model. But all of them didn't work.
\nHowever at the all most end of this competition, I found SiFT and MIXout worked well for Deberta-v3-small (Without Weighted Loss).
\nI wanted to apply these method into other models (large, xsmall, Roberta and so on), but there was no time to train…
\nSo these methods could be applied into only Deberta-v3-small model.

\n

Models used for the my best submission is below.
\n\"\"

\n

For deberta-v3-XXX set the max_len=1462, others set the max_len=512

\n

What Worked

\n\n

What Didn't Work:

\n\n

Ensemble strategy:

\n
    \n
  1. Optimize ensemble weights (target-wise) using optuna. (Trust CV)
    \nIf I use optuna for my OOF. the optuna set some Deberta-v3-base weights ZERO.
    \nI was afraid to submit, but I believed in my CV and this is the best in the PB!
    2. \n
  2. Manually optimize weights considering LB rank. (Trust LB)
    3. \n
  3. Just averaging 1. + 2. (Trust CV and LB))
    4. \n
\n

CV strategy

\n\n

Thanks.

", "rawMarkdown": "I'm new to the NLP and this is my first NLP competition.\nSo I learned a lot of things from the competition. \nThanks for hosting the competition.\n\n# **OverView**\nI ensemble 15 models (with 4 folds).\nI used orthogonal initialization, reinitialize the last 1 layer and layer wise learning rate decay for my all model.\nThese methods also worked well for me.\n\nAnd Weighted Loss slightly improved CV and LB for some models.\nThe Weighted loss is maybe my unique point.\nI shared [my code](https://www.kaggle.com/code/taruto1215/49th-place-deberta-v3-base-weightedloss/notebook) for Weighted Loss.\nThe Weighted loss tend to improve the score of targets which can be easily predictable.\nFor model diversity, I wanted to make two model (One is with Weighted Loss, other is without Weigted Loss) for each models, but There was no time to make all these model.\n\nThen I tried applying some method (AWP, SiFT, MIXout, and so on..) into the Deberta-v3-base model. But all of them didn't work.\nHowever at the all most end of this competition, I found SiFT and MIXout worked well for Deberta-v3-small (Without Weighted Loss).\nI wanted to apply these method into other models (large, xsmall, Roberta and so on), but there was no time to train...\nSo these methods could be applied into only Deberta-v3-small model.\n\nModels used for the my best submission is below.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F1f4c9ecc4ae99c559a1b43610f382a52%2F0004-12-01%2010.48.23.png?generation=1669902532219143&alt=media)\n\nFor deberta-v3-XXX set the max_len=1462, others set the max_len=512\n\n# **What Worked**\n- Orthogonal initialization [High impact]\n- Mean pooling [High impact]\n- Reinitialize the last layers [High impact]\n- Layer-wised learning rate decay [High impact] \nfor deberta-v3-base lr_mult=0.9, for the small lr_mult=0.8\n- SiFT [It depends on model (maybe it depends on hyper-parameter cause it sensitive for hyper-parameter)]\nOnly for deberta-v3-small / xsmall it works (start at 2 epoch, learning_rate=1e-3, init_perturbation=1e-2\n[https://arxiv.org/pdf/1911.03437.pdf](https://arxiv.org/pdf/1911.03437.pdf)\n[https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/sift/sift.py](https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/sift/sift.py)\n- MIXout [Medium impact for some models]\nOnly for the deberta-v3-small/xsmall, it works (mixout_prop=0.075)\n[https://arxiv.org/abs/1909.11299](https://arxiv.org/abs/1909.11299)\n- Weighted Loss [Medium impact]\nPlease reference below and [my code](https://www.kaggle.com/code/taruto1215/49th-place-deberta-v3-base-weightedloss/notebook)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F972fe868dfb919663297d2d0a4daa05d%2FWeightedLoss.png?generation=1670244987116720&alt=media)\n[https://openaccess.thecvf.com/content_cvpr_2018/papers/Kendall_Multi-Task_Learning_Using_CVPR_2018_paper.pdf](https://openaccess.thecvf.com/content_cvpr_2018/papers/Kendall_Multi-Task_Learning_Using_CVPR_2018_paper.pdf)\nI guess the weighted loss improve the score of targets which can be easily predictable.\nConsidering the characteristic of weighted loss the results reasonable.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3830852%2F4557bd4fd2607c1f6ad308cd9730a243%2F0004-12-01%2011.21.02.png?generation=1669904489190367&alt=media)\n\n- For the ensemble, below methods works (these method did not improved CV and LB)\n-- Multi-sampled dropout (For the deberta-base / large, dropout_rate=0.05)\n[https://arxiv.org/abs/1905.09788](https://arxiv.org/abs/1905.09788)\n-- AWP (Adversarial weight perturbation) (But in ensemble it work as small weight)\n[https://arxiv.org/abs/2004.05884](https://arxiv.org/abs/2004.05884)\n-- Weighted CLS pooling for the last 4 layers.\n-- Add NewLine ([BR]) token (CV improved 0.005, but LB worse, but in ensemble it works)\n\n# **What Didn't Work:**\n- Pseudo labeling using FB2+3 ( CV was much improved, but LB rank was very low, so I didn't use)\n- concatnate/weighted different layers mean pooling \n- Attention pooling\n- Masked language modeling (Pretrain)\n- SWA (Stochastic Weight Averaging)\n[https://arxiv.org/abs/1803.05407](https://arxiv.org/abs/1803.05407)\n- FGM (Fast gradient method)\n[https://arxiv.org/pdf/1806.01477.pdf](https://arxiv.org/pdf/1605.07725.pdf)\n- Random token dropout\n\n AND SO ON....\n\n# **Ensemble strategy:**\n1. Optimize ensemble weights (target-wise) using optuna. (Trust CV)\nIf I use optuna for my OOF. the optuna set some Deberta-v3-base weights ZERO. \n**I was afraid to submit, but I believed in my CV and this is the best in the PB!**\n2. Manually optimize weights considering LB rank. (Trust LB)\n3. Just averaging 1. + 2. (Trust CV and LB))\n\n# **CV strategy**\n- Just using MultilabelStratifiedKFold, Fold4, seed42\n\nThanks.\n", "votes": 8 }, { "id": 2057359, "postDate": "2022-12-07T02:56:47.403Z", "content": "

I'm curious about how you used SiFT technology in this competition, can you share the code for this part?

", "rawMarkdown": "I'm curious about how you used SiFT technology in this competition, can you share the code for this part?", "votes": 1, "replies": [ { "id": 2060658, "postDate": "2022-12-10T08:40:26.053Z", "content": "

Sorry for my delay.

\n

There are no big change from the source code.
\nI changed the KL loss into the MSE loss in the line 161.

", "rawMarkdown": "Sorry for my delay.\n\nThere are no big change from the [source code](https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/sift/sift.py).\nI changed the KL loss into the MSE loss in the line 161." }, { "id": 2060662, "postDate": "2022-12-10T08:50:43.930Z", "content": "

I changed this class like below.
\nThanks.

\n
 :\n   ():\n     adv_modules  :\n      self.adv_modules = [m  m  model.modules()  (m, PerturbationLayer)]\n    :\n      self.adv_modules = adv_modules\n    self.parameters = [p  p  model.parameters()]\n    self.model = model\n    self.rmse = RMSELoss(reduction=)\n   (): \n    self.prepare()\n     (loss_fn, ):\n      loss_fn = perturbation_loss_fns[loss_fn]\n    pert_logits = self.model(inputs) \n    pert_loss = loss_fn(pert_logits, target.detach())\n     scaler != :\n      scaler.scale(pert_loss).backward()\n    :\n      pert_loss.backward()\n     m  self.adv_modules:\n      ok = m.update_delta()\n     r,p  (self.prev, self.parameters):\n      p.requires_grad_(r)\n    pert_logits = self.model(inputs) \n    \n    pert_loss = self.rmse(pert_logits, target)\n    self.cleanup()\n    \n     pert_loss\n   ():\n    self.prev = [p.requires_grad  p  self.parameters]\n     p  self.parameters:\n      p.requires_grad_()\n     m  self.adv_modules:\n      m.adversarial_()\n   ():\n     r,p  (self.prev, self.parameters):\n      p.requires_grad_(r)\n     m  self.adv_modules:\n      m.adversarial_()\n
", "rawMarkdown": "I changed this class like below.\nThanks.\n\n```python\nclass AdversarialLearner:\n def __init__(self, model, adv_modules=None):\n if adv_modules is None:\n self.adv_modules = [m for m in model.modules() if isinstance(m, PerturbationLayer)]\n else:\n self.adv_modules = adv_modules\n self.parameters = [p for p in model.parameters()]\n self.model = model\n self.rmse = RMSELoss(reduction=\"mean\")\n def loss(self, target, loss_fn = 'symmetric-kl', inputs=None, scaler=None): # input_ids=None, attention_mask=None\n self.prepare()\n if isinstance(loss_fn, str):\n loss_fn = perturbation_loss_fns[loss_fn]\n pert_logits = self.model(inputs) #logits_fn(self.model, *wargs, **kwargs)\n pert_loss = loss_fn(pert_logits, target.detach())\n if scaler != None:\n scaler.scale(pert_loss).backward()\n else:\n pert_loss.backward()\n for m in self.adv_modules:\n ok = m.update_delta(True)\n for r,p in zip(self.prev, self.parameters):\n p.requires_grad_(r)\n pert_logits = self.model(inputs) #logits_fn(self.model, *wargs, **kwargs)\n # pert_loss = symmetric_kl(pert_logits, target)\n pert_loss = self.rmse(pert_logits, target)\n self.cleanup()\n # return pert_loss.mean()\n return pert_loss\n def prepare(self):\n self.prev = [p.requires_grad for p in self.parameters]\n for p in self.parameters:\n p.requires_grad_(False)\n for m in self.adv_modules:\n m.adversarial_(True)\n def cleanup(self):\n for r,p in zip(self.prev, self.parameters):\n p.requires_grad_(r)\n for m in self.adv_modules:\n m.adversarial_(False)\n```" } ] }, { "id": 2206853, "postDate": "2023-04-03T00:20:25.433Z", "content": "

thank you for detailed explain… your hard work so much helpful for me

", "rawMarkdown": "thank you for detailed explain... your hard work so much helpful for me" }, { "id": 2054661, "postDate": "2022-12-04T10:26:44.147Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:03:57Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9266776, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/taruto1215-49th-place-weighted-loss-etc" }