{ "id": 369559, "title": "29th Place Solution", "content": "

First of all, I would like to thank Feedback competition host and all the participants.
\nI was able to test much of my knowledge from past competitions in this competition.
\nMy solution is shown below. I have reflected that I could have made more detailed tuning to each solution method, and I hope to make the most of this in the next competition.

\n

Overview

\n
    \n
  1. I had six pre-trained models train with the given train data. AWP was used in my all training.
    \nsix pre-trained models: deberta-v3-large, deberta-v3-base, deberta-v2-xlarge, deberta-large, bigbird-roberta-large, bigbird-roberta-base
    2. \n
  2. Six trained models were used to Pseudo-Label the FB1 essay texts.
    3. \n
  3. Pseudo Labels were added to the given train data and again had six pre-trained models train.
    4. \n
  4. The predictions of the six models plus SVR were ensembled by weighted average.
    5. \n
\n

What Worked

\n\n

What Didn’t Work

\n\n

This competition was a lot of fun, with the Efficiency Prize as before, and a commemorative gift for the top 50. Thank you so much, hosts!
\nI have already started warming up for FB4😉

", "comments": [ { "id": 2050146, "author_name": "ktm", "author_url": "", "post_date": "2022-11-30T15:00:42.613000", "content": "

Congratulations!!

\n
The inference time for the test data was 40 minutes. I think this was too much time. The cause is not clear at the moment.\n
\n

What value did you set the batch size to?
\nbatch_size >1 is slower than batch_size=1 on CPU because unnecessary calculations occur in the attention_mask section.

", "votes": 1, "replies": [ { "id": 2050211, "author_name": "esprit", "author_url": "", "post_date": "2022-11-30T15:31:50.040000", "content": "

Here are the results we have tried so far.
\nbatch_size=8,16 were 40 minutes.
\nbatch_size=32 was 41 minutes.
\nbatch_size=64 was 46 minutes.

\n

And here are the result of the new trial.
\nbatch_size=1 was 30 minutes.

\n

I didn't realize that batch_size=1 was the fastest. (but still slow…)
\nThanks for letting me know!

", "votes": 0, "replies": [] } ] }, { "id": 2050036, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-11-30T13:48:34.420000", "content": "

Congratulations on 32nd place, we were very close) Thanks for the competitive resistance

", "votes": 0, "replies": [ { "id": 2050115, "author_name": "esprit", "author_url": "", "post_date": "2022-11-30T14:32:42.697000", "content": "

Thank you! It was a fun competition. Let's both do our best in the next competition.

", "votes": 0, "replies": [] } ] } ], "comment_count": 4, "messages": [ { "id": 2050005, "postDate": "2022-11-30T13:29:01.883Z", "content": "

First of all, I would like to thank Feedback competition host and all the participants.
\nI was able to test much of my knowledge from past competitions in this competition.
\nMy solution is shown below. I have reflected that I could have made more detailed tuning to each solution method, and I hope to make the most of this in the next competition.

\n

Overview

\n
    \n
  1. I had six pre-trained models train with the given train data. AWP was used in my all training.
    \nsix pre-trained models: deberta-v3-large, deberta-v3-base, deberta-v2-xlarge, deberta-large, bigbird-roberta-large, bigbird-roberta-base
    2. \n
  2. Six trained models were used to Pseudo-Label the FB1 essay texts.
    3. \n
  3. Pseudo Labels were added to the given train data and again had six pre-trained models train.
    4. \n
  4. The predictions of the six models plus SVR were ensembled by weighted average.
    5. \n
\n

What Worked

\n\n

What Didn’t Work

\n\n

This competition was a lot of fun, with the Efficiency Prize as before, and a commemorative gift for the top 50. Thank you so much, hosts!
\nI have already started warming up for FB4😉

", "rawMarkdown": "First of all, I would like to thank Feedback competition host and all the participants.\nI was able to test much of my knowledge from past competitions in this competition.\nMy solution is shown below. I have reflected that I could have made more detailed tuning to each solution method, and I hope to make the most of this in the next competition.\n# Overview\n1. I had **six pre-trained models** train with the given train data. **AWP** was used in my all training.\n**six pre-trained models**: deberta-v3-large, deberta-v3-base, deberta-v2-xlarge, deberta-large, bigbird-roberta-large, bigbird-roberta-base\n2. Six trained models were used to **Pseudo-Label** the FB1 essay texts.\n3. Pseudo Labels were added to the given train data and again had six pre-trained models train.\n4. The predictions of the six models plus SVR were ensembled by weighted average.\n# What Worked\n- **AWP**\n - I have tried AWP several times in past competitions, but all failed. And for the first time, I have succeeded in improving my score with AWP.\n - The parameters worked well with adv_lr=2e-4, adv_eps=1e-3, referring to the discussion.\ncf. https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/349594#1932041\n - CV improve (single model): 0.0007\n- **Pseudo Labeling**\n - No leaks. I knew empirically from past competitions that if I am not careful with folds, they will leak.\n - CV improve (single model): 0.0003 ~ 0.0026\n- **Diversity by training pattern**\n - I was able to create diversity by dividing the training into the following 1 and 2 training patterns. These ensembles improved the score.\n1: train one epoch with Pseudo Labels, then train several epochs with given train data\n2: train several epochs with given train data and Pseudo Labels at the same time\n- **Weighted average Ensemble**\n\n | Model | Folds | CV | Public LB | Private LB\n | --- | --- | --- | --- | ---\n | deberta-v3-large | 5 | 0.4473~0.4495 | | \n | deberta-v3-base | 5 | 0.4512~0.4522 | | \n | deberta-v2-xlarge | 5 | 0.4475~0.4522 | | \n | deberta-large | 5 | 0.4510~0.4538 | | \n | bigbird-roberta-large | 5 | 0.4577 | | \n | bigbird-roberta-base | 5 | 0.4643 | | \n | svr | 5 | 0.4526 | | \n | Ensemble | | 0.4435 | 0.437183 | 0.435225\n# What Didn’t Work\n- **Down sampling of Pseudo Labels**\n - I considered some of the Pseudo Labels to be unreliable and removed those with large std (>0.44/3) of predictions for each model.\nHowever, the CV did not improve.\n - CV (Weighted average Ensemble): 0.4435 -> 0.4437\n- **Efficiency Prize Measures**\n - For Efficiency Prize, I had deberta-v3-small train with all given train data (1 fold) and all Pseudo Labels.\n - The inference time for the test data was 40 minutes, which was too long.\nI should have set `batch_size=1` and tried deberta-v3-xsmall. (Thanks to @tmhrkt for the advice)\n - **Private Score: 0.438587** (Not bad!)\n\nThis competition was a lot of fun, with the Efficiency Prize as before, and a commemorative gift for the top 50. Thank you so much, hosts!\nI have already started warming up for FB4😉", "votes": 17 }, { "id": 2050146, "postDate": "2022-11-30T15:00:42.613Z", "content": "

Congratulations!!

\n
The inference time for the test data was 40 minutes. I think this was too much time. The cause is not clear at the moment.\n
\n

What value did you set the batch size to?
\nbatch_size >1 is slower than batch_size=1 on CPU because unnecessary calculations occur in the attention_mask section.

", "rawMarkdown": "Congratulations!! \n```\nThe inference time for the test data was 40 minutes. I think this was too much time. The cause is not clear at the moment.\n```\nWhat value did you set the batch size to? \n`batch_size >1` is slower than `batch_size=1` on CPU because unnecessary calculations occur in the `attention_mask` section. \n", "votes": 1, "replies": [ { "id": 2050211, "postDate": "2022-11-30T15:31:50.040Z", "content": "

Here are the results we have tried so far.
\nbatch_size=8,16 were 40 minutes.
\nbatch_size=32 was 41 minutes.
\nbatch_size=64 was 46 minutes.

\n

And here are the result of the new trial.
\nbatch_size=1 was 30 minutes.

\n

I didn't realize that batch_size=1 was the fastest. (but still slow…)
\nThanks for letting me know!

", "rawMarkdown": "Here are the results we have tried so far.\n`batch_size=8,16` were 40 minutes.\n`batch_size=32` was 41 minutes.\n`batch_size=64` was 46 minutes.\n\nAnd here are the result of the new trial.\n`batch_size=1` was 30 minutes.\n\nI didn't realize that batch_size=1 was the fastest. (but still slow...)\nThanks for letting me know!" } ] }, { "id": 2050036, "postDate": "2022-11-30T13:48:34.420Z", "content": "

Congratulations on 32nd place, we were very close) Thanks for the competitive resistance

", "rawMarkdown": "Congratulations on 32nd place, we were very close) Thanks for the competitive resistance", "replies": [ { "id": 2050115, "postDate": "2022-11-30T14:32:42.697Z", "content": "

Thank you! It was a fun competition. Let's both do our best in the next competition.

", "rawMarkdown": "Thank you! It was a fun competition. Let's both do our best in the next competition." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:00:37Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9096931, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/esprit-29th-place-solution" }