{ "id": 369538, "title": "20th Place Solution", "content": "

Thanks to organizers for this competition and all the competitors!
\nHere is a brief introduction of my solution with the highest Private LB score (equivalent to 20th place).

\n

Overview

\n

My solution consists of a weighted average of several BERT models using the Nelder-Mead method and pseudo-labeling with text data from feedbacks 1 and 2.

\n

my inference code

\n

What Worked

\n

High Impact Idea

\n

Pseudo-Labeling

\n

I trained deberta-v3-large on this competition data and then pseudo-labeled on the Feedback 1 and 2 data, excluding data that overlapped with this competition.
\nThe pseudo-labeled data was trained by deberta-v3-base, deberta-v3-large and deberta-v2-xlarge following a meta pseudo labels method(CV: +0.005~0.01).

\n

Nelder-Mead method

\n

The welder-mead method is one of the algorithms for optimization problems and is implemented in the minimize function of scipy.
\nThe optimal weighted average method using this technique is based on 2nd solution method in the CommonLit Competion by @takoihiraokazu.

\n

The implementation is in my solution, which is as follows:

\n
 scipy.optimize  minimize\n\ntarget_cols = [, , , , , ]\n\n ():\n    opt_preds = np.zeros((train))\n     i  ((weights)):\n        opt_preds += oof_preds[i, :, idx] * weights[i]\n    score = mean_squared_error(train[col], opt_preds, squared=)\n     score\n\nopt_oof_preds = np.zeros(((train), (target_cols)))\nopt_sub_preds = np.zeros(((test), (target_cols)))\nweight_df = []\n\n idx, col  (target_cols):\n    func =  x: optimeze_func(x, idx, col)\n    weights = [ / (oof_preds)] * (oof_preds)\n    result = minimize(func, weights,  method=)\n    weight_df.append(result.x)\n     i, weight  (result.x):\n        opt_oof_preds[:, idx] += oof_preds[i, :, idx] * weight\n        opt_sub_preds[:, idx] += sub_preds[i, :, idx] * weight\n
\n

Medium Impact Idea

\n

Ensemble of multiple BERT models

\n

The following model was used:

\n\n

The learning parameters for all models are as follows:

\n\n

What Didn't Worked

\n\n

Methods that could have been improved in accuracy if worked on

\n\n

etc.

\n

Important Citations:

\n", "comments": [ { "id": 2049931, "author_name": "gezi", "author_url": "", "post_date": "2022-11-30T12:30:42.457000", "content": "

Thanks for sharing. I'm intersted on Nelder-Mead method, how much does it improve, have you compared with like just search best weights using optuna?

", "votes": 2, "replies": [ { "id": 2050180, "author_name": "colum2131", "author_url": "", "post_date": "2022-11-30T15:17:07.380000", "content": "

Thank you for your comment!
\nEmpirically, the nelder-mead methodh can optimize faster and more accurately than optuna in weighted average of predictions for regression tasks.

", "votes": 0, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 2049895, "postDate": "2022-11-30T12:08:42.523Z", "content": "

Thanks to organizers for this competition and all the competitors!
\nHere is a brief introduction of my solution with the highest Private LB score (equivalent to 20th place).

\n

Overview

\n

My solution consists of a weighted average of several BERT models using the Nelder-Mead method and pseudo-labeling with text data from feedbacks 1 and 2.

\n

my inference code

\n

What Worked

\n

High Impact Idea

\n

Pseudo-Labeling

\n

I trained deberta-v3-large on this competition data and then pseudo-labeled on the Feedback 1 and 2 data, excluding data that overlapped with this competition.
\nThe pseudo-labeled data was trained by deberta-v3-base, deberta-v3-large and deberta-v2-xlarge following a meta pseudo labels method(CV: +0.005~0.01).

\n

Nelder-Mead method

\n

The welder-mead method is one of the algorithms for optimization problems and is implemented in the minimize function of scipy.
\nThe optimal weighted average method using this technique is based on 2nd solution method in the CommonLit Competion by @takoihiraokazu.

\n

The implementation is in my solution, which is as follows:

\n
 scipy.optimize  minimize\n\ntarget_cols = [, , , , , ]\n\n ():\n    opt_preds = np.zeros((train))\n     i  ((weights)):\n        opt_preds += oof_preds[i, :, idx] * weights[i]\n    score = mean_squared_error(train[col], opt_preds, squared=)\n     score\n\nopt_oof_preds = np.zeros(((train), (target_cols)))\nopt_sub_preds = np.zeros(((test), (target_cols)))\nweight_df = []\n\n idx, col  (target_cols):\n    func =  x: optimeze_func(x, idx, col)\n    weights = [ / (oof_preds)] * (oof_preds)\n    result = minimize(func, weights,  method=)\n    weight_df.append(result.x)\n     i, weight  (result.x):\n        opt_oof_preds[:, idx] += oof_preds[i, :, idx] * weight\n        opt_sub_preds[:, idx] += sub_preds[i, :, idx] * weight\n
\n

Medium Impact Idea

\n

Ensemble of multiple BERT models

\n

The following model was used:

\n\n

The learning parameters for all models are as follows:

\n\n

What Didn't Worked

\n\n

Methods that could have been improved in accuracy if worked on

\n\n

etc.

\n

Important Citations:

\n", "rawMarkdown": "Thanks to organizers for this competition and all the competitors!\nHere is a brief introduction of my solution with the highest Private LB score (equivalent to 20th place).\n\n# Overview\n\nMy solution consists of a weighted average of several BERT models using the Nelder-Mead method and pseudo-labeling with text data from feedbacks 1 and 2.\n\n[**my inference code**](https://www.kaggle.com/code/columbia2131/fb3-sub-exp16202122303132333638464748)\n\n# What Worked\n\n### High Impact Idea\n\n**Pseudo-Labeling**\n\nI trained deberta-v3-large on this competition data and then pseudo-labeled on the Feedback 1 and 2 data, excluding data that overlapped with this competition. \nThe pseudo-labeled data was trained by deberta-v3-base, deberta-v3-large and deberta-v2-xlarge following [a meta pseudo labels method](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/315707#:~:text=org/abs/2003.10580-,Meta%20Pseudo%20Labels,-notebook%20link%20on)(CV: +0.005~0.01).\n\n**Nelder-Mead method**\n\nThe welder-mead method is one of the algorithms for optimization problems and is implemented in the minimize function of scipy. \nThe optimal weighted average method using this technique is based on [2nd solution method in the CommonLit Competion](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328) by @takoihiraokazu.\n\nThe implementation is in [my solution](https://www.kaggle.com/code/columbia2131/fb3-sub-exp16202122303132333638464748), which is as follows:\n```python\nfrom scipy.optimize import minimize\n\ntarget_cols = ['cohesion', 'syntax', 'vocabulary', 'phraseology', 'grammar', 'conventions']\n\ndef optimeze_func(weights, idx=0, col='cohesion'):\n opt_preds = np.zeros(len(train))\n for i in range(len(weights)):\n opt_preds += oof_preds[i, :, idx] * weights[i]\n score = mean_squared_error(train[col], opt_preds, squared=False)\n return score\n\nopt_oof_preds = np.zeros((len(train), len(target_cols)))\nopt_sub_preds = np.zeros((len(test), len(target_cols)))\nweight_df = []\n\nfor idx, col in enumerate(target_cols):\n func = lambda x: optimeze_func(x, idx, col)\n weights = [1 / len(oof_preds)] * len(oof_preds)\n result = minimize(func, weights, method=\"nelder-mead\")\n weight_df.append(result.x)\n for i, weight in enumerate(result.x):\n opt_oof_preds[:, idx] += oof_preds[i, :, idx] * weight\n opt_sub_preds[:, idx] += sub_preds[i, :, idx] * weight\n\n```\n\n### Medium Impact Idea\n\n**Ensemble of multiple BERT models**\n\nThe following model was used:\n* deberta-v3-base\n* deberta-v3-large\n* deberta-large\n* muppet-roberta-large\n* roberta-base\n* roberta-large\n* funnel-transformer-large\n* funnel-transformer-xlarge\n* deberta-v2-xlarge\n\nThe learning parameters for all models are as follows:\n* max_length = 512\n* criterion = nn.SmoothL1Loss(beta=1., reduction='mean')\n* learning_rate = 1e-5\n* optimizer = AdamW(betas=(0.9, 0.98), weight_decay=2e-5)\n* scheduler = get_linear_schedule_with_warmup\n* num_warmup_steps_rate = 0.01\n* clip_grad_norm = 1000\n* all dropout = 0\n\n# What Didn't Worked\n\n* Stacking with GBDT\n\n### Methods that could have been improved in accuracy if worked on\n\n* Adversarial Weight Perturbation(AWP)\n* Layerwise Learning Rate(LLR)\n* Support Vector Regression(SVR) trained using embedding vectors of pre-trained models\n\netc.\n\n# Important Citations:\n\n* [FB3 / Deberta-v3-base baseline [train]](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) by @yasufuminakama\n* [2nd place solution](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328) in CommonLit competition by @takoihiraokazu\n", "votes": 24 }, { "id": 2049931, "postDate": "2022-11-30T12:30:42.457Z", "content": "

Thanks for sharing. I'm intersted on Nelder-Mead method, how much does it improve, have you compared with like just search best weights using optuna?

", "rawMarkdown": "Thanks for sharing. I'm intersted on Nelder-Mead method, how much does it improve, have you compared with like just search best weights using optuna? ", "votes": 2, "replies": [ { "id": 2050180, "postDate": "2022-11-30T15:17:07.380Z", "content": "

Thank you for your comment!
\nEmpirically, the nelder-mead methodh can optimize faster and more accurately than optuna in weighted average of predictions for regression tasks.

", "rawMarkdown": "Thank you for your comment!\nEmpirically, the nelder-mead methodh can optimize faster and more accurately than optuna in weighted average of predictions for regression tasks." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:00:15Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9093158, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/colum2131-20th-place-solution" }