{
"id": 370118,
"title": "Efficiency 17th Place Solution ",
"content": "Overview
\nWe are a group of three students representing Penn Data Science Group. We had a lot of fun working on this competition and thank the Competition Host and Kaggle for hosting.
\nHere goes our Efficiency Solution.
\nWe pseudolabeled the 15K essays from FB1 and FB2 using our own ensemble and then finetuned deberta-v3-xsmall for Efficiency.
\nThe ensemble has two components:
\n\n- Pretrained Embeddings: We concatenated the mean-pooled hidden states from different debertas and different layers therein and trained MLPs and KernelRidge.
\n- The hidden states add to over 20K in dimensions; interestingly, standardization / min-max scaling (which seems imperative to SVM etc.) leads to significantly worse result. This may be due to the layer normalization.
\n- KernelRidge uses
rbf as kernel, with default gamma, and with alpha set to 0.075. \n- MLP is structurally as follows:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5048151%2Ff17f5d18d746d22eff9e065799ed1823%2FWeChat%20Image_20221203000831.png?generation=1670044122478406&alt=media\")
\n- We used Ranger as optimizer for the MLP. Per recommended,
beta_1 is set to 0.95, and the learning schedule is flat for the first 60-75% and then on a cosine decay for the remaining epochs. We set the flat rate as 3e-4 for the first 10 epochs, and then used cosine decay for the final 6 epochs, finishing on a learning rate of 9e-7 (in other words, alpha is set to 3e-3). The batch size is 64, and we used a weight decay of 1e-1. \n- To stablize results, we trained 50 iterations and took the mean prediction.
\n- Finetuned Models: We kinda sucked at finetuning models, so we took kojimar's models. Specifically, we took deberta-v2-xlarge-mnli, deberta-v2-xlarge, deberta-v3-large-unscale, and 20221012-deberta-v3-large.
\n
\nThis ensemble gave us an unimpressive 594th place on the private LB w/ 0.436685 and 1061th place on the public LB w/ 0.439137.
\nThe deberta-v3-xsmall is finetuned with the 4K essays from this competition with their true labels and the 15K essays from the previous two competitions with the pseudolabels.
\nThis model gave us 17th place on the Efficiency Private LB w/ 0.442 in 1336 seconds.
\nWhat Worked for Ensemble
\nMLP & Kernel Ridge for Pretrained Embeddings
\n\n- Result: 0.446 CV as opposed to 0.451 CV for SVM.
\n- These two methods allow for a nonlinear specification whilst maintaining, if trained properly, robustness.
\n- SVM seems to be worse because of differences in objective and penalty (in direct comparison with Kernel Ridge). Actually, Bayesian Ridge with its linear specification can consistently beat SVM.
\n
\nmish and Ranger on MLP
\n\n- Result: 0.447 CV as opposed to 0.449 with swish and Ranger.
\n
\nIncluding Some Finetuned Models
\n\nWhat Worked for Efficiency
\nPsuedolabelling
\n\n- Result: The xsmall model can closely trace the CV performance of the ensemble (I don't have the data here because we could not cross-validate the entire ensemble, but just take my words that an xsmall trained with only the 4K essays do worse).
\n- Knowledge distillation.
\n
\ndeberta-v3-xsmall
\n\n- deberta-v3-xsmall performs better than deberta-v3-small, because it has 12 layers with hidden dimension as 384 as opposed to 6 layers with 768. The training/inference time is about the same, despite the increased layer counts.
\n- Interestingly, a base model I late-submitted performed much worse at 0.44674 private LB.
\n
\nEdit: Weight decay on weights
\n\n- Place no weight decay on the bias and normalization terms; useful for both finetuning and the training of the MLP.
\n
\nWhat Did Not Work
\n\n- Translation for data augmentation.
\n- Finetuning techniques (Multi Sample Dropout, Layer Weight Reinitialization, Layerwise Learning Rate Adaptation).
\n- I cannot quantify their improvements, and it seems that the increased sample size due to pseudolabelling offsets some benefits from these techniques.
\n- I submitted a version without MSD on the last date (but forgot to select it 😫); that had a private LB of 0.44 with probably a lower runtime (I didn't bother to prune the dropout, so the model technically stills feeds forward the last layer 8 times before taking an average); then again, I also changed some hyperparameters (lowering weight decay from 1e-1 to 1e-2 and increasing
beta_1 from 0.9 to 0.95).
\n
\nWhat we should have done
\n\n- Use PyTorch? It inferences faster and without the pain of memory leakage. This would also allow for the usage efficienct data pipelines from huggingface. I suspect sub-1000s would need this and some more.
\n
\nImportant Citations
\n\n- kojimar: Excellently finetuned models; we probably should have taken more models from you.
\n- Chris Deotte: As far as I can tell, the first person to share using pretrained embeddings.
\n- Ranger: Robust and reliable optimizer.
\n
\nThanks and Acknowledgement
\nWe thank the Competition Host and Kaggle for hosting and all those whom we cited for their insights.
\nTeam Members
\n",
"comments": [],
"comment_count": 0,
"messages": [
{
"id": 2053349,
"postDate": "2022-12-03T06:26:57.173Z",
"content": "Overview
\nWe are a group of three students representing Penn Data Science Group. We had a lot of fun working on this competition and thank the Competition Host and Kaggle for hosting.
\nHere goes our Efficiency Solution.
\nWe pseudolabeled the 15K essays from FB1 and FB2 using our own ensemble and then finetuned deberta-v3-xsmall for Efficiency.
\nThe ensemble has two components:
\n\n- Pretrained Embeddings: We concatenated the mean-pooled hidden states from different debertas and different layers therein and trained MLPs and KernelRidge.
\n- The hidden states add to over 20K in dimensions; interestingly, standardization / min-max scaling (which seems imperative to SVM etc.) leads to significantly worse result. This may be due to the layer normalization.
\n- KernelRidge uses
rbf as kernel, with default gamma, and with alpha set to 0.075. \n- MLP is structurally as follows:
\n \n- We used Ranger as optimizer for the MLP. Per recommended,
beta_1 is set to 0.95, and the learning schedule is flat for the first 60-75% and then on a cosine decay for the remaining epochs. We set the flat rate as 3e-4 for the first 10 epochs, and then used cosine decay for the final 6 epochs, finishing on a learning rate of 9e-7 (in other words, alpha is set to 3e-3). The batch size is 64, and we used a weight decay of 1e-1. \n- To stablize results, we trained 50 iterations and took the mean prediction.
\n- Finetuned Models: We kinda sucked at finetuning models, so we took kojimar's models. Specifically, we took deberta-v2-xlarge-mnli, deberta-v2-xlarge, deberta-v3-large-unscale, and 20221012-deberta-v3-large.
\n
\nThis ensemble gave us an unimpressive 594th place on the private LB w/ 0.436685 and 1061th place on the public LB w/ 0.439137.
\nThe deberta-v3-xsmall is finetuned with the 4K essays from this competition with their true labels and the 15K essays from the previous two competitions with the pseudolabels.
\nThis model gave us 17th place on the Efficiency Private LB w/ 0.442 in 1336 seconds.
\nWhat Worked for Ensemble
\nMLP & Kernel Ridge for Pretrained Embeddings
\n\n- Result: 0.446 CV as opposed to 0.451 CV for SVM.
\n- These two methods allow for a nonlinear specification whilst maintaining, if trained properly, robustness.
\n- SVM seems to be worse because of differences in objective and penalty (in direct comparison with Kernel Ridge). Actually, Bayesian Ridge with its linear specification can consistently beat SVM.
\n
\nmish and Ranger on MLP
\n\n- Result: 0.447 CV as opposed to 0.449 with swish and Ranger.
\n
\nIncluding Some Finetuned Models
\n\nWhat Worked for Efficiency
\nPsuedolabelling
\n\n- Result: The xsmall model can closely trace the CV performance of the ensemble (I don't have the data here because we could not cross-validate the entire ensemble, but just take my words that an xsmall trained with only the 4K essays do worse).
\n- Knowledge distillation.
\n
\ndeberta-v3-xsmall
\n\n- deberta-v3-xsmall performs better than deberta-v3-small, because it has 12 layers with hidden dimension as 384 as opposed to 6 layers with 768. The training/inference time is about the same, despite the increased layer counts.
\n- Interestingly, a base model I late-submitted performed much worse at 0.44674 private LB.
\n
\nEdit: Weight decay on weights
\n\n- Place no weight decay on the bias and normalization terms; useful for both finetuning and the training of the MLP.
\n
\nWhat Did Not Work
\n\n- Translation for data augmentation.
\n- Finetuning techniques (Multi Sample Dropout, Layer Weight Reinitialization, Layerwise Learning Rate Adaptation).
\n- I cannot quantify their improvements, and it seems that the increased sample size due to pseudolabelling offsets some benefits from these techniques.
\n- I submitted a version without MSD on the last date (but forgot to select it 😫); that had a private LB of 0.44 with probably a lower runtime (I didn't bother to prune the dropout, so the model technically stills feeds forward the last layer 8 times before taking an average); then again, I also changed some hyperparameters (lowering weight decay from 1e-1 to 1e-2 and increasing
beta_1 from 0.9 to 0.95).
\n
\nWhat we should have done
\n\n- Use PyTorch? It inferences faster and without the pain of memory leakage. This would also allow for the usage efficienct data pipelines from huggingface. I suspect sub-1000s would need this and some more.
\n
\nImportant Citations
\n\n- kojimar: Excellently finetuned models; we probably should have taken more models from you.
\n- Chris Deotte: As far as I can tell, the first person to share using pretrained embeddings.
\n- Ranger: Robust and reliable optimizer.
\n
\nThanks and Acknowledgement
\nWe thank the Competition Host and Kaggle for hosting and all those whom we cited for their insights.
\nTeam Members
\n",
"rawMarkdown": "# Overview\nWe are a group of three students representing [Penn Data Science Group](https://penndsg.com/). We had a lot of fun working on this competition and thank the Competition Host and Kaggle for hosting. \n\nHere goes our Efficiency Solution. \n\nWe pseudolabeled the 15K essays from FB1 and FB2 using our own ensemble and then finetuned deberta-v3-xsmall for Efficiency.\nThe ensemble has two components:\n1. **Pretrained Embeddings:** We concatenated the mean-pooled hidden states from different debertas and different layers therein and trained MLPs and KernelRidge.\n - The hidden states add to over 20K in dimensions; interestingly, standardization / min-max scaling (which seems imperative to SVM etc.) leads to significantly worse result. This may be due to the layer normalization.\n - KernelRidge uses `rbf` as kernel, with default `gamma`, and with `alpha` set to 0.075.\n - MLP is structurally as follows:\n\n - We used [Ranger](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer) as optimizer for the MLP. Per recommended, `beta_1` is set to 0.95, and the learning schedule is flat for the first 60-75% and then on a cosine decay for the remaining epochs. We set the flat rate as `3e-4` for the first 10 epochs, and then used cosine decay for the final 6 epochs, finishing on a learning rate of `9e-7` (in other words, `alpha` is set to `3e-3`). The batch size is 64, and we used a weight decay of `1e-1`.\n - To stablize results, we trained 50 iterations and took the mean prediction.\n2. **Finetuned Models:** We kinda sucked at finetuning models, so we took [kojimar](https://www.kaggle.com/kojimar)'s models. Specifically, we took [deberta-v2-xlarge-mnli](https://www.kaggle.com/datasets/kojimar/0919-deberta-v2-xlarge-mnli), [deberta-v2-xlarge](https://www.kaggle.com/datasets/kojimar/0919-deberta-v2-xlarge), [deberta-v3-large-unscale](https://www.kaggle.com/datasets/kojimar/0926-deberta-v3-large-unscale), and [20221012-deberta-v3-large](https://www.kaggle.com/datasets/kojimar/20221012-123357-deberta-v3-large).\n\nThis ensemble gave us an unimpressive 594th place on the private LB w/ 0.436685 and 1061th place on the public LB w/ 0.439137.\n\nThe deberta-v3-xsmall is finetuned with the 4K essays from this competition with their true labels and the 15K essays from the previous two competitions with the pseudolabels.\n\nThis model gave us 17th place on the Efficiency Private LB w/ 0.442 in 1336 seconds.\n\n# What Worked for Ensemble\n**MLP & Kernel Ridge for Pretrained Embeddings**\n- Result: 0.446 CV as opposed to 0.451 CV for SVM.\n- These two methods allow for a nonlinear specification whilst maintaining, if trained properly, robustness. \n- SVM seems to be worse because of differences in objective and penalty (in direct comparison with Kernel Ridge). Actually, Bayesian Ridge with its linear specification can consistently beat SVM.\n\n**mish and Ranger on MLP**\n- Result: 0.447 CV as opposed to 0.449 with swish and Ranger.\n\n**Including *Some* Finetuned Models**\n- Results: 0.445 CV.\n\n# What Worked for Efficiency\n**Psuedolabelling**\n- Result: The xsmall model can closely trace the CV performance of the ensemble (I don't have the data here because we could not cross-validate the entire ensemble, but just take my words that an xsmall trained with only the 4K essays do worse).\n- Knowledge distillation.\n\n**deberta-v3-xsmall**\n- deberta-v3-xsmall performs better than deberta-v3-small, because it has 12 layers with hidden dimension as 384 as opposed to 6 layers with 768. The training/inference time is about the same, despite the increased layer counts.\n- Interestingly, a base model I late-submitted performed much worse at 0.44674 private LB.\n\nEdit: **Weight decay on weights**\n- Place no weight decay on the bias and normalization terms; useful for both finetuning and the training of the MLP.\n\n# What Did Not Work\n- Translation for data augmentation.\n- Finetuning techniques (Multi Sample Dropout, Layer Weight Reinitialization, Layerwise Learning Rate Adaptation). \n - I cannot quantify their improvements, and it seems that the increased sample size due to pseudolabelling offsets some benefits from these techniques.\n - I submitted a version without MSD on the last date (but forgot to select it 😫); that had a private LB of 0.44 with probably a lower runtime (I didn't bother to prune the dropout, so the model technically stills feeds forward the last layer 8 times before taking an average); then again, I also changed some hyperparameters (lowering weight decay from 1e-1 to 1e-2 and increasing `beta_1` from 0.9 to 0.95).\n\n# What we should have done\n- Use PyTorch? It inferences faster and without the pain of memory leakage. This would also allow for the usage efficienct data pipelines from huggingface. I suspect sub-1000s would need this and some more.\n\n# Important Citations\n- [kojimar](https://www.kaggle.com/kojimar): Excellently finetuned models; we probably should have taken more models from you.\n- [Chris Deotte](https://www.kaggle.com/cdeotte): As far as I can tell, the first person to share using pretrained embeddings.\n- [Ranger](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer): Robust and reliable optimizer.\n\n# Thanks and Acknowledgement\nWe thank the Competition Host and Kaggle for hosting and all those whom we cited for their insights.\n\n# Team Members\n- [Randy Li](https://www.kaggle.com/randyli2)\n- [Mangara Paul Alexander Hutagalung](https://www.kaggle.com/mangarahutagalung)\n- [Chris Sebastian](https://www.kaggle.com/msgchris)",
"votes": 6
}
],
"complete": true,
"source": "live",
"fetched_at": "2026-10-19T20:05:03Z",
"competition_id": 38321,
"competition_slug": "feedback-prize-english-language-learning",
"team_id": 9131827,
"solution_url": "/competitions/feedback-prize-english-language-learning/writeups/pdsg-efficiency-17th-place-solution"
}