First, my contribution to the team is mostly in the process of stacking models. So, I’d like to discuss mainly in the way of ensemble.
\nPublic LB Best Models
\n| \n | CV | \nPublic | \nPrivate | \n
|---|---|---|---|
| roberta-base | \n0.4778 | \n? | \n? | \n
| roberta-large | \n0.4697 | \n0.460 | \n0.464 | \n
| electra-large | \n0.4747 | \n0.462 | \n0.474 | \n
| mpnet-base | \n0.4816 | \n? | \n? | \n
| longformer-large | \n0.4873 | \n? | \n? | \n
・Some of our models used re-initialization layers. In ‘large’ models, the number of re-init layers is 5(‘base’ is 2).
\n・This article (https://towardsdatascience.com/pitfalls-with-dropout-and-batchnorm-in-regression-problems-39e02ce08e4d) suggests that dropout and batchnorm layers should be avoided in regression problems, so we did’t use them.
\n・Other than the models mentioned in the table, we employed roberta-base x 2, roberta-large x 12, electra-large x 2, mpnet-base x 1, and longformer-large x 1 in the private best submission, but I think there is room for improvement by narrowing down the list of models.
Stacking
\nIn stacking, we added sentence embeddings (generated by sentence_transformers) with the predictions of all models for the purpose of considering not only the predicted targets but also the characteristic of the sentence itself. CV score was the best when sentence embeddings were dimensionally reduced by kpca (n_components=2, kernel=’poly’). In ensemble, we employed the average prediction of Ridge and ARDRegression, because LGBM was good at CV (about 0.432~) but seemed to be overfitted (Public LB was worse.) and other models did not achieve so much good CV scores.
Things that did not work for us
\n・Pseudo labeling
\n・Data Augmentation
\n・Some architectures like xlnet, gpt(I think one of the reasons why we missed gold medal is that we failed to handle these architectures effectively.)
Finally, I’d like to thank all the people competing with us and sharing a lot of useful insights. Thank you for reading!
", "comments": [ { "id": 1454569, "author_name": "u++", "author_url": "", "post_date": "2021-08-06T08:45:32.630000", "content": "Post from one of the teammates: I've published my github repo for this competition. https://github.com/upura/commonlitreadabilityprize
", "votes": 2, "replies": [] }, { "id": 1454104, "author_name": "", "author_url": "", "post_date": "2021-08-06T05:14:16.367000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 1425249, "postDate": "2021-08-03T06:42:10.193Z", "content": "First, my contribution to the team is mostly in the process of stacking models. So, I’d like to discuss mainly in the way of ensemble.
\nPublic LB Best Models
\n| \n | CV | \nPublic | \nPrivate | \n
|---|---|---|---|
| roberta-base | \n0.4778 | \n? | \n? | \n
| roberta-large | \n0.4697 | \n0.460 | \n0.464 | \n
| electra-large | \n0.4747 | \n0.462 | \n0.474 | \n
| mpnet-base | \n0.4816 | \n? | \n? | \n
| longformer-large | \n0.4873 | \n? | \n? | \n
・Some of our models used re-initialization layers. In ‘large’ models, the number of re-init layers is 5(‘base’ is 2).
\n・This article (https://towardsdatascience.com/pitfalls-with-dropout-and-batchnorm-in-regression-problems-39e02ce08e4d) suggests that dropout and batchnorm layers should be avoided in regression problems, so we did’t use them.
\n・Other than the models mentioned in the table, we employed roberta-base x 2, roberta-large x 12, electra-large x 2, mpnet-base x 1, and longformer-large x 1 in the private best submission, but I think there is room for improvement by narrowing down the list of models.
Stacking
\nIn stacking, we added sentence embeddings (generated by sentence_transformers) with the predictions of all models for the purpose of considering not only the predicted targets but also the characteristic of the sentence itself. CV score was the best when sentence embeddings were dimensionally reduced by kpca (n_components=2, kernel=’poly’). In ensemble, we employed the average prediction of Ridge and ARDRegression, because LGBM was good at CV (about 0.432~) but seemed to be overfitted (Public LB was worse.) and other models did not achieve so much good CV scores.
Things that did not work for us
\n・Pseudo labeling
\n・Data Augmentation
\n・Some architectures like xlnet, gpt(I think one of the reasons why we missed gold medal is that we failed to handle these architectures effectively.)
Finally, I’d like to thank all the people competing with us and sharing a lot of useful insights. Thank you for reading!
", "rawMarkdown": "First, my contribution to the team is mostly in the process of stacking models. So, I’d like to discuss mainly in the way of ensemble.\n\n**Public LB Best Models**\n| | CV| Public| Private|\n| --- | --- |\n| roberta-base | 0.4778|?|?\n| roberta-large | 0.4697|0.460|0.464\n| electra-large | 0.4747|0.462|0.474\n| mpnet-base | 0.4816|?|?\n| longformer-large | 0.4873|?|?\n\n・Some of our models used re-initialization layers. In ‘large’ models, the number of re-init layers is 5(‘base’ is 2).\n・This article ([https://towardsdatascience.com/pitfalls-with-dropout-and-batchnorm-in-regression-problems-39e02ce08e4d](url)) suggests that dropout and batchnorm layers should be avoided in regression problems, so we did’t use them.\n・Other than the models mentioned in the table, we employed roberta-base x 2, roberta-large x 12, electra-large x 2, mpnet-base x 1, and longformer-large x 1 in the private best submission, but I think there is room for improvement by narrowing down the list of models.\n\n**Stacking**\nIn stacking, we added sentence embeddings (generated by sentence_transformers) with the predictions of all models for the purpose of considering not only the predicted targets but also the characteristic of the sentence itself. CV score was the best when sentence embeddings were dimensionally reduced by kpca (n_components=2, kernel=’poly’). In ensemble, we employed the average prediction of Ridge and ARDRegression, because LGBM was good at CV (about 0.432~) but seemed to be overfitted (Public LB was worse.) and other models did not achieve so much good CV scores.\n\n**Things that did not work for us**\n・Pseudo labeling\n・Data Augmentation\n・Some architectures like xlnet, gpt(I think one of the reasons why we missed gold medal is that we failed to handle these architectures effectively.)\n\nFinally, I’d like to thank all the people competing with us and sharing a lot of useful insights. Thank you for reading!\n", "votes": 19 }, { "id": 1454569, "postDate": "2021-08-06T08:45:32.630Z", "content": "Post from one of the teammates: I've published my github repo for this competition. https://github.com/upura/commonlitreadabilityprize
", "rawMarkdown": "Post from one of the teammates: I've published my github repo for this competition. https://github.com/upura/commonlitreadabilityprize", "votes": 2 }, { "id": 1454104, "postDate": "2021-08-06T05:14:16.367Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:46:15Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 7025112, "solution_url": "/competitions/commonlitreadabilityprize/writeups/takuoko-private-25th-place-solution" }