Thank you to Kaggle and Commonlit for providing this opportunity.
\nAlso thanks to @rhtsingh and @andretugan for sharing excellent kernels.
\nWe had 24 large models and 3 base models, utilized full 3 hours of runtime. We stacked using SGD Regressor and TF-IDF features. Models in our stack described below :
\nCross Validation - All models trained with 5 folds using kincaid readability feature. All our models were using 3 seeds (42, 77, 2021). Even with 3 seeds we noticed a LB difference of ~0.005 but 3 seeds were the max we could do. Nothing specific for each fold - all folds were trained with the same head/parameters.
\nRoberta Large (3 Roberta large with 3 seeds each - total 9 models)- All Roberta models were using pretrained model using Competition data, Kids frontiers data and Simple wiki.
\nDeberta Large (3 Deberta large with 3 seeds each - total 9 models)
\nAll Deberta models were trained for 25 epochs with validation at the end. No pretraining.
Electra Large - Attention head - along with TF-IDF features with TabNet
\nErnie - Attention head - along with TF-IDF features with TabNet.
\nRoberta Base - (Pretraining on competition data) This model was almost same as public kernel, we added predictions from LGBM and Ridge to this model
\nOur best model was Roberta Large with public LB score 0.461 (CV: 0.473) and everything else was between 0.461 and 0.473. TF-IDF features gave us a boost of ~0.002 in every model.
\nThings that didnโt work -
\n-Pretraining with other datasets for Roberta Large - we tried pretraining with many many other datasets
\n-Pseudo label with Roberta Large - It worked on private LB but it was not in our stack.
\n-Pretraining for Ernie
\n-Many many different architectures and combinations of different architectures.
Disappointed that we missed gold zone by one rank, it was a pleasure collaborating with the team @kpriyanshu256 @aman1391 @gyanendradas and Adarash S - Thank you.
", "comments": [ { "id": 1445478, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-04T09:03:15.683000", "content": "Congrats on results and thanks for sharing solution @rashmibanthia and team.
", "votes": 3, "replies": [] }, { "id": 1431496, "author_name": "AK", "author_url": "", "post_date": "2021-08-03T12:33:36.797000", "content": "Thank you @rashmibanthia for the write up .. it is always an amazing experience working and collabrating with you . Always looking forward to work with you all again!!
\nPS : Gold is long overdue ๐
", "votes": 3, "replies": [] }, { "id": 1429515, "author_name": "Priyanshu Kumar", "author_url": "", "post_date": "2021-08-03T10:45:03.123000", "content": "Thank you @rashmibanthia for the write up. It was a great learning experience working with you all. Looking forward to working with you all again!!
", "votes": 3, "replies": [ { "id": 1430071, "author_name": "RB", "author_url": "", "post_date": "2021-08-03T11:16:15.610000", "content": "Likewise, Thank you !
", "votes": 3, "replies": [] } ] }, { "id": 1454172, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-06T05:49:18.980000", "content": "Good job and congratulations!
\nDisappointed that we missed gold zone by one rank\nscoreboard is not finalized yet. Keep your fingers crossed :)
", "votes": 4, "replies": [ { "id": 1454198, "author_name": "AK", "author_url": "", "post_date": "2021-08-06T06:00:37.643000", "content": "Thanks for the optimistic reply , this is what our team need the most post competition ๐
", "votes": 3, "replies": [] } ] }, { "id": 1449739, "author_name": "HinePo", "author_url": "", "post_date": "2021-08-05T00:26:55.740000", "content": "Interesting that you used TabNetRegressor. I tried this in the 2nd month of the competition, with both roberta-base and roberta-large. The results were terrible and I soon abandoned this idea ๐ . I guess I should have spent more time tuning it or trying with embeddings from other modelsโฆ it never occurred to me using TF-IDF though
", "votes": 1, "replies": [ { "id": 1449805, "author_name": "RB", "author_url": "", "post_date": "2021-08-05T00:53:44.733000", "content": "We put TF-IDF in stacking also and that gave us ~0.001 improvement.
", "votes": 1, "replies": [] } ] }, { "id": 1446913, "author_name": "Optimo", "author_url": "", "post_date": "2021-08-04T13:29:14.170000", "content": "Hello,
\nCongrats, keep going!
\nI'm just wondering what do you mean by along with TF-IDF features with TabNet ?
Could you explain a bit more please?
", "votes": 1, "replies": [ { "id": 1447245, "author_name": "RB", "author_url": "", "post_date": "2021-08-04T14:28:04.160000", "content": "We first created TF-IDF features (along with other readability features). Then instead of using a linear layer we used Tabnet.
\nThis was in our __init__(...) model.
self.tabnet = TabNet(**{\n 'n_d': 4,\n 'n_a': 4,\n 'n_steps': 2,\n 'input_dim': feature_len,\n 'output_dim': feature_hidden_size\n }\n )\nInteresting, so the boost of 0.002 you are talking about is comparing TF-IDF with an MLP+Bert vs TabNet+Bert?
", "votes": 1, "replies": [] }, { "id": 1448018, "author_name": "Priyanshu Kumar", "author_url": "", "post_date": "2021-08-04T16:31:53.950000", "content": "We had experimented with features and 100 TF-IDF features first with Deberta-Large and an MLP for the features. The features in themselves provide a boost. We observed extra boost when we replaced the MLP with TabNet. Later, we observed the boost in all the models with this sort of architecture.
", "votes": 3, "replies": [] } ] } ], "comment_count": 12, "messages": [ { "id": 1429297, "postDate": "2021-08-03T10:31:06.420Z", "content": "Thank you to Kaggle and Commonlit for providing this opportunity.
\nAlso thanks to @rhtsingh and @andretugan for sharing excellent kernels.
\nWe had 24 large models and 3 base models, utilized full 3 hours of runtime. We stacked using SGD Regressor and TF-IDF features. Models in our stack described below :
\nCross Validation - All models trained with 5 folds using kincaid readability feature. All our models were using 3 seeds (42, 77, 2021). Even with 3 seeds we noticed a LB difference of ~0.005 but 3 seeds were the max we could do. Nothing specific for each fold - all folds were trained with the same head/parameters.
\nRoberta Large (3 Roberta large with 3 seeds each - total 9 models)- All Roberta models were using pretrained model using Competition data, Kids frontiers data and Simple wiki.
\nDeberta Large (3 Deberta large with 3 seeds each - total 9 models)
\nAll Deberta models were trained for 25 epochs with validation at the end. No pretraining.
Electra Large - Attention head - along with TF-IDF features with TabNet
\nErnie - Attention head - along with TF-IDF features with TabNet.
\nRoberta Base - (Pretraining on competition data) This model was almost same as public kernel, we added predictions from LGBM and Ridge to this model
\nOur best model was Roberta Large with public LB score 0.461 (CV: 0.473) and everything else was between 0.461 and 0.473. TF-IDF features gave us a boost of ~0.002 in every model.
\nThings that didnโt work -
\n-Pretraining with other datasets for Roberta Large - we tried pretraining with many many other datasets
\n-Pseudo label with Roberta Large - It worked on private LB but it was not in our stack.
\n-Pretraining for Ernie
\n-Many many different architectures and combinations of different architectures.
Disappointed that we missed gold zone by one rank, it was a pleasure collaborating with the team @kpriyanshu256 @aman1391 @gyanendradas and Adarash S - Thank you.
", "rawMarkdown": "Thank you to Kaggle and Commonlit for providing this opportunity. \n\nAlso thanks to @rhtsingh and @andretugan for sharing excellent kernels. \n\nWe had 24 large models and 3 base models, utilized full 3 hours of runtime. We stacked using SGD Regressor and TF-IDF features. Models in our stack described below : \n\n**Cross Validation** - All models trained with 5 folds using kincaid readability feature. All our models were using 3 seeds (42, 77, 2021). Even with 3 seeds we noticed a LB difference of ~0.005 but 3 seeds were the max we could do. Nothing specific for each fold - all folds were trained with the same head/parameters. \n\n**Roberta Large** (3 Roberta large with 3 seeds each - total 9 models)- All Roberta models were using pretrained model using Competition data, Kids frontiers data and [Simple wiki](https://github.com/LGDoor/Dump-of-Simple-English-Wiki). \n- Attention head - along with TF-IDF features with TabNet - differential learning rates \n- weighted layer pooling - multisample dropouts, No differential learning rates \n\n**Deberta Large** (3 Deberta large with 3 seeds each - total 9 models) \nAll Deberta models were trained for 25 epochs with validation at the end. No pretraining. \n- Deberta Large with pseudo labeling on [Text Simplification dataset](https://cs.pomona.edu/~dkauchak/simplification/) \n- Deberta large with pooled output, attention head and multisample dropouts along with TabNet for TF-IDF features. \n\tFor the multi-sample dropout architecture, the model outputs 5 values, the mean of which correspond to our targets and their standard deviation to the standard error. Thus, both the quantities are used for training.\n\n**Electra Large** - Attention head - along with TF-IDF features with TabNet\n\n**Ernie** - Attention head - along with TF-IDF features with TabNet. \n\n**Roberta Base** - (Pretraining on competition data) This model was almost same as [public kernel](https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch), we added predictions from LGBM and Ridge to this model\n\nOur best model was Roberta Large with public LB score 0.461 (CV: 0.473) and everything else was between 0.461 and 0.473. TF-IDF features gave us a boost of ~0.002 in every model. \n\nThings that didnโt work - \n-Pretraining with other datasets for Roberta Large - we tried pretraining with many many other datasets \n-Pseudo label with Roberta Large - It worked on private LB but it was not in our stack. \n-Pretraining for Ernie \n-Many many different architectures and combinations of different architectures. \n\nDisappointed that we missed gold zone by one rank, it was a pleasure collaborating with the team @kpriyanshu256 @aman1391 @gyanendradas and Adarash S - Thank you. ", "votes": 17 }, { "id": 1445478, "postDate": "2021-08-04T09:03:15.683Z", "content": "Congrats on results and thanks for sharing solution @rashmibanthia and team.
", "rawMarkdown": "Congrats on results and thanks for sharing solution @rashmibanthia and team.", "votes": 3 }, { "id": 1431496, "postDate": "2021-08-03T12:33:36.797Z", "content": "Thank you @rashmibanthia for the write up .. it is always an amazing experience working and collabrating with you . Always looking forward to work with you all again!!
\nPS : Gold is long overdue ๐
", "rawMarkdown": "Thank you @rashmibanthia for the write up .. it is always an amazing experience working and collabrating with you . Always looking forward to work with you all again!!\n\nPS : Gold is long overdue ๐ ", "votes": 3 }, { "id": 1429515, "postDate": "2021-08-03T10:45:03.123Z", "content": "Thank you @rashmibanthia for the write up. It was a great learning experience working with you all. Looking forward to working with you all again!!
", "rawMarkdown": "Thank you @rashmibanthia for the write up. It was a great learning experience working with you all. Looking forward to working with you all again!!", "votes": 3, "replies": [ { "id": 1430071, "postDate": "2021-08-03T11:16:15.610Z", "content": "Likewise, Thank you !
", "rawMarkdown": "Likewise, Thank you !", "votes": 3 } ] }, { "id": 1454172, "postDate": "2021-08-06T05:49:18.980Z", "content": "Good job and congratulations!
\nDisappointed that we missed gold zone by one rank\nscoreboard is not finalized yet. Keep your fingers crossed :)
", "rawMarkdown": "Good job and congratulations!\n\n```\nDisappointed that we missed gold zone by one rank\n\n```\nscoreboard is not finalized yet. Keep your fingers crossed :)", "votes": 4, "replies": [ { "id": 1454198, "postDate": "2021-08-06T06:00:37.643Z", "content": "Thanks for the optimistic reply , this is what our team need the most post competition ๐
", "rawMarkdown": "Thanks for the optimistic reply , this is what our team need the most post competition ๐", "votes": 3 } ] }, { "id": 1449739, "postDate": "2021-08-05T00:26:55.740Z", "content": "Interesting that you used TabNetRegressor. I tried this in the 2nd month of the competition, with both roberta-base and roberta-large. The results were terrible and I soon abandoned this idea ๐ . I guess I should have spent more time tuning it or trying with embeddings from other modelsโฆ it never occurred to me using TF-IDF though
", "rawMarkdown": "Interesting that you used TabNetRegressor. I tried this in the 2nd month of the competition, with both roberta-base and roberta-large. The results were terrible and I soon abandoned this idea ๐ . I guess I should have spent more time tuning it or trying with embeddings from other models... it never occurred to me using TF-IDF though", "votes": 1, "replies": [ { "id": 1449805, "postDate": "2021-08-05T00:53:44.733Z", "content": "We put TF-IDF in stacking also and that gave us ~0.001 improvement.
", "rawMarkdown": "We put TF-IDF in stacking also and that gave us ~0.001 improvement. ", "votes": 1 } ] }, { "id": 1446913, "postDate": "2021-08-04T13:29:14.170Z", "content": "Hello,
\nCongrats, keep going!
\nI'm just wondering what do you mean by along with TF-IDF features with TabNet ?
Could you explain a bit more please?
", "rawMarkdown": "Hello,\n\nCongrats, keep going!\n\nI'm just wondering what do you mean by `along with TF-IDF features with TabNet` ?\n\nCould you explain a bit more please?", "votes": 1, "replies": [ { "id": 1447245, "postDate": "2021-08-04T14:28:04.160Z", "content": "We first created TF-IDF features (along with other readability features). Then instead of using a linear layer we used Tabnet.
\nThis was in our __init__(...) model.
self.tabnet = TabNet(**{\n 'n_d': 4,\n 'n_a': 4,\n 'n_steps': 2,\n 'input_dim': feature_len,\n 'output_dim': feature_hidden_size\n }\n )\nInteresting, so the boost of 0.002 you are talking about is comparing TF-IDF with an MLP+Bert vs TabNet+Bert?
", "rawMarkdown": "Interesting, so the boost of 0.002 you are talking about is comparing TF-IDF with an MLP+Bert vs TabNet+Bert? ", "votes": 1 }, { "id": 1448018, "postDate": "2021-08-04T16:31:53.950Z", "content": "We had experimented with features and 100 TF-IDF features first with Deberta-Large and an MLP for the features. The features in themselves provide a boost. We observed extra boost when we replaced the MLP with TabNet. Later, we observed the boost in all the models with this sort of architecture.
", "rawMarkdown": "We had experimented with features and 100 TF-IDF features first with Deberta-Large and an MLP for the features. The features in themselves provide a boost. We observed extra boost when we replaced the MLP with TabNet. Later, we observed the boost in all the models with this sort of architecture.", "votes": 3 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:47:42Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6874543, "solution_url": "/competitions/commonlitreadabilityprize/writeups/randomness-to-the-top-pikachu-18th-place-solution" }