Thanks host and Kaggle for an interesting challenge. Also thanks for the three submissions choices, should be a standard hereinafter.
\nCongrats all the top solutions, will be valuable source of information, with only a quick look I can see that large model and large lengths seems like a boost to the score.
\nAs many times before I had an experiment approach, the knowledge one gets can not give massive shakeup even of one’s leaderboard place were so 😉
\nI experimented with Debertav3Base w/ 512MaxL as I used T4s it was the best, my experimenting never ended so those models were used for different solution. Sadly I didn’t pick the best CV solution with CV .460 with a final LB place ~#80, no ML errors behind that choice though.
\nShort summary of the experimenting and solutions
\nFrameworks for training
\nInput to transformers
\nDifferent combinations for different models but mainly text_sum+Prompt Question. Sometime with original text and sometimes corrected, for the autocorrection I used NeuSpell: A Neural Spelling Correction Toolkit https://github.com/neuspell/neuspell which worked well and fast with cuda and batch support.
\nExceptions for other types than text where for models trained on AutoGluon MultiModalPredictor framework. In that framework one can experiment with different fusion heads for the inputs building a complete end-to-end LM model.
Features used for final tabular prediction
\nModel for tabular prediction
\nAs said before the SOTA framework AG TabularPredictor was used. Some changes were set from the standard. Used fixed set 4 fold validation with the 4 groups as folds. Fold_fitting_strategy = sequential_local as it always works without issue. Also the new feature “experimental_zeroshot_hpo_hybrid” was used to quickly get the params based on dataset in zeroshot manner, thanks. Great work @innixma 👍
\nThen a 3-level stacked ~100 trained model ensemble did the final prediction. I also manually included the first target prediction to the next target training, a.k.a Auto-Regressive training.
Other interesting experimenting
\nNext in line somewhere for testing would have been larger models and more text lengths. Next time 😊
\nThat's it! Happy Kaggling!
", "comments": [ { "id": 2481496, "author_name": "Ehsan", "author_url": "", "post_date": "2023-10-14T07:17:35.173000", "content": "Interesting, I had a very similar approach and we both got to a relatively same position in the leaderboard 😅
", "votes": 1, "replies": [] } ], "comment_count": 1, "messages": [ { "id": 2479085, "postDate": "2023-10-12T11:53:10.200Z", "content": "Thanks host and Kaggle for an interesting challenge. Also thanks for the three submissions choices, should be a standard hereinafter.
\nCongrats all the top solutions, will be valuable source of information, with only a quick look I can see that large model and large lengths seems like a boost to the score.
\nAs many times before I had an experiment approach, the knowledge one gets can not give massive shakeup even of one’s leaderboard place were so 😉
\nI experimented with Debertav3Base w/ 512MaxL as I used T4s it was the best, my experimenting never ended so those models were used for different solution. Sadly I didn’t pick the best CV solution with CV .460 with a final LB place ~#80, no ML errors behind that choice though.
\nShort summary of the experimenting and solutions
\nFrameworks for training
\nInput to transformers
\nDifferent combinations for different models but mainly text_sum+Prompt Question. Sometime with original text and sometimes corrected, for the autocorrection I used NeuSpell: A Neural Spelling Correction Toolkit https://github.com/neuspell/neuspell which worked well and fast with cuda and batch support.
\nExceptions for other types than text where for models trained on AutoGluon MultiModalPredictor framework. In that framework one can experiment with different fusion heads for the inputs building a complete end-to-end LM model.
Features used for final tabular prediction
\nModel for tabular prediction
\nAs said before the SOTA framework AG TabularPredictor was used. Some changes were set from the standard. Used fixed set 4 fold validation with the 4 groups as folds. Fold_fitting_strategy = sequential_local as it always works without issue. Also the new feature “experimental_zeroshot_hpo_hybrid” was used to quickly get the params based on dataset in zeroshot manner, thanks. Great work @innixma 👍
\nThen a 3-level stacked ~100 trained model ensemble did the final prediction. I also manually included the first target prediction to the next target training, a.k.a Auto-Regressive training.
Other interesting experimenting
\nNext in line somewhere for testing would have been larger models and more text lengths. Next time 😊
\nThat's it! Happy Kaggling!
", "rawMarkdown": "Thanks host and Kaggle for an interesting challenge. Also thanks for the three submissions choices, should be a standard hereinafter.\n\nCongrats all the top solutions, will be valuable source of information, with only a quick look I can see that large model and large lengths seems like a boost to the score. \n\nAs many times before I had an experiment approach, the knowledge one gets can not give massive shakeup even of one’s leaderboard place were so 😉\n\nI experimented with Debertav3Base w/ 512MaxL as I used T4s it was the best, my experimenting never ended so those models were used for different solution. Sadly I didn’t pick the best CV solution with CV .460 with a final LB place ~#80, no ML errors behind that choice though.\n\n**Short summary of the experimenting and solutions**\n\n**Frameworks for training**\n\n-\tHuggingface training and inference code. Only multi-target training.\n-\tPast top solutions code – feedback-prize-english-language-learning 1st place https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369457. Multi-target training with mixout augmentation etc.\n-\tAutoGluon MultiModalPredictor and TabularPredictor – for both model and later tabular training. https://auto.gluon.ai/stable/index.html. Single target training.\n\n**Input to transformers**\n\nDifferent combinations for different models but mainly text_sum+Prompt Question. Sometime with original text and sometimes corrected, for the autocorrection I used NeuSpell: A Neural Spelling Correction Toolkit https://github.com/neuspell/neuspell which worked well and fast with cuda and batch support.\nExceptions for other types than text where for models trained on AutoGluon MultiModalPredictor framework. In that framework one can experiment with different fusion heads for the inputs building a complete end-to-end LM model.\n\n**Features used for final tabular prediction**\n\n-\tThe public created 9 features.\n-\tStacking: Predictions from every model and its ensemble value.\n-\t6 feature [cohesion, syntax, vocabulary, phraseology, grammar, conventions] from the Feedback Prize - English Language Learning competition and used 1st place code for it https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369457 Even though grammar etc. shouldn’t take in count for the true human score it helped the solution.\n-\t2 features, reading level, for the Prompt Question and Prompt Text from the CommonLit Readability Prize Competition and the 2nd place code was used https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328\n-\t4 features, value of similarity between the texts. https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2 was used for it.\n\n**Model for tabular prediction**\n\nAs said before the SOTA framework AG TabularPredictor was used. Some changes were set from the standard. Used fixed set 4 fold validation with the 4 groups as folds. Fold_fitting_strategy = sequential_local as it always works without issue. Also the new feature “experimental_zeroshot_hpo_hybrid” was used to quickly get the params based on dataset in zeroshot manner, thanks. Great work @innixma 👍\nThen a 3-level stacked ~100 trained model ensemble did the final prediction. I also manually included the first target prediction to the next target training, a.k.a Auto-Regressive training.\n\n**Other interesting experimenting**\n\n-\tPseudo label in one level worked well, taking the 4 GroupKFold prediction to 24 StratifiedGroupKFold made boost to the score.\n-\tSome new findings and own-made codes along the way:\no\tA custom-made optimizer included cosine scheduler all-in-one with also a multi-learning rate for multitargets. Also added MADGRAD instead of AdamW as it worked better in the testing.\no\tA multi-target feature to a torch model that saves the best per target states within the model .self function and ensemble it between the target states to smoothing the targets best loss closer to each other. In this way I could get a Deberta Base model CV close to Deberta Large. Didn’t included it in a final solution as it was close to deadline but did a late after deadline submission and the Base model had better private score then Large model without the feature. More testing needed of course.\n\nNext in line somewhere for testing would have been larger models and more text lengths. Next time 😊\n\nThat's it! Happy Kaggling!\n", "votes": 6 }, { "id": 2481496, "postDate": "2023-10-14T07:17:35.173Z", "content": "Interesting, I had a very similar approach and we both got to a relatively same position in the leaderboard 😅
", "rawMarkdown": "Interesting, I had a very similar approach and we both got to a relatively same position in the leaderboard 😅", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:03:15Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10848921, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/kirderf-lessons-learned-and-experiments-debertav3-" }