![\"Workflow\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16611574%2Fecf091a2f0599e26d3b37d2ddbb7d210%2FPicture1.jpg?generation=1698114172221971&alt=media\")
Thank Kaggle for hosting Evaluate Student Summaries, a competition that assesses the quality of summaries written by students in grades 3-12. Despite joining this competition just a month before the deadline and being busy with another competition, I did not achieving good results but learned a lot how to train a model to automatically evaluat the student summaries in terms of the content score (how proper the content of student summaries relevant to original texts) and wording score (spellings, grammar errors, word overlapping…).
\nBusiness context: https://www.kaggle.com/competitions/kaggle-llm-science-exam
\nData context: https://www.kaggle.com/competitions/kaggle-llm-science-exam/data
The LLM-based summary evaluation system works as follows:
\nFix misspellings/ in student summaries and collect features for evaluation.
Fine-tune a pre-trained LLM and train two models on training datasets.
Predict student summary scores in test dataset using fine-tuned LLM and LightGBM models
\nThe best solution I found for the Evaluate Student Summaries competition was a combination of Deberta-v3-base and LightGBM models. This solution achieved an accuracy of 0.488, with a training time of 3.8 hours on a P100 GPU. I have published my notebook here.
Step 1. Text preprocessing collects and fixes grammar/spelling errors in student summaries.
\nText preprocessing collects features, such as the number of incorrect words, overlapping words, and named entities (people, place or medical codes), from student summaries and original texts. It then uses pyspellchecker to find the grammar errors and fix incorrect words using autocorrect. The features include (1) text length (2) misspellings (3) N-gram overlapping (tri-, bi-gram and word) (4) grammar errors (5) quote overlapping.
Step 2. Fine-tune pretrained LLM model and then train two models with training datasets for content and wording scores.
\nFine-tune the pretrained Deberta-v3-base using 4-fold group cross-validation (GroupKFold) to predict content and wording scores for student summaries training dataset. The fine-tuned model takes prompt title [SEP] prompt question [SEP] student summary as input, and predicts their content and wording scores as output, and the evaluation is based on mean squared error.
Next, train two LightGBM models on the training dataset, one to predict content score and the other to predict wording score. Both models predict using the same fold groups, based on the features from Step 1 and the scores from fine-tuned model, and are then evaluated using the same metric.
Step 3. Predict content and wording scores using the fine-tuned and LightGBM models.
\nFor the testing dataset, we predict content and wording scores for student summaries using the fine-tuned and LightGBM models, and average the predictions to produce the final score.
I tried the below approaches apart from the final solution.
\nWe used the Deberta-V3-large model without any training to predict scores on the test dataset. The score was very poor (1.08). This indicates that LLMs require further information to make accurate predictions.
We used Sentence Transformer to extract text similarity and word diversity features from student summaries and original texts, then trained XGBoost and BERT models to predict content and wording scores. BERT outperformed XGBoost with a score of 0.59, suggesting fine-tuning can improve accuracy.
\nWe applied the FastAI tabular library to simplify the training process. We used the same text processing as the final solution to extract features and fix misspellings. Next, we used a FastAI tabular learner to train a model with layers [300, 200], using mean absolute error (MAE) as the loss function. The trainer fit the training data and obtained the optimal learning rate using 30-70 cross-validation (random splits). However, the notebook threw exceptions, even though it ran successfully and produced a submission file.
The final solution was inspired by
\nThank Kaggle for hosting Evaluate Student Summaries, a competition that assesses the quality of summaries written by students in grades 3-12. Despite joining this competition just a month before the deadline and being busy with another competition, I did not achieving good results but learned a lot how to train a model to automatically evaluat the student summaries in terms of the content score (how proper the content of student summaries relevant to original texts) and wording score (spellings, grammar errors, word overlapping…).
\nBusiness context: https://www.kaggle.com/competitions/kaggle-llm-science-exam
\nData context: https://www.kaggle.com/competitions/kaggle-llm-science-exam/data
The LLM-based summary evaluation system works as follows:
\nFix misspellings/ in student summaries and collect features for evaluation.
Fine-tune a pre-trained LLM and train two models on training datasets.
Predict student summary scores in test dataset using fine-tuned LLM and LightGBM models
\nThe best solution I found for the Evaluate Student Summaries competition was a combination of Deberta-v3-base and LightGBM models. This solution achieved an accuracy of 0.488, with a training time of 3.8 hours on a P100 GPU. I have published my notebook here.
Step 1. Text preprocessing collects and fixes grammar/spelling errors in student summaries.
\nText preprocessing collects features, such as the number of incorrect words, overlapping words, and named entities (people, place or medical codes), from student summaries and original texts. It then uses pyspellchecker to find the grammar errors and fix incorrect words using autocorrect. The features include (1) text length (2) misspellings (3) N-gram overlapping (tri-, bi-gram and word) (4) grammar errors (5) quote overlapping.
Step 2. Fine-tune pretrained LLM model and then train two models with training datasets for content and wording scores.
\nFine-tune the pretrained Deberta-v3-base using 4-fold group cross-validation (GroupKFold) to predict content and wording scores for student summaries training dataset. The fine-tuned model takes prompt title [SEP] prompt question [SEP] student summary as input, and predicts their content and wording scores as output, and the evaluation is based on mean squared error.
Next, train two LightGBM models on the training dataset, one to predict content score and the other to predict wording score. Both models predict using the same fold groups, based on the features from Step 1 and the scores from fine-tuned model, and are then evaluated using the same metric.
Step 3. Predict content and wording scores using the fine-tuned and LightGBM models.
\nFor the testing dataset, we predict content and wording scores for student summaries using the fine-tuned and LightGBM models, and average the predictions to produce the final score.
I tried the below approaches apart from the final solution.
\nWe used the Deberta-V3-large model without any training to predict scores on the test dataset. The score was very poor (1.08). This indicates that LLMs require further information to make accurate predictions.
We used Sentence Transformer to extract text similarity and word diversity features from student summaries and original texts, then trained XGBoost and BERT models to predict content and wording scores. BERT outperformed XGBoost with a score of 0.59, suggesting fine-tuning can improve accuracy.
\nWe applied the FastAI tabular library to simplify the training process. We used the same text processing as the final solution to extract features and fix misspellings. Next, we used a FastAI tabular learner to train a model with layers [300, 200], using mean absolute error (MAE) as the loss function. The trainer fit the training data and obtained the optimal learning rate using 30-70 cross-validation (random splits). However, the notebook threw exceptions, even though it ran successfully and produced a submission file.
The final solution was inspired by
\n", "rawMarkdown": "Thank Kaggle for hosting Evaluate Student Summaries, a competition that assesses the quality of summaries written by students in grades 3-12. Despite joining this competition just a month before the deadline and being busy with another competition, I did not achieving good results but learned a lot how to train a model to automatically evaluat the student summaries in terms of the *content score* (how proper the content of student summaries relevant to original texts) and *wording score* (spellings, grammar errors, word overlapping...). \n\n# Context \n\nBusiness context: https://www.kaggle.com/competitions/kaggle-llm-science-exam\nData context: https://www.kaggle.com/competitions/kaggle-llm-science-exam/data\n\n# Overview of the Approach\n\n\nThe LLM-based summary evaluation system works as follows:\n1. Fix misspellings/ in student summaries and collect features for evaluation.\n2. Fine-tune a pre-trained LLM and train two models on training datasets. \n3. Predict student summary scores in test dataset using fine-tuned LLM and LightGBM models\n\n The best solution I found for the Evaluate Student Summaries competition was a combination of `Deberta-v3-base` and `LightGBM` models. This solution achieved an accuracy of 0.488, with a training time of 3.8 hours on a P100 GPU. I have published my notebook [here] (https://www.kaggle.com/code/minhsienweng/debert-v3-base-lightgbm-model-training?scriptVersionId=146058874).\n\n**Step 1.** Text preprocessing collects and fixes grammar/spelling errors in student summaries.\nText preprocessing collects features, such as the number of incorrect words, overlapping words, and named entities (people, place or medical codes), from student summaries and original texts. It then uses `pyspellchecker` to find the grammar errors and fix incorrect words using `autocorrect`. The features include (1) text length (2) misspellings (3) N-gram overlapping (tri-, bi-gram and word) (4) grammar errors (5) quote overlapping.\n\n**Step 2.** Fine-tune pretrained LLM model and then train two models with training datasets for content and wording scores.\n\nFine-tune the pretrained `Deberta-v3-base` using 4-fold group cross-validation (`GroupKFold`) to predict content and wording scores for student summaries training dataset. The fine-tuned model takes `prompt title` [SEP] `prompt question` [SEP] `student summary` as input, and predicts their content and wording scores as output, and the evaluation is based on mean squared error. \n\nNext, train two `LightGBM` models on the training dataset, one to predict content score and the other to predict wording score. Both models predict using the same fold groups, based on the features from Step 1 and the scores from fine-tuned model, and are then evaluated using the same metric.\n\n\n**Step 3.** Predict content and wording scores using the fine-tuned and LightGBM models. \nFor the testing dataset, we predict content and wording scores for student summaries using the fine-tuned and LightGBM models, and average the predictions to produce the final score.\n\n# Details of the submission\nI tried the below approaches apart from the final solution.\n\n# Deberta-V3-large model\nWe used the `Deberta-V3-large` model without any training to predict scores on the test dataset. The score was very poor (1.08). This indicates that LLMs require further information to make accurate predictions.\n\n## Sentence Transformer + XGBoot \nWe used Sentence Transformer to extract text similarity and word diversity features from student summaries and original texts, then trained XGBoost and BERT models to predict content and wording scores. BERT outperformed XGBoost with a score of 0.59, suggesting fine-tuning can improve accuracy.\n\n## Text Preprocssing + Fast-AI training\nWe applied the FastAI tabular library to simplify the training process. We used the same text processing as the final solution to extract features and fix misspellings. Next, we used a FastAI tabular learner to train a model with layers `[300, 200]`, using mean absolute error (`MAE`) as the loss function. The trainer fit the training data and obtained the optimal learning rate using 30-70 cross-validation (random splits). However, the notebook threw exceptions, even though it ran successfully and produced a submission file.\n\n\n# Sources\nThe final solution was inspired by\n- https://www.kaggle.com/code/dangnguyen97/tuned-debertav3-lgbm\n- https://www.kaggle.com/code/yusuphmustaphaladi/implementing-fastai-tabular\n\n", "votes": 2 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:09:29Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10878555, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/min-hsien-weng-429th-place-solution-for-evaluate-s" }