Thank you Learning Agency Lab and Kaggle for hosting this competition. We are relieved 😌 to survive the shake up.
\nContext: www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview/description
\nData: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data
\nOur selected submission had 17 diverse models. We used stacking with LGBM.
\nWe used one prompt per fold cv strategy. All our models had collate function which speeds up training as well as inference. Lower batch sizes were helpful as well. Most models had evaluation at the end of epoch, few models had evaluation multiple times during epoch. Many of the diverse and smaller models were distilled (described below).
\nAll of us were able to contribute models to the stack, some of us never checked individual models scores because public LB seemed unstable. Our stack seemed more stable than individual models.
\nApart from model predictions we had following features in stack:
\nword_overlap_count\nbigram_overlap_count \nbigram_overlap_ratio \ntrigram_overlap_count\ntrigram_overlap_ratio\n\n- Text stat features\nOsman\nGutierrez_polini\nDale_chall_readability_score\nflesch_kincaid_grade\nHere we describe each of the models :
\n(A) Deberta v3 large - full prompt text psuedo labeled model [ar3]
\nThis model had best cv (0.4721) in stack - This was trained with max len 1790, attention head on top. This was also using back translated pseudo labeled dataset. Every sample was translated to german and then back to english. We randomly selected 50% of this data for training. So every training fold had different ~3100 samples that were pseudo labeled , validation set had no psuedo labeled samples.
\nLoss function for this model was combination of SmoothL1Loss, RMSE Loss and RankLoss - @fightingmuscle described the loss function here
\nInput for this model - [ANSWER_START] + summary_text + [ANSWER_END] + [PROMPT_START] + prompt_text [PROMPT_END] + [QUESTION_START] + prompt_question + [QUESTION_END]
\nInference maxlen = 1024
(B)
\nThese were the same models with different seeds and mean pooled, input to these were summary_text + prompt_question, RMSE loss, validation end of epoch.
\n(C)
\nThese were the same models with different seeds and with CLS token, input to these were summary_text + prompt_question, MSE Loss,validation end of epoch. I added extra text “Summarize:” to prompt question , if we look 👀 at the data 🔎carefully, notice that there are prompts with \"cite evidence\" and these have much longer summary texts and it doesn't say summarize. After reading other solutions, should have tried some more variations to this.
\n(D)
\nAll these models were distilled - 50% oof labels and 50% true labels, with AWP added - only summary_text as input, validated multiple times during epoch.
\n(E)
\nDeberta v3 large [ar2] - Order of input to this model was different than the remaining models - text + SEP + prompt_question
Deberta v3 Base and XSmall models [ar4, ar5, ar7] - These were distilled models. We utilized oof labels from a previous stack. These models were trained using oof labels where as validation set remained as true labels.
\nBigram Signal -
\n\noutputs = .model(input_ids, attention_mask, token_type_ids)\nbigram_signal = outputs.last_hidden_state[, , ] * outputs.last_hidden_state[, , ]\nfeature = bigram_signal[, , ]\n feature\n(F) Deberta v3 large - Classification - Attention Head , Mean Pooling - (4x) [ar6]
\nAfter analyzing data, we noticed there were 1134 distinct categories of content and wording, so we added a classification model with BCEWithLogitsLoss. We trained 4 different models with different heads, averaged them together and then added to the stack.
We haven't understood how or why the splits for training/public lb/private lb were created. The distribution of prompts for training and public lb is very different than private lb. Very long prompt text in public and training set but not in private lb. Also number of summary text per prompt is very different for training/public LB and private LB, unfortunately these resulted in shake down for many many teams.
\nMany other things tried but were not part of selected submission
\nAcknowledgements
\nThank you @kononenko for this compliant textstat kernel
Thank you @tsunotsuno for LGBM features
\n🙏 Thank you to my amazing teammates, grateful to have worked together on this with you all.
\nCongratulations on your results!! Thank you for sharing your solution.
\n\n\nThis was also using back translated pseudo labeled dataset. Every sample was translated to german and then back to english.
\n
Does this mean that you translated every summary text to German and then back to English, and you assigned pseudo labels to the data using a model trained previously? In short, you did not use the original labels of the summary texts that were back-translated?
", "votes": 1, "replies": [ { "id": 2492722, "author_name": "RB", "author_url": "", "post_date": "2023-10-22T20:04:01.697000", "content": "Thank you for your question, This model had all 7165 samples provided in the competition + back translated pseudo labeled samples.
\nSo every fold had ~9000 training samples (~6000 provided by the competition + ~3000 back translated samples )
Thank you for the information. Just to clarify, when you mention 'back translated pseudo labeled samples,' did you use pseudo labels for these samples, or were the original labels used for the back-translated samples? In other words, were the labels assigned based on the original data or through a pseudo labeling process?
", "votes": 0, "replies": [] }, { "id": 2493440, "author_name": "RB", "author_url": "", "post_date": "2023-10-23T12:14:44.707000", "content": "back translated samples were pseudo labeled
", "votes": 1, "replies": [] }, { "id": 2493596, "author_name": "Yusuke Nakayama", "author_url": "", "post_date": "2023-10-23T13:49:49.010000", "content": "Thank you. I assigned the original labels to the back-translated samples, but this was not effective. I appreciate the insight!
", "votes": 2, "replies": [] } ] } ] } ], "comment_count": 5, "messages": [ { "id": 2479927, "postDate": "2023-10-13T02:25:46.567Z", "content": "Thank you Learning Agency Lab and Kaggle for hosting this competition. We are relieved 😌 to survive the shake up.
\nContext: www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview/description
\nData: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data
\nOur selected submission had 17 diverse models. We used stacking with LGBM.
\nWe used one prompt per fold cv strategy. All our models had collate function which speeds up training as well as inference. Lower batch sizes were helpful as well. Most models had evaluation at the end of epoch, few models had evaluation multiple times during epoch. Many of the diverse and smaller models were distilled (described below).
\nAll of us were able to contribute models to the stack, some of us never checked individual models scores because public LB seemed unstable. Our stack seemed more stable than individual models.
\nApart from model predictions we had following features in stack:
\nword_overlap_count\nbigram_overlap_count \nbigram_overlap_ratio \ntrigram_overlap_count\ntrigram_overlap_ratio\n\n- Text stat features\nOsman\nGutierrez_polini\nDale_chall_readability_score\nflesch_kincaid_grade\nHere we describe each of the models :
\n(A) Deberta v3 large - full prompt text psuedo labeled model [ar3]
\nThis model had best cv (0.4721) in stack - This was trained with max len 1790, attention head on top. This was also using back translated pseudo labeled dataset. Every sample was translated to german and then back to english. We randomly selected 50% of this data for training. So every training fold had different ~3100 samples that were pseudo labeled , validation set had no psuedo labeled samples.
\nLoss function for this model was combination of SmoothL1Loss, RMSE Loss and RankLoss - @fightingmuscle described the loss function here
\nInput for this model - [ANSWER_START] + summary_text + [ANSWER_END] + [PROMPT_START] + prompt_text [PROMPT_END] + [QUESTION_START] + prompt_question + [QUESTION_END]
\nInference maxlen = 1024
(B)
\nThese were the same models with different seeds and mean pooled, input to these were summary_text + prompt_question, RMSE loss, validation end of epoch.
\n(C)
\nThese were the same models with different seeds and with CLS token, input to these were summary_text + prompt_question, MSE Loss,validation end of epoch. I added extra text “Summarize:” to prompt question , if we look 👀 at the data 🔎carefully, notice that there are prompts with \"cite evidence\" and these have much longer summary texts and it doesn't say summarize. After reading other solutions, should have tried some more variations to this.
\n(D)
\nAll these models were distilled - 50% oof labels and 50% true labels, with AWP added - only summary_text as input, validated multiple times during epoch.
\n(E)
\nDeberta v3 large [ar2] - Order of input to this model was different than the remaining models - text + SEP + prompt_question
Deberta v3 Base and XSmall models [ar4, ar5, ar7] - These were distilled models. We utilized oof labels from a previous stack. These models were trained using oof labels where as validation set remained as true labels.
\nBigram Signal -
\n\noutputs = .model(input_ids, attention_mask, token_type_ids)\nbigram_signal = outputs.last_hidden_state[, , ] * outputs.last_hidden_state[, , ]\nfeature = bigram_signal[, , ]\n feature\n(F) Deberta v3 large - Classification - Attention Head , Mean Pooling - (4x) [ar6]
\nAfter analyzing data, we noticed there were 1134 distinct categories of content and wording, so we added a classification model with BCEWithLogitsLoss. We trained 4 different models with different heads, averaged them together and then added to the stack.
We haven't understood how or why the splits for training/public lb/private lb were created. The distribution of prompts for training and public lb is very different than private lb. Very long prompt text in public and training set but not in private lb. Also number of summary text per prompt is very different for training/public LB and private LB, unfortunately these resulted in shake down for many many teams.
\nMany other things tried but were not part of selected submission
\nAcknowledgements
\nThank you @kononenko for this compliant textstat kernel
Thank you @tsunotsuno for LGBM features
\n🙏 Thank you to my amazing teammates, grateful to have worked together on this with you all.
\nCongratulations on your results!! Thank you for sharing your solution.
\n\n\nThis was also using back translated pseudo labeled dataset. Every sample was translated to german and then back to english.
\n
Does this mean that you translated every summary text to German and then back to English, and you assigned pseudo labels to the data using a model trained previously? In short, you did not use the original labels of the summary texts that were back-translated?
", "rawMarkdown": "Congratulations on your results!! Thank you for sharing your solution.\n\n>This was also using back translated pseudo labeled dataset. Every sample was translated to german and then back to english. \n\nDoes this mean that you translated every summary text to German and then back to English, and you assigned pseudo labels to the data using a model trained previously? In short, you did not use the original labels of the summary texts that were back-translated?\n", "votes": 1, "replies": [ { "id": 2492722, "postDate": "2023-10-22T20:04:01.697Z", "content": "Thank you for your question, This model had all 7165 samples provided in the competition + back translated pseudo labeled samples.
\nSo every fold had ~9000 training samples (~6000 provided by the competition + ~3000 back translated samples )
Thank you for the information. Just to clarify, when you mention 'back translated pseudo labeled samples,' did you use pseudo labels for these samples, or were the original labels used for the back-translated samples? In other words, were the labels assigned based on the original data or through a pseudo labeling process?
", "rawMarkdown": "Thank you for the information. Just to clarify, when you mention 'back translated pseudo labeled samples,' did you use pseudo labels for these samples, or were the original labels used for the back-translated samples? In other words, were the labels assigned based on the original data or through a pseudo labeling process?" }, { "id": 2493440, "postDate": "2023-10-23T12:14:44.707Z", "content": "back translated samples were pseudo labeled
", "rawMarkdown": "back translated samples were pseudo labeled ", "votes": 1 }, { "id": 2493596, "postDate": "2023-10-23T13:49:49.010Z", "content": "Thank you. I assigned the original labels to the back-translated samples, but this was not effective. I appreciate the insight!
", "rawMarkdown": "Thank you. I assigned the original labels to the back-translated samples, but this was not effective. I appreciate the insight!", "votes": 2 } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:04:59Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10796316, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/barton-fink-11th-place-solution" }