First and foremost, I'd like to extend my gratitude to the host for organizing this competition. Congratulations to the winners as well! I'd like to present a brief overview of my solution that secured the 9th position.
\nThe wording of prompt_id 814d6b stood out, being notably different from other prompt_ids. When cross-validating with groupkfold using the prompt_id, predicting for this particular prompt_id didn't correlate with scores from other prompt_ids. Hypothetically, if there were any test ids with a distribution similar to 814d6b, having 814d6b in the training data might produce decent accuracy. However, there was no way to validate this. For this competition, I employed two methods of cross-validation and submitted for both:
\nCross-validation 1: groupkfold (group = prompt_id), excluding 814d6b from evaluation.
\nCross-validation 2: groupkfold (group = prompt_id), Including 814d6b in the evaluation.
\nThe final private score was better for the model from cross-validation 2, so I'll delve into the details of that model.
\nGiven that changing the seed alone resulted in variable CV scores, I performed cross-validation using three different seeds and evaluated the model based on their ensemble. The final submission was based on the predictions of models trained on all the data using three different seeds.
I tokenized the text as follows:
\ntext1: summary_text
\ntext2: prompt_question + [SEP] + prompt_text
inputs = self.tokenizer.encode_plus(text1,\n text2,\n =,\n =self.max_len,\n =,\n =,\n =)\nModel: Consisted of two deberta-v3-large and one LSTM as described below:
\n = \nTraining settings:
\ntoken_len: 4200
\nhttps://www.kaggle.com/code/takoihiraokazu/sub-ex263-seq4200?scriptVersionId=145902061
", "comments": [ { "id": 2480014, "author_name": "Cotmd", "author_url": "", "post_date": "2023-10-13T04:45:01.593000", "content": "Congrats🎉 and thanks for your cool solution!
\nI have some questions that I want to ask.
\nThe final submission was based on the predictions of models trained on all the data using three different seeds.based on predictions of models trained mean final decision was based on Public Score?Thanks for taking a look at my question
\nLearned a lot. Thanks again!!
Thank you!
\nHere are the answers to your questions:
For the model with the best public score, I calculated the CV for prompt-id other than 814d6b. It is an ensemble of two DeBERTa-v3-large models. As for 814d6b, I didn't calculate the CV, so its value remains unknown.
\nCV: 0.4598
\nPrivate score: 0.461
When evaluating the performance of the model, I use cross-validation. After the cross-validation process, I retrain on all the data using the same settings. Consequently, while I can't directly evaluate the model meant for submission, I've found that the results remain consistent across different seeds during the cross-validation phase. Additionally, the loss trajectory during training did not exhibit any problems, leading me to confidently use it for submission. I adopted this approach of training on all data based on references from discussions like the following:
\nhttps://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536
Yes, it's an ensemble solely based on models trained with different seeds. When I tried to ensemble with other models, the score for 814d6b did not improve, and even the overall CV didn't show significant improvement. Thus, I decided to submit using a single model. However, for the remaining two submissions, I did utilize ensemble.
Thx for answering!!😆
", "votes": 1, "replies": [] } ] } ] }, { "id": 2479302, "author_name": "moth", "author_url": "", "post_date": "2023-10-12T14:39:55.173000", "content": "Congratulations @takoihiraokazu for your finish!
\nThis might be a complex question, but could you elaborate on your line of thought for creating your model architecture? I imagine that the first model was used to capture all available information, second to emphasize on text summaries which were the most relevant piece, LSTM for bidirectional dependencies, pooling to summarize, etc, please correct me if I am wrong. Thanks in advance
", "votes": 4, "replies": [ { "id": 2480481, "author_name": "Takoi", "author_url": "", "post_date": "2023-10-13T10:36:24.153000", "content": "Thank you !
\nYes, your understanding is correct except for the LSTM part. The thought process behind the model architecture is as follows:
I wanted the model to capture the relationship between the summary and prompt. Therefore, I used all available information as input to the first DeBERTa model.
\nSince the prompt_id is different between the train data and test data, I believed that not including the prompt part in the pooling would improve generalization. So, I decided to only pool the summary part. Furthermore, by training only the summary part in a separate DeBERTa model before pooling, I believed it might lead to a better score. Hence, for the second DeBERTa model, I input only the summary part.
\nRegarding the LSTM, I incorporated it as it led to score improvements specifically for prompt_id: 814d6b. There isn’t a deep rationale behind this, but through various experiments, I found that including the LSTM was beneficial for improving the score for 814d6b.
Awesome, thanks again!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2480713, "author_name": "ls", "author_url": "", "post_date": "2023-10-13T14:02:30.187000", "content": "Nice solution. Congrats on gold.
", "votes": 1, "replies": [] }, { "id": 2478821, "author_name": "chumajin", "author_url": "", "post_date": "2023-10-12T08:19:26.813000", "content": "Congratulations on your strong solo gold medal!
\nIt took me about 30 minutes to understand it, but the architecture of your model is great and we could not get a cv like this! I learned a lot. Thank you very much.
Thank you! I also learned a lot from your team's solution.
", "votes": 2, "replies": [] } ] }, { "id": 2478658, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2023-10-12T06:22:03.557000", "content": "Thanks for the description!
\nSo you don't use LGBM and why?
Thank you. I tried LGBM, but I didn't adopt it because the CV didn't improve
", "votes": 3, "replies": [] } ] }, { "id": 2478647, "author_name": "Dristanta Das", "author_url": "", "post_date": "2023-10-12T06:13:23.270000", "content": "First of all, congratulations.
\nWanted to inquire more on your solution, how you incorporated 2 models (a rough idea would be suffice) and how does lstm affected your solution (as compared to non lstm) if you have tested, then shed some light onto that.
Thanks!
\nRegarding the two models, the output of one model is used as the input for the other. For details, please check the implementation of CustomModel. As for the LSTM, its inclusion resulted in an improvement in the 814d6b score.
Hi @takoihiraokazu Could you share your Exponential Moving Average code if possible.
", "votes": 0, "replies": [] }, { "id": 2576477, "author_name": "", "author_url": "", "post_date": "2023-12-27T19:21:03.973000", "content": "Hi @takoihiraokazu The difference between model1 and model2 is that model1 consists Embedding + Encoder part and model2 consists only Encoder part. Am I getting it correct ?
\nAlso if above statement is correct what difference its make like why cannot we train model2 similar to model1 ?
", "votes": 1, "replies": [ { "id": 2578610, "author_name": "Takoi", "author_url": "", "post_date": "2023-12-29T12:11:06.387000", "content": "Yes, your understanding is correct. The reason I differentiated between model1 and model2 was simply because I wanted to change the attention_mask midway through the model. By separating the models, I can achieve this just by altering the model's input.
", "votes": 0, "replies": [] } ] }, { "id": 2479325, "author_name": "", "author_url": "", "post_date": "2023-10-12T14:59:39.633000", "content": "@chumajin Congrats for solo gold. I love your solution. Looking forward to your Training code .I learn a lot from your inference book.
", "votes": 2, "replies": [] } ], "comment_count": 17, "messages": [ { "id": 2478633, "postDate": "2023-10-12T05:51:03.800Z", "content": "First and foremost, I'd like to extend my gratitude to the host for organizing this competition. Congratulations to the winners as well! I'd like to present a brief overview of my solution that secured the 9th position.
\nThe wording of prompt_id 814d6b stood out, being notably different from other prompt_ids. When cross-validating with groupkfold using the prompt_id, predicting for this particular prompt_id didn't correlate with scores from other prompt_ids. Hypothetically, if there were any test ids with a distribution similar to 814d6b, having 814d6b in the training data might produce decent accuracy. However, there was no way to validate this. For this competition, I employed two methods of cross-validation and submitted for both:
\nCross-validation 1: groupkfold (group = prompt_id), excluding 814d6b from evaluation.
\nCross-validation 2: groupkfold (group = prompt_id), Including 814d6b in the evaluation.
\nThe final private score was better for the model from cross-validation 2, so I'll delve into the details of that model.
\nGiven that changing the seed alone resulted in variable CV scores, I performed cross-validation using three different seeds and evaluated the model based on their ensemble. The final submission was based on the predictions of models trained on all the data using three different seeds.
I tokenized the text as follows:
\ntext1: summary_text
\ntext2: prompt_question + [SEP] + prompt_text
inputs = self.tokenizer.encode_plus(text1,\n text2,\n =,\n =self.max_len,\n =,\n =,\n =)\nModel: Consisted of two deberta-v3-large and one LSTM as described below:
\n = \nTraining settings:
\ntoken_len: 4200
\nhttps://www.kaggle.com/code/takoihiraokazu/sub-ex263-seq4200?scriptVersionId=145902061
", "rawMarkdown": "# 9th Place Solution\n\nFirst and foremost, I'd like to extend my gratitude to the host for organizing this competition. Congratulations to the winners as well! I'd like to present a brief overview of my solution that secured the 9th position.\n\n## Cross-Validation\nThe wording of prompt_id 814d6b stood out, being notably different from other prompt_ids. When cross-validating with groupkfold using the prompt_id, predicting for this particular prompt_id didn't correlate with scores from other prompt_ids. Hypothetically, if there were any test ids with a distribution similar to 814d6b, having 814d6b in the training data might produce decent accuracy. However, there was no way to validate this. For this competition, I employed two methods of cross-validation and submitted for both:\nCross-validation 1: groupkfold (group = prompt_id), excluding 814d6b from evaluation.\nCross-validation 2: groupkfold (group = prompt_id), Including 814d6b in the evaluation.\nThe final private score was better for the model from cross-validation 2, so I'll delve into the details of that model. \nGiven that changing the seed alone resulted in variable CV scores, I performed cross-validation using three different seeds and evaluated the model based on their ensemble. The final submission was based on the predictions of models trained on all the data using three different seeds.\n\n## Training\nI tokenized the text as follows:\ntext1: summary_text\ntext2: prompt_question + [SEP] + prompt_text\n```\n\ninputs = self.tokenizer.encode_plus(text1,\n text2,\n add_special_tokens=True,\n max_length=self.max_len,\n padding='max_length',\n truncation=True,\n return_token_type_ids=True)\n```\n\nModel: Consisted of two deberta-v3-large and one LSTM as described below:\n- The first deberta-v3-large was trained using the entire input text.\n- The second deberta-v3-large and LSTM were trained solely on the summary_text.\n\n```\n\nmodel_path = \"microsoft/deberta-v3-large\"\nclass CustomModel(nn.Module):\n def __init__(self):\n super(CustomModel, self).__init__()\n self.model = AutoModel.from_pretrained(model_path)\n config = AutoConfig.from_pretrained(\n model_path, output_hidden_states=True)\n config.num_hidden_layers = 6\n self.model2 = DebertaV2Encoder(config)\n self.lstm = nn.LSTM(input_size=1024, hidden_size=1024,\n num_layers=1, batch_first=True,\n bidirectional=True)\n self.linear1 = nn.Sequential(\n nn.Linear(1024*2, 512),\n nn.LayerNorm(512),\n nn.ReLU(),\n nn.Dropout(0.2),\n nn.Linear(512, 2))\n\n self.pool = MeanPooling()\n\n def forward(self, ids, mask, token_type_ids,\n s_mask):\n out = self.model(ids, attention_mask=mask,\n token_type_ids=token_type_ids)[\n 'last_hidden_state']\n s_mask_len = s_mask.shape[1]\n out = out[:, :s_mask_len, :]\n out = out.contiguous()\n out = self.model2(out, s_mask)[\n 'last_hidden_state']\n out_list = []\n for s in range(len(s_mask)):\n s_mask_ = s_mask[s]\n s_mask_len = torch.sum(s_mask_)\n emb = out[[s], :s_mask_len, :]\n s_mask_ = s_mask_[:s_mask_len].unsqueeze(0)\n emb, _ = self.lstm(emb)\n emb = self.pool(emb, s_mask_)\n out_list.append(emb)\n out_concat = torch.cat(out_list,\n axis=0)\n out_concat = self.linear1(out_concat)\n return out_concat\n\n```\n\nTraining settings:\n\n- token_len: 1680\n- epochs: 3\n- Loss: SmoothL1Loss\n- lr: 8e-6\n- optimizer: AdamW\n - weight_decay: 0.01\n - beta: (0.9, 0.98)\n- scheduler: get_linear_schedule_with_warmup\n - num_warmup_steps: 10% of total training steps\n- EMA (Exponential Moving Average)\n - ema_decay: 0.995\n - Using EMA helped stabilize the training.\n\n## Inference\ntoken_len: 4200\n\n## Scores\n- CV: 0.495 (ensemble of 3 seeds)\n - 814d6b: 0.604982\n - ebad26: 0.431438\n - 3b9047: 0.49692\n - 39c16e: 0.483208\n- Public Score: 0.456\n- Private Score: 0.457\n\n## Inference Code\nhttps://www.kaggle.com/code/takoihiraokazu/sub-ex263-seq4200?scriptVersionId=145902061\n", "votes": 57 }, { "id": 2480014, "postDate": "2023-10-13T04:45:01.593Z", "content": "Congrats🎉 and thanks for your cool solution!
\nI have some questions that I want to ask.
\nThe final submission was based on the predictions of models trained on all the data using three different seeds.based on predictions of models trained mean final decision was based on Public Score?Thanks for taking a look at my question
\nLearned a lot. Thanks again!!
Thank you!
\nHere are the answers to your questions:
For the model with the best public score, I calculated the CV for prompt-id other than 814d6b. It is an ensemble of two DeBERTa-v3-large models. As for 814d6b, I didn't calculate the CV, so its value remains unknown.
\nCV: 0.4598
\nPrivate score: 0.461
When evaluating the performance of the model, I use cross-validation. After the cross-validation process, I retrain on all the data using the same settings. Consequently, while I can't directly evaluate the model meant for submission, I've found that the results remain consistent across different seeds during the cross-validation phase. Additionally, the loss trajectory during training did not exhibit any problems, leading me to confidently use it for submission. I adopted this approach of training on all data based on references from discussions like the following:
\nhttps://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536
Yes, it's an ensemble solely based on models trained with different seeds. When I tried to ensemble with other models, the score for 814d6b did not improve, and even the overall CV didn't show significant improvement. Thus, I decided to submit using a single model. However, for the remaining two submissions, I did utilize ensemble.
Thx for answering!!😆
", "rawMarkdown": "Thx for answering!!😆", "votes": 1 } ] } ] }, { "id": 2479302, "postDate": "2023-10-12T14:39:55.173Z", "content": "Congratulations @takoihiraokazu for your finish!
\nThis might be a complex question, but could you elaborate on your line of thought for creating your model architecture? I imagine that the first model was used to capture all available information, second to emphasize on text summaries which were the most relevant piece, LSTM for bidirectional dependencies, pooling to summarize, etc, please correct me if I am wrong. Thanks in advance
", "rawMarkdown": "Congratulations @takoihiraokazu for your finish! \n\nThis might be a complex question, but could you elaborate on your line of thought for creating your model architecture? I imagine that the first model was used to capture all available information, second to emphasize on text summaries which were the most relevant piece, LSTM for bidirectional dependencies, pooling to summarize, etc, please correct me if I am wrong. Thanks in advance", "votes": 4, "replies": [ { "id": 2480481, "postDate": "2023-10-13T10:36:24.153Z", "content": "Thank you !
\nYes, your understanding is correct except for the LSTM part. The thought process behind the model architecture is as follows:
I wanted the model to capture the relationship between the summary and prompt. Therefore, I used all available information as input to the first DeBERTa model.
\nSince the prompt_id is different between the train data and test data, I believed that not including the prompt part in the pooling would improve generalization. So, I decided to only pool the summary part. Furthermore, by training only the summary part in a separate DeBERTa model before pooling, I believed it might lead to a better score. Hence, for the second DeBERTa model, I input only the summary part.
\nRegarding the LSTM, I incorporated it as it led to score improvements specifically for prompt_id: 814d6b. There isn’t a deep rationale behind this, but through various experiments, I found that including the LSTM was beneficial for improving the score for 814d6b.
Awesome, thanks again!
", "rawMarkdown": "Awesome, thanks again!", "votes": 1 } ] } ] }, { "id": 2480713, "postDate": "2023-10-13T14:02:30.187Z", "content": "Nice solution. Congrats on gold.
", "rawMarkdown": "Nice solution. Congrats on gold. ", "votes": 1 }, { "id": 2478821, "postDate": "2023-10-12T08:19:26.813Z", "content": "Congratulations on your strong solo gold medal!
\nIt took me about 30 minutes to understand it, but the architecture of your model is great and we could not get a cv like this! I learned a lot. Thank you very much.
Thank you! I also learned a lot from your team's solution.
", "rawMarkdown": "Thank you! I also learned a lot from your team's solution.", "votes": 2 } ] }, { "id": 2478658, "postDate": "2023-10-12T06:22:03.557Z", "content": "Thanks for the description!
\nSo you don't use LGBM and why?
Thank you. I tried LGBM, but I didn't adopt it because the CV didn't improve
", "rawMarkdown": "Thank you. I tried LGBM, but I didn't adopt it because the CV didn't improve", "votes": 3 } ] }, { "id": 2478647, "postDate": "2023-10-12T06:13:23.270Z", "content": "First of all, congratulations.
\nWanted to inquire more on your solution, how you incorporated 2 models (a rough idea would be suffice) and how does lstm affected your solution (as compared to non lstm) if you have tested, then shed some light onto that.
Thanks!
\nRegarding the two models, the output of one model is used as the input for the other. For details, please check the implementation of CustomModel. As for the LSTM, its inclusion resulted in an improvement in the 814d6b score.
Hi @takoihiraokazu Could you share your Exponential Moving Average code if possible.
", "rawMarkdown": "Hi @takoihiraokazu Could you share your Exponential Moving Average code if possible.", "isDeleted": true }, { "id": 2576477, "postDate": "2023-12-27T19:21:03.973Z", "content": "Hi @takoihiraokazu The difference between model1 and model2 is that model1 consists Embedding + Encoder part and model2 consists only Encoder part. Am I getting it correct ?
\nAlso if above statement is correct what difference its make like why cannot we train model2 similar to model1 ?
", "rawMarkdown": "Hi @takoihiraokazu The difference between model1 and model2 is that model1 consists Embedding + Encoder part and model2 consists only Encoder part. Am I getting it correct ?\n\nAlso if above statement is correct what difference its make like why cannot we train model2 similar to model1 ?", "votes": 1, "isDeleted": true, "replies": [ { "id": 2578610, "postDate": "2023-12-29T12:11:06.387Z", "content": "Yes, your understanding is correct. The reason I differentiated between model1 and model2 was simply because I wanted to change the attention_mask midway through the model. By separating the models, I can achieve this just by altering the model's input.
", "rawMarkdown": "Yes, your understanding is correct. The reason I differentiated between model1 and model2 was simply because I wanted to change the attention_mask midway through the model. By separating the models, I can achieve this just by altering the model's input." } ] }, { "id": 2479325, "postDate": "2023-10-12T14:59:39.633Z", "content": "@chumajin Congrats for solo gold. I love your solution. Looking forward to your Training code .I learn a lot from your inference book.
", "rawMarkdown": "@chumajin Congrats for solo gold. I love your solution. Looking forward to your Training code .I learn a lot from your inference book.", "votes": 2, "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:01:50Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10674642, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/rist-takoi-9th-place-solution" }