My solution is rather simple, because I used almost no complex tricks - instead I just built a reliable pipeline and found good hyperparameters. I had some brilliant ideas (at least I think so :)) - to use augmentation and generate synthetic data using some big model based on other texts from commonlit.org. But at that time, when I was still actively contributing, there was uncertainty with the license for the texts, and I did not complete the use of augmentations - I was too busy with my work (trying to get RLHF to work), so I left the competition - my last commit was a month ago. But apparently the decision was good, at least I didn’t overfit :)
\nSo let's break my solution down into parts.
\n1. Data:
\nI used a fairly standard template: prompt + question + text. At the end of my participation, I tried to dig into the data and do a good pre-processing - after all, there were quite a lot of very similar essays with exact or different scores. So, I tried to find these samples by similarity (like Levenstein) and then merge them. In addition, I decided to make augumentations based on this insight - if there are many similar essays (differing only in typos, for example) - I could use something like reverse autocorrect - randomly replace some words with its close analogues. With this technique I got 0.453 in private ONLY on fold3 (which is better than my chosen blend and probably could lead to a second place) - but I was too tired at this point so I didn't look further into augmentations. But I think augmentations could probably lead me to victory.
2. Models
\nDeberta is the king, so there's not much to say here. I tried using decoder models like Llama, but Deberta was still better. There were some techniques that gave me a boost - using EMA (honestly, without EMA it was very unstable, so it's probably just necessary) and using differential learning rates.I tried several pooling options, but the best option for me was to use concatenation of CLS token and student's text meanpooling. I also used token_type_ids to separate the prompt, question and essay.
3 Inference & train
\nI used following scheme - I tried to find a good hyperparameters on some fold (for example, fold0), and then train with exact hyperparameters on other folds. I then sumbittend the entire blend and 4 individual models (5 submission total - one day) and repeated the procedure the next day. I realized that I could use maxlen 1500 for inference (didn't research this number much, tried something like 1024 and 2048, but 1500 was better in terms of efficiency), so in my final mix I took the 10 best checkpoints across folds (some folds got 2 checkpoints, some folds got 3). First I averaged by folds, then averaged the rest. That's all.
Briefly what worked (ranked from most important to least important, IMO):
\nUsing Deberta
EMA
Augumentation
Defferentiated learning rates
Custom pooling
token_type_ids
Data cleaninig
\nWhat did not work (random order):
Decoder models
AWP
FGM
WD
Constant LR
Handcrafted features
GBT for stacking
In the end, it was a good competition for me. Last year I competed in another NLP competition and got a silver medal, but I grinded all day at that competition (I wasn't working that time, so I had a lot of free time). This time I also expected silver, which I consider a solid result, but I got 3rd place. In any case, this competition was a cakewalk for me, since I spend very little effort on it (compared to the previous competition, at least). I'm hoping this means I'll grow a lot this year - and I think that's the main goal of participating in Kaggle.
\nGood luck to all of you.
", "comments": [ { "id": 2479544, "author_name": "Kinosuke", "author_url": "", "post_date": "2023-10-12T17:16:51.047000", "content": "Thanks for the clear write-up, and congrats on solo gold and getting competition Master
\nYour solution is very simple and cool!
I would like to ask you a question about the following.
\n\n\nI tried to find these samples by similarity (like Levenstein) and then merge them
\n
How exactly did you merge similar texts?
", "votes": 3, "replies": [ { "id": 2479562, "author_name": "Dmitrii Krasnov", "author_url": "", "post_date": "2023-10-12T17:36:17.850000", "content": "Thank you, but I only got a master😀
\nI just calculated pairwise levenstein distance across folds, and then check manually very similar example (where is similarity is more that 95%, for example).
\nWhen I looked and this examples, I selected manually one of them with less number of typos and dropped the others, and if they had different scores - averaged it.
\n\nThank you, but I only got a master😀
\n
Sorry, I made a mistake.
\nThank you for your detailed answer! I understand your solution better.
\nI remembered that filtering samples with high similarity was very effective in the stable diffusion competition I previously participated in.
Yes, and when you doing this kind of preprocessing you could get some insights. For example, there was quite similar examples, ex. only differs with one word, so student #1 write this word correctly and student#2 made a mistake. So, as host explained to us, there should be identical scores for both of this essays. But in fact, this scores could be very differnt. You could even calculate this cases and get some estimate of human error on this task😁
", "votes": 3, "replies": [] }, { "id": 2479591, "author_name": "Kinosuke", "author_url": "", "post_date": "2023-10-12T18:09:57.240000", "content": "This competition was suffering from wording's unstable labeling, but this solution turned out to be a very effective way to solve that problem. I learned a lot. Thank you.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2486232, "author_name": "Urenna Okeke", "author_url": "", "post_date": "2023-10-17T19:11:28.620000", "content": "Congratulations on your win!
", "votes": 1, "replies": [] }, { "id": 2479510, "author_name": "moth", "author_url": "", "post_date": "2023-10-12T16:55:23.610000", "content": "Congratulations @koba35, by EMA you mean Exponential Moving Average of weights? Could you expand on this last item?
", "votes": 2, "replies": [ { "id": 2479519, "author_name": "Dmitrii Krasnov", "author_url": "", "post_date": "2023-10-12T17:00:15.190000", "content": "Yeah, you are right.
", "votes": 0, "replies": [] } ] }, { "id": 2489721, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-10-20T07:04:12.087000", "content": "Congratulations on 3rd place. Thanks for sharing the details of your notebook.
\nCan you share info about running time of your notebook.
Congrat! Thanks for the simple but effective solution. I also tried to use Decoder-only models (albert-large) and it did not work well. I wonder if there is any particular reason behind this.
", "votes": 0, "replies": [ { "id": 2483136, "author_name": "Dmitrii Krasnov", "author_url": "", "post_date": "2023-10-15T13:56:08.023000", "content": "Probably, it’s because you Albert. You should always use deberta nowadays, I think.
", "votes": 0, "replies": [ { "id": 2483374, "author_name": "Min-Hsien Weng", "author_url": "", "post_date": "2023-10-15T16:44:18.490000", "content": "Agree. Most top solutions use debert v3 model. Thanks.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2479983, "author_name": "Thienkaka", "author_url": "", "post_date": "2023-10-13T03:43:43.183000", "content": "I have a question about how did you apply EMA to model? I have some confuse at this point.
", "votes": 0, "replies": [ { "id": 2480165, "author_name": "Dmitrii Krasnov", "author_url": "", "post_date": "2023-10-13T06:28:04.687000", "content": "I will probably release my train code after some clean up, so stay tuned
", "votes": 2, "replies": [] } ] }, { "id": 2479797, "author_name": "Haru", "author_url": "", "post_date": "2023-10-12T22:02:35.773000", "content": "Thank you for your solution.
\nAlso, congratulations on your victory!
\nI have a question.
\n“I also used token_type_ids to separate the prompt, question and essay.”
How did you use token_type_ids?
\nThank you first.
", "votes": 0, "replies": [ { "id": 2479819, "author_name": "Dmitrii Krasnov", "author_url": "", "post_date": "2023-10-12T23:03:41.913000", "content": "In some encoder models (and in deberata too) you could pass to forward additional argument, called token_type_ids. Originally it's used in QA task, to separate questions and answers. It's working similar to positional embeddings - but it’s encoding type of tokens instead of position. Implementation is very simple- you need to add additional mask similar to attention mask, and tokens belong to prompt will be zeroes, related to question will be ones and related to essay text will be twos
", "votes": 2, "replies": [] } ] } ], "comment_count": 16, "messages": [ { "id": 2479481, "postDate": "2023-10-12T16:38:57.520Z", "content": "My solution is rather simple, because I used almost no complex tricks - instead I just built a reliable pipeline and found good hyperparameters. I had some brilliant ideas (at least I think so :)) - to use augmentation and generate synthetic data using some big model based on other texts from commonlit.org. But at that time, when I was still actively contributing, there was uncertainty with the license for the texts, and I did not complete the use of augmentations - I was too busy with my work (trying to get RLHF to work), so I left the competition - my last commit was a month ago. But apparently the decision was good, at least I didn’t overfit :)
\nSo let's break my solution down into parts.
\n1. Data:
\nI used a fairly standard template: prompt + question + text. At the end of my participation, I tried to dig into the data and do a good pre-processing - after all, there were quite a lot of very similar essays with exact or different scores. So, I tried to find these samples by similarity (like Levenstein) and then merge them. In addition, I decided to make augumentations based on this insight - if there are many similar essays (differing only in typos, for example) - I could use something like reverse autocorrect - randomly replace some words with its close analogues. With this technique I got 0.453 in private ONLY on fold3 (which is better than my chosen blend and probably could lead to a second place) - but I was too tired at this point so I didn't look further into augmentations. But I think augmentations could probably lead me to victory.
2. Models
\nDeberta is the king, so there's not much to say here. I tried using decoder models like Llama, but Deberta was still better. There were some techniques that gave me a boost - using EMA (honestly, without EMA it was very unstable, so it's probably just necessary) and using differential learning rates.I tried several pooling options, but the best option for me was to use concatenation of CLS token and student's text meanpooling. I also used token_type_ids to separate the prompt, question and essay.
3 Inference & train
\nI used following scheme - I tried to find a good hyperparameters on some fold (for example, fold0), and then train with exact hyperparameters on other folds. I then sumbittend the entire blend and 4 individual models (5 submission total - one day) and repeated the procedure the next day. I realized that I could use maxlen 1500 for inference (didn't research this number much, tried something like 1024 and 2048, but 1500 was better in terms of efficiency), so in my final mix I took the 10 best checkpoints across folds (some folds got 2 checkpoints, some folds got 3). First I averaged by folds, then averaged the rest. That's all.
Briefly what worked (ranked from most important to least important, IMO):
\nUsing Deberta
EMA
Augumentation
Defferentiated learning rates
Custom pooling
token_type_ids
Data cleaninig
\nWhat did not work (random order):
Decoder models
AWP
FGM
WD
Constant LR
Handcrafted features
GBT for stacking
In the end, it was a good competition for me. Last year I competed in another NLP competition and got a silver medal, but I grinded all day at that competition (I wasn't working that time, so I had a lot of free time). This time I also expected silver, which I consider a solid result, but I got 3rd place. In any case, this competition was a cakewalk for me, since I spend very little effort on it (compared to the previous competition, at least). I'm hoping this means I'll grow a lot this year - and I think that's the main goal of participating in Kaggle.
\nGood luck to all of you.
", "rawMarkdown": "My solution is rather simple, because I used almost no complex tricks - instead I just built a reliable pipeline and found good hyperparameters. I had some brilliant ideas (at least I think so :)) - to use augmentation and generate synthetic data using some big model based on other texts from commonlit.org. But at that time, when I was still actively contributing, there was uncertainty with the license for the texts, and I did not complete the use of augmentations - I was too busy with my work (trying to get RLHF to work), so I left the competition - my last commit was a month ago. But apparently the decision was good, at least I didn’t overfit :)\n\nSo let's break my solution down into parts.\n\n**1. Data:**\nI used a fairly standard template: prompt + question + text. At the end of my participation, I tried to dig into the data and do a good pre-processing - after all, there were quite a lot of very similar essays with exact or different scores. So, I tried to find these samples by similarity (like Levenstein) and then merge them. In addition, I decided to make augumentations based on this insight - if there are many similar essays (differing only in typos, for example) - I could use something like reverse autocorrect - randomly replace some words with its close analogues. With this technique I got 0.453 in private ONLY on fold3 (which is better than my chosen blend and probably could lead to a second place) - but I was too tired at this point so I didn't look further into augmentations. But I think augmentations could probably lead me to victory.\n\n**2. Models**\nDeberta is the king, so there's not much to say here. I tried using decoder models like Llama, but Deberta was still better. There were some techniques that gave me a boost - using EMA (honestly, without EMA it was very unstable, so it's probably just necessary) and using differential learning rates.I tried several pooling options, but the best option for me was to use concatenation of CLS token and student's text meanpooling. I also used token_type_ids to separate the prompt, question and essay.\n\n**3 Inference & train**\nI used following scheme - I tried to find a good hyperparameters on some fold (for example, fold0), and then train with exact hyperparameters on other folds. I then sumbittend the entire blend and 4 individual models (5 submission total - one day) and repeated the procedure the next day. I realized that I could use maxlen 1500 for inference (didn't research this number much, tried something like 1024 and 2048, but 1500 was better in terms of efficiency), so in my final mix I took the 10 best checkpoints across folds (some folds got 2 checkpoints, some folds got 3). First I averaged by folds, then averaged the rest. That's all. \n\nBriefly what worked (ranked from most important to least important, IMO):\n 1. Using Deberta\n 2. EMA\n 3. Augumentation\n 4. Defferentiated learning rates\n 5. Custom pooling\n 6. token_type_ids\n 7. Data cleaninig\n \n What did not work (random order):\n 1. Decoder models\n 2. AWP\n 3. FGM\n 4. WD\n 5. Constant LR\n 6. Handcrafted features\n 7. GBT for stacking\n \nIn the end, it was a good competition for me. Last year I competed in another NLP competition and got a silver medal, but I grinded all day at that competition (I wasn't working that time, so I had a lot of free time). This time I also expected silver, which I consider a solid result, but I got 3rd place. In any case, this competition was a cakewalk for me, since I spend very little effort on it (compared to the previous competition, at least). I'm hoping this means I'll grow a lot this year - and I think that's the main goal of participating in Kaggle.\n \nGood luck to all of you.", "votes": 42 }, { "id": 2479544, "postDate": "2023-10-12T17:16:51.047Z", "content": "Thanks for the clear write-up, and congrats on solo gold and getting competition Master
\nYour solution is very simple and cool!
I would like to ask you a question about the following.
\n\n\nI tried to find these samples by similarity (like Levenstein) and then merge them
\n
How exactly did you merge similar texts?
", "rawMarkdown": "Thanks for the clear write-up, and congrats on solo gold and getting competition Master ~~GM~~\nYour solution is very simple and cool!\n\nI would like to ask you a question about the following.\n> I tried to find these samples by similarity (like Levenstein) and then merge them\n\nHow exactly did you merge similar texts?", "votes": 3, "replies": [ { "id": 2479562, "postDate": "2023-10-12T17:36:17.850Z", "content": "Thank you, but I only got a master😀
\nI just calculated pairwise levenstein distance across folds, and then check manually very similar example (where is similarity is more that 95%, for example).
\nWhen I looked and this examples, I selected manually one of them with less number of typos and dropped the others, and if they had different scores - averaged it.
\n\nThank you, but I only got a master😀
\n
Sorry, I made a mistake.
\nThank you for your detailed answer! I understand your solution better.
\nI remembered that filtering samples with high similarity was very effective in the stable diffusion competition I previously participated in.
Yes, and when you doing this kind of preprocessing you could get some insights. For example, there was quite similar examples, ex. only differs with one word, so student #1 write this word correctly and student#2 made a mistake. So, as host explained to us, there should be identical scores for both of this essays. But in fact, this scores could be very differnt. You could even calculate this cases and get some estimate of human error on this task😁
", "rawMarkdown": "Yes, and when you doing this kind of preprocessing you could get some insights. For example, there was quite similar examples, ex. only differs with one word, so student #1 write this word correctly and student#2 made a mistake. So, as host explained to us, there should be identical scores for both of this essays. But in fact, this scores could be very differnt. You could even calculate this cases and get some estimate of human error on this task😁", "votes": 3 }, { "id": 2479591, "postDate": "2023-10-12T18:09:57.240Z", "content": "This competition was suffering from wording's unstable labeling, but this solution turned out to be a very effective way to solve that problem. I learned a lot. Thank you.
", "rawMarkdown": "This competition was suffering from wording's unstable labeling, but this solution turned out to be a very effective way to solve that problem. I learned a lot. Thank you." } ] } ] }, { "id": 2486232, "postDate": "2023-10-17T19:11:28.620Z", "content": "Congratulations on your win!
", "rawMarkdown": "Congratulations on your win!", "votes": 1 }, { "id": 2479510, "postDate": "2023-10-12T16:55:23.610Z", "content": "Congratulations @koba35, by EMA you mean Exponential Moving Average of weights? Could you expand on this last item?
", "rawMarkdown": "Congratulations @koba35, by EMA you mean Exponential Moving Average of weights? Could you expand on this last item?", "votes": 2, "replies": [ { "id": 2479519, "postDate": "2023-10-12T17:00:15.190Z", "content": "Yeah, you are right.
", "rawMarkdown": "Yeah, you are right." } ] }, { "id": 2489721, "postDate": "2023-10-20T07:04:12.087Z", "content": "Congratulations on 3rd place. Thanks for sharing the details of your notebook.
\nCan you share info about running time of your notebook.
Congrat! Thanks for the simple but effective solution. I also tried to use Decoder-only models (albert-large) and it did not work well. I wonder if there is any particular reason behind this.
", "rawMarkdown": "Congrat! Thanks for the simple but effective solution. I also tried to use Decoder-only models (albert-large) and it did not work well. I wonder if there is any particular reason behind this. ", "replies": [ { "id": 2483136, "postDate": "2023-10-15T13:56:08.023Z", "content": "Probably, it’s because you Albert. You should always use deberta nowadays, I think.
", "rawMarkdown": "Probably, it’s because you Albert. You should always use deberta nowadays, I think. ", "replies": [ { "id": 2483374, "postDate": "2023-10-15T16:44:18.490Z", "content": "Agree. Most top solutions use debert v3 model. Thanks.
", "rawMarkdown": "Agree. Most top solutions use debert v3 model. Thanks." } ] } ] }, { "id": 2479983, "postDate": "2023-10-13T03:43:43.183Z", "content": "I have a question about how did you apply EMA to model? I have some confuse at this point.
", "rawMarkdown": "I have a question about how did you apply EMA to model? I have some confuse at this point.", "replies": [ { "id": 2480165, "postDate": "2023-10-13T06:28:04.687Z", "content": "I will probably release my train code after some clean up, so stay tuned
", "rawMarkdown": "I will probably release my train code after some clean up, so stay tuned", "votes": 2 } ] }, { "id": 2479797, "postDate": "2023-10-12T22:02:35.773Z", "content": "Thank you for your solution.
\nAlso, congratulations on your victory!
\nI have a question.
\n“I also used token_type_ids to separate the prompt, question and essay.”
How did you use token_type_ids?
\nThank you first.
", "rawMarkdown": "Thank you for your solution.\nAlso, congratulations on your victory!\nI have a question.\n“I also used token_type_ids to separate the prompt, question and essay.”\n\nHow did you use token_type_ids?\n\nThank you first.", "replies": [ { "id": 2479819, "postDate": "2023-10-12T23:03:41.913Z", "content": "In some encoder models (and in deberata too) you could pass to forward additional argument, called token_type_ids. Originally it's used in QA task, to separate questions and answers. It's working similar to positional embeddings - but it’s encoding type of tokens instead of position. Implementation is very simple- you need to add additional mask similar to attention mask, and tokens belong to prompt will be zeroes, related to question will be ones and related to essay text will be twos
", "rawMarkdown": "In some encoder models (and in deberata too) you could pass to forward additional argument, called token_type_ids. Originally it's used in QA task, to separate questions and answers. It's working similar to positional embeddings - but it’s encoding type of tokens instead of position. Implementation is very simple- you need to add additional mask similar to attention mask, and tokens belong to prompt will be zeroes, related to question will be ones and related to essay text will be twos", "votes": 2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:04:23Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10678218, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/dmitrii-krasnov-3rd-place-solution" }