The first step to achieve this model was to train an mlm on the old dataset. I took all the data from previous feedback competition and fine-tuned deberta-v3-large using run_mlm.py script available in the transformers library. The perplexity in the end was around 4.5. I could have trained more but I didn't.
Now comes the more interesting part. I was inspired by @nbroad 's kernel that showed how we can use token classification for this problem. This was the kernel that piqued my interest in this competition and I decided to give it a go. The idea was simple but effective.
\nEssay texts were represented as follows:
\n[CLS]some text [CLS_{discourse_type}] some valid discourse text [END_{discourse_type}] ..... [SEP]\nMapping:
\ndisc_types = [\n \"Claim\",\n \"Concluding Statement\",\n \"Counterclaim\",\n \"Evidence\",\n \"Lead\",\n \"Position\",\n \"Rebuttal\",\n]\ncls_tokens_map = {label: f\"[CLS_{label.upper()}]\" for label in disc_types}\nend_tokens_map = {label: f\"[END_{label.upper()}]\" for label in disc_types}\nI arranged the data as follows:
\nO label for discourse types if they are not a valid discourseSo, now we have for each essay:
\nNow it's time to train a model. I made a crucial change in the model's embedding layer that gave a good lift. The change was to add another embedding layer for discourse_types (input_types) and add the result to the original embeddings:
\nclass DebertaV2Embeddings(nn.Module):\n \"\"\"Construct the embeddings from word, position and token_type embeddings.\"\"\"\n\n def __init__(self, config):\n super().__init__()\n pad_token_id = getattr(config, \"pad_token_id\", 0)\n self.embedding_size = getattr(config, \"embedding_size\", config.hidden_size)\n self.word_embeddings = nn.Embedding(config.vocab_size, self.embedding_size, padding_idx=pad_token_id)\n self.disc_type_embeddings = nn.Embedding(9, self.embedding_size)\n .\n .\n .\n\n def forward(\n self, input_ids=None, token_type_ids=None, disc_type_ids=None, position_ids=None, mask=None, inputs_embeds=None\n ):\n .\n .\n if self.config.disc_type_vocab_size > 0:\n disc_type_embeddings = self.disc_type_embeddings(disc_type_ids)\n embeddings += disc_type_embeddings\n .\n .\n .\n return embeddings\nOther than all this, I used polynomial learning rate scheduler and AdamW optimizer from pytorch.
\nDuring inference, I just averaged the probabilities for CLS_{discourse_type} and END_{discourse_type} tokens to get the final label.
\nThat's it. Nothing too complicated to get a good score using a single model (5-fold). Running it with some variations and averaging the models gave us 14th rank on the leaderboard.
\nThe full code to train and infer is available here: https://www.kaggle.com/code/abhishek/0-577-single-model-full-code
\nI used my own library for training the models: tez. Please give it some love here: https://github.com/abhishekkrthakur/tez
\nIf you have any questions, please feel free to ask. It was a fun competition and I learnt a lot! :)
", "comments": [ { "id": 1913246, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-08-25T08:56:02.183000", "content": "Can be your next video explaining it😬
", "votes": 4, "replies": [ { "id": 1913365, "author_name": "nymfree", "author_url": "", "post_date": "2022-08-25T09:49:46.780000", "content": "+1 on this humble request.
", "votes": 2, "replies": [] }, { "id": 1913414, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2022-08-25T10:13:04.243000", "content": "dunno if this solution is worth a video. there are many more interesting solutions that have been shared here. but ill give it a thought. the only interesting part in my solution is modification of embedding layer in deberta :)
", "votes": 4, "replies": [] }, { "id": 1913764, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-08-25T13:50:57.623000", "content": "Many thanks for the solution, your approaches are a great learning experience for beginners like me to learn and augment our knowledge @abhishek !!
", "votes": 0, "replies": [] } ] }, { "id": 1914351, "author_name": "yukiya", "author_url": "", "post_date": "2022-08-26T02:56:09.547000", "content": "Thank you for sharing. I use your baseline kernel for sentence classification, but I was out of time and couldn't push much.
\nCongrats on the neat score with token classification !
\nCan I ask about when you combine embeddings, is there any pros and cons whether you add them up or you concatenate ?
Impressive work.. Thank you for sharing!
", "votes": 1, "replies": [] }, { "id": 1913231, "author_name": "gezi", "author_url": "", "post_date": "2022-08-25T08:44:19.300000", "content": "Thanks for sharing, token type embedding is intersting!
", "votes": 1, "replies": [] }, { "id": 1916803, "author_name": "🐳鲲(China)", "author_url": "", "post_date": "2022-08-28T06:56:02.163000", "content": "I checked the run_mlm.py code, as document said perplexity is not well defined for masked language models like BERT, so it may not a good metrics?
", "votes": 0, "replies": [] }, { "id": 1914448, "author_name": "Raja Biswas", "author_url": "", "post_date": "2022-08-26T05:31:16.470000", "content": "Thanks @abhishek for sharing the great solution. I think the embedding modification is a really cool idea. I have two queries:
\nThanks again for the writeup!
", "votes": 0, "replies": [] }, { "id": 1913947, "author_name": "Julien Chaumond", "author_url": "", "post_date": "2022-08-25T15:44:03.250000", "content": "Cool explanation, thanks for sharing ❤️
", "votes": 1, "replies": [] } ], "comment_count": 10, "messages": [ { "id": 1913200, "postDate": "2022-08-25T08:09:58.110Z", "content": "The first step to achieve this model was to train an mlm on the old dataset. I took all the data from previous feedback competition and fine-tuned deberta-v3-large using run_mlm.py script available in the transformers library. The perplexity in the end was around 4.5. I could have trained more but I didn't.
Now comes the more interesting part. I was inspired by @nbroad 's kernel that showed how we can use token classification for this problem. This was the kernel that piqued my interest in this competition and I decided to give it a go. The idea was simple but effective.
\nEssay texts were represented as follows:
\n[CLS]some text [CLS_{discourse_type}] some valid discourse text [END_{discourse_type}] ..... [SEP]\nMapping:
\ndisc_types = [\n \"Claim\",\n \"Concluding Statement\",\n \"Counterclaim\",\n \"Evidence\",\n \"Lead\",\n \"Position\",\n \"Rebuttal\",\n]\ncls_tokens_map = {label: f\"[CLS_{label.upper()}]\" for label in disc_types}\nend_tokens_map = {label: f\"[END_{label.upper()}]\" for label in disc_types}\nI arranged the data as follows:
\nO label for discourse types if they are not a valid discourseSo, now we have for each essay:
\nNow it's time to train a model. I made a crucial change in the model's embedding layer that gave a good lift. The change was to add another embedding layer for discourse_types (input_types) and add the result to the original embeddings:
\nclass DebertaV2Embeddings(nn.Module):\n \"\"\"Construct the embeddings from word, position and token_type embeddings.\"\"\"\n\n def __init__(self, config):\n super().__init__()\n pad_token_id = getattr(config, \"pad_token_id\", 0)\n self.embedding_size = getattr(config, \"embedding_size\", config.hidden_size)\n self.word_embeddings = nn.Embedding(config.vocab_size, self.embedding_size, padding_idx=pad_token_id)\n self.disc_type_embeddings = nn.Embedding(9, self.embedding_size)\n .\n .\n .\n\n def forward(\n self, input_ids=None, token_type_ids=None, disc_type_ids=None, position_ids=None, mask=None, inputs_embeds=None\n ):\n .\n .\n if self.config.disc_type_vocab_size > 0:\n disc_type_embeddings = self.disc_type_embeddings(disc_type_ids)\n embeddings += disc_type_embeddings\n .\n .\n .\n return embeddings\nOther than all this, I used polynomial learning rate scheduler and AdamW optimizer from pytorch.
\nDuring inference, I just averaged the probabilities for CLS_{discourse_type} and END_{discourse_type} tokens to get the final label.
\nThat's it. Nothing too complicated to get a good score using a single model (5-fold). Running it with some variations and averaging the models gave us 14th rank on the leaderboard.
\nThe full code to train and infer is available here: https://www.kaggle.com/code/abhishek/0-577-single-model-full-code
\nI used my own library for training the models: tez. Please give it some love here: https://github.com/abhishekkrthakur/tez
\nIf you have any questions, please feel free to ask. It was a fun competition and I learnt a lot! :)
", "rawMarkdown": "The first step to achieve this model was to train an mlm on the old dataset. I took all the data from previous feedback competition and fine-tuned deberta-v3-large using `run_mlm.py` script available in the transformers library. The perplexity in the end was around 4.5. I could have trained more but I didn't.\n\nNow comes the more interesting part. I was inspired by @nbroad 's kernel that showed how we can use token classification for this problem. This was the kernel that piqued my interest in this competition and I decided to give it a go. The idea was simple but effective.\n\nEssay texts were represented as follows:\n\n```\n[CLS]some text [CLS_{discourse_type}] some valid discourse text [END_{discourse_type}] ..... [SEP]\n```\n\nMapping:\n\n```\ndisc_types = [\n \"Claim\",\n \"Concluding Statement\",\n \"Counterclaim\",\n \"Evidence\",\n \"Lead\",\n \"Position\",\n \"Rebuttal\",\n]\ncls_tokens_map = {label: f\"[CLS_{label.upper()}]\" for label in disc_types}\nend_tokens_map = {label: f\"[END_{label.upper()}]\" for label in disc_types}\n```\n\nI arranged the data as follows:\n- Convert the data to old format (I had already made my kernel public for this part)\n- For each token, add on `O` label for discourse types if they are not a valid discourse\n- For each token, add -100 as label for discourse effectiveness if they are not a valid discourse\n- If the token belongs to one of the discourse cls or end token or is a valid discourse, I added discourse type id as their label\n- If the token belongs to one of the discourse cls or end token or is a valid discourse, I added discourse effectiveness id as their label\n- discourse effectiveness label was kept only for the CLS_{discourse_type} and END_{discourse_type} tokens\n\nSo, now we have for each essay:\n- input_ids\n- input_types (discourse type ids, one for each input id)\n- input_labels (the actual label, most are -100, only start of a discourse and end have an actual label)\n- attention_mask\n\nNow it's time to train a model. I made a crucial change in the model's embedding layer that gave a good lift. The change was to add another embedding layer for discourse_types (input_types) and add the result to the original embeddings:\n\n```python\nclass DebertaV2Embeddings(nn.Module):\n \"\"\"Construct the embeddings from word, position and token_type embeddings.\"\"\"\n\n def __init__(self, config):\n super().__init__()\n pad_token_id = getattr(config, \"pad_token_id\", 0)\n self.embedding_size = getattr(config, \"embedding_size\", config.hidden_size)\n self.word_embeddings = nn.Embedding(config.vocab_size, self.embedding_size, padding_idx=pad_token_id)\n self.disc_type_embeddings = nn.Embedding(9, self.embedding_size)\n .\n .\n .\n\n def forward(\n self, input_ids=None, token_type_ids=None, disc_type_ids=None, position_ids=None, mask=None, inputs_embeds=None\n ):\n .\n .\n if self.config.disc_type_vocab_size > 0:\n disc_type_embeddings = self.disc_type_embeddings(disc_type_ids)\n embeddings += disc_type_embeddings\n .\n .\n .\n return embeddings\n\n```\n\nOther than all this, I used polynomial learning rate scheduler and AdamW optimizer from pytorch.\n\nDuring inference, I just averaged the probabilities for CLS_{discourse_type} and END_{discourse_type} tokens to get the final label.\n\nThat's it. Nothing too complicated to get a good score using a single model (5-fold). Running it with some variations and averaging the models gave us 14th rank on the leaderboard.\n\nThe full code to train and infer is available here: https://www.kaggle.com/code/abhishek/0-577-single-model-full-code\n\nI used my own library for training the models: tez. Please give it some love here: https://github.com/abhishekkrthakur/tez\n\nIf you have any questions, please feel free to ask. It was a fun competition and I learnt a lot! :)\n", "votes": 77 }, { "id": 1913246, "postDate": "2022-08-25T08:56:02.183Z", "content": "Can be your next video explaining it😬
", "rawMarkdown": "Can be your next video explaining it😬", "votes": 4, "replies": [ { "id": 1913365, "postDate": "2022-08-25T09:49:46.780Z", "content": "+1 on this humble request.
", "rawMarkdown": "+1 on this humble request.", "votes": 2 }, { "id": 1913414, "postDate": "2022-08-25T10:13:04.243Z", "content": "dunno if this solution is worth a video. there are many more interesting solutions that have been shared here. but ill give it a thought. the only interesting part in my solution is modification of embedding layer in deberta :)
", "rawMarkdown": "dunno if this solution is worth a video. there are many more interesting solutions that have been shared here. but ill give it a thought. the only interesting part in my solution is modification of embedding layer in deberta :) ", "votes": 4 }, { "id": 1913764, "postDate": "2022-08-25T13:50:57.623Z", "content": "Many thanks for the solution, your approaches are a great learning experience for beginners like me to learn and augment our knowledge @abhishek !!
", "rawMarkdown": "Many thanks for the solution, your approaches are a great learning experience for beginners like me to learn and augment our knowledge @abhishek !!" } ] }, { "id": 1914351, "postDate": "2022-08-26T02:56:09.547Z", "content": "Thank you for sharing. I use your baseline kernel for sentence classification, but I was out of time and couldn't push much.
\nCongrats on the neat score with token classification !
\nCan I ask about when you combine embeddings, is there any pros and cons whether you add them up or you concatenate ?
Impressive work.. Thank you for sharing!
", "rawMarkdown": "Impressive work.. Thank you for sharing! ", "votes": 1 }, { "id": 1913231, "postDate": "2022-08-25T08:44:19.300Z", "content": "Thanks for sharing, token type embedding is intersting!
", "rawMarkdown": "Thanks for sharing, token type embedding is intersting!", "votes": 1 }, { "id": 1916803, "postDate": "2022-08-28T06:56:02.163Z", "content": "I checked the run_mlm.py code, as document said perplexity is not well defined for masked language models like BERT, so it may not a good metrics?
", "rawMarkdown": "I checked the run_mlm.py code, as document said perplexity is not well defined for masked language models like BERT, so it may not a good metrics?" }, { "id": 1914448, "postDate": "2022-08-26T05:31:16.470Z", "content": "Thanks @abhishek for sharing the great solution. I think the embedding modification is a really cool idea. I have two queries:
\nThanks again for the writeup!
", "rawMarkdown": "Thanks @abhishek for sharing the great solution. I think the embedding modification is a really cool idea. I have two queries:\n- if you add the discourse type embeddings to tokens, won't it be too easy for the model to predict the auxiliary discourse type target?\n- will it be better to use the same modification during mlm training as well?\n\nThanks again for the writeup! " }, { "id": 1913947, "postDate": "2022-08-25T15:44:03.250Z", "content": "Cool explanation, thanks for sharing ❤️
", "rawMarkdown": "Cool explanation, thanks for sharing ❤️", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:53:35Z", "competition_id": 35308, "competition_slug": "feedback-prize-effectiveness", "team_id": 8686673, "solution_url": "/competitions/feedback-prize-effectiveness/writeups/0-577-single-model-with-full-code" }