{
  "id": 313718,
  "title": "10th solution",
  "content": "<p>First of all, much thanks to the competition hosts and Kaggle administrators for holding such an interesting competition!<br>\nHere is the our 10th solution:</p>\n<p><strong>Token classification (NER)</strong><br>\nWe created 6 deberta-large models and 1 deberta-xlarge model then took weighted average of probability of each token (the weight is according to LB).<br>\nPoints are following:</p>\n<ul>\n<li>Various versions of deberta-large models with these modifications:<ul>\n<li>Tagging of tokens (begin, inside, outside / begin, inside, end, outside / begin, inside, end, begin-of-outside, inside-of-outside, end-of-outside).</li>\n<li>Pretrain with MLM (Masked Language Model) method.</li>\n<li>Replace tokens with masked tokens randomly when training.</li>\n<li>Add LSTM before classification head.</li>\n<li>Higher and lower learning rate.</li>\n<li>Multi-task learning of ordinary NER and NER focused on specific discourses.<br>\nAdditional loss with higher weight on begin tokens.</li></ul></li>\n<li>Train deberta-xlarge<ul>\n<li>Deberta-xlarge often failed to converge. To make it converged, we used warm-up scheduler and tried several patterns of learning rate.</li></ul></li>\n<li>Ensemble<ul>\n<li>We tried various backbones like Bigbird etc., but found ensemble of deberta-large models achieves the highest score.</li>\n<li>Especially ensemble of models with different tokenizers lowered score when decomposing tokens to char-level to take weighted average of probability.</li>\n<li>Maybe ensemble of models with shared tokenizer is desirable.</li></ul></li>\n<li>Credit<ul>\n<li>Architectures of models are based on <a href=\"https://www.kaggle.com/Abhishek\" target=\"_blank\">@Abhishek</a>'s code.</li>\n<li>Experiment results of deberta etc. from <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> were very helpful.</li>\n<li>Thanks to <a href=\"https://www.kaggle.com/Abhiskek\" target=\"_blank\">@Abhiskek</a> and <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a>, we could focus on try-and-error instead of spending time on building architectures of models.</li></ul></li>\n</ul>\n<p><strong>Postprocessing with LGBM</strong><br>\nPoints are following:</p>\n<ul>\n<li>First, generate predictionstrings from results of token classification (it's same with public notebooks' \"create_submission\" function).</li>\n<li>Second, create a LGBM model that predicts whether generated predictionstrings are TP or not, then outputs the probability of TP.<ul>\n<li>Features are mainly come from aggregation of probability of token classification results.<br>\n75 aggregation features are generated from each material of ensemble (5 types of aggregation (min, max, mean, 20 percentile, 80 percentile) * 15 types of tokens (Beginning, Inside * 7 discourses + Outside))</li>\n<li>Other feats are about 15, length of prediction strings, length of essays etc.</li></ul></li>\n<li>Finally, filter predictionstrings with probability of TP.</li>\n<li>Without this postprocessing: Public: 0.716 Private 0.724, With this postprocessing Public: 0.719 Private 0.727</li>\n</ul>\n<p><strong>Other postprocessing</strong><br>\nFollowed by the filtering written in previous section , these postprocessing methods are applied.</p>\n<ul>\n<li>Define the starts of predictionstrings as these patterns: begin token or a token that differs from the previous class (exp. I-Claim, B-Claim &lt;-).</li>\n<li>Calculate mean probability (begin + inside) of predictionstrings then filter predictionstrings by it and length of predictionstrings.</li>\n<li>If multiple Lead, Position and Concluding Statement are found, keep only one with the highest mean probability.</li>\n<li>Apply <a href=\"https://www.kaggle.com/kaggleqrdl/tensorflow-longformer-ner-postprocessing\" target=\"_blank\">link evidence</a> on Evidence, Counterclaim and Rebuttal (much thanks to <a href=\"https://www.kaggle.com/kaggleqrdl\" target=\"_blank\">@kaggleqrdl</a>!).</li>\n</ul>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 1728053,
      "postDate": "2022-03-18T14:55:02.997Z",
      "content": "<p>First of all, much thanks to the competition hosts and Kaggle administrators for holding such an interesting competition!<br>\nHere is the our 10th solution:</p>\n<p><strong>Token classification (NER)</strong><br>\nWe created 6 deberta-large models and 1 deberta-xlarge model then took weighted average of probability of each token (the weight is according to LB).<br>\nPoints are following:</p>\n<ul>\n<li>Various versions of deberta-large models with these modifications:<ul>\n<li>Tagging of tokens (begin, inside, outside / begin, inside, end, outside / begin, inside, end, begin-of-outside, inside-of-outside, end-of-outside).</li>\n<li>Pretrain with MLM (Masked Language Model) method.</li>\n<li>Replace tokens with masked tokens randomly when training.</li>\n<li>Add LSTM before classification head.</li>\n<li>Higher and lower learning rate.</li>\n<li>Multi-task learning of ordinary NER and NER focused on specific discourses.<br>\nAdditional loss with higher weight on begin tokens.</li></ul></li>\n<li>Train deberta-xlarge<ul>\n<li>Deberta-xlarge often failed to converge. To make it converged, we used warm-up scheduler and tried several patterns of learning rate.</li></ul></li>\n<li>Ensemble<ul>\n<li>We tried various backbones like Bigbird etc., but found ensemble of deberta-large models achieves the highest score.</li>\n<li>Especially ensemble of models with different tokenizers lowered score when decomposing tokens to char-level to take weighted average of probability.</li>\n<li>Maybe ensemble of models with shared tokenizer is desirable.</li></ul></li>\n<li>Credit<ul>\n<li>Architectures of models are based on <a href=\"https://www.kaggle.com/Abhishek\" target=\"_blank\">@Abhishek</a>'s code.</li>\n<li>Experiment results of deberta etc. from <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> were very helpful.</li>\n<li>Thanks to <a href=\"https://www.kaggle.com/Abhiskek\" target=\"_blank\">@Abhiskek</a> and <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a>, we could focus on try-and-error instead of spending time on building architectures of models.</li></ul></li>\n</ul>\n<p><strong>Postprocessing with LGBM</strong><br>\nPoints are following:</p>\n<ul>\n<li>First, generate predictionstrings from results of token classification (it's same with public notebooks' \"create_submission\" function).</li>\n<li>Second, create a LGBM model that predicts whether generated predictionstrings are TP or not, then outputs the probability of TP.<ul>\n<li>Features are mainly come from aggregation of probability of token classification results.<br>\n75 aggregation features are generated from each material of ensemble (5 types of aggregation (min, max, mean, 20 percentile, 80 percentile) * 15 types of tokens (Beginning, Inside * 7 discourses + Outside))</li>\n<li>Other feats are about 15, length of prediction strings, length of essays etc.</li></ul></li>\n<li>Finally, filter predictionstrings with probability of TP.</li>\n<li>Without this postprocessing: Public: 0.716 Private 0.724, With this postprocessing Public: 0.719 Private 0.727</li>\n</ul>\n<p><strong>Other postprocessing</strong><br>\nFollowed by the filtering written in previous section , these postprocessing methods are applied.</p>\n<ul>\n<li>Define the starts of predictionstrings as these patterns: begin token or a token that differs from the previous class (exp. I-Claim, B-Claim &lt;-).</li>\n<li>Calculate mean probability (begin + inside) of predictionstrings then filter predictionstrings by it and length of predictionstrings.</li>\n<li>If multiple Lead, Position and Concluding Statement are found, keep only one with the highest mean probability.</li>\n<li>Apply <a href=\"https://www.kaggle.com/kaggleqrdl/tensorflow-longformer-ner-postprocessing\" target=\"_blank\">link evidence</a> on Evidence, Counterclaim and Rebuttal (much thanks to <a href=\"https://www.kaggle.com/kaggleqrdl\" target=\"_blank\">@kaggleqrdl</a>!).</li>\n</ul>",
      "rawMarkdown": "First of all, much thanks to the competition hosts and Kaggle administrators for holding such an interesting competition!\nHere is the our 10th solution:\n\n**Token classification (NER)**\nWe created 6 deberta-large models and 1 deberta-xlarge model then took weighted average of probability of each token (the weight is according to LB).\nPoints are following:\n\n- Various versions of deberta-large models with these modifications:\n  - Tagging of tokens (begin, inside, outside / begin, inside, end, outside / begin, inside, end, begin-of-outside, inside-of-outside, end-of-outside).\n  - Pretrain with MLM (Masked Language Model) method.\n  - Replace tokens with masked tokens randomly when training.\n  - Add LSTM before classification head.\n  - Higher and lower learning rate.\n  - Multi-task learning of ordinary NER and NER focused on specific discourses.\nAdditional loss with higher weight on begin tokens.\n- Train deberta-xlarge\n  - Deberta-xlarge often failed to converge. To make it converged, we used warm-up scheduler and tried several patterns of learning rate.\n- Ensemble\n  - We tried various backbones like Bigbird etc., but found ensemble of deberta-large models achieves the highest score.\n  - Especially ensemble of models with different tokenizers lowered score when decomposing tokens to char-level to take weighted average of probability.\n  - Maybe ensemble of models with shared tokenizer is desirable.\n- Credit\n  - Architectures of models are based on @Abhishek's code.\n  - Experiment results of deberta etc. from @hengck23 were very helpful.\n  - Thanks to @Abhiskek and @hengck23, we could focus on try-and-error instead of spending time on building architectures of models.\n\n**Postprocessing with LGBM**\nPoints are following:\n\n- First, generate predictionstrings from results of token classification (it's same with public notebooks' \"create_submission\" function).\n- Second, create a LGBM model that predicts whether generated predictionstrings are TP or not, then outputs the probability of TP.\n  - Features are mainly come from aggregation of probability of token classification results.\n75 aggregation features are generated from each material of ensemble (5 types of aggregation (min, max, mean, 20 percentile, 80 percentile) * 15 types of tokens (Beginning, Inside * 7 discourses + Outside))\n  - Other feats are about 15, length of prediction strings, length of essays etc.\n- Finally, filter predictionstrings with probability of TP.\n- Without this postprocessing: Public: 0.716 Private 0.724, With this postprocessing Public: 0.719 Private 0.727\n\n**Other postprocessing**\nFollowed by the filtering written in previous section , these postprocessing methods are applied.\n\n- Define the starts of predictionstrings as these patterns: begin token or a token that differs from the previous class (exp. I-Claim, B-Claim <-).\n- Calculate mean probability (begin + inside) of predictionstrings then filter predictionstrings by it and length of predictionstrings.\n- If multiple Lead, Position and Concluding Statement are found, keep only one with the highest mean probability.\n- Apply [link evidence](https://www.kaggle.com/kaggleqrdl/tensorflow-longformer-ner-postprocessing) on Evidence, Counterclaim and Rebuttal (much thanks to @kaggleqrdl!).",
      "votes": 33
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T12:37:11Z",
  "competition_id": 31779,
  "competition_slug": "feedback-prize-2021",
  "team_id": 7886854,
  "solution_url": "/competitions/feedback-prize-2021/writeups/we-hope-for-peace-10th-solution"
}