I’d like to thank the organisers for a great dataset and competition, and I’d especially like to thank my teammates @nazarov and @tikutiku who I have learnt a huge amount from.
\nWe used a 5 fold StratifiedGroupKFold method grouped by text id. Initially I was not validating with post-processing, but since PP was a major part of the final score I switched to use the min_thresh and proba_thresh methods used in the public notebooks. Initially our CV-LB agreement was good, but it seemed to diverge past the 0.710 LB boundary.
Like most other teams, we quickly found that although Longformer and BigBird were convenient, they were quite inefficient and models like deberta seemed to perform much better. @nazarov did a great job of building a pipeline to split and merge the long sequences using weighted overlaps (Tom also had a similar method before we merged) that enabled us to use the following models:
\nTypically trained for 5 epochs. We found that the results were highly sensitive to batch size, and BS=4 seemed to be a sweet spot, using a LR=1e-5 & cosine decay. I was able to train deberta-xlarge on my GPU using DeepSpeed, BS=2 and gradient accumulation=2.
\nOur models also used multisample dropout and my model used an attention layer in the head. We also trained a few models using the BIEO scheme. We also also replaced all the \"\\xa0\" characters with a space.
\nWe used a range of loss functions for diversity, Cross Entropy, a modified Dice Loss (since Dice and F1 are equivalent) and Tom was using Conditional Random Fields (CRF)
\nRandom token masking with probs in the 0.03-0.07 range.
\nThe only methods we used were the min_thresh and proba_thresh methods which were optimised using OOF predictions, the link evidence method and a method to ensure that there was only a single “Lead” prediction. Tom also made a heroic effort in the last few hours of the competition to use the essay topic clusters in PP, but we ran out of time. Unfortunately, we missed the gradient boosted trees post processing notebook.
The ensemble was tricky since the CRF method after the Viterbi decoding outputted hard labels instead of logits/probabilities . We found that taking the simple mean of the OHE CRF predictions with the softmax probabilities of the other models worked better than linear or 2nd stage stacking models.
\nI also managed to get a pipeline to create accurate token to word mappings which enabled the majority of the prediction splitting using tensors instead of expensive for/while loops. This gave quite a bit of speed up over the public notebook methods and allowed us to ensemble more models in the time limit.
\nTo choose the final ensemble models, I used a similar technique I used in CommonLit, where I start with a basket of all the models, and iteratively remove one model at a time based on how much it impacts the CV F1 score, until I have a basket of the most informative models that will fit into the time limit.
\nOur final two subs used:
\nI’d like to again thank my amazing teammates and I really enjoyed the competition.
\nUnfortunately my long wait for the GM title will have to be a little longer since I will be taking a 6-7 month break from Kaggle to thru-hike the full Pacific Crest Trail 🎒️. I’ll see you all again on the leaderboard towards the end of the year 💪️
", "comments": [ { "id": 1728167, "author_name": "RB", "author_url": "", "post_date": "2022-03-18T16:23:22.247000", "content": "Hi @anjum48 Thank you for your write up, I'm wondering which luke did you use luke-large-finetuned-conll-2003 or luke-large.
\nLuke had very different preprocessing than the rest.
We used studio-ousia/luke-large. I seem to remember there was no fast tokenizer, so we just used a regular roberta-large tokenizer.
Mind I ask how you guys manage to make CRF work for large models? I also tried CRF. For small models, the performance is alright, but for longformer and deberta-large, much worse(about 0.02 lower) than token-cls approach.
", "votes": 0, "replies": [ { "id": 1728434, "author_name": "Tom", "author_url": "", "post_date": "2022-03-18T22:26:57.997000", "content": "I used this repository : https://github.com/kmkurn/pytorch-crf and set the prohibited transition (for BIO) as
\ndef set_prohibited_transition(self):\n for end in [1,3,5,7,9,11,13]: # (I-Lead, I-Position, I-Evidence, I-Claim, I-Concluding Statement, I-Counterclaim, I-Rebuttal)\n for start in range(15):\n if start==end or start==end-1:\n continue\n nn.init.constant_(self.crf.transitions[start,end], -1e9)\nI once tried something like this with small models, converge faster but doesn't increase eval metric. I'll give another try for large models, thanks.
", "votes": 0, "replies": [] } ] } ], "comment_count": 5, "messages": [ { "id": 1724561, "postDate": "2022-03-16T10:26:50.110Z", "content": "I’d like to thank the organisers for a great dataset and competition, and I’d especially like to thank my teammates @nazarov and @tikutiku who I have learnt a huge amount from.
\nWe used a 5 fold StratifiedGroupKFold method grouped by text id. Initially I was not validating with post-processing, but since PP was a major part of the final score I switched to use the min_thresh and proba_thresh methods used in the public notebooks. Initially our CV-LB agreement was good, but it seemed to diverge past the 0.710 LB boundary.
Like most other teams, we quickly found that although Longformer and BigBird were convenient, they were quite inefficient and models like deberta seemed to perform much better. @nazarov did a great job of building a pipeline to split and merge the long sequences using weighted overlaps (Tom also had a similar method before we merged) that enabled us to use the following models:
\nTypically trained for 5 epochs. We found that the results were highly sensitive to batch size, and BS=4 seemed to be a sweet spot, using a LR=1e-5 & cosine decay. I was able to train deberta-xlarge on my GPU using DeepSpeed, BS=2 and gradient accumulation=2.
\nOur models also used multisample dropout and my model used an attention layer in the head. We also trained a few models using the BIEO scheme. We also also replaced all the \"\\xa0\" characters with a space.
\nWe used a range of loss functions for diversity, Cross Entropy, a modified Dice Loss (since Dice and F1 are equivalent) and Tom was using Conditional Random Fields (CRF)
\nRandom token masking with probs in the 0.03-0.07 range.
\nThe only methods we used were the min_thresh and proba_thresh methods which were optimised using OOF predictions, the link evidence method and a method to ensure that there was only a single “Lead” prediction. Tom also made a heroic effort in the last few hours of the competition to use the essay topic clusters in PP, but we ran out of time. Unfortunately, we missed the gradient boosted trees post processing notebook.
The ensemble was tricky since the CRF method after the Viterbi decoding outputted hard labels instead of logits/probabilities . We found that taking the simple mean of the OHE CRF predictions with the softmax probabilities of the other models worked better than linear or 2nd stage stacking models.
\nI also managed to get a pipeline to create accurate token to word mappings which enabled the majority of the prediction splitting using tensors instead of expensive for/while loops. This gave quite a bit of speed up over the public notebook methods and allowed us to ensemble more models in the time limit.
\nTo choose the final ensemble models, I used a similar technique I used in CommonLit, where I start with a basket of all the models, and iteratively remove one model at a time based on how much it impacts the CV F1 score, until I have a basket of the most informative models that will fit into the time limit.
\nOur final two subs used:
\nI’d like to again thank my amazing teammates and I really enjoyed the competition.
\nUnfortunately my long wait for the GM title will have to be a little longer since I will be taking a 6-7 month break from Kaggle to thru-hike the full Pacific Crest Trail 🎒️. I’ll see you all again on the leaderboard towards the end of the year 💪️
", "rawMarkdown": "I’d like to thank the organisers for a great dataset and competition, and I’d especially like to thank my teammates @nazarov and @tikutiku who I have learnt a huge amount from.\n\n# CV Strategy\nWe used a 5 fold `StratifiedGroupKFold` method grouped by text id. Initially I was not validating with post-processing, but since PP was a major part of the final score I switched to use the `min_thresh` and `proba_thresh` methods used in the public notebooks. Initially our CV-LB agreement was good, but it seemed to diverge past the 0.710 LB boundary.\n\n# Models\nLike most other teams, we quickly found that although Longformer and BigBird were convenient, they were quite inefficient and models like deberta seemed to perform much better. @nazarov did a great job of building a pipeline to split and merge the long sequences using weighted overlaps (Tom also had a similar method before we merged) that enabled us to use the following models:\n- deberta-large CV: ~0.697 (<- best single model public LB 0.704)\n- deberta-xlarge CV: ~0.698-0.700\n- bart-large CV: ~0.680-0.692\n- muppet-large CV: 0.670\n- luke-large CV: 0.691\n\n# Training strategy\n\n## Parameters\nTypically trained for 5 epochs. We found that the results were highly sensitive to batch size, and BS=4 seemed to be a sweet spot, using a LR=1e-5 & cosine decay. I was able to train deberta-xlarge on my GPU using DeepSpeed, BS=2 and gradient accumulation=2.\n\nOur models also used multisample dropout and my model used an attention layer in the head. We also trained a few models using the BIEO scheme. We also also replaced all the \"\\xa0\" characters with a space.\n\n## Loss\nWe used a range of loss functions for diversity, Cross Entropy, a modified Dice Loss (since Dice and F1 are equivalent) and Tom was using [Conditional Random Fields (CRF)](https://pytorch-crf.readthedocs.io/en/stable/)\n\n## Augmentation\nRandom token masking with probs in the 0.03-0.07 range.\n\n# Post-processing\nThe only methods we used were the `min_thresh` and `proba_thresh` methods which were optimised using OOF predictions, the link evidence method and a method to ensure that there was only a single “Lead” prediction. Tom also made a heroic effort in the last few hours of the competition to use the essay topic clusters in PP, but we ran out of time. Unfortunately, we missed the gradient boosted trees post processing notebook.\n\n# Ensemble\nThe ensemble was tricky since the CRF method after the Viterbi decoding outputted hard labels instead of logits/probabilities . We found that taking the simple mean of the OHE CRF predictions with the softmax probabilities of the other models worked better than linear or 2nd stage stacking models. \n\nI also managed to get a pipeline to create accurate token to word mappings which enabled the majority of the prediction splitting using tensors instead of expensive for/while loops. This gave quite a bit of speed up over the public notebook methods and allowed us to ensemble more models in the time limit.\n\nTo choose the final ensemble models, I used a similar technique I used in CommonLit, where I start with a basket of all the models, and iteratively remove one model at a time based on how much it impacts the CV F1 score, until I have a basket of the most informative models that will fit into the time limit.\n\nOur final two subs used:\n- 6 models (2x deberta-xlarge, deberta-large-mnli, deberta-large (CRF & BIEO), bart-large (CRF), muppet-large), using 3 out of 5 folds (18 checkpoints). CV: 0.716, Public: 0.710, Private: 0.723\n- 11 models (as above plus a few more variants of deberta & bart), using 2 out of 5 folds (22 checkpoints), CV: 0.717, Public: 0.710, Private: 0.721\n\n# What did not work\n- [Mixup Transformers](https://arxiv.org/pdf/2010.02394.pdf)\n- Deberta versions 2 & 3 (we built a custom wrapper for sentencepiece to get offset mappings)\n- Using essay topic clusters as either an additional target or input features\n- 2nd stage stacking models\n- Larger batch sizes\n\n# Final words\nI’d like to again thank my amazing teammates and I really enjoyed the competition.\n\nUnfortunately my long wait for the GM title will have to be a little longer since I will be taking a 6-7 month break from Kaggle to thru-hike the full Pacific Crest Trail 🎒️. I’ll see you all again on the leaderboard towards the end of the year 💪️\n", "votes": 27 }, { "id": 1728167, "postDate": "2022-03-18T16:23:22.247Z", "content": "Hi @anjum48 Thank you for your write up, I'm wondering which luke did you use luke-large-finetuned-conll-2003 or luke-large.
\nLuke had very different preprocessing than the rest.
We used studio-ousia/luke-large. I seem to remember there was no fast tokenizer, so we just used a regular roberta-large tokenizer.
Mind I ask how you guys manage to make CRF work for large models? I also tried CRF. For small models, the performance is alright, but for longformer and deberta-large, much worse(about 0.02 lower) than token-cls approach.
", "rawMarkdown": "Mind I ask how you guys manage to make CRF work for large models? I also tried CRF. For small models, the performance is alright, but for longformer and deberta-large, much worse(about 0.02 lower) than token-cls approach.", "replies": [ { "id": 1728434, "postDate": "2022-03-18T22:26:57.997Z", "content": "I used this repository : https://github.com/kmkurn/pytorch-crf and set the prohibited transition (for BIO) as
\ndef set_prohibited_transition(self):\n for end in [1,3,5,7,9,11,13]: # (I-Lead, I-Position, I-Evidence, I-Claim, I-Concluding Statement, I-Counterclaim, I-Rebuttal)\n for start in range(15):\n if start==end or start==end-1:\n continue\n nn.init.constant_(self.crf.transitions[start,end], -1e9)\nI once tried something like this with small models, converge faster but doesn't increase eval metric. I'll give another try for large models, thanks.
", "rawMarkdown": "I once tried something like this with small models, converge faster but doesn't increase eval metric. I'll give another try for large models, thanks." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:35:45Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7903729, "solution_url": "/competitions/feedback-prize-2021/writeups/datasaurus-ln-tom-26th-place-solution-datasaurus-l" }