First of all, a huge thank you to the competition hosts and Kaggle. I spent an absurd amount of time on this competition, and I’ve definitely learned a lot about QA, multilingual models, and Kaggle in general.
\nThis was basically the first time I put myself out into the public section of Kaggle, and I’m so happy for the following achievements since the start of this competition:
\nThe last model is probably the only novel thing I did, and also the reason I placed this high. I noticed that the contexts for these passages were pretty long, and from a bit of research I saw that BigBird and LongFormer were the two architectures designed for long contexts without having the compute time blow up. I also noticed that the BigBird authors indicated that they did a “warm start” from a BERT checkpoint when training their model that reduced the number of training steps, so I figured I’d do the same from XLM-R or MuRIL (see page 7 under section called Pretraining and MLM). Since MuRIL is basically identical to BERT, all of the weights (except one module) loaded in to a BigBird model without trouble. The thing that caught me up for a bit was the fact that the position embeddings (which are learned) were going from (512, 768) to (4096, 768) (max_seq_length, hidden_size). I followed the Longformer authors and ended up tiling the original position embeddings 8 times to make it fit ([see page 6 under section Position Embeddings](Position Embeddings)).
\nI trained for 150k steps on TPU v3-8 using Flax with a batch size of 1 using Masked Language Modeling as the training task. I used the Tamil, Hindi, Telugu, Urdu, and English sections of mc4 and stuck it straight into the model without cleaning. Doing whole-word masking and data cleaning would probably help significantly, but I didn’t have the time. I did the same for the large size, but I could only fit a 1024 sequence length and I only trained for 90k steps at batch size 1. The base model got up to 75% accuracy and the large model got 71% on MLM in the eval set.
\nFrom there, my MuRIL Large, XLM-R, and Muril Large BigBird models were trained first on SQuAD2 and then on XQuAD, MLQA, TyDiQA (telugu), and chaii datasets, validating only on a stratified chaii fold. Since I still had compute time, I did all of this easily and quickly on TPU v3-8 using Flax.
\nRemBERT was a pain to train on Kaggle GPUs, so I used Tensorflow and Kaggle TPUs. This made it easy, and I convinced myself that having some models in Tensorflow provided model diversity. I trained it on SQuAD2 and then TyDiQA, MLQA, XQuAD.
Here is the final notebook that got my highest score. It is messy and wasn't meant to be my final submission 😅
\nI didn't actually get a submission out with an ensemble. My highest score without ensembling different models was 6 folds of MuRIL BigBird Large. I would never be able to get 23 folds to finish in time, but I was planning on splitting the passages based on length at around 10k characters. Under 10k go to XLMR, MuRIL Large, RemBERT, and MuRIL BigBird Large. Above 10k would just be MuRIL BigBird Large. Some of those contexts are so long especially when broken up with 384/128.
\nFor ensembling, I tried voting but ran out of time before getting something to work. I ultimately converted the token logits to character logits by averaging the overlap between multiple sections and concatenating them together. Once they are character logits, they can be averaged across all models without worrying about lining up models with different tokenizers. I got this working with a few hours left, and I made one small mistake in my submission, which made the notebooks exceed the submission time. It did seem like a promising way of combining models. Here is the notebook I didn't finish in time that has character-level averaging. It's pretty messy because I was cramming stuff in earlier today to try to get it done.
\nMuch thanks to TRC for the compute time on v3-8 TPUs. All of my models are now available on my Hugging Face profile.
\nCongrats Nicholas! I'd replace that turtle with a kangaroo, your jump in the last days was impressive! Awesome execution with Longformer, I was thinking about it but gave up without trying. I vote you the MVP of this competition, thanks for all the sharing and discussions! And congrats on the new job!
", "votes": 3, "replies": [ { "id": 1586036, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-11-17T18:41:13.013000", "content": "Thanks for voting me MVP! I humbly accept the award 😄
\nI tried hard this competition to learn as much as I could and share as much as possible while keeping a few tricks up my sleeve. Your early notebooks laid the foundation that many of us used throughout the competition. Thank you for your hard work, and congratulations on the 1st place finish 🎉
\nI'll change my emoji to a kangaroo now 😊
", "votes": 1, "replies": [] } ] }, { "id": 1583729, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-11-16T03:02:16.457000", "content": "Congrats @nbroad on results, nice jump
", "votes": 3, "replies": [] }, { "id": 1589777, "author_name": "Parv Yadav", "author_url": "", "post_date": "2021-11-20T15:02:09.130000", "content": "Well written @nbroad !😊
", "votes": 1, "replies": [] }, { "id": 1587815, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-19T00:23:44.230000", "content": "Thank you for sharing @nbroad and thank you for your post.
", "votes": 1, "replies": [] }, { "id": 1586347, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-11-18T02:22:24.660000", "content": "Very well written, nice approach .. well deserved @nbroad on your win. Congratulations.
", "votes": 1, "replies": [] }, { "id": 1585168, "author_name": "adldotori", "author_url": "", "post_date": "2021-11-17T06:12:25.080000", "content": "Congrats. i'm envious of gold medal.
", "votes": 1, "replies": [ { "id": 1586368, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-11-18T02:46:25.047000", "content": "Me too! I only got a silver 😄
", "votes": 0, "replies": [] } ] }, { "id": 1584499, "author_name": "Mathurin Ache", "author_url": "", "post_date": "2021-11-16T15:07:07.250000", "content": "Well done @nbroad and thank you for your post.
", "votes": 1, "replies": [] }, { "id": 1584273, "author_name": "Zacchaeus", "author_url": "", "post_date": "2021-11-16T12:05:50.233000", "content": "Just went thru ur solution again. The BigBird \"warm start\" thing is really amazing. I tried longformer-base from MLMed scratch and it worked really bad. Also, TPU and tensorflow are really hard for me. Implementing the novel thing with TPU is even more amazing.
", "votes": 1, "replies": [ { "id": 1584360, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-11-16T13:36:28.510000", "content": "I also tried without doing a warm start and it was terrible 😄
\nAnd I didn't use Tensorflow to train the BigBird model - I used Jax/Flax. Hugging Face has a couple of examples that make it dead simple to get going. I'm sure there are more optimizations and details that I could have done, but I didn't have the time nor experience to learn them in time for this competition.
\nAlso, this would have been extremely expensive without using the TRC. I'm curious to see if Kaggle takes a stance on whether competitors can use TRC in the future
", "votes": 1, "replies": [] } ] }, { "id": 1583672, "author_name": "DrHB", "author_url": "", "post_date": "2021-11-16T01:52:58.053000", "content": "Great finish! Congratulations !
", "votes": 1, "replies": [] }, { "id": 1583659, "author_name": "Kishal Mandal", "author_url": "", "post_date": "2021-11-16T01:39:38.853000", "content": "1024!!!! blew my mind. Congratulations for that huge jump :)
", "votes": 1, "replies": [ { "id": 1583698, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-11-16T02:16:10.793000", "content": "I wanted to run a 4k seq length model… but it was too big to train!
\nSometimes the shakeups work in your favor, sometimes they don't. They more experience you have, the less likely it will be to drop. Keep practicing!
\nThanks for all of the discussion and notebooks! See you in the next one!
\nGood job for your first public competition! Compliments!
", "votes": 1, "replies": [] }, { "id": 1583576, "author_name": "Zacchaeus", "author_url": "", "post_date": "2021-11-16T00:30:06.820000", "content": "Thank you @nbroad for actively sharing during the competition. 👍
", "votes": 1, "replies": [ { "id": 1583640, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2021-11-16T01:22:13.123000", "content": "Thanks for the discussions and posts!
", "votes": 1, "replies": [] } ] }, { "id": 1583610, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T01:01:26.197000", "content": "Congratulations. Fantastic solo finish! It appears that RemBERT and MuRIL Large BigBird did very well on private LB.
Thanks Chris! It means a lot to get praise from you 😊
", "votes": 1, "replies": [] } ] }, { "id": 1693411, "author_name": "Tetiana Yakovlieva", "author_url": "", "post_date": "2022-02-16T16:53:59.253000", "content": "Thank you for sharing @nbroad !
", "votes": 0, "replies": [] } ], "comment_count": 20, "messages": [ { "id": 1583557, "postDate": "2021-11-16T00:10:10.560Z", "content": "First of all, a huge thank you to the competition hosts and Kaggle. I spent an absurd amount of time on this competition, and I’ve definitely learned a lot about QA, multilingual models, and Kaggle in general.
\nThis was basically the first time I put myself out into the public section of Kaggle, and I’m so happy for the following achievements since the start of this competition:
\nThe last model is probably the only novel thing I did, and also the reason I placed this high. I noticed that the contexts for these passages were pretty long, and from a bit of research I saw that BigBird and LongFormer were the two architectures designed for long contexts without having the compute time blow up. I also noticed that the BigBird authors indicated that they did a “warm start” from a BERT checkpoint when training their model that reduced the number of training steps, so I figured I’d do the same from XLM-R or MuRIL (see page 7 under section called Pretraining and MLM). Since MuRIL is basically identical to BERT, all of the weights (except one module) loaded in to a BigBird model without trouble. The thing that caught me up for a bit was the fact that the position embeddings (which are learned) were going from (512, 768) to (4096, 768) (max_seq_length, hidden_size). I followed the Longformer authors and ended up tiling the original position embeddings 8 times to make it fit ([see page 6 under section Position Embeddings](Position Embeddings)).
\nI trained for 150k steps on TPU v3-8 using Flax with a batch size of 1 using Masked Language Modeling as the training task. I used the Tamil, Hindi, Telugu, Urdu, and English sections of mc4 and stuck it straight into the model without cleaning. Doing whole-word masking and data cleaning would probably help significantly, but I didn’t have the time. I did the same for the large size, but I could only fit a 1024 sequence length and I only trained for 90k steps at batch size 1. The base model got up to 75% accuracy and the large model got 71% on MLM in the eval set.
\nFrom there, my MuRIL Large, XLM-R, and Muril Large BigBird models were trained first on SQuAD2 and then on XQuAD, MLQA, TyDiQA (telugu), and chaii datasets, validating only on a stratified chaii fold. Since I still had compute time, I did all of this easily and quickly on TPU v3-8 using Flax.
\nRemBERT was a pain to train on Kaggle GPUs, so I used Tensorflow and Kaggle TPUs. This made it easy, and I convinced myself that having some models in Tensorflow provided model diversity. I trained it on SQuAD2 and then TyDiQA, MLQA, XQuAD.
Here is the final notebook that got my highest score. It is messy and wasn't meant to be my final submission 😅
\nI didn't actually get a submission out with an ensemble. My highest score without ensembling different models was 6 folds of MuRIL BigBird Large. I would never be able to get 23 folds to finish in time, but I was planning on splitting the passages based on length at around 10k characters. Under 10k go to XLMR, MuRIL Large, RemBERT, and MuRIL BigBird Large. Above 10k would just be MuRIL BigBird Large. Some of those contexts are so long especially when broken up with 384/128.
\nFor ensembling, I tried voting but ran out of time before getting something to work. I ultimately converted the token logits to character logits by averaging the overlap between multiple sections and concatenating them together. Once they are character logits, they can be averaged across all models without worrying about lining up models with different tokenizers. I got this working with a few hours left, and I made one small mistake in my submission, which made the notebooks exceed the submission time. It did seem like a promising way of combining models. Here is the notebook I didn't finish in time that has character-level averaging. It's pretty messy because I was cramming stuff in earlier today to try to get it done.
\nMuch thanks to TRC for the compute time on v3-8 TPUs. All of my models are now available on my Hugging Face profile.
\nCongrats Nicholas! I'd replace that turtle with a kangaroo, your jump in the last days was impressive! Awesome execution with Longformer, I was thinking about it but gave up without trying. I vote you the MVP of this competition, thanks for all the sharing and discussions! And congrats on the new job!
", "rawMarkdown": "Congrats Nicholas! I'd replace that turtle with a kangaroo, your jump in the last days was impressive! Awesome execution with Longformer, I was thinking about it but gave up without trying. I vote you the MVP of this competition, thanks for all the sharing and discussions! And congrats on the new job!", "votes": 3, "replies": [ { "id": 1586036, "postDate": "2021-11-17T18:41:13.013Z", "content": "Thanks for voting me MVP! I humbly accept the award 😄
\nI tried hard this competition to learn as much as I could and share as much as possible while keeping a few tricks up my sleeve. Your early notebooks laid the foundation that many of us used throughout the competition. Thank you for your hard work, and congratulations on the 1st place finish 🎉
\nI'll change my emoji to a kangaroo now 😊
", "rawMarkdown": "Thanks for voting me MVP! I humbly accept the award 😄\n\nI tried hard this competition to learn as much as I could and share as much as possible while keeping a few tricks up my sleeve. Your early notebooks laid the foundation that many of us used throughout the competition. Thank you for your hard work, and congratulations on the 1st place finish 🎉\n\nI'll change my emoji to a kangaroo now 😊", "votes": 1 } ] }, { "id": 1583729, "postDate": "2021-11-16T03:02:16.457Z", "content": "Congrats @nbroad on results, nice jump
", "rawMarkdown": "Congrats @nbroad on results, nice jump", "votes": 3 }, { "id": 1589777, "postDate": "2021-11-20T15:02:09.130Z", "content": "Well written @nbroad !😊
", "rawMarkdown": "Well written @nbroad !😊", "votes": 1 }, { "id": 1587815, "postDate": "2021-11-19T00:23:44.230Z", "content": "Thank you for sharing @nbroad and thank you for your post.
", "rawMarkdown": "Thank you for sharing @nbroad and thank you for your post.", "votes": 1 }, { "id": 1586347, "postDate": "2021-11-18T02:22:24.660Z", "content": "Very well written, nice approach .. well deserved @nbroad on your win. Congratulations.
", "rawMarkdown": "Very well written, nice approach .. well deserved @nbroad on your win. Congratulations. ", "votes": 1 }, { "id": 1585168, "postDate": "2021-11-17T06:12:25.080Z", "content": "Congrats. i'm envious of gold medal.
", "rawMarkdown": "Congrats. i'm envious of gold medal.", "votes": 1, "replies": [ { "id": 1586368, "postDate": "2021-11-18T02:46:25.047Z", "content": "Me too! I only got a silver 😄
", "rawMarkdown": "Me too! I only got a silver 😄" } ] }, { "id": 1584499, "postDate": "2021-11-16T15:07:07.250Z", "content": "Well done @nbroad and thank you for your post.
", "rawMarkdown": "Well done @nbroad and thank you for your post.\n", "votes": 1 }, { "id": 1584273, "postDate": "2021-11-16T12:05:50.233Z", "content": "Just went thru ur solution again. The BigBird \"warm start\" thing is really amazing. I tried longformer-base from MLMed scratch and it worked really bad. Also, TPU and tensorflow are really hard for me. Implementing the novel thing with TPU is even more amazing.
", "rawMarkdown": "Just went thru ur solution again. The BigBird \"warm start\" thing is really amazing. I tried longformer-base from MLMed scratch and it worked really bad. Also, TPU and tensorflow are really hard for me. Implementing the novel thing with TPU is even more amazing.", "votes": 1, "replies": [ { "id": 1584360, "postDate": "2021-11-16T13:36:28.510Z", "content": "I also tried without doing a warm start and it was terrible 😄
\nAnd I didn't use Tensorflow to train the BigBird model - I used Jax/Flax. Hugging Face has a couple of examples that make it dead simple to get going. I'm sure there are more optimizations and details that I could have done, but I didn't have the time nor experience to learn them in time for this competition.
\nAlso, this would have been extremely expensive without using the TRC. I'm curious to see if Kaggle takes a stance on whether competitors can use TRC in the future
", "rawMarkdown": "I also tried without doing a warm start and it was terrible 😄\n\nAnd I didn't use Tensorflow to train the BigBird model - I used Jax/Flax. [Hugging Face has a couple of examples](https://github.com/huggingface/transformers/tree/master/examples/flax) that make it dead simple to get going. I'm sure there are more optimizations and details that I could have done, but I didn't have the time nor experience to learn them in time for this competition.\n\nAlso, this would have been extremely expensive without using the TRC. I'm curious to see if Kaggle takes a stance on whether competitors can use TRC in the future", "votes": 1 } ] }, { "id": 1583672, "postDate": "2021-11-16T01:52:58.053Z", "content": "Great finish! Congratulations !
", "rawMarkdown": "Great finish! Congratulations ! ", "votes": 1 }, { "id": 1583659, "postDate": "2021-11-16T01:39:38.853Z", "content": "1024!!!! blew my mind. Congratulations for that huge jump :)
", "rawMarkdown": "1024!!!! blew my mind. Congratulations for that huge jump :)", "votes": 1, "replies": [ { "id": 1583698, "postDate": "2021-11-16T02:16:10.793Z", "content": "I wanted to run a 4k seq length model… but it was too big to train!
\nSometimes the shakeups work in your favor, sometimes they don't. They more experience you have, the less likely it will be to drop. Keep practicing!
\nThanks for all of the discussion and notebooks! See you in the next one!
\nGood job for your first public competition! Compliments!
", "rawMarkdown": "Good job for your first public competition! Compliments!", "votes": 1 }, { "id": 1583576, "postDate": "2021-11-16T00:30:06.820Z", "content": "Thank you @nbroad for actively sharing during the competition. 👍
", "rawMarkdown": "Thank you @nbroad for actively sharing during the competition. 👍", "votes": 1, "replies": [ { "id": 1583640, "postDate": "2021-11-16T01:22:13.123Z", "content": "Thanks for the discussions and posts!
", "rawMarkdown": "Thanks for the discussions and posts! ", "votes": 1 } ] }, { "id": 1583610, "postDate": "2021-11-16T01:01:26.197Z", "content": "Congratulations. Fantastic solo finish! It appears that RemBERT and MuRIL Large BigBird did very well on private LB.
Thanks Chris! It means a lot to get praise from you 😊
", "rawMarkdown": "Thanks Chris! It means a lot to get praise from you 😊", "votes": 1 } ] }, { "id": 1693411, "postDate": "2022-02-16T16:53:59.253Z", "content": "Thank you for sharing @nbroad !
", "rawMarkdown": "Thank you for sharing @nbroad !" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:21:03Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7251286, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/nicholas-broad-17th-place-solution-bigbird" }