# 17th place solution - BigBird šŸ„ First of all, a huge thank you to the competition hosts and Kaggle. I spent an absurd amount of time on this competition, and Iā€™ve definitely learned a lot about QA, multilingual models, and Kaggle in general. This was basically the first time I put myself out into the public section of Kaggle, and Iā€™m so happy for the following achievements since the start of this competition: 3x Kaggle Expert (Datasets, Notebooks, and Discussions), soon to be 4x šŸ˜Š 2x Gold notebooks 20+ gold/silver comments Going from 600+ on the LB a few days ago to ending at 17th today (the turtle after my username was initially set to represent my progress. Slow and steady!) A new job šŸ¤— My solution XLM-R 6 folds, 384 max length, 128 stride, private/public LB 0.747/0.77, CV avg 0.718 MuRIL Large 6 folds, 384 max length, 128 stride, private/public LB 0.739/0.777, CV avg (forgot) RemBERT 5 folds, 384 max length, 128 stride, private/public LB 0.750/0.786, CV avg 0.748 MuRIL Large BigBird 6 folds, 1024 max length, 512 stride, private/public LB 0.756/0.796, CV avg 0.732, The last model is probably the only novel thing I did, and also the reason I placed this high. I noticed that the contexts for these passages were pretty long, and from a bit of research I saw that BigBird and LongFormer were the two architectures designed for long contexts without having the compute time blow up. I also noticed that the BigBird authors indicated that they did a ā€œwarm startā€ from a BERT checkpoint when training their model that reduced the number of training steps, so I figured Iā€™d do the same from XLM-R or MuRIL ( [see page 7 under section called Pretraining and MLM](assets/9e484a1e886430c7480f2d14.pdf) ). Since MuRIL is basically identical to BERT, all of the weights (except one module) loaded in to a BigBird model without trouble. The thing that caught me up for a bit was the fact that the position embeddings (which are learned) were going from (512, 768) to (4096, 768) (max_seq_length, hidden_size). I followed the Longformer authors and ended up tiling the original position embeddings 8 times to make it fit ([see page 6 under section Position Embeddings](Position Embeddings)). I trained for 150k steps on TPU v3-8 using Flax with a batch size of 1 using Masked Language Modeling as the training task. I used the Tamil, Hindi, Telugu, Urdu, and English sections of mc4 and stuck it straight into the model without cleaning. Doing whole-word masking and data cleaning would probably help significantly, but I didnā€™t have the time. I did the same for the large size, but I could only fit a 1024 sequence length and I only trained for 90k steps at batch size 1. The base model got up to 75% accuracy and the large model got 71% on MLM in the eval set. From there, my MuRIL Large, XLM-R, and Muril Large BigBird models were trained first on SQuAD2 and then on XQuAD, MLQA, TyDiQA (telugu), and chaii datasets, validating only on a stratified chaii fold. Since I still had compute time, I did all of this easily and quickly on TPU v3-8 using Flax. RemBERT was a pain to train on Kaggle GPUs, so I used Tensorflow and Kaggle TPUs. This made it easy, and I convinced myself that having some models in Tensorflow provided model diversity. I trained it on SQuAD2 and then TyDiQA, MLQA, XQuAD. [Here is the final notebook that got my highest score](https://www.kaggle.com/nbroad/test-1k-6f) . It is messy and wasn't meant to be my final submission šŸ˜… Ensembling I didn't actually get a submission out with an ensemble. My highest score without ensembling different models was 6 folds of MuRIL BigBird Large. I would never be able to get 23 folds to finish in time, but I was planning on splitting the passages based on length at around 10k characters. Under 10k go to XLMR, MuRIL Large, RemBERT, and MuRIL BigBird Large. Above 10k would just be MuRIL BigBird Large. Some of those contexts are so long especially when broken up with 384/128. For ensembling, I tried voting but ran out of time before getting something to work. I ultimately converted the token logits to character logits by averaging the overlap between multiple sections and concatenating them together. Once they are character logits, they can be averaged across all models without worrying about lining up models with different tokenizers. I got this working with a few hours left, and I made one small mistake in my submission, which made the notebooks exceed the submission time. It did seem like a promising way of combining models. [Here is the notebook I didn't finish in time that has character-level averaging.](https://www.kaggle.com/nbroad/final-sub-char-avg) It's pretty messy because I was cramming stuff in earlier today to try to get it done. Much thanks to TRC for the compute time on v3-8 TPUs. All of my models are now available on my [Hugging Face profile](https://huggingface.co/nbroad) . What didnā€™t work Attention heads on the end Conv. layers on the end Concatenating start logits to end logits Translating at test time and doing cross-lingual QA (English questions with Hindi/Tamil contexts) Monolingual models ā€œQA Pretrainingā€ on tons of other languages before finetuning: Arabic, Persian, Bengali, German, Thai, Korean. Only English, German, Hindi, Telugu, and Tamil helped. I think they were the highest quality, and most of the others were poor translations. [Splinter](https://arxiv.org/abs/2101.00438) / [FewShotQA](assets/b89cd53df745f098eda58a69.pdf) I thought this would be perfect for this scenario because it excels at small datasets, but I could never get the model to converge [They even had starter code](https://github.com/oriram/splinter) but it was in Tensorflow 1 and I stubbornly tried to turn it into PyTorch. NaturalQuestions/TriviaQA NQ is just a pain to work with because itā€™s absurdly large and the formatting is weird. I donā€™t think I ever got the training data to actually look right which is why I gave up on using it. Ran out of time before trying TriviaQA. Training a question-generating model to label more data mt5 is huge and hard to train. I could only get a base model trained and it didnā€™t seem that good. [This article indicates](assets/19c21d6997b0d59d404d824d.pdf) that using millions of pseudolabels works, but I didnā€™t think Iā€™d have the capacity to get this going.