# 4th Place Solution

[UPDATE] source code is now available at
[https://github.com/Zacchaeus14/chaii-4th-place-solution](https://github.com/Zacchaeus14/chaii-4th-place-solution)
Thank Kaggle and Google Research for hosting this fantastic competition. The CV and LB are really hard and I cannot believe we moved into the money zone. Thank my teammate
[@zzy](https://www.kaggle.com/zzy)
. Here I share our solution. Please feel free to leave comments!
TLDR:
Ensembling models with different tokenizers
Finetuning on squad
Post-processing
Models:
There are 5 models involved in the final submissions:
Zacchaeus:
model 1. Xlm-Roberta-large 5 folds
model 2. Rembert 5 folds
model 3. Xlm-Roberta-large 3 seeds
model 4. Muril-large 4 seeds
model 4'. Muril-large 5 folds
zzy:
model 5. Infoxlm 5 folds
Our submission A is an ensemble of 125 (public 0.817 private 0.767) and B 2345 (public 0.812 private 0.777). We find the best private submission is 24'5 (public 0.812 private 0.784).
Training:
I try pre-training on hi/ta wiki with vanilla MLM, hoping these data will cover the hidden test set. Doesn't work. I find that Infoxlm / Rembert / XLM-Roberta benefits from finetuning on the squad 2.0 dataset for 1 -3 epochs. Then, I translate the dataset into Tamil with Google API, but the translated dataset does not bring significant improvements. After that, I train the models on chaii + mlqa + xquad dataset with the latter two downsampled for 3 epochs. The validation Jaccard score was quite unstable, so I adapted "non-zero Jaccard percentage" as an early stopping criterion".
Ensembling:
Xlm-Roberta and Infoxlm share the same tokenizer, while Rembert and Muril have their own. I ensemble models with different tokenizers by mapping the token-level logits back to characters and then average with equal weights. After mapping and blending, I replace consecutive same numbers with -1e5 and finally do the beam search.
Post-processing:
As the training data is inconsistent. I make various post-processing heuristics to mitigate the bias. Some major methods:
Recursively strip left and right punctuations.
Add back some punctuations if necessary (inspired by
[@nbroad](https://www.kaggle.com/nbroad)
's notebook).
Extract years from mm/yyyy/dd answers using regular expression if the question asks for a year specifically.
As a non-native speaker, going through all pieces of bad cases and copying/pasting in Google Translate can take forever.
Other things interesting but not working:
Second-stage reranking model. Thanks
[@zzy](https://www.kaggle.com/zzy)
for attempting on this.
SWA.
KL loss (mentioned in the VAULT paper).
torchscript and ONNX
Acknowledgement
Thank NYU Shanghai faculty for granting me the GPU resources.
Here is the inference kernel:
[https://www.kaggle.com/zacchaeus/chaii-infer-blend-postpro-4models](https://www.kaggle.com/zacchaeus/chaii-infer-blend-postpro-4models)
. I will publish my github repository after cleaning up my code.