# 15th solution with codes

Congrats to all winners, and thanks to my teammates
! It's a long journey to finish this competition and get a silver medal, it's the first time for me to work with nlp projects.
Our solution could be summarized as:
Post processing, our idea is to group values to discrete values existing in training set. We chose columns to apply post processing by comparing oof.  Post processing could be found in
[my inference kernel](https://www.kaggle.com/jionie/models-with-optimization-v5/)
Ensemble, our final submission consists of 8 base models as you can see in the inference kernel, the best model is two part roberta base model which got 0.412 oof without post processing and 0.439 lb with post processing.
Modified pooler layer and multiple hidden_states , we used multiple hidden_states for the input of pooler layer inspired from
[this great post](https://www.kaggle.com/c/google-quest-challenge/discussion/123770)
we followed source codes in transformers for pooler class. This could also be found in my inference kernel. We chose hidden_state_layers based on validation results.
Differential learning rate, we used differential learning rate inspired by
[this great kernel](https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers)
. With differential learning rate, our models got higher oof and converged faster.
Two part model,  inspired from discussions, we trained models in two style. One is using title + question_body + answer as input for question labels and answer labels; one is use title + question_body for question labels and using title + answer for answer labels. The second style gives higher oof and lb, I think it's because more information for one problem. (Also using max_len 768 for xlnet gets higher oof is the same idea.
Data augmentation, we used nlpaug for some basic data augmentation, also we used google translate to back translate some content from other languages. I decided to do this at very early stage, so I'm sorry I can't tell how much it could boost. It won't be much I think.
Things don't work for us:
T5
Extra tokens
Large models, large models gave similar oof comparing with base models and it's hard to inference, always slow and run out of resources.
Our best single model:
[https://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization](https://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization)
Our codes:
[https://github.com/jionie/Google-Quest-Answer](https://github.com/jionie/Google-Quest-Answer)