# 19th solution

First of all, thank you for holding this great competition google and kaggle team. 
I think the balance of need for latest NLP knowledge and ideas applied to this specific task is the exciting point of this competition! I could not get the gold medal, but I’ve enjoyed and learned a lot.
I want to share my insights which I’ve achieved from more than 150 experiments. My
[codes](https://github.com/guchio3/kaggle-google-quest)
and
[kernels](https://www.kaggle.com/guchio3/refactored-bert2-robert1-xlnet1-blend-pseudo1)
are publicly available now.
preprocess
add special token to separate title and question.
ex. [CLS] title [NEW_SEP] question [SEP] answer [SEP]
add category info to the head of the title.
ex. [CLS] CAT-TECHNOLOGY title [NEW_SEP] question [SEP] answer [SEP]
This method should be better than the case just concatenating the category to the last classification layer because the hugging-face model can use the category info while it retrieves the texts.
trim head and tail part of the texts
[like this](https://github.com/guchio3/kaggle-google-quest/blob/decffc69d5657f5114970eb2ea226df8ec8cfaf6/scripts/refactor/datasets.py#L137-L157)
.
ex. I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood.

-> I have a dream that + at the table of brotherhood.
modeling (used pytorch hugging-face library)
GKF using question body
AVG pool final layer seq output and use it for the input 1 linear layer
use 2 separate models for question type label and answer type label
(title-max-len, question-max-len, answer-max-len) : (30, 239*2, 0) and (30, 0, 239*2) for each (if each part is less than the max-len, the other parts use the rest len).
Ensemble of four models (1 : 0.5 : 0.5 : 1)
bert-base-uncased (best snapshot)
robert-base (best snapshot, second-best snapshot)
xlnet-base-cased (best smapshot)
batch size 8, 6 epochs (freezing is crucial)
freeze only hugging-face model for the first 1 epoch, and unfreeze it the rest 5 epochs
BCEWithLogitsLoss
Adam w/ cosine scheduler (3e-5 -> 1e-5)
1 cycle pseudo labeling
concat soft target, hard target, and half hard target. Half hard target is the target where the threshold is applied to the col only if the application improve the oof score
postprocess
Search and apply thresholds
Get optimized thresholds using scipy.optimize.minimize for oofs. The initial value is calculated from the percentile point of the oofs. (
[ref](https://github.com/guchio3/kaggle-google-quest/blob/master/scripts/get_optR3.py)
)
Each thresholds is applied to a column only if the oof score improves by using it.
Each thresholds is applied to a column only if the cardinality of a column is more than one for public/private part of the test set.
Set 0 or 0.5 to question-type-spelling based on the host (kernel based)
What did not work for me
Larger batch size
Longer sequence size
Longer position embedding matrix than 512
Cyclic position ids (0 - 29 for title, 0 - 511 for quesiton, 0 - 511 for answer)
Shorter sequence size
Using meta features except for category
host, word counts, tokenized word counts, etc …
Using last N hidden outputs of the hugging-face models
Augmentations
Manual text preprocessings
Changing loss functions
MSE, Pair type loss, Focal loss
Changing optimizers and schedulers
AdamW, RMSProp, SGD, Step type schedulers
Using SWA
Expert models for difficult or imbalanced classes
Oversampling for the minor values of imbalanced classes
Using the other type of models for ensemble
RNN based models, GPT2, XLM
P.S.
Like some people say, I used so long time to reproduce the result of
[great TF kernel](https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer)
by pytorch. At first, I concluded that the situation is caused by the pytorch’s nature
[like this](https://discuss.pytorch.org/t/suboptimal-convergence-when-compared-with-tensorflow-model/5099/18)
. However, I noticed that the kernel use snapshot ensemble in the callback function, and this should cause the difference of the performance. I think pytorch users are not familiar with callback functions, so we should overlook it.