# 24th place - mixing all I can - BERT+XLNET+GPT2+LGB

My first share of anything in public. Got to Expert tier, so guess got to share back, even if the result is not in the very top!
The key ideas which made impact:
Pre-processing:
- Feature engineering - number of punctuation, nouns and things, number of common words between question and answer etc. Those are used in separate model (LGB).
- Two separate sets of tokenized inputs. First set consisting entirely of question title and question body and used to predict categories relying on question only (first 21 outputs). Second set consisting of question title, question body and answer, to use for remaining 9 categories related to how answer relates to question and answer only features.
Modelling:
- Train BERT, XLNET, GPT2 all in base configuration - 1 model of each kind for each of 2 sets of outputs (question only and question + answer). Use one-cycle training, standard Adam, early stopping on decrease in validation metric (spearmanr). Train for 3 folds (but using 10-fold groupkfold so that each fold has larger train set)
- Train LGB using just the engineered features, one LGB model per output variable
Post-processing:
- Mix all the models using softmaxed validation scores as weights. Using beta of 18.5 as this maximised validation score (optimised in separate kernel)
- Iterate through different rounding policies for each output variable, use best one for each. Rounding policies considered - normal with 1-5 decimal places, ceil/floor by multiplying prediction by 10^X (where X is in range 1-5) then using floor/ceil and then dividing by 10^X.
PS: the initial kernel is forked from
[Bert-base TF2.0 (now Huggingface transformer)](https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer)
, big thanks @akensert for his share! Would not have been able to accomplish such results without this as I joined relatively late (19 days before the end).
PSS: The actual code is relatively dirty and relies on saved model weights. Still in case anyone wants to have look, it is
[here](https://www.kaggle.com/bookerd/2nd-gen-predict-only-all-available-models)
(v19 is the final submission)