# 47th place solution

Thank you for Google as a host and Kaggle team! I really learned many NLP technics especially BERT form this competition.
Key point of final model
Use first and last part of text if length exceed max_seq_length
Averaging 4 BERT base-uncased model
Postprocessing: Fit target distribution with train data (a part of target columns)
Detail is in below.
Concatenate pool_output & sequence_output from bert_layer for GlobalAveragePooling1D
10 fold with MultilabelStratifiedKFold (thanks
[@ratthachat](/ratthachat)
!)
Strong heart for solo participation
Didn’t work for me
Pre-training with stackoverflow data (150,000 sentences)
Multi sample dropout
The other models
Roberta
Albert
XLNet
Concatenate question only output & answer only model
Concatenate category MLP with BERT model
LSTM head instead of Dense with BERT model
Freeze half of BertLayer for reducing model complexity
Skip half of BertLayer for reducing model complexity
USE + MLP
LSTM model with gensim embedding
custom loss
BCE & MSE
focal loss
Word count feature
Concat title and question_body as a one block (removing ["SEP"] between them)
Up-sampling for imbalance target column
My postprocessing class
```
class OptimPreds(object):
    def
init
(self, df_train):
        self.score_range_dict = {}
        for i, c in enumerate(df_train.columns[11:]):
            cnt = df_train[c].value_counts(normalize=True).sort_index()
            self.score_range_dict[i] = [cnt.index.values.tolist(), cnt.values.tolist()]
    def predict(self, preds, i):
        return pd.cut(rank_average(preds), [-np.inf] + np.cumsum(self.score_range_dict[i][1])[:-1].tolist() + [np.inf], labels = self.score_range_dict[i][0])
def optim_predict(pred, do_round=True, target=[ 2,  5,  7,  9, 11, 12, 13, 14, 15, 16, 19]):
    for i in range(pred.shape[1]):
        if i in target:
            pred[:,i] = optim.predict(pred[:,i], i)
    return pred
```