# 22nd place solution(NLP beginner)

Thanks kaggle for holding a great competition!
I'm novice at NLP, so I learned a lot from great kernels about preprocessing, how to build Bert model, etc...
My code is based on
[this kernel](https://www.kaggle.com/idv2005/pytorch-bert-baseline)
by @idv2005  and
[this kernel](https://www.kaggle.com/adityaecdrid/pytorch-bert-end-to-end-with-cv)
by @adityaecdrid ! Thank you very much!
Here is the summary of my work.(Sorry for poor English.)
Solution
bert-base-cased, bert-base-uncased, xlnet-base-uncased and Twin models(XLNet, Bert).
Twin model idea is from
[https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer](https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer)
. thanks a lot!. Twin models have Bert for Question and Bert for Answer and treat sequence output with 3 custom layers.
Average pooling for QBert/ABert => concatenate(shape=768*2) => dense(30)
Average pooling for QBert => dense(22), Average pooling for ABert => dense(8), concatenate QBert dense output and ABert dense output.
concatenate QBert/ABert sequence output(shape=512, 768*2) => Transformers => average pooling => dense(30)
Final output: a * 0.25 + b * 0.25 + c * 0.5
postprocess (thank you for
[this discussion](https://www.kaggle.com/c/google-quest-challenge/discussion/118724)
)
```
best_bin_dict = {'question_type_spelling': 4, 'question_type_instructions': 8, 'question_type_entity': 9, 'question_type_definition': 8, 'question_type_consequence': 4, 'question_type_compare': 8, 'question_type_choice': 8, 'question_opinion_seeking': 50, 'question_not_really_a_question': 4, 'question_multi_intent': 9, 'question_interestingness_self': 9, 'question_interestingness_others': 100, 'question_has_commonly_accepted_answer': 5, 'question_fact_seeking': 10, 'question_expect_short_answer': 50, 'question_conversational': 8, 'question_body_critical': 100, 'question_asker_intent_understanding': 100, 'answer_well_written': 9, 'answer_type_procedure': 50, 'answer_type_instructions': 9, 'answer_relevance': 100, 'answer_level_of_information': 50}
df_sub = pd.read_csv("../input/google-quest-challenge/sample_submission.csv")
pred_final = np.array(preds).sum(axis=0)
for i, col in enumerate(df_sub.columns[1:]):
    df_sub[col] = pred_final[:, i]
not_type_spelling_idx = df_test.query("host not in ['ell.stackexchange.com', 'english.stackexchange.com']").index
for col in df_sub.columns[1:]:
    if col in best_bin_dict:
        n_bins = best_bin_dict[col]
        binned = pd.cut(df_sub[col].values, n_bins, retbins=True, labels=np.arange(n_bins)/(n_bins-1))[0]
        if col == "question_type_spelling":
           binned[not_type_spelling_idx] = 0
        df_sub[col] = binned
df_sub.to_csv("submission.csv", index=False)
```
Not worked
DistilBert
bert-large-cased, bert-large-uncased, bert-large-uncased-whole-masking
Categories/hosts Embedding
L1loss, L2Loss, Focal Loss (Finally I use BCELoss... but I can't understand why BCELoss is best unless it is not symmetric in this case... refer=>
[https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy](https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy)
)
Subtask: predict categories.
Concat CLS token sequence and average pooling.
Others
I consumed about $200 for this competition.