# BERT -6 layer solution [LB 0.23101]

My solution was based on
[Matei's kernel](https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline)
and
[End-to-end paper](https://www.aclweb.org/anthology/D17-1018)
. 
My model is a simple BERT(no fine-tuning) + FFNN model, and the FFNN structure is from this End to End paper, out of the box.
Feature:
Word vectors of A, B and Pronoun (from BERT large layer -6);
Distance between P and A/B;
Argumentation:
Swapping A and B, I got 0.006 improvement.
Hyper parameter:
Adam optimizer learning rate=0.0004;
FFNN hidden dimensions = 40;
drop_rate=0.65;
batchsize = 16;
10 folds cross-validation;
Thanks for your reading.