# 162nd solution using only BERT

Congratulations to the winners , this was our first NLP competition we tried below things:
At first we build a simple model using
[https://www.kaggle.com/mobassir/jigsaw-google-q-a-eda](https://www.kaggle.com/mobassir/jigsaw-google-q-a-eda)
We tried to tune the hyper-parameter but that didn't had much effect.
Since we were concatenating Title, Body and Answer using 512 chars which cant understand the context , we tried to build model using word embeddings of GLove and Fasttext but didn't had much accuracy.
Then we tried to create separate embedding for Title + Body and answer with additional metadata which worked.
We finally created Title + Body and answer with additional metadata and BERT with concatenating title , body and answer with 10 folds.
Ensemble of all the outputs gave us 0.4003 on LB.
What didn't work for us:
We tried to trim Title , body and answer - Many has done that to understand the context i feel we did something wrong here.
We tried to create text summerisation of Body and Answer but the Rho remained the same and we didn't explore much of this idea.
We tried to replace last layer as regression and loss function as MSE , we could have explore this idea a bit more as many top scorer has done some kind of post processing.
*
Thing we should have done *
Pre-Processing into the data like adding category column combining with title (Don't know how i missed this one)
Post processing of the data to adjust more to "Human level scores"
Build more models like XLNet , Ro-BERT
We tried to use LAMB instead of ADAM but due to an error we couldn't progress. (TF 2.x :) )
We missed the medal by 5 ranks , hopefully we can do better next time. And not to forget thank you @vinaydoshi for continuous encouraging on working on this problem.
If anyone is interested here is our notebook.
10 fold BERT concatenating title , body and answer :
[https://www.kaggle.com/buntyshah/google-quest-q-a-labeling-8-fold](https://www.kaggle.com/buntyshah/google-quest-q-a-labeling-8-fold)
Hugging face BERT with Title + Body and Answer with additional meta-data :
[https://www.kaggle.com/vinaydoshi/tfbert-ensemble-preprocess-v1?scriptVersionId=28236333](https://www.kaggle.com/vinaydoshi/tfbert-ensemble-preprocess-v1?scriptVersionId=28236333)
BERT as regression :
[https://www.kaggle.com/buntyshah/google-quest-q-a-labeling-bert-regression?scriptVersionId=26800228](https://www.kaggle.com/buntyshah/google-quest-q-a-labeling-bert-regression?scriptVersionId=26800228)