# 128th Solution

Very interesting competition and congrats to the winners.
I was ranked 1154th on the public leaderboard and 128th on the private leaderboard. My final submission is an ensemble of 4 models, and I did not use any external data.
Generally, I used a comparison approach like many others, since I found the popular regression approach is very unstable. The first approach is a representation-based one. After encoding a batch of text, I construct pairs for comparison with some reshape tricks. Though my training batch size is 16, the actual batch size is 16*16. Since texts are encoded by BERT separately, I used an ESIM-like inference module to enhance the interaction of two semantic embedding vectors. Both regression(the target delta) and binary classification tasks were used to train models, and this provides good model diversity. You can find the code in this notebook
[https://www.kaggle.com/wuyhbb/inference-large?scriptVersionId=68357748](https://www.kaggle.com/wuyhbb/inference-large?scriptVersionId=68357748)
I also trained two interaction-based models. In these models, paired texts are encoded together. Theoretically, this approach is better than the above one, because it can inference the two sentences better. My CV results supported this theory. However, this approach has a significant drawback: it is very slow. When predicting the test set, I only need to encode the anchor excerpts, which are excerpts I sampled from the training set,  once in the above approach. In this approach, I need to encode every pair of excerpts. Code of this approach can be found in this notebook (OneBertxxx)
[https://www.kaggle.com/wuyhbb/inference-large-ensemble2?scriptVersionId=69518962](https://www.kaggle.com/wuyhbb/inference-large-ensemble2?scriptVersionId=69518962)
My CV results:
Representation-based binary classification with Roberta-Large: 0.47703
Representation-based regression with Roberta-Large: 0.47878
Interaction-based binary classification with Roberta-Large: 0.47296
Interaction-based regression with Roberta Large: 0.47432
When training the models, I used the standard error for augmentation. It can improve CV scores when the training task is binary classification.
Special Thanks to Z by HP & NVIDIA for sponsoring me a Z4 Workstation with dual RTX6000 GPU and a ZBook with RTX5000 GPU!
Training large transformer models with 24GB VRAM is awesome!