# Two BERTs are better than one (2nd place solution)

Thanks to the sponsors and kaggle for hosting such an interesting and challenging competition! Our work is an amazing team effort and sprint over a very short time period of two weeks during which we made progress each day.
Why two weeks?  Because some of us just finished the TF2 QA competition and decided to reuse whatever NLP knowledge that was acquired there. Thanks to all teammates: @christofhenkel @cpmpml @philippsinger @dott1718 @maxjeblick.
Please
also give them upvotes, we made a lottery roll of who will post the solution and everyone contributed equally to it.
Our only regret is that the final score depends too heavily on the question_type_spelling column that has only 11 non zero target values in train data.  Late submission experiments show we could easily improve, or degrade, our final rank by little tweaks on how we handle this column.
Brief Summary
Like most teams, we used pretrained models from huggingface implemented in pytorch. Our final blend consists of 5 transformer models, that have several differences to push diversity. However, they all have in common to use 2 separate inputs of up to 512 tokens length representing questions respectively answers. These 2 inputs go either through the same transformer or through 2 separate transformers. We changed the targets to an ordinal representation (setting >target to 1) resulting in 170 target columns with each original column having n-unique target columns. Predictions were then generated by calculating the expected value.
Most models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).
Dual roberta-base:
- double roberta-base
- added question title as input to answer-transformer
- individual head for each target
- mean of last layer output
Dual roberta-base:
- double roberta-base
- added question title as input to answer-transformer
- individual head for each target
- mean of last layer output
Siamese roberta-base with softmax layer weight:
- double roberta-base
- added question title as input to answer-transformer
- individual head for each target
- mean of last layer output
Dual roberta-large:
- only use 256 tokens for question / answer
- double roberta-large
- only fits in memory with batchsize 1 and fp16
Dual xlnet:
- same as dual roberta-base but with xlnet-base backbone
- We used a simple average of probabiity predictions to ensemble our models.
Longer Summary
Binary Encoded Targets
Given the metric is rank-based, and given targets are not binary, it seemed important to be able to predict values that are neither 0 or 1 correctly.  We tried mse loss and other variants but results were not satisfactory.  We then decided to use binary cross-entropy with binary targets.  The first try was to use on hot encoding of targets, given that targets have a small number of distinct values.  This wasn’t satisfactory either, because this representation loses the ordering of values. 
We ended up using an encoding of the form
(t > v)
for all v values of target t, except last value. For instance, if a target t has unique values
[0, ⅓, ⅔, 1]
then we would get 3 binary targets:
t > 0, t> ⅓
, and
t > ⅔
.  Assuming we get perfect probability predictions
p(0), p(⅓ )
, and
p(⅔)
for the binary targets,  we can compute predictions for each value of the original target as:
t(0)= 1 - p(0)
t(⅓)= p(0) - p(⅓)
t(⅔)= p(⅓) - p(⅔)
t(1)= p(⅔) - 0
And the original target value as
t = 0 * t(0) +  ⅓ * t(⅓) + ⅔ * t(⅔) + 1*t(1)
We keep that computation with the actual predictions
p(0), p(⅓ )
, and
p(⅔)
for the binary targets.
When the values are evenly spaced, as in this example, then the formula simplifies into:
t = mean(p(0), p(⅓ ), p(⅔))
If we assume that what matters is only the order of values, then we can stick to this simplified form.
We used the simplified form throughout the competition as it was simpler to code, and faster.  In our final two selected subs we reverted to the exact computation because it was better on both cv and public LB, but this didn’t change private LB significantly (0.0004 difference).
Validation scheme
Similar to most teams, we use a 5-fold GroupKFold on question body for fitting our models and evaluation. After a bunch of experiments we saw though, that this is not sufficient mainly due to the following aspects:
- Test data is different to training data in the sense that it only has one question-answer pair sampled out of a group of questions in train data. There can be stark noise for labels of the same question, which is why this needs to be addressed robustly.
- There are a few columns with very rare events and also a lot of noise within those rare events. The prime example is spelling which has a huge impact on CV and LB, but can completely blind your view when trying to judge the overall strength of your model.
That is why we settled after some time on the following full validation scheme:
- Use 5-fold GroupKFold on question body
- For each validation fold, sample 100 times randomly a single question-answer pair out of multiple questions.
- Calculate the median score across these 100 samples and report.
- Ignore spelling column.
- Final CV is a mean of 5 folds.
This setup also allowed us to properly test any type of postprocessing in a realistic manner as described next.
Postprocessing
Even though we had really strong models, post-processing was very important to us and we spent significant time on it to find a robust way that we can trust. It is based on threshold clipping. 
For each target column, we separately attempt to find optimal thresholds from both sides (starting from lowest predictions and highest predictions) based on which we clip the data. So let’s assume our predictions look like
x=[0.01, 0.015, 0.02, 0.03, 0.04]
and our optimal thresholds are
coefs=[0.016, 0.029]
then we would clip the data with
np.clip(x, coefs[0], coefs[1])
leading to
x=[0.016, 0.016, 0.02, 0.029, 0.029]
effectively generating ties at the edges.
The CV setup from above gave us a perfect way to fully validate any of our approaches by simply generating the thresholds on training folds, and applying them on truncated samples of the validation folds. For final PP, we get the thresholds on full oof and apply them on the test set.
In a nutshell, our routine looks like the following:
- Sample 1000 times single question-answer pairs from multiple questions
- Generate thresholds that optimize the median score of all 1000 samples
We tested quite a few different strategies, but this one was the most robust one we found and we are quite happy with it. Only for spelling, it was still a bit shaky, which is why our two final subs just differ in how we handle spelling column, one uses the calculated thresholds, and one hard-sets the 6 highest predictions in private LB to 1 and rest to 0 which is based on experiments on samples.
Unfortunately, if we would have post-processed all columns except spelling and would have kept spelling as is, we would have reached 0.432 on private LB as apparently spelling is very differently distributed on private. No CV experiment would have let us make this decision though. It would have been so much better though to not include this column into the competition.
Architectures and training models
Similar to other recent competitions at first it was quite difficult to beat public kernels, which we normally see as a kind of baseline. All “normal” tricks that worked in past computer vision or other NLP competitions (e.g. concat of Max and Mean pooling) did not improve cv. Also using a sliding window approach to capture more text did not work.
We saw a first big improvement when using 2 transformers instead of one (+0.01), but still, any slightly more complex architectures lead to a worse result. That changed when we tried to freeze the transformers and only train the model head for 1 epoch, before fine-tuning. With this approach, we were able to try more fancy things and that is also how we finally came to the two main architectures. 
We further improved than this 2 step approach by using different learning rates for transformer and head, together with a warm up schedule, which enabled us to get rid of the freezing step in general. We illustrate the two main architectures in the following figure (For the sake of simplicity we show the architecture for the original 30 targets.  It was adapted to work with the binarized targets.).
![](assets/41906eab6f115ddd041212cc.png)
Dual Transformer:
The upper architecture shows the dual transformer model, a combo model where one transformer handles the question text, while a second transformer handles the answer text. the output of the last layer is then averaged over the 512 tokens and both resulting tensors are concatenated. The resulting representation is then fed into 30 little 2 fully connected layers which result in the target predictions. 30 little heads enable each target to have its own head and gain individuality.
Siamese Transformer with soft weighted layers:
It is inspired by the
Elmo paper
, where the final embedding representation is a weighted average of all LSTM layers. We use the output of every layer of a single transformer model in which we put 512 question and 512 answer tokens. For roberta-base that will give us twelve 512x768 tensors (for roberta large it would be 24 512x1024 tensors). We then average over the 512 tokens for each layer which results in twelve 768 representations. We then take a weighted sum of these 12 representations (where the weights for adding the representations are trainable!). This results in a 768 representation. The weighted average of all layer outputs enables to capture low level features in the final representation which was quite important for some answer related targets. Finally, we add a single prediction head for getting our targets.
Our final ensemble contains 5 models which belong to the one or the other architecture, and only differ in used pretrained backbone. 
We ended up with 
- 2x dual roberta-base
- dual roberta-large (2x 256 tokens)
- dual xnet-base
- siamese roberta-large with weighted averaged layers
as this combination had the best cv score while fitting in the 2h kernel runtime requirement.
We even trained one dual roberta-large model which on a V100 can only be trained when using a batch size of 1 and fp16. and was our best single model. Although this model could not fit into the ensemble due to runtime reasons, the promising cv results let us to a daring experiment: reducing the number of tokens used. There we saw something quite surprising. We could reduce the number of tokens to 256 for question as well as for answer without losing much quality.
Most models were trained with differential learning rate, where the transformer gets a lr of 3-e5 and the model head(s) a lr of 0.005 and used a cosine schedule for training 3 epochs including warmup over one epoch. Optimizer were either AdamW or RAdam with weight decay of 0.01 using an effective batchsize of 8 (gradient accumulation).
Blending
Our submission includes an equally weighted blend between all of above mentioned 5 models (leading to 25 test predictions with 5-fold). The blend is conducted on raw expected values calculated as described above. We experimented quite a bit with ranked blending as it seemed natural, but results were always worse.
Our final blend is both our best one on CV, public LB and private LB meaning that we had a robust CV setup in the end and our selection was solid.
Wrapping things up and putting into kernel
Training was conducted offline and inference in kernel. Our final runtime was close to two hours and we had to spend some efforts to squeeze the 5 models in. All our models are implemented in Pytorch using the amazing Huggingface library. Fitting was done either locally or on cloud providers.
Thanks for reading.