# 1st place solution: Metric learning and GPT

The code is available here:
[https://github.com/suicao/coleridge-gpt/](https://github.com/suicao/coleridge-gpt/)
My inference kernel is available here:
[https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765](https://www.kaggle.com/suicaokhoailang/submit-gpt-spacy?scriptVersionId=66488765)
.
Overview
I'll lay out a few notable features from our solution here for now, kudos to my teammate
[@dathudeptrai](https://www.kaggle.com/dathudeptrai)
for his immense effort in this competition, especially in the second half.
His solution is metric learning with usual MLM backbones, I'll leave it to him to give more details later. We could not find a way to make our ensemble to look stable so we ended up submitting each of ours separately, my final sub scored 0.565 while his was 0.576.
Mine is a text extraction model with CLM backbone and beamsearch, in this case GPT. I also tried to modify Roberta and Scibert with future masking, basically turning them into CLMs but so far GPT worked the best.
Why GPT
The reason why GPT worked and Roberta failed so hard is that Roberta is too clever for its own good. When you try to build a text extraction model with a single query of finding dataset names, Roberta doesn't need to care about the context and will try to find a substring that most resemble a dataset name, thus lead to overfitting.
In my GPT + beamsearch model, GPT will be forced to predict
whether the next token would be the start of dataset name
, given only the previous context, and then predict
wherether the next token would be the end of the mentio
n, given the starting point and the extracted content so far. To make it more robust, I used a few sources of dataset and replaced 95% of mentions in the original training set with these random labels. Funnily,
[https://www.kaggle.com/datasets](https://www.kaggle.com/datasets)
was the best source I could find for augmentation, although it has zero overlapping with the public (and possibly private) labels.
Modeling
The architecture is pretty much the same as what I used in the Tweet extraction competition. We reused XLNet’s decoder head for SQuAD, instead of predicting start and end indices independently, we first predict the start index, and use that information to predict the corresponding end.
![](https://i.imgur.com/hdVe7bV.png)
For training, we predicting the start index normally.
We take the hidden representation at the target index (ignoring the predicted) and concat it into the representations at every position. The new presentation is then fed to a MLP to predict the end index:
Inference is a bit more complicated:
We take top-k hidden states corresponding to top-k start indices with highest probabilities, this is normalized with sigmoid, not softmax like training.
Each hidden state is then concatenated into the representations at every position.
The new representation is fed to a MLP, similar to training. We then select top-k end indices for each selected hidden state, resulting in k*k top start-end pairs.
We then calculate the joint probablities of every start-end pairs and take any pairs with a score large than 0.8.
![](https://i.imgur.com/nyP25it.png)
Final results and some gotchas
My selected submission score 0.565 on the private LB, but my best one (that I know of) was 0.594. I only took predictions that appeared at least 4 times in the corpus, the rare ones were only selected if they contained words like 'Data', 'Survey'.. etc. I tried to add more predictions using scispacy's abbreviation detector for my final sub but this actually lowered my score, should have trusted my model in the first place.