# 16th Place Solution 

First I would like to thank my teammates
[@abhishek](/abhishek)
,
[@atsunorifujita](/atsunorifujita)
,
[@pvduy23](/pvduy23)
for a very good team work.
Solution Highlights
Preprocessing
We didn't really do any preprocessing to the text, we tried some cleaning which were aimed especially to make the parts with code more readable (shorter) but it didn't make any difference.
Loss function
I'm surprised I didn't see more referance to this in the winning posts, but here we did a little trick. Instead of using the target columns as-is we first ranked and normalized them to [0,1] and then used them as targets, something like
> target = scipy.stats.rankdata(target,'average')
target = (target-target.min())/ (target.max()-target.min())
(I saw CV improvement of about 0.05 - 0.01)
As the loss itself I used MSE and some other teammates used MSE+BCE
Models
The models we used where RoBERTa-Large, XLNet, BERT-base-uncased. 
We also tried some other models, like BERT-large, RoBERTa-base, XLMRoBERTa and others.
(Not choosing  XLMRoBERTa was a mistake - will be discussed later)
We also tried to pre-train and fine-tune bert-base on stackexchange  corpus, but it didn't seem to improve CV.
To handle long text, I used the full
question-title
+ equal parts from
question-body
and
answer
, truncated by taking the beginning and the end.
We 5 folds for every model
Augmentation
I Tried several augmentations:
* replacing random tokens with [PAD] or with another token - 20% [PAD} seems to give nice results
* Translating to German and Back - didn't work (Russian also didn't work)
* Truncating random parts when the text is too long - didn't do any difference
Post - Processing
Postprocessing was very important in this competition due to the special characteristics of spearman correlation
Spearman
We ended up with clipping some of the output columns. The columns we clipped where columns were most of the targets values were 0 or where most of them where 1. The clipping threshold value was chosen to preserve the same non zero values in test as there are in train. This method improved CV and LB by ~0.03-0.04
Question-Type-Spelling
As the number of unique non-zero examples for this class is very low, we understood no ML model could really do a reliable work here. So we tried some heuristics - we ended -up with setting all question where the host was 'english.stackexchange.com' or  'ell.stackexchange.com' to 1. 
We also tried a more ambitions trick, by selecting different key words, but this didn't work on Private LB.
Stacking
We tried different types of  stacking and averaging. At the end it looks like simple averaging was as good as any other method.
Where did we fail?
We didn't use enough models - we used 5 folds for every model which limit the amount of different models we could run in 2h, for this reason we (I !) neglected some good models like XLMRoBERTa, which gave good results, but didn't seem better then other models. The best practice if probably more diversified models, and dropping some of the folds from each model.
CV - It might be because of the spearman metric or because the training set was too small for 30 classes, or we did something wrong but the CV wasn't reliable enough for us (compared to itself, Public LB and Private LB), which meant we didn't do the right choices in selecting models and parameters
My Last observation
At some points in this competition it seems like the competition is not about NLP but about tricks.
This is a mistake. This competition is about NLP
and eventually tricks were only a small part of the full picture, and the winning teams were the teams with the best models
As always I want to thank Kaggle and the competition team for a well organized competition.