# 9th public and private solution: 30 models, 1 for each column?

First of all, I'd thank the organizers and my teammate,
[@murphy89](/murphy89)
for this competition, and congrats to all the teams participating. I know the metric is not ideal, but I guess we all gained something from the competition.
Secondly, well, that is a click bait title, but we did something sort of the same when coming to blending parts. Anyway here is the overview of our solution:
Overview
There is no special pre-processing of text, just basic cleaning like replacing & with & and < with < and some more broken signs.
Models
We all split the models into 2 parts: models for predicting first 21 question columns and for last 9 answer columns.
My models
Question part
Question models take only text from question title and question body. If the text length is long, use head and tail method.
Using special tokens like site pages, categories...
Freezing last 4 layers, concatenating them and feeding into 1-layer LSTM before taking the average pool and max pool. Then 1-2 more FC layers at the end.
I used bert base, bert large and xlnet. Then from each model I vary a bit like different layers to freeze or instead of concatenating layers, just take the sum.
Sum of BCELogitsLoss on each column work better for me. I guess it's similar to weighted loss.
Answer part
Answer models take text from all 3 columns and also use head-tail method.
Not using special tokens, as my CV is worse with special tokens in answer part.
Freezing last 4 layers, and doing almost the same as question part, except that this time it's most ly 2-layer BiLSTM.
Mostly bert base and bert large of simply varying structures from the main model.
Normal BCELogitsLoss.
Blending
I believe this is the key part in boosting our score. We discovered the right blending strategy only on the last day, when we saved the oof predictions and cross validated offline. Although our single models have different CV score 0.411-0.426, they seemed better when blending together at certain weights. Later I found out that each model better in predicting some columns, so I use heuristic search to search for best CV score of each column. For those columns that depending heavily on post-processing, the search is on the final values after post-processing. This is why there is such title: 30 models, since I believe tuning some specific models for some columns would be better with post-processing part. In fact, some models with initial lower CV score seem to have higher weightage in this part. The best CV score after blending is 0.435, and after post-processing is 0.478.
The only regret is that when we were doing this, it was only 4 hours till the deadline, so some of our columns are not maximized in the final solution. In fact, when I tried again after the deadline, our CV could reach 0.482 with post-processing.
Post-processing
Our post-processing is not different from others'; on the contrary, I think it's even less effective since it's the simple rounding of predictions of some columns to the nearest values with certain thresholds.
What did not work for me
Translation augmentation. I had translated back and forth in French and Italian, but adding them did not help.
SWA.
Multi-dropout for me, but it worked for my teammate's, Morphy, architecture.
Other models: xlnet-large, albert... The scores are worse in my case.