# 45th Place Simple Solution - 4 model average 

Thanks everyone for posting the elegant solutions . Thanks to the organizers and my team mates 
@iru538 @tarandro @mobassir and @shahules
I had joined the competition early and then left it for PKU .After  failing to secure a medal spot in PKU  , I joined back the competition and teamed up with Shahul and Mobassir .  Both of them had by that time already trained around 13+  vanilla (public Kernel) TF bert + xlnet models and doing prediction at the range of .385 to .387 , but we were not able to creack .400 mark . Then I applied the below post processing (collected from one of the public Kernel )
```
test_preds = final_predictions
y_train = df_train[output_categories].values
for column_ind in range(30):
    curr_column = y_train[:, column_ind]
    values = np.unique(curr_column)
    map_quantiles = []
    for val in values:
        occurrence = np.mean(curr_column == val)
        cummulative = sum(el['occurrence'] for el in map_quantiles)
        map_quantiles.append({'value': val, 'occurrence': occurrence, 'cummulative': cummulative})
for quant in map_quantiles:
    pred_col = test_preds[:, column_ind]
    q1, q2 = np.quantile(pred_col, quant['cummulative']), np.quantile(pred_col, min(quant['cummulative'] + quant['occurrence'], 1))
    pred_col[(pred_col >= q1) & (pred_col <= q2)] = quant['value']
    test_preds[:, column_ind] = pred_col
```
We crossed the .401 first time and was able to extend it further with XLNET and Roberta etc till .417
In parallel Toru joined our team and had a great single pytorch model with .409 score without post processing . 
Since I was more comfortable with pytorch , I used that model as base and started training the GPT2 and other pytroch BERT models . This took us to .410 based on pytorch models .
After Alexis joined he pointed us to a post processing trick used in Data Sceience Bowl 2018 1st place solution ."Data Science is Important- Bert says" . Here is our OptimizedRounder
```
from functools import partial
import scipy as sp
class OptimizedRounder(object):
    def
init
(self,correlation):
        self.correlation = correlation
        self.coef_ = 0
        self.score = 0
def _kappa_loss(self, coef, X, y):
    a= X.copy()
    b=y.copy()
    X_p = pd.cut(a, [-np.inf] + list(np.sort(coef)) + [np.inf], labels = [0,1,2])

    a[X_p == 0] = 0
    a[X_p == 2] = 1 

    #print("validation score = {}".format(spearmanr(a, b).correlation))
    if spearmanr(a, b).correlation < self.correlation:
        self.score = spearmanr(a, b).correlation
        return - spearmanr(a, b).correlation + (self.correlation - spearmanr(a, b).correlation + 1)**10
    else:
        self.score = spearmanr(a, b).correlation
        return - spearmanr(a, b).correlation

def fit(self, X, y,coef_ini):
    loss_partial = partial(self._kappa_loss, X=X, y=y)
    initial_coef = coef_ini
    self.coef_ = sp.optimize.minimize(loss_partial, initial_coef, method='nelder-mead')

def coefficients(self):
    return self.coef_['x']
def score_fc(self):
    return self.score
```
Once Alexis was onboarded he started applying post processing to our pytorch models and it gave a lot of boost . So we scrapped the TF models (As it was taking more GPU as well) .
Our final selected solution consists of 
Ensemble of : 
1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture) + Statistical Features (Boost the score a lot).
2. Sequence model with Statistical Features 
2. Distillbert Embedding Sequence Model 
3. Distillbert Embedding LSTM model
We got a slightly better Private Score using GPT2 in place of Sequential Model , but we didnt select that as final score . Our highest scoring Kernel has : 
1.BERT_Base 5 Folds (Question Model and Answer Model separate)(Tried three different architecture)
2. GPT2 Base (10 Folds Question and Answer Separate Model)
3.  Simple LSTM model 
4.Sequence Model with USE features
Here is a Kernel with the Inference which will give some idea of the architectures :
[https://www.kaggle.com/phoenix9032/threshold-base-4toru?scriptVersionId=28450007](https://www.kaggle.com/phoenix9032/threshold-base-4toru?scriptVersionId=28450007)
We have not used any trick in training time .