# 3rd place solution

First of all, I'd like to say thank you to the organizers and all teams, and congrats to all the winners.
Here is a brief summary of my solution.
- Solution overview
- Model architectures
- Training strategy
- Post-processing
- Blending strategy
- What didn't work for me
Solution overview
text columns + category column
various text truncation
pre-truncate
post-truncate
head + tail tokens (
[](https://arxiv.org/abs/1905.05583)
[https://arxiv.org/abs/1905.05583](https://arxiv.org/abs/1905.05583)
)
assign longer max_len to answer
ensemble of following models:
LSTM + Universal Sentence Encoder
BERT base uncased * 2
BERT base cased
BERT large uncased * 2
BERT large cased * 2
ALBERT base
RoBERTa base
GPT2 base
XLNet base
Model architectures
Surprisingly, stacking 2 Linear layers
without
activation performed better than single Linear layer in some cases. (I found it by a mistake. :P)
Some of the following models have category embedding structure additionally.
LSTM model
You can check my LSTM code
[here](https://www.kaggle.com/sakami/google-quest-single-lstm?scriptVersionId=28487242)
.
![](assets/3a3b55c2838ca713c68d8e40.png)
BERT models
I used 2 types of model architectures.
![](assets/9bb70f45ab528b5193f4ff0b.png)
![](assets/5b2c5e525463141514bddfde.png)
ALBERT model
![](assets/772644f5a1d2b21ebb890fcc.png)
RoBERTa model
![](assets/a26438ce9b649ae275bfe9f5.png)
GPT2 model
![](assets/fde50777ba757c5e6de2b25c.png)
XLNet model
![](assets/ca45e0a664faca96d8aa75fd.png)
Training strategy
Min-Max target scaling
t_max = train_y.max(axis=0)[np.newaxis, :]
t_min = train_y.min(axis=0)[np.newaxis, :]
train_y = (train_y - t_min) / (t_max - t_min)
Large weights for minor positive or negative samples
zero_inflated = (train_y > 0).mean(axis=0) < 0.1
positive_weighted = np.tile(zero_inflated, (len(train_y), 1))
positive_weighted *= (train_y > 0)
one_inflated = (train_y < 1).mean(axis=0) < 0.1
negative_weighted = np.tile(one_inflated, (len(train_y), 1))
negative_weighted *= (train_y < 1)
train_weights = np.where(positive_weighted + negative_weighted, 2., 1.)
gelu_new activation for BERT-based models
cosine warmup scheduler
EMA
Post-processing
I just clipped predictions. The thresholds are decided by golden section search.
This improve my score from 0.4434 to 0.4852. (CV score)
before post-processing scores:
![](assets/6cb8faef9abf089fb052774e.png)
after post-processing scores:
![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F04f51a001c9f734fed93003cfe8e935b%2Fnewplot%20(1)
.png?generation=1581416716829926&alt=media)
```
class OptimizedRounder(BaseEstimator, TransformerMixin):
def __init__(self):
    self.threshold = [0., 1.]
    self.ab_start = [(0., 0.2), (0.8, 1.)]

def fit(self, train_labels, train_preds):
    assert train_labels.shape == train_preds.shape
    assert train_labels.ndim == 1

    self.best_score = self.score(train_labels, train_preds)
    self._golden_section_search(train_labels, train_preds, 0)  # lower threshold
    score = self.score(train_labels, train_preds)
    if score > self.best_score + 1e-3:
        self.best_score = score
    else:
        self.threshold[0] = 0.

    self._golden_section_search(train_labels, train_preds, 1)  # higher threshold
    score = self.score(train_labels, train_preds)
    if score > self.best_score + 1e-3:
        self.best_score = score
    else:
        self.threshold[1] = 1.

def _golden_section_search(self, train_labels, train_preds, idx):
    # idx == 0 -> lower threshold search
    # idx == 1 -> higher threshold search
    golden1 = 0.618
    golden2 = 1 - golden1
    for _ in range(10):
        a, b = self.ab_start[idx]
        # calc losses
        self.threshold[idx] = a
        la = -self.score(train_labels, train_preds)
        self.threshold[idx] = b
        lb = -self.score(train_labels, train_preds)
        for _ in range(20):
            # choose value
            if la > lb:
                a = b - (b - a) * golden1
                self.threshold[idx] = a
                la = -self.score(train_labels, train_preds)
            else:
                b = b - (b - a) * golden2
                self.threshold[idx] = b
                lb = -self.score(train_labels, train_preds)

def transform(self, preds):
    transformed = np.clip(preds, *self.threshold)
    if np.unique(transformed).size == 1:
        return preds
    return transformed

def score(self, labels, preds):
    p = self.transform(preds)
    score = scipy.stats.spearmanr(labels, p)[0]
    return score
```
Blending strategy
I used TPE optimization, same method as
[previous solution](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/97471#562472)
.
Blending weights:
test_preds = lstm_test_preds * 0.1145 + \
             bert_uncased_test_preds * 0.0112 + \
             bert_uncased_2_test_preds * 0.0911 + \
             bert_cased_test_preds * 0.0446 + \
             bert_large_uncased_test_preds * 0.1670 + \
             bert_large_uncased_2_test_preds * 0.0487 + \
             bert_large_cased_test_preds * 0.1094 + \
             bert_large_cased_2_test_preds * 0.1369 + \
             gpt2_test_preds * 0.0478 + \
             albert_test_preds * 0.0120 + \
             xlnet_test_preds * 0.1607 + \
             roberta_test_preds * 0.0560
What didn't work for me
Pre-training
masked LM
input-response prediction (
[](https://arxiv.org/abs/1705.00652)
[https://arxiv.org/abs/1705.00652](https://arxiv.org/abs/1705.00652)
)
Data augmentation
inverse translation (
[](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038)
[https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038)
)
manifold mixup
up-sample minor labels