# 26 place solution

Congrats to all winners! And thanks to organizers!
The competition was very very difficult for me, but fortunately I got 26th place.
I share my gotten knowledge.
modeling
I used a huggingface transformer and customized the head of the model.
![](assets/9d09a7f517e409a2cbae6c64.png)
As you know, I imitated the first solution of the
[jigsaw](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/103280)
.
BERT-base-cased, BERT-base-uncased and XLNet-base-case with this header were weighted at 2:4:4.
Parameter
10 GroupFold
epoch: 3
batch size: 8
optimizer: AdamW
OneCycleLR
max_lr=5e-5 (decay*0.9)
loss: BCE + MSE(MSE weighted by label frequency)
Token
I add category token. like  below;
[CLS] [CATEGORY_TOKEN] question_title [SEP] question_body [SEP] ansewr [SEP]
It raised 0.05 my CV.
Optimize binning
I read
[this Discussion](https://www.kaggle.com/c/google-quest-challenge/discussion/119177)
.
I calculated the common denominator and binned the predicted values.
Optimized binning with features like
[OptimizeRounder](https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/76107)
.
not work for me
pseudo label
MultiSample Dropout
SWA
AdaBound
FocalLoss
RoBERTa, ALBERT
Prediction with Special Token for each target
my code
train:
[https://github.com/trtd56/KaggleQuest](https://github.com/trtd56/KaggleQuest)
predict :
[https://www.kaggle.com/takamichitoda/26th-place-solution?scriptVersionId=28524791](https://www.kaggle.com/takamichitoda/26th-place-solution?scriptVersionId=28524791)
Thank you!!