# 8th place solution (OneFormer works)

Summary
Models:
OneFormer
, effnet, MaxViT, resnetrs, nfnetf5
Train on
soft labels and pseudo label
Optimizing Ensemble Weights
Loss: BCE
Only random image cropping without TTA and without mask shift
Introduction
Our team would like to thank the organizers and Kaggle for making this competition possible. Also, I want to express my gratitude to my outstanding teammates
[@cnumber](https://www.kaggle.com/cnumber)
and
[@yukiokumura1](https://www.kaggle.com/yukiokumura1)
for their incredible contribution toward our final result.
Details
Our main strategy is the adoption and optimization of various models and the use of soft and pseudo labels.
Models:
In the beginning of the competition, we mainly experimented with a combination of smp and timm, but from the middle of the competition, we also started using OneFormer.
OneFormer demonstrated the best results.
private lb score of single oneformer (dinat-l) : 0.70204
With CV score of holdout dice, effnet b7: 0.671, oneformer(dinat-l): 0.693
.
Model resolutions are below: (inference settings)
OneFormer
effnetb7, b8
MaxViT-Tiny, Base
resnetrs
nfnetf5
1024
640
512
512
640
Data: Soft and Pseudo Labels:
Using soft labels (the average of all annotator labels) during training.  We also introduced pseudo labels for some models. When both are used together, training is performed randomly on images with the 0th through 7th time frame, with soft label used only for the 4th time frame and pseudo label for all other cases.
Optimizing Ensemble Weights:
To maximized the performance of ensembling, we incorporated optimization using Optuna within the CV environment. This facilitated the efficient verification of the best model combinations.
Be open to consideration
Interplay Between a Strong Backbone and Pseudo Labels:
The impact of using pseudo labels
, when trained with pseudo label of when trained with pseudo labels *, effnet b7: 0.685, oneformer(dinat-l): 0.695
*The average of each model trained in holdout is used as pseudo label.
However, the enhancements were relatively modest with OneFormer.
One hypothesis suggests that pseudo labels might bring about a knowledge distillation effect. The impact of this might reach a limit when the backbone is of a certain strength.