# 11th Place Solution Write-up

Summary [TLDR]
Weighted ensemble (optimal weights by Optuna)
Multi-stage long training runs (70-100 epochs)
Pseudo-labeling (time slices = 1-4, 6-8)
SWA [3]
TTA d4 / FlipTransforms [4]
Image size x2
Modelling Approach
We used
human_pixel_masks
as ground truth. We tried to incorporate
human_individual_labels
in different ways, but couldn’t improve using them. We didn’t try to average them and train on the soft labels. Our ensemble includes a model that calculates the losses on all individual masks and includes the one with the largest loss as an aux loss.
We tried to improve a single model using temporal information and also couldn’t improve over 2d setup, so we moved to create a diverse ensemble using diff encoders and decoders. We used different CNNs and transformers as backbones and we also had models with a transformer decoder (UNetFormer).
Many experiments conducted to utilise the additional data in the form of additional time periods so we incorporate that information in at least some way if not directly modelling for it. What ended up working for the pseudo setup in terms of both improving the single model and adding to the ensemble was the following:
Balance the batch to continue ½ pseudo ½ real examples.
Apply heavy augmentation on the pseudo data and light to normal augmentation on real data.
Mixup the pseudo examples with the real examples.
Training setup
Optimizer: AdamW
LR: 1.0e-3 or 1.0e-4
Scheduler: Cosine Decay with 2 epochs warmup.
Augs: Flips / RandomResizedCrop / RandAugment / Mixup / Mosaic
Mixed loss: BCE / Dice / Focal
CV strategy
For most of our experiments we used the split provided by the hosts, i.e. train on full train data and evaluate on
validation.csv
. We did earlier some experiments with KFold but OOF CV was overestimated and the CV scores on the holdout (validation.csv) weren’t good.
Evaluation
We used 2 variants for evaluation against ground truth labels (256x256):
i) Model outputs x2 size -->
interpolate(256, mode='nearest')
--> calculate Dice score
ii) Model outputs directly 256 size
PS: Most of our experiments used
(i)
, although we believe 2nd setup without interpolation is more reliable
Architectures / Backbones
Final ensemble
CV: 0.7026
/
Public LB:  0.71868
/
Private LB: 0.71059
Backbone
Architecture
CV
TTA
Pseudo
Mixup
efficientnet-b7
Unet (SMP)
0.6428
Yes
No
Yes
tf_efficientnet_b8
Unet++ (Timm)
0.6704
No
No
Yes
tf_efficientnet_b8
Unet++ (Timm)
0.6752
Yes
No
Yes
maxvit_small_tf_512
Unet (Timm)
0.6677
No
No
Yes
convnext_base
Unet++ (Timm)
0.6618
No
Yes
Yes
convnext_large
Unet (Timm)
0.6670
No
Yes
Yes
timm-resnest200e
Unet
0.6869
Yes
No
No
timm-resnest200e
Unet
0.6853
Yes
No
Yes
timm-resnest200e
Unet
0.6885
Yes
Yes
Yes
convnext_large_384_in22ft1k
UnetFormer
0.6788
Yes
No
Yes
tf_efficientnet_b6
UnetFormer
0.6532
No
No
No
eva02_large_patch14_448
Unet
0.6542
Yes
No
No
convnext_large_384_in22ft1k
UnetFormer
0.6841
Yes
No
Yes
The selected ensemble was our
best CV
/
best Public LB (6th place)
and the
best PVT LB (11th place)
CV-LB Plot
![](assets/1eb967418f60c2c1bc0522a1.png)
![](assets/d38c9b5b26a32c30a64bb154.png)
Things didn't work
Use of ConvLSTM, UTAE, 3D-Unets to leverage information from other time slices
Post-processing with t-2, t-1, t slices
diff image resolutions (384, 768, 1024)
Acknowledgements
We would like to thank hosts and Kaggle team for organizing such an interesting research competition. We are also grateful to all kagglers that share their ideas in discussions and notebooks.
Libraries
[1]
[Timm](https://github.com/huggingface/pytorch-image-models)
[2]
[segmentation_models.pytorch (SMP)](https://github.com/qubvel/segmentation_models.pytorch)
[3]
[SWA implementation](https://github.com/izmailovpavel/contrib_swa_examples)
[4]
[ttach](https://github.com/qubvel/ttach)
Team GIRYN
![](assets/552fef8e05268ff29d815d6d.png)
[Rohit Singh (Dracarys)](https://www.kaggle.com/rohitsingh9990)
[Nihjar (doomsday)](https://www.kaggle.com/phoenix9032)
[Ioannis Meintanis](https://www.kaggle.com/imeintanis)
[Yousef Rabi](https://www.kaggle.com/yousof9)
[Giba](https://www.kaggle.com/titericz)