# 30th Place Solution for the Google Research - Identify Contrails to Reduce Global Warming Competition

First of all, we would like to thank Kaggle and the host for such a funny (and
nothing-works-well-for-us
) competition. This is not a fancy solution but we feel proud of taking the right steps to generalize well to the private test set (it was indeed, 42 positions shake up 😃).
1.
Context
Business context
:
[https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview](https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview)
Data context
:
[https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/data](https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/data)
2.
Overview of the Approach
Our final solution was an ensemble of 8 models which combines different encoders and image sizes. Each model configuration was trained in turn 3 times with different random seeds (24 models in total) and always with all available data (train+validation).
Data preprocessing
ASH false color images, only 2D models with labeled frame and hard labels.
Validation schema
Provided train/validation split.
3.
Overview of the Approach
3.1.
Individual models and final ensemble
All models are based on UNet architecture from SMP. We combine them using a weighted average of the probabilities dropped by single models, where weights were determined optimizing the global dice score on the validation set using Optuna. The final CV score was
0.6825
(which may be a little biased because as we mentioned we optimized the weights directly using the validation set).
Weight
Encoder
Image size
LR
Epochs
BS
CV score
0.02
timm-resnest101e
384
5e-4
20
48
0.64397
0.07
maxvit_base_tf_384.in21k_ft_in1k
384
1e-4
20
16
0.65699
0.18
maxvit_base_tf_512
512
1e-4
40
8*
0.66414
0.02
tf_efficientnetv2_b3.in1k
768
5e-4
35
24*
0.65178
0.23
tf_efficientnetv2_m.in21k_ft_in1k
512
5e-4
35
24*
0.6525
0.14
tf_efficientnetv2_l.in21k_ft_in1k
384
5e-4
35
24*
0.64538
0.11
maxvit_large_tf_224.in1k
448
1e-4
40
8*
0.66407
0.23
maxvit_large_tf_384.in1k
384
1e-4
40
8*
0.66782
*gradient accumulation of 2.
3.2.
Threshold selection
The optimal threshold for the ensemble was 0.43, which was determined running a for loop varying the threshold from 0 to 1 with 0.1 increments and calculating the corresponding global dice. We observed a relatively wide maximum peaking at 0.43 which give us enough confidence to choose that threshold.
3.3.
Training procedure
We used CosineAnnealingLR and AdamW. Maybe it is important to note that we always tried to choose the number of epochs such that the last epoch is the best one. This generalizes better than choosing the best checkpoint during training, especially in our case, where we observed an irregular behavior of the training curves. Finally, once a model was validated, we took the same configuration and train it again using all the available data.
3.4.
Augmentations
We didn't notice the mask shift, so flipping and rotation didn't work for us. Other augmentations that worked were:
A.Compose([  
     A.RandomResizedCrop(height=, width=, p=),  
     A.RandomBrightnessContrast(p=)  
])  
CutMix(p=)  
MixUp(p=)
These are our implementations of CutMix and MixUp:
():
    W = size[]
    H = size[]
    cut_rat = np.sqrt( - lam)
    cut_w = (W * cut_rat)
    cut_h = (H * cut_rat)

    
    cx = np.random.randint(W)
    cy = np.random.randint(H)

    bbx1 = np.clip(cx - cut_w // , , W)
    bby1 = np.clip(cy - cut_h // , , H)
    bbx2 = np.clip(cx + cut_w // , , W)
    bby2 = np.clip(cy + cut_h // , , H)
     bbx1, bby1, bbx2, bby2

 ():
    indices = torch.randperm(data.size())
    
    shuffled_target = target[indices]

    lam = np.clip(np.random.beta(alpha, alpha),,)
    bbx1, bby1, bbx2, bby2 = rand_bbox(data.size(), lam)
    new_data = data.clone()
    new_data[:, :, bby1:bby2, bbx1:bbx2] = data[indices, :, bby1:bby2, bbx1:bbx2]
    
    lam =  - ((bbx2 - bbx1) * (bby2 - bby1) / (data.size()[-] * data.size()[-]))
    
     new_data, target, shuffled_target, lam

 ():
     alpha > , 
     x.size() > , 

    lam = np.random.beta(alpha, alpha)
    rand_index = torch.randperm(x.size()[])
    mixed_x = lam * x + ( - lam) * x[rand_index, :]
    target_a, target_b = y, y[rand_index]
     mixed_x, target_a, target_b, lam
And losses are calculated as:
self.cfg.mixup  torch.rand()[] < self.cfg.mixup_p:
    mix_images, target_a, target_b, lam = mixup(images, masks, alpha=self.cfg.mixup_alpha)
    logits = self(mix_images)
    loss = self.loss_fn(logits, target_a) * lam + ( - lam) * self.loss_fn(logits, target_b)

 self.cfg.cutmix  torch.rand()[] < self.cfg.cutmix_p:
    mix_images, target_a, target_b, lam = cutmix(images, masks, alpha=self.cfg.cutmix_alpha)
    logits = self(mix_images)
    loss = self.loss_fn(logits, target_a) * lam + ( - lam) * self.loss_fn(logits, target_b)
3.5.
What didn't work
A ton of things:
Using other bands/other data schema.
TTA.
We tried to combine the former and the latter frames with the labeled one at each stage of the UNet without success (only tried concatenation).
At the end of the competition we discovered that pseudolabels give a big boost for some models (not the strongest ones) but we didn't have enough time to properly experiment with it.
…
References
[ASH color scheme](https://www.kaggle.com/code/inversion/visualizing-contrails)
[OpenContrails: Benchmarking Contrail Detection on GOES-16 ABI](https://arxiv.org/abs/2304.02122)
[@egortrushin](https://www.kaggle.com/egortrushin)
[public notebook](https://www.kaggle.com/code/egortrushin/gr-icrgw-training-with-4-folds)
was our starting point.
Acknowledgments
To my wonderful teammates
[@maxdiazbattan](https://www.kaggle.com/maxdiazbattan)
[@edomingo](https://www.kaggle.com/edomingo)
[@juansensio](https://www.kaggle.com/juansensio)
(if I forgot something please feel free to write it in the comments).