# 2nd place solution

Summary
Customized U-shape network with CoaT, NeXtViT, SAM-B, and tf_efficientnetv2_s backbones
×2 (with pixel shuffle up-scaling of predictions to 256×256) and ×4 input upscaling: pixel-level accuracy of predictions is critical
Sequence of images: LSTM, Transformer, Convolutional temporal mixing
No flip and 90-degree rotation augmentation (nor TTA) because masks are shifted
Train on soft labels: label = average of all annotators
BCE + dice Lovasz loss
Code
My and
[@drhabib](https://www.kaggle.com/drhabib)
's
[part](https://github.com/DrHB/2nd-place-contrails)
(includes the best performing CoaT_ULSTM model)
[@theoviel](https://www.kaggle.com/theoviel)
's
[part](https://github.com/TheoViel/kaggle_contrails)
[Inference](https://www.kaggle.com/code/theoviel/contrails-inference-comb)
Introduction
Our team would like to thank the organizers and Kaggle for making this competition possible. Also, I want to express my gratitude to my outstanding teammates
[@drhabib](https://www.kaggle.com/drhabib)
and
[@theoviel](https://www.kaggle.com/theoviel)
for their incredible contribution toward our final result.
Details
This competition has two main challenges: (1) noisy labels and (2) pixel-level accuracy requirements. (1) If one checks the annotation from all annotators, a major disagreement is quite apparent. This label noise challenge may be partially addressed by using soft labels (the average of all annotator labels) during training. However, since the evaluation requires hard labels, the major model failure is the prediction of contrails near the decision boundary, which cannot be avoided. (2) The thing that could really be addressed by the models is achieving pixel-level accuracy of predictions. Pixel-level accuracy is quite important since contrails are only several pixels thick, and even a mistake by one pixel in the mask boundary in the lateral direction may result in a significant decrease in the dice score, used as a metric. For such a task, a typical solution is up-sampling the input of the model or replacing the final linear up-sampling in segmentation models with transposed convolution or pixel shuffle up-sampling. This modification gave us a substantial boost in initial experiments over the direct use of models on the original resolution.
Data
Following the
[organizer's publication](assets/1941d66763373b5ec6fdcd9c.pdf)
, we used “ash” false color images considering the 12 μm band, the difference between 12 and 11 μm bands, and the difference between 11 and 8 μm bands, respectively. We also tried to consider all bands or expand the “ash” color images with 8, 10, and 12 μm, but it resulted in lower performance (the pre-trained weights of the first convolution were replicated). We hypothesize that the best performance of ash color images may be a consequence of the use of the ash color images by annotators, and biased label noise. In our experiments, we up-sample the input with bi-cubic interpolation by ×2 for CoaT and NeXtViT models, and by ×4 for SAM model. EfficientNet model uses the original input but the stride in the first convolutional block is set to 1, which is equivalent to ×4 input up-sampling.
Model
The interesting thing about this competition is that the model is required to do both (1) tracking the global pixel dependencies because contrails are quite elongated, and (2) capable of generating predictions with pixel-level accuracy because contrails are only several pixels thick. (1) can be addressed with transformers, while (2) is addressed with convolutional networks or local attention. In the competition, we tried to address these points independently with SAM-B ViT backbone and EfficientNet v2 purely convolutional network. In addition, there is a class of networks, transformers with a hierarchical structure, which addresses both aspects of the considered problem, (1) + (2), simultaneously. This class includes CoaT, which resulted in the best single model performance in our experiments, NeXtViT, MaxViT (which we missed), and MiT transformer from Segformer (unfortunately, it cannot be used in the competition because of the license).
For the decoder we either considered a vanilla U-Net decoder (EfficientNet models) or a more customized set of U-blocks using pixel shuffle up-sampling and factorized FPN (CoaT, NeXtViT, SAM). In these blocks, we also replaced the BatchNorm with LayerNorm2d to have proper model convergence at the low batch size and used GELU instead of ReLU activation.
The models are schematically illustrated in the figure below.
![](assets/5926386981ba6e6a4b73af97.png)
Temporal mixing
In addition to our initial runs with single-frame models, we performed experiments with image sequences. In our best setup image, multi-frame models get ~0.01 improvement.
Since the images are taken with a considerable temporal delay, and the displacement of clouds between frames is huge (~10-30 pixels) and is deteriorated by input up-sampling, video models based on 3D convolutions or window attention (VideoSwin) are not effective in the context of the considered problem. For the same reason, early mixing or mixing the predictions of sequential models is also not expected to work. Therefore, mixing at the intermediate feature map scales, such as res/32 and res/16, is most promising: even if clouds/contrails are misplaced, the feature maps at low resolution are reasonably aligned. In our experiments with the CoaT model, the best performance is achieved if two low-resolution feature maps (res/32 and res/16) are considered. The output of the temporal mixing modules as well as the output of the remaining feature maps are pooled at frame 4 and the decoder sees only the input corresponding to the considered frame, as illustrated in Figure 1.
As temporal mixing modules, we considered LSTMs (which achieved the best performance), 1D temporal convolutions, and Transformers. LSTM (1 layer) and 1D temporal convolutions perform mixing along the temporal dimension only, while the spatial dependences between features are considered in the backbone and the decoder. The transformer-based mixing was proposed to perform an implicit feature map registration for a large displacement of contrails/clouds because it works with temporal and spatial mixing of features simultaneously and can match them even if they are not spatially aligned.
This mixing is performed in the following way. To each feature map, corresponding to a given frame, we add encoding to distinguish it from others, then we flatten tokens from all frames into a single sequence and process it with 2 transformer decoder blocks. Key and value input to the transformer is represented by the concatenated sequence of all frames, while the query is a sequence of tokens from the 4-th frame only. Unfortunately, transformer-based mixing ended up with slightly lower performance, and the training with it is less stable. However, we kept both approaches in the final ensemble, benefitting from the improved diversity.
In our experiments with CoaT/NeXtViT we followed the
[organizer's publication](assets/1941d66763373b5ec6fdcd9c.pdf)
and selected the first 5 frames as the input. Both LSTM and Transformer mixing were considered. In our experiments with EfficientNet we considered 4 frame input (2nd – 5th, 2 frames before and 1 frame after the annotated frame) with bi-directional LSTM. This frame selection is dictated by the instruction to annotators to have a contrail at least on two subsequent frames. With the SAM model, we considered 1D-convolutional mixing with 3 frames (1 frame before and 1 frame after the annotation) because of the heavy VRAM requirement for this model, which was also an issue in inference.
One idea we had is explicit registration of input frames to have contrails and clouds aligned between frames (temporal mixing is the most effective), for example considering optical flow. Unfortunately, the images contain not only clouds but also the Earth's surface, which does not move, or several layers of clouds moving in different directions. Therefore, we could not come up with any methods to proceed with the frame registration task. We were also considering performing registration based on the predicted contrails, but since contrails appear and disappear from frame to frame, this approach is also not feasible.
Pseudo-labels (PL)
In addition to the competition data, we have collected a dataset “Contrails GOES16 Images May” (single frame setup) [9]. We split the images into 256×256 tiles with partial overlap and generated the masks based on the ensemble of best single-frame models. This data is shared as a Kaggle dataset [10]. This data was used for pretraining the models in a single-frame setup in some experiments, followed by finetuning the models either in single-frame or multi-frame setups. The use of PL has drastically improved the performance of individual models (folds), but the diversity of the models is reduced even if each fold of the finetuned model uses an independently trained PL model. Therefore, the performance on the average prediction over all folds is comparable to one for the setup trained without PL.
We also have experimented with PL training on both external data + unannotated competition frames, but it did not give any benefit in finetuning a single frame model. Since such pretraining could affect multi-frame models, in the production experiments we used only external data at the PL step.
Training
CoaT and NeXtViT models: Training is performed for 24 epochs with Over9000 (Radam+LAMB+LookAhead) optimizer, learning rate of 3.5e-4, weight decay of 0.01. It appeared that this optimizer has superior compatibility with CoaT. In the case of PL pretraining, it is performed for 12 epochs on the external data, and then the models are finetuned for 12 epochs for CoaT and 18 epochs for NeXtViT models.
EfficientNet models: Pretraining is performed in a single frame setup for 100 epochs with additional CutMix augmentation, learning rate is 1e-3, AdamW optimizer, weight decay 0.2. During pretraining the fraction of PL data is linearly decreased to zero. Then the model is fine-tuned in the image sequence setup with 3e-5 for the encoder and 1e-4 for the decoder. The addition of external data for pretraining resulted in much stronger 2D models, hence multi-frame EfficientNet models were not considered in the final ensemble. For diversity, we also pre-trained models for 200 epochs, and in a full-fit setup (training on all data, including validation set).
SAM models: Our final submission includes only single-frame SAM-B ViT-based models. During training, we, first, train the model with a frozen encoder for 10 epochs, and then we unfreeze the model and continue for 20 more epochs. AdamW optimizer is used with the learning rate of 4e-6 for the encoder and 4e-5 for the decoder.
During training, we used ShiftRotateScale, RandomGamma, RandomBrightnessContrast, MotionBlur, and GaussianBlur augmentations. Flip and 90-rotate augmentation were not used because they resulted in worse performance. The training of all production models is performed for 5-6 folds our folds are trained on the entire train dataset with different random seeds, while the evaluation is performed on the provided validation set. This strategy is used because evaluation in a standard K-fold split of the training data overestimates CV (cross-validation), which may be caused by spatial overlaps of tiles in the training dataset.
Loss
As a loss function, we used BCE + dice Lovasz loss. The second term is a modified version of a Lovasz loss [12] where (1) ReLU is replaced with 1 + ELU, (2) a symmetric version is used, and (3) instead of IoU we use a surrogate function maximizing dice. The second term provides an insignificant boost to the individual model performance. However, this term gives a wider maximum of the dice with respect to the threshold selection, i.e. weaker dependence of the performance on the threshold selection. This property is vital to avoid shakeup at the private leaderboard because of the improper threshold and is preferable for more effective ensembling.
Results
The performance of our final models is summarized in the table below. Ex refers to models trained in PL setup (EfficientNet is trained with PL only). The postfix U refers to the single frame model, UT – transformer-based temporal mixing, and ULSTM – LSTM-based temporal mixing. The threshold is selected based on the search maximizing CV value. For most models, it is 0.46-0.50.
![](assets/198456c51bb7f0efed090ba5.png)
Best Single model
Our best single mode is CoaT, which for 5 folds gets 0.7039 CV, 0.71790 private, and 0.71243 public LB. Single fold CV is 0.6960+-0.0003 evaluated on val set (our folds are trained on the train set with different seeds because eval in a standard Kfold split of train data overestimates CV due to possible spatial overlaps).
We could take top-4 in the competition with this single model
.
Ensemble
Our best ensemble consists of 8 models with weights selected to make approximately equal contributions from all components: CV is 0.7140 (on eval set), 0.72574 public and 0.72304 private LB. The models and their weight are summarized in the table below.
![](assets/5ff725a819e27cf8116009c1.png)
EfficientNet includes 3 model setups (100 epochs, 200 epochs, and full-fit) with the total efficient weight equal to 1.
Model Execution Time
CoaT and NeXtViT models: Training time is 2 hours for single-frame and 8 hours for multi-frame models per fold on 2×RTX4090 GPUs. The inference time at Kaggle (P100) for multi-frame models is 35 minutes for 5 folds.
EfficientNet models: Training time is 3 hours 45 minutes for 100 epochs on 8×V100 GPUs per fold. Inference at Kaggle (P100) takes 20 minutes for 6 folds including data loading.
SAM models: Training time is 16 hours per fold on 2×A6000 GPUs. Inference at Kaggle (P100) takes 48 minutes for 5 folds including data loading.
Things we missed
(1) During the competition one surprising finding for us was that flips and 90-degree rotation augmentation in model training resulted in worse performance. It was quite surprising, but, unfortunately, we did not think sufficiently about the origin of this behavior and attributed it to prevailing winds and a particular shape of clouds and contrails. After the competition, it appeared that the masks were shifted by 0.5 pixels, which made any flips inapplicable, as we discovered. If the mask issue was fixed in our setup, flip/90-rotation augmentation could lead to an 8-fold increase of the effective training dataset size and enables further performance boost at the inference stage due to test time augmentation. So, potentially the performance of our models can be noticeably improved.
(2) PL single model pertaining on the external data gives moderate CV improvement to the single-frame model (interesting that at private LB this improvement is quite large). Meanwhile, this single-frame pretraining didn't give much to sequence models. So, external sequence data collection + proper reprojection could potentially server as a better pertaining strategy for sequence models, and we may expect a boost comparable to single-frame models.