# Learning Spectral Representations with SimMIM and Swin Transformer V2

Overview
Our solution follows a two-stage training strategy combining self-supervised learning and supervised fine-tuning. Since the competition dataset contains multispectral Sentinel imagery with limited labeled samples, we first learned general spectral representations using a SimMIM-style masked image modeling task and then fine-tuned the pretrained encoder for disease classification.
The final pipeline consists of:
Multispectral preprocessing and normalization
Self-supervised pretraining with SimMIM
Supervised fine-tuning using class-weighted cross-entropy
Test-time augmentation (TTA) during inference
Data Preparation
The dataset consists of Sentinel imagery containing 12 spectral bands:
B1, B2, B3, B4, B5, B6, B7, B8, B8A, B9, B11, and B12.
Each band was resized to 256×256 using bilinear interpolation. Band-specific normalization was applied using predefined mean and standard deviation statistics.
For robustness, missing or unreadable bands were replaced with zero-valued arrays.
During training, only spatial augmentations were used:
Horizontal Flip
Vertical Flip
Random Rotation
No spectral perturbation was applied because preserving physically meaningful spectral information was considered important for multispectral imagery.
Model Architecture
The backbone of the solution is Swin Transformer V2 Tiny (swinv2_tiny_window16_256).
The original ImageNet-pretrained model accepts RGB input (3 channels). Since Sentinel imagery contains 12 channels, the patch embedding layer was modified to accept 12-band input.
Instead of random initialization, pretrained RGB weights were replicated across all spectral channels. This provided a stable starting point and allowed the model to benefit from ImageNet pretraining despite the different input modality.
Self-Supervised Learning (SimMIM)
Before supervised training, a SimMIM-inspired masked image modeling stage was performed.
Masking Strategy
A random mask ratio of 60% was used. Masked regions were removed from the input image while visible regions remained unchanged.
The encoder was required to reconstruct the missing multispectral content.
Reconstruction Head
A lightweight convolutional decoder was attached to the Swin encoder. PixelShuffle was used to recover full image resolution.
Loss Function
Only masked regions contributed to the reconstruction objective.
An L1 reconstruction loss was computed between the reconstructed image and the original image:
Loss evaluated only on masked pixels
Visible pixels ignored during optimization
Training Configuration
Optimizer: AdamW
Learning Rate: 5e-4
Epochs: 30
After pretraining, only the encoder weights were retained. The decoder was discarded.
Supervised Fine-Tuning
The pretrained encoder weights were loaded into a classification model.
A new classification head was attached for the four disease categories:
Aphid
Blast
RPH
Rust
Class Imbalance Handling
The dataset exhibited noticeable class imbalance. To address this issue, weighted cross-entropy loss was used:
Aphid: 1.0
Blast: 4.0
RPH: 0.6
Rust: 12.0
Higher weights were assigned to minority classes to improve their contribution during optimization.
Training Configuration
Optimizer: AdamW
Learning Rate: 1e-4
Scheduler: Cosine Annealing
Epochs: 20
The entire labeled dataset was used for supervised training.
Inference
For inference, Test Time Augmentation (TTA) was applied.
Predictions were generated from six transformed versions of each image:
Original image
Horizontal flip
Vertical flip
90° rotation
180° rotation
270° rotation
Softmax probabilities from all transformations were averaged, and the final class prediction was obtained using argmax.
Final Pipeline
Sentinel-2 Multispectral Images
→ Band Normalization
→ SimMIM Self-Supervised Pretraining
→ Swin Transformer Encoder
→ Supervised Fine-Tuning
→ Class-Weighted Cross-Entropy
→ Test Time Augmentation
→ Final Prediction
Conclusion
The most important component of the solution was the self-supervised pretraining stage. By leveraging all available imagery through masked image modeling, the Swin encoder learned useful spectral representations before receiving class labels.
Combined with class-weighted fine-tuning and test-time augmentation, this approach produced a robust classification pipeline for multispectral crop disease recognition.