# 9th place recap

Share some personal experiences in this competition.
CV strategy
No CV, only public leaderboard. It just works.
At early stage, I tried to build CV and even wasted lots of submissions to do leaderboard probing. Then I realized that CV is just waste of time. I just assume unlabeled soundscapes, public test set and private test set share similar data distribution later on.
Data Augmentation
Only cutmix and mixup. Only one model with time & freq mask and it overfits.
Traditional data augmentation tends to overfit the train data, because it teaches the model to output the same representation for data from single audio clip (single domain knowledge). While things like mixup have some cross domain knowledge, because data comes from different audio clips. (not really sure, just guessing)
![nice pic from OPTIMAL REPRESENTATIONS FOR COVARIATE SHIFT](assets/9c3030a456229a8cd1d517cc.png)
Some Model Design
Almost last year's model. Add a bottleneck layer to let the representation more compact.
self.bottleneck_layer = nn.Sequential(
            nn.Linear(backbone_out, cfg.bottleneck_dim),
            nn.BatchNorm1d(cfg.bottleneck_dim),
            nn.ReLU())
bottleneck_dim=1024(actually 512 is better). This can improve score a little(less than 0.01).
With eca_nfnet_l0 as backbone and above tips, my baseline public score is 0.65(close to 0.66).
Contrastive Adversarial Domain (CAD) bottleneck
from the paper
[OPTIMAL REPRESENTATIONS FOR COVARIATE SHIFT](https://arxiv.org/pdf/2201.00057)
.
![model](assets/34dbf277ac94cabb6971f3f9.png)
Honestly, I don't understand the math behind that(read the paper if you are interest). Practically, it use data domains as labels (in our case, train data: 0, unlabeled_soundscapes: 1), and bottleneck_layer's output as features to do SupCon learning. The total loss is traditional BCE loss + weight*SupCon loss. The weight and the temperature in SupCon loss is tuned base on public leaderboard.
weight
Public
Private
1e-2
0.56
0.57
1e-3
0.66
0.63
1e-4
0.68
0.66
1e-5
0.65
0.64
temperature
Public
Private
0.07
0.62
0.62
0.09
0.65
0.65
0.10
0.68
0.66
0.11
0.65
0.66
![supcon loss](assets/a5feac3dc9fc6958caa27965.png)
At the early stage, BCE loss dominates, then SupCon loss starts to kick in.
Post-process trick
Average prediction with previous and next window. Improvement depends(0.01~0.02).
Model ensemble and selection
My best ensemble is actually two eca_nfnet_l0s with different data split and SupCon loss weight, it scores 0.68 in private, but I didn't select it.
I further ensemble these two with a tf_efficientnetv2_b0(public 0.65, private 0.62, with time & freq mask) and rexnet_100(public 0.63, private 0.64). Don't have enough time to finetune these two, maybe they can improve.
Things didn't work(overfitting mostly)
pretrain (on 22&23 data, maybe I should try pretrain on unlabeled_soundscapes)
knowledge distillation (only try train data, maybe I should try knowledge distillation on unlabeled_soundscapes)
Entmin (definitely useless)
DANN (no improvement)
focal loss (it actually works solely, but no improvement with CAD)
Final thought
This competition is very hard. On domain adaptation/generalization leaderboard Domainbed, simple baseline is hard to beat. In practice, no current methods for learning representation uniformly outperform the baseline. I think that's reason for the big shakeup, and why public notebooks outperform carefully trained models.