# Public 20th | Private 15th solution 

Thanks to competition host for hosting this year BirdClef a bit different  as compared to previous competitions. Data provided has resulted in exploring some SFDA (Source Free Domain Adaptation) approaches. Ultimately I want to realize SFDA as mentioned in
[https://research.google/blog/in-search-of-a-generalizable-method-for-source-free-domain-adaptation/](https://research.google/blog/in-search-of-a-generalizable-method-for-source-free-domain-adaptation/)
but not able to do so.
Need to use public LB score as CV. Not a good situation but local CV is quite high and there is evident domain shift in the test data
Phase 1:
Pretraining using XCM data (
[https://huggingface.co/datasets/DBD-research-group/BirdSet)](https://huggingface.co/datasets/DBD-research-group/BirdSet))
, BirdClef 2021, 2022, 2023
Training using Birdclef 2024 data. Training and augmentation as mentioned in
[https://www.kaggle.com/code/salmanahmedtamu/training-0-65-0-66](https://www.kaggle.com/code/salmanahmedtamu/training-0-65-0-66)
is used
Max public/private score: 0.66 / 0.63
Phase 2: Pseudo Labels
General methodology used is in
[https://arxiv.org/pdf/2011.13439](https://arxiv.org/pdf/2011.13439)
(UNSUPERVISED DOMAIN ADAPTATION FOR SPEECH RECOGNITION VIA UNCERTAINTY DRIVEN SELF-TRAINING)
step 1) Generate pseudo labels in eval mode
step 2) Generate pseudo label in train mode with dropout (noisy), generated 8 samples with different seed
step 3) Calculate cosine distance of pairs (eval mode prediction, train mode prediction)
step 4) Select ulabeled frames ( file + start) having cosine distance < ( mean(cosine) - std(cosine))
It selected around 21 K frames for unlabeled data
`
preds = []
preds1=train_fold(True)
preds.append(preds1)
for i in range(8):
set_seed(i)
preds2=train_fold(False)
preds.append(preds2)
predsa=np.stack([preds],axis=3)
diffs = []
for i in range(predsa.shape[1]):
p0 = predsa[0,i][:,0]
darr = []
for j in range(1,9):
pp = predsa[j,i][:,0]
d = cosine(p0,pp)
darr.append(d)
diffs.append(darr)
diffs = np.array(diffs)
soundscape_meta_df['cosine'] = diffs.sum(axis=1)
soundscape_meta_df['preds'] = [ p for p in predsa[0]]
`
Phase 3: Using filtered unlabeled data with pseudo labels and competition data
step 1) Introduced augmentation by added noises generated from nocall + ESC 50. Training with noise now didn't hurt LB score which was earlier occurring only with competition data.
step 2) Pretraining with noise augmentation
step 3) Training with noise augmentation, trained 2 models with eca_nfnet_l0 and effnet_b0 backends
step 4) Ensemble of two gives following public/private LB: 0.702/ 0.666
![](assets/7c7b841f03488b98adc0d3f7.png)
Opportunity lost, I was thinking to create ensemble of some other 2 models, but driven by LB to select the maximum scoring ensemble
![](assets/2ee596571d15da23419d4b05.png)
After reading discussions I should have used openvino to create better ensembles.. I was using pytorch script and limited to 2 models.