# 22nd Place Solution Congratulations on all teams that have managed to secure top places and develop well performed solutions. It was a tough problem with main challenges the shift domain & noisy/weak labelling as described by many others. What follows is not rocket science, mostly is based on ideas from previous audio competitions, however I thought that It might be useful to describe it. TLDR Our team's submission is a weighted ensemble of 3 groups of models (CNNs + SED) with different backbones, training procedures, durations etc + PP (threshold optimisation) Read about our team's ensemble approach by [@rohitsingh9990](https://www.kaggle.com/rohitsingh9990) Read about our [9th Public solution outline & PP](https://www.kaggle.com/c/birdclef-2021/discussion/243349) by [@jaideepvalani](https://www.kaggle.com/jaideepvalani) ![](assets/a54cebbedc401c30cd53e295.png) Modelling approach (my journey) Before I start working on this problem I gave some time to read past year's top-10 solutions and view/analysed many many spectrograms. In overall, I performed more than 60 training experiments (single fold most of them) with different pipelines. Next, I describe briefly the main approaches 1) CNNs trained with precomputed mel-specs (5, 7, 10 sec) 2) CNNs with different pooling/heads a) attention; b) plain dense head; that takes the raw audio as input (10, 15 sec) and compute mel-spec transformations on the fly. This increased training time but gave more flexibility trying different params as the penultimate goal was to add some diversity to the ensemble. 3) SED models (few experiments at start of comp. however none of them used in team's ensemble, [@rsinda](https://www.kaggle.com/rsinda) had better results with sed models so I leave it for him to describe this part) For all three cases above numerous backbones tested but the following worked better for me: backbones: effnet-B0, regnetx, densenet121 The basis for all training pipelines was the 2nd solution code from last year's comp shared generously by [@vlomme](https://www.kaggle.com/vlomme) with many modifications along the road - I guess the most important one was to use the audio ratings as loss weights (sample wise) to account for the "noisy" low rating recordings. Mel-spectrograms: n_fft=2048 , n_mels=128 , with diff hop_lengths = [375, 512, 878] 5, 7, 10 sec audio crops, started with 3 channels but towards the end I used single channel mels Optimizer/Loss: Adam / BCELogits & FocallLoss (tried many others but mainly these two) Metrics to monitor: f1 at 03,05,07, max_f1, precision, recall (during training and on SC) Training - other tips use of secondary labels pink + white noise use Time-Freq masking Mixup / Mixor (for some of them) use Multi-sample dropout a.k.a. big dropout [(see paper)](https://arxiv.org/abs/1905.09788) for better generalisation (tried on last days - haven't tested on LB alone but didn't see improvement using it in team's ensemble) used wandb for logging experiments & excel sheet with more details check heatmaps with model predictions for each Train SC (eg see figure below) ![img](assets/2479bb102de7c85ac04ceb3d.png) Things that didn't work for me / didn't show improvement training a second stage model on hot-slices (selected by the best at the time 1st stage model) training with reduced bird classes ~300-330 using deltas for 3-channel using diff random_power() - ie, change contrast finetuning checkpoints from previous year comp (trained on 264 classes) from [@theoviel](https://www.kaggle.com/theoviel) 3rd place solution tweaking [@iafoss](https://www.kaggle.com/iafoss) type model from last year comp [described here](https://www.kaggle.com/iafoss/cornell-birdcall) Acknowledgments Thanks to hosts and kaggle for organizing this competition & every single kaggler that shared his notebooks/tips/concerns in public. Thanks to my teammates [@rohitsingh9990](https://www.kaggle.com/rohitsingh9990) , [@jaideepvalani](https://www.kaggle.com/jaideepvalani) , [@rsinda](https://www.kaggle.com/rsinda) , [@tomohiroh](https://www.kaggle.com/tomohiroh) and especially our team leader [@rohitsingh9990](https://www.kaggle.com/rohitsingh9990) for his efforts to integrate/test all models to our final ensemble.