# Part of the 18th place solution (Iafoss's model)

Summary
Sequence based model trained on weak labels
Multi-head self-attention applied to entire sequences
Train-Segment-Shift steps: generation of PL, building a segmentation model, and domain shift mitigation.
Rainforest noise
postprocessing based on 2.5s shifted sequence
best
single model performance 0.7598/0.6486
at public/private LB
Introduction
Congratulation to all participants. To begin with, our team would like to thank organizers and Kaggle team for making this competition possible. I also would like to express my gratitude to my teammates:
[@urvishp80](https://www.kaggle.com/urvishp80)
,
[@whurobin](https://www.kaggle.com/whurobin)
,
[@philipkd](https://www.kaggle.com/philipkd)
, and especially to
[@naoism](https://www.kaggle.com/naoism)
: he helped me today until 3:30 AM of his time to assemble the final submissions. Below I'll share the key point of our approach, and
[@naoism](https://www.kaggle.com/naoism)
may publish a separate post describing specific details of his part.
This competition was relatively short for me:  I had just ~2 weeks after finishing another competition. Below I provide the key points of my portion of our solution.
Model
My part of the solution is largely influenced by the method
[I proposed in 2020 Cornell Birdcall Identification challene](https://www.kaggle.com/c/birdsong-recognition/discussion/183258)
targeted at building a segmentation model based on weak classification labels.
![](assets/cafed61d945ecb0bdbfa1eb2.png)
I collapse the frequency domain into dim of size 1 and then consider the produced tensor as a sequence and then apply multi-head self-attention blocks to it, like in transformers. The produced output with stride of ~0.3s is merged with logsumexp (LSE) pooling to produce the prediction global prediction. In this year I quite modified this approach, and the model predictions look much better in comparison with ones from the previous year. First, following
[this article](assets/4edfad84047796a30b85b2aa.pdf)
I introduced temperature to LSE equal to 5. Another trick is enforcing the minimum of predicted sequence to be 0: bird call is only present in a short time frame. It is achieved by an additional LSE output with -1 temperature followed by applying a loss with zero GT label. These modifications drastically boosted the model performance. Another important thing is using sufficiently short audio segments during training. Long segments help to overcome label weakness, but I realized that a model trained on long segments learns only easy sounds ignoring the rest (it is often just enough to find an easy example in a long sequence to assign the corresponding label). To encourage the model to learn difficult examples I used 5s chunks during training (but I used PL based sampling to deal with weak labels). Important component of the model is a multi-head self-attention applied to entire sequence:  in my experiments this component boosted F1 of sequence based predictions (10 min) by 0.01-0.02 in comparison with CV evaluated based solely on 5s chunks.
Additional details:
Backbone: ResNeXt50
Loss: Focal loss, corrected to be suitable for soft labels
Augmentation:
[MixUp with max label](https://www.kaggle.com/c/birdsong-recognition/discussion/183199)
(proposed by
[@theoviel](https://www.kaggle.com/theoviel)
), pink noise, external noise
[important]
n_ftt = 2048. It appeared that models trained with n_ftt = 1024 are very susceptible to the domain shift, and n_ftt = 2048 leads to much better performance on soundscapes.
Train-Segment-Shift
It is the concept that was used to address the competition problem.
There are 2 major issues to address in this challenge: (1) label weakness and noise, and (2) domain mismatch
between train and test data. My approach consists of 3 steps: train, segment, and shift.
(1) I begin with creating a segmentation model based on the original labels assigned to entire clips. The main difficulty is that clip labels are not equal to chunk labels because the clips may have no bird calls in particular parts or bird calls from other species. So
direct training of a model on clip labels assigned to chunks cannot produce a good model
, but it gives PL for the next step.
(2) Run similar setup as step 1 but do sampling based on PL segmentation, making sure that the primary label bird call is included in each selected audio segment. In addition, I applied segmentation loss to region having high confidence pseudo labels, while masking regions where confidence is not sufficient. It is done to guide the model, and it improved the quality of segmentation labels at the end.
The image below shows the model predictions from the same audio for top 5 classes. Blue color corresponds to the primary label, while orange corresponds to the secondary label. The bottom image shows an example of segmentation PL used at step2 and step3. Red bars indicate high confidence PL, while orange ones correspond to low confidence PL (used for masking).
![](assets/35d2638775aded643e543114.png)
At this step I got ~0.85 F1 CV (based on short audio) evaluated by taking max values for predictions within entire clip train files (so the provided weak labels can be used for model evaluation). However, if I try to apply this model directly to train soundscapes (which are similar to test data), I get only 0.67 CV (soundscapes), which indicated the domain mismatch between short audio and soundscapes.
(3) That's why the next step is crucial: domain shift accommodation. In this competition the mismatch is not as bad as one year ago (in that case soundscapes were recorded with 16 kHz rate only). Though, if you plot the spectrograms or listen soundscapes, they are still drastically different from short train clips.
Initially I tried to add noise extracted from train soundscapes (with proper CV split) to each loaded audio segment (I didn't use train soundscapes directly, only noise from them) + pink noise. It didn't work well giving 0.7231/0.6401 at public/private LB. Next, I realized that using
[Rain Forest data](https://www.kaggle.com/c/rfcx-species-audio-detection/data)
works really well to eliminate FP. This data contains multiple background sounds and sounds similar to bird calls but not bird calls.
So the model learns to ignore sounds similar to bird calls as well as bird calls from unknown species.
This approach boosted the single model performance to  0.7598/0.6486 at public/private LB.
For example, the image below shows a single model single fold predictions for the first two minutes of 10534_SSW_20170429.ogg soundscape for top 10 classes. The predictions look really confident for detecting nocalls. There is some confusion between some classes though. Also, I noticed that GT labels missed several calls: it is quite clear that the calls are present, but probably experts were not confident enough about the species because of the noise.
![](assets/acad20ea4719f647f795bcd0.png)
One may ask, why do we need step2 and high accuracy segmentation PL. Without noise the model is able to localize the areas with corresponding bird calls. Meanwhile if extensive noise is added, the model needs hints on localization of birdcalls and ignoring the rest. Segmentation PL help to guide training.
Postprocessing
One important portion of our solution is postprocessing. We averaged the prediction for a given 5s chunk with predictions for chunks shifted by 2.5 seconds forward and backward (one can consider a shifted audio), so
p = 0.5*p0 + 0.25*p_r + 0.25*pl
. This postprocessing is especially effective to deal with cases when a birdcall is located at a chunk boundary. In addition,
[@naoism](https://www.kaggle.com/naoism)
was using a postprocessing based on multiplying all chunk predictions for a given class by 1.3 if the maximum predicted value for the given class over the entire sequence is reaching a particular threshold.
Ensembling
Our best final submission is combination of 2 models having 0.7471/0.6206 and 0.7598/0.6486 public LB and private LB scores which gives 0.7736/0.6605 for weighted ensemble.