# 4th place solution 

First of all, I would like to thank the organizers of the competition and the Kaggle team for the competition hosting. I would also like to thank all the participants who have shared their knowledge so generously, including the previous Birdcall competitions. Two weeks before the end of this competition, I didn't expect it to be so hard competition.
Keys of my solution
Ensemble
Max 62model (Best private: 47model)
Inference with global information for SED model
Post-processing
Training
Preprocess
The logmelspectrograms were calculated using TorchAudio and normalised using the means and variances per a image.
logmelspec_extractor = nn.Sequential(
            MelSpectrogram(
                ,
                n_mels=,
                f_min=,
                n_fft=,
                hop_length=,
                normalized=,
            ),
            AmplitudeToDB(top_db=),
            NormalizeMelSpec(),
        )
For Augmentation in waveform, gaussian and uniform, pink noise are added random.
As an augmentation on logmelspec, I used Mixup. As a parameter,  I used a probability of 0.2~0.5 and alpha=0.8.
Modeling
I used the SED model like
[@hidehisaarai1213 's kernel](https://www.kaggle.com/hidehisaarai1213/pytorch-training-birdclef2021-starter)
.
Using various backbones, the seconds(max 30s, min 10s) used training has been adjusted so that each has a batchsize of 36.
Training
clipwise_pred
is optimized directly. By doing this, I was able to suppress the generation of NaN on backward.
loss = bce_with_logits(torch.logit(clipwise_pred), target) +  * bce_with_logits((framewise_logit.()[], target)
Other points are that
use a secondary label
soft labels such as 0.5 did not work
trained 40~50 epochs
Pseudo labeling
I have tried some patterns using a combination of the following steps.
Simple per-audio file relabeling
Use clipwise_pred and threshold = 0.2
Inspired by
[@hidehisaarai1213 's solution](https://www.kaggle.com/c/birdsong-recognition/discussion/183204)
in the last competition, I saved
framewise_pred
and
time_att
with the whole audio file as input, and computed
clipwise_pred
by cropping the period used for training.
Create a pseudo label using the second method only for audio files without a secondary_label
If the pseudo label was below a certain value (0.05), the probability of 0.1 was used to set the primary and secondary labels to 0.
All patterns improved the performance of the single models by about 0.01, and although the score on train_soundscape did not contribute to ensemble, it did work somewhat on the public LB.
30s finetuning
Larger models are trained with shorter segments, such as 10s. Therefore, the labels become noisy. To solve this, only the Attention module was trained with 30s segments on 10 additional epochs. There is a slight improvement in score with train_soundscape, and it is included in ensemble.
Checkpoint selection
The checkpoint with the smallest oof loss was selected for validation using the same number of seconds as for training.
Inference with global information
This is my favorite and most effective part of my solution.
To begin with, the idea that I had before joining this competition was that if the SED model could accurately perform "Sound Event Detection", then it would be possible to improve predictions for the shorter time segment by cropping from the longer time segment the necessary parts of predictions.
In this way, longer term features can be used for inference.
framewise_pred_5s = self.fix_scale(feat[:, :, start:end])
att_5s = torch.softmax(time_att[:, :, start:end], dim=-)
clipwise_pred_5s = torch.(torch.sigmoid(framewise_pred_5s) * att_5s, dim=-,)
By implementing this idea, I can improve both score and public LB in train_soundscape by about 0.03. In practice, I used 30s as the segment for the longer period and implemented it so that the 5s interval I want to predict is in the center. This idea is also used to create a pseudo label in the second pattern.
Secondly, in
[7th place solution in previous Birdcall competition](https://www.kaggle.com/c/birdsong-recognition/discussion/183571)
,  the birds that were observed to appear using a high threshold for each audio file can be expected to appear in short segments with a high probability, and the sensitivity can be increased by lowering the threshold.
Inspired by this observation, I used a high threshold (0.05) in
clipwise_pred_30s
to generate a list of possible birds and a low threshold (0.025) in
clipwise_pred_5s
,  to perform AND operations.
((clipwise_pred_30s > high_threshold) + (clipwise_pred_5s > low_threshold)) >=
To get an idea of these thresholds, I used
scipy.optimize.dual_annealing
to optimize for train_soundscape. This seems rather risky, but I thought it was somewhat reasonable due to the simple public LB probing described below.
This double thresholding improved the score of public LB and train_soundscape by 0.03.
In the ensemble I took a simple average. Initially, hard voting was considered, but the score did not change much, so the simple method was chosen.
Post-processing with location and date
A list of birds observed within 450 meters around each site and a list of birds appearing in each month was made, and those not appearing were removed from the submission.
Here, because the two sites "COR" and "COL" are relatively close, the birds have only been removed if both are zero.
I have also quadrupled the thresholds for some rare classes of birds observed in the vicinity of the sites.
This post-processing resulted in a consistent improvement of less than 0.01 for both train_soundscape and public LB.
Simple public LB probing
Since train_soundscape and public LB were correlated to some extent, I was concerned that the distribution of public LB might be extremely similar to train_soundscape.
To verify this, I changed the site and bird predictions not included in the train_soundscape to nobird.
If there is no change in the score, it is assumed that train_soundscape and public LB have the same distribution, but the score has decreased considerably. (bird: 0.71->0.64, site: 0.71->0.58)
So I can see that public LB contains sites and birds that are not included in train_soundscape.
From this, I forcefully predicted that public and private would be randomly split. (This is strictly uncertain, but there was nothing else I could do.)
Experimental code and the inference notebook for best submission
Code:
[https://github.com/tattaka/birdclef-2021](https://github.com/tattaka/birdclef-2021)
Inference notebook:
[https://www.kaggle.com/tattaka/birdclef2021-submissions-pp-ave?scriptVersionId=64016465](https://www.kaggle.com/tattaka/birdclef2021-submissions-pp-ave?scriptVersionId=64016465)
Aggregation of the number of birds for post-processing:
[https://www.kaggle.com/tattaka/make-month-and-site-mask](https://www.kaggle.com/tattaka/make-month-and-site-mask)