# My journey (11th solution, with code)

I decided to join early this competition because I was frustrated by the previous two bird song competitions.  In Cornell competition I joined too late. In Rainforest competition,  missed some key train/test distribution insights and stagnated in the LB after  a great start.
The same 5 folds CV model (efficientnet b3 on first and last 5 seconds mel spectrograms) as in
[Cornell competition](https://www.kaggle.com/c/birdsong-recognition/discussion/183219)
plus
[improvements from my Rainforest solution](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220304)
gave 0.69 public LB (0.60 private LB) at my first submission.  This made me very happy as I took the lead on the public LB with it.
The main difference with Cornell competition is that we were provided train soundscape.  As many I trained models on short audio records and tuned rounding thresholds on soundscape.  For CV I used the same as in my Cornell solution. Compute F1score on no call rows separately from f1 score on bird call rows, then compute final score with:
score_all = 0.54 * score_nocall + (1 - 0.54) * score_birds
This way a nocall sub CV is the same as the nocall submission LB.
This made my CV and LB identical for most of my submissions.
Tuning thresholds improved my public LB to 0.75 (0.64 private) at my 5th submission, after 2 days.  CV and LB were identical so far.  At the time everyone else score was well below 0.70.
This great start made me will to hide my score and I stopped submitting until someone matched my score.  It took 2 weeks.  At this time I submitted the same model bagged twice, i.e. the original  model averaged with a second model trained with a different seed.   This scored 0.78 on public LB (0.64 LB).  I knew it was lucky as CV was 0.76, but it looked great still. It took another 2 weeks for someone to beat this public score.  Later, after improving a bit the training procedure I got 0.80 public LB (0.66 private) with a 2 seeds x 5 folds submission of my baseline.
It means that my baseline alone gives me a top 20 final rank.
Given single models were so great I assumed ensembling would move me ahead further and I decided to focus on creating a wide range of individual models.  This is my main mistake, I should have worked on ensembling way earlier.
I only submitted ensembles after the submission outage, 3 days before deadline, and discovered that it was hard to have a blending ensemble that beats all its individual component models.  I beat my best individual model only in my last submission.  It has both a CV and a public LB of 0.80 and is also my best private LB at 0.67.
I started a stacking model last day, but this was too late…  So be it.
While I was holding top public score I decided to not submit and I explored lots of different models.  In particular I explored vision transformers.  I started with ViT and Deit.  First try with 384x384 mel spectrograms were disappointing.  Then I realized that i could use other image dimensions instead of a 24x24 grid of 16x16 patches.  I tried 12x48, i.e. a 192x768 spectrogram, and also a 16x36 grid (256x576 spectrogram).  The only trick is to modify the position embeddings to match the new grid dimension.
This led to 0.75 public LB (0.64 private).
But the most interesting one was to forget about square patches altogether.  A 196x576 spectrogram can be seen as 576 time slices of size 196.  Each slice contains 16x16 entries.  It means that I could just use the time slices as input patches.  Here is how this input looks once it is reshaped as a 24x24 grid of 16x16 patches:
![time slices](https://i.imgur.com/F3zVGVM.pngd)
Maybe surprisingly, vision transformers are happy with this input.  The main advantage is that there is no longer any issue with translation on the frequency axis, which is the main issue with CNNs applied to spectrograms.
Blending Deit trained on this input with my baseline gave my best sub.  The Deit model alone scores 0.77 on public LB and 0.66 on private LB.
Although I am disappointed by my final result, I am happy to have explored lots of vision models and devised some new ways to use them.  And being disappointed by a solo gold in a deep learning competition is something I would not have imagined one year ago anyway ;)
Special thanks to Ross Wightman for his timm package.  It made my model exploration seamless.
Edit. I shared code and the paper I submitted to the workshop:
[https://github.com/jfpuget/STFT_Transformer](https://github.com/jfpuget/STFT_Transformer)