# 7th Place Solution for the BirdCLEF 2024 Competition Congratulations to all the winners! Thanks to Kaggle and Cornell Lab of Ornithology for hosting this interesting competition. This year's birdclef is really a hard one. Unlike padded CMAP which suppress the impact of species with very few positive labels, AUC treat every species equally and is very sensitive. I am happy to survive the final shake. Let me introduce my solution. Thanks to every competitor who gave me inspiration. Special thanks to atsunorifujita, martynoveduard, loonypenguin, vladimirsydor, anonamename Context Business context: [https://www.kaggle.com/competitions/birdclef-2024/overview](https://www.kaggle.com/competitions/birdclef-2024/overview) Data context: [https://www.kaggle.com/competitions/birdclef-2024/data](https://www.kaggle.com/competitions/birdclef-2024/data) Overview of the approach My final submission is a combination of 2 SED model and 1 CNN model. The training scheme is almost the same as my [last year's 2nd place solution](https://www.kaggle.com/competitions/birdclef-2023/discussion/412707) . The new stuffs I added include Extract training sample from soundscape using Birdnet and Bird-vocalization-classifier Knowledge distillation proposed by [last year's 4th place](https://www.kaggle.com/competitions/birdclef-2023/discussion/412753) Sumixup proposed by [last year's 7th place](https://www.kaggle.com/competitions/birdclef-2023/discussion/412922) Int8 quantize Split species to serveral subset (most important for me) The key to win this competition is to split species to several subset and train the subset separately, which enables model to focus more on each species compared to train them all. My final submission is an ensemble of 2 models with all species and 1 model with 66 species which have relatively small sample size. Actually I had an experiment which split species to 3 subset, which gave me 0.690440(1st place) in private LB, but it was too risky to choose because I was not sure whether I was just overfitting the public LB. Details of the submission Models SED with tf_efficientnetv2_s_in21k (all species) SED with seresnext26t_32x4d (66 rare species) CNN with resnet34d (all species) Extra data on xeno-canto Like last year's situation, I collected extra audio from xeno-canto and trained a baseline SED v2s model. The baseline score was 0.65. Extracting train sample from soundscape using Birdnet and Bird-vocalization-classifier Birdnet covers 181 of 182 species and Bird-vocalization-classifier covers 180 of 182 species. I mainly used birdnet and extracted niwpig1 with bird-vocalization-classifier. I extracted the 15 second audio clip with threshold 0.3. This boosted my baseline score to 0.68. Knowledge Distillation I implemented the same knowledge distillation scheme as last year's 4th place. This further boosted my baseline score to 0.70. Further Extracting train sample from soundscape using trained models I used my own trained models to further extract train samples from soundscape, this gave me a little boost in LB. Species subset Inspired by [3rd place in birdclef 2022](https://www.kaggle.com/competitions/birdclef-2022/discussion/327193) , I tried to train a model with 66 species which have relatively small sample size, and I found that this model outperformed model trained with all species on 43 of 66 species. I was not sure whether this will also happen in private LB, but I decided to ensemble 1 model trained on 66 species into my final submission. I also tried to split species into 3 subsets with some overlap. I trained them separately and then created a submission with 3 subset model ensemble. This gave me 0.690440(1st place) in private LB, thus maybe splitting birds to subset is the key to win. But it was too risky to choose because I was not sure whether I was just overfitting the public LB. Int8 quantize Thanks to the unlabeled soundscape, we can perform quantization this year. I used nncf to quantize encoder part of sed model. Quantizing CNN model led to 0.01 decrease in LB score, so I didn't perform quantization for CNN. Quantization reduced inference time by 20 minutes for sed v2s model. (100 minutes to 80 minutes) nncf openvino ov nncf_dataset = nncf.Dataset(pytorch_dataset, transform_fn) model = ov.Core().read_model(path_to_model) quantized_model = nncf.quantize( model, nncf_dataset, target_device=nncf.TargetDevice.CPU, subset_size=, fast_bias_correction=, preset=nncf.QuantizationPreset.MIXED, ) ov.save_model(quantized_model, quantized_model_path) ensemble strategy logit average of 3 models rank average of 3 models Things didn't work for me. Too many… Adding validation data to training led to 0.01 LB decrease. It is very weird, but I have to accept it and be faithful… Calculating species weight to deal with label shift and covariance shift Giving more weight to audios recorded in india according to geometric information (latitude 8~21, longitude 72.5 ~ 79) changing weight of the sampler significantly decrease the LB. Sampling every species equally performed best Filtering out 2024 train data duplicates not only by id but also by 'author + primary_label'(proposed [here](https://www.kaggle.com/competitions/birdclef-2023/discussion/412808) ) using embedding extracted from birdnet and bird-vocalization-classifier proposed by [last year's 6th place](https://www.kaggle.com/competitions/birdclef-2023/discussion/412708) Sources [https://www.kaggle.com/competitions/birdclef-2023/discussion/412707](https://www.kaggle.com/competitions/birdclef-2023/discussion/412707) [https://www.kaggle.com/competitions/birdclef-2023/discussion/412753](https://www.kaggle.com/competitions/birdclef-2023/discussion/412753) [https://www.kaggle.com/competitions/birdclef-2023/discussion/412922](https://www.kaggle.com/competitions/birdclef-2023/discussion/412922) [https://www.kaggle.com/competitions/birdclef-2023/discussion/412808](https://www.kaggle.com/competitions/birdclef-2023/discussion/412808) [https://www.kaggle.com/competitions/birdclef-2023/discussion/412708](https://www.kaggle.com/competitions/birdclef-2023/discussion/412708) [https://www.kaggle.com/competitions/birdclef-2022/discussion/327193](https://www.kaggle.com/competitions/birdclef-2022/discussion/327193) [https://docs.openvino.ai/2023.3/basic_quantization_flow.html](https://docs.openvino.ai/2023.3/basic_quantization_flow.html)