# 9th place solution: smaller and faster

First, I would like to congratulate all winers and participants of this competition and thank kaggle and the organizers for posting this interesting challenge and providing resources for participating in it.  Also, I want to express my highest gratitude and appreciation to my teammates:
[@suicaokhoailang](/suicaokhoailang)
,
[@theoviel](/theoviel)
, and
[@yanglehan](/yanglehan)
for working with me on this challenge.  It was the first time when I worked with audio related task, and it was a month of quite intense learning. I would like to give special thanks to
[@theoviel](/theoviel)
for working quite hard with me on optimization of our models and prepare the final submissions.
The code used for training our final models is available now at
[this link](https://www.kaggle.com/theoviel/9th-place-modeling-kernel)
(private LB 0.72171 score, top 42, with a single model (5 CV folds) trained for 7 hours). Further details of our solution can be found in our
[DCASE2019 technical report](assets/96f1f98151d5a745cfd1c227.pdf)
.
The
key points
of our solutions are the following (see details below): (1) Use of
small and fast models with optimized architecture
, (2) Use of
64 mels
instead of 128 (sometimes less gives more) with
4s duration
, (3) Use
data augmentation and noisy data
for pretraining.
Stage 1
: we started the competition, like many participants, with experimenting with common computer vision models. The input size of spectrograms was 256x512 pixels, with upscaling the input image along first dimension by a factor of 2. With this setup the best performance was demonstrated by DenseNet models: they outperformed the baseline model published in
[this kernel](https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch)
and successfully used in the competition a year ago, also Dnet121 was faster. With using pretraining on full noisy set, spectral augmentation, and MixUp, CV could reach ~0.85+, and public score for a single fold, 4 folds, and ensemble of 4 models are 0.67-0.68, ~0.70, 0.717, respectively. Despite these models are not used for our submission, these experiments have provided important insights for the next stage.
Stage 2
:
Use of noisy data
: It is the main point of this competition that organizers wanted us to focus on (no external data, no pretrained models, no test data use policies). We have used 2 strategies: (1) pretraining on full noisy data and (2) pretraining on a mixture of the curated data with most confidently labeled noisy data. In both cases the pretraining is followed by fine tuning on curated only data. The most confident labels are identified based on a model trained on curated data, and further details can be provided by
[@theoviel](/theoviel)
. For our best setup we have the following values of CV (in stage 2 we use 5 fold scheme): training on curated data only - 0.858, pretraining on full noisy data - 0.866, curated data + 15k best noisy data examples - 0.865, curated data + 5k best noisy data examples - 0.872. We have utilized all 3 strategies of noisy data use to create a variety in our ensemble.
Preprocessing
: According to our experiments, big models, like Dnet121, work better on 128 mels and even higher image resolution, while the default model reaches the best performance for 64 mels. This setup also decreases training time and improves convergence speed. 32 mels also could be considered, but the performence drops to 0.856 for our best setup. Use of 4s intervals instead of traditional 2s has gave also a considerable boost. The input image size for the model is 64x256x1. We tried both normalization of data based on image and global train set statistics, and the results were similar. Though, our best CV is reached for global normalization. In final models we used both strategies to crease a diversity. We also tried to experiment with the fft window size but did not see a significant difference and stayed with 1920. One thing to try we didn't have time for is using different window sizes mels as channels of the produced image. In particular,
[this paper](https://arxiv.org/abs/1706.07156)
shows that some classes prefer longer while other shorter window size. The preprocessing pipeline is similar to one described in
[this kernel](https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data)
.
Model architecture
: At stage 2 we used the model from
[this kernel](https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch)
as a starting point. The performance of this base model for our best setup is 0.855 CV. Based on our prior positive experience with DensNet, we added dense connections inside convolution blocks and concatenate pooling that boosted the performance to 0.868 in our best experiments (model M1):
```
class ConvBlock(nn.Module):
    def
init
(self, in_channels, out_channels, kernel_size=3, pool=True):
        super().
init
()
padding = kernel_size // 2
    self.pool = pool

    self.conv1 = nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size,
            stride=1, padding=padding),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(),
    )
    self.conv2 = nn.Sequential(
        nn.Conv2d(out_channels + in_channels, out_channels, 
            kernel_size=kernel_size, stride=1, padding=padding),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(),
    )

def forward(self, x): # x.shape = [batch_size, in_channels, a, b]
    x1 = self.conv1(x)
    x = self.conv2(torch.cat([x, x1],1))
    if(self.pool): x = F.avg_pool2d(x, 2)
    return x   # x.shape = [batch_size, out_channels, a//2, b//2]
```
The increase of the number of convolution blocks from 4 to 5 gave only 0.865 CV. Use of a pyramidal pooling for 2,3 and 4-th conv blocks (M2) gave slightly worse result than M1. Finally, our ultimate setup (M3) consists of 5 conv blocks with pyramidal pooling reached 0.872 CV. DenseNet121 in the same pipeline reached only 0.836 (DenseNet121 requires higher image resolution, 256x512, to reach 0.85+ CV). From the experiments, it looks for audio it is important to have nonlinear operations before size reduction by pooling, though we did not check it in details. We used M1, M2, and M3 to create variability in our ensemble. Because of the submission limit we checked performance of only a few models in public LB, with the best single model score 0.715.
This plot illustrates architecture of M3 model:
![](assets/1eb2a6cc91efe06fb3bb0b40.png)
Here, all tested models are summarized:
![](assets/537e74129a08f3e8df9b586b.png)
And here the model performance for our best setup:
![](assets/037865167efb70d059bd1d22.png)
Data augmentation
:
The main thing for working with audio data is MixUp. In contrast to images of real objects,
[sounds are transparent](https://towardsdatascience.com/whats-wrong-with-spectrograms-and-cnns-for-audio-processing-311377d7ccd)
: they do not overshadow each other. Therefore, MixUp is so efficient for audio and gives 0.01-0.015 CV boost. At the stage 1 the best results were achieved for alpha MixUp parameter equal to 1.0, while at the stage 2 we used 0.4.
[Spectral augmentation](https://arxiv.org/abs/1904.08779)
(with 2 masks for frequency and time with the coverage range between zero and 0.15 and 0.3, respectively) gave about 0.005 CV boost. We did not use stretching the spectrograms in time domain because it gave lower model performance. In several model we also used
[Multisample Dropout](https://arxiv.org/abs/1905.09788)
(other models were trained without dropout); though, it decreased CV by ~0.002. We did not apply horizontal flip since it decreased CV and also is not natural: I do not think that people would be able to recognize sounds played from the back. It is the same as training ImageNet with use of vertical flip.
Training
: At the pretraining stage we used one cycle of cosine annealing with warm up. The maximum lr is 0.001, and the number of epochs is ranged between 50 and 70 for different setups. At the stage of fine tuning we applied ReduceLROnPlateau several times to alternate high and low lr. The code is implemented with Pytorch. The total time of training for one fold is 1-2 hours, so the training of entire model is 6-9 hours. Almost all our models were trained at kaggle, and the kernel is at
[this link](https://www.kaggle.com/theoviel/9th-place-modeling-kernel)
.
![](assets/5e94c8ec559f3f76e293b0f5.png)
Submission
: 
Use of 64 mels has allowed us also to shorten the inference time. In particular, each 5-fold model takes only 1-1.5 min for generation of a prediction with 8TTA. Therefore, we were able to use an ensemble of 15 models to generate the final submissions. 
The final remark about the possible reason of the gap between CV and public LB:
[this paper](assets/664fcc562465fb1a3f9b0c91.pdf)
(Task 1B) reports 0.15 lwlrap drop for test performed on data recorded with different device vs the same device as used for training data. So, the public LB data may be selected for a different device, but we never know if it is true or not.