# 18th place

Summary
I used various training setups (spectrogram settings, model types, data augmentation) to perform logit distillation and assembled an ensemble of four distillation models with different configurations.
Distillation
Using the $L_{MLD}$ from
[Multi-Label Knowledge Distillation](https://arxiv.org/abs/2308.06453)
led to an improvement in accuracy.
() -> torch.Tensor:
        
        loss_focal = self.focal_weight * self.focal_loss(logits_student, target.())

        
        prob_student = torch.sigmoid(logits_student)
        prob_teacher = torch.sigmoid(logits_teacher)
        prob_student = torch.clamp(prob_student, =self.eps, =-self.eps)
        prob_teacher = torch.clamp(prob_teacher, =self.eps, =-self.eps)
        loss_mld = self.kl_div_loss(torch.log(prob_student), prob_teacher) + self.kl_div_loss(torch.log( - prob_student),  - prob_teacher)
        loss_mld = loss_mld.mean()
        loss_mld = (epoch / self.warmup, ) * loss_mld

         loss_focal + loss_mld
◆Pattern 1
Model
Public
Private
ECA NFNet L0 (teacher)
0.685730
0.637945
EfficientNet B0 (student)
0.681507
0.638541
Focal + KD
0.674996
0.639671
Focal + DKD
0.657009
0.629826
Focal + MLD
0.695467
0.655356
◆Pattern 2
Model
Public
Private
ECA NFNet L1 (teacher)
0.690174
0.630097
EfficientNet B0 (student)
0.661563
0.622148
Focal + KD
N/A
N/A
Focal + DKD
N/A
N/A
Focal + MLD
0.704317
0.633722
◆Pattern 3
Model
Public
Private
ECA NFNet L1 (teacher)
0.680272
0.658853
EfficientNet B1 (student)
0.661441
0.620913
Focal + KD
N/A
N/A
Focal + DKD
N/A
N/A
Focal + MLD
0.693386
0.667936
KD: Logit KD, DKD:
[Decoupled Knowledge Distillation](https://arxiv.org/abs/2203.08679)
, MLD: $L_{MLD}$ from
[Multi-Label Knowledge Distillation](https://arxiv.org/abs/2308.06453)
Reflections
✅ Distillation with MLD Loss proved effective.
✅ Having models with varied training settings helped us withstand shake and create a slightly more accurate solution than the public notebooks.
❌ We struggled to determine the best approach.
Appendix
Code for converting
eca_nfnet_l*
from timm to OpenVINO format.
copy
 functools  reduce

 torch.nn  nn
 torch.nn.functional  F

 timm


 () -> nn.Module:
    names = access_string.split(sep=)
     reduce(, names, module)


 () -> nn.Module:
    converted_model = copy.deepcopy(model)
    module_table = (converted_model.named_modules())
     name, m  converted_model.named_modules():
         (m, timm.layers.std_conv.ScaledStdConv2d):
            scaled_weight = F.batch_norm(
                m.weight.reshape(, m.out_channels, -), , ,
                weight=(m.gain * m.scale).view(-),
                training=, momentum=, eps=m.eps).reshape_as(m.weight).detach()

            bias = m.bias   
            conv = nn.Conv2d(m.in_channels, m.out_channels, m.kernel_size,
                             stride=m.stride, padding=m.padding, dilation=m.dilation,
                             groups=m.groups, bias=bias, padding_mode=m.padding_mode)
            conv.weight.data = scaled_weight
             bias:
                conv.bias.data = m.bias

            
            parent_name, child_name = name.rsplit(, )
            parent_module = module_table[parent_name]
            (parent_module, child_name, conv)

     converted_model