# Top 8% Bronze Medal Solution

Many congratulations to all the winners in this competition. Huge thanks to all the discussion topics, public notebooks and datasets - most (if not all of) of my learning came from them.
First I would like to thank the organizers of this wonderful competition.
To say that I learned a lot by competing in this competition would really be an understatement. I wanted to take on the challenge of using Tensorflow in this competition and boy did it test me. If there was one thing that gave me hope in this competition it was
[@markwijkhuizen](https://www.kaggle.com/markwijkhuizen)
's amazing public
[notebook](https://www.kaggle.com/code/markwijkhuizen/gislr-tf-data-processing-transformer-training)
. It was the foundation to my entire work in this competition.
Initially I was using my own Transformer architectures in this competition which were not bad themselves but as
[@tatamikenn](https://www.kaggle.com/tatamikenn)
pointed it, the public notebooks had really good Transformer architectures, with a few tweaks and code cleanups I was able to extract good results both on CV and LB, this gave me a significant increase in CV and LB right on my first experiment.
I have open sourced the code base on GitHub ->
[https://github.com/abhinand5/isolated-sign-language-recognition](https://github.com/abhinand5/isolated-sign-language-recognition)
.
Leave a ⭐️ if you like my work.
What worked for me?
Flip augmentation (
aug_prob = 0.4
)
10 Fold Cross Validation
label_smoothing = 0.3
embed_dim = 256
input_size = 32
num_transformer_blocks = 2
(Using 1 was under-fitting and 3 was over fitting even after several experiments of trying to tune)
Attention heads:
MHA_HEADS = 4
and
MHA_HEADS = 8
Train on the entire dataset before submission
FP16 Quantization
GeLU activation
What did NOT work for me?
input_size = 64
Knowledge Distillation (wasted a lot of time in this)
Rotate augmentations (degraded CV and LB)
Adding Gaussian noise to the inputs (degraded CV and LB)
Removing
LayerNorm
from Transformer blocks (although the arch is shallow it did not work for me)
Strip Pruning
Dynamic Range Quantization (increased the inference time)
2D Affine transforms
Final Solution:
There were two different model architectures in the final solution.
Dataset:
Only Competition Data
Model 1:
Training
epochs = 50
Add LayerNorm
CV Strategy:
StratifiedGroupKFold(k=10)
Optuna tuning -> Best OOF on all folds
SEED = 555
Ensemble of Best Folds:
[3,6,7,9]
(based on OOF score)
QUANTIZE_MODEL = True
QUANT_METHOD = "float16"
No Augmentations
Below is the complete model conf:
Model 2:
Training
epochs = 100
Add LayerNorm
Single Split -> Optuna Tuning -> Use best hyper params to train on entire train data
SEED = 555
QUANTIZE_MODEL = True
QUANT_METHOD = "float16"
Flip Augmentations (
aug_prob=0.4
)
Below is the complete model conf:
Submission:
The submission is a simple ensemble of 5 models
Model 1 (folds=[3,6,7,9]) + Model 2

Private Score: 0.82815

Public Score: 0.73896
Ensembling code can be found below:
(tf.Module):
     ():
        (TFLiteEnsembleModel, self).__init__()

        
        self.preprocess_layer = preprocess_layer
        self.models = models

        
        self.weights = weights  [/(models)] * (models)
        self.weights = tf.reshape(self.weights, (-, ))


     ():
        
        x, non_empty_frame_idxs = self.proprocess_inputs(inputs)

        outputs = []
        
         _model  self.models:
            output = _model({ : x, : non_empty_frame_idxs })
            outputs.append(output)

        
        outputs = tf.concat(outputs, axis=)
        weighted_outputs = outputs * self.weights
        weighted_sum = tf.reduce_sum(weighted_outputs, axis=, keepdims=)

        
        outputs = tf.squeeze(weighted_sum, axis=)

        
         {: outputs}

     ():
        
        x, non_empty_frame_idxs = self.preprocess_layer(inputs)
        
        x = tf.expand_dims(x, axis=)
        non_empty_frame_idxs = tf.expand_dims(non_empty_frame_idxs, axis=)

         x, non_empty_frame_idxs