# 449th Position Solution for Google ASLFR Competition - Adversarial Regularization, Quantization Aware Training, and KD Code for Tensorflow! Hi Everyone! I'm just posting my solution here, in case it could help someone! I used a lot of techniques, but didn't have the computing power to push the boundaries required for a very high rank. Nevertheless, I learnt a lot of techniques, and am writing this solution here to help provide useful code! Competition Context The context of this solution is the Google - American Sign Language Fingerspelling Recognition Competition: [https://www.kaggle.com/competitions/asl-fingerspelling/overview](https://www.kaggle.com/competitions/asl-fingerspelling/overview) The data of this competition can be found here: [https://www.kaggle.com/competitions/asl-fingerspelling/data](https://www.kaggle.com/competitions/asl-fingerspelling/data) My Approach My approach was identical to most people, I referred to the really awesome notebook by Rohit Ingilela here: [https://www.kaggle.com/code/irohith/aslfr-ctc-based-on-prev-comp-1st-place](https://www.kaggle.com/code/irohith/aslfr-ctc-based-on-prev-comp-1st-place) I made several additions and changes, namely by adding Quantization Aware Training, AWP, and Knowledge distillation (not necessarily in the same model). I tested a lot of parameters as well and made a lot of models. My idea was to use Quantization to increase model size. I found that after 24 Million parameters, even though the model size was small enough, the performance became too poor to run within the 5 hours allocated for testing. I found that KD didn't improve my performance too much, so I didn't explore it a lot. I tried AWP only at the end, and even though it improved performance by around 0.003, I didn't get to test it a lot. Nevertheless, here is some code for each idea: Quantization: !pip install -q tensorflow-model-optimization tensorflow_model_optimization tfmot tensorflow.keras layers x = tfmot.quantization.keras.quantize_annotate_layer(layers.Dense(channels, activation = activation, kernel_regularizer = regularizer))(inputs) tflitemodel_base = TFLiteModel(model) tflitemodel_base(frames)[].shape keras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflitemodel_base) keras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT] keras_model_converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS] tflite_model = keras_model_converter.convert() (, ) f: f.write(tflite_model) Adversarial Regularization in Training ! pip install -q neural-structured-learning neural_structured_learning nsl adv_config = nsl.configs.make_adv_reg_config( multiplier = , adv_step_size = , adv_grad_norm = ) (): {: image, : label} train_dataset = train_dataset.(convert_to_dictionaries) val_dataset = val_dataset.(convert_to_dictionaries) strategy.scope(): model = nsl.keras.AdversarialRegularization( model, label_keys = [], adv_config = adv_config ) loss = CTCLoss optimizer = tfa.optimizers.RectifiedAdam(sma_threshold = ) optimizer = tfa.optimizers.Lookahead(optimizer, sync_period = ) model.(loss = loss, optimizer = optimizer) Knowledge Distillation (keras.Model): (): ().__init__() self.student = student self.teacher = teacher (): ().(optimizer=optimizer) self.loss = CTCLoss self.distiller_loss_fn = keras.losses.KLDivergence() self.alpha = alpha self.temp = temperature (): x, y = data is_batch_small = tf.math.less(tf.shape(x)[], train_batch_size) is_batch_small = tf.math.reduce_all(is_batch_small) (): tf.constant(), tf.constant() (): teacher_pred = self.teacher(x, training = ) tf.GradientTape() tape: student_pred = self.student(x, training = ) student_loss = self.loss(y, student_pred) distilled_loss = self.distiller_loss_fn( tf.nn.softmax(student_pred / self.temp), tf.nn.softmax(teacher_pred / self.temp) ) * (self.temp ** ) loss = self.alpha * student_loss + ( - self.alpha) * distilled_loss trainable_vars = self.student.trainable_variables gradients = tape.gradient(loss, trainable_vars) self.optimizer.apply_gradients((gradients, trainable_vars)) self.compiled_metrics.update_state(y, student_pred) student_loss, distilled_loss student_loss, distilled_loss = tf.cond(is_batch_small, pass_batch, normal) results = {m.name : m.result() m self.metrics} results.update( {: student_loss, : distilled_loss} ) results (): () x, y = data is_batch_small = tf.less(tf.shape(x)[], val_batch_size) is_batch_small = tf.reduce_all(is_batch_small) (): tf.constant(), tf.constant() (): y_pred = self.student(x, training = ) teacher_pred = self.teacher(x, training = ) student_loss = self.loss(y, y_pred) distilled_loss = self.distiller_loss_fn( tf.nn.softmax(y_pred / self.temp), tf.nn.softmax(teacher_pred / self.temp) ) * (self.temp ** ) self.compiled_metrics.update_state(y, y_pred) student_loss, distilled_loss student_loss, distilled_loss = tf.cond(is_batch_small, pass_batch, normal) results = {m.name : m.result() m self.metrics} results.update( {: student_loss, : distilled_loss} ) results While I didn't get a super high rank, I did learn a lot and used a plethora of techniques! I wrote most things from scratch, to learn things myself, and referred to other's work only to get a better understanding of what they did and improve my own code as a result. Validation Quite standard - kept a holdout set and computer Normalized Levenshtein distance on it after every epoch. No CV due to low computational resources. Improvements I am quite disappointed that the things I missed out were very trivial - such as not training for more than 300 epochs (I could only manage 100 to try enough models and variations out) and that keeping some missing hands frames was important. I also didn't use enough augmentations which was important, even though I used all the ones used by top submissions except cutout. I also read about Squeezeformer, but got demotivated because I saw high scoring public submissions without Squeezeformer :( To further improve, I'll correct all the points above. Final Note I am quite happy to have participated in this fun and awesome contest. I really learnt a lot, and experimented a lot. I am certain that what I did here will help me in future competitions, research and work. If you could give me any suggestions/feedback on my code above, please do so. It will help me improve and learn, and I will be very grateful for your help!