# 41th Place Solution

41th Place Solution
First, thanks Kaggle and the competition hosts for a challenging competition Google AI4Code – Understand Code in Python Notebooks: Predict the  relationship between code and comments. This summarizes the effort of our  team and it was a great experience working with my teammates! This is a  writeup of our collective team effort.
Overview
Based on Khôi's initial idea, we modify a little bit to get more global context information. We sample 45 cells with shorter code cell's length (10 tokens). If the notebook has less than 45 cells, we dynamically adjust the max length of code cell based on the number of code cell.
Ensemble from 2 training setting (our setting and Khoi Nguyen setting - 20 code cell with 23 tokens each) also give us a boost on CV and LB. (we don't remember the exact number).
Model
: Weighted Average of some Point-wise models such as: Distilbert-base-uncased, Deberta-v3-base and Codebert-base
Data
: All training competition data, no external data. Cross Validation Split: 90/10
Hardware
: Two A6000-48GB. The largest model training for 3-4 days, the lighting model like distilbert is 1-2 days.
Some of the results on local CV are detailed in the photos below:
Initial experimental
Initial experimentals didn't beat Khoi's baseline:
[](https://postimg.cc/DSSp3ZXN)
Dynamic tokenizer based on number of code cells and FGM training help boost score on both Local CV and LB:
[](https://postimg.cc/JyJTt2Kc)
Final & best experimental
The best model on both Local CV and LB is Deberta-v3-base.
[](https://postimg.cc/JsHxdFwV)
Inference code:
[Inference Ensemble Final Submission](https://www.kaggle.com/code/dunglt/inference-pct-rank-model?scriptVersionId=101081766)
What worked
[High Impact] Idea
Dynamic tokenizer based on number of code cells: Local + 0.007 and LB + 0.005
Local tokenizer to boost training speed
[Medium Impact] Idea
MLM pretrained (15% probability random mask token) on all data except 1000 val example: Local and LB + 0.002 (not boost much and much time to pretrained)
FGM training: Boost Local and CV + 0.002 but 2x time training.
What didn’t work
Larger model: Deberta-v3-large, Deberta-v3-xlarge
AWP/Fast AWP training: not boost score on local CV and LB
EMA training
Custom head:
mean/max pooling
concat mean-max pooling
concat last 4 layers
Attention pooling
LSTM/ Bi-LSTM
Additional context (idea not have time to try):
We really want to test the ideas in the papers below, but we really don't have enough time and computational resources.
Deep Attentive Ranking Networks for Learning to Order Sentences -
[paper](https://arxiv.org/abs/2001.00056)
Efficient Relational Sentence Ordering Network    -
[paper](https://ieeexplore.ieee.org/document/9444816)
Neural Sentence Ordering Based on Constraint Graphs -
[paper](https://arxiv.org/abs/2101.11178)
Is Everything in Order? A Simple Way to Order Sentences -
[paper](assets/8d157226e4604db26e44b016.pdf)
Sentence Ordering and Coherence Modeling using Recurrent Neural Networks -
[paper](https://arxiv.org/abs/1611.02654)
Knowledge Distillation (KD)
Augmentation technique: Mixup, Random mask
Thanks and Acknowledgements:
We want to thanks the Kaggle staff team & Competition Host for coordinating this great competition!
Thanks for sharing the solution from all the teams, especially the top teams, we really learned a lot from their solutions.
Finally, thanks for the hard work of my teammates
[@dunglt](https://www.kaggle.com/dunglt)
Team Members:
Khanh Vu Duy
[KhanhVD](https://www.kaggle.com/duykhanh99)
Dung Le
[Dzung](https://www.kaggle.com/dunglt)