# 10th place solution

Overview
First, I would like to thank Kaggle and Google and X for hosting this interesting competition.
In my approach, all the markdown and code cells in a notebook are tokenized and concatenated to form a long input sequence, with all the markdown cells go first followed by all the code cells, and they are separated with a [SEP] token. There are also special separator tokens added for markdown (MKDS) and code (CDS) respectively. Once the input goes through the transformer, tokens of each cell is aggregated by meanpooling and a linear layer to map to a single value. During training, the output of each cell is penalized with a pairwise ranking loss which considers all the combinations of cells within a notebook. For inference we can simply sort the output of each cell for correct order.
![](assets/4bb5533f4d294356fa05ffef.png)
Source Code
[https://github.com/GuanshuoXu/Google-AI4Code-Understand-Code-in-Python-Notebooks](https://github.com/GuanshuoXu/Google-AI4Code-Understand-Code-in-Python-Notebooks)
What Worked
torch.nn.MarginRankingLoss works better than BCE and MSE loss
deberta-v3-large is the best transformer in Huggingface for this task
longer training sequence (2048) and even longer inference (4096 for validation and 4352 for submission).
compress markdown cells with
[https://gist.github.com/lorey/eb15a7f3338f959a78cc3661fbc255fe](https://gist.github.com/lorey/eb15a7f3338f959a78cc3661fbc255fe)
compress code cells with
[https://www.kaggle.com/code/haithamaliryan/ai4code-extract-all-functions-variables-names/notebook](https://www.kaggle.com/code/haithamaliryan/ai4code-extract-all-functions-variables-names/notebook)
What didn't Work
seq-to-seq model like T5
multilingual models