# 28th Place Solution

Overview
Based on the baselines' ideas, we propose a two-phase inference algorithm to predict the exact position of the markdown cells.
Phase 1: We first use rank-based and pairwise-based methods to predict the markdown cells' approximate position. Two rank-based models and one pairwise-based model are used. We use softmax with temperature for the ensemble.
Phase 2: Once we obtain the approximate position of the markdown cells, we move to phase 2 to predict the exact position of the cells. We use a rank-based model (window size is 20, detecting around the position phase 1 predicted) and two pairwise-based models for prediction. Finally, another softmax is applied for the ensemble.
Code
[https://www.kaggle.com/code/tide525/ai4code-final-ensemble/notebook?scriptVersionId=102987081](https://www.kaggle.com/code/tide525/ai4code-final-ensemble/notebook?scriptVersionId=102987081)
Solution Pipeline Visualization
![](assets/7541b2a744f02a7c9ea39e1d.jpeg)
Model
rank:
In the rank-based model, we estimate the relative position of a markdown cell by feeding the markdown cell and all the code cells sampled in the correct order. The model is trained on regression of the normalized rank of markdown cells corresponding to that of code cells. We align the markdowns with the model by inserting each markdown at the position of its predicted rank.
pairwise
The pairwise-based model predicts the similarity between a markdown cell and a code cell. We compute the similarity to a markdown cell for each code cell and insert the markdown cell next to the code cell with the highest similarity.
For training the model, we adopt contrastive learning with in-batch negatives, which is used in DPR. Specifically, pairs of a markdown cell and a code cell next to each other are given as positive examples, and all the other combinations in a batch are considered negative examples. We use contrastive loss as an objective.
What Worked
Idea
Data augmentation
The original baseline sample 20 code cells in total for more global information. If there are 400 cells in total, the baseline will sample cells whose idx are 0, 20, 40, … 400.
We propose a data augmentation method, which collects cells {0, 20, 40, …}, {1, 21, 41, …}, and {2, 22, 42, …} separately, making the data triple.
cv: 0.8436 -> 0.8506
Idea
Two-phase prediction: from wide to narrow
In some notebooks with many cells, it is hard to estimate accurate position. Then, we take a two-phase prediction. First, all cells are inputted and predict the approximate position of the markdown cells is. Second,  we search for a neighborhood of the first-phase prediction. "Neighborhood" is determined based on experiments as a window size. We experiment from 5 to 40 and found that 20 is the best for the window size. The neighborhood searching model is independently trained. For this training, we split the train data into subsets; each subset has window-size cells.
Idea
Extra data(KGTorrent)
We first train our model for one epoch using KGTorent and AI4Code data, then only use AI4Code data for 3 epochs.
cv += 0.002-0.004
What Did not Work
Idea
Changing input code length dynamically
Each notebook has a different number of cells though the max input length is always 512. Then, we hypothesized that it would be better to change the input length of each cell based on the number of cells. For notebooks with many cells, the input length for each cell should be short; for those with a few cells, the input length should be longer. However, stable input length models perform better. One possible cause is that the model could not cope with dynamic changes in the input length.
Idea
Cubert
Cubert (
[https://github.com/zhihu/cuBERT](https://github.com/zhihu/cuBERT)
) has a longer max_length (2048) and larger size (BERT-large configuration), which are attractive to this task. However, it is too heavy for T4 GPU, so we abandoned this model.
Idea
Data preprocessing
We tried (1) excluding all notebooks with over 200 cells (which are difficult to predict) cv: 0.8436 -> 0.8406 (2) transforming markdown to pure text (by
[https://gist.github.com/lorey/eb15a7f3338f959a78cc3661fbc255fe](https://gist.github.com/lorey/eb15a7f3338f959a78cc3661fbc255fe)
) cv: 0.8436 -> 0.8431
Idea
Translating multilingual to English
There are a lot of other languages in the dataset, such as Russian and Korean; we thought preprocessing all foreign languages into English could make a better dataset. However, statistics-based machine translation models perform poorly, and generation-based models take long in beam search. Moreover, the biggest problem is that it is challenging to deal with languages mixing. Therefore, we did not apply machine translation to the dataset.
Idea
Extra data(GitHub repositories)
We collected Jupyter Notebooks from GitHub, but they are not high quality, leading to poor results.
Additional Context
We used KGTorrent for an additional dataset. Its dataset has 248,761 Jupyter Notebooks written in Python. This dataset covers the period between November 2015 and October 2020. ( KGTorrent is available at
[https://zenodo.org/record/4468523#.Y39guXbP2Hs](https://zenodo.org/record/4468523#.Y39guXbP2Hs)
)                                           Paper:
[https://arxiv.org/abs/2103.10558](https://arxiv.org/abs/2103.10558)
Github:
[https://github.com/collab-uniba/KGTorrent](https://github.com/collab-uniba/KGTorrent)
Additional data were also collected from GitHub repositories using GitHub REST API. We first cloned GitHub repositories, including the Jupyter Notebook file, in order of GitHub stars. Then, we filtered out huge and short files and deduplicated files. The final dataset has about 40000 Jupyter Notebooks.
These additional data were formatted into the contest data format. To make it identical to the original data, we assigned a unique 8-digit hexadecimal number to cell_id and a unique 14-digit hexadecimal number to id.
Important Citations
The following baselines help us a lot.
[https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells](https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells)
[https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-inference](https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-inference)
Thanks and Acknowledgements
Grateful to the Kaggle team & competition host for this exciting NLP competition!
And thanks for the hard work of my teammates!
Team Members
This team represents Kawahara Lab, Waseda University. We focus on researching Japanese NLP. Please check our homepage if you are interested:
[https://nlp-waseda.jp/en/](https://nlp-waseda.jp/en/)
[@conan1024](https://www.kaggle.com/conan1024)
[@tide525](https://www.kaggle.com/tide525)
[@hirofumishimizu](https://www.kaggle.com/hirofumishimizu)
[@mkmkmk1](https://www.kaggle.com/mkmkmk1)
[@emurata](https://www.kaggle.com/emurata)