# Public LB 14th Solution

First and foremost, I want to thank Kaggle and the Kaggle community for providing us with the chance to learn via experience and collaboration. This was my first time competing in a Kaggle competition, and I'm quite pleased because I learned a lot about various model Archs (such as transformers, Pointer Networks, and so on) and NLP in general.
We have three approaches for ranking problems (correct me if I'm wrong!): point-wise, pair-wise, and list-wise. The absence of global context is a concern with point-wise, however, some participants attempted to fix it by attaching a subset of code cells to each markdown cell. The pair-wise strategy appears to be highly strong, but it comes at the cost of a long run-time. Finally, list-wise attempts to rank cells using a mix of global and local information.
I picked the list-wise method since it was faster than the pair-wise method and seemed more natural to me.
The following image depicts the model architecture:
![](assets/1bde337176ec723529a9b0a8.png)
Using
sentence-transformers/all-mpnet-base-v2
, compute cell features separately: I choose the first and last 64 words of each cell's text after preprocessing it to get a feel of how the cell begins and finishes. This is then sent into the sentence transformer model, which produces a 768d feature vector.
I then append the following 8 features to each vector:
is_code: 1 if cell is code else 0
is_markdown: 1 if cell is markdown else 0
code_rank: rank of the code cell if cell is code else 0
toc_rank: rank of the markdown cell in the table of contents if cell is markdown and ToC exists else 0
numberd_rank: rank of the md cell if it has numbered title (like 1.1, 1.2) else 0
has_heading_1: 1 if markdown cell has heading 1 else 0
has_heading_2: 1 if markdown cell has heading 2 else 0
has_heading_3: 1 if markdown cell has heading 3 else 0
Then I use an 8-head, 10-layer encoder-only transformer to compute the final features for each cell.
Finally, I have two outputs for the code cells and four outputs for the markdown cells:
rank: this is the ranking output that I use in the inference. I apply a sigmoid on it.
code_rank: rank of the code cell among other code cells.
markdown_rank: rank of the markdown cell among other md cells
toc_rank: rank of the md cell in the toc
numbered_rank: rank of the md cell among other numbered md cells
Loss function:
L1Loss is used, however instead of using it on the [0, 1] range, I use it on the [0, #cells] range. To do this, I first apply sigmoid, then multiply by the number of cells and use the result as the predicted output. This is due to the fact that longer notebooks contribute more to the scoring function, thus we should pay more attention to their flaws.
training time:
Training the notebook transformer is pretty fast. One epoch of training with a batch size of 10 notebooks and 17k steps takes less than 30 mins on a P100.
Inference time:
The entire inference procedure, including my preprocessing, takes around two hours. However, the model itself takes less than an hour to complete.
Preprocessing
Fix encodings
Fix HTML in markdown cells
Detect ToC rankings
Detect numbered rankings
Detect language
Translation of non-English languages using MarianMT
Challenges
My main challenge was that I couldn't train my entire model end2end on all the notebooks since my GPU (Colab P100 or V100) couldn't handle notebooks larger than 20 cells 😄
External datasets
I crawled around 50k additional notebooks from Kaggle, but it seems they didn't overlap much with the Private test set cause it only improved my score by 0.01!