# 9th Place Solution 

Overview
The final submission of us is a blend of three models with same architecture and different training data. Three models share the same pipline. We applied a two stage method for the competition.
Training a bert model (Unixcoder) to get sentence embedding for each cell;
Training a simple transformer with sentence embedding (or cell embedding) input.
![](assets/b0fecf7d6344fb5ca75e46b6.png)
Our Code
:
Submit and training code :
[https://www.kaggle.com/code/qiaoshiji/fork-of-ai4code-infer-22e56e/data?scriptVersionId=103071263](https://www.kaggle.com/code/qiaoshiji/fork-of-ai4code-infer-22e56e/data?scriptVersionId=103071263)
What Worked
Using unixcoder as first stage pretrained model. The inital score of different model is shown as following:
codebert 0.8226
graphcodebert 0.8482
Plbart 0.8492
codet5 0.8475
unixcoder 0.8594
Pretrain first stage model using three different types of the task as following:
Randomly selecting N successive cells and training rerank task;
Randomly selecting one cell and predicting next cell;
Randomly selecting N successive cells and choosing one cell as query, rest as answer.
Using longer sequence input in second stage, 512 code cells and 512 markdown cells for our final model
Multi task for second stage model:
Position as regression task
The position of markdown between which two code cells
The number of markdown cells between every two code cells
Using more data:
Raw data for model1
Raw data + ext data1 for model2
Raw data + ext data2 for model3
Ext data is from KGTorrent:
[https://github.com/collab-uniba/KGTorrent](https://github.com/collab-uniba/KGTorrent)
What Didn’t Work
Classify model as ensemble model
Making group by the number of cells and training models for each group
Important Citations
KGTorrent
Github:
[https://github.com/collab-uniba/KGTorrent](https://github.com/collab-uniba/KGTorrent)
Paper:
[https://arxiv.org/abs/2103.10558](https://arxiv.org/abs/2103.10558)
UniXcoder
Github:
[https://github.com/microsoft/CodeBERT/tree/master/UniXcoder#1-code-and-nl-embeddings](https://github.com/microsoft/CodeBERT/tree/master/UniXcoder#1-code-and-nl-embeddings)
Thanks and Acknowledgements
Many thanks to organizer for hosting such a great competition.
Team Members
:
[@guoday](https://www.kaggle.com/guoday)
[@mahluo](https://www.kaggle.com/mahluo)