# A High Schooler’s Solo Journey to get a Silver Medal.

Model Architecture
The model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.
The model architecture is as follows:
![](assets/bbbf76fe2c391df0b6d3351e.png)
Firstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected:
x_code = self.code_bert(code_tokens.flatten(0,1),
                                attention_mask=code_mask.flatten(0,1).to(device))\
                .last_hidden_state.view(*code_tokens.shape[:3],-1)[:,:,0,:]
Then, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.
Model Training
The models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.
def spearman_p_loss(pred, target_ranks, **kw):
    d2 = 0.
    n = 0.
    for p,t in zip(pred.flatten(1),target_ranks.flatten(1)):
        mask = t != -1
        rp = soft_rank(p[mask].unsqueeze(0), **kw) - 1.
        rt = t[mask].unsqueeze(0)
        d2 += (rt-rp).square().sum()
        n  += mask.sum()
    return 4. * d2/ ((n*(n-1)))
[https://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case](https://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case)
Fast Kendall Tau computation
I noticed the code for Kendall Tau was a bit slow, even the version that was based on bisect, so as a fun experiment I made it faster and implemented it without any loops (and batched):
def update(self,preds,index_gt):
        masks = index_gt != -1
        _preds=preds.clone()
        _index_gt = index_gt.clone()
        m = masks.flatten(1)
        _index_gt[~masks] = MAX_CELLS*2
        _preds[~masks] = np.inf
        pred_ranks = _preds.flatten(1).argsort(dim=1)
        ranks=torch.gather(_index_gt.flatten(1),1,pred_ranks)
        self.inversions += torch.triu((ranks.unsqueeze(1).permute(0,2,1)>ranks.unsqueeze(1))).sum()
        len_gt = masks.sum(dim=(1,2))
        self.total_2max +=((len_gt-1)*len_gt).sum()

    def compute(self):
        return  1. - 4. * self.inversions /  self.total_2max
Closing thoughts
Having entered the competition only one month prior to the deadline I ran out of computational power and time, therefore my model ended up being undertrained.
I really enjoyed this competition, see you guys on the next one!
[https://github.com/antorbus/Google-AI4Code](https://github.com/antorbus/Google-AI4Code)