\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F58d2dcbe3cd1933a229f290470528f7f%2FTree%20diagrams%20(1).png?generation=1663797416217958&alt=media\")
Model Architecture
\nThe model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.
\nThe model architecture is as follows:
\n
Firstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected:
\nx_code = self.code_bert(code_tokens.flatten(0,1),\n attention_mask=code_mask.flatten(0,1).to(device))\\\n .last_hidden_state.view(*code_tokens.shape[:3],-1)[:,:,0,:]\n
\nThen, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.
Model Training
\nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.
\ndef spearman_p_loss(pred, target_ranks, **kw):\n d2 = 0.\n n = 0.\n for p,t in zip(pred.flatten(1),target_ranks.flatten(1)):\n mask = t != -1\n rp = soft_rank(p[mask].unsqueeze(0), **kw) - 1.\n rt = t[mask].unsqueeze(0)\n d2 += (rt-rp).square().sum()\n n += mask.sum()\n return 4. * d2/ ((n*(n-1)))\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case
\nFast Kendall Tau computation
\nI noticed the code for Kendall Tau was a bit slow, even the version that was based on bisect, so as a fun experiment I made it faster and implemented it without any loops (and batched):
\n def update(self,preds,index_gt):\n masks = index_gt != -1\n _preds=preds.clone()\n _index_gt = index_gt.clone()\n m = masks.flatten(1)\n _index_gt[~masks] = MAX_CELLS*2\n _preds[~masks] = np.inf\n pred_ranks = _preds.flatten(1).argsort(dim=1)\n ranks=torch.gather(_index_gt.flatten(1),1,pred_ranks)\n self.inversions += torch.triu((ranks.unsqueeze(1).permute(0,2,1)>ranks.unsqueeze(1))).sum()\n len_gt = masks.sum(dim=(1,2))\n self.total_2max +=((len_gt-1)*len_gt).sum()\n\n def compute(self):\n return 1. - 4. * self.inversions / self.total_2max\nClosing thoughts
\nHaving entered the competition only one month prior to the deadline I ran out of computational power and time, therefore my model ended up being undertrained.
\nI really enjoyed this competition, see you guys on the next one!
\nhttps://github.com/antorbus/Google-AI4Code
", "messages": [ { "id": "1949777", "postDate": "09/21/2022 22:01:11", "content": "Model Architecture
\nThe model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.
\nThe model architecture is as follows:
\n
Firstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected:
\nx_code = self.code_bert(code_tokens.flatten(0,1),\n attention_mask=code_mask.flatten(0,1).to(device))\\\n .last_hidden_state.view(*code_tokens.shape[:3],-1)[:,:,0,:]\n
\nThen, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.
Model Training
\nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.
\ndef spearman_p_loss(pred, target_ranks, **kw):\n d2 = 0.\n n = 0.\n for p,t in zip(pred.flatten(1),target_ranks.flatten(1)):\n mask = t != -1\n rp = soft_rank(p[mask].unsqueeze(0), **kw) - 1.\n rt = t[mask].unsqueeze(0)\n d2 += (rt-rp).square().sum()\n n += mask.sum()\n return 4. * d2/ ((n*(n-1)))\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case
\nFast Kendall Tau computation
\nI noticed the code for Kendall Tau was a bit slow, even the version that was based on bisect, so as a fun experiment I made it faster and implemented it without any loops (and batched):
\n def update(self,preds,index_gt):\n masks = index_gt != -1\n _preds=preds.clone()\n _index_gt = index_gt.clone()\n m = masks.flatten(1)\n _index_gt[~masks] = MAX_CELLS*2\n _preds[~masks] = np.inf\n pred_ranks = _preds.flatten(1).argsort(dim=1)\n ranks=torch.gather(_index_gt.flatten(1),1,pred_ranks)\n self.inversions += torch.triu((ranks.unsqueeze(1).permute(0,2,1)>ranks.unsqueeze(1))).sum()\n len_gt = masks.sum(dim=(1,2))\n self.total_2max +=((len_gt-1)*len_gt).sum()\n\n def compute(self):\n return 1. - 4. * self.inversions / self.total_2max\nClosing thoughts
\nHaving entered the competition only one month prior to the deadline I ran out of computational power and time, therefore my model ended up being undertrained.
\nI really enjoyed this competition, see you guys on the next one!
\nhttps://github.com/antorbus/Google-AI4Code
", "rawMarkdown": "*Model Architecture*\n\nThe model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.\n\nThe model architecture is as follows:\n.png?generation=1663797416217958&alt=media)\n\nFirstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected: \n```\nx_code = self.code_bert(code_tokens.flatten(0,1),\n attention_mask=code_mask.flatten(0,1).to(device))\\\n .last_hidden_state.view(*code_tokens.shape[:3],-1)[:,:,0,:]\n``` \nThen, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.\n\n*Model Training*\n\nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.\n```\ndef spearman_p_loss(pred, target_ranks, **kw):\n d2 = 0.\n n = 0.\n for p,t in zip(pred.flatten(1),target_ranks.flatten(1)):\n mask = t != -1\n rp = soft_rank(p[mask].unsqueeze(0), **kw) - 1.\n rt = t[mask].unsqueeze(0)\n d2 += (rt-rp).square().sum()\n n += mask.sum()\n return 4. * d2/ ((n*(n-1)))\n```\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case \n\n*Fast Kendall Tau computation*\n\nI noticed the code for Kendall Tau was a bit slow, even the version that was based on bisect, so as a fun experiment I made it faster and implemented it without any loops (and batched):\n```\n def update(self,preds,index_gt):\n masks = index_gt != -1\n _preds=preds.clone()\n _index_gt = index_gt.clone()\n m = masks.flatten(1)\n _index_gt[~masks] = MAX_CELLS*2\n _preds[~masks] = np.inf\n pred_ranks = _preds.flatten(1).argsort(dim=1)\n ranks=torch.gather(_index_gt.flatten(1),1,pred_ranks)\n self.inversions += torch.triu((ranks.unsqueeze(1).permute(0,2,1)>ranks.unsqueeze(1))).sum()\n len_gt = masks.sum(dim=(1,2))\n self.total_2max +=((len_gt-1)*len_gt).sum()\n\n def compute(self):\n return 1. - 4. * self.inversions / self.total_2max\n```\n\n*Closing thoughts*\n\nHaving entered the competition only one month prior to the deadline I ran out of computational power and time, therefore my model ended up being undertrained. \n\nI really enjoyed this competition, see you guys on the next one!\n\nhttps://github.com/antorbus/Google-AI4Code", "votes": null }, { "id": "1950774", "postDate": "09/22/2022 14:24:39", "content": "Great result! This is a wonderful achievement @antorino! Hearty congratulations and all the best!
", "rawMarkdown": "Great result! This is a wonderful achievement @antorino! Hearty congratulations and all the best!", "votes": null }, { "id": "1950847", "postDate": "09/22/2022 15:06:28", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "2026819", "postDate": "11/12/2022 11:05:48", "content": "Great solution!
\nHow much time did it take for training this model?
Around 4 days with 6x 3090.
", "rawMarkdown": "Around 4 days with 6x 3090.", "votes": null } ], "comments": [ { "id": 1950774, "author_name": "ravi20076", "author_url": "", "post_date": "09/22/2022 14:24:39", "content": "Great result! This is a wonderful achievement @antorino! Hearty congratulations and all the best!
", "votes": null, "replies": [ { "id": 1950847, "author_name": "antorino", "author_url": "", "post_date": "09/22/2022 15:06:28", "content": "Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 2026819, "author_name": "nguynthhi", "author_url": "", "post_date": "11/12/2022 11:05:48", "content": "Great solution!
\nHow much time did it take for training this model?
Around 4 days with 6x 3090.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1949777": "*Model Architecture*\n\nThe model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.\n\nThe model architecture is as follows:\n.png?generation=1663797416217958&alt=media)\n\nFirstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected: \n```\nx_code = self.code_bert(code_tokens.flatten(0,1),\n attention_mask=code_mask.flatten(0,1).to(device))\\\n .last_hidden_state.view(*code_tokens.shape[:3],-1)[:,:,0,:]\n``` \nThen, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.\n\n*Model Training*\n\nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.\n```\ndef spearman_p_loss(pred, target_ranks, **kw):\n d2 = 0.\n n = 0.\n for p,t in zip(pred.flatten(1),target_ranks.flatten(1)):\n mask = t != -1\n rp = soft_rank(p[mask].unsqueeze(0), **kw) - 1.\n rt = t[mask].unsqueeze(0)\n d2 += (rt-rp).square().sum()\n n += mask.sum()\n return 4. * d2/ ((n*(n-1)))\n```\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case \n\n*Fast Kendall Tau computation*\n\nI noticed the code for Kendall Tau was a bit slow, even the version that was based on bisect, so as a fun experiment I made it faster and implemented it without any loops (and batched):\n```\n def update(self,preds,index_gt):\n masks = index_gt != -1\n _preds=preds.clone()\n _index_gt = index_gt.clone()\n m = masks.flatten(1)\n _index_gt[~masks] = MAX_CELLS*2\n _preds[~masks] = np.inf\n pred_ranks = _preds.flatten(1).argsort(dim=1)\n ranks=torch.gather(_index_gt.flatten(1),1,pred_ranks)\n self.inversions += torch.triu((ranks.unsqueeze(1).permute(0,2,1)>ranks.unsqueeze(1))).sum()\n len_gt = masks.sum(dim=(1,2))\n self.total_2max +=((len_gt-1)*len_gt).sum()\n\n def compute(self):\n return 1. - 4. * self.inversions / self.total_2max\n```\n\n*Closing thoughts*\n\nHaving entered the competition only one month prior to the deadline I ran out of computational power and time, therefore my model ended up being undertrained. \n\nI really enjoyed this competition, see you guys on the next one!\n\nhttps://github.com/antorbus/Google-AI4Code", "1950774": "Great result! This is a wonderful achievement @antorino! Hearty congratulations and all the best!", "1950847": "Thanks a lot!", "2026819": "Great solution!\nHow much time did it take for training this model?", "2026973": "Around 4 days with 6x 3090." }, "source": "existing_discussion", "complete": true, "competition_id": 35887, "competition_slug": "AI4Code", "team_id": 8851936, "solution_url": "/competitions/AI4Code/writeups/andres-torrubia-bustos-a-high-schooler-s-solo-jour" }