{ "id": 368729, "title": "18th place solution", "url": "/competitions/AI4Code/discussion/368729", "author_name": "antorino", "post_date": "2022-11-27T11:31:03.032000", "votes": 8, "comment_count": 0, "views": 0, "content": "

Overview

\n

https://github.com/antorbus/Google-AI4Code

\n

The model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.

\n

Model Architecture
\n\"\"

\n

Firstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected. Then, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.

\n

Training
\nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.

\n

What Worked

\n\n

What Didn’t Work

\n\n

Further Improvements
\nAfter post-competition testing of my approximate Kendall Tau Loss I have tuned it so that it maps much more closely to the Kendall Tau. The two graphs below show a comparison of the actual Kendall Tau against both Loss functions. Each point on the graph represents a synthetically constructed sample of two lists similar in size to the competition's, one ordered and the other with a varying percentage of random permutations.
\n\"\"
\n\"\"

\n

References
\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case

\n

Thanks and Acknowledgements
\nThanks to Kaggle, Google and X for organizing this competition.

\n

Team Members
\nAndres Torrubia Bustos

", "messages": [ { "id": 2045487, "postDate": "2022-11-27T11:31:03.033Z", "content": "

Overview

\n

https://github.com/antorbus/Google-AI4Code

\n

The model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.

\n

Model Architecture
\n\"\"

\n

Firstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected. Then, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.

\n

Training
\nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable.

\n

What Worked

\n\n

What Didn’t Work

\n\n

Further Improvements
\nAfter post-competition testing of my approximate Kendall Tau Loss I have tuned it so that it maps much more closely to the Kendall Tau. The two graphs below show a comparison of the actual Kendall Tau against both Loss functions. Each point on the graph represents a synthetically constructed sample of two lists similar in size to the competition's, one ordered and the other with a varying percentage of random permutations.
\n\"\"
\n\"\"

\n

References
\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case

\n

Thanks and Acknowledgements
\nThanks to Kaggle, Google and X for organizing this competition.

\n

Team Members
\nAndres Torrubia Bustos

", "rawMarkdown": "**Overview**\n\nhttps://github.com/antorbus/Google-AI4Code\n\nThe model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.\n\n**Model Architecture**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F0b05c90da80f650d838a0fa5ee402dea%2Fmmm.png?generation=1669479397375090&alt=media)\n\nFirstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected. Then, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.\n\n**Training** \nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable. \n\n**What Worked**\n- The approximate Kendall Tau Loss over MSE/MAE\n- Increasing model and input size\n- Pre-Tokenizing for faster testing\n- Ensembling (marginal improvements)\n\n**What Didn’t Work**\n- Making the model smaller by removing transformer layers\n- Attempting to augment the training data by splitting large cells and merging smaller consecutive ones\n\n**Further Improvements** \nAfter post-competition testing of my approximate Kendall Tau Loss I have tuned it so that it maps much more closely to the Kendall Tau. The two graphs below show a comparison of the actual Kendall Tau against both Loss functions. Each point on the graph represents a synthetically constructed sample of two lists similar in size to the competition's, one ordered and the other with a varying percentage of random permutations.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F9affcb62ea2b48e54cff0c74f290f6ea%2Fapprox.png?generation=1669480496182649&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F0be005ce55911d522099074a154c5ac7%2Fdownload.png?generation=1669548555418840&alt=media)\n\n**References**\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case \n\n**Thanks and Acknowledgements**\nThanks to Kaggle, Google and X for organizing this competition.\n\n**Team Members**\nAndres Torrubia Bustos", "votes": 8 } ], "comments": [], "raw_markdown_by_id": { "2045487": "**Overview**\n\nhttps://github.com/antorbus/Google-AI4Code\n\nThe model is based on treating the problem as a regression task, where the known relative code cell positions are assigned a value between 0 and 1 and the predicted markdown cell positions can take a value between -0.2 and 1.2, this way I impose an inductive bias on the code cells and allow the markdown cells to be place anywhere before and after the code cells.\n\n**Model Architecture**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F0b05c90da80f650d838a0fa5ee402dea%2Fmmm.png?generation=1669479397375090&alt=media)\n\nFirstly, the tokenized (CodeBERT tokenizer) code and markdown cells are both fed separately into CodeBERT. Only the first latent code and markdown output vectors from CodeBert are selected. Then, positional encodings are added to the latent code vector, this is the input to the encoder. The output from the encoder is then input (as context) alongside the latent markdown vector into the decoder which can attend to the interactions between all cells, code and markdown. The output of the decoder is finally fed into the regressor which outputs the markdown cell position predictions.\n\n**Training** \nThe models were trained only with the dataset provided by the competition. The only notable difference in training with regards to other solutions is that instead of using MSE or MAE as a loss function I tried to approximate Kendall Tau using a loss function inspired by Spearman Rho which internally uses softrank, making it differentiable. \n\n**What Worked**\n- The approximate Kendall Tau Loss over MSE/MAE\n- Increasing model and input size\n- Pre-Tokenizing for faster testing\n- Ensembling (marginal improvements)\n\n**What Didn’t Work**\n- Making the model smaller by removing transformer layers\n- Attempting to augment the training data by splitting large cells and merging smaller consecutive ones\n\n**Further Improvements** \nAfter post-competition testing of my approximate Kendall Tau Loss I have tuned it so that it maps much more closely to the Kendall Tau. The two graphs below show a comparison of the actual Kendall Tau against both Loss functions. Each point on the graph represents a synthetically constructed sample of two lists similar in size to the competition's, one ordered and the other with a varying percentage of random permutations.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F9affcb62ea2b48e54cff0c74f290f6ea%2Fapprox.png?generation=1669480496182649&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8807553%2F0be005ce55911d522099074a154c5ac7%2Fdownload.png?generation=1669548555418840&alt=media)\n\n**References**\nhttps://en.wikipedia.org/wiki/Rank_correlation#Spearman%E2%80%99s_%CF%81_as_a_particular_case \n\n**Thanks and Acknowledgements**\nThanks to Kaggle, Google and X for organizing this competition.\n\n**Team Members**\nAndres Torrubia Bustos" } }