{ "id": 367600, "title": "3rd Place Solution", "url": "/competitions/AI4Code/discussion/367600", "author_name": "yufuin", "post_date": "2022-11-21T11:56:07.374000", "votes": 19, "comment_count": 0, "views": 0, "content": "

Overview

\n

Main Features
\n1) The cosine-similarity based classification model to predict which consecutive two code cells the given markdown cell should be placed between.
\n2) Postprocessing to bridge the gap between the predicted probability distribution and the task metric.
\n3) Additional notebooks crawled from Kaggle as external training dataset.
\nMinor Features
\n4) Additional pretraining of MLM
\n5) Markdown sorting model for when multiple markdowns are placed between the same code pairs. (“Markdown sorting model” in the figure below.)
\n6) Some techniques to speed up the inference.

\n

Inference code: https://www.kaggle.com/code/yufuin/ai4code-public-3rd-solution-inference/notebook?scriptVersionId=103005501
\nModel: https://www.kaggle.com/models/yufuin/ai4code-parallel-bert

\n

Here is the overview of my model.
\n\"\"

\n

What Worked

\n
    \n

  1. Main model: classification model to predict which consecutive two code cells the given markdown cell should be placed between.

    \n
    2. \n

  2. Postprocessing

    \n
    3. \n

  3. External dataset

    \n
    4. \n

  4. MLM

    \n
    5. \n

  5. Markdown sorting model

    \n\n
        (torch.nn.Module):\n         ():\n            h1 = .multi_head_att(mk_attended, mk_attended, mk_attended, mask)[]\n            h2 = mk_attended + .linear1(h1)\n             .linear2(h2).squeeze(-)\n
    \n
    6. \n

  6. Speed up the inference

    \n
    7. \n
\n

What Didn’t Work

\n\n

Additional Context

\n\n

Important Citations

\n\n

Thanks and Acknowledgements

\n

I appreciate the task organizer for hosting this exciting competition.
\nI also thank other competitors for insightful discussions.

", "messages": [ { "id": 2038469, "postDate": "2022-11-21T11:56:07.373Z", "content": "

Overview

\n

Main Features
\n1) The cosine-similarity based classification model to predict which consecutive two code cells the given markdown cell should be placed between.
\n2) Postprocessing to bridge the gap between the predicted probability distribution and the task metric.
\n3) Additional notebooks crawled from Kaggle as external training dataset.
\nMinor Features
\n4) Additional pretraining of MLM
\n5) Markdown sorting model for when multiple markdowns are placed between the same code pairs. (“Markdown sorting model” in the figure below.)
\n6) Some techniques to speed up the inference.

\n

Inference code: https://www.kaggle.com/code/yufuin/ai4code-public-3rd-solution-inference/notebook?scriptVersionId=103005501
\nModel: https://www.kaggle.com/models/yufuin/ai4code-parallel-bert

\n

Here is the overview of my model.
\n\"\"

\n

What Worked

\n
    \n

  1. Main model: classification model to predict which consecutive two code cells the given markdown cell should be placed between.

    \n
    2. \n

  2. Postprocessing

    \n
    3. \n

  3. External dataset

    \n
    4. \n

  4. MLM

    \n
    5. \n

  5. Markdown sorting model

    \n\n
        (torch.nn.Module):\n         ():\n            h1 = .multi_head_att(mk_attended, mk_attended, mk_attended, mask)[]\n            h2 = mk_attended + .linear1(h1)\n             .linear2(h2).squeeze(-)\n
    \n
    6. \n

  6. Speed up the inference

    \n
    7. \n
\n

What Didn’t Work

\n\n

Additional Context

\n\n

Important Citations

\n\n

Thanks and Acknowledgements

\n

I appreciate the task organizer for hosting this exciting competition.
\nI also thank other competitors for insightful discussions.

", "rawMarkdown": "# Overview\n**Main Features**\n1) The cosine-similarity based classification model to predict which consecutive two code cells the given markdown cell should be placed between.\n2) Postprocessing to bridge the gap between the predicted probability distribution and the task metric.\n3) Additional notebooks crawled from Kaggle as external training dataset.\n**Minor Features**\n4) Additional pretraining of MLM\n5) Markdown sorting model for when multiple markdowns are placed between the same code pairs. (“Markdown sorting model” in the figure below.)\n6) Some techniques to speed up the inference.\n\nInference code: https://www.kaggle.com/code/yufuin/ai4code-public-3rd-solution-inference/notebook?scriptVersionId=103005501\nModel: https://www.kaggle.com/models/yufuin/ai4code-parallel-bert\n\nHere is the overview of my model.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2430820%2F4793ecab00bc55b7a0f90cec5b239341%2FAI4Code_3rd_sorlution_model.PNG?generation=1669030267549699&alt=media)\n\n# What Worked\n1. Main model: classification model to predict which consecutive two code cells the given markdown cell should be placed between. \n * Result: 88.5% on dev (Single deberta-v3-large model, w/o {MLM, postprocess, external dataset})\n * This model builds representations of each consecutive two code cells and each markdown cells, then predicts the probability distribution from cosine similarities between a code and a markdown representations. \n * Loss: CrossEntropyLoss\n * The weights of BERT to encode markdowns and codes are shared.\n * The most important part of the model architecture is LSTM at code encoding. Without LSTM, I only got the dev score of 86~87%.\n * I trained three checkpoints for the best submission.\n * One is initialized by deberta-v3-large, and the other two are initialized by distilroberta.\n * All of them used the same train-dev split.\n * The ensemble is done by weighted averaging the output probability distributions. The ensemble weight is tuned by the grid search on the dev set.\n * I used checkpoint ensembling by EMA for deberta-v3-large, and by the strict average for distilroberta.\n * In fact, the strict average always got better results than EMA even for deberta-v3-large.\n * However, the time to converge is considerably longer than EMA (2~3 times longer), and unfortunately, I found a problem at training a week before the end of the competition and I couldn’t finish re-training for deberta-v3-large with the strict average by the deadline, so I submitted the EMA version for deberta-v3-large instead.\n * To prevent out-of-memory during training, I sampled markdown and code cells.\n * The larger the sampling size, the better the dev/test score.\n * The maximum sequence length of subwords for each single cell is 128.\n * Since a code-pair is constructed from two cells, it could have 256 subwords + special tokens.\n\n1. Postprocessing\n * Result: +0.8～1% on dev\n * The task score of this competition, Kendall’s tau, has a property that the decrement from the ideal score (=100%) is proportional to the total deviations of the predicted markdown positions from the correct positions (if we ignore the decrement caused by the incorrect order of markdown cells).\n * For more information: https://www.kaggle.com/code/yufuin/ai4code-linearity-of-kendall-s-tau-english\n * My model outputs the probability distribution of the position of the markdown. According to the predicted probability, we can estimate the expected deviation for each position.\n * Choosing the minimum expected deviation position is a much better strategy to minimize the decrement of the task score than naively choosing by argmax over the probability.\n\n1. External dataset\n * Result: >+0.42% on Private LB.\n * I crawled kaggle’s public notebooks myself based on the KGTorrent code\n * KGTorrent: https://github.com/collab-uniba/KGTorrent\n * My best submission used only external dataset for training to prevent duplication with original training dataset.\n * For more information: https://www.kaggle.com/competitions/AI4Code/discussion/343658\n\n1. MLM\n * Result: ~+1% on dev (for deberta-v3-large)\n * I trained for 5 days for distilroberta, and 40 days 😂😂😂 for deberta-v3-large by 3090.\n * For deberta-v3-large, the longer MLM training, the better the dev/test scores.\n * I used only the original competition’s training dataset for MLM training.\n\n1. Markdown sorting model\n * Result: +0.15% on Public LB (for deberta-v3-large)\n * My main model cannot resolve the order of markdown cells when they are placed between the same code-pair.\n * However, I could improve a maximum +0.6% on dev by perfect ordering than putting randomly. The markdown sorting model addresses this.\n * the actual boost I made is only a quoter of it, though.\n * The pseudo code of the model architecture is as follows\n ```python3\n class MKSortModel(torch.nn.Module):\n def forward(self, mk_attended, mask):\n h1 = self.multi_head_att(mk_attended, mk_attended, mk_attended, mask)[0]\n h2 = mk_attended + self.linear1(h1)\n return self.linear2(h2).squeeze(-1)\n ```\n * Input: a set of markdown vectors from the attention layer of the main model.\n * Here, the set is constructed of markdown cells only that are predicted to be placed between the same code-pair (i.e., not the all markdown cells in the notebook).\n * Output: the percent ranks among the input set.\n * Loss: BCEWithLogitsLoss.\n\n1. Speed up the inference\n * The model weights are converted into FP16.\n * I sorted the input sequences by the sequence length and built minibatches before inputting to BERT. I restored the order after pooling.\n\n# What Didn’t Work\n* Distorting the predicted probabilities at postprocessing stage\n * What I did: to maximize the task score with postprocessing, using the different temperature parameter of softmax other than the one used at training (i.e., tuning the temperature parameter after training w.r.t. the task score).\n * At the end of the day, I found using the same temperature as the one used at training is the best option.\n\n# Additional Context\n* Most experiments were done on 3090, except for the models for the final submission (they are trained on A6000).\n* The most appealing point of this competition for me is that the sources of the competition dataset are Kaggle notebooks, which are under Apache License 2.0. It’s quite attractive that learned knowledge and techniques through this competition, and even datasets and models are basically free to use by rebuilding the dataset ourselves (I currently have no plan to use, though). This is a big reason why I decided to crawl the dataset myself.\n* The two-stage competition is mentally very good. We have almost no concern about the leakage. Otherwise I couldn’t run 40 days training in my living room.\n\n# Important Citations\n* L1 loss idea\n * Disucission: https://www.kaggle.com/competitions/AI4Code/discussion/325272\n * Linearity of Kendall's Tau: https://www.kaggle.com/code/yufuin/ai4code-linearity-of-kendall-s-tau-english\n* KGTorrent\n * Discussion: https://www.kaggle.com/competitions/AI4Code/discussion/326520\n * Github: https://github.com/collab-uniba/KGTorrent\n * Paper: https://arxiv.org/abs/2103.10558\n\n# Thanks and Acknowledgements\nI appreciate the task organizer for hosting this exciting competition.\nI also thank other competitors for insightful discussions.", "votes": 19 } ], "comments": [], "raw_markdown_by_id": { "2038469": "# Overview\n**Main Features**\n1) The cosine-similarity based classification model to predict which consecutive two code cells the given markdown cell should be placed between.\n2) Postprocessing to bridge the gap between the predicted probability distribution and the task metric.\n3) Additional notebooks crawled from Kaggle as external training dataset.\n**Minor Features**\n4) Additional pretraining of MLM\n5) Markdown sorting model for when multiple markdowns are placed between the same code pairs. (“Markdown sorting model” in the figure below.)\n6) Some techniques to speed up the inference.\n\nInference code: https://www.kaggle.com/code/yufuin/ai4code-public-3rd-solution-inference/notebook?scriptVersionId=103005501\nModel: https://www.kaggle.com/models/yufuin/ai4code-parallel-bert\n\nHere is the overview of my model.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2430820%2F4793ecab00bc55b7a0f90cec5b239341%2FAI4Code_3rd_sorlution_model.PNG?generation=1669030267549699&alt=media)\n\n# What Worked\n1. Main model: classification model to predict which consecutive two code cells the given markdown cell should be placed between. \n * Result: 88.5% on dev (Single deberta-v3-large model, w/o {MLM, postprocess, external dataset})\n * This model builds representations of each consecutive two code cells and each markdown cells, then predicts the probability distribution from cosine similarities between a code and a markdown representations. \n * Loss: CrossEntropyLoss\n * The weights of BERT to encode markdowns and codes are shared.\n * The most important part of the model architecture is LSTM at code encoding. Without LSTM, I only got the dev score of 86~87%.\n * I trained three checkpoints for the best submission.\n * One is initialized by deberta-v3-large, and the other two are initialized by distilroberta.\n * All of them used the same train-dev split.\n * The ensemble is done by weighted averaging the output probability distributions. The ensemble weight is tuned by the grid search on the dev set.\n * I used checkpoint ensembling by EMA for deberta-v3-large, and by the strict average for distilroberta.\n * In fact, the strict average always got better results than EMA even for deberta-v3-large.\n * However, the time to converge is considerably longer than EMA (2~3 times longer), and unfortunately, I found a problem at training a week before the end of the competition and I couldn’t finish re-training for deberta-v3-large with the strict average by the deadline, so I submitted the EMA version for deberta-v3-large instead.\n * To prevent out-of-memory during training, I sampled markdown and code cells.\n * The larger the sampling size, the better the dev/test score.\n * The maximum sequence length of subwords for each single cell is 128.\n * Since a code-pair is constructed from two cells, it could have 256 subwords + special tokens.\n\n1. Postprocessing\n * Result: +0.8～1% on dev\n * The task score of this competition, Kendall’s tau, has a property that the decrement from the ideal score (=100%) is proportional to the total deviations of the predicted markdown positions from the correct positions (if we ignore the decrement caused by the incorrect order of markdown cells).\n * For more information: https://www.kaggle.com/code/yufuin/ai4code-linearity-of-kendall-s-tau-english\n * My model outputs the probability distribution of the position of the markdown. According to the predicted probability, we can estimate the expected deviation for each position.\n * Choosing the minimum expected deviation position is a much better strategy to minimize the decrement of the task score than naively choosing by argmax over the probability.\n\n1. External dataset\n * Result: >+0.42% on Private LB.\n * I crawled kaggle’s public notebooks myself based on the KGTorrent code\n * KGTorrent: https://github.com/collab-uniba/KGTorrent\n * My best submission used only external dataset for training to prevent duplication with original training dataset.\n * For more information: https://www.kaggle.com/competitions/AI4Code/discussion/343658\n\n1. MLM\n * Result: ~+1% on dev (for deberta-v3-large)\n * I trained for 5 days for distilroberta, and 40 days 😂😂😂 for deberta-v3-large by 3090.\n * For deberta-v3-large, the longer MLM training, the better the dev/test scores.\n * I used only the original competition’s training dataset for MLM training.\n\n1. Markdown sorting model\n * Result: +0.15% on Public LB (for deberta-v3-large)\n * My main model cannot resolve the order of markdown cells when they are placed between the same code-pair.\n * However, I could improve a maximum +0.6% on dev by perfect ordering than putting randomly. The markdown sorting model addresses this.\n * the actual boost I made is only a quoter of it, though.\n * The pseudo code of the model architecture is as follows\n ```python3\n class MKSortModel(torch.nn.Module):\n def forward(self, mk_attended, mask):\n h1 = self.multi_head_att(mk_attended, mk_attended, mk_attended, mask)[0]\n h2 = mk_attended + self.linear1(h1)\n return self.linear2(h2).squeeze(-1)\n ```\n * Input: a set of markdown vectors from the attention layer of the main model.\n * Here, the set is constructed of markdown cells only that are predicted to be placed between the same code-pair (i.e., not the all markdown cells in the notebook).\n * Output: the percent ranks among the input set.\n * Loss: BCEWithLogitsLoss.\n\n1. Speed up the inference\n * The model weights are converted into FP16.\n * I sorted the input sequences by the sequence length and built minibatches before inputting to BERT. I restored the order after pooling.\n\n# What Didn’t Work\n* Distorting the predicted probabilities at postprocessing stage\n * What I did: to maximize the task score with postprocessing, using the different temperature parameter of softmax other than the one used at training (i.e., tuning the temperature parameter after training w.r.t. the task score).\n * At the end of the day, I found using the same temperature as the one used at training is the best option.\n\n# Additional Context\n* Most experiments were done on 3090, except for the models for the final submission (they are trained on A6000).\n* The most appealing point of this competition for me is that the sources of the competition dataset are Kaggle notebooks, which are under Apache License 2.0. It’s quite attractive that learned knowledge and techniques through this competition, and even datasets and models are basically free to use by rebuilding the dataset ourselves (I currently have no plan to use, though). This is a big reason why I decided to crawl the dataset myself.\n* The two-stage competition is mentally very good. We have almost no concern about the leakage. Otherwise I couldn’t run 40 days training in my living room.\n\n# Important Citations\n* L1 loss idea\n * Disucission: https://www.kaggle.com/competitions/AI4Code/discussion/325272\n * Linearity of Kendall's Tau: https://www.kaggle.com/code/yufuin/ai4code-linearity-of-kendall-s-tau-english\n* KGTorrent\n * Discussion: https://www.kaggle.com/competitions/AI4Code/discussion/326520\n * Github: https://github.com/collab-uniba/KGTorrent\n * Paper: https://arxiv.org/abs/2103.10558\n\n# Thanks and Acknowledgements\nI appreciate the task organizer for hosting this exciting competition.\nI also thank other competitors for insightful discussions." } }