This is a new domain to me. I couldn’t grasp the details of the neat solutions presented and had to take a step back to understand what was going on under-the-hood. I am documenting my findings in case it helps somebody in a position similar to mine. Credit goes to the original authors especially Yuanzhe Zhou and many others who have shared their brilliant ideas.
\nThe stated goal of this competition is somewhat interesting. The first line states that the goal is to understand the relationship between the code and comment cells. The second line challenges us to reconstruct the order of the markdown cells and this is purely how the winner is measured (the markdowns need to be ordered AND put in the right slots between the code…so the first goal does matter indirectly). We need to rank the markdown cells - identify which comes after the other & put them in he right slot between the appropriate code cells (which are already ordered correctly). The evaluation is based on Kendell Tau - and since folks have already shared some good links explaining the same, we needn’t go deep into the same. It is just an indicator between -1 to +1 on how well the predicted and the actual ranks match.
\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem. Now this provides a great relief as there is an extensive body of work under the “Learning to Rank” framework which can now be leveraged. Of course, language models also come into picture. Most of the solutions presented revolve around these 2 items - leveraging language models and leveraging the Learning to Rank framework. We now have a clear sense of direction and can dig into both these aspects and how they help us design a suitable model. But before that - What could be the intent of this competition? This was an interesting question posted https://www.kaggle.com/competitions/AI4Code/discussion/325205 in one of the discussions and my take on it is that this could just be an extension of Google search to cater more efficiently to Code results. We have a query (this is the equivalent of the markdown cell), we have content (this will be the code cell) and what we want is the code cell that is best matched to the query. This is an exciting area. Facebook is doing a lot of work in neural code search, Microsoft has CodeBert and is also neck-deep in the code search domain. Tying natural language to code has immense downstream possibilities.
\nComing to the potential approaches, we have the following:
\nLastly, let us look at how the language models can be leveraged:
\nSRK had suggested the approach of Entailment. Typically, entailment tasks involve a three-way classification of “entailment”, “contradiction”, or “neutral” (i.e., neither). In fact relevance, semantic equivalence, paraphrase, entailment are all similar tasks and with a language model like BERT, looks like the approach may not change much. The standard input template continues to be: [[CLS], s1, [SEP], s2, [SEP]] where s1 and s2 are the two inputs. As long as it is fine-tuned with annotated data that capture the nuances of the target task, the model should be able to figure out how to model the relevant relationship, be it entailment, paraphrase, or query–document relevance.
\nSpeaking of language models, this discussion is incomplete without CodeBERT and couple of its more recent peers. CodeBERT is a “bi-modal” pre-trained model where the language model (I think it is same as Roberta) is built using bi-modal data - one being natural language (NL) and the other being Programming Language (PL). The idea is to capture semantic connections between them and produce general purpose representations which can then be used for interesting applications like code-document generation, code search queries etc. In the past bi-modal data has been used in ViLBERT where the two inputs were pairs of language and image and the model learns the implicit alignment between them and predicts whether the caption describes the image content or not. A small difference is that CodeBERT leverages uni-modal data as well as part of training. In other words the inputs are not just pairs of NL/PL as we shall see but also includes raw code which does not have a NL counterpart. Bi-Modal data is processed with standard MLM objective. To process the uni-modal data (ie code without documentation), they use a RTD (replaced token detection objective) which also has side-effect of building generators. So we can generate excellent documentation from the code. The training input is: [ CLS, NLtext 1, NL text2….SEP, code snippet1, code snippet 2… EOS]. CLS is used for classification or ranking. Maybe it is a great delight to have any of the authors of CodeBert participate and share their valuable insights at the end of the competition. Recently, GraphCodeBERT incorporates the data flow extracted from the code structure into CodeBERT.
\nMost of the time we discussed about converting the 768 dimensions of the CLS vector into a single score. We could also use the same directly to measure cosine distance between them. For e.g. CodeBert is a great unified encoder. We could take a Late Fusion approach as follows - Use CodeBert to encode the Markdown cells and the code cells separately, and then calculate the similarity with a dot-product. In this way, code search is equivalent to find the nearest codes in the shared vector space. In this case, CodeBERT can be fine-tuned with the objective of maximizing the dot-product of the ground truth while minimizing the dot-product of distractors (see eq 15 of the CodeBERT paper for the exact loss). This part is not open-sourced but I guess it may not be too difficult to create it from scratch.
\nAround the same time as CodeBERT, we also had the release of CuBERT https://arxiv.org/pdf/2001.00059.pdf. This is trained on Python-only codes (originally, though now maybe other languages are added and originally used only MLM) and may prove to be a neat alternative to CodeBERT as far as this competition is concerned. Unlike CodeBERT, here the inputs i.e the code and comments are not separated with a SEP token but can be mixed freely. The original code is in Tensorflow.
\nThe latest alternative is Salesforce - CodeT5 https://arxiv.org/pdf/2109.00859.pdf which is more recent. In general, I haven't explored T5 series in the past, but I believe these series of models use denoising objectives that corrupt the source input and require the decoder to recover them (MSP - Masked Span Prediction objective). So it is pre-trained with both encoder and decoder pieces thereby giving certain advantages. On top of it they have neat techniques for custom tags in code as well as leverage the tree structure of code well. At the pre-training stage, the model receives either PL-only or NL-PL as inputs depending on whether the code snippet has accompanying NL descriptions or not. For the NL-PL bimodal inputs, they concatenate them into the standard sequence we saw earlier - ([CLS], w1, …, wn, [SEP], c1, …, cm, [SEP]), where n and m denote the number of NL word tokens and PL code tokens, respectively. The NL word sequence will be empty for PL-only unimodal inputs. On top of it they have neat ways to tokenize function names, tree syntax etc. My bets lie with this model
\nThis is an interesting competition and though it needs high end machine to process the whole data, a competitive attempt can be made with just 15-20% of the data and some neat tricks and hacks. All the very best to all the participants.
", "messages": [ { "id": 1810444, "postDate": "2022-06-03T15:10:27.057Z", "content": "This is a new domain to me. I couldn’t grasp the details of the neat solutions presented and had to take a step back to understand what was going on under-the-hood. I am documenting my findings in case it helps somebody in a position similar to mine. Credit goes to the original authors especially Yuanzhe Zhou and many others who have shared their brilliant ideas.
\nThe stated goal of this competition is somewhat interesting. The first line states that the goal is to understand the relationship between the code and comment cells. The second line challenges us to reconstruct the order of the markdown cells and this is purely how the winner is measured (the markdowns need to be ordered AND put in the right slots between the code…so the first goal does matter indirectly). We need to rank the markdown cells - identify which comes after the other & put them in he right slot between the appropriate code cells (which are already ordered correctly). The evaluation is based on Kendell Tau - and since folks have already shared some good links explaining the same, we needn’t go deep into the same. It is just an indicator between -1 to +1 on how well the predicted and the actual ranks match.
\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem. Now this provides a great relief as there is an extensive body of work under the “Learning to Rank” framework which can now be leveraged. Of course, language models also come into picture. Most of the solutions presented revolve around these 2 items - leveraging language models and leveraging the Learning to Rank framework. We now have a clear sense of direction and can dig into both these aspects and how they help us design a suitable model. But before that - What could be the intent of this competition? This was an interesting question posted https://www.kaggle.com/competitions/AI4Code/discussion/325205 in one of the discussions and my take on it is that this could just be an extension of Google search to cater more efficiently to Code results. We have a query (this is the equivalent of the markdown cell), we have content (this will be the code cell) and what we want is the code cell that is best matched to the query. This is an exciting area. Facebook is doing a lot of work in neural code search, Microsoft has CodeBert and is also neck-deep in the code search domain. Tying natural language to code has immense downstream possibilities.
\nComing to the potential approaches, we have the following:
\nLastly, let us look at how the language models can be leveraged:
\nSRK had suggested the approach of Entailment. Typically, entailment tasks involve a three-way classification of “entailment”, “contradiction”, or “neutral” (i.e., neither). In fact relevance, semantic equivalence, paraphrase, entailment are all similar tasks and with a language model like BERT, looks like the approach may not change much. The standard input template continues to be: [[CLS], s1, [SEP], s2, [SEP]] where s1 and s2 are the two inputs. As long as it is fine-tuned with annotated data that capture the nuances of the target task, the model should be able to figure out how to model the relevant relationship, be it entailment, paraphrase, or query–document relevance.
\nSpeaking of language models, this discussion is incomplete without CodeBERT and couple of its more recent peers. CodeBERT is a “bi-modal” pre-trained model where the language model (I think it is same as Roberta) is built using bi-modal data - one being natural language (NL) and the other being Programming Language (PL). The idea is to capture semantic connections between them and produce general purpose representations which can then be used for interesting applications like code-document generation, code search queries etc. In the past bi-modal data has been used in ViLBERT where the two inputs were pairs of language and image and the model learns the implicit alignment between them and predicts whether the caption describes the image content or not. A small difference is that CodeBERT leverages uni-modal data as well as part of training. In other words the inputs are not just pairs of NL/PL as we shall see but also includes raw code which does not have a NL counterpart. Bi-Modal data is processed with standard MLM objective. To process the uni-modal data (ie code without documentation), they use a RTD (replaced token detection objective) which also has side-effect of building generators. So we can generate excellent documentation from the code. The training input is: [ CLS, NLtext 1, NL text2….SEP, code snippet1, code snippet 2… EOS]. CLS is used for classification or ranking. Maybe it is a great delight to have any of the authors of CodeBert participate and share their valuable insights at the end of the competition. Recently, GraphCodeBERT incorporates the data flow extracted from the code structure into CodeBERT.
\nMost of the time we discussed about converting the 768 dimensions of the CLS vector into a single score. We could also use the same directly to measure cosine distance between them. For e.g. CodeBert is a great unified encoder. We could take a Late Fusion approach as follows - Use CodeBert to encode the Markdown cells and the code cells separately, and then calculate the similarity with a dot-product. In this way, code search is equivalent to find the nearest codes in the shared vector space. In this case, CodeBERT can be fine-tuned with the objective of maximizing the dot-product of the ground truth while minimizing the dot-product of distractors (see eq 15 of the CodeBERT paper for the exact loss). This part is not open-sourced but I guess it may not be too difficult to create it from scratch.
\nAround the same time as CodeBERT, we also had the release of CuBERT https://arxiv.org/pdf/2001.00059.pdf. This is trained on Python-only codes (originally, though now maybe other languages are added and originally used only MLM) and may prove to be a neat alternative to CodeBERT as far as this competition is concerned. Unlike CodeBERT, here the inputs i.e the code and comments are not separated with a SEP token but can be mixed freely. The original code is in Tensorflow.
\nThe latest alternative is Salesforce - CodeT5 https://arxiv.org/pdf/2109.00859.pdf which is more recent. In general, I haven't explored T5 series in the past, but I believe these series of models use denoising objectives that corrupt the source input and require the decoder to recover them (MSP - Masked Span Prediction objective). So it is pre-trained with both encoder and decoder pieces thereby giving certain advantages. On top of it they have neat techniques for custom tags in code as well as leverage the tree structure of code well. At the pre-training stage, the model receives either PL-only or NL-PL as inputs depending on whether the code snippet has accompanying NL descriptions or not. For the NL-PL bimodal inputs, they concatenate them into the standard sequence we saw earlier - ([CLS], w1, …, wn, [SEP], c1, …, cm, [SEP]), where n and m denote the number of NL word tokens and PL code tokens, respectively. The NL word sequence will be empty for PL-only unimodal inputs. On top of it they have neat ways to tokenize function names, tree syntax etc. My bets lie with this model
\nThis is an interesting competition and though it needs high end machine to process the whole data, a competitive attempt can be made with just 15-20% of the data and some neat tricks and hacks. All the very best to all the participants.
", "rawMarkdown": "This is a new domain to me. I couldn’t grasp the details of the neat solutions presented and had to take a step back to understand what was going on under-the-hood. I am documenting my findings in case it helps somebody in a position similar to mine. Credit goes to the original authors especially Yuanzhe Zhou and many others who have shared their brilliant ideas.\n\n##AN INTERESTING GOAL\nThe stated goal of this competition is somewhat interesting. The first line states that the goal is to understand the relationship between the code and comment cells. The second line challenges us to reconstruct the order of the markdown cells and this is purely how the winner is measured (~~the first goal does not matter as far as evaluations concerned~~the markdowns need to be ordered AND put in the right slots between the code...so the first goal does matter indirectly). We need to rank the markdown cells - identify which comes after the other & put them in he right slot between the appropriate code cells (which are already ordered correctly). The evaluation is based on Kendell Tau - and since folks have already shared some good links explaining the same, we needn’t go deep into the same. It is just an indicator between -1 to +1 on how well the predicted and the actual ranks match.\n\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem. Now this provides a great relief as there is an extensive body of work under the “Learning to Rank” framework which can now be leveraged. Of course, language models also come into picture. Most of the solutions presented revolve around these 2 items - leveraging language models and leveraging the Learning to Rank framework. We now have a clear sense of direction and can dig into both these aspects and how they help us design a suitable model. But before that - What could be the intent of this competition? This was an interesting question posted https://www.kaggle.com/competitions/AI4Code/discussion/325205 in one of the discussions and my take on it is that this could just be an extension of Google search to cater more efficiently to Code results. We have a **query** (this is the equivalent of the markdown cell), we have **content** (this will be the code cell) and what we want is the code cell that is best matched to the query. This is an exciting area. Facebook is doing a lot of work in neural code search, Microsoft has CodeBert and is also neck-deep in the code search domain. Tying natural language to code has immense downstream possibilities. \n\nComing to the potential approaches, we have the following:\n- Rank only the Markdowns without paying heed to the code cells. Use plain NLP for this combined with the LTR (Learning To Rank) frameworks. Surprisingly this seems to get OK results. When given a \"context\" in the form of code cells these work really well\n- Tie the Markdown to the code. In this case we dont need to develop a model for ranking because the code cells are already ranked. As long as we are able to match the code to markdown, we directly get the markdown order. \n- See how we can leverage both the above approaches.\n\n##Learning to Rank framework: \n- Easiest is to treat this as a regression problem by assigning each markdown cell a score and ranking is obtained by a simple sort. MSE can be the loss function. This is the Pointwise approach. Basically each Markdown cell is scored separately. Though I say regression, we can use Classification, Ordinal regression etc as long as items to be ranked are treated in isolation. This can be further enhanced by providing a \"context\" in the form of code cells\n- Then we have the Pair-wise approach: Here we take pairs of 2 markdowns and simply rank which comes first. We now bring relative scores into the picture. This seems cleaner. Cost function typically minimises misclassification of pairwise difference vectors. The challenge is we have too many permutations and combinations while training as well as inference. Moreover, during training we have to identify negative pairs and as of now, there is no great way to decide on what makes a good -ve pair. The interesting part is that in a normal scenario (like in a search query ranking), Pairwise models have a major disadvantage - Every document pair is treated equally important, such setting is not useful in real world scenario because we expect our search systems to answer correctly in top 10 items and not really care about the remaining 11-100 results that much. But in this comp, all the ranks are important and so pair-wise approach becomes much more powerful. \n- Then there is the Listwise approach: Idea is to take the whole set and rank every entry in the list at one shot by optimizing ranking metrics directly which is a great thought except that these usually are not continuous and back-prop will not work. So there are interesting proxy approaches to get gradients for these cases to enable us to compute loss and decide the best order.\n\nLastly, let us look at how the language models can be leveraged:\n- A plain vanilla approach is to take the query and document (which in our case is the markdown and the code cell), jointly encode the same and fine-tune the model’s classification component to provide a ranking score. Basically tuning is done in such a way that related markdown and code provide better scores compared to unrelated markdown and code. So basically the input to the model is: [ CLS, Markdown text, SEP, code snippet, EOS ]. The vector representation corresponding to the [CLS] token is the one that is used to infer a query-document relevance score. To project the vector of the [CLS] token into a scalar value a single-layer, a fully connected neural network is used which maps from (say) 768 Bert dimensions to one score. If the score is high, it means the markdown and code cells are linked else we check other combinations. In a sense this is like a pair-wise approach except that we are leveraging the language model here\n- Point-wise approach as discussed above would be to use regression/MSE (or MAE which is better) to rank markdown cells. By giving it a context which contains the code cells this performs really well. So input = md+separator+context\n- It might be interesting to see if we can bring in list-wise approaches to the language model. See for e.g. https://ai.googleblog.com/2021/07/advances-in-tf-ranking.html. There they learn an independent BERT representation for each\n\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem
\n
Since the code has already been ordered, one can also treat this as a slot problem. Given a markdown and some slots, which slot does this markdown belongs to?
\nThe pairwise method actually treat this problem as the slot problem not ranking. And the solution open sourced by Yuanzhe Zhou only comes with one side of the \"slot\". In that solution, the \"pair\" goes to code + markdown, but we can bring more to this: code + markdown + code, or even code + code + markdown + code + code
Yes, my bad. I explained general \"Learn to Rank\" framework in beginning. But coming specifically to Zhou code, I agree here pair-wise takes a slightly diff meaning. He does not rank 2 pieces of markdown pairwise but the pair here stands for code and markdown. Each such pair gets a score and this is how model learns to get related pairs. Once pairs are learnt, the markdowns can be ranked easily because the codes are already ranked and it is matter of pushing the corresponding markdown to one slot above the code with highest match. This is a simple method, though Zhou does some very interesting softmax logic by scaling the probs 20 times (possibly to account for the 5% -ve sampling? not sure?) & also summing up the product of softmax and code rank (maybe it \"pulls\" the rank towards the areas where a set of good scores lie and not just tie to the max score)
\nBTW, congrats on your good \"Rank\" in the competition :)
\nEdit: The multiplier by 20 may be just empirical. It drives the scores to get more clearly segregated which means that the distribution will be more \"spiky\", the differences between the order statistics will get bigger. multiplying by < 1 does the opposite - tends to push this distribution uniform.
\nhttps://math.stackexchange.com/questions/2471528/understanding-the-softmax-function.
\nso looks like this has nothing to fo with the -ve sampling 5% statistic :)
Thank you too for this very informative post. It helps me a lot since I'm new to NLP and transformer(s)
", "rawMarkdown": "Thank you too for this very informative post. It helps me a lot since I'm new to NLP and transformer(s)", "votes": 1 } ] }, { "id": 1812775, "postDate": "2022-06-06T07:40:45.983Z", "content": "Thanks all for your comments. I would like to share one additional info which happened recently. There was a community comp recently on analyzing software code where CodeBert was extensively used. Perhaps some of the discussions/code there may be helpful in finetuning: https://www.kaggle.com/competitions/wherethereiscodethereisbug/overview
", "rawMarkdown": "Thanks all for your comments. I would like to share one additional info which happened recently. There was a community comp recently on analyzing software code where CodeBert was extensively used. Perhaps some of the discussions/code there may be helpful in finetuning: https://www.kaggle.com/competitions/wherethereiscodethereisbug/overview", "votes": 3 }, { "id": 1844164, "postDate": "2022-07-05T11:43:06.340Z", "content": "Insightful especially with the input consisting of code snippets and also I had been looking for alternatives to CodeBERT. Thanks for the information.
", "rawMarkdown": "Insightful especially with the input consisting of code snippets and also I had been looking for alternatives to CodeBERT. Thanks for the information.", "votes": 1 }, { "id": 1840454, "postDate": "2022-07-02T09:03:25.693Z", "content": "Thank you for sharing! I wonder how could I use fully-trained pairwise model to generate a ordered notebook.
", "rawMarkdown": "Thank you for sharing! I wonder how could I use fully-trained pairwise model to generate a ordered notebook. ", "votes": 1, "replies": [ { "id": 1840495, "postDate": "2022-07-02T09:44:05.277Z", "content": "Thanks. I didnt get time to work on this comp due to professional commitments. Maybe I will pick up in coming weeks. Both models have been shared publicly (Zhou and Khoi). If you filter on best scores in public kernels you should be getting both of these.
", "rawMarkdown": "Thanks. I didnt get time to work on this comp due to professional commitments. Maybe I will pick up in coming weeks. Both models have been shared publicly (Zhou and Khoi). If you filter on best scores in public kernels you should be getting both of these. " } ] }, { "id": 1813434, "postDate": "2022-06-06T20:10:05.383Z", "content": "@allohvk Thanks for sharing such a valuable resource. Its a must read for anyone making an attempt for this challenge especially newbies. 🎉
\nIn my opinion, a plain vanilla approach combined with list-wise approach will be most suitable for this challenge.
\nCode-Aware Language Models section here is a jewel. ✨✨✨
", "rawMarkdown": "@allohvk Thanks for sharing such a valuable resource. Its a must read for anyone making an attempt for this challenge especially newbies. 🎉\n\nIn my opinion, a plain vanilla approach combined with list-wise approach will be most suitable for this challenge.\n\nCode-Aware Language Models section here is a jewel. ✨✨✨\n\n", "votes": 1 }, { "id": 1812664, "postDate": "2022-06-06T05:11:29.623Z", "content": "Thanks for sharing @allohvk ! superb article ! I guess majority of us are new to this too :)
", "rawMarkdown": "Thanks for sharing @allohvk ! superb article ! I guess majority of us are new to this too :)", "votes": 1 }, { "id": 1812490, "postDate": "2022-06-05T22:27:02.203Z", "content": "Thank you for sharing your thoughts! I tried to sort markdown cells at first and then spread them onto code ones. But what I didn't like in that approach is if for example kernel's author tries out several in one kernel you have virtually no chance to guess which order of methods he/she uses. So I'm currently thinking of a way to firstly map some md-cells to code and then insert all the other markdown, but I haven't yet come up with anything :(
", "rawMarkdown": "Thank you for sharing your thoughts! I tried to sort markdown cells at first and then spread them onto code ones. But what I didn't like in that approach is if for example kernel's author tries out several in one kernel you have virtually no chance to guess which order of methods he/she uses. So I'm currently thinking of a way to firstly map some md-cells to code and then insert all the other markdown, but I haven't yet come up with anything :(", "votes": 1 }, { "id": 1811506, "postDate": "2022-06-04T18:40:40.817Z", "content": "Thanks! very informative! new to this too
", "rawMarkdown": "Thanks! very informative! new to this too", "votes": 1 }, { "id": 1811504, "postDate": "2022-06-04T18:39:17.733Z", "content": "Interesting and lot to take it!! I havent explored NLP much but highly interested. This is very helpful. Keeping all papers in my reading list.. thanks for sharing.🙏
", "rawMarkdown": "Interesting and lot to take it!! I havent explored NLP much but highly interested. This is very helpful. Keeping all papers in my reading list.. thanks for sharing.🙏", "votes": 1 }, { "id": 1810599, "postDate": "2022-06-03T17:54:02.010Z", "content": "Thanks for sharing @allohvk. This is a new domain for me too and I learned a lot from your post.
", "rawMarkdown": "Thanks for sharing @allohvk. This is a new domain for me too and I learned a lot from your post. ", "votes": 1 }, { "id": 1812718, "postDate": "2022-06-06T06:34:00.470Z", "content": "Very Informative, thanks for sharing. Bi-Modal/Multi-Modal could be an important topic in this competition. There are a lot research recently for both bimodal including visual-text pair and text-text pair(like sentence and table data), and it will be very interesting to see techniques from those research to be applied in this competition, such as the cross-modality attention mechanism
", "rawMarkdown": "Very Informative, thanks for sharing. Bi-Modal/Multi-Modal could be an important topic in this competition. There are a lot research recently for both bimodal including visual-text pair and text-text pair(like sentence and table data), and it will be very interesting to see techniques from those research to be applied in this competition, such as the cross-modality attention mechanism", "votes": 2 }, { "id": 1839797, "postDate": "2022-07-01T17:36:42.540Z", "content": "I was wondering how to finetune the codeBert?We don't know which markdown is for which code cell, cus all the code cells are ranked before the markdown in the dataset
", "rawMarkdown": "I was wondering how to finetune the codeBert?We don't know which markdown is for which code cell, cus all the code cells are ranked before the markdown in the dataset", "votes": 1, "isDeleted": true, "replies": [ { "id": 1840493, "postDate": "2022-07-02T09:41:11.980Z", "content": "Pl check https://www.kaggle.com/competitions/AI4Code/discussion/326970
", "rawMarkdown": "Pl check https://www.kaggle.com/competitions/AI4Code/discussion/326970" }, { "id": 1846226, "postDate": "2022-07-07T00:58:44.823Z", "content": "Tks!That really helps
", "rawMarkdown": "Tks!That really helps", "votes": 1, "isDeleted": true } ] }, { "id": 1815051, "postDate": "2022-06-08T16:04:25.840Z", "content": "Thank you very much for your sharing!
", "rawMarkdown": "Thank you very much for your sharing! ", "votes": 1 } ], "comments": [ { "id": 1817930, "author_name": "Chenglu", "author_url": "", "post_date": "2022-06-12T03:48:18.483000", "content": "\n\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem
\n
Since the code has already been ordered, one can also treat this as a slot problem. Given a markdown and some slots, which slot does this markdown belongs to?
\nThe pairwise method actually treat this problem as the slot problem not ranking. And the solution open sourced by Yuanzhe Zhou only comes with one side of the \"slot\". In that solution, the \"pair\" goes to code + markdown, but we can bring more to this: code + markdown + code, or even code + code + markdown + code + code
Yes, my bad. I explained general \"Learn to Rank\" framework in beginning. But coming specifically to Zhou code, I agree here pair-wise takes a slightly diff meaning. He does not rank 2 pieces of markdown pairwise but the pair here stands for code and markdown. Each such pair gets a score and this is how model learns to get related pairs. Once pairs are learnt, the markdowns can be ranked easily because the codes are already ranked and it is matter of pushing the corresponding markdown to one slot above the code with highest match. This is a simple method, though Zhou does some very interesting softmax logic by scaling the probs 20 times (possibly to account for the 5% -ve sampling? not sure?) & also summing up the product of softmax and code rank (maybe it \"pulls\" the rank towards the areas where a set of good scores lie and not just tie to the max score)
\nBTW, congrats on your good \"Rank\" in the competition :)
\nEdit: The multiplier by 20 may be just empirical. It drives the scores to get more clearly segregated which means that the distribution will be more \"spiky\", the differences between the order statistics will get bigger. multiplying by < 1 does the opposite - tends to push this distribution uniform.
\nhttps://math.stackexchange.com/questions/2471528/understanding-the-softmax-function.
\nso looks like this has nothing to fo with the -ve sampling 5% statistic :)
Thank you too for this very informative post. It helps me a lot since I'm new to NLP and transformer(s)
", "votes": 1, "replies": [] } ] }, { "id": 1812775, "author_name": "Allohvk", "author_url": "", "post_date": "2022-06-06T07:40:45.983000", "content": "Thanks all for your comments. I would like to share one additional info which happened recently. There was a community comp recently on analyzing software code where CodeBert was extensively used. Perhaps some of the discussions/code there may be helpful in finetuning: https://www.kaggle.com/competitions/wherethereiscodethereisbug/overview
", "votes": 3, "replies": [] }, { "id": 1844164, "author_name": "samu2505", "author_url": "", "post_date": "2022-07-05T11:43:06.340000", "content": "Insightful especially with the input consisting of code snippets and also I had been looking for alternatives to CodeBERT. Thanks for the information.
", "votes": 1, "replies": [] }, { "id": 1840454, "author_name": "Teemo", "author_url": "", "post_date": "2022-07-02T09:03:25.693000", "content": "Thank you for sharing! I wonder how could I use fully-trained pairwise model to generate a ordered notebook.
", "votes": 1, "replies": [ { "id": 1840495, "author_name": "Allohvk", "author_url": "", "post_date": "2022-07-02T09:44:05.277000", "content": "Thanks. I didnt get time to work on this comp due to professional commitments. Maybe I will pick up in coming weeks. Both models have been shared publicly (Zhou and Khoi). If you filter on best scores in public kernels you should be getting both of these.
", "votes": 0, "replies": [] } ] }, { "id": 1813434, "author_name": "Muhammad Irfan Azam", "author_url": "", "post_date": "2022-06-06T20:10:05.383000", "content": "@allohvk Thanks for sharing such a valuable resource. Its a must read for anyone making an attempt for this challenge especially newbies. 🎉
\nIn my opinion, a plain vanilla approach combined with list-wise approach will be most suitable for this challenge.
\nCode-Aware Language Models section here is a jewel. ✨✨✨
", "votes": 1, "replies": [] }, { "id": 1812664, "author_name": "Old Monk", "author_url": "", "post_date": "2022-06-06T05:11:29.623000", "content": "Thanks for sharing @allohvk ! superb article ! I guess majority of us are new to this too :)
", "votes": 1, "replies": [] }, { "id": 1812490, "author_name": "Sergey Dulikov", "author_url": "", "post_date": "2022-06-05T22:27:02.203000", "content": "Thank you for sharing your thoughts! I tried to sort markdown cells at first and then spread them onto code ones. But what I didn't like in that approach is if for example kernel's author tries out several in one kernel you have virtually no chance to guess which order of methods he/she uses. So I'm currently thinking of a way to firstly map some md-cells to code and then insert all the other markdown, but I haven't yet come up with anything :(
", "votes": 1, "replies": [] }, { "id": 1811506, "author_name": "TheStoneMX", "author_url": "", "post_date": "2022-06-04T18:40:40.817000", "content": "Thanks! very informative! new to this too
", "votes": 1, "replies": [] }, { "id": 1811504, "author_name": "Chirag Desai", "author_url": "", "post_date": "2022-06-04T18:39:17.733000", "content": "Interesting and lot to take it!! I havent explored NLP much but highly interested. This is very helpful. Keeping all papers in my reading list.. thanks for sharing.🙏
", "votes": 1, "replies": [] }, { "id": 1810599, "author_name": "Jose Carmona", "author_url": "", "post_date": "2022-06-03T17:54:02.010000", "content": "Thanks for sharing @allohvk. This is a new domain for me too and I learned a lot from your post.
", "votes": 1, "replies": [] }, { "id": 1812718, "author_name": "Diego", "author_url": "", "post_date": "2022-06-06T06:34:00.470000", "content": "Very Informative, thanks for sharing. Bi-Modal/Multi-Modal could be an important topic in this competition. There are a lot research recently for both bimodal including visual-text pair and text-text pair(like sentence and table data), and it will be very interesting to see techniques from those research to be applied in this competition, such as the cross-modality attention mechanism
", "votes": 2, "replies": [] }, { "id": 1839797, "author_name": "", "author_url": "", "post_date": "2022-07-01T17:36:42.540000", "content": "I was wondering how to finetune the codeBert?We don't know which markdown is for which code cell, cus all the code cells are ranked before the markdown in the dataset
", "votes": 1, "replies": [ { "id": 1840493, "author_name": "Allohvk", "author_url": "", "post_date": "2022-07-02T09:41:11.980000", "content": "Pl check https://www.kaggle.com/competitions/AI4Code/discussion/326970
", "votes": 0, "replies": [] }, { "id": 1846226, "author_name": "", "author_url": "", "post_date": "2022-07-07T00:58:44.823000", "content": "Tks!That really helps
", "votes": 1, "replies": [] } ] }, { "id": 1815051, "author_name": "Quyen Truong", "author_url": "", "post_date": "2022-06-08T16:04:25.840000", "content": "Thank you very much for your sharing!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1810444": "This is a new domain to me. I couldn’t grasp the details of the neat solutions presented and had to take a step back to understand what was going on under-the-hood. I am documenting my findings in case it helps somebody in a position similar to mine. Credit goes to the original authors especially Yuanzhe Zhou and many others who have shared their brilliant ideas.\n\n##AN INTERESTING GOAL\nThe stated goal of this competition is somewhat interesting. The first line states that the goal is to understand the relationship between the code and comment cells. The second line challenges us to reconstruct the order of the markdown cells and this is purely how the winner is measured (~~the first goal does not matter as far as evaluations concerned~~the markdowns need to be ordered AND put in the right slots between the code...so the first goal does matter indirectly). We need to rank the markdown cells - identify which comes after the other & put them in he right slot between the appropriate code cells (which are already ordered correctly). The evaluation is based on Kendell Tau - and since folks have already shared some good links explaining the same, we needn’t go deep into the same. It is just an indicator between -1 to +1 on how well the predicted and the actual ranks match.\n\nFor a moment, keeping the code part aside, this essentially boils down to ranking a series of texts in the right order. In that sense, it can be reframed as a ranking problem. Now this provides a great relief as there is an extensive body of work under the “Learning to Rank” framework which can now be leveraged. Of course, language models also come into picture. Most of the solutions presented revolve around these 2 items - leveraging language models and leveraging the Learning to Rank framework. We now have a clear sense of direction and can dig into both these aspects and how they help us design a suitable model. But before that - What could be the intent of this competition? This was an interesting question posted https://www.kaggle.com/competitions/AI4Code/discussion/325205 in one of the discussions and my take on it is that this could just be an extension of Google search to cater more efficiently to Code results. We have a **query** (this is the equivalent of the markdown cell), we have **content** (this will be the code cell) and what we want is the code cell that is best matched to the query. This is an exciting area. Facebook is doing a lot of work in neural code search, Microsoft has CodeBert and is also neck-deep in the code search domain. Tying natural language to code has immense downstream possibilities. \n\nComing to the potential approaches, we have the following:\n- Rank only the Markdowns without paying heed to the code cells. Use plain NLP for this combined with the LTR (Learning To Rank) frameworks. Surprisingly this seems to get OK results. When given a \"context\" in the form of code cells these work really well\n- Tie the Markdown to the code. In this case we dont need to develop a model for ranking because the code cells are already ranked. As long as we are able to match the code to markdown, we directly get the markdown order. \n- See how we can leverage both the above approaches.\n\n##Learning to Rank framework: \n- Easiest is to treat this as a regression problem by assigning each markdown cell a score and ranking is obtained by a simple sort. MSE can be the loss function. This is the Pointwise approach. Basically each Markdown cell is scored separately. Though I say regression, we can use Classification, Ordinal regression etc as long as items to be ranked are treated in isolation. This can be further enhanced by providing a \"context\" in the form of code cells\n- Then we have the Pair-wise approach: Here we take pairs of 2 markdowns and simply rank which comes first. We now bring relative scores into the picture. This seems cleaner. Cost function typically minimises misclassification of pairwise difference vectors. The challenge is we have too many permutations and combinations while training as well as inference. Moreover, during training we have to identify negative pairs and as of now, there is no great way to decide on what makes a good -ve pair. The interesting part is that in a normal scenario (like in a search query ranking), Pairwise models have a major disadvantage - Every document pair is treated equally important, such setting is not useful in real world scenario because we expect our search systems to answer correctly in top 10 items and not really care about the remaining 11-100 results that much. But in this comp, all the ranks are important and so pair-wise approach becomes much more powerful. \n- Then there is the Listwise approach: Idea is to take the whole set and rank every entry in the list at one shot by optimizing ranking metrics directly which is a great thought except that these usually are not continuous and back-prop will not work. So there are interesting proxy approaches to get gradients for these cases to enable us to compute loss and decide the best order.\n\nLastly, let us look at how the language models can be leveraged:\n- A plain vanilla approach is to take the query and document (which in our case is the markdown and the code cell), jointly encode the same and fine-tune the model’s classification component to provide a ranking score. Basically tuning is done in such a way that related markdown and code provide better scores compared to unrelated markdown and code. So basically the input to the model is: [ CLS, Markdown text, SEP, code snippet, EOS ]. The vector representation corresponding to the [CLS] token is the one that is used to infer a query-document relevance score. To project the vector of the [CLS] token into a scalar value a single-layer, a fully connected neural network is used which maps from (say) 768 Bert dimensions to one score. If the score is high, it means the markdown and code cells are linked else we check other combinations. In a sense this is like a pair-wise approach except that we are leveraging the language model here\n- Point-wise approach as discussed above would be to use regression/MSE (or MAE which is better) to rank markdown cells. By giving it a context which contains the code cells this performs really well. So input = md+separator+context\n- It might be interesting to see if we can bring in list-wise approaches to the language model. See for e.g. https://ai.googleblog.com/2021/07/advances-in-tf-ranking.html. There they learn an independent BERT representation for each