(from the: Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments)
\nCore: The core solution is well outlined in the public kernel by Nguyen, the basic idea is to add a ‘context’ which helps the model rank the cells better.
Pre-Processing 1: Retaining the last few tokens in the markdown (i.e. last few words) helped in a small way. In other words, instead of simply truncating a lengthy markdown, we retain a few ending tokens. These last few words/tokens seem to be more intimately connected to the code cell that follows and helps the model make better connections.
Pre-Processing 2: Grabbing the class names & function names from the code cells and moving them to the top of the code cell (to avoid truncation) also helps slightly. Moving ‘all comments in the code cell’ to the top of the code cell didn’t help although I have a strong feeling this would have helped larger models (mine was only 768 tokens)
Data Selection: Kernels selected should be relevant & match with actual test data. This is important from 3 perspectives:
\n** Fits into my compute budget
\n** I can run more experiments
\n** In absence of strong validation data, I felt this will play key role when actual results start rolling in
\nWhat helped:
\n** Kaggle increased memory size from 5-20 GB 3 years back. I could identify a bunch of old kernels by simply searching for markdowns on “You can write up to 5GB to\"
\n** Identify R kernels, TF1 kernels etc which use old libraries
\n** Identify common ancestors/parent kernels etc
\n** Many times, kernels are not forked, however they are copied manually. So identify first few cells (say 2 markdowns and 3 code cells) and remove duplicates
\n** When sampling take very few samples from above categories (however don’t make it 0%). For remaining kernels, take a healthy sample.
\n** In all, I could go up to around 70% samples and yet get a decent result. This 70%, I further split into training and validation
\n** After tuning, I trained on validation data as well
\n** In past 3 months, as new data kept rolling in, each run has improved my score slightly
Minor point during Ensembling: A very minor (4th decimal) increase during ensemble time by checking if the last cell is a markdown. Pair-wise model does not rank such markdowns correctly (due to the way triplet generation happens). So if the last cell is a markdown, don’t average between pair-wise & point wise & instead just take the point-wise data.
Having a good ‘context’ is the key to getting a good score (with this model). With 768 tokens one can squeeze in maybe 20 tokens for each code cell assuming an average of 30 code cells per kernel with a few tokens remaining for the markdown cell in question. 20 tokens covers barely 1-2 lines of code even after preprocessing. Is there a way to increase this ‘context’ without increasing the model size? I thought that both BigBird and Longformer, being sparse attention models would be the right fit to apply to the problem at hand. In Longformer, tokens attend to tokens in the vicinity defined by a window that slides as we move across the text during self-attention. The receptive field can be further increased by applying dilation to the local window (basically the token simply skips certain tokens) thus allowing attention to reach farther tokens. In addition, we also have global tokens which are attended to by everybody. Similarly Bigbird has a combination of local (sliding window), global & random attention. I felt AI4C domain was a classic case for usage of these models since the relationship between kernel cells can be captured beautifully given the constraints of limited token space:
\nIt has been four months since the competition ended and I could not retrieve the exact empirical details I maintained, however based on the available notes/kernels, I have tried to give a gist of my experience w.r.t what worked and what didn’t. I apologize for not posting this earlier as I felt that I didn’t have anything significant to contribute in terms of ‘What worked’. However Mark Mc Donald’s template made me realize that perhaps I did have something to share as far as “What didn’t work” is concerned 😊
\nI want to Thank the Kaggle staff team for coordinating this great competition and ensuring it goes off seamlessly over a timeframe of 6 months! I want to thank Nguyen for raising the bar high with the initial codebase he shared. This competition had leading industry experts, leading Kaggle grandmasters (many from top 20) & (maybe) folks from academia. I would like to Thank all of them for sharing their thoughts and views during the course of this competition.
\nAllohvk
", "messages": [ { "id": 2035897, "postDate": "2022-11-19T09:35:18.823Z", "content": "(from the: Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments)
\nCore: The core solution is well outlined in the public kernel by Nguyen, the basic idea is to add a ‘context’ which helps the model rank the cells better.
Pre-Processing 1: Retaining the last few tokens in the markdown (i.e. last few words) helped in a small way. In other words, instead of simply truncating a lengthy markdown, we retain a few ending tokens. These last few words/tokens seem to be more intimately connected to the code cell that follows and helps the model make better connections.
Pre-Processing 2: Grabbing the class names & function names from the code cells and moving them to the top of the code cell (to avoid truncation) also helps slightly. Moving ‘all comments in the code cell’ to the top of the code cell didn’t help although I have a strong feeling this would have helped larger models (mine was only 768 tokens)
Data Selection: Kernels selected should be relevant & match with actual test data. This is important from 3 perspectives:
\n** Fits into my compute budget
\n** I can run more experiments
\n** In absence of strong validation data, I felt this will play key role when actual results start rolling in
\nWhat helped:
\n** Kaggle increased memory size from 5-20 GB 3 years back. I could identify a bunch of old kernels by simply searching for markdowns on “You can write up to 5GB to\"
\n** Identify R kernels, TF1 kernels etc which use old libraries
\n** Identify common ancestors/parent kernels etc
\n** Many times, kernels are not forked, however they are copied manually. So identify first few cells (say 2 markdowns and 3 code cells) and remove duplicates
\n** When sampling take very few samples from above categories (however don’t make it 0%). For remaining kernels, take a healthy sample.
\n** In all, I could go up to around 70% samples and yet get a decent result. This 70%, I further split into training and validation
\n** After tuning, I trained on validation data as well
\n** In past 3 months, as new data kept rolling in, each run has improved my score slightly
Minor point during Ensembling: A very minor (4th decimal) increase during ensemble time by checking if the last cell is a markdown. Pair-wise model does not rank such markdowns correctly (due to the way triplet generation happens). So if the last cell is a markdown, don’t average between pair-wise & point wise & instead just take the point-wise data.
Having a good ‘context’ is the key to getting a good score (with this model). With 768 tokens one can squeeze in maybe 20 tokens for each code cell assuming an average of 30 code cells per kernel with a few tokens remaining for the markdown cell in question. 20 tokens covers barely 1-2 lines of code even after preprocessing. Is there a way to increase this ‘context’ without increasing the model size? I thought that both BigBird and Longformer, being sparse attention models would be the right fit to apply to the problem at hand. In Longformer, tokens attend to tokens in the vicinity defined by a window that slides as we move across the text during self-attention. The receptive field can be further increased by applying dilation to the local window (basically the token simply skips certain tokens) thus allowing attention to reach farther tokens. In addition, we also have global tokens which are attended to by everybody. Similarly Bigbird has a combination of local (sliding window), global & random attention. I felt AI4C domain was a classic case for usage of these models since the relationship between kernel cells can be captured beautifully given the constraints of limited token space:
\nIt has been four months since the competition ended and I could not retrieve the exact empirical details I maintained, however based on the available notes/kernels, I have tried to give a gist of my experience w.r.t what worked and what didn’t. I apologize for not posting this earlier as I felt that I didn’t have anything significant to contribute in terms of ‘What worked’. However Mark Mc Donald’s template made me realize that perhaps I did have something to share as far as “What didn’t work” is concerned 😊
\nI want to Thank the Kaggle staff team for coordinating this great competition and ensuring it goes off seamlessly over a timeframe of 6 months! I want to thank Nguyen for raising the bar high with the initial codebase he shared. This competition had leading industry experts, leading Kaggle grandmasters (many from top 20) & (maybe) folks from academia. I would like to Thank all of them for sharing their thoughts and views during the course of this competition.
\nAllohvk
", "rawMarkdown": "#### 38th Place Solution\n(from the: Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments)\n\n#### Overview:\n-\t**Model**: Simple blend of point-wise (DeBERTa-V3-Small & GraphCodeBERT-base….thanks Guo) & [pair-wise](https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training/notebook) ensemble just for diversity. Around 20% weightage to pair-wise.\n-\t**Data**: Use around 70% of data by careful selection of notebooks. No external data.\n-\t**Compute**: Free Kaggle & Google Colab. Rented A5000-24GB intermittently during last couple of weeks).\nLightweight models due to limited compute and the fact that I was only working on weekends:\n\n#### What worked:\n-\t**Core**: The core solution is well outlined in the public [kernel ](https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells)by Nguyen, the basic idea is to add a ‘context’ which helps the model rank the cells better.\n-\t**Pre-Processing 1**: Retaining the last few tokens in the markdown (i.e. last few words) helped in a small way. In other words, instead of simply truncating a lengthy markdown, we retain a few ending tokens. These last few words/tokens seem to be more intimately connected to the code cell that follows and helps the model make better connections.\n-\t**Pre-Processing 2**: Grabbing the class names & function names from the code cells and moving them to the top of the code cell (to avoid truncation) also helps slightly. Moving ‘all comments in the code cell’ to the top of the code cell didn’t help although I have a strong feeling this would have helped larger models (mine was only 768 tokens)\n-\t**Data Selection:** Kernels selected should be relevant & match with actual test data. This is important from 3 perspectives:\n** Fits into my compute budget\n** I can run more experiments\n** In absence of strong validation data, I felt this will play key role when actual results start rolling in\n****What helped:****\n** Kaggle increased memory size from 5-20 GB 3 years back. I could identify a bunch of old kernels by simply searching for markdowns on “You can write up to 5GB to\"\n** Identify R kernels, TF1 kernels etc which use old libraries\n** Identify common ancestors/parent kernels etc\n** Many times, kernels are not forked, however they are copied manually. So identify first few cells (say 2 markdowns and 3 code cells) and remove duplicates\n** When sampling take very few samples from above categories (however don’t make it 0%). For remaining kernels, take a healthy sample. \n** In all, I could go up to around 70% samples and yet get a decent result. This 70%, I further split into training and validation\n** After tuning, I trained on validation data as well\n** In past 3 months, as new data kept rolling in, each run has improved my score slightly\n\n-\t**Minor point during Ensembling**: A very minor (4th decimal) increase during ensemble time by checking if the last cell is a markdown. Pair-wise model does not rank such markdowns correctly (due to the way triplet generation happens). So if the last cell is a markdown, don’t average between pair-wise & point wise & instead just take the point-wise data.\n\n#### What didn’t work:\n-\tA variety of attention heads (at least 5) were tried & none gave a significantly better result than default one\n-\tBi-LSTM on top didn’t help either\n-\tIn pair-wise, Arcface, Cosface was used to sharpen the embeddings but didn’t show great results. In fact I had trouble in reproducing results consistently for this model\n-\tThe biggest item in the “didn’t work” category would be my experiences with BigBird and Longformer outlined in the ‘Additional context’ section.\n-\t2 costly mistakes: Wasted lot of token space as I didn’t catch this [bug](https://www.kaggle.com/competitions/AI4Code/discussion/343801#1896253). I felt this could have pushed up my score significantly. Secondly, last few days had to deal with memory errors. I am reasonably convinced that root cause could be [this](https://www.kaggle.com/competitions/AI4Code/discussion/343471).\n\n#### Additional context (on a failed experiment):\nHaving a good ‘context’ is the key to getting a good score (with this model). With 768 tokens one can squeeze in maybe 20 tokens for each code cell assuming an average of 30 code cells per kernel with a few tokens remaining for the markdown cell in question. 20 tokens covers barely 1-2 lines of code even after preprocessing. Is there a way to increase this ‘context’ without increasing the model size? I thought that both BigBird and Longformer, being sparse attention models would be the right fit to apply to the problem at hand. In Longformer, tokens attend to tokens in the vicinity defined by a window that slides as we move across the text during self-attention. The receptive field can be further increased by applying dilation to the local window (basically the token simply skips certain tokens) thus allowing attention to reach farther tokens. In addition, we also have global tokens which are attended to by everybody. Similarly Bigbird has a combination of local (sliding window), global & random attention. I felt AI4C domain was a classic case for usage of these models since the relationship between kernel cells can be captured beautifully given the constraints of limited token space:\n-\tThe sliding window will ensure to capture relation between 8-10 consecutive cells. These matter the most. Random tokens do the rest of the job.\n-\tGlobal tokens could be identified that attend to all cells. The tokens in the markdown cell would all be global & will attend to each and every code cell token.\n-\tThis way the ‘context’ becomes really really powerful. Relationship between the markdown in question and each and every code cell token can be established while program continuity aspect and relation between a code cell & the cells above and below it can be captured by the sliding window. Random attention further bring in unanticipated connections across cell tokens .\nHowever looks like both models perform well only in excess of 4096 tokens and do not work for anything less than that. Some experiments and discussion with HF on this aspect is captured [here](https://github.com/huggingface/transformers/issues/18234).\n\n#### Thanks and Acknowledgements:\nIt has been four months since the competition ended and I could not retrieve the exact empirical details I maintained, however based on the available notes/kernels, I have tried to give a gist of my experience w.r.t what worked and what didn’t. I apologize for not posting this earlier as I felt that I didn’t have anything significant to contribute in terms of ‘What worked’. However Mark Mc Donald’s template made me realize that perhaps I did have something to share as far as “What didn’t work” is concerned 😊\n\nI want to Thank the Kaggle staff team for coordinating this great competition and ensuring it goes off seamlessly over a timeframe of 6 months! I want to thank Nguyen for raising the bar high with the initial codebase he shared. This competition had leading industry experts, leading Kaggle grandmasters (many from top 20) & (maybe) folks from academia. I would like to Thank all of them for sharing their thoughts and views during the course of this competition. \n\n#### Team Members:\nAllohvk", "votes": 5 } ], "comments": [], "raw_markdown_by_id": { "2035897": "#### 38th Place Solution\n(from the: Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments)\n\n#### Overview:\n-\t**Model**: Simple blend of point-wise (DeBERTa-V3-Small & GraphCodeBERT-base….thanks Guo) & [pair-wise](https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training/notebook) ensemble just for diversity. Around 20% weightage to pair-wise.\n-\t**Data**: Use around 70% of data by careful selection of notebooks. No external data.\n-\t**Compute**: Free Kaggle & Google Colab. Rented A5000-24GB intermittently during last couple of weeks).\nLightweight models due to limited compute and the fact that I was only working on weekends:\n\n#### What worked:\n-\t**Core**: The core solution is well outlined in the public [kernel ](https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells)by Nguyen, the basic idea is to add a ‘context’ which helps the model rank the cells better.\n-\t**Pre-Processing 1**: Retaining the last few tokens in the markdown (i.e. last few words) helped in a small way. In other words, instead of simply truncating a lengthy markdown, we retain a few ending tokens. These last few words/tokens seem to be more intimately connected to the code cell that follows and helps the model make better connections.\n-\t**Pre-Processing 2**: Grabbing the class names & function names from the code cells and moving them to the top of the code cell (to avoid truncation) also helps slightly. Moving ‘all comments in the code cell’ to the top of the code cell didn’t help although I have a strong feeling this would have helped larger models (mine was only 768 tokens)\n-\t**Data Selection:** Kernels selected should be relevant & match with actual test data. This is important from 3 perspectives:\n** Fits into my compute budget\n** I can run more experiments\n** In absence of strong validation data, I felt this will play key role when actual results start rolling in\n****What helped:****\n** Kaggle increased memory size from 5-20 GB 3 years back. I could identify a bunch of old kernels by simply searching for markdowns on “You can write up to 5GB to\"\n** Identify R kernels, TF1 kernels etc which use old libraries\n** Identify common ancestors/parent kernels etc\n** Many times, kernels are not forked, however they are copied manually. So identify first few cells (say 2 markdowns and 3 code cells) and remove duplicates\n** When sampling take very few samples from above categories (however don’t make it 0%). For remaining kernels, take a healthy sample. \n** In all, I could go up to around 70% samples and yet get a decent result. This 70%, I further split into training and validation\n** After tuning, I trained on validation data as well\n** In past 3 months, as new data kept rolling in, each run has improved my score slightly\n\n-\t**Minor point during Ensembling**: A very minor (4th decimal) increase during ensemble time by checking if the last cell is a markdown. Pair-wise model does not rank such markdowns correctly (due to the way triplet generation happens). So if the last cell is a markdown, don’t average between pair-wise & point wise & instead just take the point-wise data.\n\n#### What didn’t work:\n-\tA variety of attention heads (at least 5) were tried & none gave a significantly better result than default one\n-\tBi-LSTM on top didn’t help either\n-\tIn pair-wise, Arcface, Cosface was used to sharpen the embeddings but didn’t show great results. In fact I had trouble in reproducing results consistently for this model\n-\tThe biggest item in the “didn’t work” category would be my experiences with BigBird and Longformer outlined in the ‘Additional context’ section.\n-\t2 costly mistakes: Wasted lot of token space as I didn’t catch this [bug](https://www.kaggle.com/competitions/AI4Code/discussion/343801#1896253). I felt this could have pushed up my score significantly. Secondly, last few days had to deal with memory errors. I am reasonably convinced that root cause could be [this](https://www.kaggle.com/competitions/AI4Code/discussion/343471).\n\n#### Additional context (on a failed experiment):\nHaving a good ‘context’ is the key to getting a good score (with this model). With 768 tokens one can squeeze in maybe 20 tokens for each code cell assuming an average of 30 code cells per kernel with a few tokens remaining for the markdown cell in question. 20 tokens covers barely 1-2 lines of code even after preprocessing. Is there a way to increase this ‘context’ without increasing the model size? I thought that both BigBird and Longformer, being sparse attention models would be the right fit to apply to the problem at hand. In Longformer, tokens attend to tokens in the vicinity defined by a window that slides as we move across the text during self-attention. The receptive field can be further increased by applying dilation to the local window (basically the token simply skips certain tokens) thus allowing attention to reach farther tokens. In addition, we also have global tokens which are attended to by everybody. Similarly Bigbird has a combination of local (sliding window), global & random attention. I felt AI4C domain was a classic case for usage of these models since the relationship between kernel cells can be captured beautifully given the constraints of limited token space:\n-\tThe sliding window will ensure to capture relation between 8-10 consecutive cells. These matter the most. Random tokens do the rest of the job.\n-\tGlobal tokens could be identified that attend to all cells. The tokens in the markdown cell would all be global & will attend to each and every code cell token.\n-\tThis way the ‘context’ becomes really really powerful. Relationship between the markdown in question and each and every code cell token can be established while program continuity aspect and relation between a code cell & the cells above and below it can be captured by the sliding window. Random attention further bring in unanticipated connections across cell tokens .\nHowever looks like both models perform well only in excess of 4096 tokens and do not work for anything less than that. Some experiments and discussion with HF on this aspect is captured [here](https://github.com/huggingface/transformers/issues/18234).\n\n#### Thanks and Acknowledgements:\nIt has been four months since the competition ended and I could not retrieve the exact empirical details I maintained, however based on the available notes/kernels, I have tried to give a gist of my experience w.r.t what worked and what didn’t. I apologize for not posting this earlier as I felt that I didn’t have anything significant to contribute in terms of ‘What worked’. However Mark Mc Donald’s template made me realize that perhaps I did have something to share as far as “What didn’t work” is concerned 😊\n\nI want to Thank the Kaggle staff team for coordinating this great competition and ensuring it goes off seamlessly over a timeframe of 6 months! I want to thank Nguyen for raising the bar high with the initial codebase he shared. This competition had leading industry experts, leading Kaggle grandmasters (many from top 20) & (maybe) folks from academia. I would like to Thank all of them for sharing their thoughts and views during the course of this competition. \n\n#### Team Members:\nAllohvk" } }