Many thanks to Kaggle and Google X for an exciting unique challenge. Congrats to all participants!
\nI used a point-wise ranking approach. Context for each markdown cell is dynamic, aiming to get as many meaningful tokens as possible. Specifically we reserve a maximum 128 tokens for markdown cell (query) and the rest goes for context. Context pool contains all code cells followed by all markdown cells except the query. Code cells in the context pool are always ordered, markdown cells are shuffled for each example. Sampling from a context pool is based on evenly distributed amounts of tokens taken from each cell. Specifically we start from 1 token per cell. If the total amount of sampled tokens is less than max length we try to take 2 tokens from each cell, and so on. Eventually if we cannot collect enough tokens padding is applied.
\nI tried several backbones and found GraphCodeBERT-base and CodeT5-base being the best ones. They have similar performance on 512 length while CodeT5 was slightly better in my experiments. CodeT5 with 1024 max length demonstrated significant improvement over 512 max length, about 0.014. Single checkpoint takes 4 hours for inference and gets the score around 0.8700. I got my final score with an ensemble of 2 checkpoints from different folds, but they still were undertrained so probably a single checkpoint can reach even higher.
\nI did not use any additional data. I believe the official training set is already representative enough. I used a validation set of 27k notebooks obtained by group split. CV scores correlate very well, being systematically 0.003 higher than LB. Source cleaning is minimalistic. I just removed extra spaces including all indents from the code to make it more compact.
\nAcknowledgment. Huge thanks to the TRC program for TPU resources.
\nBest of luck to all in the 2nd phase rerun!
\nHopefully our models will not encounter spherical endless notebooks especially in vacuum.
EDIT: Solution code
", "messages": [ { "id": "1895849", "postDate": "08/12/2022 12:03:05", "content": "Many thanks to Kaggle and Google X for an exciting unique challenge. Congrats to all participants!
\nI used a point-wise ranking approach. Context for each markdown cell is dynamic, aiming to get as many meaningful tokens as possible. Specifically we reserve a maximum 128 tokens for markdown cell (query) and the rest goes for context. Context pool contains all code cells followed by all markdown cells except the query. Code cells in the context pool are always ordered, markdown cells are shuffled for each example. Sampling from a context pool is based on evenly distributed amounts of tokens taken from each cell. Specifically we start from 1 token per cell. If the total amount of sampled tokens is less than max length we try to take 2 tokens from each cell, and so on. Eventually if we cannot collect enough tokens padding is applied.
\nI tried several backbones and found GraphCodeBERT-base and CodeT5-base being the best ones. They have similar performance on 512 length while CodeT5 was slightly better in my experiments. CodeT5 with 1024 max length demonstrated significant improvement over 512 max length, about 0.014. Single checkpoint takes 4 hours for inference and gets the score around 0.8700. I got my final score with an ensemble of 2 checkpoints from different folds, but they still were undertrained so probably a single checkpoint can reach even higher.
\nI did not use any additional data. I believe the official training set is already representative enough. I used a validation set of 27k notebooks obtained by group split. CV scores correlate very well, being systematically 0.003 higher than LB. Source cleaning is minimalistic. I just removed extra spaces including all indents from the code to make it more compact.
\nAcknowledgment. Huge thanks to the TRC program for TPU resources.
\nBest of luck to all in the 2nd phase rerun!
\nHopefully our models will not encounter spherical endless notebooks especially in vacuum.
EDIT: Solution code
", "rawMarkdown": "Many thanks to Kaggle and Google X for an exciting unique challenge. Congrats to all participants!\n\nI used a point-wise ranking approach. Context for each markdown cell is dynamic, aiming to get as many meaningful tokens as possible. Specifically we reserve a maximum 128 tokens for markdown cell (query) and the rest goes for context. Context pool contains all code cells followed by all markdown cells except the query. Code cells in the context pool are always ordered, markdown cells are shuffled for each example. Sampling from a context pool is based on evenly distributed amounts of tokens taken from each cell. Specifically we start from 1 token per cell. If the total amount of sampled tokens is less than max length we try to take 2 tokens from each cell, and so on. Eventually if we cannot collect enough tokens padding is applied.\n\nI tried several backbones and found GraphCodeBERT-base and CodeT5-base being the best ones. They have similar performance on 512 length while CodeT5 was slightly better in my experiments. CodeT5 with 1024 max length demonstrated significant improvement over 512 max length, about 0.014. Single checkpoint takes 4 hours for inference and gets the score around 0.8700. I got my final score with an ensemble of 2 checkpoints from different folds, but they still were undertrained so probably a single checkpoint can reach even higher.\n\nI did not use any additional data. I believe the official training set is already representative enough. I used a validation set of 27k notebooks obtained by group split. CV scores correlate very well, being systematically 0.003 higher than LB. Source cleaning is minimalistic. I just removed extra spaces including all indents from the code to make it more compact.\n\nAcknowledgment. Huge thanks to the [TRC](https://sites.research.google/trc/about/) program for TPU resources.\n\nBest of luck to all in the 2nd phase rerun! \nHopefully our models will not encounter spherical endless notebooks especially in vacuum.\n\nEDIT: [Solution code](https://github.com/vecxoz/ai4code)", "votes": null }, { "id": "1896140", "postDate": "08/12/2022 16:03:11", "content": "Good to see a CodeT5 model(encoder side). Congrats on your strong finish. Did you find any major benefits in putting the (unordered) markdowns in the context?
", "rawMarkdown": "Good to see a CodeT5 model(encoder side). Congrats on your strong finish. Did you find any major benefits in putting the (unordered) markdowns in the context?", "votes": null }, { "id": "1896171", "postDate": "08/12/2022 16:19:15", "content": "Thanks and congrats! Addition of markdown cells into context gave me a significant improvement (don't remember how much exactly). I shuffle them because during inference their order is unknown.
", "rawMarkdown": "Thanks and congrats! Addition of markdown cells into context gave me a significant improvement (don't remember how much exactly). I shuffle them because during inference their order is unknown.", "votes": null }, { "id": "1896537", "postDate": "08/12/2022 22:19:02", "content": "Congratulations (мої вітання)! Переходжу на англиську, щоб інші учасники змагання змогли зрозуміти.
\nWe also tried to train model with a higher max length (1024), but we don't have much time. It would probably improve us as you from your experiments by a lot.
\nThank you for the great solutions!
\"TPU\" then I guess your code is based on TensorFlow or Jax?
", "rawMarkdown": "\"TPU\" then I guess your code is based on TensorFlow or Jax?", "votes": null }, { "id": "1896768", "postDate": "08/13/2022 06:22:11", "content": "Thank you for your comment. I was wondering why this arrangement works till it finally stuck me just now. When we do a self-attention, every token attends to every other token and tries to build relationships. By putting the markdowns into the context, we give more information to model to build these relationships. Relationships are further \"strengthened\" by adding the positional embedding. Sure, these markdowns are unordered and so the strengthening does not happen but definitely the first part happens (building of relations). This leads to stronger score, I suspect.
\nVery strong innovation from your side!
Thanks! Yes, I had similar thoughts.
", "rawMarkdown": "Thanks! Yes, I had similar thoughts.", "votes": null }, { "id": "1896908", "postDate": "08/13/2022 09:03:12", "content": "Thanks a lot and congrats! Each competition is a great space for learning, especially when solving complicated tasks like this one. Good luck in the next competitions!
", "rawMarkdown": "Thanks a lot and congrats! Each competition is a great space for learning, especially when solving complicated tasks like this one. Good luck in the next competitions!", "votes": null }, { "id": "1896913", "postDate": "08/13/2022 09:04:22", "content": "Congrats on the high spot! Yes, 100% Tensorflow.
", "rawMarkdown": "Congrats on the high spot! Yes, 100% Tensorflow.", "votes": null }, { "id": "1899080", "postDate": "08/15/2022 04:00:16", "content": "Hi, amazing approach, and congrats on the strong finish!
\nI am not sure I understood exactly how you structured the problem at the end.
\nWhat exactly is the target of the model? Is it training on a single cell (indicated by the query) to predict a simple integer as the rank of that cell on the notebook?
\nAlso, do you plan to release the code? It is interesting to see how you actually implemented this.
Thanks and congrats again.
\nThe Devastator.
Thanks and congrats! Basically you are correct. I trained the model to predict the relative rank of each markdown cell within any given notebook. For example, say our notebook has 4 markdown cells and 4 code cells, and we want to create a training example for the very first markdown cell. In this case our training example would look like the following:
\n[md1] + [sep] + [code1][code2][code3][code4][md4][md2][md3] and target rank is 0.0
\nI plan to release the code, but not very soon. Need time for some cleaning. I will update this thread when it is ready.
", "rawMarkdown": "Thanks and congrats! Basically you are correct. I trained the model to predict the relative rank of each markdown cell within any given notebook. For example, say our notebook has 4 markdown cells and 4 code cells, and we want to create a training example for the very first markdown cell. In this case our training example would look like the following:\n\n[md1] + [sep] + [code1][code2][code3][code4][md4][md2][md3] and target rank is 0.0\n\nI plan to release the code, but not very soon. Need time for some cleaning. I will update this thread when it is ready.", "votes": null }, { "id": "2241181", "postDate": "05/01/2023 09:06:29", "content": "Recently returned to this project to run some experiments. Really great dataset.
\nSorry, it’s a bit late, but still, here’s my solution code. It’s a minimalist TF/TPU pipeline. @thedevastator
Good to see a CodeT5 model(encoder side). Congrats on your strong finish. Did you find any major benefits in putting the (unordered) markdowns in the context?
", "votes": null, "replies": [ { "id": 1896171, "author_name": "vecxoz", "author_url": "", "post_date": "08/12/2022 16:19:15", "content": "Thanks and congrats! Addition of markdown cells into context gave me a significant improvement (don't remember how much exactly). I shuffle them because during inference their order is unknown.
", "votes": null, "replies": [] }, { "id": 1896768, "author_name": "allohvk", "author_url": "", "post_date": "08/13/2022 06:22:11", "content": "Thank you for your comment. I was wondering why this arrangement works till it finally stuck me just now. When we do a self-attention, every token attends to every other token and tries to build relationships. By putting the markdowns into the context, we give more information to model to build these relationships. Relationships are further \"strengthened\" by adding the positional embedding. Sure, these markdowns are unordered and so the strengthening does not happen but definitely the first part happens (building of relations). This leads to stronger score, I suspect.
\nVery strong innovation from your side!
Thanks! Yes, I had similar thoughts.
", "votes": null, "replies": [] } ] }, { "id": 1896537, "author_name": "vad13irt", "author_url": "", "post_date": "08/12/2022 22:19:02", "content": "Congratulations (мої вітання)! Переходжу на англиську, щоб інші учасники змагання змогли зрозуміти.
\nWe also tried to train model with a higher max length (1024), but we don't have much time. It would probably improve us as you from your experiments by a lot.
\nThank you for the great solutions!
Thanks a lot and congrats! Each competition is a great space for learning, especially when solving complicated tasks like this one. Good luck in the next competitions!
", "votes": null, "replies": [] } ] }, { "id": 1896603, "author_name": "snaker", "author_url": "", "post_date": "08/13/2022 00:59:22", "content": "\"TPU\" then I guess your code is based on TensorFlow or Jax?
", "votes": null, "replies": [ { "id": 1896913, "author_name": "vecxoz", "author_url": "", "post_date": "08/13/2022 09:04:22", "content": "Congrats on the high spot! Yes, 100% Tensorflow.
", "votes": null, "replies": [] } ] }, { "id": 1899080, "author_name": "thedevastator", "author_url": "", "post_date": "08/15/2022 04:00:16", "content": "Hi, amazing approach, and congrats on the strong finish!
\nI am not sure I understood exactly how you structured the problem at the end.
\nWhat exactly is the target of the model? Is it training on a single cell (indicated by the query) to predict a simple integer as the rank of that cell on the notebook?
\nAlso, do you plan to release the code? It is interesting to see how you actually implemented this.
Thanks and congrats again.
\nThe Devastator.
Thanks and congrats! Basically you are correct. I trained the model to predict the relative rank of each markdown cell within any given notebook. For example, say our notebook has 4 markdown cells and 4 code cells, and we want to create a training example for the very first markdown cell. In this case our training example would look like the following:
\n[md1] + [sep] + [code1][code2][code3][code4][md4][md2][md3] and target rank is 0.0
\nI plan to release the code, but not very soon. Need time for some cleaning. I will update this thread when it is ready.
", "votes": null, "replies": [] }, { "id": 2241181, "author_name": "vecxoz", "author_url": "", "post_date": "05/01/2023 09:06:29", "content": "Recently returned to this project to run some experiments. Really great dataset.
\nSorry, it’s a bit late, but still, here’s my solution code. It’s a minimalist TF/TPU pipeline. @thedevastator