Hi all,
\nI had a great experience participating in this competition and would like to thank the organizers, Kaggle staff, everyone who shared their insights on the discussion boards and in the public notebooks, and most importantly - my amazing teammates.
Background
\nI wasn’t intending to seriously participate in this competition, due to a lack of free time during the summer. However, by late July I came up with a direction that seemed appealing due to being lightweight in terms of development time, training time and inference time. Therefore, I decided to pursue it and see what happens. When I saw that the results are promising, I teamed up with a few other participants. After teaming up, I focused mainly on ensembling the team’s models. I present below some information about my single model, my ensembling method, the team solution, the tools that I used and some of my work process.
Single Model
\nI implemented a Bi-encoder model that, given a markdown cell and a text cell, predicts whether they are adjacent or not, based on the cosine similarity of the encoded cells. Here are some details.
Ensembling
\nFirst of all, I found it helpful to combine the fractional predictions rather than the ranks as they appear in the submission file, as the quantization to integer ranks introduces bias. To calculate the model weights, I optimize for a maximal Kendall Tau value by using the Nelder-Mead method. On top of this, I also implemented an adaptive scheme where each individual prediction is further weighted by a confidence score that is based on how sharp its logit distribution is. The The improvement from this adaptive weighting scheme over a simple linear combination is 0.0037.
Team Solution
\nOur best solution (LB 0.8824) is two of my bi-encoder models (CodeBert, GraphCodeBert) ensembled with a single pointwise Deberta-v3-base model that my teammates had developed.
Tooling
\nWork Methodology
\nConsistent preprocessing of data between training and inference was achieved by:
Procedure for testing new ideas:
\nThanks for reading and good luck to all participants in the upcoming rerun!
", "messages": [ { "id": "1895096", "postDate": "08/12/2022 00:40:57", "content": "Hi all,
\nI had a great experience participating in this competition and would like to thank the organizers, Kaggle staff, everyone who shared their insights on the discussion boards and in the public notebooks, and most importantly - my amazing teammates.
Background
\nI wasn’t intending to seriously participate in this competition, due to a lack of free time during the summer. However, by late July I came up with a direction that seemed appealing due to being lightweight in terms of development time, training time and inference time. Therefore, I decided to pursue it and see what happens. When I saw that the results are promising, I teamed up with a few other participants. After teaming up, I focused mainly on ensembling the team’s models. I present below some information about my single model, my ensembling method, the team solution, the tools that I used and some of my work process.
Single Model
\nI implemented a Bi-encoder model that, given a markdown cell and a text cell, predicts whether they are adjacent or not, based on the cosine similarity of the encoded cells. Here are some details.
Ensembling
\nFirst of all, I found it helpful to combine the fractional predictions rather than the ranks as they appear in the submission file, as the quantization to integer ranks introduces bias. To calculate the model weights, I optimize for a maximal Kendall Tau value by using the Nelder-Mead method. On top of this, I also implemented an adaptive scheme where each individual prediction is further weighted by a confidence score that is based on how sharp its logit distribution is. The The improvement from this adaptive weighting scheme over a simple linear combination is 0.0037.
Team Solution
\nOur best solution (LB 0.8824) is two of my bi-encoder models (CodeBert, GraphCodeBert) ensembled with a single pointwise Deberta-v3-base model that my teammates had developed.
Tooling
\nWork Methodology
\nConsistent preprocessing of data between training and inference was achieved by:
Procedure for testing new ideas:
\nThanks for reading and good luck to all participants in the upcoming rerun!
", "rawMarkdown": "Hi all,\nI had a great experience participating in this competition and would like to thank the organizers, Kaggle staff, everyone who shared their insights on the discussion boards and in the public notebooks, and most importantly - my amazing teammates.\n\n**Background**\nI wasn’t intending to seriously participate in this competition, due to a lack of free time during the summer. However, by late July I came up with a direction that seemed appealing due to being lightweight in terms of development time, training time and inference time. Therefore, I decided to pursue it and see what happens. When I saw that the results are promising, I teamed up with a few other participants. After teaming up, I focused mainly on ensembling the team’s models. I present below some information about my single model, my ensembling method, the team solution, the tools that I used and some of my work process.\n\n**Single Model**\nI implemented a Bi-encoder model that, given a markdown cell and a text cell, predicts whether they are adjacent or not, based on the cosine similarity of the encoded cells. Here are some details.\n- Loss function: multiple negatives ranking loss, with additional hard negatives sampled from the same notebook as the positive example.\n- *Main modeling trick*: I added fake code cells to represent the beginning and end of notebook, in order to better predict for introduction and summary markdown cells (LB improvement 0.01)\n- Inference time improvement: Loaded the input data in chunks of 50 notebooks, sorted the cells by their text length to minimize the time being spent encoding pad tokens. Ran the encoding with a batch size of 128.\n- Dataset: 80k notebooks, 90/10 train/test split\n- Sequence length: 256 tokens for each cell, so 256 markdown and 256 code \n- Data preprocessing: None!\n- Number of training epochs: 3\n- Training runtime: 17h on a A6000, \n- Inference runtime: 1h20m\n- Single model performance: Public and Private LB both at 0.8555 with a GraphCodeBert backbone.\n\n**Ensembling**\nFirst of all, I found it helpful to combine the fractional predictions rather than the ranks as they appear in the submission file, as the quantization to integer ranks introduces bias. To calculate the model weights, I optimize for a maximal Kendall Tau value by using the Nelder-Mead method. On top of this, I also implemented an adaptive scheme where each individual prediction is further weighted by a confidence score that is based on how sharp its logit distribution is. The The improvement from this adaptive weighting scheme over a simple linear combination is 0.0037.\n\n**Team Solution**\nOur best solution (LB 0.8824) is two of my bi-encoder models (CodeBert, GraphCodeBert) ensembled with a single pointwise Deberta-v3-base model that my teammates had developed.\n\n**Tooling**\n- Experiment tracking with W&B\n- Configuration management with Hydra\n- Team communication via Discord\n- Code version management with Github\n- PyCharm for development\n- Model training on Kaggle, Colab and eventually JarvisLabs\n\n**Work Methodology**\nConsistent preprocessing of data between training and inference was achieved by:\n1. Packaging my GitHub project as a Pip Wheel file, saving it as a dataset and then using it to run inference. The inference notebook doesn’t have any Python code in it, it only invokes a script from the package.\n2. The training script saves the hyperparameters alongside the model weights. In inference, this hyperparameters file is being applied to the configuration.\n3. The training script and the inference code packaging script both save a git commit hash, this allows me to make sure that the versions are aligned.\n\nProcedure for testing new ideas:\n1. Prototyping In a feature branch, to avoid introducing problems into the main code. In Git, it can be hard to undo a bad commit that is far back in the log.\n2. Running in the IDE line by line to debug.\n3. Training on a small dataset first to see if the results are in a reasonable range.\n4. Running a prediction analysis notebook that checks for prediction error by: language, notebook length, true rank quantile of cell, cell text length, etc.\n\nThanks for reading and good luck to all participants in the upcoming rerun!", "votes": null }, { "id": "1895137", "postDate": "08/12/2022 01:35:06", "content": "Thanks for your kindly sharing，I have learned a lot from your work methodology
", "rawMarkdown": "Thanks for your kindly sharing，I have learned a lot from your work methodology", "votes": null }, { "id": "1895161", "postDate": "08/12/2022 02:02:48", "content": "Very nice all around!
", "rawMarkdown": "Very nice all around!", "votes": null }, { "id": "1906421", "postDate": "08/19/2022 21:33:47", "content": "I want to say that this was an excellent post and very informative.
\nI particularly liked hearing about your work methodology and how you test new ideas. This is something that I think is very important, but often overlooked.
\nGreat job on your success in the competition and thanks for sharing!
The Devastator.
", "rawMarkdown": "I want to say that this was an excellent post and very informative.\nI particularly liked hearing about your work methodology and how you test new ideas. This is something that I think is very important, but often overlooked. \nGreat job on your success in the competition and thanks for sharing!\n\nThe Devastator.", "votes": null } ], "comments": [ { "id": 1895137, "author_name": "liampay", "author_url": "", "post_date": "08/12/2022 01:35:06", "content": "Thanks for your kindly sharing，I have learned a lot from your work methodology
", "votes": null, "replies": [] }, { "id": 1895161, "author_name": "happycube", "author_url": "", "post_date": "08/12/2022 02:02:48", "content": "Very nice all around!
", "votes": null, "replies": [] }, { "id": 1906421, "author_name": "thedevastator", "author_url": "", "post_date": "08/19/2022 21:33:47", "content": "I want to say that this was an excellent post and very informative.
\nI particularly liked hearing about your work methodology and how you test new ideas. This is something that I think is very important, but often overlooked.
\nGreat job on your success in the competition and thanks for sharing!
The Devastator.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1895096": "Hi all,\nI had a great experience participating in this competition and would like to thank the organizers, Kaggle staff, everyone who shared their insights on the discussion boards and in the public notebooks, and most importantly - my amazing teammates.\n\n**Background**\nI wasn’t intending to seriously participate in this competition, due to a lack of free time during the summer. However, by late July I came up with a direction that seemed appealing due to being lightweight in terms of development time, training time and inference time. Therefore, I decided to pursue it and see what happens. When I saw that the results are promising, I teamed up with a few other participants. After teaming up, I focused mainly on ensembling the team’s models. I present below some information about my single model, my ensembling method, the team solution, the tools that I used and some of my work process.\n\n**Single Model**\nI implemented a Bi-encoder model that, given a markdown cell and a text cell, predicts whether they are adjacent or not, based on the cosine similarity of the encoded cells. Here are some details.\n- Loss function: multiple negatives ranking loss, with additional hard negatives sampled from the same notebook as the positive example.\n- *Main modeling trick*: I added fake code cells to represent the beginning and end of notebook, in order to better predict for introduction and summary markdown cells (LB improvement 0.01)\n- Inference time improvement: Loaded the input data in chunks of 50 notebooks, sorted the cells by their text length to minimize the time being spent encoding pad tokens. Ran the encoding with a batch size of 128.\n- Dataset: 80k notebooks, 90/10 train/test split\n- Sequence length: 256 tokens for each cell, so 256 markdown and 256 code \n- Data preprocessing: None!\n- Number of training epochs: 3\n- Training runtime: 17h on a A6000, \n- Inference runtime: 1h20m\n- Single model performance: Public and Private LB both at 0.8555 with a GraphCodeBert backbone.\n\n**Ensembling**\nFirst of all, I found it helpful to combine the fractional predictions rather than the ranks as they appear in the submission file, as the quantization to integer ranks introduces bias. To calculate the model weights, I optimize for a maximal Kendall Tau value by using the Nelder-Mead method. On top of this, I also implemented an adaptive scheme where each individual prediction is further weighted by a confidence score that is based on how sharp its logit distribution is. The The improvement from this adaptive weighting scheme over a simple linear combination is 0.0037.\n\n**Team Solution**\nOur best solution (LB 0.8824) is two of my bi-encoder models (CodeBert, GraphCodeBert) ensembled with a single pointwise Deberta-v3-base model that my teammates had developed.\n\n**Tooling**\n- Experiment tracking with W&B\n- Configuration management with Hydra\n- Team communication via Discord\n- Code version management with Github\n- PyCharm for development\n- Model training on Kaggle, Colab and eventually JarvisLabs\n\n**Work Methodology**\nConsistent preprocessing of data between training and inference was achieved by:\n1. Packaging my GitHub project as a Pip Wheel file, saving it as a dataset and then using it to run inference. The inference notebook doesn’t have any Python code in it, it only invokes a script from the package.\n2. The training script saves the hyperparameters alongside the model weights. In inference, this hyperparameters file is being applied to the configuration.\n3. The training script and the inference code packaging script both save a git commit hash, this allows me to make sure that the versions are aligned.\n\nProcedure for testing new ideas:\n1. Prototyping In a feature branch, to avoid introducing problems into the main code. In Git, it can be hard to undo a bad commit that is far back in the log.\n2. Running in the IDE line by line to debug.\n3. Training on a small dataset first to see if the results are in a reasonable range.\n4. Running a prediction analysis notebook that checks for prediction error by: language, notebook length, true rank quantile of cell, cell text length, etc.\n\nThanks for reading and good luck to all participants in the upcoming rerun!", "1895137": "Thanks for your kindly sharing，I have learned a lot from your work methodology", "1895161": "Very nice all around!", "1906421": "I want to say that this was an excellent post and very informative.\nI particularly liked hearing about your work methodology and how you test new ideas. This is something that I think is very important, but often overlooked. \nGreat job on your success in the competition and thanks for sharing!\n\nThe Devastator." }, "source": "existing_discussion", "complete": true, "competition_id": 35887, "competition_slug": "AI4Code", "team_id": 8796179, "solution_url": "/competitions/AI4Code/writeups/punch-cards-are-all-you-need-my-part-of-team-s-sol" }