![\"alt](\"https://raw.githubusercontent.com/Shujun-He/Google_AI4code_6th_place_solution/main/graphics/relative2absolute.png\")
First, thanks to the organizers for hosting a challenging and interesting competition. Next, congrats to all winners! I placed better than I thought I would but unfortunately, I still placed right outside of the prize zone. Luckily, there is still $1000 prize for top 10 teams.
\nMy inference code is released at: https://www.kaggle.com/code/shujun717/add-decorator-ai4code-test32-add-break-26-code?scriptVersionId=103045432
\nMy training code is released at: https://github.com/Shujun-He/Google_AI4code_6th_place_solution
Please see below for details of my solution:
\n
(I typed this in latex and just screenshot it. Sorry for the quality. For a better version, see write_up_6th_solution.pdf in my code repo)
My network architecture consists mainly of a deberta encoder and a GRU layer; an overview is shown in the Figure below. First, markdown blocks and code blocks are tokenized and concatenated. During training, 10 markdown blocks with 20 code blocks with a max length of 1100 are used first, then 10 markdown blocks with 40 code blocks with a max length of 1600 are used to train another 5 epochs, and then the weights are retrained again with 20 markdown blocks with 80 code blocks with a max length of 3160. Following concatenation of markdown and cell blocks, a deberta large encoder is used to encode the text, and then each markdown and code block is pooled. Next, features are concatenated to pooled feature vectors (discussed in a later section). Each markdown block feature vector is then concatenated to the sequence of code block feature vectors, and then an GRU network is used to predict the relative position of that markdown block to each code blocks. Lastly, the absolute position of the markdown cell is computed according to Equation 1. MAE loss is used during training.
\n![\"alt](\"https://raw.githubusercontent.com/Shujun-He/Google_AI4code_6th_place_solution/main/graphics/architecture.png\")
The following features are used:
\nThese features are concatenated with pooled feature vectors of markdown cells. Pooled code cells feature vectors are also concatenated with their absolute positions.
\nMasked langauge modeling pretraining is used, where the input text is arranged in the same manner as in finetuning. MLM probability was set to 0.15 and the model is trained for 10 epochs. A max sequence length of 768 is used with at most 5 markdown cells and 10 code cells.
\nDuring inference, I tried to predict positions of as many markdown cells as possible. I was able to use a max length of 4250 with 100 code blocks and 30 markdown blocks. Code blocks have at most 26 tokens and markdown blocks 64. Notebooks are sorted based on number of markdown/cell blocks to reduce inference time. Because for longer sequences where there are more than 30 markdown blocks, I cannot make predictions for all of them at the same time, and multiple forward passes are needed to make predictions for all markdown blocks. As a result, the model never sees all the markdown blocks at the same time. To counteract that, I do test time augmentation by randomly shuffling the markdown cells each time a new inference cycle is started and therefore the model will see different combinations of markdown cells in long sequences. For shorter sequences, this simply changes the order of markdown cells during inference. For my best inference notebook, a total of 3 inference cycles are used (3xTTA). 3xTTA improves from score from 0.9067 to 0.9100.
\nI used the solution from Khoi Nguyen as a baseline: https://www.kaggle.com/competitions/AI4Code/discussion/326970.
", "messages": [ { "id": "2043578", "postDate": "11/25/2022 19:13:24", "content": "First, thanks to the organizers for hosting a challenging and interesting competition. Next, congrats to all winners! I placed better than I thought I would but unfortunately, I still placed right outside of the prize zone. Luckily, there is still $1000 prize for top 10 teams.
\nMy inference code is released at: https://www.kaggle.com/code/shujun717/add-decorator-ai4code-test32-add-break-26-code?scriptVersionId=103045432
\nMy training code is released at: https://github.com/Shujun-He/Google_AI4code_6th_place_solution
Please see below for details of my solution:
\n
(I typed this in latex and just screenshot it. Sorry for the quality. For a better version, see write_up_6th_solution.pdf in my code repo)
My network architecture consists mainly of a deberta encoder and a GRU layer; an overview is shown in the Figure below. First, markdown blocks and code blocks are tokenized and concatenated. During training, 10 markdown blocks with 20 code blocks with a max length of 1100 are used first, then 10 markdown blocks with 40 code blocks with a max length of 1600 are used to train another 5 epochs, and then the weights are retrained again with 20 markdown blocks with 80 code blocks with a max length of 3160. Following concatenation of markdown and cell blocks, a deberta large encoder is used to encode the text, and then each markdown and code block is pooled. Next, features are concatenated to pooled feature vectors (discussed in a later section). Each markdown block feature vector is then concatenated to the sequence of code block feature vectors, and then an GRU network is used to predict the relative position of that markdown block to each code blocks. Lastly, the absolute position of the markdown cell is computed according to Equation 1. MAE loss is used during training.
\n
The following features are used:
\nThese features are concatenated with pooled feature vectors of markdown cells. Pooled code cells feature vectors are also concatenated with their absolute positions.
\nMasked langauge modeling pretraining is used, where the input text is arranged in the same manner as in finetuning. MLM probability was set to 0.15 and the model is trained for 10 epochs. A max sequence length of 768 is used with at most 5 markdown cells and 10 code cells.
\nDuring inference, I tried to predict positions of as many markdown cells as possible. I was able to use a max length of 4250 with 100 code blocks and 30 markdown blocks. Code blocks have at most 26 tokens and markdown blocks 64. Notebooks are sorted based on number of markdown/cell blocks to reduce inference time. Because for longer sequences where there are more than 30 markdown blocks, I cannot make predictions for all of them at the same time, and multiple forward passes are needed to make predictions for all markdown blocks. As a result, the model never sees all the markdown blocks at the same time. To counteract that, I do test time augmentation by randomly shuffling the markdown cells each time a new inference cycle is started and therefore the model will see different combinations of markdown cells in long sequences. For shorter sequences, this simply changes the order of markdown cells during inference. For my best inference notebook, a total of 3 inference cycles are used (3xTTA). 3xTTA improves from score from 0.9067 to 0.9100.
\nI used the solution from Khoi Nguyen as a baseline: https://www.kaggle.com/competitions/AI4Code/discussion/326970.
", "rawMarkdown": "First, thanks to the organizers for hosting a challenging and interesting competition. Next, congrats to all winners! I placed better than I thought I would but unfortunately, I still placed right outside of the prize zone. Luckily, there is still $1000 prize for top 10 teams.\n\nMy inference code is released at: https://www.kaggle.com/code/shujun717/add-decorator-ai4code-test32-add-break-26-code?scriptVersionId=103045432\nMy training code is released at: https://github.com/Shujun-He/Google_AI4code_6th_place_solution\n\nPlease see below for details of my solution:\n\n### Relative to absolute position prediction\n\n\n\n\n(I typed this in latex and just screenshot it. Sorry for the quality. For a better version, see `write_up_6th_solution.pdf` in my code repo)\n\n### Network achitecture\nMy network architecture consists mainly of a deberta encoder and a GRU layer; an overview is shown in the Figure below. First, markdown blocks and code blocks are tokenized and concatenated. During training, 10 markdown blocks with 20 code blocks with a max length of 1100 are used first, then 10 markdown blocks with 40 code blocks with a max length of 1600 are used to train another 5 epochs, and then the weights are retrained again with 20 markdown blocks with 80 code blocks with a max length of 3160. Following concatenation of markdown and cell blocks, a deberta large encoder is used to encode the text, and then each markdown and code block is pooled. Next, features are concatenated to pooled feature vectors (discussed in a later section). Each markdown block feature vector is then concatenated to the sequence of code block feature vectors, and then an GRU network is used to predict the relative position of that markdown block to each code blocks. Lastly, the absolute position of the markdown cell is computed according to Equation 1. MAE loss is used during training.\n\n\n\n### Features\n\nThe following features are used:\n\n1. number of markdown cells in given notebook (normalized by 128)\n2. number of code cells in given notebook (normalized by 128)\n3. ratio of markdown cells to total number of cells\n\nThese features are concatenated with pooled feature vectors of markdown cells. Pooled code cells feature vectors are also concatenated with their absolute positions.\n\n### MLM\nMasked langauge modeling pretraining is used, where the input text is arranged in the same manner as in finetuning. MLM probability was set to 0.15 and the model is trained for 10 epochs. A max sequence length of 768 is used with at most 5 markdown cells and 10 code cells.\n\n### Inference and TTA\nDuring inference, I tried to predict positions of as many markdown cells as possible. I was able to use a max length of 4250 with 100 code blocks and 30 markdown blocks. Code blocks have at most 26 tokens and markdown blocks 64. Notebooks are sorted based on number of markdown/cell blocks to reduce inference time. Because for longer sequences where there are more than 30 markdown blocks, I cannot make predictions for all of them at the same time, and multiple forward passes are needed to make predictions for all markdown blocks. As a result, the model never sees all the markdown blocks at the same time. To counteract that, I do test time augmentation by randomly shuffling the markdown cells each time a new inference cycle is started and therefore the model will see different combinations of markdown cells in long sequences. For shorter sequences, this simply changes the order of markdown cells during inference. For my best inference notebook, a total of 3 inference cycles are used (3xTTA). 3xTTA improves from score from 0.9067 to 0.9100.\n\n### What didn’t work\n1. Trying to translate markdown cells that are not in English did not result in any improve-\nments for me. I tried facebook’s m2m100 418M.\n2. Separating code and markdown cells and doing two forward passes through deberta for\nconcatenated code and markdown cells.\n3. Cubert did not work because I couldn’t fix the tokenizer.\n4. Maxpooling instead of meanpooling\n\n### References\nI used the solution from Khoi Nguyen as a baseline: https://www.kaggle.com/competitions/AI4Code/discussion/326970.", "votes": null }, { "id": "2043579", "postDate": "11/25/2022 19:14:35", "content": "Great result @shujun717, this is a great approach note too. Congratulations and best regards!
", "rawMarkdown": "Great result @shujun717, this is a great approach note too. Congratulations and best regards!", "votes": null } ], "comments": [ { "id": 2043579, "author_name": "ravi20076", "author_url": "", "post_date": "11/25/2022 19:14:35", "content": "Great result @shujun717, this is a great approach note too. Congratulations and best regards!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2043578": "First, thanks to the organizers for hosting a challenging and interesting competition. Next, congrats to all winners! I placed better than I thought I would but unfortunately, I still placed right outside of the prize zone. Luckily, there is still $1000 prize for top 10 teams.\n\nMy inference code is released at: https://www.kaggle.com/code/shujun717/add-decorator-ai4code-test32-add-break-26-code?scriptVersionId=103045432\nMy training code is released at: https://github.com/Shujun-He/Google_AI4code_6th_place_solution\n\nPlease see below for details of my solution:\n\n### Relative to absolute position prediction\n\n\n\n\n(I typed this in latex and just screenshot it. Sorry for the quality. For a better version, see `write_up_6th_solution.pdf` in my code repo)\n\n### Network achitecture\nMy network architecture consists mainly of a deberta encoder and a GRU layer; an overview is shown in the Figure below. First, markdown blocks and code blocks are tokenized and concatenated. During training, 10 markdown blocks with 20 code blocks with a max length of 1100 are used first, then 10 markdown blocks with 40 code blocks with a max length of 1600 are used to train another 5 epochs, and then the weights are retrained again with 20 markdown blocks with 80 code blocks with a max length of 3160. Following concatenation of markdown and cell blocks, a deberta large encoder is used to encode the text, and then each markdown and code block is pooled. Next, features are concatenated to pooled feature vectors (discussed in a later section). Each markdown block feature vector is then concatenated to the sequence of code block feature vectors, and then an GRU network is used to predict the relative position of that markdown block to each code blocks. Lastly, the absolute position of the markdown cell is computed according to Equation 1. MAE loss is used during training.\n\n\n\n### Features\n\nThe following features are used:\n\n1. number of markdown cells in given notebook (normalized by 128)\n2. number of code cells in given notebook (normalized by 128)\n3. ratio of markdown cells to total number of cells\n\nThese features are concatenated with pooled feature vectors of markdown cells. Pooled code cells feature vectors are also concatenated with their absolute positions.\n\n### MLM\nMasked langauge modeling pretraining is used, where the input text is arranged in the same manner as in finetuning. MLM probability was set to 0.15 and the model is trained for 10 epochs. A max sequence length of 768 is used with at most 5 markdown cells and 10 code cells.\n\n### Inference and TTA\nDuring inference, I tried to predict positions of as many markdown cells as possible. I was able to use a max length of 4250 with 100 code blocks and 30 markdown blocks. Code blocks have at most 26 tokens and markdown blocks 64. Notebooks are sorted based on number of markdown/cell blocks to reduce inference time. Because for longer sequences where there are more than 30 markdown blocks, I cannot make predictions for all of them at the same time, and multiple forward passes are needed to make predictions for all markdown blocks. As a result, the model never sees all the markdown blocks at the same time. To counteract that, I do test time augmentation by randomly shuffling the markdown cells each time a new inference cycle is started and therefore the model will see different combinations of markdown cells in long sequences. For shorter sequences, this simply changes the order of markdown cells during inference. For my best inference notebook, a total of 3 inference cycles are used (3xTTA). 3xTTA improves from score from 0.9067 to 0.9100.\n\n### What didn’t work\n1. Trying to translate markdown cells that are not in English did not result in any improve-\nments for me. I tried facebook’s m2m100 418M.\n2. Separating code and markdown cells and doing two forward passes through deberta for\nconcatenated code and markdown cells.\n3. Cubert did not work because I couldn’t fix the tokenizer.\n4. Maxpooling instead of meanpooling\n\n### References\nI used the solution from Khoi Nguyen as a baseline: https://www.kaggle.com/competitions/AI4Code/discussion/326970.", "2043579": "Great result @shujun717, this is a great approach note too. Congratulations and best regards!" }, "source": "meta" }