\n
![\"Nested](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2202881%2Faad7b5931ba26287e467c535a69d595a%2Fnested_tfm.png?generation=1660286753619476&alt=media\")
We called our network architecture \"Nested Transformers\", since it contains two levels of transformers:
\nSource codes: https://github.com/ShinSiangChoong/kaggle_ai4code_nested_transformers
\n<code><sep><source_tokens> OR <mark><sep><source_tokens>We would like to thank the competition host(s) and Kaggle for organizing this interesting competition.
\n@css919
\n@erniechiew
\n@hexomiter
We called our network architecture \"Nested Transformers\", since it contains two levels of transformers:
\nSource codes: https://github.com/ShinSiangChoong/kaggle_ai4code_nested_transformers
\n<code><sep><source_tokens> OR <mark><sep><source_tokens>We would like to thank the competition host(s) and Kaggle for organizing this interesting competition.
\n@css919
\n@erniechiew
\n@hexomiter
` OR ``\n- Thanks @cabbage972 for verifying the usefulness of this idea. Please refer to this [comment](https://www.kaggle.com/competitions/AI4Code/discussion/343680#1907307). \n\n#### Cell Features: \n- The code cell order is known, while the markdown order is unknown, hence a straightforward positional embedding scheme in the notebook transformer won't work. The cell features aim to inform the model about the known code cell positions/order.\n- We use 2 features for each cell, i.e. cell type (code -> 1, markdown -> 0) and percentile rank among the code cells: If the cell is a markdown cell, this feature = 0\n- The 2 features are \"expanded\" by FC layers, such that they are represented as an embedding that has the same dimension with that of the cell token embeddings. \n- These cell feature embeddings are then concatenated with the token embeddings and passed into the Aggregation Attention Layer together.\n\n#### Aggregation Attention\n- We believe different tokens (and the cell features) have different importance to represent the cell. An aggregation attention layer is used to learn the weights of each token embedding (and the cell feature embedding). \n- The weights are normalized by a Softmax layer, such that the final cell embedding is a weighted average of its token and feature embeddings.\n- Thanks @cabbage972 for verifying the usefulness of this idea. Please refer to this [comment](https://www.kaggle.com/competitions/AI4Code/discussion/343680#1909812). \n\n#### Double Head Architecture\n- Pointwise Head: output the percentile ranks of each cell. L1 Loss is applied.\n- Pairwise Head: For each pair of cells A & B, predict whether cell B is the next cell of cell A. Cross-entropy loss is applied.\n- We only use the pointwise head for inference, i.e. the cells are ranked by the predicted percentile rank. \n- The pairwise head is not used during inference, however, in our experiment, inclusion of the pairwise head helps in boosting the validation score by around 0.03.\n\n#### Code Order Correction\n- Since our model predicts a percentile rank for each cell (including code cells), the code cell order (which is given) may be violated. \n- If this happens, we swap the code cells based on the given order.\n- This idea only marginally boosts the score. The model actually learns the code order pretty well given the cell features.\n\n# What Didn't Work\n- Make use of the pairwise head prediction in a greedy manner while fulfilling the code order constraint, i.e. keep determining the next cell by selecting the cell with the highest prediction score.\n- Make use of the pairwise head prediction as a distance matrix, then use LKH to solve a Sequential Ordering Problem (TSP with precedence constraints)\n- Train 2 cell transformer backbones together end to end, rely on the Aggregation Attention to aggregate the token embeddings from the 2 backbones.\n\n# Additional Context\n- The final model is an ensemble of 4 different cell transformer backbones, i.e. CodeBERT, Deberta-large, Deberta-base, and DistilBERT. \n- For the notebook transformer, we used 6 transformer encoder layers here, each with 8 self-attention heads. This layer \"adjusts\" the embedding of each cell based on the embeddings of other cells using the self-attention mechanism.\n\n# Acknowledgements\nWe would like to thank the competition host(s) and Kaggle for organizing this interesting competition.\n\n# Team Members\n@css919 \n@erniechiew \n@hexomiter",
"votes": null
},
{
"id": "1895432",
"postDate": "08/12/2022 06:13:45",
"content": "Great work! @css919 since notebook may contain like 512 cells, does your model end2end encodes all of them ?
",
"rawMarkdown": "Great work! @css919 since notebook may contain like 512 cells, does your model end2end encodes all of them ?",
"votes": null
},
{
"id": "1895440",
"postDate": "08/12/2022 06:21:33",
"content": "To save training time, we trained on shorter notebooks first (<=126 cells), then only further finetune on longer notebooks, it depends on the model size and VRAM to decide whether to include notebooks with certain number of cells for training.
",
"rawMarkdown": "To save training time, we trained on shorter notebooks first (<=126 cells), then only further finetune on longer notebooks, it depends on the model size and VRAM to decide whether to include notebooks with certain number of cells for training.",
"votes": null
},
{
"id": "1895460",
"postDate": "08/12/2022 06:42:22",
"content": "Thanks for your explain!
",
"rawMarkdown": "Thanks for your explain!",
"votes": null
},
{
"id": "1895470",
"postDate": "08/12/2022 06:52:01",
"content": "I found with the gradient chekpointing and predicting by a few smaller groups, I can fit the larger notebooks even with the 11GB VRAM of 2080ti. My initial solution was to try to the forward/back pass on the notebook, if it failed to freeze bert encoders and to train only the nested transformer. But with checkpointing it was not really necessary.
",
"rawMarkdown": "I found with the gradient chekpointing and predicting by a few smaller groups, I can fit the larger notebooks even with the 11GB VRAM of 2080ti. My initial solution was to try to the forward/back pass on the notebook, if it failed to freeze bert encoders and to train only the nested transformer. But with checkpointing it was not really necessary.",
"votes": null
},
{
"id": "1895498",
"postDate": "08/12/2022 07:13:43",
"content": "Details are added. Thanks for reading!
",
"rawMarkdown": "Details are added. Thanks for reading!",
"votes": null
},
{
"id": "1895500",
"postDate": "08/12/2022 07:15:02",
"content": "@css919 which backbone produce best single model result? For my experiments deberta still much better.
",
"rawMarkdown": "css919 which backbone produce best single model result? For my experiments deberta still much better.",
"votes": null
},
{
"id": "1895502",
"postDate": "08/12/2022 07:21:14",
"content": "@goldenlock CodeBERT produced the best results for us, followed by DistilBERT. We were expecting Deberta to produce even better results, but we were sadly disappointed. It could be that we did not adjust the learning rate schedules appropriately.
",
"rawMarkdown": "goldenlock CodeBERT produced the best results for us, followed by DistilBERT. We were expecting Deberta to produce even better results, but we were sadly disappointed. It could be that we did not adjust the learning rate schedules appropriately.",
"votes": null
},
{
"id": "1895505",
"postDate": "08/12/2022 07:23:59",
"content": "We do not have enough submissions to test every single model after the training is done. In our case, CodeBERT and DistilBERT have similar performance, the two debertas slightly underperformed them, but to be fair, we trained deberta-base and deberta-large with smaller number of epochs (they are much larger and require longer training time per epoch).
",
"rawMarkdown": "We do not have enough submissions to test every single model after the training is done. In our case, CodeBERT and DistilBERT have similar performance, the two debertas slightly underperformed them, but to be fair, we trained deberta-base and deberta-large with smaller number of epochs (they are much larger and require longer training time per epoch).",
"votes": null
},
{
"id": "1895543",
"postDate": "08/12/2022 07:55:26",
"content": "Thanks for your info. Actually I found dev3-small(smaller) better then CodeBERT(larger), intersting.
",
"rawMarkdown": "Thanks for your info. Actually I found dev3-small(smaller) better then CodeBERT(larger), intersting.",
"votes": null
},
{
"id": "1895961",
"postDate": "08/12/2022 13:13:39",
"content": "Great work. We did so many same things.😄
",
"rawMarkdown": "Great work. We did so many same things.😄",
"votes": null
},
{
"id": "1896655",
"postDate": "08/13/2022 02:58:22",
"content": "Nice solution @erniechiew and team.
",
"rawMarkdown": "Nice solution @erniechiew and team.",
"votes": null
},
{
"id": "1896901",
"postDate": "08/13/2022 08:55:27",
"content": "wow great work
",
"rawMarkdown": "wow great work",
"votes": null
},
{
"id": "1897655",
"postDate": "08/14/2022 01:25:26",
"content": "Fantastic work! Appreciate the detailed solution.
",
"rawMarkdown": "Fantastic work! Appreciate the detailed solution.",
"votes": null
},
{
"id": "1897712",
"postDate": "08/14/2022 03:12:18",
"content": "Great work and a very helpful explanation!
\nI've just one question: how do you organize the notebooks into batches for the trainer? One notebook per batch or something else?
",
"rawMarkdown": "Great work and a very helpful explanation! \n\nI've just one question: how do you organize the notebooks into batches for the trainer? One notebook per batch or something else?",
"votes": null
},
{
"id": "1899143",
"postDate": "08/15/2022 05:01:47",
"content": "great work!!👍
",
"rawMarkdown": "great work!!👍",
"votes": null
},
{
"id": "1899191",
"postDate": "08/15/2022 05:43:34",
"content": "depending on the backbone and notebook length, we use batch size within [1:4]
",
"rawMarkdown": "depending on the backbone and notebook length, we use batch size within [1:4]",
"votes": null
},
{
"id": "1900229",
"postDate": "08/15/2022 20:01:01",
"content": "Oh, so you got over 0.92 by 2080ti? That's really interesting. I thought most of the top competitors used strong GPUs.
",
"rawMarkdown": "Oh, so you got over 0.92 by 2080ti? That's really interesting. I thought most of the top competitors used strong GPUs.",
"votes": null
},
{
"id": "1901775",
"postDate": "08/16/2022 22:40:42",
"content": "I used a few 3090 and 2080ti, so I'm not an exception here and I also used stronger GPUs.
",
"rawMarkdown": "I used a few 3090 and 2080ti, so I'm not an exception here and I also used stronger GPUs.",
"votes": null
},
{
"id": "1903341",
"postDate": "08/17/2022 10:30:41",
"content": "I see, thanks for the reply! I also used the similar environment, most experiments are done on 3090 and for the final model I used stronger GPUs.
",
"rawMarkdown": "I see, thanks for the reply! I also used the similar environment, most experiments are done on 3090 and for the final model I used stronger GPUs.",
"votes": null
},
{
"id": "1903579",
"postDate": "08/17/2022 14:01:00",
"content": "I tried to implement this solution myself and have a few more questions:
\n\n- Did you guys use the same learning rate for the cell transformer and the notebook transformer?
\n- Did you train the two losses with equal weights (e.g. loss=loss1+loss2)?
\n- Is there any specific part of the architecture that you noticed to give a big boost to the score?
\n
",
"rawMarkdown": "I tried to implement this solution myself and have a few more questions:\n1. Did you guys use the same learning rate for the cell transformer and the notebook transformer?\n2. Did you train the two losses with equal weights (e.g. loss=loss1+loss2)?\n3. Is there any specific part of the architecture that you noticed to give a big boost to the score?",
"votes": null
},
{
"id": "1903850",
"postDate": "08/17/2022 17:36:25",
"content": "Awesome work !!! Cell Feature Embeddings and Nested Transformers are very innovative for me.
\nIt would be really helpful , if you could elaborate a bit on Aggregation Attention Layer. Any kind of literature about that would be awesome. Thank you.
",
"rawMarkdown": "Awesome work !!! Cell Feature Embeddings and Nested Transformers are very innovative for me. \nIt would be really helpful , if you could elaborate a bit on Aggregation Attention Layer. Any kind of literature about that would be awesome. Thank you.",
"votes": null
},
{
"id": "1903913",
"postDate": "08/17/2022 18:44:16",
"content": "It's just a simple way of deciding the aggregation weights based on the data itself. Implementation can be something like:
\nclass AttentionPooler(nn.Module):\n def __init__(self, emb_dim):\n super().__init__()\n self.emb_dim = emb_dim'\n\n self.attention = nn.Sequential(\n nn.Linear(self.emb_dim, 512),\n nn.Tanh(),\n nn.Linear(512, 1),\n nn.Softmax(dim=1)\n )\n\n def forward(self, embeddings, mask):\n weight = self.attention(embeddings)\n input_mask_expanded = mask.unsqueeze(-1).expand(x.size()).float()\n return torch.sum(weight * embeddings * input_mask_expanded, 1) / torch.clamp((weight * input_mask_expanded).sum(1), min=1e-9)\n
",
"rawMarkdown": "It's just a simple way of deciding the aggregation weights based on the data itself. Implementation can be something like: \n\n\n```\nclass AttentionPooler(nn.Module):\n def __init__(self, emb_dim):\n super().__init__()\n self.emb_dim = emb_dim'\n\n self.attention = nn.Sequential(\n nn.Linear(self.emb_dim, 512),\n nn.Tanh(),\n nn.Linear(512, 1),\n nn.Softmax(dim=1)\n )\n\n def forward(self, embeddings, mask):\n weight = self.attention(embeddings)\n input_mask_expanded = mask.unsqueeze(-1).expand(x.size()).float()\n return torch.sum(weight * embeddings * input_mask_expanded, 1) / torch.clamp((weight * input_mask_expanded).sum(1), min=1e-9)\n```",
"votes": null
},
{
"id": "1903925",
"postDate": "08/17/2022 18:58:54",
"content": "\n- We trained the model from end to end, hence they share the same learning rate and optimizer state.
\n- Yes, but the l1loss is weighted by the number of markdown cells in the notebook.
\n- We did not have the time to quantify the effect of each and every component, the most significant one should be the pairwise head which gives a boost around 0.02-0.03.
\n
",
"rawMarkdown": "1. We trained the model from end to end, hence they share the same learning rate and optimizer state.\n2. Yes, but the l1loss is weighted by the number of markdown cells in the notebook.\n3. We did not have the time to quantify the effect of each and every component, the most significant one should be the pairwise head which gives a boost around 0.02-0.03.",
"votes": null
},
{
"id": "1903929",
"postDate": "08/17/2022 19:02:53",
"content": "Yes, we used something similar. To be precise, it is equation (3) of this paper.
",
"rawMarkdown": "Yes, we used something similar. To be precise, it is equation (3) of this [paper](https://aclanthology.org/2020.emnlp-main.511.pdf).",
"votes": null
},
{
"id": "1904424",
"postDate": "08/18/2022 07:12:03",
"content": "Thanks a lot, I will give it a try.
",
"rawMarkdown": "Thanks a lot, I will give it a try.",
"votes": null
},
{
"id": "1904425",
"postDate": "08/18/2022 07:12:38",
"content": "Thanks for the paper as well , I will go through that
",
"rawMarkdown": "Thanks for the paper as well , I will go through that",
"votes": null
},
{
"id": "1906428",
"postDate": "08/19/2022 21:49:23",
"content": "Congratulations on your success in this competition! Your solution is really interesting!
",
"rawMarkdown": "Congratulations on your success in this competition! Your solution is really interesting!",
"votes": null
},
{
"id": "1906554",
"postDate": "08/20/2022 01:40:25",
"content": "Thanks for the answers! I hope I'm not asking too many questions, but it seems remarkable to me that the same architecture is able to process both markdown and code cells so well, using only the cell type and position features to differentiate between the two. I'm wondering if you guys tried a few different ways to represent the cell type difference and the the best one was on the token level? In my experiments, when I add the cell features on the cell level rather than on the token level, the prediction accuracy for markdown is relatively weak.
",
"rawMarkdown": "Thanks for the answers! I hope I'm not asking too many questions, but it seems remarkable to me that the same architecture is able to process both markdown and code cells so well, using only the cell type and position features to differentiate between the two. I'm wondering if you guys tried a few different ways to represent the cell type difference and the the best one was on the token level? In my experiments, when I add the cell features on the cell level rather than on the token level, the prediction accuracy for markdown is relatively weak.",
"votes": null
},
{
"id": "1906655",
"postDate": "08/20/2022 05:20:46",
"content": "Forgot to mention that besides the cell features, we also add 2 tokens (cell_type and separator tokens) to the text of each cell, either code <sep> (source) or mark <sep> (source).
",
"rawMarkdown": "Forgot to mention that besides the cell features, we also add 2 tokens (cell_type and separator tokens) to the text of each cell, either `code (source)` or `mark (source)`.",
"votes": null
},
{
"id": "1907307",
"postDate": "08/20/2022 16:52:36",
"content": "In my implementation of your solution, adding the cell type as you described improves the score by a whopping 0.09.
",
"rawMarkdown": "In my implementation of your solution, adding the cell type as you described improves the score by a whopping 0.09.",
"votes": null
},
{
"id": "1907356",
"postDate": "08/20/2022 17:40:52",
"content": "impressive 😹
",
"rawMarkdown": "impressive 😹",
"votes": null
},
{
"id": "1907651",
"postDate": "08/21/2022 00:39:45",
"content": "It's interesting, have you tried to use the simpler aggregation like averaging over all tokens? I tried to use the cls token output (which may also have the attention-weighted activations over other tokens) and average over all tokens and received similar results, so I decided to use simple averaging.
",
"rawMarkdown": "It's interesting, have you tried to use the simpler aggregation like averaging over all tokens? I tried to use the cls token output (which may also have the attention-weighted activations over other tokens) and average over all tokens and received similar results, so I decided to use simple averaging.",
"votes": null
},
{
"id": "1907971",
"postDate": "08/21/2022 08:45:40",
"content": "I didn't try simpler aggregation, but I agreed that it is interesting to know the score difference. In fact this kind of analysis to any design choice is interesting, hopefully I will have the bandwidth to try them in the future :)
",
"rawMarkdown": "I didn't try simpler aggregation, but I agreed that it is interesting to know the score difference. In fact this kind of analysis to any design choice is interesting, hopefully I will have the bandwidth to try them in the future :)",
"votes": null
},
{
"id": "1909812",
"postDate": "08/23/2022 00:48:15",
"content": "When I was implementing this solution, I compared between using mean-pooling and attention-pooling. The difference was quite dramatic.
\nI'm pretty sure that it's because the cell features (cell type, code cell position) are represented as extra tokens. If we just average them together with the cell token embeddings, the signal of the cell features gets drowned out.
\nWhen instead I tried adding the cell features after the pooling operation, the pooler type didn't matter as much.
",
"rawMarkdown": "When I was implementing this solution, I compared between using mean-pooling and attention-pooling. The difference was quite dramatic. \n\nI'm pretty sure that it's because the cell features (cell type, code cell position) are represented as extra tokens. If we just average them together with the cell token embeddings, the signal of the cell features gets drowned out.\n\nWhen instead I tried adding the cell features after the pooling operation, the pooler type didn't matter as much.",
"votes": null
},
{
"id": "1915032",
"postDate": "08/26/2022 16:07:04",
"content": "A massive thanks for your sharing. It's fascinating to me. I'm trying to re-implement your solution and I have a question that: In the Cell Transformer block, how did you use both the Transformer encoder and the BERT models (the tokens are fed to the encoder or the BERT models first)?
",
"rawMarkdown": "A massive thanks for your sharing. It's fascinating to me. I'm trying to re-implement your solution and I have a question that: In the Cell Transformer block, how did you use both the Transformer encoder and the BERT models (the tokens are fed to the encoder or the BERT models first)?",
"votes": null
},
{
"id": "1915038",
"postDate": "08/26/2022 16:12:59",
"content": "BERT is just stacks of transformer encoder? For Cell Transformer we are using huggingface models only.
",
"rawMarkdown": "BERT is just stacks of transformer encoder? For Cell Transformer we are using huggingface models only.",
"votes": null
},
{
"id": "1915062",
"postDate": "08/26/2022 16:25:35",
"content": "Take CodeBERT as an example, when the tokens of a cell go through the model, the output dim is 768. This is a 1-shape tensor but you guys said the output's shape is max_len*emb_dim. Could you please tell me why? Your answer will be very helpful for me.
",
"rawMarkdown": "Take CodeBERT as an example, when the tokens of a cell go through the model, the output dim is 768. This is a 1-shape tensor but you guys said the output's shape is max_len*emb_dim. Could you please tell me why? Your answer will be very helpful for me.",
"votes": null
},
{
"id": "1915186",
"postDate": "08/26/2022 18:19:43",
"content": "The output of CodeBert is token embeddings and the dimension of the output is therefore num_tokens * 768. This data is then fed into a pooler that creates a \"sentence\" embedding that has dimension 1*768.
",
"rawMarkdown": "The output of CodeBert is token embeddings and the dimension of the output is therefore num_tokens * 768. This data is then fed into a pooler that creates a \"sentence\" embedding that has dimension 1*768.",
"votes": null
},
{
"id": "1915235",
"postDate": "08/26/2022 19:27:15",
"content": "emb_dim = 768
\nembedding of 1 token (output of huggingface model): emb_dim
\nembedding of max_len tokens: max_len * emb_dim
\nafter concatenate with the code feature embedding: (max_len+1) * emb_dim
\nThis happens on every cell, for all cells: n_cells * (max_len+1) * emb_dim
\nAfter the attention aggregation (pooler): n_cells * emb_dim
",
"rawMarkdown": "emb_dim = 768\nembedding of 1 token (output of huggingface model): emb_dim\nembedding of max_len tokens: max_len * emb_dim\nafter concatenate with the code feature embedding: (max_len+1) * emb_dim\nThis happens on every cell, for all cells: n_cells * (max_len+1) * emb_dim\nAfter the attention aggregation (pooler): n_cells * emb_dim",
"votes": null
},
{
"id": "1915525",
"postDate": "08/27/2022 03:43:14",
"content": "Awesome!!! So, you mean you take each word of a cell and then pass it to the BERT (not the whole cell's content), right?
",
"rawMarkdown": "Awesome!!! So, you mean you take each word of a cell and then pass it to the BERT (not the whole cell's content), right?",
"votes": null
},
{
"id": "1919121",
"postDate": "08/30/2022 06:45:01",
"content": "Fantastic work!
\nDid you use a variable called max_num_cells while training?
",
"rawMarkdown": "Fantastic work!\nDid you use a variable called `max_num_cells` while training?",
"votes": null
},
{
"id": "2026180",
"postDate": "11/11/2022 20:01:13",
"content": "Updated the post with the link to the Github repository.
",
"rawMarkdown": "Updated the post with the link to the Github repository.",
"votes": null
},
{
"id": "2026821",
"postDate": "11/12/2022 11:09:23",
"content": "Sorry, I can't see the NotebookModel class. Could you please show me its place in your repo?
",
"rawMarkdown": "Sorry, I can't see the NotebookModel class. Could you please show me its place in your repo?",
"votes": null
},
{
"id": "2026944",
"postDate": "11/12/2022 12:48:30",
"content": "I'm sorry, I forgot to add back certain files after renaming. Have just pushed a new tested version.
",
"rawMarkdown": "I'm sorry, I forgot to add back certain files after renaming. Have just pushed a new tested version.",
"votes": null
},
{
"id": "2026951",
"postDate": "11/12/2022 12:53:20",
"content": "Love to hear that from you!
",
"rawMarkdown": "Love to hear that from you!",
"votes": null
}
],
"comments": [
{
"id": 1895432,
"author_name": "goldenlock",
"author_url": "",
"post_date": "08/12/2022 06:13:45",
"content": "Great work! @css919 since notebook may contain like 512 cells, does your model end2end encodes all of them ?
",
"votes": null,
"replies": [
{
"id": 1895440,
"author_name": "css919",
"author_url": "",
"post_date": "08/12/2022 06:21:33",
"content": "To save training time, we trained on shorter notebooks first (<=126 cells), then only further finetune on longer notebooks, it depends on the model size and VRAM to decide whether to include notebooks with certain number of cells for training.
",
"votes": null,
"replies": []
},
{
"id": 1895460,
"author_name": "goldenlock",
"author_url": "",
"post_date": "08/12/2022 06:42:22",
"content": "Thanks for your explain!
",
"votes": null,
"replies": []
},
{
"id": 1895470,
"author_name": "dmytropoplavskiy",
"author_url": "",
"post_date": "08/12/2022 06:52:01",
"content": "I found with the gradient chekpointing and predicting by a few smaller groups, I can fit the larger notebooks even with the 11GB VRAM of 2080ti. My initial solution was to try to the forward/back pass on the notebook, if it failed to freeze bert encoders and to train only the nested transformer. But with checkpointing it was not really necessary.
",
"votes": null,
"replies": []
},
{
"id": 1900229,
"author_name": "yufuin",
"author_url": "",
"post_date": "08/15/2022 20:01:01",
"content": "Oh, so you got over 0.92 by 2080ti? That's really interesting. I thought most of the top competitors used strong GPUs.
",
"votes": null,
"replies": []
},
{
"id": 1901775,
"author_name": "dmytropoplavskiy",
"author_url": "",
"post_date": "08/16/2022 22:40:42",
"content": "I used a few 3090 and 2080ti, so I'm not an exception here and I also used stronger GPUs.
",
"votes": null,
"replies": []
},
{
"id": 1903341,
"author_name": "yufuin",
"author_url": "",
"post_date": "08/17/2022 10:30:41",
"content": "I see, thanks for the reply! I also used the similar environment, most experiments are done on 3090 and for the final model I used stronger GPUs.
",
"votes": null,
"replies": []
}
]
},
{
"id": 1895498,
"author_name": "css919",
"author_url": "",
"post_date": "08/12/2022 07:13:43",
"content": "Details are added. Thanks for reading!
",
"votes": null,
"replies": []
},
{
"id": 1895500,
"author_name": "goldenlock",
"author_url": "",
"post_date": "08/12/2022 07:15:02",
"content": "@css919 which backbone produce best single model result? For my experiments deberta still much better.
",
"votes": null,
"replies": [
{
"id": 1895502,
"author_name": "erniechiew",
"author_url": "",
"post_date": "08/12/2022 07:21:14",
"content": "@goldenlock CodeBERT produced the best results for us, followed by DistilBERT. We were expecting Deberta to produce even better results, but we were sadly disappointed. It could be that we did not adjust the learning rate schedules appropriately.
",
"votes": null,
"replies": []
},
{
"id": 1895505,
"author_name": "css919",
"author_url": "",
"post_date": "08/12/2022 07:23:59",
"content": "We do not have enough submissions to test every single model after the training is done. In our case, CodeBERT and DistilBERT have similar performance, the two debertas slightly underperformed them, but to be fair, we trained deberta-base and deberta-large with smaller number of epochs (they are much larger and require longer training time per epoch).
",
"votes": null,
"replies": []
},
{
"id": 1895543,
"author_name": "goldenlock",
"author_url": "",
"post_date": "08/12/2022 07:55:26",
"content": "Thanks for your info. Actually I found dev3-small(smaller) better then CodeBERT(larger), intersting.
",
"votes": null,
"replies": []
}
]
},
{
"id": 1895961,
"author_name": "hydantess",
"author_url": "",
"post_date": "08/12/2022 13:13:39",
"content": "Great work. We did so many same things.😄
",
"votes": null,
"replies": []
},
{
"id": 1896655,
"author_name": "duykhanh99",
"author_url": "",
"post_date": "08/13/2022 02:58:22",
"content": "Nice solution @erniechiew and team.
",
"votes": null,
"replies": []
},
{
"id": 1896901,
"author_name": "bilalsuppal",
"author_url": "",
"post_date": "08/13/2022 08:55:27",
"content": "wow great work
",
"votes": null,
"replies": []
},
{
"id": 1897655,
"author_name": "harrymapodile",
"author_url": "",
"post_date": "08/14/2022 01:25:26",
"content": "Fantastic work! Appreciate the detailed solution.
",
"votes": null,
"replies": []
},
{
"id": 1897712,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/14/2022 03:12:18",
"content": "Great work and a very helpful explanation!
\nI've just one question: how do you organize the notebooks into batches for the trainer? One notebook per batch or something else?
",
"votes": null,
"replies": [
{
"id": 1899191,
"author_name": "css919",
"author_url": "",
"post_date": "08/15/2022 05:43:34",
"content": "depending on the backbone and notebook length, we use batch size within [1:4]
",
"votes": null,
"replies": []
}
]
},
{
"id": 1899143,
"author_name": "rajg28",
"author_url": "",
"post_date": "08/15/2022 05:01:47",
"content": "great work!!👍
",
"votes": null,
"replies": []
},
{
"id": 1903579,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/17/2022 14:01:00",
"content": "I tried to implement this solution myself and have a few more questions:
\n\n- Did you guys use the same learning rate for the cell transformer and the notebook transformer?
\n- Did you train the two losses with equal weights (e.g. loss=loss1+loss2)?
\n- Is there any specific part of the architecture that you noticed to give a big boost to the score?
\n
",
"votes": null,
"replies": [
{
"id": 1903925,
"author_name": "css919",
"author_url": "",
"post_date": "08/17/2022 18:58:54",
"content": "\n- We trained the model from end to end, hence they share the same learning rate and optimizer state.
\n- Yes, but the l1loss is weighted by the number of markdown cells in the notebook.
\n- We did not have the time to quantify the effect of each and every component, the most significant one should be the pairwise head which gives a boost around 0.02-0.03.
\n
",
"votes": null,
"replies": []
},
{
"id": 1906554,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/20/2022 01:40:25",
"content": "Thanks for the answers! I hope I'm not asking too many questions, but it seems remarkable to me that the same architecture is able to process both markdown and code cells so well, using only the cell type and position features to differentiate between the two. I'm wondering if you guys tried a few different ways to represent the cell type difference and the the best one was on the token level? In my experiments, when I add the cell features on the cell level rather than on the token level, the prediction accuracy for markdown is relatively weak.
",
"votes": null,
"replies": []
},
{
"id": 1906655,
"author_name": "css919",
"author_url": "",
"post_date": "08/20/2022 05:20:46",
"content": "Forgot to mention that besides the cell features, we also add 2 tokens (cell_type and separator tokens) to the text of each cell, either code <sep> (source) or mark <sep> (source).
",
"votes": null,
"replies": []
},
{
"id": 1907307,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/20/2022 16:52:36",
"content": "In my implementation of your solution, adding the cell type as you described improves the score by a whopping 0.09.
",
"votes": null,
"replies": []
},
{
"id": 1907356,
"author_name": "css919",
"author_url": "",
"post_date": "08/20/2022 17:40:52",
"content": "impressive 😹
",
"votes": null,
"replies": []
}
]
},
{
"id": 1903850,
"author_name": "kuntalbhowmick",
"author_url": "",
"post_date": "08/17/2022 17:36:25",
"content": "Awesome work !!! Cell Feature Embeddings and Nested Transformers are very innovative for me.
\nIt would be really helpful , if you could elaborate a bit on Aggregation Attention Layer. Any kind of literature about that would be awesome. Thank you.
",
"votes": null,
"replies": [
{
"id": 1903913,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/17/2022 18:44:16",
"content": "It's just a simple way of deciding the aggregation weights based on the data itself. Implementation can be something like:
\nclass AttentionPooler(nn.Module):\n def __init__(self, emb_dim):\n super().__init__()\n self.emb_dim = emb_dim'\n\n self.attention = nn.Sequential(\n nn.Linear(self.emb_dim, 512),\n nn.Tanh(),\n nn.Linear(512, 1),\n nn.Softmax(dim=1)\n )\n\n def forward(self, embeddings, mask):\n weight = self.attention(embeddings)\n input_mask_expanded = mask.unsqueeze(-1).expand(x.size()).float()\n return torch.sum(weight * embeddings * input_mask_expanded, 1) / torch.clamp((weight * input_mask_expanded).sum(1), min=1e-9)\n
",
"votes": null,
"replies": []
},
{
"id": 1903929,
"author_name": "css919",
"author_url": "",
"post_date": "08/17/2022 19:02:53",
"content": "Yes, we used something similar. To be precise, it is equation (3) of this paper.
",
"votes": null,
"replies": []
},
{
"id": 1904424,
"author_name": "kuntalbhowmick",
"author_url": "",
"post_date": "08/18/2022 07:12:03",
"content": "Thanks a lot, I will give it a try.
",
"votes": null,
"replies": []
},
{
"id": 1904425,
"author_name": "kuntalbhowmick",
"author_url": "",
"post_date": "08/18/2022 07:12:38",
"content": "Thanks for the paper as well , I will go through that
",
"votes": null,
"replies": []
},
{
"id": 1907651,
"author_name": "dmytropoplavskiy",
"author_url": "",
"post_date": "08/21/2022 00:39:45",
"content": "It's interesting, have you tried to use the simpler aggregation like averaging over all tokens? I tried to use the cls token output (which may also have the attention-weighted activations over other tokens) and average over all tokens and received similar results, so I decided to use simple averaging.
",
"votes": null,
"replies": []
},
{
"id": 1907971,
"author_name": "css919",
"author_url": "",
"post_date": "08/21/2022 08:45:40",
"content": "I didn't try simpler aggregation, but I agreed that it is interesting to know the score difference. In fact this kind of analysis to any design choice is interesting, hopefully I will have the bandwidth to try them in the future :)
",
"votes": null,
"replies": []
},
{
"id": 1909812,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/23/2022 00:48:15",
"content": "When I was implementing this solution, I compared between using mean-pooling and attention-pooling. The difference was quite dramatic.
\nI'm pretty sure that it's because the cell features (cell type, code cell position) are represented as extra tokens. If we just average them together with the cell token embeddings, the signal of the cell features gets drowned out.
\nWhen instead I tried adding the cell features after the pooling operation, the pooler type didn't matter as much.
",
"votes": null,
"replies": []
}
]
},
{
"id": 1906428,
"author_name": "thedevastator",
"author_url": "",
"post_date": "08/19/2022 21:49:23",
"content": "Congratulations on your success in this competition! Your solution is really interesting!
",
"votes": null,
"replies": []
},
{
"id": 1915032,
"author_name": "nguynthhi",
"author_url": "",
"post_date": "08/26/2022 16:07:04",
"content": "A massive thanks for your sharing. It's fascinating to me. I'm trying to re-implement your solution and I have a question that: In the Cell Transformer block, how did you use both the Transformer encoder and the BERT models (the tokens are fed to the encoder or the BERT models first)?
",
"votes": null,
"replies": [
{
"id": 1915038,
"author_name": "css919",
"author_url": "",
"post_date": "08/26/2022 16:12:59",
"content": "BERT is just stacks of transformer encoder? For Cell Transformer we are using huggingface models only.
",
"votes": null,
"replies": []
},
{
"id": 1915062,
"author_name": "nguynthhi",
"author_url": "",
"post_date": "08/26/2022 16:25:35",
"content": "Take CodeBERT as an example, when the tokens of a cell go through the model, the output dim is 768. This is a 1-shape tensor but you guys said the output's shape is max_len*emb_dim. Could you please tell me why? Your answer will be very helpful for me.
",
"votes": null,
"replies": []
},
{
"id": 1915186,
"author_name": "cabbage972",
"author_url": "",
"post_date": "08/26/2022 18:19:43",
"content": "The output of CodeBert is token embeddings and the dimension of the output is therefore num_tokens * 768. This data is then fed into a pooler that creates a \"sentence\" embedding that has dimension 1*768.
",
"votes": null,
"replies": []
},
{
"id": 1915235,
"author_name": "css919",
"author_url": "",
"post_date": "08/26/2022 19:27:15",
"content": "emb_dim = 768
\nembedding of 1 token (output of huggingface model): emb_dim
\nembedding of max_len tokens: max_len * emb_dim
\nafter concatenate with the code feature embedding: (max_len+1) * emb_dim
\nThis happens on every cell, for all cells: n_cells * (max_len+1) * emb_dim
\nAfter the attention aggregation (pooler): n_cells * emb_dim
",
"votes": null,
"replies": []
},
{
"id": 1915525,
"author_name": "nguynthhi",
"author_url": "",
"post_date": "08/27/2022 03:43:14",
"content": "Awesome!!! So, you mean you take each word of a cell and then pass it to the BERT (not the whole cell's content), right?
",
"votes": null,
"replies": []
}
]
},
{
"id": 1919121,
"author_name": "nguynthhi",
"author_url": "",
"post_date": "08/30/2022 06:45:01",
"content": "Fantastic work!
\nDid you use a variable called max_num_cells while training?
",
"votes": null,
"replies": []
},
{
"id": 2026180,
"author_name": "css919",
"author_url": "",
"post_date": "11/11/2022 20:01:13",
"content": "Updated the post with the link to the Github repository.
",
"votes": null,
"replies": [
{
"id": 2026821,
"author_name": "nguynthhi",
"author_url": "",
"post_date": "11/12/2022 11:09:23",
"content": "Sorry, I can't see the NotebookModel class. Could you please show me its place in your repo?
",
"votes": null,
"replies": []
},
{
"id": 2026944,
"author_name": "css919",
"author_url": "",
"post_date": "11/12/2022 12:48:30",
"content": "I'm sorry, I forgot to add back certain files after renaming. Have just pushed a new tested version.
",
"votes": null,
"replies": []
},
{
"id": 2026951,
"author_name": "nguynthhi",
"author_url": "",
"post_date": "11/12/2022 12:53:20",
"content": "Love to hear that from you!
",
"votes": null,
"replies": []
}
]
}
],
"raw_markdown_by_id": {
"1895389": "# Overview\nWe called our network architecture \"Nested Transformers\", since it contains two levels of transformers:\n- Cell transformer: A transformer encoder to encode cell tokens, this can be a typical transformer for NLP.\n- Notebook transformer: A transformer encoder with cell*cell self-attention to learn the interaction among cells.\n\n\nSource codes: https://github.com/ShinSiangChoong/kaggle_ai4code_nested_transformers\n\n# What Worked\n#### Modified Format of the Input to Cell Transformer\n- The input is in the following format: `` OR ``\n- Thanks @cabbage972 for verifying the usefulness of this idea. Please refer to this [comment](https://www.kaggle.com/competitions/AI4Code/discussion/343680#1907307). \n\n#### Cell Features: \n- The code cell order is known, while the markdown order is unknown, hence a straightforward positional embedding scheme in the notebook transformer won't work. The cell features aim to inform the model about the known code cell positions/order.\n- We use 2 features for each cell, i.e. cell type (code -> 1, markdown -> 0) and percentile rank among the code cells: If the cell is a markdown cell, this feature = 0\n- The 2 features are \"expanded\" by FC layers, such that they are represented as an embedding that has the same dimension with that of the cell token embeddings. \n- These cell feature embeddings are then concatenated with the token embeddings and passed into the Aggregation Attention Layer together.\n\n#### Aggregation Attention\n- We believe different tokens (and the cell features) have different importance to represent the cell. An aggregation attention layer is used to learn the weights of each token embedding (and the cell feature embedding). \n- The weights are normalized by a Softmax layer, such that the final cell embedding is a weighted average of its token and feature embeddings.\n- Thanks @cabbage972 for verifying the usefulness of this idea. Please refer to this [comment](https://www.kaggle.com/competitions/AI4Code/discussion/343680#1909812). \n\n#### Double Head Architecture\n- Pointwise Head: output the percentile ranks of each cell. L1 Loss is applied.\n- Pairwise Head: For each pair of cells A & B, predict whether cell B is the next cell of cell A. Cross-entropy loss is applied.\n- We only use the pointwise head for inference, i.e. the cells are ranked by the predicted percentile rank. \n- The pairwise head is not used during inference, however, in our experiment, inclusion of the pairwise head helps in boosting the validation score by around 0.03.\n\n#### Code Order Correction\n- Since our model predicts a percentile rank for each cell (including code cells), the code cell order (which is given) may be violated. \n- If this happens, we swap the code cells based on the given order.\n- This idea only marginally boosts the score. The model actually learns the code order pretty well given the cell features.\n\n# What Didn't Work\n- Make use of the pairwise head prediction in a greedy manner while fulfilling the code order constraint, i.e. keep determining the next cell by selecting the cell with the highest prediction score.\n- Make use of the pairwise head prediction as a distance matrix, then use LKH to solve a Sequential Ordering Problem (TSP with precedence constraints)\n- Train 2 cell transformer backbones together end to end, rely on the Aggregation Attention to aggregate the token embeddings from the 2 backbones.\n\n# Additional Context\n- The final model is an ensemble of 4 different cell transformer backbones, i.e. CodeBERT, Deberta-large, Deberta-base, and DistilBERT. \n- For the notebook transformer, we used 6 transformer encoder layers here, each with 8 self-attention heads. This layer \"adjusts\" the embedding of each cell based on the embeddings of other cells using the self-attention mechanism.\n\n# Acknowledgements\nWe would like to thank the competition host(s) and Kaggle for organizing this interesting competition.\n\n# Team Members\n@css919 \n@erniechiew \n@hexomiter",
"1895432": "Great work! @css919 since notebook may contain like 512 cells, does your model end2end encodes all of them ?",
"1895440": "To save training time, we trained on shorter notebooks first (<=126 cells), then only further finetune on longer notebooks, it depends on the model size and VRAM to decide whether to include notebooks with certain number of cells for training.",
"1895460": "Thanks for your explain!",
"1895470": "I found with the gradient chekpointing and predicting by a few smaller groups, I can fit the larger notebooks even with the 11GB VRAM of 2080ti. My initial solution was to try to the forward/back pass on the notebook, if it failed to freeze bert encoders and to train only the nested transformer. But with checkpointing it was not really necessary.",
"1895498": "Details are added. Thanks for reading!",
"1895500": "css919 which backbone produce best single model result? For my experiments deberta still much better.",
"1895502": "goldenlock CodeBERT produced the best results for us, followed by DistilBERT. We were expecting Deberta to produce even better results, but we were sadly disappointed. It could be that we did not adjust the learning rate schedules appropriately.",
"1895505": "We do not have enough submissions to test every single model after the training is done. In our case, CodeBERT and DistilBERT have similar performance, the two debertas slightly underperformed them, but to be fair, we trained deberta-base and deberta-large with smaller number of epochs (they are much larger and require longer training time per epoch).",
"1895543": "Thanks for your info. Actually I found dev3-small(smaller) better then CodeBERT(larger), intersting.",
"1895961": "Great work. We did so many same things.😄",
"1896655": "Nice solution @erniechiew and team.",
"1896901": "wow great work",
"1897655": "Fantastic work! Appreciate the detailed solution.",
"1897712": "Great work and a very helpful explanation! \n\nI've just one question: how do you organize the notebooks into batches for the trainer? One notebook per batch or something else?",
"1899143": "great work!!👍",
"1899191": "depending on the backbone and notebook length, we use batch size within [1:4]",
"1900229": "Oh, so you got over 0.92 by 2080ti? That's really interesting. I thought most of the top competitors used strong GPUs.",
"1901775": "I used a few 3090 and 2080ti, so I'm not an exception here and I also used stronger GPUs.",
"1903341": "I see, thanks for the reply! I also used the similar environment, most experiments are done on 3090 and for the final model I used stronger GPUs.",
"1903579": "I tried to implement this solution myself and have a few more questions:\n1. Did you guys use the same learning rate for the cell transformer and the notebook transformer?\n2. Did you train the two losses with equal weights (e.g. loss=loss1+loss2)?\n3. Is there any specific part of the architecture that you noticed to give a big boost to the score?",
"1903850": "Awesome work !!! Cell Feature Embeddings and Nested Transformers are very innovative for me. \nIt would be really helpful , if you could elaborate a bit on Aggregation Attention Layer. Any kind of literature about that would be awesome. Thank you.",
"1903913": "It's just a simple way of deciding the aggregation weights based on the data itself. Implementation can be something like: \n\n\n```\nclass AttentionPooler(nn.Module):\n def __init__(self, emb_dim):\n super().__init__()\n self.emb_dim = emb_dim'\n\n self.attention = nn.Sequential(\n nn.Linear(self.emb_dim, 512),\n nn.Tanh(),\n nn.Linear(512, 1),\n nn.Softmax(dim=1)\n )\n\n def forward(self, embeddings, mask):\n weight = self.attention(embeddings)\n input_mask_expanded = mask.unsqueeze(-1).expand(x.size()).float()\n return torch.sum(weight * embeddings * input_mask_expanded, 1) / torch.clamp((weight * input_mask_expanded).sum(1), min=1e-9)\n```",
"1903925": "1. We trained the model from end to end, hence they share the same learning rate and optimizer state.\n2. Yes, but the l1loss is weighted by the number of markdown cells in the notebook.\n3. We did not have the time to quantify the effect of each and every component, the most significant one should be the pairwise head which gives a boost around 0.02-0.03.",
"1903929": "Yes, we used something similar. To be precise, it is equation (3) of this [paper](https://aclanthology.org/2020.emnlp-main.511.pdf).",
"1904424": "Thanks a lot, I will give it a try.",
"1904425": "Thanks for the paper as well , I will go through that",
"1906428": "Congratulations on your success in this competition! Your solution is really interesting!",
"1906554": "Thanks for the answers! I hope I'm not asking too many questions, but it seems remarkable to me that the same architecture is able to process both markdown and code cells so well, using only the cell type and position features to differentiate between the two. I'm wondering if you guys tried a few different ways to represent the cell type difference and the the best one was on the token level? In my experiments, when I add the cell features on the cell level rather than on the token level, the prediction accuracy for markdown is relatively weak.",
"1906655": "Forgot to mention that besides the cell features, we also add 2 tokens (cell_type and separator tokens) to the text of each cell, either `code (source)` or `mark (source)`.",
"1907307": "In my implementation of your solution, adding the cell type as you described improves the score by a whopping 0.09.",
"1907356": "impressive 😹",
"1907651": "It's interesting, have you tried to use the simpler aggregation like averaging over all tokens? I tried to use the cls token output (which may also have the attention-weighted activations over other tokens) and average over all tokens and received similar results, so I decided to use simple averaging.",
"1907971": "I didn't try simpler aggregation, but I agreed that it is interesting to know the score difference. In fact this kind of analysis to any design choice is interesting, hopefully I will have the bandwidth to try them in the future :)",
"1909812": "When I was implementing this solution, I compared between using mean-pooling and attention-pooling. The difference was quite dramatic. \n\nI'm pretty sure that it's because the cell features (cell type, code cell position) are represented as extra tokens. If we just average them together with the cell token embeddings, the signal of the cell features gets drowned out.\n\nWhen instead I tried adding the cell features after the pooling operation, the pooler type didn't matter as much.",
"1915032": "A massive thanks for your sharing. It's fascinating to me. I'm trying to re-implement your solution and I have a question that: In the Cell Transformer block, how did you use both the Transformer encoder and the BERT models (the tokens are fed to the encoder or the BERT models first)?",
"1915038": "BERT is just stacks of transformer encoder? For Cell Transformer we are using huggingface models only.",
"1915062": "Take CodeBERT as an example, when the tokens of a cell go through the model, the output dim is 768. This is a 1-shape tensor but you guys said the output's shape is max_len*emb_dim. Could you please tell me why? Your answer will be very helpful for me.",
"1915186": "The output of CodeBert is token embeddings and the dimension of the output is therefore num_tokens * 768. This data is then fed into a pooler that creates a \"sentence\" embedding that has dimension 1*768.",
"1915235": "emb_dim = 768\nembedding of 1 token (output of huggingface model): emb_dim\nembedding of max_len tokens: max_len * emb_dim\nafter concatenate with the code feature embedding: (max_len+1) * emb_dim\nThis happens on every cell, for all cells: n_cells * (max_len+1) * emb_dim\nAfter the attention aggregation (pooler): n_cells * emb_dim",
"1915525": "Awesome!!! So, you mean you take each word of a cell and then pass it to the BERT (not the whole cell's content), right?",
"1919121": "Fantastic work!\nDid you use a variable called `max_num_cells` while training?",
"2026180": "Updated the post with the link to the Github repository.",
"2026821": "Sorry, I can't see the NotebookModel class. Could you please show me its place in your repo?",
"2026944": "I'm sorry, I forgot to add back certain files after renaming. Have just pushed a new tested version.",
"2026951": "Love to hear that from you!"
},
"source": "meta"
}