![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F743064%2Fc42b70aafb899ed52cf2ef7ed554e4f7%2Fcode_bert.png?generation=1660273106105753&alt=media\")
The solution is based on the bi-encoder or poly-encoder like approach, when every cell is individually processed by the codebert, with the code/md interaction modelled with the pair of parallel TransformerDecoderLayer:
\n
I either used the shared CodeBERT based encoder for both code and markdown cells or CodeBERT for code and multi language bert for markdown. The resulting activations are averaged over tokens.
\nThe code cell activations are passed to 1D convolution with zero padding, to convert from the N code cell positions to N+1 positions between cells (bins to put mb cell to), with every point combined activations of the cell before and after.
\nAfter adding the pos encoding to the code activations (the combination of the absolute and relative encodings).
\nThe next stage is a few layers (2 or 6) of transformer decoders, for code cell to use the self attention and attention to md cells as a memory, the same for markdown: self attention and the memory query over the output of the code cell:
\nfor step in range(self.num_decoder_layers):\n x_code = self.code_decoders[step](x_code, x_md)\n x_md = self.md_decoders[step](x_md, x_code)\nOutputs of the code and markdown decoders are combined with matmul similarly to attention in a transformer,
\nwith 3 outputs:
(nb_code+1) * nb_markdown shape, is markdown cell i before the code bin j
(nb_code+1) * nb_markdown shape, is markdown cell i at the code bin j
nb_markdown * nb_markdown is the markdown cell i before another markdown cell j
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F743064%2F863249a74f079b19943a7f6a46689e8e%2Fmodel_outputs.png?generation=1668982985055751&alt=media\")
The first two outputs are used to put the md cell to the right bin between the code cells, and the last output is used to sort md cells within the single bin. BCE loss was used during training.
\nMD cell is put to the right bin between code cells using the first prediction: probability of the MD cell i is before code bin j. The position is selected to minimize the sum of probabilities of the wrong placement over all bins:
\nmd_cell_bin_num = np.argmin([\n -1 * md_after_code[md_idx, :i+1].sum() + md_after_code[md_idx, i+1:].sum()\n for i in range(nb_bins)\n])\nSuch approach to select the markdown cell position worked much better compared to the simple \"is markdown cell i at the code bin j\" output + softmax/argmax. The intuition behind the loss and sorting approach was to have a proxy for expected number of cell swaps used in the competition metric.
\nThe model was trained end to end, with the batch equal to a single notebook, I guess this makes the optimisation task harder due to the variable size of notebooks. Gradient clipping helped.
\nThe loss weighting and notebooks sampling also had a significant impact on the score. Large notebooks have a larger impact on the final score (due to the large number of position inversions for wrong predictions) but the wrong prediction error is averaged over a large number of nb_md * nb_code predictions. To address this I sampled larger notebooks more often (most useful) and for some models added avg_loss + a * max_loss + b * sum_loss (with a very small a of 0.02, adding sum_loss helped for initial experiments but was disabled later).
\nI tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.
\nExample of the performance difference between approaches (from the local validation, on one of folds):
\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652
\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825
The multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).
\nI initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.
\nTo increase the ensemble side, I also trained the separate encoder models (using per-cell predictions from the first level transformers). Such encoder only parts of models were much faster to train (single per notebook transformer compared to multiple ber cell transformers, allowed to faster experiment with different architectures) and to predict (L1 per cell model predictions are re-used).
\nThe ensemble - to average 3 outputs between 4 folds and two models, including two versions of L2 encoders.
\nFor ensembling I used the simple weighted averaging of model predictions (weighting was not very important, used higher weight for better performing models but the result was almost the same as for unweighted average).
Models have been trained on the original dataset.
\nI tried to use the published external set of notebooks with acceptable licenses https://zenodo.org/record/6383115, it seems to improve results. The dataset was mentioned closer to the end of the challenge so I did not have enough time to re-train the full models. I trained the additional encoders using the external dataset notebooks, the result of individual improved, but the improvement of the final ensemble seems to be quite small.
Time permitting, I would re-train models using the external data.
\nThe dataset converted to the challenge format: https://www.kaggle.com/datasets/dmytropoplavskiy/the-dataset-of-properlylicensed-jupyter-notebooks
\nT5 based models performed worse
Different pre-processing of code and markdown cells had little impact, so used unprocessed.
Different ways to combine per cell model outputs performed in a similar way (using average, avg+max, output of the special code token).
Using GRU/LSTM instead of transformer, for the encoder part or GRU for code with transformer encoder for md/code. The performance was worse compared to the dual branch transformer decoder.
The solution is based on the bi-encoder or poly-encoder like approach, when every cell is individually processed by the codebert, with the code/md interaction modelled with the pair of parallel TransformerDecoderLayer:
\n
I either used the shared CodeBERT based encoder for both code and markdown cells or CodeBERT for code and multi language bert for markdown. The resulting activations are averaged over tokens.
\nThe code cell activations are passed to 1D convolution with zero padding, to convert from the N code cell positions to N+1 positions between cells (bins to put mb cell to), with every point combined activations of the cell before and after.
\nAfter adding the pos encoding to the code activations (the combination of the absolute and relative encodings).
\nThe next stage is a few layers (2 or 6) of transformer decoders, for code cell to use the self attention and attention to md cells as a memory, the same for markdown: self attention and the memory query over the output of the code cell:
\nfor step in range(self.num_decoder_layers):\n x_code = self.code_decoders[step](x_code, x_md)\n x_md = self.md_decoders[step](x_md, x_code)\nOutputs of the code and markdown decoders are combined with matmul similarly to attention in a transformer,
\nwith 3 outputs:
(nb_code+1) * nb_markdown shape, is markdown cell i before the code bin j
(nb_code+1) * nb_markdown shape, is markdown cell i at the code bin j
nb_markdown * nb_markdown is the markdown cell i before another markdown cell j
The first two outputs are used to put the md cell to the right bin between the code cells, and the last output is used to sort md cells within the single bin. BCE loss was used during training.
\nMD cell is put to the right bin between code cells using the first prediction: probability of the MD cell i is before code bin j. The position is selected to minimize the sum of probabilities of the wrong placement over all bins:
\nmd_cell_bin_num = np.argmin([\n -1 * md_after_code[md_idx, :i+1].sum() + md_after_code[md_idx, i+1:].sum()\n for i in range(nb_bins)\n])\nSuch approach to select the markdown cell position worked much better compared to the simple \"is markdown cell i at the code bin j\" output + softmax/argmax. The intuition behind the loss and sorting approach was to have a proxy for expected number of cell swaps used in the competition metric.
\nThe model was trained end to end, with the batch equal to a single notebook, I guess this makes the optimisation task harder due to the variable size of notebooks. Gradient clipping helped.
\nThe loss weighting and notebooks sampling also had a significant impact on the score. Large notebooks have a larger impact on the final score (due to the large number of position inversions for wrong predictions) but the wrong prediction error is averaged over a large number of nb_md * nb_code predictions. To address this I sampled larger notebooks more often (most useful) and for some models added avg_loss + a * max_loss + b * sum_loss (with a very small a of 0.02, adding sum_loss helped for initial experiments but was disabled later).
\nI tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.
\nExample of the performance difference between approaches (from the local validation, on one of folds):
\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652
\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825
The multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).
\nI initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.
\nTo increase the ensemble side, I also trained the separate encoder models (using per-cell predictions from the first level transformers). Such encoder only parts of models were much faster to train (single per notebook transformer compared to multiple ber cell transformers, allowed to faster experiment with different architectures) and to predict (L1 per cell model predictions are re-used).
\nThe ensemble - to average 3 outputs between 4 folds and two models, including two versions of L2 encoders.
\nFor ensembling I used the simple weighted averaging of model predictions (weighting was not very important, used higher weight for better performing models but the result was almost the same as for unweighted average).
Models have been trained on the original dataset.
\nI tried to use the published external set of notebooks with acceptable licenses https://zenodo.org/record/6383115, it seems to improve results. The dataset was mentioned closer to the end of the challenge so I did not have enough time to re-train the full models. I trained the additional encoders using the external dataset notebooks, the result of individual improved, but the improvement of the final ensemble seems to be quite small.
Time permitting, I would re-train models using the external data.
\nThe dataset converted to the challenge format: https://www.kaggle.com/datasets/dmytropoplavskiy/the-dataset-of-properlylicensed-jupyter-notebooks
\nT5 based models performed worse
Different pre-processing of code and markdown cells had little impact, so used unprocessed.
Different ways to combine per cell model outputs performed in a similar way (using average, avg+max, output of the special code token).
Using GRU/LSTM instead of transformer, for the encoder part or GRU for code with transformer encoder for md/code. The performance was worse compared to the dual branch transformer decoder.
Congratulations！ Really creative and elegant solution, thanks for sharing.
", "rawMarkdown": "Congratulations！ Really creative and elegant solution, thanks for sharing.", "votes": 3 }, { "id": 1895687, "postDate": "2022-08-12T10:16:37.257Z", "content": "Thank you. Will you share your code maybe after the competition ends? It will be helpful
", "rawMarkdown": "Thank you. Will you share your code maybe after the competition ends? It will be helpful ", "votes": 4 }, { "id": 1896510, "postDate": "2022-08-12T21:23:14.270Z", "content": "Great job!
", "rawMarkdown": "Great job! ", "votes": 1 }, { "id": 1895941, "postDate": "2022-08-12T13:00:25.240Z", "content": "Wow, learned something new. Thanks for sharing 😊
", "rawMarkdown": "Wow, learned something new. Thanks for sharing 😊", "votes": 1 }, { "id": 1895583, "postDate": "2022-08-12T08:50:15.040Z", "content": "Congratulations! An elegant end-to-end solution that takes into account the inherent order between Markdown cells. I learned a lot from it.
", "rawMarkdown": "Congratulations! An elegant end-to-end solution that takes into account the inherent order between Markdown cells. I learned a lot from it.", "votes": 1 }, { "id": 1895437, "postDate": "2022-08-12T06:20:37.760Z", "content": "Amazing solution.
", "rawMarkdown": "Amazing solution.", "votes": 1 }, { "id": 1895428, "postDate": "2022-08-12T06:10:05.997Z", "content": "Fantastic explanation!! Very new and innovative approach!!
\nCongrats for the results, your approach is a learning experience for people like me
Congrats . This post was very helpful for AI4Code competition.
", "rawMarkdown": "Congrats . This post was very helpful for AI4Code competition.", "votes": 1 }, { "id": 1896622, "postDate": "2022-08-13T01:49:57.083Z", "content": "Super interesting architecture, thank you for sharing!
\nI have a a couple of questions about the details:
\nAnd a more general question: what was the thinking process when you decided to go for this kind of solution, which is relatively off the beaten path?
", "rawMarkdown": "Super interesting architecture, thank you for sharing!\n\nI have a a couple of questions about the details:\n1. Why did you create model heads both for predicting if markdown cell i is before the code bin j and for if markdown cell i is at the code bin j?\n2. Have you tried running this model without the decoder layers? I'd be curious to know how much they contributed.\n\nAnd a more general question: what was the thinking process when you decided to go for this kind of solution, which is relatively off the beaten path?", "votes": 2, "replies": [ { "id": 1896735, "postDate": "2022-08-13T05:34:41.467Z", "content": "I experimented with using both md before code (implemented it first) and md exactly at the code bin.
\nDuring inference the md before code approach worked much better (I select the bin which minimizes the total sum of probabilities of the wrong prediction, as a reasonable proxy for the competition metric).
\nWhen I added the softmax based \"md at the code bin\" output, I had a slightly better result, but it may be the side effect of larger loss values for notebooks with larger number of code cells.
\nOverall to was the hard part, how to design the loss and how to batch/sample notebooks, with such irregular notebook size. The larger notebooks have much bigger impact on the final score, but I'd expect the large variations between notebook sizes would make optimization harder. It helped to sample larger notebooks more often (relative to the notebook size + constant) but still average over all outputs. When I tried to train only the decoder part, I also experimented with using the sum of all outputs and accumulating notebooks gradients up until some total nb_md*nb_code threshold, this also worked. It also helped to add gradient clipping.
\nI have not tried to run models without encoders, it would not be able to model dependencies between different code and md cells. I tried to use GRU instead of code branch transformers and the result was somewhat worse.
\nIt was the first time I dealt with NLP task so I did not really know about the common approaches (it also means I most likely missed some obvious improvements). I tried to build the model with a suitable information flow and all the necessary information available for prediction. I was not sure if attention-like design would work well for ordering of md/code cells, but it seems to work. I tried to improve the architecture (like calculation of the mean/max activations for cells before and after the current one, to keep the transformer capacity for more complex tasks but the result was the same), I found the training hyperparameters had a bigger impact on the result.
", "rawMarkdown": "I experimented with using both md before code (implemented it first) and md exactly at the code bin.\n\nDuring inference the md before code approach worked much better (I select the bin which minimizes the total sum of probabilities of the wrong prediction, as a reasonable proxy for the competition metric).\n\nWhen I added the softmax based \"md at the code bin\" output, I had a slightly better result, but it may be the side effect of larger loss values for notebooks with larger number of code cells.\n\nOverall to was the hard part, how to design the loss and how to batch/sample notebooks, with such irregular notebook size. The larger notebooks have much bigger impact on the final score, but I'd expect the large variations between notebook sizes would make optimization harder. It helped to sample larger notebooks more often (relative to the notebook size + constant) but still average over all outputs. When I tried to train only the decoder part, I also experimented with using the sum of all outputs and accumulating notebooks gradients up until some total nb_md*nb_code threshold, this also worked. It also helped to add gradient clipping.\n\nI have not tried to run models without encoders, it would not be able to model dependencies between different code and md cells. I tried to use GRU instead of code branch transformers and the result was somewhat worse.\n\nIt was the first time I dealt with NLP task so I did not really know about the common approaches (it also means I most likely missed some obvious improvements). I tried to build the model with a suitable information flow and all the necessary information available for prediction. I was not sure if attention-like design would work well for ordering of md/code cells, but it seems to work. I tried to improve the architecture (like calculation of the mean/max activations for cells before and after the current one, to keep the transformer capacity for more complex tasks but the result was the same), I found the training hyperparameters had a bigger impact on the result. \n", "votes": 5 } ] }, { "id": 1895443, "postDate": "2022-08-12T06:24:01.977Z", "content": "Great work. I can't believe it works.👍
", "rawMarkdown": "Great work. I can't believe it works.👍", "votes": 2 }, { "id": 1980820, "postDate": "2022-10-10T12:09:11.460Z", "content": "Congrats on the result!
\nWould you please mind citing my article from 3 months ago, in case that was the source of the design?
@dmytropoplavskiy I'm wondering if this solution was inspired by recent work in computer vision regarding decoders for multi-label classification, e.g. DETR, UNIT, Query2Label, ML-Decoder?
", "rawMarkdown": "@dmytropoplavskiy I'm wondering if this solution was inspired by recent work in computer vision regarding decoders for multi-label classification, e.g. DETR, UNIT, Query2Label, ML-Decoder?", "replies": [ { "id": 1951394, "postDate": "2022-09-23T03:26:18.147Z", "content": "Hi, probably to some extent at least, I was impressed with how much DETR simplified the usual detection pipeline, with anchors, NMS etc. I tried to use something similar for the Lift challenge but could not beat the simple CNN baseline. It's interesting to see of how much the NLP and CV approaches started to overlap.
", "rawMarkdown": "Hi, probably to some extent at least, I was impressed with how much DETR simplified the usual detection pipeline, with anchors, NMS etc. I tried to use something similar for the Lift challenge but could not beat the simple CNN baseline. It's interesting to see of how much the NLP and CV approaches started to overlap.", "votes": 1 } ] }, { "id": 1901796, "postDate": "2022-08-16T23:26:41.423Z", "content": "Great work!
", "rawMarkdown": "Great work!" }, { "id": 1900200, "postDate": "2022-08-15T19:14:15.893Z", "content": "Great work!
", "rawMarkdown": "Great work!" }, { "id": 1900138, "postDate": "2022-08-15T18:20:39.770Z", "content": "insightful
", "rawMarkdown": "insightful" }, { "id": 1900030, "postDate": "2022-08-15T16:52:37.123Z", "content": "very interesting solution! And congrats!
\nAnd, I really like to learn from your code as well if you don't mind sharing it…
Wow this is a bit complex
", "rawMarkdown": "Wow this is a bit complex" }, { "id": 1899260, "postDate": "2022-08-15T06:42:13.200Z", "content": "nice work
", "rawMarkdown": "nice work\n" }, { "id": 1899060, "postDate": "2022-08-15T03:35:19.620Z", "content": "Thank you for such useful sharing! Learned many things and can't wait to apply it
", "rawMarkdown": "Thank you for such useful sharing! Learned many things and can't wait to apply it" }, { "id": 1898878, "postDate": "2022-08-14T23:25:12.700Z", "content": "Thanks for sharing your approach. Very insightful.
", "rawMarkdown": "Thanks for sharing your approach. Very insightful." }, { "id": 1898148, "postDate": "2022-08-14T10:56:40.903Z", "content": "This was so insightful! Thanks for sharing💯
", "rawMarkdown": "This was so insightful! Thanks for sharing💯" }, { "id": 1897654, "postDate": "2022-08-14T01:19:20.520Z", "content": "Thanks for the explanation, very helpful.
", "rawMarkdown": "Thanks for the explanation, very helpful. " }, { "id": 1897344, "postDate": "2022-08-13T17:09:14.653Z", "content": "Cool solution
", "rawMarkdown": "Cool solution" }, { "id": 1897222, "postDate": "2022-08-13T15:00:24.580Z", "content": "Thanks for sharing, interesting @dmytropoplavskiy
", "rawMarkdown": "Thanks for sharing, interesting @dmytropoplavskiy " }, { "id": 1933980, "postDate": "2022-09-11T03:25:17.330Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1910139, "postDate": "2022-08-23T08:06:06.390Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1899251, "postDate": "2022-08-15T06:31:03.293Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1898876, "postDate": "2022-08-14T23:10:01.447Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1895319, "postDate": "2022-08-12T04:36:23.677Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1927066, "postDate": "2022-09-05T11:05:46.327Z", "content": "Thank you for your creative ideas
", "rawMarkdown": "Thank you for your creative ideas" }, { "id": 1896896, "postDate": "2022-08-13T08:54:07.330Z", "content": "thanks for shering
", "rawMarkdown": "thanks for shering\n" } ], "comments": [ { "id": 1895279, "author_name": "gezi", "author_url": "", "post_date": "2022-08-12T04:02:46.130000", "content": "Congratulations！ Really creative and elegant solution, thanks for sharing.
", "votes": 3, "replies": [] }, { "id": 1895687, "author_name": "RoBi", "author_url": "", "post_date": "2022-08-12T10:16:37.257000", "content": "Thank you. Will you share your code maybe after the competition ends? It will be helpful
", "votes": 4, "replies": [] }, { "id": 1896510, "author_name": "li hui", "author_url": "", "post_date": "2022-08-12T21:23:14.270000", "content": "Great job!
", "votes": 1, "replies": [] }, { "id": 1895941, "author_name": "Subhadeep Debnath", "author_url": "", "post_date": "2022-08-12T13:00:25.240000", "content": "Wow, learned something new. Thanks for sharing 😊
", "votes": 1, "replies": [] }, { "id": 1895583, "author_name": "Bob Simth", "author_url": "", "post_date": "2022-08-12T08:50:15.040000", "content": "Congratulations! An elegant end-to-end solution that takes into account the inherent order between Markdown cells. I learned a lot from it.
", "votes": 1, "replies": [] }, { "id": 1895437, "author_name": "[START]", "author_url": "", "post_date": "2022-08-12T06:20:37.760000", "content": "Amazing solution.
", "votes": 1, "replies": [] }, { "id": 1895428, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-08-12T06:10:05.997000", "content": "Fantastic explanation!! Very new and innovative approach!!
\nCongrats for the results, your approach is a learning experience for people like me
Congrats . This post was very helpful for AI4Code competition.
", "votes": 1, "replies": [] }, { "id": 1896622, "author_name": "Cabbage972", "author_url": "", "post_date": "2022-08-13T01:49:57.083000", "content": "Super interesting architecture, thank you for sharing!
\nI have a a couple of questions about the details:
\nAnd a more general question: what was the thinking process when you decided to go for this kind of solution, which is relatively off the beaten path?
", "votes": 2, "replies": [ { "id": 1896735, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2022-08-13T05:34:41.467000", "content": "I experimented with using both md before code (implemented it first) and md exactly at the code bin.
\nDuring inference the md before code approach worked much better (I select the bin which minimizes the total sum of probabilities of the wrong prediction, as a reasonable proxy for the competition metric).
\nWhen I added the softmax based \"md at the code bin\" output, I had a slightly better result, but it may be the side effect of larger loss values for notebooks with larger number of code cells.
\nOverall to was the hard part, how to design the loss and how to batch/sample notebooks, with such irregular notebook size. The larger notebooks have much bigger impact on the final score, but I'd expect the large variations between notebook sizes would make optimization harder. It helped to sample larger notebooks more often (relative to the notebook size + constant) but still average over all outputs. When I tried to train only the decoder part, I also experimented with using the sum of all outputs and accumulating notebooks gradients up until some total nb_md*nb_code threshold, this also worked. It also helped to add gradient clipping.
\nI have not tried to run models without encoders, it would not be able to model dependencies between different code and md cells. I tried to use GRU instead of code branch transformers and the result was somewhat worse.
\nIt was the first time I dealt with NLP task so I did not really know about the common approaches (it also means I most likely missed some obvious improvements). I tried to build the model with a suitable information flow and all the necessary information available for prediction. I was not sure if attention-like design would work well for ordering of md/code cells, but it seems to work. I tried to improve the architecture (like calculation of the mean/max activations for cells before and after the current one, to keep the transformer capacity for more complex tasks but the result was the same), I found the training hyperparameters had a bigger impact on the result.
", "votes": 5, "replies": [] } ] }, { "id": 1895443, "author_name": "hyd", "author_url": "", "post_date": "2022-08-12T06:24:01.977000", "content": "Great work. I can't believe it works.👍
", "votes": 2, "replies": [] }, { "id": 1980820, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2022-10-10T12:09:11.460000", "content": "Congrats on the result!
\nWould you please mind citing my article from 3 months ago, in case that was the source of the design?
@dmytropoplavskiy I'm wondering if this solution was inspired by recent work in computer vision regarding decoders for multi-label classification, e.g. DETR, UNIT, Query2Label, ML-Decoder?
", "votes": 0, "replies": [ { "id": 1951394, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2022-09-23T03:26:18.147000", "content": "Hi, probably to some extent at least, I was impressed with how much DETR simplified the usual detection pipeline, with anchors, NMS etc. I tried to use something similar for the Lift challenge but could not beat the simple CNN baseline. It's interesting to see of how much the NLP and CV approaches started to overlap.
", "votes": 1, "replies": [] } ] }, { "id": 1901796, "author_name": "Igor Araújo de Melo Monticello", "author_url": "", "post_date": "2022-08-16T23:26:41.423000", "content": "Great work!
", "votes": 0, "replies": [] }, { "id": 1900200, "author_name": "Clément DECAUX", "author_url": "", "post_date": "2022-08-15T19:14:15.893000", "content": "Great work!
", "votes": 0, "replies": [] }, { "id": 1900138, "author_name": "Shivam", "author_url": "", "post_date": "2022-08-15T18:20:39.770000", "content": "insightful
", "votes": 0, "replies": [] }, { "id": 1900030, "author_name": "kyle", "author_url": "", "post_date": "2022-08-15T16:52:37.123000", "content": "very interesting solution! And congrats!
\nAnd, I really like to learn from your code as well if you don't mind sharing it…
Wow this is a bit complex
", "votes": 0, "replies": [] }, { "id": 1899260, "author_name": "AMAN BARKAT BHANVADIA", "author_url": "", "post_date": "2022-08-15T06:42:13.200000", "content": "nice work
", "votes": 0, "replies": [] }, { "id": 1899060, "author_name": "Kaijie Huang", "author_url": "", "post_date": "2022-08-15T03:35:19.620000", "content": "Thank you for such useful sharing! Learned many things and can't wait to apply it
", "votes": 0, "replies": [] }, { "id": 1898878, "author_name": "Oscar Aguilar", "author_url": "", "post_date": "2022-08-14T23:25:12.700000", "content": "Thanks for sharing your approach. Very insightful.
", "votes": 0, "replies": [] }, { "id": 1898148, "author_name": "Jasleen Sondhi", "author_url": "", "post_date": "2022-08-14T10:56:40.903000", "content": "This was so insightful! Thanks for sharing💯
", "votes": 0, "replies": [] }, { "id": 1897654, "author_name": "Harry Mapodile", "author_url": "", "post_date": "2022-08-14T01:19:20.520000", "content": "Thanks for the explanation, very helpful.
", "votes": 0, "replies": [] }, { "id": 1897344, "author_name": "Kiwiegg", "author_url": "", "post_date": "2022-08-13T17:09:14.653000", "content": "Cool solution
", "votes": 0, "replies": [] }, { "id": 1897222, "author_name": "Muhammed Tausif", "author_url": "", "post_date": "2022-08-13T15:00:24.580000", "content": "Thanks for sharing, interesting @dmytropoplavskiy
", "votes": 0, "replies": [] }, { "id": 1933980, "author_name": "", "author_url": "", "post_date": "2022-09-11T03:25:17.330000", "content": "", "votes": 0, "replies": [] }, { "id": 1910139, "author_name": "", "author_url": "", "post_date": "2022-08-23T08:06:06.390000", "content": "", "votes": 0, "replies": [] }, { "id": 1899251, "author_name": "", "author_url": "", "post_date": "2022-08-15T06:31:03.293000", "content": "", "votes": 0, "replies": [] }, { "id": 1898876, "author_name": "", "author_url": "", "post_date": "2022-08-14T23:10:01.447000", "content": "", "votes": 0, "replies": [] }, { "id": 1895319, "author_name": "", "author_url": "", "post_date": "2022-08-12T04:36:23.677000", "content": "", "votes": 0, "replies": [] }, { "id": 1927066, "author_name": "", "author_url": "", "post_date": "2022-09-05T11:05:46.327000", "content": "", "votes": 0, "replies": [] }, { "id": 1896896, "author_name": "", "author_url": "", "post_date": "2022-08-13T08:54:07.330000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1895262": "The solution is based on the bi-encoder or poly-encoder like approach, when every cell is individually processed by the codebert, with the code/md interaction modelled with the pair of parallel TransformerDecoderLayer:\n\n\n\n\nI either used the shared CodeBERT based encoder for both code and markdown cells or CodeBERT for code and multi language bert for markdown. The resulting activations are averaged over tokens.\n\nThe code cell activations are passed to 1D convolution with zero padding, to convert from the N code cell positions to N+1 positions between cells (bins to put mb cell to), with every point combined activations of the cell before and after.\n\nAfter adding the pos encoding to the code activations (the combination of the absolute and relative encodings).\n\nThe next stage is a few layers (2 or 6) of transformer decoders, for code cell to use the self attention and attention to md cells as a memory, the same for markdown: self attention and the memory query over the output of the code cell:\n \n\n for step in range(self.num_decoder_layers):\n x_code = self.code_decoders[step](x_code, x_md)\n x_md = self.md_decoders[step](x_md, x_code)\n\n\n\nOutput and loss\n===========\n\nOutputs of the code and markdown decoders are combined with matmul similarly to attention in a transformer,\nwith 3 outputs:\n\n* (nb_code+1) * nb_markdown shape, is markdown cell i before the code bin j\n\n* (nb_code+1) * nb_markdown shape, is markdown cell i at the code bin j\n\n* nb_markdown * nb_markdown is the markdown cell i before another markdown cell j\n\n\n\nThe first two outputs are used to put the md cell to the right bin between the code cells, and the last output is used to sort md cells within the single bin. BCE loss was used during training.\n\nPost-processing\n============\n\nMD cell is put to the right bin between code cells using the first prediction: probability of the MD cell i is before code bin j. The position is selected to minimize the sum of probabilities of the wrong placement over all bins:\n\n\n md_cell_bin_num = np.argmin([\n -1 * md_after_code[md_idx, :i+1].sum() + md_after_code[md_idx, i+1:].sum()\n for i in range(nb_bins)\n ])\n\nSuch approach to select the markdown cell position worked much better compared to the simple \"is markdown cell i at the code bin j\" output + softmax/argmax. The intuition behind the loss and sorting approach was to have a proxy for expected number of cell swaps used in the competition metric.\n\nThe model was trained end to end, with the batch equal to a single notebook, I guess this makes the optimisation task harder due to the variable size of notebooks. Gradient clipping helped.\n\nThe loss weighting and notebooks sampling also had a significant impact on the score. Large notebooks have a larger impact on the final score (due to the large number of position inversions for wrong predictions) but the wrong prediction error is averaged over a large number of nb_md * nb_code predictions. To address this I sampled larger notebooks more often (most useful) and for some models added avg_loss + a * max_loss + b * sum_loss (with a very small a of 0.02, adding sum_loss helped for initial experiments but was disabled later).\n\nNon-English notebooks handling\n=======================\n\nI tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.\n\nExample of the performance difference between approaches (from the local validation, on one of folds):\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825\n\nThe multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).\n\nI initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.\n\nEnsembling\n========\n\nTo increase the ensemble side, I also trained the separate encoder models (using per-cell predictions from the first level transformers). Such encoder only parts of models were much faster to train (single per notebook transformer compared to multiple ber cell transformers, allowed to faster experiment with different architectures) and to predict (L1 per cell model predictions are re-used).\n\nThe ensemble - to average 3 outputs between 4 folds and two models, including two versions of L2 encoders.\nFor ensembling I used the simple weighted averaging of model predictions (weighting was not very important, used higher weight for better performing models but the result was almost the same as for unweighted average).\n\nExternal data\n==========\n\nModels have been trained on the original dataset.\nI tried to use the published external set of notebooks with acceptable licenses https://zenodo.org/record/6383115, it seems to improve results. The dataset was mentioned closer to the end of the challenge so I did not have enough time to re-train the full models. I trained the additional encoders using the external dataset notebooks, the result of individual improved, but the improvement of the final ensemble seems to be quite small.\n\nTime permitting, I would re-train models using the external data.\n\nThe dataset converted to the challenge format: https://www.kaggle.com/datasets/dmytropoplavskiy/the-dataset-of-properlylicensed-jupyter-notebooks\n\n\nWhat did not work or had no significant impact\n=================================\n\n* T5 based models performed worse\n\n* Different pre-processing of code and markdown cells had little impact, so used unprocessed.\n\n* Different ways to combine per cell model outputs performed in a similar way (using average, avg+max, output of the special code token).\n\n* Using GRU/LSTM instead of transformer, for the encoder part or GRU for code with transformer encoder for md/code. The performance was worse compared to the dual branch transformer decoder.\n\n\nSource code\n=========\n\n", "1895279": "Congratulations！ Really creative and elegant solution, thanks for sharing.", "1895687": "Thank you. Will you share your code maybe after the competition ends? It will be helpful ", "1896510": "Great job! ", "1895941": "Wow, learned something new. Thanks for sharing 😊", "1895583": "Congratulations! An elegant end-to-end solution that takes into account the inherent order between Markdown cells. I learned a lot from it.", "1895437": "Amazing solution.", "1895428": "Fantastic explanation!! Very new and innovative approach!! \nCongrats for the results, your approach is a learning experience for people like me", "1895341": "Congrats . This post was very helpful for AI4Code competition.", "1896622": "Super interesting architecture, thank you for sharing!\n\nI have a a couple of questions about the details:\n1. Why did you create model heads both for predicting if markdown cell i is before the code bin j and for if markdown cell i is at the code bin j?\n2. Have you tried running this model without the decoder layers? I'd be curious to know how much they contributed.\n\nAnd a more general question: what was the thinking process when you decided to go for this kind of solution, which is relatively off the beaten path?", "1895443": "Great work. I can't believe it works.👍", "1980820": "Congrats on the result! \nWould you please mind citing my [article](https://www.kaggle.com/competitions/AI4Code/discussion/336756) from 3 months ago, in case that was the source of the design? ", "1948297": "@dmytropoplavskiy I'm wondering if this solution was inspired by recent work in computer vision regarding decoders for multi-label classification, e.g. DETR, UNIT, Query2Label, ML-Decoder?", "1901796": "Great work!", "1900200": "Great work!", "1900138": "insightful", "1900030": "very interesting solution! And congrats! \nAnd, I really like to learn from your code as well if you don't mind sharing it...", "1899816": "Wow this is a bit complex", "1899260": "nice work\n", "1899060": "Thank you for such useful sharing! Learned many things and can't wait to apply it", "1898878": "Thanks for sharing your approach. Very insightful.", "1898148": "This was so insightful! Thanks for sharing💯", "1897654": "Thanks for the explanation, very helpful. ", "1897344": "Cool solution", "1897222": "Thanks for sharing, interesting @dmytropoplavskiy ", "1933980": "", "1910139": "", "1899251": "", "1898876": "", "1895319": "", "1927066": "Thank you for your creative ideas", "1896896": "thanks for shering\n" } }