![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F478989%2Ff7de356d26ff88204073ee34d459e7b5%2FUntitled%20Diagram.drawio.png?generation=1669608703048539&alt=media\")
First, I would like to thank Kaggle and Google and X for hosting this interesting competition.
\nIn my approach, all the markdown and code cells in a notebook are tokenized and concatenated to form a long input sequence, with all the markdown cells go first followed by all the code cells, and they are separated with a [SEP] token. There are also special separator tokens added for markdown (MKDS) and code (CDS) respectively. Once the input goes through the transformer, tokens of each cell is aggregated by meanpooling and a linear layer to map to a single value. During training, the output of each cell is penalized with a pairwise ranking loss which considers all the combinations of cells within a notebook. For inference we can simply sort the output of each cell for correct order.
https://github.com/GuanshuoXu/Google-AI4Code-Understand-Code-in-Python-Notebooks
\nFirst, I would like to thank Kaggle and Google and X for hosting this interesting competition.
\nIn my approach, all the markdown and code cells in a notebook are tokenized and concatenated to form a long input sequence, with all the markdown cells go first followed by all the code cells, and they are separated with a [SEP] token. There are also special separator tokens added for markdown (MKDS) and code (CDS) respectively. Once the input goes through the transformer, tokens of each cell is aggregated by meanpooling and a linear layer to map to a single value. During training, the output of each cell is penalized with a pairwise ranking loss which considers all the combinations of cells within a notebook. For inference we can simply sort the output of each cell for correct order.
https://github.com/GuanshuoXu/Google-AI4Code-Understand-Code-in-Python-Notebooks
\nMany people don't realize that not only your code wins Kaggle competitions.
\nIn terms of code quality: You are on a different level than nearly everyone else on Kaggle.
\nYou get to these results with 10% of the amount of code in comparison to other \"High scoring\" solutions, your solution to Jigsaw had been unbelievably useful for me simply because it is minimalistic:
\nOnly what is actually needed is included. No huge configs you can not replicate. Everything is encapsulated in its own function / simple-file.
Didn't go through this repository yet but I am sure it is 10/10.
\nKeep humiliating us all and thank you for sharing! @wowfattie
", "rawMarkdown": "Many people don't realize that not only your code wins Kaggle competitions. \n\n**In terms of code quality: You are on a different level than nearly everyone else on Kaggle.**\n\nYou get to these results with 10% of the amount of code in comparison to other \"High scoring\" solutions, your solution to Jigsaw had been unbelievably useful for me simply because it is minimalistic: \nOnly what is actually needed is included. No huge configs you can not replicate. Everything is encapsulated in its own function / simple-file.\n\nDidn't go through this repository yet but I am sure it is 10/10. \n\nKeep humiliating us all and thank you for sharing! @wowfattie", "votes": null } ], "comments": [ { "id": 2047174, "author_name": "thedevastator", "author_url": "", "post_date": "11/28/2022 17:00:30", "content": "Many people don't realize that not only your code wins Kaggle competitions.
\nIn terms of code quality: You are on a different level than nearly everyone else on Kaggle.
\nYou get to these results with 10% of the amount of code in comparison to other \"High scoring\" solutions, your solution to Jigsaw had been unbelievably useful for me simply because it is minimalistic:
\nOnly what is actually needed is included. No huge configs you can not replicate. Everything is encapsulated in its own function / simple-file.
Didn't go through this repository yet but I am sure it is 10/10.
\nKeep humiliating us all and thank you for sharing! @wowfattie
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2047147": "#### Overview\nFirst, I would like to thank Kaggle and Google and X for hosting this interesting competition. \nIn my approach, all the markdown and code cells in a notebook are tokenized and concatenated to form a long input sequence, with all the markdown cells go first followed by all the code cells, and they are separated with a [SEP] token. There are also special separator tokens added for markdown (MKDS) and code (CDS) respectively. Once the input goes through the transformer, tokens of each cell is aggregated by meanpooling and a linear layer to map to a single value. During training, the output of each cell is penalized with a pairwise ranking loss which considers all the combinations of cells within a notebook. For inference we can simply sort the output of each cell for correct order. \n\n\n\n#### Source Code\nhttps://github.com/GuanshuoXu/Google-AI4Code-Understand-Code-in-Python-Notebooks\n\n#### What Worked\n- torch.nn.MarginRankingLoss works better than BCE and MSE loss\n- deberta-v3-large is the best transformer in Huggingface for this task\n- longer training sequence (2048) and even longer inference (4096 for validation and 4352 for submission). \n- compress markdown cells with https://gist.github.com/lorey/eb15a7f3338f959a78cc3661fbc255fe\n- compress code cells with https://www.kaggle.com/code/haithamaliryan/ai4code-extract-all-functions-variables-names/notebook\n\n#### What didn't Work\n- seq-to-seq model like T5\n- multilingual models", "2047174": "Many people don't realize that not only your code wins Kaggle competitions. \n\n**In terms of code quality: You are on a different level than nearly everyone else on Kaggle.**\n\nYou get to these results with 10% of the amount of code in comparison to other \"High scoring\" solutions, your solution to Jigsaw had been unbelievably useful for me simply because it is minimalistic: \nOnly what is actually needed is included. No huge configs you can not replicate. Everything is encapsulated in its own function / simple-file.\n\nDidn't go through this repository yet but I am sure it is 10/10. \n\nKeep humiliating us all and thank you for sharing! @wowfattie" }, "source": "meta" }