I've published a complete, end-to-end pipeline for this competition including 3 notebooks and 2 datasets. The final model gets public score of 0.8137. The whole training part takes only 4 hours on Kaggle TPU.
\nThis is basically a translation of Khoi Nguyen's works [1, 2] from PyTorch to TensorFlow with a handful of changes and updates for TPU support. Huge thanks to @suicaokhoailang for his original work!
\nThere's a lot of great public notebooks here already, and yet it seems that none of those really make it possible to enrol the competition using only resources provided by Kaggle. There's even a discussion on whether it's possible to take part without your own GPU.
\nI tried to speed up some of PyTorch public notebooks using PyTorch XLA. It seems I'm not so good at it, so 9 hours per epoch was the best I could get from it. Instead, I decided to take @suicaokhoailang's work and translated PyTorch to TensorFlow instead leaving preprocessing part unchanged. At the end of the day I got a slightly worse score and enormous performance boost: only 4 hours for the entire training process.
", "messages": [ { "id": "1829561", "postDate": "06/22/2022 17:44:34", "content": "I've published a complete, end-to-end pipeline for this competition including 3 notebooks and 2 datasets. The final model gets public score of 0.8137. The whole training part takes only 4 hours on Kaggle TPU.
\nThis is basically a translation of Khoi Nguyen's works [1, 2] from PyTorch to TensorFlow with a handful of changes and updates for TPU support. Huge thanks to @suicaokhoailang for his original work!
\nThere's a lot of great public notebooks here already, and yet it seems that none of those really make it possible to enrol the competition using only resources provided by Kaggle. There's even a discussion on whether it's possible to take part without your own GPU.
\nI tried to speed up some of PyTorch public notebooks using PyTorch XLA. It seems I'm not so good at it, so 9 hours per epoch was the best I could get from it. Instead, I decided to take @suicaokhoailang's work and translated PyTorch to TensorFlow instead leaving preprocessing part unchanged. At the end of the day I got a slightly worse score and enormous performance boost: only 4 hours for the entire training process.
", "rawMarkdown": "# TL;DR\n\nI've published a complete, end-to-end pipeline for this competition including 3 notebooks and 2 datasets. The final model gets **public score of 0.8137**. The whole **training part takes only 4 hours** on Kaggle TPU.\n\n### Notebooks\n\n1. **[AI4Code TF TPU CodeBert - Data Preparation][1]** (~5 hours)\n2. **[AI4Code TF TPU CodeBert - Training][2]** (~4 hours)\n3. **[AI4Code TF TPU CodeBert - Inference][3]** (~2 hours)\n\n### Datasets\n\n1. **[AI4Code CodeBert Tokens][4]** (pre-tokenised inputs and tfrecords)\n2. **[AI4Code CodeBert Weights][5]** (weights of pre-trained CodeBert Base model scoring 0.8137)\n\nThis is basically a translation of **[Khoi Nguyen's][6]** works [[1][7], [2][8]] from PyTorch to TensorFlow with a handful of changes and updates for TPU support. Huge thanks to @suicaokhoailang for his original work!\n\n# About Solution\n\n- Input data: markdown + code context (512 tokens) + features\n - Markdown (up to 64 tokens)\n - Code context (all code cells or up to 20 code cells each up to 23 tokens)\n - Features: markdown cells to total cells ratio (appended to backbone outputs)\n- Model and hyperparameters\n - CodeBert Base model\n - L1 loss (MAE)\n - AdamW optimizer\n - Learning rate schedule with warmup and linear decay\n - Total 5 epochs\n\n# Motivation\n\nThere's a lot of great public notebooks here already, and yet it seems that none of those really make it possible to enrol the competition using only resources provided by Kaggle. There's even a **[discussion][9]** on whether it's possible to take part without your own GPU.\n\nI tried to speed up some of PyTorch public notebooks using PyTorch XLA. It seems I'm not so good at it, so 9 hours per epoch was the best I could get from it. Instead, I decided to take @suicaokhoailang's work and translated PyTorch to TensorFlow instead leaving preprocessing part unchanged. At the end of the day I got a slightly worse score and enormous performance boost: only 4 hours for the entire training process.\n\n[1]: https://www.kaggle.com/nickuzmenkov/ai4code-tf-tpu-codebert-data-preparation\n[2]: https://www.kaggle.com/code/nickuzmenkov/ai4code-tf-tpu-codebert-training\n[3]: https://www.kaggle.com/nickuzmenkov/ai4code-tf-tpu-codebert-inference\n[4]: https://www.kaggle.com/datasets/nickuzmenkov/ai4code-codebert-tokens\n[5]: https://www.kaggle.com/datasets/nickuzmenkov/ai4code-codebert-weights\n[6]: https://www.kaggle.com/suicaokhoailang\n[7]: https://github.com/suicao/ai4code-baseline/tree/main/code\n[8]: https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells\n[9]: https://www.kaggle.com/competitions/AI4Code/discussion/329848", "votes": null }, { "id": "1829686", "postDate": "06/22/2022 20:21:16", "content": "Thanks a lot for sharing this great work!!!!
", "rawMarkdown": "Thanks a lot for sharing this great work!!!!", "votes": null }, { "id": "1829933", "postDate": "06/23/2022 03:58:31", "content": "it's nice when see what's tensorflow on tpu can do.
", "rawMarkdown": "it's nice when see what's tensorflow on tpu can do.", "votes": null }, { "id": "1830490", "postDate": "06/23/2022 13:22:42", "content": "Amazing! Thank you for sharing to everyone.
", "rawMarkdown": "Amazing! Thank you for sharing to everyone.", "votes": null }, { "id": "1834651", "postDate": "06/27/2022 05:39:05", "content": "I should take this opportunity to study TPU. Thank you for the good writing.π
", "rawMarkdown": "I should take this opportunity to study TPU. Thank you for the good writing.π", "votes": null }, { "id": "1835602", "postDate": "06/27/2022 22:47:22", "content": "very amazing workπ
", "rawMarkdown": "very amazing workπ", "votes": null }, { "id": "1842887", "postDate": "07/04/2022 11:21:32", "content": "@nickuzmenkov
\nI ran GPU-based code which took ~7hours to complete of 1 epoch on 1xV100. But your TPU-based code is big improvement on that in terms of training speed. Great work, thanks for sharing. And congrats for becoming Master.
@ipythonx
\nThanks π
Good work! Do you have any speculation on what caused the score drop relative to the pytorch model you adapted? Do you think it was the training batch size alone?
", "rawMarkdown": "Good work! Do you have any speculation on what caused the score drop relative to the pytorch model you adapted? Do you think it was the training batch size alone?", "votes": null }, { "id": "1843805", "postDate": "07/05/2022 06:22:13", "content": "Hi! Thanks
\nIt's hard to tell for sure, but most likely it's due to a couple of factors:
\nI tried a couple of workarounds to make my solution closer to the original and ended up with a custom training loop solution which turned out to be hands down worse than the published baseline. I must have missed something, so I still continue my experiments.
", "rawMarkdown": "Hi! Thanks\n\nIt's hard to tell for sure, but most likely it's due to a couple of factors:\n\n- even for almost identical pipelines PyTorch often yields a better score compared to TensorFlow. This sounds like magic, but I've experienced it many times\n- smaller batch size means more training steps per epoch and a bit more explorational power, so I completely agree with you on this point\n- it's really hard to set up a proper step-wise (not epoch-wise) learning rate schedule in TensorFlow. If you add learning rate monitor to my notebook you'll see that despite properly defined learning rate scheduler, the optimiser doesn't actually update learning rate as expected. This is really annoying thing about TensorFlow, but step-wise learning rate schedules only work in custom training loops\n\nI tried a couple of workarounds to make my solution closer to the original and ended up with a custom training loop solution which turned out to be hands down worse than the published baseline. I must have missed something, so I still continue my experiments.", "votes": null }, { "id": "1847463", "postDate": "07/08/2022 00:29:48", "content": "thanks for sharing. I am using gpu. It seems too slow for training.
", "rawMarkdown": "thanks for sharing. I am using gpu. It seems too slow for training.", "votes": null }, { "id": "1848875", "postDate": "07/09/2022 02:48:02", "content": "Could someone help on checkpoint for TPU on colab? I tried the regular callback, (\"local\" is not implemented) error message
", "rawMarkdown": "Could someone help on checkpoint for TPU on colab? I tried the regular callback, (\"local\" is not implemented) error message", "votes": null }, { "id": "1868151", "postDate": "07/23/2022 18:39:30", "content": "You can get a slightly better score if you increase the number of epochs to 10. Updated solution scores 0.8179 PB. You can visit it here:
\nThanks a lot for sharing this great work!!!!
", "votes": null, "replies": [] }, { "id": 1829933, "author_name": "locbaop", "author_url": "", "post_date": "06/23/2022 03:58:31", "content": "it's nice when see what's tensorflow on tpu can do.
", "votes": null, "replies": [] }, { "id": 1830490, "author_name": "ronaldokun", "author_url": "", "post_date": "06/23/2022 13:22:42", "content": "Amazing! Thank you for sharing to everyone.
", "votes": null, "replies": [] }, { "id": 1834651, "author_name": "heuingttiang", "author_url": "", "post_date": "06/27/2022 05:39:05", "content": "I should take this opportunity to study TPU. Thank you for the good writing.π
", "votes": null, "replies": [] }, { "id": 1835602, "author_name": "jieminren", "author_url": "", "post_date": "06/27/2022 22:47:22", "content": "very amazing workπ
", "votes": null, "replies": [] }, { "id": 1842887, "author_name": "ipythonx", "author_url": "", "post_date": "07/04/2022 11:21:32", "content": "@nickuzmenkov
\nI ran GPU-based code which took ~7hours to complete of 1 epoch on 1xV100. But your TPU-based code is big improvement on that in terms of training speed. Great work, thanks for sharing. And congrats for becoming Master.
@ipythonx
\nThanks π
Good work! Do you have any speculation on what caused the score drop relative to the pytorch model you adapted? Do you think it was the training batch size alone?
", "votes": null, "replies": [ { "id": 1843805, "author_name": "nickuzmenkov", "author_url": "", "post_date": "07/05/2022 06:22:13", "content": "Hi! Thanks
\nIt's hard to tell for sure, but most likely it's due to a couple of factors:
\nI tried a couple of workarounds to make my solution closer to the original and ended up with a custom training loop solution which turned out to be hands down worse than the published baseline. I must have missed something, so I still continue my experiments.
", "votes": null, "replies": [] } ] }, { "id": 1847463, "author_name": "dragonzhang", "author_url": "", "post_date": "07/08/2022 00:29:48", "content": "thanks for sharing. I am using gpu. It seems too slow for training.
", "votes": null, "replies": [] }, { "id": 1848875, "author_name": "dragonzhang", "author_url": "", "post_date": "07/09/2022 02:48:02", "content": "Could someone help on checkpoint for TPU on colab? I tried the regular callback, (\"local\" is not implemented) error message
", "votes": null, "replies": [] }, { "id": 1868151, "author_name": "nickuzmenkov", "author_url": "", "post_date": "07/23/2022 18:39:30", "content": "You can get a slightly better score if you increase the number of epochs to 10. Updated solution scores 0.8179 PB. You can visit it here:
\n