As a TensorFlow user, I was upset that I didn't find a single TensorFlow public notebook for this competition. So I decided to translate this AI4Code Pytorch DistilBert Baseline from PyTorch to TensorFlow:
\nIt's configured to run on TPU taking only 6 minutes per epoch for more than 2M training samples, leaving you with a plenty of room for experiments.
\nMany thanks to @aerdem4 for the original work!
", "messages": [ { "id": "1816668", "postDate": "06/10/2022 12:35:46", "content": "As a TensorFlow user, I was upset that I didn't find a single TensorFlow public notebook for this competition. So I decided to translate this AI4Code Pytorch DistilBert Baseline from PyTorch to TensorFlow:
\nIt's configured to run on TPU taking only 6 minutes per epoch for more than 2M training samples, leaving you with a plenty of room for experiments.
\nMany thanks to @aerdem4 for the original work!
", "rawMarkdown": "### Hello!\n\nAs a TensorFlow user, I was upset that I didn't find a single TensorFlow public notebook for this competition. So I decided to translate this **[AI4Code Pytorch DistilBert Baseline][0]** from PyTorch to TensorFlow:\n\n### [AI4Code TensorFlow DistilBert Baseline][1]\n\nIt's configured to run on TPU taking only 6 minutes per epoch for more than 2M training samples, leaving you with a plenty of room for experiments.\n\nMany thanks to @aerdem4 for the original work!\n\n[0]: https://www.kaggle.com/code/aerdem4/ai4code-pytorch-distilbert-baseline\n[1]: https://www.kaggle.com/code/nickuzmenkov/ai4code-tensorflow-distilbert-baseline", "votes": null }, { "id": "1817968", "postDate": "06/12/2022 04:46:24", "content": "Thanks very much for this! Anyone on a budget (even a Colab Pro+ budget ;) ) should look at this, seriously.
\nI'm looking at this on colab, and increasing the batch size (8192 looks good to me) appears to bring a notable speed improvement - considering that data has to be hauled over to the TPU in chunks, that makes sense.
", "rawMarkdown": "Thanks very much for this! Anyone on a budget (even a Colab Pro+ budget ;) ) should look at this, seriously.\n\nI'm looking at this on colab, and increasing the batch size (8192 looks good to me) appears to bring a notable speed improvement - considering that data has to be hauled over to the TPU in chunks, that makes sense.", "votes": null }, { "id": "1818107", "postDate": "06/12/2022 09:25:45", "content": "\n\nI didn't find a single TensorFlow public notebook for this competition
\n
Same here. Nobody seems to use TF these days. There seems to be a fundamental problem with using TF in competitions. :/
", "rawMarkdown": "> I didn't find a single TensorFlow public notebook for this competition\n\nSame here. Nobody seems to use TF these days. There seems to be a fundamental problem with using TF in competitions. :/", "votes": null }, { "id": "1819880", "postDate": "06/14/2022 08:00:33", "content": "You're right.
\nI haven't used Torch a lot, but at first glance it seems to be more user friendly and flexible despite it requires way more code for the same task and thus seems harder to quickstart compared to Keras. However, if that's critical, Torch Lightning totally eliminates this.
\nTorch also has more straightforward and intuitive API. I've seen somebody complaining here that there's exactly 5 ways to initialise convolutional layer in TF and Keras. Sounds like a joke, but it's true.
\nTF gives a huge privilege of TPU support out of the box, but unfortunately, TPU comes with the same 16 Gb of memory as all other kernels, which makes it impossible to unleash all its power.
\nBut I guess the vital issue about TF + TPU in NLP competitions is that some transformers models aren't compatible with TPU (e.g. LongFormer), so it's worthless to go for TF.
", "rawMarkdown": "You're right. \n\nI haven't used Torch a lot, but at first glance it seems to be more user friendly and flexible despite it requires way more code for the same task and thus seems harder to quickstart compared to Keras. However, if that's critical, Torch Lightning totally eliminates this.\n\nTorch also has more straightforward and intuitive API. I've seen somebody complaining here that there's exactly 5 ways to initialise convolutional layer in TF and Keras. Sounds like a joke, but it's true.\n\nTF gives a huge privilege of TPU support out of the box, but unfortunately, TPU comes with the same 16 Gb of memory as all other kernels, which makes it impossible to unleash all its power.\n\nBut I guess the vital issue about TF + TPU in NLP competitions is that some transformers models aren't compatible with TPU (e.g. LongFormer), so it's worthless to go for TF.", "votes": null }, { "id": "1819893", "postDate": "06/14/2022 08:06:59", "content": "I always feel enthusiastic about making use of all the free resources provided by Kaggle :)
\n\n\nincreasing the batch size (8192 looks good to me) appears to bring a notable speed improvement
\n
That's a good point, thanks! I'll give it a try too
", "rawMarkdown": "I always feel enthusiastic about making use of all the free resources provided by Kaggle :)\n\n> increasing the batch size (8192 looks good to me) appears to bring a notable speed improvement \n\nThat's a good point, thanks! I'll give it a try too", "votes": null }, { "id": "1820069", "postDate": "06/14/2022 10:43:48", "content": "This is truly wonderful.
\nThank you for making it a public notebook.
I think Jax is better choice than TF now.
", "rawMarkdown": "I think Jax is better choice than TF now.", "votes": null }, { "id": "1822011", "postDate": "06/16/2022 02:27:53", "content": "Are you using jax?
", "rawMarkdown": "Are you using jax?", "votes": null }, { "id": "1822047", "postDate": "06/16/2022 03:24:31", "content": "Thanks for the clue, I'll check it out!
", "rawMarkdown": "Thanks for the clue, I'll check it out!", "votes": null } ], "comments": [ { "id": 1817968, "author_name": "happycube", "author_url": "", "post_date": "06/12/2022 04:46:24", "content": "Thanks very much for this! Anyone on a budget (even a Colab Pro+ budget ;) ) should look at this, seriously.
\nI'm looking at this on colab, and increasing the batch size (8192 looks good to me) appears to bring a notable speed improvement - considering that data has to be hauled over to the TPU in chunks, that makes sense.
", "votes": null, "replies": [ { "id": 1819893, "author_name": "nickuzmenkov", "author_url": "", "post_date": "06/14/2022 08:06:59", "content": "I always feel enthusiastic about making use of all the free resources provided by Kaggle :)
\n\n\nincreasing the batch size (8192 looks good to me) appears to bring a notable speed improvement
\n
That's a good point, thanks! I'll give it a try too
", "votes": null, "replies": [] } ] }, { "id": 1818107, "author_name": "tornikeonoprishvili", "author_url": "", "post_date": "06/12/2022 09:25:45", "content": "\n\nI didn't find a single TensorFlow public notebook for this competition
\n
Same here. Nobody seems to use TF these days. There seems to be a fundamental problem with using TF in competitions. :/
", "votes": null, "replies": [ { "id": 1819880, "author_name": "nickuzmenkov", "author_url": "", "post_date": "06/14/2022 08:00:33", "content": "You're right.
\nI haven't used Torch a lot, but at first glance it seems to be more user friendly and flexible despite it requires way more code for the same task and thus seems harder to quickstart compared to Keras. However, if that's critical, Torch Lightning totally eliminates this.
\nTorch also has more straightforward and intuitive API. I've seen somebody complaining here that there's exactly 5 ways to initialise convolutional layer in TF and Keras. Sounds like a joke, but it's true.
\nTF gives a huge privilege of TPU support out of the box, but unfortunately, TPU comes with the same 16 Gb of memory as all other kernels, which makes it impossible to unleash all its power.
\nBut I guess the vital issue about TF + TPU in NLP competitions is that some transformers models aren't compatible with TPU (e.g. LongFormer), so it's worthless to go for TF.
", "votes": null, "replies": [] }, { "id": 1821505, "author_name": "snaker", "author_url": "", "post_date": "06/15/2022 15:45:02", "content": "I think Jax is better choice than TF now.
", "votes": null, "replies": [] }, { "id": 1822011, "author_name": "yuanzhezhou", "author_url": "", "post_date": "06/16/2022 02:27:53", "content": "Are you using jax?
", "votes": null, "replies": [] }, { "id": 1822047, "author_name": "nickuzmenkov", "author_url": "", "post_date": "06/16/2022 03:24:31", "content": "Thanks for the clue, I'll check it out!
", "votes": null, "replies": [] } ] }, { "id": 1820069, "author_name": "thedevastator", "author_url": "", "post_date": "06/14/2022 10:43:48", "content": "This is truly wonderful.
\nThank you for making it a public notebook.