![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Fed9fe1a5c1ec2231e094c739a561cd8b%2FScreen%20Shot%202019-07-23%20at%202.03.21%20PM.png?generation=1563887189468734&alt=media\")
Hi,\nI have read many kernels for this competition, and I have the feeling that most of the people are using PyTorch... So my question is: why are you using PyTorch instead of TensorFlow?\nKind regards,\nKaharon
", "messages": [ { "id": 573256, "postDate": "2019-07-12T03:23:00.967Z", "content": "Hi,\nI have read many kernels for this competition, and I have the feeling that most of the people are using PyTorch... So my question is: why are you using PyTorch instead of TensorFlow?\nKind regards,\nKaharon
", "rawMarkdown": "Hi,\nI have read many kernels for this competition, and I have the feeling that most of the people are using PyTorch... So my question is: why are you using PyTorch instead of TensorFlow?\nKind regards,\nKaharon", "votes": 12 }, { "id": 575317, "postDate": "2019-07-15T09:16:12.223Z", "content": "I use TF for a quite long time since it 1st appeared. recently, I noticed that pytorch are dominant at research papers. I begin to try pytorch since than. I found out that several things that make people chose pytorch over TF:\n1. you dont have to control graph and session in pytorch. It gets a little bit messy when u have 2/more graphs in TF\n2. you have (to the best of my knowledge) to provide the string of the node/tensor name to access them, which in pytorch, you can access them using object manner.\n3. running on multiple GPUs is far easier to be configured in pytorch than TF\nthese are 3 huge diffs in my opinion. I wish to hear opinions from pro kagglers.
", "rawMarkdown": "I use TF for a quite long time since it 1st appeared. recently, I noticed that pytorch are dominant at research papers. I begin to try pytorch since than. I found out that several things that make people chose pytorch over TF:\n1. you dont have to control graph and session in pytorch. It gets a little bit messy when u have 2/more graphs in TF\n2. you have (to the best of my knowledge) to provide the string of the node/tensor name to access them, which in pytorch, you can access them using object manner.\n3. running on multiple GPUs is far easier to be configured in pytorch than TF\nthese are 3 huge diffs in my opinion. I wish to hear opinions from pro kagglers.", "votes": 9, "replies": [ { "id": 582321, "postDate": "2019-07-23T03:57:10.380Z", "content": "Your second point get my particular attention.
", "rawMarkdown": "Your second point get my particular attention." }, { "id": 643022, "postDate": "2019-10-07T00:57:09.667Z", "content": "For point 3 Distributed Training and in general debate over static and dynamic graphs is obsolete by tensorflow 2.0 update. Still, I think PyTorch has got the competition to TensorFlow amped.
", "rawMarkdown": "For point 3 [Distributed Training](https://www.tensorflow.org/guide/distributed_training) and in [general debate over static and dynamic graphs](https://www.tensorflow.org/guide/eager) is obsolete by tensorflow 2.0 update. Still, I think PyTorch has got the competition to TensorFlow amped. " } ] }, { "id": 577218, "postDate": "2019-07-16T13:45:44.047Z", "content": "Currently in Kaggle there are 4 most used DL framework. I will rank them based on its complexity (in my opinion), the higher the more complex.
\n\nThe general rule of thumb is more complex = more customization power. Another rule of thumb is if somebody already implemented the functionality on the official repo it is most likely optimized and you could assume bug-free. This is why fastai and keras seems really popular when the competition has a straight forward objective (loss minimization, simple metric maximization e.g. AUC, accuracy). It really saves you a lot of time by using a more bug-free by using implemented training routine (implemented in keras and fastai).
\n\nHowever, it's a bit different with this competition, training gan is a game between discriminator and generator and the performance metric optimization has little to no correlation with our loss minimization on the discriminator and generator optimization. Now, let's say we are reduced to between keras and torch. When you train gan with keras since you have to switch between generator and discriminator you have to use .train_on_batch() and you cannot use .fit(). When you do this basically this is similar to working with torch (basically you are implementing your own training routine), but you already sacrificed a lot of customization power offered by torch.
\n\nLastly, here are also several reasons why torch seems a nice framework for this competition.\n1. Torch is pythonic\n2. Easy to understand the flow of the computation graph\n3. Many examples can be found on the internet.\n4. Encourages you to do well-organized coding practices (most of the codes written in torch are based on modules)hence a model is basically a collection of modules. You can do this also with keras, but for me it is more satisfying when you look at a torch code rather than keras, since when you summarized it (model.summary()) they dont group unlike torch.
\n\nThis is purely my opinion alone and might not represent the whole kaggle community.
", "rawMarkdown": "Currently in Kaggle there are 4 most used DL framework. I will rank them based on its complexity (in my opinion), the higher the more complex.\n\n1. Tensorflow\n2. Pytorch\n3. Keras\n4. Fastai\n\nThe general rule of thumb is more complex = more customization power. Another rule of thumb is if somebody already implemented the functionality on the official repo it is most likely optimized and you could assume bug-free. This is why fastai and keras seems really popular when the competition has a straight forward objective (loss minimization, simple metric maximization e.g. AUC, accuracy). It really saves you a lot of time by using a more bug-free by using implemented training routine (implemented in keras and fastai). \n\nHowever, it's a bit different with this competition, training gan is a game between discriminator and generator and the performance metric optimization has little to no correlation with our loss minimization on the discriminator and generator optimization. Now, let's say we are reduced to between keras and torch. When you train gan with keras since you have to switch between generator and discriminator you have to use .train_on_batch() and you cannot use .fit(). When you do this basically this is similar to working with torch (basically you are implementing your own training routine), but you already sacrificed a lot of customization power offered by torch.\n\nLastly, here are also several reasons why torch seems a nice framework for this competition.\n1. Torch is pythonic\n2. Easy to understand the flow of the computation graph\n3. Many examples can be found on the internet.\n4. Encourages you to do well-organized coding practices (most of the codes written in torch are based on modules)hence a model is basically a collection of modules. You can do this also with keras, but for me it is more satisfying when you look at a torch code rather than keras, since when you summarized it (model.summary()) they dont group unlike torch.\n\nThis is purely my opinion alone and might not represent the whole kaggle community.\n", "votes": 7, "replies": [ { "id": 582317, "postDate": "2019-07-23T03:49:11.463Z", "content": "Thank you, the well-organized code is something I am looking for! :smile:
", "rawMarkdown": "Thank you, the well-organized code is something I am looking for! :smile:" } ] }, { "id": 573670, "postDate": "2019-07-12T15:30:42.353Z", "content": "The fact that most kernels are in PyTorch doesn't mean that most people use it ;-) As Chris stated earlier Tensorflow requires some advanced stuff but there is enough example material to be found on the internet.
\n\nI presume that for every major ML framework there are multiple examples waiting on the internet to be found. Just use that whatever you are most comfortable with. I personally use Keras + Tensorflow.
", "rawMarkdown": "The fact that most kernels are in PyTorch doesn't mean that most people use it ;-) As Chris stated earlier Tensorflow requires some advanced stuff but there is enough example material to be found on the internet.\n\nI presume that for every major ML framework there are multiple examples waiting on the internet to be found. Just use that whatever you are most comfortable with. I personally use Keras + Tensorflow.\n", "votes": 5 }, { "id": 573512, "postDate": "2019-07-12T11:02:30.073Z", "content": "Using TensorFlow requires using the advanced TF feature GradientTape because GANs have complex loss functions compared to basic CNN. And if you want to use Keras alongside TF, you need to use TF eager_execution, another advanced feature. TensorFlow provides a great tutorial here. And there is a wonderful Kaggle kernel that uses TF here.
\n\nI've published 3 kernels using TF, but they don't show you how to build a basic traditional GAN.
", "rawMarkdown": "Using TensorFlow requires using the advanced TF feature [GradientTape][1] because GANs have complex loss functions compared to basic CNN. And if you want to use Keras alongside TF, you need to use TF [eager_execution][2], another advanced feature. TensorFlow provides a great tutorial [here][3]. And there is a wonderful Kaggle kernel that uses TF [here][4].\n\nI've published 3 kernels using TF, but they don't show you how to build a basic traditional GAN.\n\n[1]: https://www.tensorflow.org/api_docs/python/tf/GradientTape\n[2]: https://www.tensorflow.org/guide/eager\n[3]: https://www.tensorflow.org/beta/tutorials/generative/dcgan\n[4]: https://www.kaggle.com/cmalla94/dcgan-generating-dog-images-with-tensorflow", "votes": 5, "replies": [ { "id": 573541, "postDate": "2019-07-12T11:52:03.180Z", "content": "You can also build GANs in Keras/TensorFlow without using GradientTape and EagerExecution. You build two separate Keras models, a Generator and Discriminator. You compile the Discriminator and compile a combined GAN. You train the Discriminator like normal. To train the Generator, you train the GAN (combined model) with the Discriminator's weights set to untrainable. Here and here are two tutorials online. I haven't compared and contrasted the advantages of using GradientTape and not using GradientTape.
", "rawMarkdown": "You can also build GANs in Keras/TensorFlow without using GradientTape and EagerExecution. You build two separate Keras models, a Generator and Discriminator. You compile the Discriminator and compile a combined GAN. You train the Discriminator like normal. To train the Generator, you train the GAN (combined model) with the Discriminator's weights set to untrainable. [Here][1] and [here][2] are two tutorials online. I haven't compared and contrasted the advantages of using GradientTape and not using GradientTape.\n\n[1]: https://medium.com/datadriveninvestor/generative-adversarial-network-gan-using-keras-ce1c05cfdfd3\n[2]: https://machinelearningmastery.com/how-to-develop-a-conditional-generative-adversarial-network-from-scratch/", "votes": 7 }, { "id": 573969, "postDate": "2019-07-13T03:43:13.517Z", "content": "Thanks you so much for all these responses! I will read it. :)
", "rawMarkdown": "Thanks you so much for all these responses! I will read it. :)" }, { "id": 574475, "postDate": "2019-07-13T23:03:26.963Z", "content": "Note since Kaggle uses TF 1.14.1, you need to add tf.contrib.eager.defun(train_step) to speed up your eager_execution code. Or @tf.function.
EDIT: this speeds up the code but prevents convergence. Not sure why :-(
\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)
", "rawMarkdown": "Note since Kaggle uses TF 1.14.1, you need to add `tf.contrib.eager.defun(train_step)` to speed up your eager_execution code. Or `@tf.function`.\n\nEDIT: this speeds up the code but prevents convergence. Not sure why :-(\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)", "votes": 1 } ] }, { "id": 573279, "postDate": "2019-07-12T04:32:20.167Z", "content": "I think one of the reasons is the starter kernel uses Pytorch https://www.kaggle.com/wendykan/gan-dogs-starter
", "rawMarkdown": "I think one of the reasons is the starter kernel uses Pytorch https://www.kaggle.com/wendykan/gan-dogs-starter", "votes": 5 }, { "id": 575497, "postDate": "2019-07-15T13:52:17.220Z", "content": "I really like this word so I'm gonna use it again: pytorch is more pythonic :)
\n\nso it's very easy to write and code compared to TF or Keras
\n\nplease check my post here @Kaharon
\n\nhttps://www.kaggle.com/getting-started/96953#latest-562249
", "rawMarkdown": "I really like this word so I'm gonna use it again: pytorch is more pythonic :) \n\nso it's very easy to write and code compared to TF or Keras\n\nplease check my post here @Kaharon\n\nhttps://www.kaggle.com/getting-started/96953#latest-562249", "votes": 3 }, { "id": 574474, "postDate": "2019-07-13T23:01:39.230Z", "content": "Here's a huge TF tip:
\n\nIf you're using TensorFlow in eager_execution mode then add train_step = tf.contrib.eager.defun(train_step) after your function that trains with GradientTape. That will make your code go twice as fast ! Or @tf.function just before.
EDIT: this speeds up the code but prevents convergence. Not sure why :-(
\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)
", "rawMarkdown": "Here's a huge TF tip:\n\nIf you're using TensorFlow in eager_execution mode then add `train_step = tf.contrib.eager.defun(train_step)` after your function that trains with GradientTape. That will make your code go twice as fast ! Or `@tf.function` just before.\n\nEDIT: this speeds up the code but prevents convergence. Not sure why :-(\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)", "votes": 3 }, { "id": 573264, "postDate": "2019-07-12T03:33:08.707Z", "content": "I find it easier to code as a beginner . But there are a lot of tensorflow Kernels as well and they are quite efficient . Have a look at the most upvoted Kernels .
", "rawMarkdown": "I find it easier to code as a beginner . But there are a lot of tensorflow Kernels as well and they are quite efficient . Have a look at the most upvoted Kernels .", "votes": 3 }, { "id": 582689, "postDate": "2019-07-23T13:07:21.443Z", "content": "@kaharon A lot of people have already expressed their opinions, I would like to direct you to Lecture 8 cs231n Stanford University. There is an entire lecture dedicated to Deep learning frameworks. http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture8.pdf
\n\n
Thanks for the link. The full PDF is an excellent tutorial on computational graphs. And it compares TensorFlow to PyTorch well.
\n\nIt also explains why frameworks like TensorFlow and PyTorch are needed. Neural networks are just sequences of matrix math. It's easy to create NNs from scratch to perform forward propagation. The difficult part is back prop. To train a neural network we need the gradients (derivatives) of variables with respect to other variables. Computational graphs is a mathematical framework that automatically finds gradients (derivatives) for us. It's very cool.
", "rawMarkdown": "Thanks for the link. The full PDF is an excellent tutorial on computational graphs. And it compares TensorFlow to PyTorch well.\n\nIt also explains why frameworks like TensorFlow and PyTorch are needed. Neural networks are just sequences of matrix math. It's easy to create NNs from scratch to perform forward propagation. The difficult part is back prop. To train a neural network we need the gradients (derivatives) of variables with respect to other variables. Computational graphs is a mathematical framework that automatically finds gradients (derivatives) for us. It's very cool.", "votes": 4 }, { "id": 582727, "postDate": "2019-07-23T13:57:56.570Z", "content": "Thanks a ton..
", "rawMarkdown": "Thanks a ton.." } ] }, { "id": 574339, "postDate": "2019-07-13T16:40:51.383Z", "content": "I provide some TensorFlow/Keras starter code here. It's a RaLS-CGAN, but you can remove the categorical variable breed input and turn it into a RaLSGAN easily if you wish.
", "rawMarkdown": "I provide some TensorFlow/Keras starter code [here][1]. It's a RaLS-CGAN, but you can remove the categorical variable breed input and turn it into a RaLSGAN easily if you wish.\n\n[1]: https://www.kaggle.com/cdeotte/dog-breed-cgan", "votes": 2, "replies": [ { "id": 574344, "postDate": "2019-07-13T16:47:10.173Z", "content": "If you remove the category inputs and change the loss functions to the following, you'll have a basic DCGAN in Keras/TensorFlow
\n\nbce = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n\n# Basic Discriminator Loss from logits\ndef errD(fake,real):\n real_loss = bce(tf.ones_like(real), real)\n fake_loss = bce(tf.zeros_like(fake), fake)\n return real_loss + fake_loss\ndef errG(fake,real):\n return bce(tf.ones_like(fake), fake)\nWow! Thank you so much for your code!
", "rawMarkdown": "Wow! Thank you so much for your code!" } ] }, { "id": 574704, "postDate": "2019-07-14T10:57:32.470Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 574723, "postDate": "2019-07-14T11:42:31.097Z", "content": "TensorFlow is making changes to address some of those concerns. Starting in version 2.0, eager_execution is on by default. This makes it behave like Python and NumPy and team up with them easier. To use Sessions and Graphs, you'll need to explicitly turn them on.
\n\n\nIt teams up better with Python, and the resulting code is more \"clean\".
\n
This seems to be one of the main reason why people prefer PyTorch rather than TF according to the comments of this topic.\nThank you for your response.
", "rawMarkdown": "> It teams up better with Python, and the resulting code is more \"clean\".\n\nThis seems to be one of the main reason why people prefer PyTorch rather than TF according to the comments of this topic.\nThank you for your response." }, { "id": 643027, "postDate": "2019-10-07T01:06:16.733Z", "content": "As @cdeotte has already mentioned TensorFlow has been making changes regarding some concerns.
\n\nAs of today, TF2.0 is released and I have tried to list down and review some important changes in TF2.0 Kernel here
\n\nI do think TF vs PyTorch adaption/competition will be largely beneficial for deep learning coders which is good to have.
", "rawMarkdown": "As @cdeotte has already mentioned TensorFlow has been making changes regarding some concerns. \n\nAs of today, TF2.0 is released and I have tried to list down and review some important changes in [TF2.0 Kernel here](https://www.kaggle.com/vivekwisdom/tensorflow-2-0-touchdown)\n\nI do think TF vs PyTorch adaption/competition will be largely beneficial for deep learning coders which is good to have.", "votes": 2 } ] } ], "comments": [ { "id": 575317, "author_name": "TryanAditya", "author_url": "", "post_date": "2019-07-15T09:16:12.223000", "content": "I use TF for a quite long time since it 1st appeared. recently, I noticed that pytorch are dominant at research papers. I begin to try pytorch since than. I found out that several things that make people chose pytorch over TF:\n1. you dont have to control graph and session in pytorch. It gets a little bit messy when u have 2/more graphs in TF\n2. you have (to the best of my knowledge) to provide the string of the node/tensor name to access them, which in pytorch, you can access them using object manner.\n3. running on multiple GPUs is far easier to be configured in pytorch than TF\nthese are 3 huge diffs in my opinion. I wish to hear opinions from pro kagglers.
", "votes": 9, "replies": [ { "id": 582321, "author_name": "Kaharon", "author_url": "", "post_date": "2019-07-23T03:57:10.380000", "content": "Your second point get my particular attention.
", "votes": 0, "replies": [] }, { "id": 643022, "author_name": "sapientiae", "author_url": "", "post_date": "2019-10-07T00:57:09.667000", "content": "For point 3 Distributed Training and in general debate over static and dynamic graphs is obsolete by tensorflow 2.0 update. Still, I think PyTorch has got the competition to TensorFlow amped.
", "votes": 0, "replies": [] } ] }, { "id": 577218, "author_name": "Yohanes Alfredo", "author_url": "", "post_date": "2019-07-16T13:45:44.047000", "content": "Currently in Kaggle there are 4 most used DL framework. I will rank them based on its complexity (in my opinion), the higher the more complex.
\n\nThe general rule of thumb is more complex = more customization power. Another rule of thumb is if somebody already implemented the functionality on the official repo it is most likely optimized and you could assume bug-free. This is why fastai and keras seems really popular when the competition has a straight forward objective (loss minimization, simple metric maximization e.g. AUC, accuracy). It really saves you a lot of time by using a more bug-free by using implemented training routine (implemented in keras and fastai).
\n\nHowever, it's a bit different with this competition, training gan is a game between discriminator and generator and the performance metric optimization has little to no correlation with our loss minimization on the discriminator and generator optimization. Now, let's say we are reduced to between keras and torch. When you train gan with keras since you have to switch between generator and discriminator you have to use .train_on_batch() and you cannot use .fit(). When you do this basically this is similar to working with torch (basically you are implementing your own training routine), but you already sacrificed a lot of customization power offered by torch.
\n\nLastly, here are also several reasons why torch seems a nice framework for this competition.\n1. Torch is pythonic\n2. Easy to understand the flow of the computation graph\n3. Many examples can be found on the internet.\n4. Encourages you to do well-organized coding practices (most of the codes written in torch are based on modules)hence a model is basically a collection of modules. You can do this also with keras, but for me it is more satisfying when you look at a torch code rather than keras, since when you summarized it (model.summary()) they dont group unlike torch.
\n\nThis is purely my opinion alone and might not represent the whole kaggle community.
", "votes": 7, "replies": [ { "id": 582317, "author_name": "Kaharon", "author_url": "", "post_date": "2019-07-23T03:49:11.463000", "content": "Thank you, the well-organized code is something I am looking for! :smile:
", "votes": 0, "replies": [] } ] }, { "id": 573670, "author_name": "Robin Smits", "author_url": "", "post_date": "2019-07-12T15:30:42.353000", "content": "The fact that most kernels are in PyTorch doesn't mean that most people use it ;-) As Chris stated earlier Tensorflow requires some advanced stuff but there is enough example material to be found on the internet.
\n\nI presume that for every major ML framework there are multiple examples waiting on the internet to be found. Just use that whatever you are most comfortable with. I personally use Keras + Tensorflow.
", "votes": 5, "replies": [] }, { "id": 573512, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-12T11:02:30.073000", "content": "Using TensorFlow requires using the advanced TF feature GradientTape because GANs have complex loss functions compared to basic CNN. And if you want to use Keras alongside TF, you need to use TF eager_execution, another advanced feature. TensorFlow provides a great tutorial here. And there is a wonderful Kaggle kernel that uses TF here.
\n\nI've published 3 kernels using TF, but they don't show you how to build a basic traditional GAN.
", "votes": 5, "replies": [ { "id": 573541, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-12T11:52:03.180000", "content": "You can also build GANs in Keras/TensorFlow without using GradientTape and EagerExecution. You build two separate Keras models, a Generator and Discriminator. You compile the Discriminator and compile a combined GAN. You train the Discriminator like normal. To train the Generator, you train the GAN (combined model) with the Discriminator's weights set to untrainable. Here and here are two tutorials online. I haven't compared and contrasted the advantages of using GradientTape and not using GradientTape.
", "votes": 7, "replies": [] }, { "id": 573969, "author_name": "Kaharon", "author_url": "", "post_date": "2019-07-13T03:43:13.517000", "content": "Thanks you so much for all these responses! I will read it. :)
", "votes": 0, "replies": [] }, { "id": 574475, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-13T23:03:26.963000", "content": "Note since Kaggle uses TF 1.14.1, you need to add tf.contrib.eager.defun(train_step) to speed up your eager_execution code. Or @tf.function.
EDIT: this speeds up the code but prevents convergence. Not sure why :-(
\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)
", "votes": 1, "replies": [] } ] }, { "id": 573279, "author_name": "Tom", "author_url": "", "post_date": "2019-07-12T04:32:20.167000", "content": "I think one of the reasons is the starter kernel uses Pytorch https://www.kaggle.com/wendykan/gan-dogs-starter
", "votes": 5, "replies": [] }, { "id": 575497, "author_name": "Firat Gonen", "author_url": "", "post_date": "2019-07-15T13:52:17.220000", "content": "I really like this word so I'm gonna use it again: pytorch is more pythonic :)
\n\nso it's very easy to write and code compared to TF or Keras
\n\nplease check my post here @Kaharon
\n\nhttps://www.kaggle.com/getting-started/96953#latest-562249
", "votes": 3, "replies": [] }, { "id": 574474, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-13T23:01:39.230000", "content": "Here's a huge TF tip:
\n\nIf you're using TensorFlow in eager_execution mode then add train_step = tf.contrib.eager.defun(train_step) after your function that trains with GradientTape. That will make your code go twice as fast ! Or @tf.function just before.
EDIT: this speeds up the code but prevents convergence. Not sure why :-(
\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)
", "votes": 3, "replies": [] }, { "id": 573264, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2019-07-12T03:33:08.707000", "content": "I find it easier to code as a beginner . But there are a lot of tensorflow Kernels as well and they are quite efficient . Have a look at the most upvoted Kernels .
", "votes": 3, "replies": [] }, { "id": 582689, "author_name": "cyberia", "author_url": "", "post_date": "2019-07-23T13:07:21.443000", "content": "@kaharon A lot of people have already expressed their opinions, I would like to direct you to Lecture 8 cs231n Stanford University. There is an entire lecture dedicated to Deep learning frameworks. http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture8.pdf
\n\n
Thanks for the link. The full PDF is an excellent tutorial on computational graphs. And it compares TensorFlow to PyTorch well.
\n\nIt also explains why frameworks like TensorFlow and PyTorch are needed. Neural networks are just sequences of matrix math. It's easy to create NNs from scratch to perform forward propagation. The difficult part is back prop. To train a neural network we need the gradients (derivatives) of variables with respect to other variables. Computational graphs is a mathematical framework that automatically finds gradients (derivatives) for us. It's very cool.
", "votes": 4, "replies": [] }, { "id": 582727, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2019-07-23T13:57:56.570000", "content": "Thanks a ton..
", "votes": 0, "replies": [] } ] }, { "id": 574339, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-13T16:40:51.383000", "content": "I provide some TensorFlow/Keras starter code here. It's a RaLS-CGAN, but you can remove the categorical variable breed input and turn it into a RaLSGAN easily if you wish.
", "votes": 2, "replies": [ { "id": 574344, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-13T16:47:10.173000", "content": "If you remove the category inputs and change the loss functions to the following, you'll have a basic DCGAN in Keras/TensorFlow
\n\nbce = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n\n# Basic Discriminator Loss from logits\ndef errD(fake,real):\n real_loss = bce(tf.ones_like(real), real)\n fake_loss = bce(tf.zeros_like(fake), fake)\n return real_loss + fake_loss\ndef errG(fake,real):\n return bce(tf.ones_like(fake), fake)\nWow! Thank you so much for your code!
", "votes": 0, "replies": [] } ] }, { "id": 574704, "author_name": "", "author_url": "", "post_date": "2019-07-14T10:57:32.470000", "content": "", "votes": 2, "replies": [ { "id": 574723, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-07-14T11:42:31.097000", "content": "TensorFlow is making changes to address some of those concerns. Starting in version 2.0, eager_execution is on by default. This makes it behave like Python and NumPy and team up with them easier. To use Sessions and Graphs, you'll need to explicitly turn them on.
\n\n\nIt teams up better with Python, and the resulting code is more \"clean\".
\n
This seems to be one of the main reason why people prefer PyTorch rather than TF according to the comments of this topic.\nThank you for your response.
", "votes": 0, "replies": [] }, { "id": 643027, "author_name": "sapientiae", "author_url": "", "post_date": "2019-10-07T01:06:16.733000", "content": "As @cdeotte has already mentioned TensorFlow has been making changes regarding some concerns.
\n\nAs of today, TF2.0 is released and I have tried to list down and review some important changes in TF2.0 Kernel here
\n\nI do think TF vs PyTorch adaption/competition will be largely beneficial for deep learning coders which is good to have.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "573256": "Hi,\nI have read many kernels for this competition, and I have the feeling that most of the people are using PyTorch... So my question is: why are you using PyTorch instead of TensorFlow?\nKind regards,\nKaharon", "575317": "I use TF for a quite long time since it 1st appeared. recently, I noticed that pytorch are dominant at research papers. I begin to try pytorch since than. I found out that several things that make people chose pytorch over TF:\n1. you dont have to control graph and session in pytorch. It gets a little bit messy when u have 2/more graphs in TF\n2. you have (to the best of my knowledge) to provide the string of the node/tensor name to access them, which in pytorch, you can access them using object manner.\n3. running on multiple GPUs is far easier to be configured in pytorch than TF\nthese are 3 huge diffs in my opinion. I wish to hear opinions from pro kagglers.", "577218": "Currently in Kaggle there are 4 most used DL framework. I will rank them based on its complexity (in my opinion), the higher the more complex.\n\n1. Tensorflow\n2. Pytorch\n3. Keras\n4. Fastai\n\nThe general rule of thumb is more complex = more customization power. Another rule of thumb is if somebody already implemented the functionality on the official repo it is most likely optimized and you could assume bug-free. This is why fastai and keras seems really popular when the competition has a straight forward objective (loss minimization, simple metric maximization e.g. AUC, accuracy). It really saves you a lot of time by using a more bug-free by using implemented training routine (implemented in keras and fastai). \n\nHowever, it's a bit different with this competition, training gan is a game between discriminator and generator and the performance metric optimization has little to no correlation with our loss minimization on the discriminator and generator optimization. Now, let's say we are reduced to between keras and torch. When you train gan with keras since you have to switch between generator and discriminator you have to use .train_on_batch() and you cannot use .fit(). When you do this basically this is similar to working with torch (basically you are implementing your own training routine), but you already sacrificed a lot of customization power offered by torch.\n\nLastly, here are also several reasons why torch seems a nice framework for this competition.\n1. Torch is pythonic\n2. Easy to understand the flow of the computation graph\n3. Many examples can be found on the internet.\n4. Encourages you to do well-organized coding practices (most of the codes written in torch are based on modules)hence a model is basically a collection of modules. You can do this also with keras, but for me it is more satisfying when you look at a torch code rather than keras, since when you summarized it (model.summary()) they dont group unlike torch.\n\nThis is purely my opinion alone and might not represent the whole kaggle community.\n", "573670": "The fact that most kernels are in PyTorch doesn't mean that most people use it ;-) As Chris stated earlier Tensorflow requires some advanced stuff but there is enough example material to be found on the internet.\n\nI presume that for every major ML framework there are multiple examples waiting on the internet to be found. Just use that whatever you are most comfortable with. I personally use Keras + Tensorflow.\n", "573512": "Using TensorFlow requires using the advanced TF feature [GradientTape][1] because GANs have complex loss functions compared to basic CNN. And if you want to use Keras alongside TF, you need to use TF [eager_execution][2], another advanced feature. TensorFlow provides a great tutorial [here][3]. And there is a wonderful Kaggle kernel that uses TF [here][4].\n\nI've published 3 kernels using TF, but they don't show you how to build a basic traditional GAN.\n\n[1]: https://www.tensorflow.org/api_docs/python/tf/GradientTape\n[2]: https://www.tensorflow.org/guide/eager\n[3]: https://www.tensorflow.org/beta/tutorials/generative/dcgan\n[4]: https://www.kaggle.com/cmalla94/dcgan-generating-dog-images-with-tensorflow", "573279": "I think one of the reasons is the starter kernel uses Pytorch https://www.kaggle.com/wendykan/gan-dogs-starter", "575497": "I really like this word so I'm gonna use it again: pytorch is more pythonic :) \n\nso it's very easy to write and code compared to TF or Keras\n\nplease check my post here @Kaharon\n\nhttps://www.kaggle.com/getting-started/96953#latest-562249", "574474": "Here's a huge TF tip:\n\nIf you're using TensorFlow in eager_execution mode then add `train_step = tf.contrib.eager.defun(train_step)` after your function that trains with GradientTape. That will make your code go twice as fast ! Or `@tf.function` just before.\n\nEDIT: this speeds up the code but prevents convergence. Not sure why :-(\n\nUPDATE: I got this to work by being careful to use TensorFlow variables and to not access global variables inside my @tf.function :-)", "573264": "I find it easier to code as a beginner . But there are a lot of tensorflow Kernels as well and they are quite efficient . Have a look at the most upvoted Kernels .", "582689": "@kaharon A lot of people have already expressed their opinions, I would like to direct you to Lecture 8 cs231n Stanford University. There is an entire lecture dedicated to Deep learning frameworks. http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture8.pdf\n\n\n", "574339": "I provide some TensorFlow/Keras starter code [here][1]. It's a RaLS-CGAN, but you can remove the categorical variable breed input and turn it into a RaLSGAN easily if you wish.\n\n[1]: https://www.kaggle.com/cdeotte/dog-breed-cgan", "574704": "" } }