I've created a set of notebooks that show how anyone can achieve a quite decent loss in this competition by using the power of transfer learning and a few lines of code.
\n\nYou can follow 2 different approaches:
\n\nUse bottlenecks
\n\nUse generator.
\n\nYou can find the notebooks here and attached.
\n\nA standard anaconda distribution + Keras (Tensorflow backend) should be enough to get this running.
\n\nI hope that people getting started in competitions (like me) will find this helpfull.
", "messages": [ { "id": 157349, "postDate": "2017-01-20T12:09:50.983Z", "content": "I've created a set of notebooks that show how anyone can achieve a quite decent loss in this competition by using the power of transfer learning and a few lines of code.
\n\nYou can follow 2 different approaches:
\n\nUse bottlenecks
\n\nUse generator.
\n\nYou can find the notebooks here and attached.
\n\nA standard anaconda distribution + Keras (Tensorflow backend) should be enough to get this running.
\n\nI hope that people getting started in competitions (like me) will find this helpfull.
", "rawMarkdown": "I've created a set of notebooks that show how anyone can achieve a quite decent loss in this competition by using the power of [transfer learning][1] and a few lines of code.\r\n\r\nYou can follow 2 different approaches:\r\n\r\n\r\n----------\r\n\r\n\r\n- Notebooks 01->02->03->04\r\n\r\nUse bottlenecks\r\n\r\n- Notebooks 01->02A->04\r\n\r\nUse generator.\r\n\r\n\r\n----------\r\n\r\n\r\nYou can find the [notebooks here][3] and attached.\r\n\r\nA standard [anaconda distribution][4] + [Keras (Tensorflow backend)][5] should be enough to get this running.\r\n\r\nI hope that people getting started in competitions (like me) will find this helpfull.\r\n\r\n\r\n [1]: http://cs231n.github.io/transfer-learning/\r\n [3]: https://github.com/daavoo/kaggle_solutions/tree/master/dogs_vs_cats\r\n [4]: https://www.continuum.io/downloads\r\n [5]: https://keras.io/#installation", "votes": 22 }, { "id": 158338, "postDate": "2017-01-26T23:04:10.783Z", "content": "@Ronin If I find the time I'll upload a versión using fit_generator instead off computing the bottlenecks. This method is slower if you are going yo try diferents things but it requieres less memory.
", "rawMarkdown": "@Ronin If I find the time I'll upload a versión using fit_generator instead off computing the bottlenecks. This method is slower if you are going yo try diferents things but it requieres less memory.", "votes": 3 }, { "id": 158334, "postDate": "2017-01-26T22:44:44.243Z", "content": "sadly my 8mb machine barfed on this at the computing training bottlenecks. (even after pairing down to 500 images) Still thank you for posting. Very helpful.
\n\nedit--by shrinking the input size down in the 2nd file to 64x64 i was able to run without issue on the full set. So for those of you with lesser machines (I have 8gb memory and GTX 650 Ti) tryin smaller input sizes on the resnet input shape.
\n\nAlso a reminder to setup you files structure correctly--otherwise the '02' notebook will not pick up files correctly (I didn't have the 'test' subfolder in 'test'
\n\nworking-dir \n train\n dogs\n cats\n valid\n dogs\n cats\n test\n test\nThanks - I was missing the test/test subdirectory!
", "rawMarkdown": "Thanks - I was missing the test/test subdirectory!" } ] }, { "id": 220433, "postDate": "2017-09-12T04:59:37.607Z", "content": "0.06 ?? \nIs this the leaderboard score.\nI have fine tuned vgg16 to get validation loss of 0.0001.\nBut at the leaderboard i am getting 1.06...\nCan you suggest what might be wrong?
", "rawMarkdown": "0.06 ?? \nIs this the leaderboard score.\nI have fine tuned vgg16 to get validation loss of 0.0001.\nBut at the leaderboard i am getting 1.06...\nCan you suggest what might be wrong?", "replies": [ { "id": 235214, "postDate": "2017-10-24T22:55:53.237Z", "content": "Your validation loss is too low. You probably have training examples in your validation set. You should recreate your validation set and make sure that you don't have duplicate images.\nAlso don't forget to submit probabilities instead of labels. Clipping predictions in the interval from 0.01 to 0.99 (eps) might also help.
", "rawMarkdown": "Your validation loss is too low. You probably have training examples in your validation set. You should recreate your validation set and make sure that you don't have duplicate images.\nAlso don't forget to submit probabilities instead of labels. Clipping predictions in the interval from 0.01 to 0.99 (eps) might also help.", "votes": 1 } ] }, { "id": 180755, "postDate": "2017-05-07T01:13:35.183Z", "content": "Thank you for posting the guidance! Super helpful, your github link doesn't work anymore though!
", "rawMarkdown": "Thank you for posting the guidance! Super helpful, your github link doesn't work anymore though! " }, { "id": 177952, "postDate": "2017-04-26T08:50:32.163Z", "content": "@Daivd,
\n\nThanks for this awesome stuff. How come you don't rescale images with 1/255. transformation when creating Bottleneck features? Isn't resnet expecting a transformation like this that rescales the data?
", "rawMarkdown": "@Daivd,\n\nThanks for this awesome stuff. How come you don't rescale images with 1/255. transformation when creating Bottleneck features? Isn't resnet expecting a transformation like this that rescales the data? " }, { "id": 164800, "postDate": "2017-03-02T12:06:05.237Z", "content": "Great post. \nI tried similar approach but with Squeezenet (the keras SN: https://github.com/rcmalli/keras-squeezenet), it has fewer parameters thus runs faster.
\n\nI wonder why you included resnet50.py? \nKeras has ResNet50 implementation in applications already: https://keras.io/applications/\nIt can download the weights automatically like:
\n\nmodel = ResNet50(weights='imagenet')\nIs there any reason you used the included implementation?
\n\np.s. what's the reasoning behind bottleneck vs CNN codes?
", "rawMarkdown": "Great post. \nI tried similar approach but with Squeezenet (the keras SN: https://github.com/rcmalli/keras-squeezenet), it has fewer parameters thus runs faster.\n\nI wonder why you included resnet50.py? \nKeras has ResNet50 implementation in applications already: https://keras.io/applications/\nIt can download the weights automatically like:\n\n model = ResNet50(weights='imagenet')\n\nIs there any reason you used the included implementation?\n\np.s. what's the reasoning behind bottleneck vs CNN codes?", "replies": [ { "id": 165957, "postDate": "2017-03-07T19:38:55.560Z", "content": "The main reason was that I wanted to re-implement the architecture from scratch, only looking at the original code when I got stucked, in order to theck if I understand the architecture correctly.\nThe other reason was that I included the imagenet preprocessing function inside the model.
\n\nI don't fully understand your question about the bottleneck reasoning but I'll asume that you are asking about what are the differences between those approaches.
\n\nUsing bottlenecks you are renouncing to some things:
\n\nBut, in exchange, you can train many different \"heads\" (trying different parameters, etc) on top of those bottelenecks, quite fast.
", "rawMarkdown": "The main reason was that I wanted to re-implement the architecture from scratch, only looking at the original code when I got stucked, in order to theck if I understand the architecture correctly.\nThe other reason was that I included the imagenet preprocessing function inside the model.\n\nI don't fully understand your question about the bottleneck reasoning but I'll asume that you are asking about what are the differences between those approaches.\n\nUsing bottlenecks you are renouncing to some things:\n\n - You won't be retraining any convolutional layer. In datasets bigger and/or not so similar to imagenet features it's usually helpfull to do this.\n- You won't be able to use on-line data augmentation. This is also important to build more robust models.\n\nBut, in exchange, you can train many different \"heads\" (trying different parameters, etc) on top of those bottelenecks, quite fast." } ] }, { "id": 163275, "postDate": "2017-02-23T09:45:24.133Z", "content": "Thank you so much for sharing, this is exactly what I needed to learn at this point.
\n\nOne question, why in notebook 03 do you train at lr=1e-5 first before bumping back up to 1e-3? I tried starting at 1e-3 directly and got exploding gradient pretty much every time. It seems mostly people start training at a high lr and reduce it slowly afterwards. What's your reasoning? Thanks!
", "rawMarkdown": "Thank you so much for sharing, this is exactly what I needed to learn at this point.\n\nOne question, why in notebook 03 do you train at lr=1e-5 first before bumping back up to 1e-3? I tried starting at 1e-3 directly and got exploding gradient pretty much every time. It seems mostly people start training at a high lr and reduce it slowly afterwards. What's your reasoning? Thanks!", "replies": [ { "id": 165959, "postDate": "2017-03-07T19:40:55.400Z", "content": "I train at low learning rate on first epoch in order to \"warm up\" and avoid exploding gradients. Yes it's usual to follow that procedure when training neural nets but in this case we are training a very tiny layer on top on some powerfull features.
", "rawMarkdown": "I train at low learning rate on first epoch in order to \"warm up\" and avoid exploding gradients. Yes it's usual to follow that procedure when training neural nets but in this case we are training a very tiny layer on top on some powerfull features." } ] }, { "id": 160618, "postDate": "2017-02-08T19:37:35.010Z", "content": "Hi, can you share what version of CuDnn, CUDA and Tensorflow you use to run this code? Thank you~
", "rawMarkdown": "Hi, can you share what version of CuDnn, CUDA and Tensorflow you use to run this code? Thank you~", "replies": [ { "id": 160624, "postDate": "2017-02-08T20:48:24.750Z", "content": "I can confirm that the notebooks work with the lastest version of Tensorflow, Cuda and Cudnn.
\n\nTry to follow the Tensorflow instalation instructions.
", "rawMarkdown": "I can confirm that the notebooks work with the lastest version of Tensorflow, Cuda and Cudnn.\n\nTry to follow the Tensorflow instalation instructions." }, { "id": 160641, "postDate": "2017-02-08T22:06:07.830Z", "content": "For my machine (which ran the code)
\n\nThank you for sharing. I am wondering why you didn't import ResNet thru Keras (https://keras.io/applications/#inceptionv3) but downloaded the resnet50.py and the weights to run the code instead? I tried to follow the Inception V3 transfer learning, but there was error message when it comes to training, and the error had something to do with CuDNN. So Im curious if others run into the same issue.
", "rawMarkdown": "Thank you for sharing. I am wondering why you didn't import ResNet thru Keras (https://keras.io/applications/#inceptionv3) but downloaded the resnet50.py and the weights to run the code instead? I tried to follow the Inception V3 transfer learning, but there was error message when it comes to training, and the error had something to do with CuDNN. So Im curious if others run into the same issue.", "replies": [ { "id": 160622, "postDate": "2017-02-08T20:43:17.150Z", "content": "I rewrited the code of the resnet50 file to check my understanding of the code. My version also include the imagenet preprocessing step inside the model definition.
", "rawMarkdown": "I rewrited the code of the resnet50 file to check my understanding of the code. My version also include the imagenet preprocessing step inside the model definition. " } ] }, { "id": 160445, "postDate": "2017-02-07T18:30:19.790Z", "content": "Would've been nicer to have it all in one notebook.
", "rawMarkdown": "Would've been nicer to have it all in one notebook." }, { "id": 159889, "postDate": "2017-02-04T16:43:10.373Z", "content": "Got a bit of time to work through your example and it is 100% reproducible. Well done!
\n\nMy question now is, why is this so much more different than the results I would get with the inbuilt Keras Resnet50 model?
\n\nWith that I can only scrape a 0.80 loss Notebook here
", "rawMarkdown": "Got a bit of time to work through your example and it is 100% reproducible. Well done!\n\nMy question now is, why is this so much more different than the results I would get with the inbuilt Keras [Resnet50][1] model? \n\nWith that I can only scrape a 0.80 loss [Notebook here][2]\n\n\n [1]: https://keras.io/applications/#resnet50\n [2]: https://www.dropbox.com/s/h8gbliyt76bsurr/vgg_own.ipynb?dl=0", "replies": [ { "id": 159991, "postDate": "2017-02-05T11:11:04.800Z", "content": "I'm not sure about what is the problem and I didin't look into your notebook in detail but I noticed 2 things:
\n\n1 - You are appending the new Dense layer with 2 outputs on top of the Resnet fc1000 layer.
The usual thing to do (as I do in the example notebooks) is to replace the fc1000 layer instead of appending the new layer on top of it.
This is because the weights of the original resnet 'fc1000' layer are trained to predict the 1000 categories of the Imagenet Challenge.
\n\nAlthough many of this 1000 categories are types of dogs and cats it just work better if you replace the 1000 fc layer with a new layer that you train from scratch.
\n\n2 - You are clipping the scores too much.
\n\nAlthough when dealing with log loss is usually a good idea to \"clip\" your predictions to avoid been penalized by the overconfidence of the model, the clip is usually in the order of 0.95 / 0.05 or so.
\n\nI see at the end of your notebook that you are clipping the scores to 0.825548 / 0.175611 so even when you have guessed the right label your logloss is still quite high.
\n\nI think that if you to change those 2 things your logloss should decrease.
", "rawMarkdown": "I'm not sure about what is the problem and I didin't look into your notebook in detail but I noticed 2 things:\n\n----------\n\n1 - You are appending the new `Dense` layer with 2 outputs on top of the Resnet `fc1000` layer.\n\nThe usual thing to do (as I do in the example notebooks) is to **replace** the `fc1000` layer instead of appending the new layer on top of it. \n\nThis is because the weights of the original resnet 'fc1000' layer are trained to predict the 1000 categories of the Imagenet Challenge. \n\nAlthough many of this 1000 categories are types of dogs and cats it just work better if you replace the 1000 fc layer with a new layer that you train from scratch.\n\n----------\n\n2 - You are clipping the scores too much.\n\nAlthough when dealing with log loss is usually a good idea to \"clip\" your predictions to avoid been penalized by the overconfidence of the model, the clip is usually in the order of 0.95 / 0.05 or so. \n\nI see at the end of your notebook that you are clipping the scores to 0.825548 / 0.175611 so even when you have guessed the right label your logloss is still quite high.\n\n\nI think that if you to change those 2 things your logloss should decrease.\n", "votes": 1 } ] }, { "id": 159815, "postDate": "2017-02-04T04:32:37.357Z", "content": "Hi, has anyone submitted to test? I used the fit_generator method which validation error is about 0.04 but the test error is very big like 4.0, anyone knows why? The only difference here is I manually split the training and validation image.
", "rawMarkdown": "Hi, has anyone submitted to test? I used the fit_generator method which validation error is about 0.04 but the test error is very big like 4.0, anyone knows why? The only difference here is I manually split the training and validation image. ", "replies": [ { "id": 159854, "postDate": "2017-02-04T11:45:28.033Z", "content": "There is probably something wrong with your manual split.
\n\nThe fit_generator method reorder the images internally different than your operating system.
\n\nThat's why in the submission notebook I get the order of the filenames in the generator by calling test_batches.filenames wich returns a list with the proper order.
Not yet done any submissions; but very useful!
", "rawMarkdown": "Not yet done any submissions; but very useful!", "replies": [ { "id": 159720, "postDate": "2017-02-03T15:16:21.520Z", "content": "Thanks!. What are you waiting for??? :)
", "rawMarkdown": "Thanks!. What are you waiting for??? :)" } ] }, { "id": 159684, "postDate": "2017-02-03T10:54:50.157Z", "content": "Really nice job and give me many inspirations. The code is right and I can reproduce the results. \nThanks a lot for sharing.
", "rawMarkdown": "Really nice job and give me many inspirations. The code is right and I can reproduce the results. \nThanks a lot for sharing. ", "replies": [ { "id": 159719, "postDate": "2017-02-03T15:16:00.200Z", "content": "Thank you. Glad to hear that you didn't have troubles to run the code.
", "rawMarkdown": "Thank you. Glad to hear that you didn't have troubles to run the code." } ] }, { "id": 159321, "postDate": "2017-02-01T19:55:50.037Z", "content": "@blargl No there is no downside. In the previous version the GAP layer was added to the model in Notebook 03 so the training results should be the same.
", "rawMarkdown": "@blargl No there is no downside. In the previous version the GAP layer was added to the model in Notebook 03 so the training results should be the same." }, { "id": 159236, "postDate": "2017-02-01T13:29:07.730Z", "content": "@Vlad @Ronin Updated with:
\n\nNotebook 02A using generator approach and data augmentation
Notebook 02 now including the G.A.P layer in bottlenecks
With the update in notebook 2 the bottleneck file now has a size of 300MB wich make the bottleneck approach usable in computers with little RAM.
", "rawMarkdown": "@Vlad @Ronin Updated with:\r\n\r\n- Notebook 02A using generator approach and data augmentation\r\n\r\n- Notebook 02 now including the G.A.P layer in bottlenecks\r\n\r\nWith the update in notebook 2 the bottleneck file now has a size of 300MB wich make the bottleneck approach usable in computers with little RAM.\r\n\r\n", "replies": [ { "id": 159284, "postDate": "2017-02-01T17:45:54.903Z", "content": "\n\n\nNotebook 02 now including the G.A.P layer in bottlenecks
\n \nWith the update in notebook 2 the bottleneck file now has a size of 300MB wich make the bottleneck approach usable in computers with little RAM.
\n
Is there a downside to this compared to the multi-GB bottleneck files without the GAP layer?
", "rawMarkdown": "> Notebook 02 now including the G.A.P layer in bottlenecks\r\n\r\n> With the update in notebook 2 the bottleneck file now has a size of 300MB wich make the bottleneck approach usable in computers with little RAM.\r\n\r\nIs there a downside to this compared to the multi-GB bottleneck files without the GAP layer?" } ] }, { "id": 159119, "postDate": "2017-01-31T22:21:26.793Z", "content": "Well I got an example working. Will post shortly. I'll link it when complete.
\n\n", "rawMarkdown": "Well I got an example working. Will post shortly. I'll link it when complete.\r\n\r\nLinky:\r\nhttps://www.kaggle.com/rdslater/dogs-vs-cats-redux-kernels-edition/data-streaming-for-convoluted-neural-net/" }, { "id": 158995, "postDate": "2017-01-31T06:56:20.150Z", "content": "@Ronin, I always bump into that problem when trying to use fit_generator, so I'm very keen to see how people get over this.
", "rawMarkdown": "@Ronin, I always bump into that problem when trying to use fit_generator, so I'm very keen to see how people get over this. " }, { "id": 158978, "postDate": "2017-01-31T03:22:53.827Z", "content": "If anyone runs into issues getting 02.generatebottlenecks working and issues related to tensorflow, here's some instructions on re-installing everything, that worked for me.
\n\n", "rawMarkdown": "If anyone runs into issues getting 02.generatebottlenecks working and issues related to tensorflow, here's some instructions on re-installing everything, that worked for me.\r\n\r\nhttps://aichamp.wordpress.com/2017/01/10/tensorflow-0-12-1-cuda-8-0-and-cuda-5-1-on-ubuntu-16-04-with-titan-x/" }, { "id": 158968, "postDate": "2017-01-31T01:32:40.113Z", "content": "I've tried the fit_generater without much success. I seem to get stuck at having the wrong input dimensions. I am troubleshooting on the keras/stackoverflow but I am not having much luck. I am able to generate batches of data, but the first layer doesn't want to accept them as it expects an input of (none, ROWS, COLS, CHANNELS) while my generator gives it (Samples, ROWS, COLS, CHANNELS). The two don't seem to want to play well with each other. I cannot figure out the command to tell the input layer \"Expect 32 samples\" for fit_generator.
", "rawMarkdown": "I've tried the fit_generater without much success. I seem to get stuck at having the wrong input dimensions. I am troubleshooting on the keras/stackoverflow but I am not having much luck. I am able to generate batches of data, but the first layer doesn't want to accept them as it expects an input of (none, ROWS, COLS, CHANNELS) while my generator gives it (Samples, ROWS, COLS, CHANNELS). The two don't seem to want to play well with each other. I cannot figure out the command to tell the input layer \"Expect 32 samples\" for fit_generator." }, { "id": 158685, "postDate": "2017-01-28T23:25:36.073Z", "content": "@Vlad Indeed the fit_generator approach is designed to this cases where the dataset does not fit the memory. It also comes with the benefit of performing real time data augmentation. I'm intented to upload the notebook tomorrow.
\n\nHowever there are another posibilities to reduce the size of the bottleneck file like include the GlobalAveragePooling layer in the bottleneck generation. I'll upload the notebook with this alternative aswell.
", "rawMarkdown": "@Vlad Indeed the fit_generator approach is designed to this cases where the dataset does not fit the memory. It also comes with the benefit of performing real time data augmentation. I'm intented to upload the notebook tomorrow. \r\n\r\nHowever there are another posibilities to reduce the size of the bottleneck file like include the GlobalAveragePooling layer in the bottleneck generation. I'll upload the notebook with this alternative aswell." }, { "id": 158679, "postDate": "2017-01-28T22:12:50.953Z", "content": "Ah, I ran into another issue. The 300_bottlenecks.h5 file that was generated is 30gb... This means that it can't load into memory for notebook 03. I assume the fit_generator approach would help?
\n\nIf yes, do you have nay update on that Ronin?
", "rawMarkdown": "Ah, I ran into another issue. The 300_bottlenecks.h5 file that was generated is 30gb... This means that it can't load into memory for notebook 03. I assume the fit_generator approach would help?\r\n\r\nIf yes, do you have nay update on that Ronin?" }, { "id": 158366, "postDate": "2017-01-27T03:14:38.993Z", "content": "So I was thinking \"It would be a great idea if you could apply online learning to neural nets\" and then low and behold that is what fit_generator does. I will pursue that myself.
\n\nThis really was a nice learning post despite my slightly grumpy experience earlier. Between the fit_generator and using transfer learning you get a gold star from me (not that it is worth much!)
\n\nThanks again!
", "rawMarkdown": "So I was thinking \"It would be a great idea if you could apply online learning to neural nets\" and then low and behold that is what fit_generator does. I will pursue that myself.\r\n\r\nThis really was a nice learning post despite my slightly grumpy experience earlier. Between the fit_generator and using transfer learning you get a gold star from me (not that it is worth much!) \r\n\r\nThanks again! " }, { "id": 158342, "postDate": "2017-01-26T23:15:53.717Z", "content": "Actually it appears to be a problem with my system--I had to pair down image to about 4 and when monitoring my memory, I wasn't even close to load. There is a bug in my installs somewhere so that the program thinks it is out of memory--but according to my system monitor, things aren't even close to full (or even half).
\n\nI do have it on good authority (I can't say why as I work for them--just not as a data scientist) that this method you posted is the way many problems are being solved--pre-trained networks are becoming very big! It is a applicable solution. If I had more upvotes, I'd throw em your way.
", "rawMarkdown": "Actually it appears to be a problem with my system--I had to pair down image to about 4 and when monitoring my memory, I wasn't even close to load. There is a bug in my installs somewhere so that the program thinks it is out of memory--but according to my system monitor, things aren't even close to full (or even half).\r\n\r\nI do have it on good authority (I can't say why as I work for them--just not as a data scientist) that this method you posted is the way many problems are being solved--pre-trained networks are becoming very big! It is a applicable solution. If I had more upvotes, I'd throw em your way." }, { "id": 158238, "postDate": "2017-01-26T13:16:25.937Z", "content": "@Bin Yang I cant't know what is the problem you are facing if you don't give me more details. Are you following all the notebooks or are you changing/tuning some parameters??
\n\nTo make sure everithing works fine I just re-run the 4 notebooks from scratch in another computer and I actually get a better score on the Leaderboard (0.05799) without changing any line of code.
", "rawMarkdown": "@Bin Yang I cant't know what is the problem you are facing if you don't give me more details. Are you following all the notebooks or are you changing/tuning some parameters??\r\n\r\nTo make sure everithing works fine I just re-run the 4 notebooks from scratch in another computer and I actually get a better score on the Leaderboard (0.05799) without changing any line of code." }, { "id": 158231, "postDate": "2017-01-26T12:23:35.153Z", "content": "@Vlad Jiman I can't be sure about this but given that the X_train second dimension is 1000 my guess is that in the generate_bottlenecks notebook you have changed the argument include_top to True when creating the model. Or perhaps you are not using the resnet model that I uploaded here and you are using the included in keras by default.
Please check in that notebook that the output shape of the model is a 4D array:
\n\nmodel.input_shape[1:3], model.output_shape\nOut[5]:\n((300, 300), (None, 10, 10, 2048))\nHi, David, in your training, how do you decrease the loss from 0.5 to 0.05. In my case, even after about 50 epoch, the loss still around 0.5-0.6, decreased very slowly.
", "rawMarkdown": "Hi, David, in your training, how do you decrease the loss from 0.5 to 0.05. In my case, even after about 50 epoch, the loss still around 0.5-0.6, decreased very slowly. " }, { "id": 157999, "postDate": "2017-01-24T21:12:50.100Z", "content": "Hi David,
\n\nYes, I followed the previous notebooks exactly. The result of print(X_train.shape) is (23000, 1000).
\n\nAny idea?
\n\nVlad
", "rawMarkdown": "Hi David,\r\n\r\nYes, I followed the previous notebooks exactly. The result of print(X_train.shape) is (23000, 1000).\r\n\r\nAny idea?\r\n\r\nVlad" }, { "id": 157702, "postDate": "2017-01-23T08:44:27.393Z", "content": "@Vlad Jiman Seems to be a problem with the input_shape of the model.
Did you follow the steps in the previous notebook where the Bottlenecks for the train set were created.
\n\nAnd if so, could you tell me the result of calling (print(X_train.shape)) before you define de model?
Hi,
\n\nI get the following error in your third notebook when defining \"model\":
\n\n\n\n\n\n \n---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\n<ipython-input-17-3464986be4c7> in <module>()\n 1 model = Sequential([GlobalAveragePooling2D(input_shape=X_train.shape[1:]),\n 2 Dropout(0.3),\n----> 3 Dense(2, activation=\"softmax\")])\n\n/Users/vladjiman/anaconda/envs/fastaip3/lib/python3.5/site-packages/keras/models.py\nin init(self, layers, name)\n 271 if layers:\n 272 for layer in layers:\n --> 273 self.add(layer)\n 274 \n 275 def add(self, layer):
\n\n\n \n/Users/vladjiman/anaconda/envs/fastaip3/lib/python3.5/site-packages/keras/models.py\nin add(self, layer)\n 297 else:\n 298 input_dtype = None\n --> 299 layer.create_input_layer(batch_input_shape, input_dtype)\n 300 \n 301 if len(layer.inbound_nodes) != 1:
\n\n\n \n/Users/vladjiman/anaconda/envs/fastaip3/lib/python3.5/site-packages/keras/engine/topology.py\nin create_input_layer(self, batch_input_shape, input_dtype, name)\n 399 # and create the node connecting the current layer\n 400 # to the input layer we just created.\n --> 401 self(x)\n 402 \n 403 def add_weight(self, shape, initializer, name=None,
\n\n\n \n/Users/vladjiman/anaconda/envs/fastaip3/lib/python3.5/site-packages/keras/engine/topology.py\nin call(self, x, mask)\n 527 # Raise exceptions in case the input is not compatible\n 528 # with the input_spec specified in the layer constructor.\n --> 529 self.assert_input_compatibility(x)\n 530 \n 531 # Collect input shapes to build layer.
\n\n\n \n/Users/vladjiman/anaconda/envs/fastaip3/lib/python3.5/site-packages/keras/engine/topology.py\nin assert_input_compatibility(self, input)\n 467 self.name + ': expected ndim=' +\n 468 str(spec.ndim) + ', found ndim=' +\n --> 469 str(K.ndim(x)))\n 470 if spec.dtype is not None:\n 471 if K.dtype(x) != spec.dtype:
\n\n\nValueError: Input 0 is incompatible with layer globalaveragepooling2d_7: expected ndim=4, found ndim=2\n
Any idea why?
", "rawMarkdown": "Hi,\r\n\r\nI get the following error in your third notebook when defining \"model\":\r\n\r\n> ---------------------------------------------------------------------------\r\n> ValueError Traceback (most recent call last)\r\n>Thanks for the good info!
", "rawMarkdown": "Thanks for the good info!" }, { "id": 160135, "postDate": "2017-02-06T08:31:10.423Z", "content": "thanks.
", "rawMarkdown": "thanks." } ], "comments": [ { "id": 158338, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-01-26T23:04:10.783000", "content": "@Ronin If I find the time I'll upload a versión using fit_generator instead off computing the bottlenecks. This method is slower if you are going yo try diferents things but it requieres less memory.
", "votes": 3, "replies": [] }, { "id": 158334, "author_name": "Ronin", "author_url": "", "post_date": "2017-01-26T22:44:44.243000", "content": "sadly my 8mb machine barfed on this at the computing training bottlenecks. (even after pairing down to 500 images) Still thank you for posting. Very helpful.
\n\nedit--by shrinking the input size down in the 2nd file to 64x64 i was able to run without issue on the full set. So for those of you with lesser machines (I have 8gb memory and GTX 650 Ti) tryin smaller input sizes on the resnet input shape.
\n\nAlso a reminder to setup you files structure correctly--otherwise the '02' notebook will not pick up files correctly (I didn't have the 'test' subfolder in 'test'
\n\nworking-dir \n train\n dogs\n cats\n valid\n dogs\n cats\n test\n test\nThanks - I was missing the test/test subdirectory!
", "votes": 0, "replies": [] } ] }, { "id": 220433, "author_name": "abhay kumar", "author_url": "", "post_date": "2017-09-12T04:59:37.607000", "content": "0.06 ?? \nIs this the leaderboard score.\nI have fine tuned vgg16 to get validation loss of 0.0001.\nBut at the leaderboard i am getting 1.06...\nCan you suggest what might be wrong?
", "votes": 0, "replies": [ { "id": 235214, "author_name": "Roman Kornev", "author_url": "", "post_date": "2017-10-24T22:55:53.237000", "content": "Your validation loss is too low. You probably have training examples in your validation set. You should recreate your validation set and make sure that you don't have duplicate images.\nAlso don't forget to submit probabilities instead of labels. Clipping predictions in the interval from 0.01 to 0.99 (eps) might also help.
", "votes": 1, "replies": [] } ] }, { "id": 180755, "author_name": "Qichao Zhao", "author_url": "", "post_date": "2017-05-07T01:13:35.183000", "content": "Thank you for posting the guidance! Super helpful, your github link doesn't work anymore though!
", "votes": 0, "replies": [] }, { "id": 177952, "author_name": "pEman", "author_url": "", "post_date": "2017-04-26T08:50:32.163000", "content": "@Daivd,
\n\nThanks for this awesome stuff. How come you don't rescale images with 1/255. transformation when creating Bottleneck features? Isn't resnet expecting a transformation like this that rescales the data?
", "votes": 0, "replies": [] }, { "id": 164800, "author_name": "Emsi", "author_url": "", "post_date": "2017-03-02T12:06:05.237000", "content": "Great post. \nI tried similar approach but with Squeezenet (the keras SN: https://github.com/rcmalli/keras-squeezenet), it has fewer parameters thus runs faster.
\n\nI wonder why you included resnet50.py? \nKeras has ResNet50 implementation in applications already: https://keras.io/applications/\nIt can download the weights automatically like:
\n\nmodel = ResNet50(weights='imagenet')\nIs there any reason you used the included implementation?
\n\np.s. what's the reasoning behind bottleneck vs CNN codes?
", "votes": 0, "replies": [ { "id": 165957, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-03-07T19:38:55.560000", "content": "The main reason was that I wanted to re-implement the architecture from scratch, only looking at the original code when I got stucked, in order to theck if I understand the architecture correctly.\nThe other reason was that I included the imagenet preprocessing function inside the model.
\n\nI don't fully understand your question about the bottleneck reasoning but I'll asume that you are asking about what are the differences between those approaches.
\n\nUsing bottlenecks you are renouncing to some things:
\n\nBut, in exchange, you can train many different \"heads\" (trying different parameters, etc) on top of those bottelenecks, quite fast.
", "votes": 0, "replies": [] } ] }, { "id": 163275, "author_name": "kubilai", "author_url": "", "post_date": "2017-02-23T09:45:24.133000", "content": "Thank you so much for sharing, this is exactly what I needed to learn at this point.
\n\nOne question, why in notebook 03 do you train at lr=1e-5 first before bumping back up to 1e-3? I tried starting at 1e-3 directly and got exploding gradient pretty much every time. It seems mostly people start training at a high lr and reduce it slowly afterwards. What's your reasoning? Thanks!
", "votes": 0, "replies": [ { "id": 165959, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-03-07T19:40:55.400000", "content": "I train at low learning rate on first epoch in order to \"warm up\" and avoid exploding gradients. Yes it's usual to follow that procedure when training neural nets but in this case we are training a very tiny layer on top on some powerfull features.
", "votes": 0, "replies": [] } ] }, { "id": 160618, "author_name": "JennyYu", "author_url": "", "post_date": "2017-02-08T19:37:35.010000", "content": "Hi, can you share what version of CuDnn, CUDA and Tensorflow you use to run this code? Thank you~
", "votes": 0, "replies": [ { "id": 160624, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-08T20:48:24.750000", "content": "I can confirm that the notebooks work with the lastest version of Tensorflow, Cuda and Cudnn.
\n\nTry to follow the Tensorflow instalation instructions.
", "votes": 0, "replies": [] }, { "id": 160641, "author_name": "Ronin", "author_url": "", "post_date": "2017-02-08T22:06:07.830000", "content": "For my machine (which ran the code)
\n\nThank you for sharing. I am wondering why you didn't import ResNet thru Keras (https://keras.io/applications/#inceptionv3) but downloaded the resnet50.py and the weights to run the code instead? I tried to follow the Inception V3 transfer learning, but there was error message when it comes to training, and the error had something to do with CuDNN. So Im curious if others run into the same issue.
", "votes": 0, "replies": [ { "id": 160622, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-08T20:43:17.150000", "content": "I rewrited the code of the resnet50 file to check my understanding of the code. My version also include the imagenet preprocessing step inside the model definition.
", "votes": 0, "replies": [] } ] }, { "id": 160445, "author_name": "Zaid Daba'een", "author_url": "", "post_date": "2017-02-07T18:30:19.790000", "content": "Would've been nicer to have it all in one notebook.
", "votes": 0, "replies": [] }, { "id": 159889, "author_name": "Vlad Jiman", "author_url": "", "post_date": "2017-02-04T16:43:10.373000", "content": "Got a bit of time to work through your example and it is 100% reproducible. Well done!
\n\nMy question now is, why is this so much more different than the results I would get with the inbuilt Keras Resnet50 model?
\n\nWith that I can only scrape a 0.80 loss Notebook here
", "votes": 0, "replies": [ { "id": 159991, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-05T11:11:04.800000", "content": "I'm not sure about what is the problem and I didin't look into your notebook in detail but I noticed 2 things:
\n\n1 - You are appending the new Dense layer with 2 outputs on top of the Resnet fc1000 layer.
The usual thing to do (as I do in the example notebooks) is to replace the fc1000 layer instead of appending the new layer on top of it.
This is because the weights of the original resnet 'fc1000' layer are trained to predict the 1000 categories of the Imagenet Challenge.
\n\nAlthough many of this 1000 categories are types of dogs and cats it just work better if you replace the 1000 fc layer with a new layer that you train from scratch.
\n\n2 - You are clipping the scores too much.
\n\nAlthough when dealing with log loss is usually a good idea to \"clip\" your predictions to avoid been penalized by the overconfidence of the model, the clip is usually in the order of 0.95 / 0.05 or so.
\n\nI see at the end of your notebook that you are clipping the scores to 0.825548 / 0.175611 so even when you have guessed the right label your logloss is still quite high.
\n\nI think that if you to change those 2 things your logloss should decrease.
", "votes": 1, "replies": [] } ] }, { "id": 159815, "author_name": "Bin Yang", "author_url": "", "post_date": "2017-02-04T04:32:37.357000", "content": "Hi, has anyone submitted to test? I used the fit_generator method which validation error is about 0.04 but the test error is very big like 4.0, anyone knows why? The only difference here is I manually split the training and validation image.
", "votes": 0, "replies": [ { "id": 159854, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-04T11:45:28.033000", "content": "There is probably something wrong with your manual split.
\n\nThe fit_generator method reorder the images internally different than your operating system.
\n\nThat's why in the submission notebook I get the order of the filenames in the generator by calling test_batches.filenames wich returns a list with the proper order.
Not yet done any submissions; but very useful!
", "votes": 0, "replies": [ { "id": 159720, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-03T15:16:21.520000", "content": "Thanks!. What are you waiting for??? :)
", "votes": 0, "replies": [] } ] }, { "id": 159684, "author_name": "LifeMatrix", "author_url": "", "post_date": "2017-02-03T10:54:50.157000", "content": "Really nice job and give me many inspirations. The code is right and I can reproduce the results. \nThanks a lot for sharing.
", "votes": 0, "replies": [ { "id": 159719, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-03T15:16:00.200000", "content": "Thank you. Glad to hear that you didn't have troubles to run the code.
", "votes": 0, "replies": [] } ] }, { "id": 159321, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-01T19:55:50.037000", "content": "@blargl No there is no downside. In the previous version the GAP layer was added to the model in Notebook 03 so the training results should be the same.
", "votes": 0, "replies": [] }, { "id": 159236, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-02-01T13:29:07.730000", "content": "@Vlad @Ronin Updated with:
\n\nNotebook 02A using generator approach and data augmentation
Notebook 02 now including the G.A.P layer in bottlenecks
With the update in notebook 2 the bottleneck file now has a size of 300MB wich make the bottleneck approach usable in computers with little RAM.
", "votes": 0, "replies": [ { "id": 159284, "author_name": "blargl", "author_url": "", "post_date": "2017-02-01T17:45:54.903000", "content": "\n\n\nNotebook 02 now including the G.A.P layer in bottlenecks
\n \nWith the update in notebook 2 the bottleneck file now has a size of 300MB wich make the bottleneck approach usable in computers with little RAM.
\n
Is there a downside to this compared to the multi-GB bottleneck files without the GAP layer?
", "votes": 0, "replies": [] } ] }, { "id": 159119, "author_name": "Ronin", "author_url": "", "post_date": "2017-01-31T22:21:26.793000", "content": "Well I got an example working. Will post shortly. I'll link it when complete.
\n\n", "votes": 0, "replies": [] }, { "id": 158995, "author_name": "Vlad Jiman", "author_url": "", "post_date": "2017-01-31T06:56:20.150000", "content": "@Ronin, I always bump into that problem when trying to use fit_generator, so I'm very keen to see how people get over this.
", "votes": 0, "replies": [] }, { "id": 158978, "author_name": "Rainer", "author_url": "", "post_date": "2017-01-31T03:22:53.827000", "content": "If anyone runs into issues getting 02.generatebottlenecks working and issues related to tensorflow, here's some instructions on re-installing everything, that worked for me.
\n\n", "votes": 0, "replies": [] }, { "id": 158968, "author_name": "Ronin", "author_url": "", "post_date": "2017-01-31T01:32:40.113000", "content": "I've tried the fit_generater without much success. I seem to get stuck at having the wrong input dimensions. I am troubleshooting on the keras/stackoverflow but I am not having much luck. I am able to generate batches of data, but the first layer doesn't want to accept them as it expects an input of (none, ROWS, COLS, CHANNELS) while my generator gives it (Samples, ROWS, COLS, CHANNELS). The two don't seem to want to play well with each other. I cannot figure out the command to tell the input layer \"Expect 32 samples\" for fit_generator.
", "votes": 0, "replies": [] }, { "id": 158685, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-01-28T23:25:36.073000", "content": "@Vlad Indeed the fit_generator approach is designed to this cases where the dataset does not fit the memory. It also comes with the benefit of performing real time data augmentation. I'm intented to upload the notebook tomorrow.
\n\nHowever there are another posibilities to reduce the size of the bottleneck file like include the GlobalAveragePooling layer in the bottleneck generation. I'll upload the notebook with this alternative aswell.
", "votes": 0, "replies": [] }, { "id": 158679, "author_name": "Vlad Jiman", "author_url": "", "post_date": "2017-01-28T22:12:50.953000", "content": "Ah, I ran into another issue. The 300_bottlenecks.h5 file that was generated is 30gb... This means that it can't load into memory for notebook 03. I assume the fit_generator approach would help?
\n\nIf yes, do you have nay update on that Ronin?
", "votes": 0, "replies": [] }, { "id": 158366, "author_name": "Ronin", "author_url": "", "post_date": "2017-01-27T03:14:38.993000", "content": "So I was thinking \"It would be a great idea if you could apply online learning to neural nets\" and then low and behold that is what fit_generator does. I will pursue that myself.
\n\nThis really was a nice learning post despite my slightly grumpy experience earlier. Between the fit_generator and using transfer learning you get a gold star from me (not that it is worth much!)
\n\nThanks again!
", "votes": 0, "replies": [] }, { "id": 158342, "author_name": "Ronin", "author_url": "", "post_date": "2017-01-26T23:15:53.717000", "content": "Actually it appears to be a problem with my system--I had to pair down image to about 4 and when monitoring my memory, I wasn't even close to load. There is a bug in my installs somewhere so that the program thinks it is out of memory--but according to my system monitor, things aren't even close to full (or even half).
\n\nI do have it on good authority (I can't say why as I work for them--just not as a data scientist) that this method you posted is the way many problems are being solved--pre-trained networks are becoming very big! It is a applicable solution. If I had more upvotes, I'd throw em your way.
", "votes": 0, "replies": [] }, { "id": 158238, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-01-26T13:16:25.937000", "content": "@Bin Yang I cant't know what is the problem you are facing if you don't give me more details. Are you following all the notebooks or are you changing/tuning some parameters??
\n\nTo make sure everithing works fine I just re-run the 4 notebooks from scratch in another computer and I actually get a better score on the Leaderboard (0.05799) without changing any line of code.
", "votes": 0, "replies": [] }, { "id": 158231, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-01-26T12:23:35.153000", "content": "@Vlad Jiman I can't be sure about this but given that the X_train second dimension is 1000 my guess is that in the generate_bottlenecks notebook you have changed the argument include_top to True when creating the model. Or perhaps you are not using the resnet model that I uploaded here and you are using the included in keras by default.
Please check in that notebook that the output shape of the model is a 4D array:
\n\nmodel.input_shape[1:3], model.output_shape\nOut[5]:\n((300, 300), (None, 10, 10, 2048))\nHi, David, in your training, how do you decrease the loss from 0.5 to 0.05. In my case, even after about 50 epoch, the loss still around 0.5-0.6, decreased very slowly.
", "votes": 0, "replies": [] }, { "id": 157999, "author_name": "Vlad Jiman", "author_url": "", "post_date": "2017-01-24T21:12:50.100000", "content": "Hi David,
\n\nYes, I followed the previous notebooks exactly. The result of print(X_train.shape) is (23000, 1000).
\n\nAny idea?
\n\nVlad
", "votes": 0, "replies": [] }, { "id": 157702, "author_name": "David de la Iglesia Castro", "author_url": "", "post_date": "2017-01-23T08:44:27.393000", "content": "@Vlad Jiman Seems to be a problem with the input_shape of the model.
Did you follow the steps in the previous notebook where the Bottlenecks for the train set were created.
\n\nAnd if so, could you tell me the result of calling (print(X_train.shape)) before you define de model?