Hey everyone,\nThese are the results for all popular models available. Let me know if I missed any:
\n\n 1. VGGNet 19 : \n Accuracy: 83%\n Log Loss: 0.56 \n\n 2. Inception V3: \n Accuracy: 87%\n Log Loss: 0.47\n\n 3. ResNet50: \n Accuracy: 90%\n Log Loss: 0.38\n\n 4. Xception: \n Accuracy: 89%\n Log Loss: 0.42\n\n 5. DenseNet:\n Accuracy: 91%\n Log Loss: 0.36\n\n 6. SENet:\n Accuracy: 89%\n Log Loss: 0.38\n\n 7. ResNext:\n Accuracy: 93%\n Log Loss: 0.22\n\n 8. InceptionV4:\n Accuracy: 94%\n Log Loss: 0.20\n\n 9. InceptionResnetV2:\n Accuracy: 95%\n Log Loss: 0.19\n\n 10. Ensembling InceptionResNetV2, InceptionV4 and ResNext:\n Accuracy: 96%\n Log Loss: 0.16\nBest hyper-parameters I found were:
\n\nImage Sizes: Between 299 - 400
\n\nLearning Rate: Play with it. 0.001 worked for InceptionResNet but for ResNext 0.004 worked better.
\n\nFreezing/Unfreezing layers: Didn't helped for most models but helped for ResNet and InceptionV3. Also play with BatchNorm and Dropout.
\n\nPyTorch or Keras: PyTorch
", "messages": [ { "id": 261360, "postDate": "2017-12-22T12:03:45.490Z", "content": "Hey everyone,\nThese are the results for all popular models available. Let me know if I missed any:
\n\n 1. VGGNet 19 : \n Accuracy: 83%\n Log Loss: 0.56 \n\n 2. Inception V3: \n Accuracy: 87%\n Log Loss: 0.47\n\n 3. ResNet50: \n Accuracy: 90%\n Log Loss: 0.38\n\n 4. Xception: \n Accuracy: 89%\n Log Loss: 0.42\n\n 5. DenseNet:\n Accuracy: 91%\n Log Loss: 0.36\n\n 6. SENet:\n Accuracy: 89%\n Log Loss: 0.38\n\n 7. ResNext:\n Accuracy: 93%\n Log Loss: 0.22\n\n 8. InceptionV4:\n Accuracy: 94%\n Log Loss: 0.20\n\n 9. InceptionResnetV2:\n Accuracy: 95%\n Log Loss: 0.19\n\n 10. Ensembling InceptionResNetV2, InceptionV4 and ResNext:\n Accuracy: 96%\n Log Loss: 0.16\nBest hyper-parameters I found were:
\n\nImage Sizes: Between 299 - 400
\n\nLearning Rate: Play with it. 0.001 worked for InceptionResNet but for ResNext 0.004 worked better.
\n\nFreezing/Unfreezing layers: Didn't helped for most models but helped for ResNet and InceptionV3. Also play with BatchNorm and Dropout.
\n\nPyTorch or Keras: PyTorch
", "rawMarkdown": "Hey everyone,\nThese are the results for all popular models available. Let me know if I missed any:\n\n 1. VGGNet 19 : \n Accuracy: 83%\n Log Loss: 0.56 \n \n 2. Inception V3: \n Accuracy: 87%\n Log Loss: 0.47\n \n 3. ResNet50: \n Accuracy: 90%\n Log Loss: 0.38\n \n 4. Xception: \n Accuracy: 89%\n Log Loss: 0.42\n \n 5. DenseNet:\n Accuracy: 91%\n Log Loss: 0.36\n \n 6. SENet:\n Accuracy: 89%\n Log Loss: 0.38\n \n 7. ResNext:\n Accuracy: 93%\n Log Loss: 0.22\n \n 8. InceptionV4:\n Accuracy: 94%\n Log Loss: 0.20\n \n 9. InceptionResnetV2:\n Accuracy: 95%\n Log Loss: 0.19\n \n 10. Ensembling InceptionResNetV2, InceptionV4 and ResNext:\n Accuracy: 96%\n Log Loss: 0.16\n\nBest hyper-parameters I found were:\n\nImage Sizes: Between 299 - 400 \n\nLearning Rate: Play with it. 0.001 worked for InceptionResNet but for ResNext 0.004 worked better. \n\nFreezing/Unfreezing layers: Didn't helped for most models but helped for ResNet and InceptionV3. Also play with BatchNorm and Dropout. \n\nPyTorch or Keras: PyTorch", "votes": 12 }, { "id": 1059218, "postDate": "2020-10-24T19:35:52.963Z", "content": "I heard a lot how resnet is one of the greatest model out there. How come it only gets 90%. I my self got 94. Can I improve on it by adding more layers to it?
", "rawMarkdown": "I heard a lot how resnet is one of the greatest model out there. How come it only gets 90%. I my self got 94. Can I improve on it by adding more layers to it? " }, { "id": 275712, "postDate": "2018-01-29T20:53:49.323Z", "content": "Hi Kumar,
\n\nNice summary. \nCould you please describe how did you manage to reach 0.47 with InceptionV3? I just can get closer to it. (I'm at around 0.67). I'm chasing the 0.47 but getting frustrated about none of my improvement ideas were good enough.:(\nI use augmentation. I've tried image preprocessing, changing resolution 299-->400 (maybe not the right way) , tried dropout, batchnormalization, adding only 1 or more 2,3,4 FC layers with different combination of shapes on top of the InceptionV3 (without top layer, with 'avg' pooling), adam, SGD, different LRs and WDs but I didn't find the right combination. I have the feeling I tried everything but didn't find the final point to reach 0.47.\nCould you please add the followings to your post:\n-Do you use augmentation. If yes, what are your params?\n-What is the topX layer you added to the InceprtionV3?\n-What are the params you've been using to compile, fit, etc (hyper params)?\n-What are the fitting params?
\n\nThank you, Daniel
", "rawMarkdown": "Hi Kumar,\n\nNice summary. \nCould you please describe how did you manage to reach 0.47 with InceptionV3? I just can get closer to it. (I'm at around 0.67). I'm chasing the 0.47 but getting frustrated about none of my improvement ideas were good enough.:(\nI use augmentation. I've tried image preprocessing, changing resolution 299-->400 (maybe not the right way) , tried dropout, batchnormalization, adding only 1 or more 2,3,4 FC layers with different combination of shapes on top of the InceptionV3 (without top layer, with 'avg' pooling), adam, SGD, different LRs and WDs but I didn't find the right combination. I have the feeling I tried everything but didn't find the final point to reach 0.47.\nCould you please add the followings to your post:\n-Do you use augmentation. If yes, what are your params?\n-What is the topX layer you added to the InceprtionV3?\n-What are the params you've been using to compile, fit, etc (hyper params)?\n-What are the fitting params?\n\nThank you, Daniel", "replies": [ { "id": 276907, "postDate": "2018-02-01T19:15:25.160Z", "content": "Reply to myself. :)\nHavin only one InceptionV3 (without top layer) with 2 extra layers on top of it - typical transfer learning - I managed to achieve 0.28374.\nI went with a very basic setup.\nBatch size: 32\nResolution 400\nTrain/valid split: 80:20 (using only the original train dataset published on Kaggle)\nImage preprocessing: only rescale (1/255.) \nTop layers:\n = cnnmodel_inception.output\nx = Dropout(0.3)(x)\nx = Dense(1204, activation='relu')(x)\nx = Dropout(0.5)(x)\npredictions = Dense(120, activation=\"softmax\")(x)
\n\noptimizers : SGD(lr=0.001, momentum=0.9)\nmonitoring val_loss for checkpoint and earlystop\nCNN: frozen, only newly added FCs were trained
", "rawMarkdown": "Reply to myself. :)\nHavin only one InceptionV3 (without top layer) with 2 extra layers on top of it - typical transfer learning - I managed to achieve 0.28374.\nI went with a very basic setup.\nBatch size: 32\nResolution 400\nTrain/valid split: 80:20 (using only the original train dataset published on Kaggle)\nImage preprocessing: only rescale (1/255.) \nTop layers:\n = cnnmodel_inception.output\nx = Dropout(0.3)(x)\nx = Dense(1204, activation='relu')(x)\nx = Dropout(0.5)(x)\npredictions = Dense(120, activation=\"softmax\")(x)\n\noptimizers : SGD(lr=0.001, momentum=0.9)\nmonitoring val_loss for checkpoint and earlystop\nCNN: frozen, only newly added FCs were trained\n\n\n\n", "votes": 1 }, { "id": 572591, "postDate": "2019-07-11T06:34:10.913Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 272333, "postDate": "2018-01-22T20:56:30.060Z", "content": "How are you ensembling? Just average? And just one run per model? I'm just getting into ensembling techniques and I assume most will train 100s of each model on high end GPUs, and then try various blending functions. Do they also grid search some of the parameters of these models? I could see these ensemble techniques taking days to train.
\n\nPerhaps a dual-path network?
", "rawMarkdown": "How are you ensembling? Just average? And just one run per model? I'm just getting into ensembling techniques and I assume most will train 100s of each model on high end GPUs, and then try various blending functions. Do they also grid search some of the parameters of these models? I could see these ensemble techniques taking days to train.\n\nPerhaps a dual-path network?" }, { "id": 270790, "postDate": "2018-01-18T23:30:41.763Z", "content": "Right you are! Using Image size 420 with Ensembling InceptionResNetV2 and InceptionV4 I got to 0.177.
", "rawMarkdown": "Right you are! Using Image size 420 with Ensembling InceptionResNetV2 and InceptionV4 I got to 0.177. " }, { "id": 266877, "postDate": "2018-01-10T03:31:23.130Z", "content": "I am training with a VGG19 model, but getting very less validation error. Below is the specification.
\nimage_size = 90 * 90 * 3
\nbatch_size = 16
\nnum_epochs = 30
\noptimizer = 'adam', lr = 0.001
\nAdded 1 FC with 1024 size and one softmax layer with 120 outputs.
\n
\nUsing 80-20 training and validation split.
\nUsing dropout and L2 regularization in last 2 layers.
\nBut always getting validation accuracy around 16-17%.
\n
\nI have also tried trained last few layers of convnet also.
\n
\nAny suggestion would be really helpful.
Note that in the post above the input image sizes used were between 299 and 400. Presumably the small size of your input makes a big difference but even then 16-17% seems a bit low. Are you sure your validation set is representative?
", "rawMarkdown": "Note that in the post above the input image sizes used were between 299 and 400. Presumably the small size of your input makes a big difference but even then 16-17% seems a bit low. Are you sure your validation set is representative?" } ] } ], "comments": [ { "id": 1059218, "author_name": "Amir Reza", "author_url": "", "post_date": "2020-10-24T19:35:52.963000", "content": "I heard a lot how resnet is one of the greatest model out there. How come it only gets 90%. I my self got 94. Can I improve on it by adding more layers to it?
", "votes": 0, "replies": [] }, { "id": 275712, "author_name": "Daniel Morvay", "author_url": "", "post_date": "2018-01-29T20:53:49.323000", "content": "Hi Kumar,
\n\nNice summary. \nCould you please describe how did you manage to reach 0.47 with InceptionV3? I just can get closer to it. (I'm at around 0.67). I'm chasing the 0.47 but getting frustrated about none of my improvement ideas were good enough.:(\nI use augmentation. I've tried image preprocessing, changing resolution 299-->400 (maybe not the right way) , tried dropout, batchnormalization, adding only 1 or more 2,3,4 FC layers with different combination of shapes on top of the InceptionV3 (without top layer, with 'avg' pooling), adam, SGD, different LRs and WDs but I didn't find the right combination. I have the feeling I tried everything but didn't find the final point to reach 0.47.\nCould you please add the followings to your post:\n-Do you use augmentation. If yes, what are your params?\n-What is the topX layer you added to the InceprtionV3?\n-What are the params you've been using to compile, fit, etc (hyper params)?\n-What are the fitting params?
\n\nThank you, Daniel
", "votes": 0, "replies": [ { "id": 276907, "author_name": "Daniel Morvay", "author_url": "", "post_date": "2018-02-01T19:15:25.160000", "content": "Reply to myself. :)\nHavin only one InceptionV3 (without top layer) with 2 extra layers on top of it - typical transfer learning - I managed to achieve 0.28374.\nI went with a very basic setup.\nBatch size: 32\nResolution 400\nTrain/valid split: 80:20 (using only the original train dataset published on Kaggle)\nImage preprocessing: only rescale (1/255.) \nTop layers:\n = cnnmodel_inception.output\nx = Dropout(0.3)(x)\nx = Dense(1204, activation='relu')(x)\nx = Dropout(0.5)(x)\npredictions = Dense(120, activation=\"softmax\")(x)
\n\noptimizers : SGD(lr=0.001, momentum=0.9)\nmonitoring val_loss for checkpoint and earlystop\nCNN: frozen, only newly added FCs were trained
", "votes": 1, "replies": [] }, { "id": 572591, "author_name": "", "author_url": "", "post_date": "2019-07-11T06:34:10.913000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 272333, "author_name": "NeuroGuy", "author_url": "", "post_date": "2018-01-22T20:56:30.060000", "content": "How are you ensembling? Just average? And just one run per model? I'm just getting into ensembling techniques and I assume most will train 100s of each model on high end GPUs, and then try various blending functions. Do they also grid search some of the parameters of these models? I could see these ensemble techniques taking days to train.
\n\nPerhaps a dual-path network?
", "votes": 0, "replies": [] }, { "id": 270790, "author_name": "Bart Fish", "author_url": "", "post_date": "2018-01-18T23:30:41.763000", "content": "Right you are! Using Image size 420 with Ensembling InceptionResNetV2 and InceptionV4 I got to 0.177.
", "votes": 0, "replies": [] }, { "id": 266877, "author_name": "Swaraj Khadanga", "author_url": "", "post_date": "2018-01-10T03:31:23.130000", "content": "I am training with a VGG19 model, but getting very less validation error. Below is the specification.
\nimage_size = 90 * 90 * 3
\nbatch_size = 16
\nnum_epochs = 30
\noptimizer = 'adam', lr = 0.001
\nAdded 1 FC with 1024 size and one softmax layer with 120 outputs.
\n
\nUsing 80-20 training and validation split.
\nUsing dropout and L2 regularization in last 2 layers.
\nBut always getting validation accuracy around 16-17%.
\n
\nI have also tried trained last few layers of convnet also.
\n
\nAny suggestion would be really helpful.
Note that in the post above the input image sizes used were between 299 and 400. Presumably the small size of your input makes a big difference but even then 16-17% seems a bit low. Are you sure your validation set is representative?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "261360": "Hey everyone,\nThese are the results for all popular models available. Let me know if I missed any:\n\n 1. VGGNet 19 : \n Accuracy: 83%\n Log Loss: 0.56 \n \n 2. Inception V3: \n Accuracy: 87%\n Log Loss: 0.47\n \n 3. ResNet50: \n Accuracy: 90%\n Log Loss: 0.38\n \n 4. Xception: \n Accuracy: 89%\n Log Loss: 0.42\n \n 5. DenseNet:\n Accuracy: 91%\n Log Loss: 0.36\n \n 6. SENet:\n Accuracy: 89%\n Log Loss: 0.38\n \n 7. ResNext:\n Accuracy: 93%\n Log Loss: 0.22\n \n 8. InceptionV4:\n Accuracy: 94%\n Log Loss: 0.20\n \n 9. InceptionResnetV2:\n Accuracy: 95%\n Log Loss: 0.19\n \n 10. Ensembling InceptionResNetV2, InceptionV4 and ResNext:\n Accuracy: 96%\n Log Loss: 0.16\n\nBest hyper-parameters I found were:\n\nImage Sizes: Between 299 - 400 \n\nLearning Rate: Play with it. 0.001 worked for InceptionResNet but for ResNext 0.004 worked better. \n\nFreezing/Unfreezing layers: Didn't helped for most models but helped for ResNet and InceptionV3. Also play with BatchNorm and Dropout. \n\nPyTorch or Keras: PyTorch", "1059218": "I heard a lot how resnet is one of the greatest model out there. How come it only gets 90%. I my self got 94. Can I improve on it by adding more layers to it? ", "275712": "Hi Kumar,\n\nNice summary. \nCould you please describe how did you manage to reach 0.47 with InceptionV3? I just can get closer to it. (I'm at around 0.67). I'm chasing the 0.47 but getting frustrated about none of my improvement ideas were good enough.:(\nI use augmentation. I've tried image preprocessing, changing resolution 299-->400 (maybe not the right way) , tried dropout, batchnormalization, adding only 1 or more 2,3,4 FC layers with different combination of shapes on top of the InceptionV3 (without top layer, with 'avg' pooling), adam, SGD, different LRs and WDs but I didn't find the right combination. I have the feeling I tried everything but didn't find the final point to reach 0.47.\nCould you please add the followings to your post:\n-Do you use augmentation. If yes, what are your params?\n-What is the topX layer you added to the InceprtionV3?\n-What are the params you've been using to compile, fit, etc (hyper params)?\n-What are the fitting params?\n\nThank you, Daniel", "272333": "How are you ensembling? Just average? And just one run per model? I'm just getting into ensembling techniques and I assume most will train 100s of each model on high end GPUs, and then try various blending functions. Do they also grid search some of the parameters of these models? I could see these ensemble techniques taking days to train.\n\nPerhaps a dual-path network?", "270790": "Right you are! Using Image size 420 with Ensembling InceptionResNetV2 and InceptionV4 I got to 0.177. ", "266877": "I am training with a VGG19 model, but getting very less validation error. Below is the specification.