{
  "id": 43203,
  "title": "Accuracy sharing",
  "url": "/competitions/dog-breed-identification/discussion/43203",
  "author_name": "",
  "post_date": "2017-11-11T10:26:07.481616400Z",
  "votes": 16,
  "comment_count": 26,
  "views": 0,
  "content": "<p>Models log loss on a whole training set (acquired through 5 fold cv, between folds score might very from less than 0.17 to 0.26):</p>\n\n<ul>\n<li>inception_4_300    0.228</li>\n<li>inception_4_350    0.211</li>\n<li>inception_4_400    0.204</li>\n<li>inception_4_450    0.223</li>\n<li>inceptionresnet_2_300      0.239</li>\n<li>inceptionresnet_2_350      0.217</li>\n<li>inceptionresnet_2_400      0.215</li>\n<li>inceptionresnet_2_450      0.222</li>\n</ul>\n\n<p>Simple averaging all models except for inceptionresnet_2_300, inception_4_450 gives 0.181 on a training set and 0.172 on a leaderboard.</p>",
  "messages": [
    {
      "id": "242357",
      "postDate": "11/11/2017 10:26:07",
      "content": "<p>Models log loss on a whole training set (acquired through 5 fold cv, between folds score might very from less than 0.17 to 0.26):</p>\n\n<ul>\n<li>inception_4_300    0.228</li>\n<li>inception_4_350    0.211</li>\n<li>inception_4_400    0.204</li>\n<li>inception_4_450    0.223</li>\n<li>inceptionresnet_2_300      0.239</li>\n<li>inceptionresnet_2_350      0.217</li>\n<li>inceptionresnet_2_400      0.215</li>\n<li>inceptionresnet_2_450      0.222</li>\n</ul>\n\n<p>Simple averaging all models except for inceptionresnet_2_300, inception_4_450 gives 0.181 on a training set and 0.172 on a leaderboard.</p>",
      "rawMarkdown": "Models log loss on a whole training set (acquired through 5 fold cv, between folds score might very from less than 0.17 to 0.26):\n\n* inception_4_300 \t 0.228\n* inception_4_350 \t 0.211\n* inception_4_400 \t 0.204\n* inception_4_450 \t 0.223\n* inceptionresnet_2_300 \t 0.239\n* inceptionresnet_2_350 \t 0.217\n* inceptionresnet_2_400 \t 0.215\n* inceptionresnet_2_450 \t 0.222\n\nSimple averaging all models except for inceptionresnet_2_300, inception_4_450 gives 0.181 on a training set and 0.172 on a leaderboard.",
      "votes": null
    },
    {
      "id": "244518",
      "postDate": "11/16/2017 13:41:07",
      "content": "<p>Did you use any regularization when fine-tuning?</p>",
      "rawMarkdown": "Did you use any regularization when fine-tuning?",
      "votes": null
    },
    {
      "id": "244620",
      "postDate": "11/16/2017 16:39:39",
      "content": "<p>Only dropout. </p>",
      "rawMarkdown": "Only dropout.",
      "votes": null
    },
    {
      "id": "245488",
      "postDate": "11/18/2017 16:08:05",
      "content": "<p>Are those models pre-trained only on Imagenet?</p>",
      "rawMarkdown": "Are those models pre-trained only on Imagenet?",
      "votes": null
    },
    {
      "id": "245489",
      "postDate": "11/18/2017 16:10:55",
      "content": "<p>Convolutional layers pre-trained on Imagenet, correct. FC layers tuned on train set. No cheating )</p>",
      "rawMarkdown": "Convolutional layers pre-trained on Imagenet, correct. FC layers tuned on train set. No cheating )",
      "votes": null
    },
    {
      "id": "245506",
      "postDate": "11/18/2017 17:24:46",
      "content": "<p>I see :). So, if I understood correctly, you froze the convolutional weights and only fine-tuned FC layers?</p>",
      "rawMarkdown": "I see :). So, if I understood correctly, you froze the convolutional weights and only fine-tuned FC layers?",
      "votes": null
    },
    {
      "id": "245744",
      "postDate": "11/19/2017 14:23:05",
      "content": "<p>Yes, thats right.</p>",
      "rawMarkdown": "Yes, thats right.",
      "votes": null
    },
    {
      "id": "245815",
      "postDate": "11/19/2017 19:50:21",
      "content": "<p>Please forgive my ignorance, but what do you mean by averaging? Are you saving the weights for each model and then averaging over all of them? </p>",
      "rawMarkdown": "Please forgive my ignorance, but what do you mean by averaging? Are you saving the weights for each model and then averaging over all of them?",
      "votes": null
    },
    {
      "id": "246080",
      "postDate": "11/20/2017 14:28:16",
      "content": "<p>Averaging is related to cross-validation (cv) technique. Basically, one can split training data into <code>n</code> subparts (folds), each fold is then used as a validation set while the <code>n-1</code> remaining folds form the training set. For instance, in case of 5-fold cross-validation we have 5 splits (0, 1, 2, 3, 4) of the training data. We choose the model (VGG, Inception, etc) and train it on 0-3 folds checking its performance on validation data (4th fold). The state (weights) of the model producing the smallest validation error is saved (model1). Then, we take another set of folds (let's assume 0, 1, 2, 4) to create new training and validation (3rd fold) datasets respectively and train our model from the same starting point as we used previously but on this new data. It will give us another model state (model2) which is optimal for this particular validation set. Following the same procedure, one can easily get 5 models. Using those models (states), we can predict probabilities on <code>test</code> data for each model independently, then summing predictions up and dividing by the number of models (folds) leads to the final result. So, this is where averaging comes on stage.</p>",
      "rawMarkdown": "Averaging is related to cross-validation (cv) technique. Basically, one can split training data into `n` subparts (folds), each fold is then used as a validation set while the `n-1` remaining folds form the training set. For instance, in case of 5-fold cross-validation we have 5 splits (0, 1, 2, 3, 4) of the training data. We choose the model (VGG, Inception, etc) and train it on 0-3 folds checking its performance on validation data (4th fold). The state (weights) of the model producing the smallest validation error is saved (model1). Then, we take another set of folds (let's assume 0, 1, 2, 4) to create new training and validation (3rd fold) datasets respectively and train our model from the same starting point as we used previously but on this new data. It will give us another model state (model2) which is optimal for this particular validation set. Following the same procedure, one can easily get 5 models. Using those models (states), we can predict probabilities on `test` data for each model independently, then summing predictions up and dividing by the number of models (folds) leads to the final result. So, this is where averaging comes on stage.",
      "votes": null
    },
    {
      "id": "248517",
      "postDate": "11/26/2017 10:29:30",
      "content": "<p>Hello, it looks like CV helps a lot (at least in this competition). Is there something to be done in all deep learning competitions ? </p>",
      "rawMarkdown": "Hello, it looks like CV helps a lot (at least in this competition). Is there something to be done in all deep learning competitions ?",
      "votes": null
    },
    {
      "id": "251172",
      "postDate": "11/30/2017 19:16:51",
      "content": "<p>Thanks Iaroslav! </p>\n\n<p>Do you run each model (VGG, Inception, etc) on each set of folds ((0,1,2,3), (0,2,3,4), etc) and pick the best model and parameters for each validation set?</p>\n\n<p>Do you know of any good resources for learning to implement this in python?</p>",
      "rawMarkdown": "Thanks Iaroslav! \n\nDo you run each model (VGG, Inception, etc) on each set of folds ((0,1,2,3), (0,2,3,4), etc) and pick the best model and parameters for each validation set?\n\nDo you know of any good resources for learning to implement this in python?",
      "votes": null
    },
    {
      "id": "251404",
      "postDate": "12/01/2017 05:54:59",
      "content": "<p>Hi, thanks a lot for your sharing. But I'm a very newbie of machine learning, and I have two questions that might be stupid. So I know there are Keras pretrained models InceptionV3 and InceptionResNetV2, but what do you mean by \"inception_4_300\" or \"inceptionresnet_2_450\"? And by \"0.181 on a training set\", do you mean you calculated the mean of the (five) validation set errors and it's 0.181?\nThanks for helping me!</p>",
      "rawMarkdown": "Hi, thanks a lot for your sharing. But I'm a very newbie of machine learning, and I have two questions that might be stupid. So I know there are Keras pretrained models InceptionV3 and InceptionResNetV2, but what do you mean by \"inception_4_300\" or \"inceptionresnet_2_450\"? And by \"0.181 on a training set\", do you mean you calculated the mean of the (five) validation set errors and it's 0.181?\nThanks for helping me!",
      "votes": null
    },
    {
      "id": "251407",
      "postDate": "12/01/2017 06:01:13",
      "content": "<p>_300, _450 - are image sizes. \n0.181 on training set means on training set. On each CV cycle I do predictions both for train and valid (which is 1 part of original train) sets. So after the whole CV process I have predictions for the whole train set. So I can measure this error based on train set and not on the leaderboard. </p>",
      "rawMarkdown": "_300, _450 - are image sizes. \n0.181 on training set means on training set. On each CV cycle I do predictions both for train and valid (which is 1 part of original train) sets. So after the whole CV process I have predictions for the whole train set. So I can measure this error based on train set and not on the leaderboard.",
      "votes": null
    },
    {
      "id": "251409",
      "postDate": "12/01/2017 06:06:18",
      "content": "<p>Thanks. But I guess extracting features from images with size 450 * 450 can be very time-consuming? Did you do this on gpu?</p>",
      "rawMarkdown": "Thanks. But I guess extracting features from images with size 450 * 450 can be very time-consuming? Did you do this on gpu?",
      "votes": null
    },
    {
      "id": "251470",
      "postDate": "12/01/2017 08:34:02",
      "content": "<p>Pretty much everything in image recognition is better to do on gpu. Take a look at aws p2 instances. You can rent one for $0.4-0.5 per hour. </p>",
      "rawMarkdown": "Pretty much everything in image recognition is better to do on gpu. Take a look at aws p2 instances. You can rent one for $0.4-0.5 per hour.",
      "votes": null
    },
    {
      "id": "251796",
      "postDate": "12/01/2017 18:17:59",
      "content": "<p>Thanks! Your post has helped me a lot!</p>",
      "rawMarkdown": "Thanks! Your post has helped me a lot!",
      "votes": null
    },
    {
      "id": "252257",
      "postDate": "12/02/2017 16:00:31",
      "content": "<p>Hi Vedder,</p>\n\n<p>Actually, not. I experimented with only Inception-v4 and cross-validation described above. To implement cross-validation, you can take a look at <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html\">StratifiedKFold</a> function in scikit-learn.</p>",
      "rawMarkdown": "Hi Vedder,\n\nActually, not. I experimented with only Inception-v4 and cross-validation described above. To implement cross-validation, you can take a look at [StratifiedKFold][1] function in scikit-learn.\n\n\n  [1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html",
      "votes": null
    },
    {
      "id": "252288",
      "postDate": "12/02/2017 17:18:10",
      "content": "<p>Awesome. Thanks again!</p>",
      "rawMarkdown": "Awesome. Thanks again!",
      "votes": null
    },
    {
      "id": "252569",
      "postDate": "12/03/2017 08:59:13",
      "content": "<p>Hi laroslav. It seems to me that InceptionV4 is not available in Keras.applications? How did you implement it?</p>",
      "rawMarkdown": "Hi laroslav. It seems to me that InceptionV4 is not available in Keras.applications? How did you implement it?",
      "votes": null
    },
    {
      "id": "252668",
      "postDate": "12/03/2017 14:04:47",
      "content": "<p>Hi Sheeyidre,</p>\n\n<p>I do not use Keras for deep learning but Pytorch. There is a very nice <a href=\"https://github.com/Cadene/pretrained-models.pytorch\">repository</a> with multiple pre-trained models for that framework.</p>",
      "rawMarkdown": "Hi Sheeyidre,\n\nI do not use Keras for deep learning but Pytorch. There is a very nice [repository][1] with multiple pre-trained models for that framework.\n\n\n  [1]: https://github.com/Cadene/pretrained-models.pytorch",
      "votes": null
    },
    {
      "id": "253499",
      "postDate": "12/05/2017 03:43:08",
      "content": "<p>Thanks! I'm not familiar with Pytorch, but I'll take a look though.</p>",
      "rawMarkdown": "Thanks! I'm not familiar with Pytorch, but I'll take a look though.",
      "votes": null
    },
    {
      "id": "258585",
      "postDate": "12/16/2017 14:09:12",
      "content": "<p>Excellent</p>",
      "rawMarkdown": "Excellent",
      "votes": null
    },
    {
      "id": "288427",
      "postDate": "02/27/2018 13:10:52",
      "content": "<p>Back to the original question: \nI believe it is averaging the predictions of the models listed and not about cross validation.</p>",
      "rawMarkdown": "Back to the original question: \nI believe it is averaging the predictions of the models listed and not about cross validation.",
      "votes": null
    },
    {
      "id": "288447",
      "postDate": "02/27/2018 14:09:03",
      "content": "<p>I have seen you finally achieved a 0.13987 score without using the Stanford dataset. \nCould you kindly share how did you manage to go from 0.172 to 0.139? \nThank you!</p>",
      "rawMarkdown": "I have seen you finally achieved a 0.13987 score without using the Stanford dataset. \nCould you kindly share how did you manage to go from 0.172 to 0.139? \nThank you!",
      "votes": null
    },
    {
      "id": "288453",
      "postDate": "02/27/2018 14:27:35",
      "content": "<p>NASNET model was very good</p>",
      "rawMarkdown": "NASNET model was very good",
      "votes": null
    },
    {
      "id": "288454",
      "postDate": "02/27/2018 14:29:17",
      "content": "<p>Thank you! And congrats for the results!</p>",
      "rawMarkdown": "Thank you! And congrats for the results!",
      "votes": null
    },
    {
      "id": "663940",
      "postDate": "11/02/2019 22:47:09",
      "content": "<p>Excuse me, How do I download images for comparison?</p>",
      "rawMarkdown": "Excuse me, How do I download images for comparison?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 244518,
      "author_name": "hanspinckaers",
      "author_url": "",
      "post_date": "11/16/2017 13:41:07",
      "content": "<p>Did you use any regularization when fine-tuning?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 244620,
      "author_name": "sermakarevich",
      "author_url": "",
      "post_date": "11/16/2017 16:39:39",
      "content": "<p>Only dropout. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 245488,
      "author_name": "yaby87",
      "author_url": "",
      "post_date": "11/18/2017 16:08:05",
      "content": "<p>Are those models pre-trained only on Imagenet?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 245489,
      "author_name": "sermakarevich",
      "author_url": "",
      "post_date": "11/18/2017 16:10:55",
      "content": "<p>Convolutional layers pre-trained on Imagenet, correct. FC layers tuned on train set. No cheating )</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 245506,
      "author_name": "yaby87",
      "author_url": "",
      "post_date": "11/18/2017 17:24:46",
      "content": "<p>I see :). So, if I understood correctly, you froze the convolutional weights and only fine-tuned FC layers?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 245744,
      "author_name": "sermakarevich",
      "author_url": "",
      "post_date": "11/19/2017 14:23:05",
      "content": "<p>Yes, thats right.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 245815,
      "author_name": "vedder",
      "author_url": "",
      "post_date": "11/19/2017 19:50:21",
      "content": "<p>Please forgive my ignorance, but what do you mean by averaging? Are you saving the weights for each model and then averaging over all of them? </p>",
      "votes": null,
      "replies": [
        {
          "id": 246080,
          "author_name": "yaby87",
          "author_url": "",
          "post_date": "11/20/2017 14:28:16",
          "content": "<p>Averaging is related to cross-validation (cv) technique. Basically, one can split training data into <code>n</code> subparts (folds), each fold is then used as a validation set while the <code>n-1</code> remaining folds form the training set. For instance, in case of 5-fold cross-validation we have 5 splits (0, 1, 2, 3, 4) of the training data. We choose the model (VGG, Inception, etc) and train it on 0-3 folds checking its performance on validation data (4th fold). The state (weights) of the model producing the smallest validation error is saved (model1). Then, we take another set of folds (let's assume 0, 1, 2, 4) to create new training and validation (3rd fold) datasets respectively and train our model from the same starting point as we used previously but on this new data. It will give us another model state (model2) which is optimal for this particular validation set. Following the same procedure, one can easily get 5 models. Using those models (states), we can predict probabilities on <code>test</code> data for each model independently, then summing predictions up and dividing by the number of models (folds) leads to the final result. So, this is where averaging comes on stage.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 248517,
          "author_name": "pierreguillou",
          "author_url": "",
          "post_date": "11/26/2017 10:29:30",
          "content": "<p>Hello, it looks like CV helps a lot (at least in this competition). Is there something to be done in all deep learning competitions ? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 251172,
          "author_name": "vedder",
          "author_url": "",
          "post_date": "11/30/2017 19:16:51",
          "content": "<p>Thanks Iaroslav! </p>\n\n<p>Do you run each model (VGG, Inception, etc) on each set of folds ((0,1,2,3), (0,2,3,4), etc) and pick the best model and parameters for each validation set?</p>\n\n<p>Do you know of any good resources for learning to implement this in python?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 252257,
          "author_name": "yaby87",
          "author_url": "",
          "post_date": "12/02/2017 16:00:31",
          "content": "<p>Hi Vedder,</p>\n\n<p>Actually, not. I experimented with only Inception-v4 and cross-validation described above. To implement cross-validation, you can take a look at <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html\">StratifiedKFold</a> function in scikit-learn.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 252288,
          "author_name": "vedder",
          "author_url": "",
          "post_date": "12/02/2017 17:18:10",
          "content": "<p>Awesome. Thanks again!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 252569,
          "author_name": "sheeyidre",
          "author_url": "",
          "post_date": "12/03/2017 08:59:13",
          "content": "<p>Hi laroslav. It seems to me that InceptionV4 is not available in Keras.applications? How did you implement it?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 252668,
          "author_name": "yaby87",
          "author_url": "",
          "post_date": "12/03/2017 14:04:47",
          "content": "<p>Hi Sheeyidre,</p>\n\n<p>I do not use Keras for deep learning but Pytorch. There is a very nice <a href=\"https://github.com/Cadene/pretrained-models.pytorch\">repository</a> with multiple pre-trained models for that framework.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 253499,
          "author_name": "sheeyidre",
          "author_url": "",
          "post_date": "12/05/2017 03:43:08",
          "content": "<p>Thanks! I'm not familiar with Pytorch, but I'll take a look though.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 288427,
          "author_name": "andraskuti",
          "author_url": "",
          "post_date": "02/27/2018 13:10:52",
          "content": "<p>Back to the original question: \nI believe it is averaging the predictions of the models listed and not about cross validation.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 251404,
      "author_name": "sheeyidre",
      "author_url": "",
      "post_date": "12/01/2017 05:54:59",
      "content": "<p>Hi, thanks a lot for your sharing. But I'm a very newbie of machine learning, and I have two questions that might be stupid. So I know there are Keras pretrained models InceptionV3 and InceptionResNetV2, but what do you mean by \"inception_4_300\" or \"inceptionresnet_2_450\"? And by \"0.181 on a training set\", do you mean you calculated the mean of the (five) validation set errors and it's 0.181?\nThanks for helping me!</p>",
      "votes": null,
      "replies": [
        {
          "id": 251407,
          "author_name": "sermakarevich",
          "author_url": "",
          "post_date": "12/01/2017 06:01:13",
          "content": "<p>_300, _450 - are image sizes. \n0.181 on training set means on training set. On each CV cycle I do predictions both for train and valid (which is 1 part of original train) sets. So after the whole CV process I have predictions for the whole train set. So I can measure this error based on train set and not on the leaderboard. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 251409,
          "author_name": "sheeyidre",
          "author_url": "",
          "post_date": "12/01/2017 06:06:18",
          "content": "<p>Thanks. But I guess extracting features from images with size 450 * 450 can be very time-consuming? Did you do this on gpu?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 251470,
          "author_name": "sermakarevich",
          "author_url": "",
          "post_date": "12/01/2017 08:34:02",
          "content": "<p>Pretty much everything in image recognition is better to do on gpu. Take a look at aws p2 instances. You can rent one for $0.4-0.5 per hour. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 251796,
          "author_name": "sheeyidre",
          "author_url": "",
          "post_date": "12/01/2017 18:17:59",
          "content": "<p>Thanks! Your post has helped me a lot!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 258585,
      "author_name": "classtag",
      "author_url": "",
      "post_date": "12/16/2017 14:09:12",
      "content": "<p>Excellent</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 288447,
      "author_name": "andraskuti",
      "author_url": "",
      "post_date": "02/27/2018 14:09:03",
      "content": "<p>I have seen you finally achieved a 0.13987 score without using the Stanford dataset. \nCould you kindly share how did you manage to go from 0.172 to 0.139? \nThank you!</p>",
      "votes": null,
      "replies": [
        {
          "id": 288453,
          "author_name": "sermakarevich",
          "author_url": "",
          "post_date": "02/27/2018 14:27:35",
          "content": "<p>NASNET model was very good</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 288454,
          "author_name": "andraskuti",
          "author_url": "",
          "post_date": "02/27/2018 14:29:17",
          "content": "<p>Thank you! And congrats for the results!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 663940,
      "author_name": "yizhen064",
      "author_url": "",
      "post_date": "11/02/2019 22:47:09",
      "content": "<p>Excuse me, How do I download images for comparison?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "242357": "Models log loss on a whole training set (acquired through 5 fold cv, between folds score might very from less than 0.17 to 0.26):\n\n* inception_4_300 \t 0.228\n* inception_4_350 \t 0.211\n* inception_4_400 \t 0.204\n* inception_4_450 \t 0.223\n* inceptionresnet_2_300 \t 0.239\n* inceptionresnet_2_350 \t 0.217\n* inceptionresnet_2_400 \t 0.215\n* inceptionresnet_2_450 \t 0.222\n\nSimple averaging all models except for inceptionresnet_2_300, inception_4_450 gives 0.181 on a training set and 0.172 on a leaderboard.",
    "244518": "Did you use any regularization when fine-tuning?",
    "244620": "Only dropout.",
    "245488": "Are those models pre-trained only on Imagenet?",
    "245489": "Convolutional layers pre-trained on Imagenet, correct. FC layers tuned on train set. No cheating )",
    "245506": "I see :). So, if I understood correctly, you froze the convolutional weights and only fine-tuned FC layers?",
    "245744": "Yes, thats right.",
    "245815": "Please forgive my ignorance, but what do you mean by averaging? Are you saving the weights for each model and then averaging over all of them?",
    "246080": "Averaging is related to cross-validation (cv) technique. Basically, one can split training data into `n` subparts (folds), each fold is then used as a validation set while the `n-1` remaining folds form the training set. For instance, in case of 5-fold cross-validation we have 5 splits (0, 1, 2, 3, 4) of the training data. We choose the model (VGG, Inception, etc) and train it on 0-3 folds checking its performance on validation data (4th fold). The state (weights) of the model producing the smallest validation error is saved (model1). Then, we take another set of folds (let's assume 0, 1, 2, 4) to create new training and validation (3rd fold) datasets respectively and train our model from the same starting point as we used previously but on this new data. It will give us another model state (model2) which is optimal for this particular validation set. Following the same procedure, one can easily get 5 models. Using those models (states), we can predict probabilities on `test` data for each model independently, then summing predictions up and dividing by the number of models (folds) leads to the final result. So, this is where averaging comes on stage.",
    "248517": "Hello, it looks like CV helps a lot (at least in this competition). Is there something to be done in all deep learning competitions ?",
    "251172": "Thanks Iaroslav! \n\nDo you run each model (VGG, Inception, etc) on each set of folds ((0,1,2,3), (0,2,3,4), etc) and pick the best model and parameters for each validation set?\n\nDo you know of any good resources for learning to implement this in python?",
    "251404": "Hi, thanks a lot for your sharing. But I'm a very newbie of machine learning, and I have two questions that might be stupid. So I know there are Keras pretrained models InceptionV3 and InceptionResNetV2, but what do you mean by \"inception_4_300\" or \"inceptionresnet_2_450\"? And by \"0.181 on a training set\", do you mean you calculated the mean of the (five) validation set errors and it's 0.181?\nThanks for helping me!",
    "251407": "_300, _450 - are image sizes. \n0.181 on training set means on training set. On each CV cycle I do predictions both for train and valid (which is 1 part of original train) sets. So after the whole CV process I have predictions for the whole train set. So I can measure this error based on train set and not on the leaderboard.",
    "251409": "Thanks. But I guess extracting features from images with size 450 * 450 can be very time-consuming? Did you do this on gpu?",
    "251470": "Pretty much everything in image recognition is better to do on gpu. Take a look at aws p2 instances. You can rent one for $0.4-0.5 per hour.",
    "251796": "Thanks! Your post has helped me a lot!",
    "252257": "Hi Vedder,\n\nActually, not. I experimented with only Inception-v4 and cross-validation described above. To implement cross-validation, you can take a look at [StratifiedKFold][1] function in scikit-learn.\n\n\n  [1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html",
    "252288": "Awesome. Thanks again!",
    "252569": "Hi laroslav. It seems to me that InceptionV4 is not available in Keras.applications? How did you implement it?",
    "252668": "Hi Sheeyidre,\n\nI do not use Keras for deep learning but Pytorch. There is a very nice [repository][1] with multiple pre-trained models for that framework.\n\n\n  [1]: https://github.com/Cadene/pretrained-models.pytorch",
    "253499": "Thanks! I'm not familiar with Pytorch, but I'll take a look though.",
    "258585": "Excellent",
    "288427": "Back to the original question: \nI believe it is averaging the predictions of the models listed and not about cross validation.",
    "288447": "I have seen you finally achieved a 0.13987 score without using the Stanford dataset. \nCould you kindly share how did you manage to go from 0.172 to 0.139? \nThank you!",
    "288453": "NASNET model was very good",
    "288454": "Thank you! And congrats for the results!",
    "663940": "Excuse me, How do I download images for comparison?"
  },
  "source": "meta"
}