{
  "id": 44537,
  "title": "can fine-tuning( in the fine-tuning strategy all weights are changed when training on the new task) increase the accuracy of classification?",
  "url": "/competitions/dog-breed-identification/discussion/44537",
  "author_name": "",
  "post_date": "2017-11-29T23:57:50.418779900Z",
  "votes": 5,
  "comment_count": 5,
  "views": 0,
  "content": "<p>as you see in this competition the first man in the leader board have earned a very low loss in the , actually it has very high accuracy therefore, it reached to my mind that, maybe fine tuning is one of this reason,  please share your idea here. </p>",
  "messages": [
    {
      "id": "250227",
      "postDate": "11/29/2017 23:57:50",
      "content": "<p>as you see in this competition the first man in the leader board have earned a very low loss in the , actually it has very high accuracy therefore, it reached to my mind that, maybe fine tuning is one of this reason,  please share your idea here. </p>",
      "rawMarkdown": "as you see in this competition the first man in the leader board have earned a very low loss in the , actually it has very high accuracy therefore, it reached to my mind that, maybe fine tuning is one of this reason,  please share your idea here.",
      "votes": null
    },
    {
      "id": "250229",
      "postDate": "11/30/2017 00:00:10",
      "content": "<p>it is probable, because we have low number of instance for each class.</p>",
      "rawMarkdown": "it is probable, because we have low number of instance for each class.",
      "votes": null
    },
    {
      "id": "262349",
      "postDate": "12/26/2017 09:49:06",
      "content": "<p>I think you may want to check his code on his github (<a href=\"https://github.com/ypwhs\">https://github.com/ypwhs</a>).\nAs for your question, I think fine tuning is very import in this case. I have two reasons to share with you and some of my experiment results.</p>\n\n<ol>\n<li>Small number of training data (even we do data augmentation) cannot support the big model used in ImageNet (for instance Densenet 121)</li>\n<li>Fine tuning is very efficient for task like this. It is easy to understand this intuitively. As we know, the last layers of CNN are used mostly for choosing features (which feature should have higher weight). If we freeze very front layers in the CNN, it tells CNN to extract the same features (like eyes, tails, etc.) but it requires CNN to rank importance of these features (for example, in this case, dogs' eyes are more important than dogs' tails. Not very precise but you get the main idea).</li>\n</ol>\n\n<p>And I do some experiments to support my claim.\nIn Densenet 121, if we do transfer learning from imagenet and don't freeze any of the layers, we have the log loss: 1.6.\nHowever, if we freeze the some layers and only update fourth of all the parameters, the log loss is about 1.0.</p>",
      "rawMarkdown": "I think you may want to check his code on his github (https://github.com/ypwhs).\nAs for your question, I think fine tuning is very import in this case. I have two reasons to share with you and some of my experiment results.\n\n 1. Small number of training data (even we do data augmentation) cannot support the big model used in ImageNet (for instance Densenet 121)\n 2. Fine tuning is very efficient for task like this. It is easy to understand this intuitively. As we know, the last layers of CNN are used mostly for choosing features (which feature should have higher weight). If we freeze very front layers in the CNN, it tells CNN to extract the same features (like eyes, tails, etc.) but it requires CNN to rank importance of these features (for example, in this case, dogs' eyes are more important than dogs' tails. Not very precise but you get the main idea).\n\nAnd I do some experiments to support my claim.\nIn Densenet 121, if we do transfer learning from imagenet and don't freeze any of the layers, we have the log loss: 1.6.\nHowever, if we freeze the some layers and only update fourth of all the parameters, the log loss is about 1.0.",
      "votes": null
    },
    {
      "id": "262623",
      "postDate": "12/27/2017 10:02:25",
      "content": "<p>thank you so much for sharing of experiment here! actually, in your view, if we implementing fine tuning  with data augmentation, you mean that i  we will have good performance rather than using the fine tuning without data augmentation?</p>",
      "rawMarkdown": "thank you so much for sharing of experiment here! actually, in your view, if we implementing fine tuning  with data augmentation, you mean that i  we will have good performance rather than using the fine tuning without data augmentation?",
      "votes": null
    },
    {
      "id": "262831",
      "postDate": "12/28/2017 03:18:26",
      "content": "<p>In my opinion, whether data augmentation would affect the results of fine tuning depends on which layer we choose to freeze.\nData augmentation doesn't provide any additional information to the CNN but it forces our algorithms to catch features that are irrelevant to the position, angle, etc. (since we force the CNN to catch similar features between to a left-direction cat or a right-direction cat)\nAs a result, if we freeze all the CNN layers and only update parameters in the Fully-Connected layers, I don't think data augmentation would help much (changing the FC layers doesn't force the networks to learn similar things between same image in different direction, FC layers are used to combine all these features with matrix).\nHowever, if we still leave some CNN layers' parameters to update, I think data augmentation actually helps.</p>",
      "rawMarkdown": "In my opinion, whether data augmentation would affect the results of fine tuning depends on which layer we choose to freeze.\nData augmentation doesn't provide any additional information to the CNN but it forces our algorithms to catch features that are irrelevant to the position, angle, etc. (since we force the CNN to catch similar features between to a left-direction cat or a right-direction cat)\nAs a result, if we freeze all the CNN layers and only update parameters in the Fully-Connected layers, I don't think data augmentation would help much (changing the FC layers doesn't force the networks to learn similar things between same image in different direction, FC layers are used to combine all these features with matrix).\nHowever, if we still leave some CNN layers' parameters to update, I think data augmentation actually helps.",
      "votes": null
    },
    {
      "id": "276332",
      "postDate": "01/31/2018 08:27:40",
      "content": "<p>Thank you</p>",
      "rawMarkdown": "Thank you",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 250229,
      "author_name": "beyoghlu",
      "author_url": "",
      "post_date": "11/30/2017 00:00:10",
      "content": "<p>it is probable, because we have low number of instance for each class.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 262349,
      "author_name": "wangxiao1105",
      "author_url": "",
      "post_date": "12/26/2017 09:49:06",
      "content": "<p>I think you may want to check his code on his github (<a href=\"https://github.com/ypwhs\">https://github.com/ypwhs</a>).\nAs for your question, I think fine tuning is very import in this case. I have two reasons to share with you and some of my experiment results.</p>\n\n<ol>\n<li>Small number of training data (even we do data augmentation) cannot support the big model used in ImageNet (for instance Densenet 121)</li>\n<li>Fine tuning is very efficient for task like this. It is easy to understand this intuitively. As we know, the last layers of CNN are used mostly for choosing features (which feature should have higher weight). If we freeze very front layers in the CNN, it tells CNN to extract the same features (like eyes, tails, etc.) but it requires CNN to rank importance of these features (for example, in this case, dogs' eyes are more important than dogs' tails. Not very precise but you get the main idea).</li>\n</ol>\n\n<p>And I do some experiments to support my claim.\nIn Densenet 121, if we do transfer learning from imagenet and don't freeze any of the layers, we have the log loss: 1.6.\nHowever, if we freeze the some layers and only update fourth of all the parameters, the log loss is about 1.0.</p>",
      "votes": null,
      "replies": [
        {
          "id": 262623,
          "author_name": "ayanzadeh93",
          "author_url": "",
          "post_date": "12/27/2017 10:02:25",
          "content": "<p>thank you so much for sharing of experiment here! actually, in your view, if we implementing fine tuning  with data augmentation, you mean that i  we will have good performance rather than using the fine tuning without data augmentation?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 262831,
          "author_name": "wangxiao1105",
          "author_url": "",
          "post_date": "12/28/2017 03:18:26",
          "content": "<p>In my opinion, whether data augmentation would affect the results of fine tuning depends on which layer we choose to freeze.\nData augmentation doesn't provide any additional information to the CNN but it forces our algorithms to catch features that are irrelevant to the position, angle, etc. (since we force the CNN to catch similar features between to a left-direction cat or a right-direction cat)\nAs a result, if we freeze all the CNN layers and only update parameters in the Fully-Connected layers, I don't think data augmentation would help much (changing the FC layers doesn't force the networks to learn similar things between same image in different direction, FC layers are used to combine all these features with matrix).\nHowever, if we still leave some CNN layers' parameters to update, I think data augmentation actually helps.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 276332,
          "author_name": "airqin",
          "author_url": "",
          "post_date": "01/31/2018 08:27:40",
          "content": "<p>Thank you</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "250227": "as you see in this competition the first man in the leader board have earned a very low loss in the , actually it has very high accuracy therefore, it reached to my mind that, maybe fine tuning is one of this reason,  please share your idea here.",
    "250229": "it is probable, because we have low number of instance for each class.",
    "262349": "I think you may want to check his code on his github (https://github.com/ypwhs).\nAs for your question, I think fine tuning is very import in this case. I have two reasons to share with you and some of my experiment results.\n\n 1. Small number of training data (even we do data augmentation) cannot support the big model used in ImageNet (for instance Densenet 121)\n 2. Fine tuning is very efficient for task like this. It is easy to understand this intuitively. As we know, the last layers of CNN are used mostly for choosing features (which feature should have higher weight). If we freeze very front layers in the CNN, it tells CNN to extract the same features (like eyes, tails, etc.) but it requires CNN to rank importance of these features (for example, in this case, dogs' eyes are more important than dogs' tails. Not very precise but you get the main idea).\n\nAnd I do some experiments to support my claim.\nIn Densenet 121, if we do transfer learning from imagenet and don't freeze any of the layers, we have the log loss: 1.6.\nHowever, if we freeze the some layers and only update fourth of all the parameters, the log loss is about 1.0.",
    "262623": "thank you so much for sharing of experiment here! actually, in your view, if we implementing fine tuning  with data augmentation, you mean that i  we will have good performance rather than using the fine tuning without data augmentation?",
    "262831": "In my opinion, whether data augmentation would affect the results of fine tuning depends on which layer we choose to freeze.\nData augmentation doesn't provide any additional information to the CNN but it forces our algorithms to catch features that are irrelevant to the position, angle, etc. (since we force the CNN to catch similar features between to a left-direction cat or a right-direction cat)\nAs a result, if we freeze all the CNN layers and only update parameters in the Fully-Connected layers, I don't think data augmentation would help much (changing the FC layers doesn't force the networks to learn similar things between same image in different direction, FC layers are used to combine all these features with matrix).\nHowever, if we still leave some CNN layers' parameters to update, I think data augmentation actually helps.",
    "276332": "Thank you"
  },
  "source": "meta"
}