{
  "id": 12709,
  "title": "Tuning CNN for data augmentation",
  "url": "/competitions/datasciencebowl/discussion/12709",
  "author_name": "",
  "post_date": "2015-03-06T10:32:36.153Z",
  "votes": 2,
  "comment_count": 3,
  "views": 2873,
  "content": "<p>Hi, guys. As I described in a previous post,&nbsp;I deploy a CNN with Nolearn + Lasagne with a single GPU in some cloud computing platform. I left 10% training data as my validation data using sklearn.train_test_split. And I get my validation accuracy nearly 70% within an hour. I submit this result and get a roughly 1.2 logloss. Now I begin to consider data augmentation and expect such strategy could gain me a boost of validation accuracy. However, this is what I get.&nbsp;</p>\n<p>http://imgur.com/y7F723Z</p>\n<p>Note that the validation loss/accuracy continues to decrease instead of increasing as we &nbsp;expected. My first action is to turn down the learning rate. However it is still going down, only more slowly. Could anyone give some advice ? Thank you !</p>",
  "messages": [
    {
      "id": "65587",
      "postDate": "03/06/2015 10:32:36",
      "content": "<p>Hi, guys. As I described in a previous post,&nbsp;I deploy a CNN with Nolearn + Lasagne with a single GPU in some cloud computing platform. I left 10% training data as my validation data using sklearn.train_test_split. And I get my validation accuracy nearly 70% within an hour. I submit this result and get a roughly 1.2 logloss. Now I begin to consider data augmentation and expect such strategy could gain me a boost of validation accuracy. However, this is what I get.&nbsp;</p>\n<p>http://imgur.com/y7F723Z</p>\n<p>Note that the validation loss/accuracy continues to decrease instead of increasing as we &nbsp;expected. My first action is to turn down the learning rate. However it is still going down, only more slowly. Could anyone give some advice ? Thank you !</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65589",
      "postDate": "03/06/2015 11:22:54",
      "content": "<p>The standard strategy is to train with data augmentation from the beginning, not after model converges. And even turn it off on the last steps as suggested Dr. Graham&nbsp;http://blog.kaggle.com/2015/01/02/cifar-10-competition-winners-interviews-with-dr-ben-graham-phil-culliton-zygmunt-zajac/ :</p>\n<p><em>Q:Anything of note on the competition and/or dataset that you found surprising? An approach that worked unexpectedly well, or perhaps did not work for you?</em></p>\n<p><em>A: I was very surprised how much of a difference fine-tuning (finishing off training the network using a small number of training epochs with a low learning rate and without data augmentation) made.</em></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65596",
      "postDate": "03/06/2015 11:47:39",
      "content": "<p>@ old-ufo, thank you for your reply. And I have noticed that you are already been top 10 for this competition. Wa! Another experiment that I have made is that I only train the the network augmented data(rotate 90 degrees) from scratch. As a result, it performs pretty bad on the same validation data as low as 20%. So, what's wrong with that?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65597",
      "postDate": "03/06/2015 11:56:19",
      "content": "<p>I don`t know what is wrong :) &nbsp;I recommend you to perform experiments with data aug on small network like LeNet which converges in 10-15 minutes to find best data augmentation strategy. And read topic&nbsp;https://www.kaggle.com/c/datasciencebowl/forums/t/11360/data-augmentation by long-time top-1, now top-2&nbsp;xudong cao.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 65589,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/06/2015 11:22:54",
      "content": "<p>The standard strategy is to train with data augmentation from the beginning, not after model converges. And even turn it off on the last steps as suggested Dr. Graham&nbsp;http://blog.kaggle.com/2015/01/02/cifar-10-competition-winners-interviews-with-dr-ben-graham-phil-culliton-zygmunt-zajac/ :</p>\n<p><em>Q:Anything of note on the competition and/or dataset that you found surprising? An approach that worked unexpectedly well, or perhaps did not work for you?</em></p>\n<p><em>A: I was very surprised how much of a difference fine-tuning (finishing off training the network using a small number of training epochs with a low learning rate and without data augmentation) made.</em></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65596,
      "author_name": "pengpai",
      "author_url": "",
      "post_date": "03/06/2015 11:47:39",
      "content": "<p>@ old-ufo, thank you for your reply. And I have noticed that you are already been top 10 for this competition. Wa! Another experiment that I have made is that I only train the the network augmented data(rotate 90 degrees) from scratch. As a result, it performs pretty bad on the same validation data as low as 20%. So, what's wrong with that?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65597,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/06/2015 11:56:19",
      "content": "<p>I don`t know what is wrong :) &nbsp;I recommend you to perform experiments with data aug on small network like LeNet which converges in 10-15 minutes to find best data augmentation strategy. And read topic&nbsp;https://www.kaggle.com/c/datasciencebowl/forums/t/11360/data-augmentation by long-time top-1, now top-2&nbsp;xudong cao.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "65587": "",
    "65589": "",
    "65596": "",
    "65597": ""
  },
  "source": "meta"
}