{
  "id": 50685,
  "title": "Sharing lessons what I've learned",
  "url": "/competitions/acoustic-scene-2018/discussion/50685",
  "author_name": "daisukelab",
  "post_date": "2018-02-28T04:51:48.690000",
  "votes": 14,
  "comment_count": 10,
  "views": 0,
  "content": "<p>First, thank you for sharing opportunity to participate in this competition.</p>\n\n<p>I'm a 20+ year experienced software engineer, very recently pivoted to focus on machine learning.\nSince my career have started with sound engineering tool for digital music, I set my goal to handle sound with powerful ML technology.</p>\n\n<p>Now it's almost time to finish this challenge, then I feel I should share lessons what I've learned from this challenge.</p>\n\n<p>Basically I have created some models which describe and learn problem from different perspective, then ensembled models as many Kagglers do in other challenges.\nMy attempts for various combination between data preprocessings/ models/ learning methods exceeded 20, tried about 10 models.\nMy score started from awful 0.41 or so by making mistake on pre-processing test set, started over with 0.71. And then current final score is 0.84, it's about 13% improvement after all.</p>\n\n<h2>Lessons I've learnied</h2>\n\n<ol>\n<li>Ensemble different models really works, 10+% effective.</li>\n<li>Model design makes -10+% to +3% difference. I saw very bad models if I started from scratch, or if I choose famous model like AlexNet, it've shown good results. Then if I tuned based on successful model, it improved about 3% maximum. After repeating try and error as well as carefully designing layers to match to the problem, my almost-scratch models started working better.</li>\n<li>Preprocessing by domain knowledge to get different perspective enabled ensemble improvement. Especially by taking mean by time, extracting sound events, or simply handling whole data as is.</li>\n<li>Weighted ensemble, model predictions are weight-averaged for each row &amp; class. And this weight optimization was done by scipy.optimize.minimize though I used 'Nelder-Mead' instead of 'SLSQP' as <a href=\"https://www.kaggle.com/hsperr/finding-ensamble-weights\">commented here</a>.</li>\n<li>Augmentation matters, I used mainly ImageDataGenerator of Keras, as well as mixup &amp; cutout/random erasing as described in my <a href=\"https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment\">kernel</a>.</li>\n<li>Learning rate and methods affects results also so much. <a href=\"https://github.com/bckenstler/CLR\">Cyclic Learning Rate</a>'d helped finding better minimum, comparing to fixed learning rate. Then lowering LR to get fine-tuned final result, it worked.</li>\n<li>Final model should learn entire training set if you divide original train set to train/valid set. After I got confidence for a model, I switch to use entire training set and did the same training flow, to get final results.</li>\n</ol>\n\n<h2>Kernels shared</h2>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/daisukelab/simple-cnn-approach\">Simple CNN approach</a></li>\n<li><a href=\"https://www.kaggle.com/daisukelab/sound-event-based-approach\">Sound event based approach</a></li>\n<li><a href=\"https://www.kaggle.com/daisukelab/time-wise-mean-approach\">Time-wise mean approach</a></li>\n<li><a href=\"https://www.kaggle.com/daisukelab/example-of-weighted-ensemble\">Example of weighted ensemble</a></li>\n<li>Regarding augmentation, <a href=\"https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment\">mixup &amp; cutout/random erasing</a></li>\n</ul>\n\n<p>I thank competition organizers very much, it is great opportunity and I could learn really a lot!</p>",
  "messages": [
    {
      "id": 288724,
      "postDate": "2018-02-28T04:51:48.690Z",
      "content": "<p>First, thank you for sharing opportunity to participate in this competition.</p>\n\n<p>I'm a 20+ year experienced software engineer, very recently pivoted to focus on machine learning.\nSince my career have started with sound engineering tool for digital music, I set my goal to handle sound with powerful ML technology.</p>\n\n<p>Now it's almost time to finish this challenge, then I feel I should share lessons what I've learned from this challenge.</p>\n\n<p>Basically I have created some models which describe and learn problem from different perspective, then ensembled models as many Kagglers do in other challenges.\nMy attempts for various combination between data preprocessings/ models/ learning methods exceeded 20, tried about 10 models.\nMy score started from awful 0.41 or so by making mistake on pre-processing test set, started over with 0.71. And then current final score is 0.84, it's about 13% improvement after all.</p>\n\n<h2>Lessons I've learnied</h2>\n\n<ol>\n<li>Ensemble different models really works, 10+% effective.</li>\n<li>Model design makes -10+% to +3% difference. I saw very bad models if I started from scratch, or if I choose famous model like AlexNet, it've shown good results. Then if I tuned based on successful model, it improved about 3% maximum. After repeating try and error as well as carefully designing layers to match to the problem, my almost-scratch models started working better.</li>\n<li>Preprocessing by domain knowledge to get different perspective enabled ensemble improvement. Especially by taking mean by time, extracting sound events, or simply handling whole data as is.</li>\n<li>Weighted ensemble, model predictions are weight-averaged for each row &amp; class. And this weight optimization was done by scipy.optimize.minimize though I used 'Nelder-Mead' instead of 'SLSQP' as <a href=\"https://www.kaggle.com/hsperr/finding-ensamble-weights\">commented here</a>.</li>\n<li>Augmentation matters, I used mainly ImageDataGenerator of Keras, as well as mixup &amp; cutout/random erasing as described in my <a href=\"https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment\">kernel</a>.</li>\n<li>Learning rate and methods affects results also so much. <a href=\"https://github.com/bckenstler/CLR\">Cyclic Learning Rate</a>'d helped finding better minimum, comparing to fixed learning rate. Then lowering LR to get fine-tuned final result, it worked.</li>\n<li>Final model should learn entire training set if you divide original train set to train/valid set. After I got confidence for a model, I switch to use entire training set and did the same training flow, to get final results.</li>\n</ol>\n\n<h2>Kernels shared</h2>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/daisukelab/simple-cnn-approach\">Simple CNN approach</a></li>\n<li><a href=\"https://www.kaggle.com/daisukelab/sound-event-based-approach\">Sound event based approach</a></li>\n<li><a href=\"https://www.kaggle.com/daisukelab/time-wise-mean-approach\">Time-wise mean approach</a></li>\n<li><a href=\"https://www.kaggle.com/daisukelab/example-of-weighted-ensemble\">Example of weighted ensemble</a></li>\n<li>Regarding augmentation, <a href=\"https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment\">mixup &amp; cutout/random erasing</a></li>\n</ul>\n\n<p>I thank competition organizers very much, it is great opportunity and I could learn really a lot!</p>",
      "rawMarkdown": "First, thank you for sharing opportunity to participate in this competition.\n\nI'm a 20+ year experienced software engineer, very recently pivoted to focus on machine learning.\nSince my career have started with sound engineering tool for digital music, I set my goal to handle sound with powerful ML technology.\n\nNow it's almost time to finish this challenge, then I feel I should share lessons what I've learned from this challenge.\n\nBasically I have created some models which describe and learn problem from different perspective, then ensembled models as many Kagglers do in other challenges.\nMy attempts for various combination between data preprocessings/ models/ learning methods exceeded 20, tried about 10 models.\nMy score started from awful 0.41 or so by making mistake on pre-processing test set, started over with 0.71. And then current final score is 0.84, it's about 13% improvement after all.\n\n## Lessons I've learnied\n1. Ensemble different models really works, 10+% effective.\n2. Model design makes -10+% to +3% difference. I saw very bad models if I started from scratch, or if I choose famous model like AlexNet, it've shown good results. Then if I tuned based on successful model, it improved about 3% maximum. After repeating try and error as well as carefully designing layers to match to the problem, my almost-scratch models started working better.\n3. Preprocessing by domain knowledge to get different perspective enabled ensemble improvement. Especially by taking mean by time, extracting sound events, or simply handling whole data as is.\n4. Weighted ensemble, model predictions are weight-averaged for each row &amp; class. And this weight optimization was done by scipy.optimize.minimize though I used 'Nelder-Mead' instead of 'SLSQP' as [commented here](https://www.kaggle.com/hsperr/finding-ensamble-weights).\n5. Augmentation matters, I used mainly ImageDataGenerator of Keras, as well as mixup &amp; cutout/random erasing as described in my [kernel](https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment).\n6. Learning rate and methods affects results also so much. [Cyclic Learning Rate](https://github.com/bckenstler/CLR)'d helped finding better minimum, comparing to fixed learning rate. Then lowering LR to get fine-tuned final result, it worked.\n7. Final model should learn entire training set if you divide original train set to train/valid set. After I got confidence for a model, I switch to use entire training set and did the same training flow, to get final results.\n\n## Kernels shared\n- [Simple CNN approach](https://www.kaggle.com/daisukelab/simple-cnn-approach)\n- [Sound event based approach](https://www.kaggle.com/daisukelab/sound-event-based-approach)\n- [Time-wise mean approach](https://www.kaggle.com/daisukelab/time-wise-mean-approach)\n- [Example of weighted ensemble](https://www.kaggle.com/daisukelab/example-of-weighted-ensemble)\n- Regarding augmentation, [mixup &amp; cutout/random erasing](https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment)\n\nI thank competition organizers very much, it is great opportunity and I could learn really a lot!",
      "votes": 14
    },
    {
      "id": 289636,
      "postDate": "2018-03-02T01:00:19.800Z",
      "content": "<p>Yes I'm going to show some of my code after competition is over.\nOne would be entire code to summarize in kernel, please let me know if I should not be sharing one of working code entirely.</p>",
      "rawMarkdown": "Yes I'm going to show some of my code after competition is over.\nOne would be entire code to summarize in kernel, please let me know if I should not be sharing one of working code entirely.",
      "votes": 1,
      "replies": [
        {
          "id": 289785,
          "postDate": "2018-03-02T09:03:36.267Z",
          "content": "<p>Sure, kernel submission would be very welcome. Everyone is eagerly waiting to hear your secrets!</p>\n\n<p>And, congratulations for winning this competition! </p>\n\n<p>Heikki</p>",
          "rawMarkdown": "Sure, kernel submission would be very welcome. Everyone is eagerly waiting to hear your secrets!\n\nAnd, congratulations for winning this competition! \n\nHeikki",
          "votes": 1
        },
        {
          "id": 289895,
          "postDate": "2018-03-02T15:12:02.600Z",
          "content": "<p>Thank you, I opened 3 kernels that are basically the ones ensembled for the final submission.\nI'm planning to make some more comments with code Sunday night here (in Japan).</p>",
          "rawMarkdown": "Thank you, I opened 3 kernels that are basically the ones ensembled for the final submission.\nI'm planning to make some more comments with code Sunday night here (in Japan)."
        },
        {
          "id": 290625,
          "postDate": "2018-03-04T15:22:45.450Z",
          "content": "<p>I have updated regarding kernels, though most of them are still running.</p>",
          "rawMarkdown": "I have updated regarding kernels, though most of them are still running."
        }
      ]
    },
    {
      "id": 288774,
      "postDate": "2018-02-28T06:35:55.397Z",
      "content": "<p>Thank you for very insightful comments! I'm sure many participants were wondering which kind of techniques you were using. During the course we discussed model averaging so I think many will attempt to use that (and other things you mention) during the last two days.</p>\n\n<p>I hope to hear even more detailed description when the competition is ober.</p>\n\n<p>Heikki</p>",
      "rawMarkdown": "Thank you for very insightful comments! I'm sure many participants were wondering which kind of techniques you were using. During the course we discussed model averaging so I think many will attempt to use that (and other things you mention) during the last two days.\n\nI hope to hear even more detailed description when the competition is ober.\n\nHeikki\n",
      "votes": 1
    },
    {
      "id": 294633,
      "postDate": "2018-03-12T10:14:22.310Z",
      "content": "<p>Hi all, I have uploaded all the kernels.\nAnd the ensemble kernel was not correctly opened to public.\nExcuse me if you have tried, now it should be fine.</p>\n\n<p>@Heikki This is all from me. Thanks again!</p>",
      "rawMarkdown": "Hi all, I have uploaded all the kernels.\nAnd the ensemble kernel was not correctly opened to public.\nExcuse me if you have tried, now it should be fine.\n\n@Heikki This is all from me. Thanks again!",
      "replies": [
        {
          "id": 294771,
          "postDate": "2018-03-12T16:06:52.283Z",
          "content": "<p>Thank you @daisukelab. Very informative and reproducible.</p>\n\n<p>We are considering a conference paper on the results. Would you like to join as an author? We would do all the writing but would ask if anything in your method is unclear.</p>\n\n<p>Similar to <a href=\"http://www.cs.tut.fi/~hehu/HeadPose.pdf\">this one</a></p>",
          "rawMarkdown": "Thank you @daisukelab. Very informative and reproducible.\n\nWe are considering a conference paper on the results. Would you like to join as an author? We would do all the writing but would ask if anything in your method is unclear.\n\nSimilar to [this one][1]\n\n\n  [1]: http://www.cs.tut.fi/~hehu/HeadPose.pdf"
        },
        {
          "id": 295228,
          "postDate": "2018-03-13T10:08:53.020Z",
          "content": "<p>Thank you for your offer, of course I'm OK, please proceed.\nI will also try to make my things clear.</p>",
          "rawMarkdown": "Thank you for your offer, of course I'm OK, please proceed.\nI will also try to make my things clear."
        },
        {
          "id": 310760,
          "postDate": "2018-04-08T14:20:21.117Z",
          "content": "<p>Hi @Heikki, I sent you an email regarding my info. Thanks.</p>",
          "rawMarkdown": "Hi @Heikki, I sent you an email regarding my info. Thanks."
        }
      ]
    },
    {
      "id": 290263,
      "postDate": "2018-03-03T14:40:32.297Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 289636,
      "author_name": "daisukelab",
      "author_url": "",
      "post_date": "2018-03-02T01:00:19.800000",
      "content": "<p>Yes I'm going to show some of my code after competition is over.\nOne would be entire code to summarize in kernel, please let me know if I should not be sharing one of working code entirely.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 289785,
          "author_name": "Heikki Huttunen",
          "author_url": "",
          "post_date": "2018-03-02T09:03:36.267000",
          "content": "<p>Sure, kernel submission would be very welcome. Everyone is eagerly waiting to hear your secrets!</p>\n\n<p>And, congratulations for winning this competition! </p>\n\n<p>Heikki</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 289895,
          "author_name": "daisukelab",
          "author_url": "",
          "post_date": "2018-03-02T15:12:02.600000",
          "content": "<p>Thank you, I opened 3 kernels that are basically the ones ensembled for the final submission.\nI'm planning to make some more comments with code Sunday night here (in Japan).</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 290625,
          "author_name": "daisukelab",
          "author_url": "",
          "post_date": "2018-03-04T15:22:45.450000",
          "content": "<p>I have updated regarding kernels, though most of them are still running.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 288774,
      "author_name": "Heikki Huttunen",
      "author_url": "",
      "post_date": "2018-02-28T06:35:55.397000",
      "content": "<p>Thank you for very insightful comments! I'm sure many participants were wondering which kind of techniques you were using. During the course we discussed model averaging so I think many will attempt to use that (and other things you mention) during the last two days.</p>\n\n<p>I hope to hear even more detailed description when the competition is ober.</p>\n\n<p>Heikki</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 294633,
      "author_name": "daisukelab",
      "author_url": "",
      "post_date": "2018-03-12T10:14:22.310000",
      "content": "<p>Hi all, I have uploaded all the kernels.\nAnd the ensemble kernel was not correctly opened to public.\nExcuse me if you have tried, now it should be fine.</p>\n\n<p>@Heikki This is all from me. Thanks again!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 294771,
          "author_name": "Heikki Huttunen",
          "author_url": "",
          "post_date": "2018-03-12T16:06:52.283000",
          "content": "<p>Thank you @daisukelab. Very informative and reproducible.</p>\n\n<p>We are considering a conference paper on the results. Would you like to join as an author? We would do all the writing but would ask if anything in your method is unclear.</p>\n\n<p>Similar to <a href=\"http://www.cs.tut.fi/~hehu/HeadPose.pdf\">this one</a></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 295228,
          "author_name": "daisukelab",
          "author_url": "",
          "post_date": "2018-03-13T10:08:53.020000",
          "content": "<p>Thank you for your offer, of course I'm OK, please proceed.\nI will also try to make my things clear.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 310760,
          "author_name": "daisukelab",
          "author_url": "",
          "post_date": "2018-04-08T14:20:21.117000",
          "content": "<p>Hi @Heikki, I sent you an email regarding my info. Thanks.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 290263,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-03-03T14:40:32.297000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "288724": "First, thank you for sharing opportunity to participate in this competition.\n\nI'm a 20+ year experienced software engineer, very recently pivoted to focus on machine learning.\nSince my career have started with sound engineering tool for digital music, I set my goal to handle sound with powerful ML technology.\n\nNow it's almost time to finish this challenge, then I feel I should share lessons what I've learned from this challenge.\n\nBasically I have created some models which describe and learn problem from different perspective, then ensembled models as many Kagglers do in other challenges.\nMy attempts for various combination between data preprocessings/ models/ learning methods exceeded 20, tried about 10 models.\nMy score started from awful 0.41 or so by making mistake on pre-processing test set, started over with 0.71. And then current final score is 0.84, it's about 13% improvement after all.\n\n## Lessons I've learnied\n1. Ensemble different models really works, 10+% effective.\n2. Model design makes -10+% to +3% difference. I saw very bad models if I started from scratch, or if I choose famous model like AlexNet, it've shown good results. Then if I tuned based on successful model, it improved about 3% maximum. After repeating try and error as well as carefully designing layers to match to the problem, my almost-scratch models started working better.\n3. Preprocessing by domain knowledge to get different perspective enabled ensemble improvement. Especially by taking mean by time, extracting sound events, or simply handling whole data as is.\n4. Weighted ensemble, model predictions are weight-averaged for each row &amp; class. And this weight optimization was done by scipy.optimize.minimize though I used 'Nelder-Mead' instead of 'SLSQP' as [commented here](https://www.kaggle.com/hsperr/finding-ensamble-weights).\n5. Augmentation matters, I used mainly ImageDataGenerator of Keras, as well as mixup &amp; cutout/random erasing as described in my [kernel](https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment).\n6. Learning rate and methods affects results also so much. [Cyclic Learning Rate](https://github.com/bckenstler/CLR)'d helped finding better minimum, comparing to fixed learning rate. Then lowering LR to get fine-tuned final result, it worked.\n7. Final model should learn entire training set if you divide original train set to train/valid set. After I got confidence for a model, I switch to use entire training set and did the same training flow, to get final results.\n\n## Kernels shared\n- [Simple CNN approach](https://www.kaggle.com/daisukelab/simple-cnn-approach)\n- [Sound event based approach](https://www.kaggle.com/daisukelab/sound-event-based-approach)\n- [Time-wise mean approach](https://www.kaggle.com/daisukelab/time-wise-mean-approach)\n- [Example of weighted ensemble](https://www.kaggle.com/daisukelab/example-of-weighted-ensemble)\n- Regarding augmentation, [mixup &amp; cutout/random erasing](https://www.kaggle.com/daisukelab/mixup-cutout-or-random-erasing-to-augment)\n\nI thank competition organizers very much, it is great opportunity and I could learn really a lot!",
    "289636": "Yes I'm going to show some of my code after competition is over.\nOne would be entire code to summarize in kernel, please let me know if I should not be sharing one of working code entirely.",
    "288774": "Thank you for very insightful comments! I'm sure many participants were wondering which kind of techniques you were using. During the course we discussed model averaging so I think many will attempt to use that (and other things you mention) during the last two days.\n\nI hope to hear even more detailed description when the competition is ober.\n\nHeikki\n",
    "294633": "Hi all, I have uploaded all the kernels.\nAnd the ensemble kernel was not correctly opened to public.\nExcuse me if you have tried, now it should be fine.\n\n@Heikki This is all from me. Thanks again!",
    "290263": ""
  }
}