{
  "id": 12844,
  "title": "Blog post describing the winning solution",
  "url": "/competitions/datasciencebowl/discussion/12844",
  "author_name": "sedielem",
  "post_date": "2015-03-17T00:07:04.800000",
  "votes": 65,
  "comment_count": 35,
  "views": 10339,
  "content": "<p>Hey everyone,</p>\n<p>Thanks to Kaggle, Booz Allen and the Hatfield Marine Science Center for a very interesting competition. Congratulations to the other winners!</p>\n<p>Here's a blog post describing our approach in some detail:&nbsp;http://benanne.github.io/2015/03/17/plankton.html</p>\n<p>Code and documentation to reproduce the winning solution will follow soon!</p>\n<p>UPDATE:</p>\n<p>Code and documentation are now available on GitHub:&nbsp;https://github.com/benanne/kaggle-ndsb</p>",
  "messages": [
    {
      "id": 66576,
      "postDate": "2015-03-17T00:07:04.800Z",
      "content": "<p>Hey everyone,</p>\n<p>Thanks to Kaggle, Booz Allen and the Hatfield Marine Science Center for a very interesting competition. Congratulations to the other winners!</p>\n<p>Here's a blog post describing our approach in some detail:&nbsp;http://benanne.github.io/2015/03/17/plankton.html</p>\n<p>Code and documentation to reproduce the winning solution will follow soon!</p>\n<p>UPDATE:</p>\n<p>Code and documentation are now available on GitHub:&nbsp;https://github.com/benanne/kaggle-ndsb</p>",
      "votes": 65
    },
    {
      "id": 67191,
      "postDate": "2015-03-19T09:41:11.783Z",
      "content": "<p>This is actually quite a challenge as there were usually a bunch of people working on different things in parallel at any given time. So it was not a linear process :)</p>\n<p>It's also not a 'discrete' process in the sense that all decisions we made were approximate. We never had perfect information about whether something worked better than something else, due the noise on our validation score estimates (this is always a problem with Kaggle competitions, but in this case it was more severe than I'm used to). So when we decided to use something, it was because it was &quot;probably&quot; better, but it's also possible that it wasn't making any difference. A good example are the untied biases we ended up using - I'm still not 100% convinced that this actually helps, but it didn't hurt, so why not :)</p>\n<p>There's also another important Kaggle lesson that follows from this: don't discard an idea you believe in after a single failed experiment. Keep it around and try it again later, maybe you'll be pleasantly surprised.</p>\n<p>Anyway, here's a rough, incomplete overview of what I did. This is mostly network architecture stuff, which I was more or less in charge of together with Aaron.</p>\n<p>--</p>\n<p>I don't have a great record of the very first few things we tried (first two weeks or so) because we only started properly validating things and recording experimental results after that. But the very first network I tried to train had 5 layers: 3 convolutional with 32 3x3 filters each and 2 dense (one hidden layer with 128 units, one output layer with 121 units). It had pooling layers after the 1st and 2nd convolutional layer and 50x50 input, just the 'center patch' of the images, randomly rotated (so no rescaling). This got us ~1.6 on the leaderboard.</p>\n<p>(That's another important Kaggle lesson by the way, start small so you can iterate fast! Being able to try a ton of things is much more important than having a big model that gets epsilon better performance.)</p>\n<p>We gradually made this bigger by adding more layer, more filters, more augmentation and increasing the input size. A 7-layer network applied to 68x68 patches gave us a score of ~1.1. A 9-layer net with 76x76 input and more augmentation gave us ~1.0.</p>\n<p>When we started to experiment with rescaling the images, our score improved&nbsp;quite a bit. We tried a bunch of different strategies: initially I thought rescaling based on image area would be a good idea, but it turned out that scaling based on the maximal side of the image worked a lot better.</p>\n<p>A network with 8 convolutional layers, a discrete learning rate schedule and an area-based rescaling strategy got us down to 0.79. Then we started experimenting with test-time augmentation and saw a huge drop to ~0.71.</p>\n<p>Adding an extra dense layer and changing the rescaling strategy to be based on the maximal side got us down to ~0.68.</p>\n<p>At this point, I was trying many architectural improvements in parallel so the scores are hard to compare one to one sometimes. I believe cyclic pooling and adding a roll operation only after the first dense layer got us from ~0.67 to ~0.66, so a modest improvement.</p>\n<p>Our first attempt at a multiscale network got us ~0.64, but I don't immediately remember what model this should be compared to, so I don't know how big the jump was.</p>\n<p>I think our best single model eventually got about 0.607 on the public leaderboard (with test-time augmentation). After that we started submitting blends almost exclusively, as well as networks trained with pseudo-labeling.</p>\n<p>--</p>\n<p>This is about the best I can do, based on our submission history. Hopefully it's helpful. Some general guidelines are:</p>\n<p>* Iteration speed is crucially important. keep the ideas that slow down training too much for later in the competition. Initially it's crucial that you are able to try a ton of things. Maybe 80% of them will fail, but that's normal.</p>\n<p>* Don't discard ideas based on a single failed experiment, as mentioned before.</p>\n<p>* Keep track of your ideas in a list, and constantly re-prioritize them based on new information from experiments, or just from thinking about them really hard :) Write EVERYTHING down because you will forget things.</p>\n<p>* Keep track of your results in a list as well. In a team, this is especially important. We used a Google Drive spreadsheet for this, where every team member had their own sheet.</p>\n<p>* Look at the data and what your models learn about it (visualize filters / activations). One thing I find especially useful is to look at validation set examples that the model makes the worst predictions on. This is great for generating ideas. Find patterns in the mistakes the model makes and try to fix them. Write a bunch of data visualization tools while the experiments are running, you won't regret it.</p>",
      "votes": 10
    },
    {
      "id": 66604,
      "postDate": "2015-03-17T01:26:05.580Z",
      "content": "<p>Thanks for sharing! Amazing result!!</p>\n<p>TL;DR:&nbsp;log-uniform&nbsp;rescaling/scaling, modified VGG16,&nbsp;Cyclic pooling:&nbsp;4 rotations in parallel with RMS pooling at the top, Rolling: combinig features from each of the parallel models in with&nbsp;rolling order,&nbsp;leaky ReLU: y=max(x,x/3), side features add at the last layer, NAG better than ADAM,&nbsp;unsupervised CAE pre-training, Pseudo-labeling: use best LB predictions as input to new&nbsp;model, TTA:&nbsp; modest random augmentation for test images&nbsp;</p>",
      "votes": 6
    },
    {
      "id": 66675,
      "postDate": "2015-03-17T09:49:57.617Z",
      "content": "<p>Thanks for the kind words everyone :) Thanks zzspar also for the effective tl;dr summary!</p>\n<p>Personally, my next project is going to be to finally finish my dissertation, which Kaggle has been distracting me from ;)&nbsp;I don't know about the rest of the team though!</p>",
      "votes": 3
    },
    {
      "id": 68270,
      "postDate": "2015-03-25T17:20:37.010Z",
      "content": "<p>Hi everyone,</p>\n<p>The code to reproduce our winning solution as well as documentation are now available on GitHub:&nbsp;https://github.com/benanne/kaggle-ndsb</p>",
      "votes": 4
    },
    {
      "id": 67165,
      "postDate": "2015-03-19T07:07:55.953Z",
      "content": "<p>@sedielem could you please also share what method makes your improvement during competition process? For example first big improvement comes from what method, second comes from what and so on. I think everyone feel interesting in your team thinking process. Thanks a lot.</p>",
      "votes": 1
    },
    {
      "id": 66596,
      "postDate": "2015-03-17T00:58:18.193Z",
      "content": "<p>Congratulations! That is a truly impressive approach.</p>",
      "votes": 1
    },
    {
      "id": 66587,
      "postDate": "2015-03-17T00:34:25.077Z",
      "content": "<p>Congratulations! Thanks for sharing the insights.&nbsp;</p>",
      "votes": 1
    },
    {
      "id": 66586,
      "postDate": "2015-03-17T00:31:39.660Z",
      "content": "<p>Incredible!</p>",
      "votes": 1
    },
    {
      "id": 66859,
      "postDate": "2015-03-17T22:42:55.703Z",
      "content": "<p>Just to clarify, with a &quot;gradient step&quot; I meant a single update of the parameters, i.e. a single forward and backward pass through the network. It sounds like you might be interpreting it as a single pass through the training data, which indeed took a lot longer! I don't have numbers for this since we randomly sampled data points for each minibatch, but I guess you can extrapolate that it took more than 100 seconds for the network to have seen ~30000 training examples for almost all of our models.</p>\n<p>To determine when the CNNs converged, we just kind of eyeballed the learning curves at the start of the competition. Because we were using dropout, the validation error tended not to increase as the network started overfitting, so it wasn't a disaster to leave it training for a bit too long. We settled on 215k updates + 2 learning rate decreases pretty quickly,&nbsp;and any attempts to come up with a better learning schedule&nbsp;didn't really improve things. It's all very &quot;ad hoc&quot;, but it's simple and it works.</p>",
      "votes": 2
    },
    {
      "id": 68272,
      "postDate": "2015-03-25T17:24:23.903Z",
      "content": "<p>Right, I was going to do that next ;)</p>"
    },
    {
      "id": 68271,
      "postDate": "2015-03-25T17:22:44.297Z",
      "content": "<p>Thanks for posting! Mind linking it to your team on the leaderboard via https://www.kaggle.com/c/datasciencebowl/github ?&nbsp;</p>"
    },
    {
      "id": 67209,
      "postDate": "2015-03-19T12:01:31.910Z",
      "content": "<p>Great work sedielem et al. I started out using Lasagne, which is a great library, but for some reason using the same architecture in Graphlab's CNN I easily achieved 0.9 where I struggled to get past 1.1 with Lasagne. I can't put my finger on what the differences were that caused this discrepancy.</p>\n<p>And then again for other problems Lasagne does better and provides more versatility than Graphlab. I'm looking forward to implementing aspects of your solution on Lasagne to see where the hickup was for the databowl problem.</p>"
    },
    {
      "id": 66999,
      "postDate": "2015-03-18T16:11:24.330Z",
      "content": "<p>Congratulations!</p>\n<p>Thanks for sharing&nbsp;:)&nbsp;</p>"
    },
    {
      "id": 66866,
      "postDate": "2015-03-17T22:59:26.877Z",
      "content": "<p>@sedielem</p>\n<p>Yes you are right, I assumed the complete training data pass. 2 days with 100 seconds per iteration thru the complete data set, can we say you iteration was around 1700-1800?</p>\n<p>I think what I am stuck is that I tried various CNN architectures using Bing's cxxnet framework and it seem logloss start to oscillate after 100th iteration in all those CNN architectures I created, it could have been going down very slightly but I did not observe the same decrease after ~100th iteration. Of course I haven't use majority of the cool solutions you described in your blog. So I was wondering whether I gave up too early in iterations of 300-400 or not used the correct CNN architecture. Probably the second one it seems.</p>\n<p>Thanks</p>"
    },
    {
      "id": 66842,
      "postDate": "2015-03-17T21:54:17.060Z",
      "content": "<p>@sedielem</p>\n<p>Congratulations and thanks for the blog post. One thing i noticed that in your blog:&quot;We trained most of the models with about 215000 gradient steps&quot;</p>\n<p>since training took less than 2 days, that means your HW completed a single complete iteration in your CNN less than 1 sec, right?</p>\n<p>Kind of shocked since my GTX 860M does this in 40+ secs and practically reaching your iteration seem to require running more than 90 days. On top of that my laptop warms up in 10-15 minutes and need to give a break for a cool down so it seems impractical with my current laptop.</p>\n<p>One more question: How did you decide when your CNN converged? Not sure I understand &quot;eventually settled on a discrete learning rate schedule with two 10-fold decreases&quot;.</p>\n<p>Thanks</p>"
    },
    {
      "id": 66735,
      "postDate": "2015-03-17T16:11:52.420Z",
      "content": "<p>Great post and congratulations&nbsp;</p>"
    },
    {
      "id": 66712,
      "postDate": "2015-03-17T14:46:56.280Z",
      "content": "<p>I will definitely try to include them! Probably as appendices though, since they sort of fall outside of the main story, which will be about deep learning for music.</p>"
    },
    {
      "id": 66705,
      "postDate": "2015-03-17T14:15:33.853Z",
      "content": "<p>[quote=sedielem;66675]</p>\n<p>Thanks for the kind words everyone :) Thanks zzspar also for the effective tl;dr summary!</p>\n<p>Personally, my next project is going to be to finally finish my dissertation, which Kaggle has been distracting me from ;)&nbsp;I don't know about the rest of the team though!</p>\n<p>[/quote]</p>\n<p>Congrats Sander and your team! You deserve to be the winner.</p>\n<p>Btw, I had the same personal situation as yours :). Will you put your two wins into your dissertation?</p>\n<p>I bet they are strongly supportive.</p>"
    },
    {
      "id": 66696,
      "postDate": "2015-03-17T13:21:08.627Z",
      "content": "<p>Hi Sander, Congratulations to you and your team! And to other prize winners as well!</p>"
    },
    {
      "id": 66681,
      "postDate": "2015-03-17T10:45:44.453Z",
      "content": "<p>[quote=old-ufo;66676]</p>\n<p>@sedielem, could you please also share score of the best-single model (with or without TTA)?</p>\n<p>[/quote]</p>\n<p>We'll see what we can do :) It's probably going to be one of those in the ensemble anyway, so probably yes!</p>"
    },
    {
      "id": 66678,
      "postDate": "2015-03-17T10:13:57.433Z",
      "content": "<p>Congratulations, and thanks a lot for the great blog post!</p>"
    },
    {
      "id": 66676,
      "postDate": "2015-03-17T09:51:39.437Z",
      "content": "<p>@sedielem, could you please also share score of the best-single model (with or without TTA)?</p>"
    },
    {
      "id": 66665,
      "postDate": "2015-03-17T08:30:04.807Z",
      "content": "<p>Congratulations sedielem, you truly rock!<br>I think you and your lab&nbsp;should&nbsp;consider&nbsp;going for&nbsp;ILSVRC2015 as your next project.</p>"
    },
    {
      "id": 66657,
      "postDate": "2015-03-17T07:48:09.797Z",
      "content": "<p>Great team work!</p>"
    },
    {
      "id": 66643,
      "postDate": "2015-03-17T06:19:19.930Z",
      "content": "<p>Congratulations :)</p>"
    },
    {
      "id": 66641,
      "postDate": "2015-03-17T05:42:55.237Z",
      "content": "<p>Congratulations! Solid ideas!</p>",
      "replies": [
        {
          "id": 1220667,
          "postDate": "2021-02-28T08:28:21.340Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 66636,
      "postDate": "2015-03-17T05:13:54.523Z",
      "content": "<p>Congratulations on the team win. I read the write up. Well written and useful. Thanks.</p>"
    },
    {
      "id": 66613,
      "postDate": "2015-03-17T01:52:46.067Z",
      "content": "<p>Congratulations! And thanks for sharing the approaches.&nbsp;</p>"
    },
    {
      "id": 66593,
      "postDate": "2015-03-17T00:54:44.510Z",
      "content": "<p>Congratulations ! Thank you for sharing.</p>"
    },
    {
      "id": 66592,
      "postDate": "2015-03-17T00:53:05.660Z",
      "content": "<p>Thank you for the great post. If a code is shared, I hope I could go through all the setup process this time. I tried to run&nbsp;sedielem's&nbsp;<a href=\"https://github.com/benanne/kaggle-galaxies\">galaxy</a>&nbsp;winning solution but cannot fix a bug in setup.</p>"
    },
    {
      "id": 66590,
      "postDate": "2015-03-17T00:44:22.797Z",
      "content": "<p>Congratulations and thanks for sharing.</p>"
    },
    {
      "id": 66584,
      "postDate": "2015-03-17T00:31:21.173Z",
      "content": "<p>Congratulations! That's great posting!</p>"
    },
    {
      "id": 66582,
      "postDate": "2015-03-17T00:28:14.987Z",
      "content": "<p>Hello, thank you for sharing, that was quick :) Congratulations!</p>"
    },
    {
      "id": 66579,
      "postDate": "2015-03-17T00:22:36.167Z",
      "content": "<p>Congratulations!</p>\n<p>Thanks for the timely&nbsp;description.</p>"
    }
  ],
  "comments": [
    {
      "id": 67191,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-19T09:41:11.783000",
      "content": "<p>This is actually quite a challenge as there were usually a bunch of people working on different things in parallel at any given time. So it was not a linear process :)</p>\n<p>It's also not a 'discrete' process in the sense that all decisions we made were approximate. We never had perfect information about whether something worked better than something else, due the noise on our validation score estimates (this is always a problem with Kaggle competitions, but in this case it was more severe than I'm used to). So when we decided to use something, it was because it was &quot;probably&quot; better, but it's also possible that it wasn't making any difference. A good example are the untied biases we ended up using - I'm still not 100% convinced that this actually helps, but it didn't hurt, so why not :)</p>\n<p>There's also another important Kaggle lesson that follows from this: don't discard an idea you believe in after a single failed experiment. Keep it around and try it again later, maybe you'll be pleasantly surprised.</p>\n<p>Anyway, here's a rough, incomplete overview of what I did. This is mostly network architecture stuff, which I was more or less in charge of together with Aaron.</p>\n<p>--</p>\n<p>I don't have a great record of the very first few things we tried (first two weeks or so) because we only started properly validating things and recording experimental results after that. But the very first network I tried to train had 5 layers: 3 convolutional with 32 3x3 filters each and 2 dense (one hidden layer with 128 units, one output layer with 121 units). It had pooling layers after the 1st and 2nd convolutional layer and 50x50 input, just the 'center patch' of the images, randomly rotated (so no rescaling). This got us ~1.6 on the leaderboard.</p>\n<p>(That's another important Kaggle lesson by the way, start small so you can iterate fast! Being able to try a ton of things is much more important than having a big model that gets epsilon better performance.)</p>\n<p>We gradually made this bigger by adding more layer, more filters, more augmentation and increasing the input size. A 7-layer network applied to 68x68 patches gave us a score of ~1.1. A 9-layer net with 76x76 input and more augmentation gave us ~1.0.</p>\n<p>When we started to experiment with rescaling the images, our score improved&nbsp;quite a bit. We tried a bunch of different strategies: initially I thought rescaling based on image area would be a good idea, but it turned out that scaling based on the maximal side of the image worked a lot better.</p>\n<p>A network with 8 convolutional layers, a discrete learning rate schedule and an area-based rescaling strategy got us down to 0.79. Then we started experimenting with test-time augmentation and saw a huge drop to ~0.71.</p>\n<p>Adding an extra dense layer and changing the rescaling strategy to be based on the maximal side got us down to ~0.68.</p>\n<p>At this point, I was trying many architectural improvements in parallel so the scores are hard to compare one to one sometimes. I believe cyclic pooling and adding a roll operation only after the first dense layer got us from ~0.67 to ~0.66, so a modest improvement.</p>\n<p>Our first attempt at a multiscale network got us ~0.64, but I don't immediately remember what model this should be compared to, so I don't know how big the jump was.</p>\n<p>I think our best single model eventually got about 0.607 on the public leaderboard (with test-time augmentation). After that we started submitting blends almost exclusively, as well as networks trained with pseudo-labeling.</p>\n<p>--</p>\n<p>This is about the best I can do, based on our submission history. Hopefully it's helpful. Some general guidelines are:</p>\n<p>* Iteration speed is crucially important. keep the ideas that slow down training too much for later in the competition. Initially it's crucial that you are able to try a ton of things. Maybe 80% of them will fail, but that's normal.</p>\n<p>* Don't discard ideas based on a single failed experiment, as mentioned before.</p>\n<p>* Keep track of your ideas in a list, and constantly re-prioritize them based on new information from experiments, or just from thinking about them really hard :) Write EVERYTHING down because you will forget things.</p>\n<p>* Keep track of your results in a list as well. In a team, this is especially important. We used a Google Drive spreadsheet for this, where every team member had their own sheet.</p>\n<p>* Look at the data and what your models learn about it (visualize filters / activations). One thing I find especially useful is to look at validation set examples that the model makes the worst predictions on. This is great for generating ideas. Find patterns in the mistakes the model makes and try to fix them. Write a bunch of data visualization tools while the experiments are running, you won't regret it.</p>",
      "votes": 10,
      "replies": []
    },
    {
      "id": 66604,
      "author_name": "udibr",
      "author_url": "",
      "post_date": "2015-03-17T01:26:05.580000",
      "content": "<p>Thanks for sharing! Amazing result!!</p>\n<p>TL;DR:&nbsp;log-uniform&nbsp;rescaling/scaling, modified VGG16,&nbsp;Cyclic pooling:&nbsp;4 rotations in parallel with RMS pooling at the top, Rolling: combinig features from each of the parallel models in with&nbsp;rolling order,&nbsp;leaky ReLU: y=max(x,x/3), side features add at the last layer, NAG better than ADAM,&nbsp;unsupervised CAE pre-training, Pseudo-labeling: use best LB predictions as input to new&nbsp;model, TTA:&nbsp; modest random augmentation for test images&nbsp;</p>",
      "votes": 6,
      "replies": []
    },
    {
      "id": 66675,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-17T09:49:57.617000",
      "content": "<p>Thanks for the kind words everyone :) Thanks zzspar also for the effective tl;dr summary!</p>\n<p>Personally, my next project is going to be to finally finish my dissertation, which Kaggle has been distracting me from ;)&nbsp;I don't know about the rest of the team though!</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 68270,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-25T17:20:37.010000",
      "content": "<p>Hi everyone,</p>\n<p>The code to reproduce our winning solution as well as documentation are now available on GitHub:&nbsp;https://github.com/benanne/kaggle-ndsb</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 67165,
      "author_name": "Liang Pang",
      "author_url": "",
      "post_date": "2015-03-19T07:07:55.953000",
      "content": "<p>@sedielem could you please also share what method makes your improvement during competition process? For example first big improvement comes from what method, second comes from what and so on. I think everyone feel interesting in your team thinking process. Thanks a lot.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 66596,
      "author_name": "Zachary Kann",
      "author_url": "",
      "post_date": "2015-03-17T00:58:18.193000",
      "content": "<p>Congratulations! That is a truly impressive approach.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 66587,
      "author_name": "Patrick Chan",
      "author_url": "",
      "post_date": "2015-03-17T00:34:25.077000",
      "content": "<p>Congratulations! Thanks for sharing the insights.&nbsp;</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 66586,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "2015-03-17T00:31:39.660000",
      "content": "<p>Incredible!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 66859,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-17T22:42:55.703000",
      "content": "<p>Just to clarify, with a &quot;gradient step&quot; I meant a single update of the parameters, i.e. a single forward and backward pass through the network. It sounds like you might be interpreting it as a single pass through the training data, which indeed took a lot longer! I don't have numbers for this since we randomly sampled data points for each minibatch, but I guess you can extrapolate that it took more than 100 seconds for the network to have seen ~30000 training examples for almost all of our models.</p>\n<p>To determine when the CNNs converged, we just kind of eyeballed the learning curves at the start of the competition. Because we were using dropout, the validation error tended not to increase as the network started overfitting, so it wasn't a disaster to leave it training for a bit too long. We settled on 215k updates + 2 learning rate decreases pretty quickly,&nbsp;and any attempts to come up with a better learning schedule&nbsp;didn't really improve things. It's all very &quot;ad hoc&quot;, but it's simple and it works.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 68272,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-25T17:24:23.903000",
      "content": "<p>Right, I was going to do that next ;)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 68271,
      "author_name": "Will Cukierski",
      "author_url": "",
      "post_date": "2015-03-25T17:22:44.297000",
      "content": "<p>Thanks for posting! Mind linking it to your team on the leaderboard via https://www.kaggle.com/c/datasciencebowl/github ?&nbsp;</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 67209,
      "author_name": "Unas",
      "author_url": "",
      "post_date": "2015-03-19T12:01:31.910000",
      "content": "<p>Great work sedielem et al. I started out using Lasagne, which is a great library, but for some reason using the same architecture in Graphlab's CNN I easily achieved 0.9 where I struggled to get past 1.1 with Lasagne. I can't put my finger on what the differences were that caused this discrepancy.</p>\n<p>And then again for other problems Lasagne does better and provides more versatility than Graphlab. I'm looking forward to implementing aspects of your solution on Lasagne to see where the hickup was for the databowl problem.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66999,
      "author_name": "Jae Hyun Lim",
      "author_url": "",
      "post_date": "2015-03-18T16:11:24.330000",
      "content": "<p>Congratulations!</p>\n<p>Thanks for sharing&nbsp;:)&nbsp;</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66866,
      "author_name": "Ertuka",
      "author_url": "",
      "post_date": "2015-03-17T22:59:26.877000",
      "content": "<p>@sedielem</p>\n<p>Yes you are right, I assumed the complete training data pass. 2 days with 100 seconds per iteration thru the complete data set, can we say you iteration was around 1700-1800?</p>\n<p>I think what I am stuck is that I tried various CNN architectures using Bing's cxxnet framework and it seem logloss start to oscillate after 100th iteration in all those CNN architectures I created, it could have been going down very slightly but I did not observe the same decrease after ~100th iteration. Of course I haven't use majority of the cool solutions you described in your blog. So I was wondering whether I gave up too early in iterations of 300-400 or not used the correct CNN architecture. Probably the second one it seems.</p>\n<p>Thanks</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66842,
      "author_name": "Ertuka",
      "author_url": "",
      "post_date": "2015-03-17T21:54:17.060000",
      "content": "<p>@sedielem</p>\n<p>Congratulations and thanks for the blog post. One thing i noticed that in your blog:&quot;We trained most of the models with about 215000 gradient steps&quot;</p>\n<p>since training took less than 2 days, that means your HW completed a single complete iteration in your CNN less than 1 sec, right?</p>\n<p>Kind of shocked since my GTX 860M does this in 40+ secs and practically reaching your iteration seem to require running more than 90 days. On top of that my laptop warms up in 10-15 minutes and need to give a break for a cool down so it seems impractical with my current laptop.</p>\n<p>One more question: How did you decide when your CNN converged? Not sure I understand &quot;eventually settled on a discrete learning rate schedule with two 10-fold decreases&quot;.</p>\n<p>Thanks</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66735,
      "author_name": "AhmedMohamedou",
      "author_url": "",
      "post_date": "2015-03-17T16:11:52.420000",
      "content": "<p>Great post and congratulations&nbsp;</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66712,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-17T14:46:56.280000",
      "content": "<p>I will definitely try to include them! Probably as appendices though, since they sort of fall outside of the main story, which will be about deep learning for music.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66705,
      "author_name": "tund",
      "author_url": "",
      "post_date": "2015-03-17T14:15:33.853000",
      "content": "<p>[quote=sedielem;66675]</p>\n<p>Thanks for the kind words everyone :) Thanks zzspar also for the effective tl;dr summary!</p>\n<p>Personally, my next project is going to be to finally finish my dissertation, which Kaggle has been distracting me from ;)&nbsp;I don't know about the rest of the team though!</p>\n<p>[/quote]</p>\n<p>Congrats Sander and your team! You deserve to be the winner.</p>\n<p>Btw, I had the same personal situation as yours :). Will you put your two wins into your dissertation?</p>\n<p>I bet they are strongly supportive.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66696,
      "author_name": "Maxim Milakov",
      "author_url": "",
      "post_date": "2015-03-17T13:21:08.627000",
      "content": "<p>Hi Sander, Congratulations to you and your team! And to other prize winners as well!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66681,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "2015-03-17T10:45:44.453000",
      "content": "<p>[quote=old-ufo;66676]</p>\n<p>@sedielem, could you please also share score of the best-single model (with or without TTA)?</p>\n<p>[/quote]</p>\n<p>We'll see what we can do :) It's probably going to be one of those in the ensemble anyway, so probably yes!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66678,
      "author_name": "Eelke",
      "author_url": "",
      "post_date": "2015-03-17T10:13:57.433000",
      "content": "<p>Congratulations, and thanks a lot for the great blog post!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66676,
      "author_name": "old-ufo",
      "author_url": "",
      "post_date": "2015-03-17T09:51:39.437000",
      "content": "<p>@sedielem, could you please also share score of the best-single model (with or without TTA)?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66665,
      "author_name": "David Beniaguev",
      "author_url": "",
      "post_date": "2015-03-17T08:30:04.807000",
      "content": "<p>Congratulations sedielem, you truly rock!<br>I think you and your lab&nbsp;should&nbsp;consider&nbsp;going for&nbsp;ILSVRC2015 as your next project.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66657,
      "author_name": "Elena Cuoco",
      "author_url": "",
      "post_date": "2015-03-17T07:48:09.797000",
      "content": "<p>Great team work!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66643,
      "author_name": "old-ufo",
      "author_url": "",
      "post_date": "2015-03-17T06:19:19.930000",
      "content": "<p>Congratulations :)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66641,
      "author_name": "wf",
      "author_url": "",
      "post_date": "2015-03-17T05:42:55.237000",
      "content": "<p>Congratulations! Solid ideas!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1220667,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-02-28T08:28:21.340000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 66636,
      "author_name": "Charles Roberson",
      "author_url": "",
      "post_date": "2015-03-17T05:13:54.523000",
      "content": "<p>Congratulations on the team win. I read the write up. Well written and useful. Thanks.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66613,
      "author_name": "Junonia",
      "author_url": "",
      "post_date": "2015-03-17T01:52:46.067000",
      "content": "<p>Congratulations! And thanks for sharing the approaches.&nbsp;</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66593,
      "author_name": "WalkingMoose",
      "author_url": "",
      "post_date": "2015-03-17T00:54:44.510000",
      "content": "<p>Congratulations ! Thank you for sharing.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66592,
      "author_name": "",
      "author_url": "",
      "post_date": "2015-03-17T00:53:05.660000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66590,
      "author_name": "",
      "author_url": "",
      "post_date": "2015-03-17T00:44:22.797000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66584,
      "author_name": "",
      "author_url": "",
      "post_date": "2015-03-17T00:31:21.173000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66582,
      "author_name": "",
      "author_url": "",
      "post_date": "2015-03-17T00:28:14.987000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 66579,
      "author_name": "",
      "author_url": "",
      "post_date": "2015-03-17T00:22:36.167000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "66576": "",
    "67191": "",
    "66604": "",
    "66675": "",
    "68270": "",
    "67165": "",
    "66596": "",
    "66587": "",
    "66586": "",
    "66859": "",
    "68272": "",
    "68271": "",
    "67209": "",
    "66999": "",
    "66866": "",
    "66842": "",
    "66735": "",
    "66712": "",
    "66705": "",
    "66696": "",
    "66681": "",
    "66678": "",
    "66676": "",
    "66665": "",
    "66657": "",
    "66643": "",
    "66641": "",
    "66636": "",
    "66613": "",
    "66593": "",
    "66592": "",
    "66590": "",
    "66584": "",
    "66582": "",
    "66579": ""
  }
}