![\"block](\"https://storage.googleapis.com/kaggle-competitions/kaggle/4414/media/block_diagram.png\")
We've posted our code to github here.
\n\nOur approach was simple: we developed a neural net that scored ~0.971 on the leaderboard (I discussed the structure of this net previously in this post ), generated a bunch of variations on this net and then just averaged the results together. We weighted nets that we knew or strongly suspected would score well (>0.97) on the public leaderboard higher and we discarded poorly performing (0.963) nets, but other than that we didn't really have a second level model.\nThe highest scoring single net that made it into our ensemble score 0.97147 on the private leaderboard. However, we didn't submit this net individually till after the close of the competition so, we didn't know that until recently.
\n\nWe learned several things when we went to clean up, write up and recreate our scores. Two important ones were, first that we had undertrained many of our nets. Training them for longer times generally resulted in better scores, this was particularly true when paired with the improved validation setup we put together near the end of the competition. Second, we didn't keep track of the exact parameters we used when training some of the nets. As a result, when we went to recreate everything we couldn't recreate the results of those early nets exactly and had to content ourselves with a score slightly above the score we submitted for third place.
\n\nA total of 34 nets made it into the final ensemble, and training all of these nets and generating a solution takes several days of runtime on machine with two high powered GPUs. Using a large ensemble of nets is probably impractical, but a single net or even a few nets averaged together good probably make predictions fairly efficiently. This is because the output top several levels of the net, where most of the computation takes place, could update their outputs incrementally rather than recomputing them from scratch each time. The bookkeeping involved in implementing this would likely be something of headache but it seems possible.
\n\nIf you have any trouble recreating any of the nets, please let us know and as soon as @deep is done with his Thesis/Paper marathon and I'm done moving, we'll get right back to:-) (That is probably Tuesday or Wednesday).
\n\nThanks for the opportunity to compete in the competition with this great dataset. It was a lot of fun.
", "messages": [ { "id": "93058", "postDate": "09/21/2015 02:34:09", "content": "We've posted our code to github here.
\n\nOur approach was simple: we developed a neural net that scored ~0.971 on the leaderboard (I discussed the structure of this net previously in this post ), generated a bunch of variations on this net and then just averaged the results together. We weighted nets that we knew or strongly suspected would score well (>0.97) on the public leaderboard higher and we discarded poorly performing (0.963) nets, but other than that we didn't really have a second level model.\nThe highest scoring single net that made it into our ensemble score 0.97147 on the private leaderboard. However, we didn't submit this net individually till after the close of the competition so, we didn't know that until recently.
\n\nWe learned several things when we went to clean up, write up and recreate our scores. Two important ones were, first that we had undertrained many of our nets. Training them for longer times generally resulted in better scores, this was particularly true when paired with the improved validation setup we put together near the end of the competition. Second, we didn't keep track of the exact parameters we used when training some of the nets. As a result, when we went to recreate everything we couldn't recreate the results of those early nets exactly and had to content ourselves with a score slightly above the score we submitted for third place.
\n\nA total of 34 nets made it into the final ensemble, and training all of these nets and generating a solution takes several days of runtime on machine with two high powered GPUs. Using a large ensemble of nets is probably impractical, but a single net or even a few nets averaged together good probably make predictions fairly efficiently. This is because the output top several levels of the net, where most of the computation takes place, could update their outputs incrementally rather than recomputing them from scratch each time. The bookkeeping involved in implementing this would likely be something of headache but it seems possible.
\n\nIf you have any trouble recreating any of the nets, please let us know and as soon as @deep is done with his Thesis/Paper marathon and I'm done moving, we'll get right back to:-) (That is probably Tuesday or Wednesday).
\n\nThanks for the opportunity to compete in the competition with this great dataset. It was a lot of fun.
", "rawMarkdown": "We've posted our code to github [here](https://github.com/bitsofbits/kaggle_grasp_and_lift_eeg_detection). \r\n\r\nOur approach was simple: we developed a neural net that scored ~0.971 on the leaderboard (I discussed the structure of this net previously in this [post](https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/16288/structure-of-our-best-net-hedj) ), generated a bunch of variations on this net and then just averaged the results together. We weighted nets that we knew or strongly suspected would score well (>0.97) on the public leaderboard higher and we discarded poorly performing (0.963) nets, but other than that we didn't really have a second level model.\r\nThe highest scoring single net that made it into our ensemble score 0.97147 on the private leaderboard. However, we didn't submit this net individually till after the close of the competition so, we didn't know that until recently.\r\n\r\nWe learned several things when we went to clean up, write up and recreate our scores. Two important ones were, first that we had undertrained many of our nets. Training them for longer times generally resulted in better scores, this was particularly true when paired with the improved validation setup we put together near the end of the competition. Second, we didn't keep track of the exact parameters we used when training some of the nets. As a result, when we went to recreate everything we couldn't recreate the results of those early nets exactly and had to content ourselves with a score slightly above the score we submitted for third place. \r\n\r\nA total of 34 nets made it into the final ensemble, and training all of these nets and generating a solution takes several days of runtime on machine with two high powered GPUs. Using a large ensemble of nets is probably impractical, but a single net or even a few nets averaged together good probably make predictions fairly efficiently. This is because the output top several levels of the net, where most of the computation takes place, could update their outputs incrementally rather than recomputing them from scratch each time. The bookkeeping involved in implementing this would likely be something of headache but it seems possible.\r\n\r\nIf you have any trouble recreating any of the nets, please let us know and as soon as @deep is done with his Thesis/Paper marathon and I'm done moving, we'll get right back to:-) (That is probably Tuesday or Wednesday).\r\n\r\nThanks for the opportunity to compete in the competition with this great dataset. It was a lot of fun.", "votes": null }, { "id": "93691", "postDate": "09/30/2015 11:51:03", "content": "Quick update: I'm done training most of your models now. However I ran into a strange Segfault on one of them (see github issue for details). But after that is resolved I should soon be able to reproduce your results and check your solution for rule violations.
", "rawMarkdown": "Quick update: I'm done training most of your models now. However I ran into a strange Segfault on one of them (see github issue for details). But after that is resolved I should soon be able to reproduce your results and check your solution for rule violations.", "votes": null }, { "id": "95215", "postDate": "10/06/2015 00:33:16", "content": "Team HEDJ put together a truly excellent blog post on their approach. I encourage everyone who participated in the competition to take a look! You can read it here.
\n\n
"Our final solution was an ensemble of 34 nets."
", "rawMarkdown": "Team HEDJ put together a truly excellent blog post on their approach. I encourage everyone who participated in the competition to take a look! You can read it [here][1].\r\n\r\n![block diagram of ensemble][2]\r\n\r\n\"Our final solution was an ensemble of 34 nets.\"\r\n\r\n\r\n [1]: http://blog.kaggle.com/2015/10/05/grasp-and-lift-eeg-detection-winners-interview-3rd-place-team-hedj/\r\n [2]: https://kaggle2.blob.core.windows.net/competitions/kaggle/4414/media/block_diagram.png", "votes": null } ], "comments": [ { "id": 93691, "author_name": "qwlouse", "author_url": "", "post_date": "09/30/2015 11:51:03", "content": "Quick update: I'm done training most of your models now. However I ran into a strange Segfault on one of them (see github issue for details). But after that is resolved I should soon be able to reproduce your results and check your solution for rule violations.
", "votes": null, "replies": [] }, { "id": 95215, "author_name": "annavictoria", "author_url": "", "post_date": "10/06/2015 00:33:16", "content": "Team HEDJ put together a truly excellent blog post on their approach. I encourage everyone who participated in the competition to take a look! You can read it here.
\n\n
"Our final solution was an ensemble of 34 nets."
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "93058": "We've posted our code to github [here](https://github.com/bitsofbits/kaggle_grasp_and_lift_eeg_detection). \r\n\r\nOur approach was simple: we developed a neural net that scored ~0.971 on the leaderboard (I discussed the structure of this net previously in this [post](https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/16288/structure-of-our-best-net-hedj) ), generated a bunch of variations on this net and then just averaged the results together. We weighted nets that we knew or strongly suspected would score well (>0.97) on the public leaderboard higher and we discarded poorly performing (0.963) nets, but other than that we didn't really have a second level model.\r\nThe highest scoring single net that made it into our ensemble score 0.97147 on the private leaderboard. However, we didn't submit this net individually till after the close of the competition so, we didn't know that until recently.\r\n\r\nWe learned several things when we went to clean up, write up and recreate our scores. Two important ones were, first that we had undertrained many of our nets. Training them for longer times generally resulted in better scores, this was particularly true when paired with the improved validation setup we put together near the end of the competition. Second, we didn't keep track of the exact parameters we used when training some of the nets. As a result, when we went to recreate everything we couldn't recreate the results of those early nets exactly and had to content ourselves with a score slightly above the score we submitted for third place. \r\n\r\nA total of 34 nets made it into the final ensemble, and training all of these nets and generating a solution takes several days of runtime on machine with two high powered GPUs. Using a large ensemble of nets is probably impractical, but a single net or even a few nets averaged together good probably make predictions fairly efficiently. This is because the output top several levels of the net, where most of the computation takes place, could update their outputs incrementally rather than recomputing them from scratch each time. The bookkeeping involved in implementing this would likely be something of headache but it seems possible.\r\n\r\nIf you have any trouble recreating any of the nets, please let us know and as soon as @deep is done with his Thesis/Paper marathon and I'm done moving, we'll get right back to:-) (That is probably Tuesday or Wednesday).\r\n\r\nThanks for the opportunity to compete in the competition with this great dataset. It was a lot of fun.", "93691": "Quick update: I'm done training most of your models now. However I ran into a strange Segfault on one of them (see github issue for details). But after that is resolved I should soon be able to reproduce your results and check your solution for rule violations.", "95215": "Team HEDJ put together a truly excellent blog post on their approach. I encourage everyone who participated in the competition to take a look! You can read it [here][1].\r\n\r\n![block diagram of ensemble][2]\r\n\r\n\"Our final solution was an ensemble of 34 nets.\"\r\n\r\n\r\n [1]: http://blog.kaggle.com/2015/10/05/grasp-and-lift-eeg-detection-winners-interview-3rd-place-team-hedj/\r\n [2]: https://kaggle2.blob.core.windows.net/competitions/kaggle/4414/media/block_diagram.png" }, "source": "existing_discussion", "complete": true, "competition_id": 4477, "competition_slug": "grasp-and-lift-eeg-detection", "team_id": 197772, "solution_url": "/competitions/grasp-and-lift-eeg-detection/writeups/hedj-team-hedj-solution" }