What's your single model performance so far? Let's hear it for those who don't mind.
", "messages": [ { "id": "90198", "postDate": "08/24/2015 03:27:57", "content": "What's your single model performance so far? Let's hear it for those who don't mind.
", "rawMarkdown": "What's your single model performance so far? Let's hear it for those who don't mind.", "votes": null }, { "id": "90221", "postDate": "08/24/2015 11:16:36", "content": "Currently at .94300.
\n\nModification of Tim h.'s NN with some light feature engineering. Big issue for me seems to be performance. Trying to work on that now actually (after a vacation break)
", "rawMarkdown": "Currently at .94300. \r\n\r\nModification of Tim h.'s NN with some light feature engineering. Big issue for me seems to be performance. Trying to work on that now actually (after a vacation break)", "votes": null }, { "id": "91079", "postDate": "08/31/2015 19:15:37", "content": "@jamesdipadua, sorry to leave you hanging like, we've been buried under a pile neural nets and I haven't had a time to check the forum much. Anyway, our best single model scores a bit over 0.971. It shares a common ancestor with the code I posted, but the important parts are not very similar to that code any more.
", "rawMarkdown": "jamesdipadua, sorry to leave you hanging like, we've been buried under a pile neural nets and I haven't had a time to check the forum much. Anyway, our best single model scores a bit over 0.971. It shares a common ancestor with the code I posted, but the important parts are not very similar to that code any more.", "votes": null }, { "id": "91080", "postDate": "08/31/2015 19:27:41", "content": "oh, no worries tim. i think what was needed was a bit of feature creation. not sure if that's what you mean by "important parts."
\n\nwas never really able to get the performance where i wanted (but went to croatia for a vacation and worked on some other projects).
", "rawMarkdown": "oh, no worries tim. i think what was needed was a bit of feature creation. not sure if that's what you mean by \"important parts.\" \r\n\r\nwas never really able to get the performance where i wanted (but went to croatia for a vacation and worked on some other projects).", "votes": null }, { "id": "91099", "postDate": "08/31/2015 21:06:52", "content": "All will be revealed shortly, at least if we didn't dramatically overfit on public leaderboard, but I suspect you'll be surprised....
", "rawMarkdown": "All will be revealed shortly, at least if we didn't dramatically overfit on public leaderboard, but I suspect you'll be surprised....", "votes": null }, { "id": "91102", "postDate": "08/31/2015 21:30:03", "content": "You have my curiosity !
\n\nWill we be surprised by your single model ranking above 0.98 or by your thousands of weak models ?
", "rawMarkdown": "You have my curiosity ! \r\n\r\nWill we be surprised by your single model ranking above 0.98 or by your thousands of weak models ?", "votes": null }, { "id": "91121", "postDate": "09/01/2015 00:05:19", "content": "Actually, neither of the above, we had a few dozen intermediate (~0.97) models, although we would have trained more if we had time. But what I was referring to was James' suspicion that our secret was some sort of hybrid feature engineering / neural net scheme.
\n\nWe wanted to try some fancy preprocessing, but we ran out of time, so the only preprocessing we did was some light filtering to take out the very low and very high frequencies. Other than that the models are just neural nets. So what's surprising is, in some sense, how boring our models are....
\n\nI did figure out a couple of neat tricks on the neural net end that improved the results quite a bit. But most of the work was just tuning. And then more tuning...
", "rawMarkdown": "Actually, neither of the above, we had a few dozen intermediate (~0.97) models, although we would have trained more if we had time. But what I was referring to was James' suspicion that our secret was some sort of hybrid feature engineering / neural net scheme. \r\n\r\nWe *wanted* to try some fancy preprocessing, but we ran out of time, so the only preprocessing we did was some light filtering to take out the very low and very high frequencies. Other than that the models are just neural nets. So what's surprising is, in some sense, how boring our models are....\r\n\r\nI did figure out a couple of neat tricks on the neural net end that improved the results quite a bit. But most of the work was just tuning. And then more tuning...", "votes": null }, { "id": "91123", "postDate": "09/01/2015 00:06:35", "content": "Congrats everyone. Not much LB movement (we didn't move at all) and congrats to Sky Library who has become a Master
", "rawMarkdown": "Congrats everyone. Not much LB movement (we didn't move at all) and congrats to Sky Library who has become a Master", "votes": null }, { "id": "91168", "postDate": "09/01/2015 10:01:38", "content": "Hey,
\n\nCongratulations! Thank you all very much for this awesome challenge :-)
\n\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)
\n\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.
", "rawMarkdown": "Hey,\r\n\r\nCongratulations! Thank you all very much for this awesome challenge :-)\r\n\r\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)\r\n\r\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.", "votes": null }, { "id": "91178", "postDate": "09/01/2015 12:57:15", "content": "@blaine: Congratulations! That's very gratifying that you ended up using some of my code; it makes me feel better about all the ideas I ended up borrowing from Alexandre's posts.
\n\nSuddenly I'm feeling pretty good about our best single model. Now that I can see the private leaderboard scores I can see that the score for our best single model held up pretty well: 0.97117 / 0.97088 public/private leaderboard. It'll be interesting to see if there are any single models out there that beat that.
", "rawMarkdown": "blaine: Congratulations! That's very gratifying that you ended up using some of my code; it makes me feel better about all the ideas I ended up borrowing from Alexandre's posts.\r\n\r\nSuddenly I'm feeling pretty good about our best single model. Now that I can see the private leaderboard scores I can see that the score for our best single model held up pretty well: 0.97117 / 0.97088 public/private leaderboard. It'll be interesting to see if there are any single models out there that beat that.", "votes": null }, { "id": "91186", "postDate": "09/01/2015 13:45:49", "content": "Congratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.
\n\nOur best "single" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.
\n\nThe observation is that events are inter-dependent and temporally dependent, which is quite obvious. Please find our code on github, here
", "rawMarkdown": "Congratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.\r\n\r\nOur best \"single\" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.\r\n\r\nThe observation is that events are inter-dependent and temporally dependent, which is quite obvious. Please find our code on github, [here][1]\r\n\r\n\r\n [1]: https://github.com/daxiongshu/Grasp-and-Lift", "votes": null }, { "id": "91202", "postDate": "09/01/2015 15:19:36", "content": "Congrats to the winners, We learned a lot from your top players script and discussion. I am also glad that several kagglers become kaggle master after this competition.
\n\n@blaine Thank you, it's very nice of you to say that :)
\n\n@Bluefool Thank you for being an awesome teammate~
\n\n[quote=blaine;91168]
\n\nCongratulations! Thank you all very much for this awesome challenge :-)
\n\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)
\n\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.
\n\n[/quote]
", "rawMarkdown": "Congrats to the winners, We learned a lot from your top players script and discussion. I am also glad that several kagglers become kaggle master after this competition.\r\n\r\n@blaine Thank you, it's very nice of you to say that :)\r\n\r\n@Bluefool Thank you for being an awesome teammate~\r\n\r\n[quote=blaine;91168]\r\n\r\nCongratulations! Thank you all very much for this awesome challenge :-)\r\n\r\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)\r\n\r\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.\r\n\r\n[/quote]", "votes": null }, { "id": "91221", "postDate": "09/01/2015 17:11:45", "content": "The best single model of our team was 0.964.
\n\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC.
", "rawMarkdown": "The best single model of our team was 0.964. \r\n\r\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC.", "votes": null }, { "id": "91239", "postDate": "09/01/2015 17:59:43", "content": "[quote=Guocong Song;91221]
\n\nThe best single model of our team was 0.964.
\n\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC.
\n\n[/quote]\nCould you explain what you refer with "ordinal regression objective " and "multi-class objective function"? Are the Mean Squared Error and CrossEntropy?
", "rawMarkdown": "[quote=Guocong Song;91221]\r\n\r\nThe best single model of our team was 0.964. \r\n\r\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC. \r\n\r\n[/quote]\r\nCould you explain what you refer with \"ordinal regression objective \" and \"multi-class objective function\"? Are the Mean Squared Error and CrossEntropy?", "votes": null }, { "id": "91244", "postDate": "09/01/2015 18:24:09", "content": "@Guocong Song, that's interesting. I experimented a little bit with using a multi-class objective function early on, but it had the nasty habit of making my more "interesting" nets become unstable. So, I stopped using it. I always intended to come back and give it another try at some point, but there never enough time!
\n\n@rcarson, Thanks for the compliment, but I don't think I qualify as a NN Master. I'm working on it, but there's still a ton I don't know! I'm glad a lot of people got some use out my code though; that's always gratifying. Also, I'm always impressed by schemes where one stacks a bunch of predictors up into something strong. You have my respect sir.
\n\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!
", "rawMarkdown": "Guocong Song, that's interesting. I experimented a little bit with using a multi-class objective function early on, but it had the nasty habit of making my more \"interesting\" nets become unstable. So, I stopped using it. I always intended to come back and give it another try at some point, but there never enough time!\r\n\r\n@rcarson, Thanks for the compliment, but I don't think I qualify as a NN Master. I'm working on it, but there's still a ton I don't know! I'm glad a lot of people got some use out my code though; that's always gratifying. Also, I'm always impressed by schemes where one stacks a bunch of predictors up into something strong. You have my respect sir.\r\n\r\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!", "votes": null }, { "id": "91247", "postDate": "09/01/2015 18:34:55", "content": "@Incalo, FWIW, I interpreted that as binary cross entropy combined with sigmoid nonlinearity on the output in the first case (since that is what I did). This allows the net to predict the values of the different activities independently. However the activities were not independent, there were only 12 or so different combinations. So one could instead encode the 6 event values onto one of those twelve possible states. This has the effect of reducing the output space that one is trying to predict in from 2^6 (64) down to about 12, reducing the problem space considerably. One would use categorical cross entropy combined with a softmax output function in this case. As I said, I had only so-so results with this, but wanted to try it again.
\n\nOf course I may be totally misrepresenting what @Guocong Song meant, but that's what I tried early on.
", "rawMarkdown": "Incalo, FWIW, I interpreted that as binary cross entropy combined with sigmoid nonlinearity on the output in the first case (since that is what I did). This allows the net to predict the values of the different activities independently. However the activities were not independent, there were only 12 or so different combinations. So one could instead encode the 6 event values onto one of those twelve possible states. This has the effect of reducing the output space that one is trying to predict in from 2^6 (64) down to about 12, reducing the problem space considerably. One would use categorical cross entropy combined with a softmax output function in this case. As I said, I had only so-so results with this, but wanted to try it again.\r\n\r\nOf course I may be totally misrepresenting what @Guocong Song meant, but that's what I tried early on.", "votes": null }, { "id": "91252", "postDate": "09/01/2015 18:43:40", "content": "Thank you Tim. You have a wonderful team : ) \nI hope to see your winning code soon. Best regards.
", "rawMarkdown": "Thank you Tim. You have a wonderful team : ) \r\nI hope to see your winning code soon. Best regards.", "votes": null }, { "id": "91265", "postDate": "09/01/2015 20:16:22", "content": "@Tim you rightfully feel good about your single model, it's very impressive. As well as rcarson's and Guocong's, very solid scores.
\n\nIt's hard to call them "single models" (I apologize for a small offtopic then), but in our case the true workhorses of the solution were Recurrent Neural Nets and XGBoosts trained on 1st level metafeatures (i.e. outputs of models trained on raw data). We had lots of diversity in metafeatures, so 2nd level models could truly shine here.
\n\nRecurrent Neural Nets naturally take into consideration time progression of data, which allowed the model to infer the sequence of events. Of course we trained them to output predictions of all events simultaneously, so they also capture dependencies between output classes. Our best not bagged RNN scores 0.97946 public/private (very stable). Unfortunately we didn't have much luck (and patience :-) ) in training/tuning a well-scoring RNN on raw data.
\n\nAs for XGBoosts, we added a time-unrolling preprocessing step - we used past samples along with the current time sample to let it also capture sequence of events, and they were trained per-event. Best not bagged XGB scores 0.9756 private / 0.9764 public.
\n\nOn a side note, probably inputting your powerful single models as metafeatures to our RNNs and XGBs could give excellent AUC :-)
", "rawMarkdown": "Tim you rightfully feel good about your single model, it's very impressive. As well as rcarson's and Guocong's, very solid scores.\r\n\r\nIt's hard to call them \"single models\" (I apologize for a small offtopic then), but in our case the true workhorses of the solution were Recurrent Neural Nets and XGBoosts trained on 1st level metafeatures (i.e. outputs of models trained on raw data). We had lots of diversity in metafeatures, so 2nd level models could truly shine here. \r\n\r\nRecurrent Neural Nets naturally take into consideration time progression of data, which allowed the model to infer the sequence of events. Of course we trained them to output predictions of all events simultaneously, so they also capture dependencies between output classes. Our best not bagged RNN scores 0.97946 public/private (very stable). Unfortunately we didn't have much luck (and patience :-) ) in training/tuning a well-scoring RNN on raw data.\r\n\r\nAs for XGBoosts, we added a time-unrolling preprocessing step - we used past samples along with the current time sample to let it also capture sequence of events, and they were trained per-event. Best not bagged XGB scores 0.9756 private / 0.9764 public.\r\n\r\nOn a side note, probably inputting your powerful single models as metafeatures to our RNNs and XGBs could give excellent AUC :-)", "votes": null }, { "id": "91281", "postDate": "09/01/2015 22:57:40", "content": "Congrats everyone! The results of everyone here are pretty impressive! The methods are even more stellar!
\n\n@rcarson, your effort of stacking many first level predictors and turn them in to such strong ensemble is quite impressive! Our 'ensembles' were the good old weighted averaging :) and we didn't even try and train per subject although with proper caution it could have helped much.
\n\n@blaine, your idea of using 3 layers of models is quite amazing! The use of RNNs and XGB in that manner seems quite intesresting and I cannot wait till I see your code :)
\n\nIf you are interested in using your two level stacking on some of our models and see what the results could have been, we could try that sometime in the future once the dusts settle and we all get our code up on github.
", "rawMarkdown": "Congrats everyone! The results of everyone here are pretty impressive! The methods are even more stellar!\r\n\r\n@rcarson, your effort of stacking many first level predictors and turn them in to such strong ensemble is quite impressive! Our 'ensembles' were the good old weighted averaging :) and we didn't even try and train per subject although with proper caution it could have helped much.\r\n\r\n@blaine, your idea of using 3 layers of models is quite amazing! The use of RNNs and XGB in that manner seems quite intesresting and I cannot wait till I see your code :) \r\n\r\nIf you are interested in using your two level stacking on some of our models and see what the results could have been, we could try that sometime in the future once the dusts settle and we all get our code up on github.", "votes": null }, { "id": "91289", "postDate": "09/02/2015 00:33:56", "content": "@blaine, I look forward to getting a look at your RNN. I really wanted to try that since it seems like a natural fit for this problem. However, I've never set one of those up before and so decided discretion was the better part of valor and went with what I knew.
\n\nAnyway, it was great fun and I enjoyed it a lot. It would indeed be interesting to see what would happen if our NN models got thrown into your machine. After the just settles a bit though, we have a bunch of clean up to do first!
", "rawMarkdown": "blaine, I look forward to getting a look at your RNN. I really wanted to try that since it seems like a natural fit for this problem. However, I've never set one of those up before and so decided discretion was the better part of valor and went with what I knew. \r\n\r\nAnyway, it was great fun and I enjoyed it a lot. It would indeed be interesting to see what would happen if our NN models got thrown into your machine. After the just settles a bit though, we have a bunch of clean up to do first!", "votes": null }, { "id": "91300", "postDate": "09/02/2015 04:17:49", "content": "Congrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot !
\n\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on "Kaggle Master" badge.
\n\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.!
", "rawMarkdown": "Congrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot ! \r\n\r\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on \"Kaggle Master\" badge. \r\n\r\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.!", "votes": null }, { "id": "91305", "postDate": "09/02/2015 07:47:00", "content": "[quote=SRK;91300]
\n\nCongrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot !
\n\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on "Kaggle Master" badge.
\n\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.!
\n\n[/quote]\nThanks a lot SRK! I'm happy to have been useful.
", "rawMarkdown": "[quote=SRK;91300]\r\n\r\nCongrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot ! \r\n\r\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on \"Kaggle Master\" badge. \r\n\r\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.! \r\n\r\n[/quote]\r\nThanks a lot SRK! I'm happy to have been useful.", "votes": null }, { "id": "91306", "postDate": "09/02/2015 07:51:21", "content": "[quote=rcarson;91186]
\n\nCongratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.
\n\nOur best "single" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.
\n\n[/quote]\nYou are welcome rcarson! It is useful to read how you used my script, and I'm happy for your results. I did not succeed in obtaining very good results with my script. I was so concentrated in preprocessing the data, remove artifact, avoid aliasing and so on, that I did not explore other ways to work with those data until Tim posted his script and showed how NN worked so well with data with his idea of CNN.
", "rawMarkdown": "[quote=rcarson;91186]\r\n\r\nCongratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.\r\n\r\nOur best \"single\" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.\r\n\r\n \r\n\r\n[/quote]\r\nYou are welcome rcarson! It is useful to read how you used my script, and I'm happy for your results. I did not succeed in obtaining very good results with my script. I was so concentrated in preprocessing the data, remove artifact, avoid aliasing and so on, that I did not explore other ways to work with those data until Tim posted his script and showed how NN worked so well with data with his idea of CNN.", "votes": null }, { "id": "91307", "postDate": "09/02/2015 07:55:11", "content": "[quote=Tim Hochberg;91244]
\n\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!
\n\n[/quote]\nThanks to you Tim! It was really a pleasure to be part of your team and learn so much from you. Thanks to all the team too!and congratulations.
", "rawMarkdown": "[quote=Tim Hochberg;91244]\r\n\r\n \r\n\r\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!\r\n\r\n\r\n\r\n[/quote]\r\nThanks to you Tim! It was really a pleasure to be part of your team and learn so much from you. Thanks to all the team too!and congratulations.", "votes": null }, { "id": "91311", "postDate": "09/02/2015 09:40:05", "content": "Congratulations to the winners,
\n\nI'm surprised at the fact that, apparently, all the top-performers used a pure ML approach.
\n\nI didn't have enough spare time to participate in the competition,but if I had, I would've used an approach based on a finite-state machine: since the sequences of events is known in advance, this fact imposes some constraints in the temporal sequence of labels. This could've been use to refine the initial predictions based on independent predictors for each labels, by running a forward-backward algorithm - or, since the "backward" thing is forbidden by the rules, just the "forward" step.
\n\nWhat do you think? Did anybody try this approach?
", "rawMarkdown": "Congratulations to the winners, \r\n\r\nI'm surprised at the fact that, apparently, all the top-performers used a pure ML approach. \r\n\r\nI didn't have enough spare time to participate in the competition,but if I had, I would've used an approach based on a finite-state machine: since the sequences of events is known in advance, this fact imposes some constraints in the temporal sequence of labels. This could've been use to refine the initial predictions based on independent predictors for each labels, by running a forward-backward algorithm - or, since the \"backward\" thing is forbidden by the rules, just the \"forward\" step. \r\n\r\nWhat do you think? Did anybody try this approach?", "votes": null }, { "id": "91321", "postDate": "09/02/2015 11:19:57", "content": "@Jose, I think that's a good idea, too bad you didn't have time to try it out. As for capturing the temporal sequence of labels - this is exactly the reasoning that led us to use Recurrent Neural Nets. They were extremely effective, but we had success with them rather on metafeatures than on raw data (they were very computationally intensive on raw data and required much tuning, so at some point we decided to focus our CPU/GPU power on other things).
\n\nWe chose RNNs over HMMs (where HMM is very close to being a FSM), because RNNs have a bigger learning capacity and can model complicated nonlinear dynamics, e.g. AFAIK they achieve better results in speech recognition tasks (e.g. last page of this article). However, we haven't done any experiments with FSMs/HMMs on this data, so they might have as well turned out to be superior than RNNs in this setting.
", "rawMarkdown": "Jose, I think that's a good idea, too bad you didn't have time to try it out. As for capturing the temporal sequence of labels - this is exactly the reasoning that led us to use Recurrent Neural Nets. They were extremely effective, but we had success with them rather on metafeatures than on raw data (they were very computationally intensive on raw data and required much tuning, so at some point we decided to focus our CPU/GPU power on other things).\r\n\r\nWe chose RNNs over HMMs (where HMM is very close to being a FSM), because RNNs have a bigger learning capacity and can model complicated nonlinear dynamics, e.g. AFAIK they achieve better results in speech recognition tasks (e.g. [last page of this article][1]). However, we haven't done any experiments with FSMs/HMMs on this data, so they might have as well turned out to be superior than RNNs in this setting.\r\n\r\n [1]: http://www.cs.toronto.edu/~asamir/papers/icassp10.pdf", "votes": null }, { "id": "91358", "postDate": "09/02/2015 15:46:20", "content": "@deep @Elena Cuoco @SkyLibrary, congratulations! See you in the master competition :D
", "rawMarkdown": "deep @Elena Cuoco @SkyLibrary, congratulations! See you in the master competition :D", "votes": null } ], "comments": [ { "id": 90221, "author_name": "messels", "author_url": "", "post_date": "08/24/2015 11:16:36", "content": "Currently at .94300.
\n\nModification of Tim h.'s NN with some light feature engineering. Big issue for me seems to be performance. Trying to work on that now actually (after a vacation break)
", "votes": null, "replies": [] }, { "id": 91079, "author_name": "bitsofbits", "author_url": "", "post_date": "08/31/2015 19:15:37", "content": "@jamesdipadua, sorry to leave you hanging like, we've been buried under a pile neural nets and I haven't had a time to check the forum much. Anyway, our best single model scores a bit over 0.971. It shares a common ancestor with the code I posted, but the important parts are not very similar to that code any more.
", "votes": null, "replies": [] }, { "id": 91080, "author_name": "messels", "author_url": "", "post_date": "08/31/2015 19:27:41", "content": "oh, no worries tim. i think what was needed was a bit of feature creation. not sure if that's what you mean by "important parts."
\n\nwas never really able to get the performance where i wanted (but went to croatia for a vacation and worked on some other projects).
", "votes": null, "replies": [] }, { "id": 91099, "author_name": "bitsofbits", "author_url": "", "post_date": "08/31/2015 21:06:52", "content": "All will be revealed shortly, at least if we didn't dramatically overfit on public leaderboard, but I suspect you'll be surprised....
", "votes": null, "replies": [] }, { "id": 91102, "author_name": "alexandrebarachant", "author_url": "", "post_date": "08/31/2015 21:30:03", "content": "You have my curiosity !
\n\nWill we be surprised by your single model ranking above 0.98 or by your thousands of weak models ?
", "votes": null, "replies": [] }, { "id": 91121, "author_name": "bitsofbits", "author_url": "", "post_date": "09/01/2015 00:05:19", "content": "Actually, neither of the above, we had a few dozen intermediate (~0.97) models, although we would have trained more if we had time. But what I was referring to was James' suspicion that our secret was some sort of hybrid feature engineering / neural net scheme.
\n\nWe wanted to try some fancy preprocessing, but we ran out of time, so the only preprocessing we did was some light filtering to take out the very low and very high frequencies. Other than that the models are just neural nets. So what's surprising is, in some sense, how boring our models are....
\n\nI did figure out a couple of neat tricks on the neural net end that improved the results quite a bit. But most of the work was just tuning. And then more tuning...
", "votes": null, "replies": [] }, { "id": 91123, "author_name": "domcastro", "author_url": "", "post_date": "09/01/2015 00:06:35", "content": "Congrats everyone. Not much LB movement (we didn't move at all) and congrats to Sky Library who has become a Master
", "votes": null, "replies": [] }, { "id": 91168, "author_name": "rafalcycon", "author_url": "", "post_date": "09/01/2015 10:01:38", "content": "Hey,
\n\nCongratulations! Thank you all very much for this awesome challenge :-)
\n\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)
\n\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.
", "votes": null, "replies": [] }, { "id": 91178, "author_name": "bitsofbits", "author_url": "", "post_date": "09/01/2015 12:57:15", "content": "@blaine: Congratulations! That's very gratifying that you ended up using some of my code; it makes me feel better about all the ideas I ended up borrowing from Alexandre's posts.
\n\nSuddenly I'm feeling pretty good about our best single model. Now that I can see the private leaderboard scores I can see that the score for our best single model held up pretty well: 0.97117 / 0.97088 public/private leaderboard. It'll be interesting to see if there are any single models out there that beat that.
", "votes": null, "replies": [] }, { "id": 91186, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2015 13:45:49", "content": "Congratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.
\n\nOur best "single" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.
\n\nThe observation is that events are inter-dependent and temporally dependent, which is quite obvious. Please find our code on github, here
", "votes": null, "replies": [] }, { "id": 91202, "author_name": "skylibrary", "author_url": "", "post_date": "09/01/2015 15:19:36", "content": "Congrats to the winners, We learned a lot from your top players script and discussion. I am also glad that several kagglers become kaggle master after this competition.
\n\n@blaine Thank you, it's very nice of you to say that :)
\n\n@Bluefool Thank you for being an awesome teammate~
\n\n[quote=blaine;91168]
\n\nCongratulations! Thank you all very much for this awesome challenge :-)
\n\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)
\n\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 91221, "author_name": "songgc", "author_url": "", "post_date": "09/01/2015 17:11:45", "content": "The best single model of our team was 0.964.
\n\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC.
", "votes": null, "replies": [] }, { "id": 91239, "author_name": "lnicalo", "author_url": "", "post_date": "09/01/2015 17:59:43", "content": "[quote=Guocong Song;91221]
\n\nThe best single model of our team was 0.964.
\n\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC.
\n\n[/quote]\nCould you explain what you refer with "ordinal regression objective " and "multi-class objective function"? Are the Mean Squared Error and CrossEntropy?
", "votes": null, "replies": [] }, { "id": 91244, "author_name": "bitsofbits", "author_url": "", "post_date": "09/01/2015 18:24:09", "content": "@Guocong Song, that's interesting. I experimented a little bit with using a multi-class objective function early on, but it had the nasty habit of making my more "interesting" nets become unstable. So, I stopped using it. I always intended to come back and give it another try at some point, but there never enough time!
\n\n@rcarson, Thanks for the compliment, but I don't think I qualify as a NN Master. I'm working on it, but there's still a ton I don't know! I'm glad a lot of people got some use out my code though; that's always gratifying. Also, I'm always impressed by schemes where one stacks a bunch of predictors up into something strong. You have my respect sir.
\n\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!
", "votes": null, "replies": [] }, { "id": 91247, "author_name": "bitsofbits", "author_url": "", "post_date": "09/01/2015 18:34:55", "content": "@Incalo, FWIW, I interpreted that as binary cross entropy combined with sigmoid nonlinearity on the output in the first case (since that is what I did). This allows the net to predict the values of the different activities independently. However the activities were not independent, there were only 12 or so different combinations. So one could instead encode the 6 event values onto one of those twelve possible states. This has the effect of reducing the output space that one is trying to predict in from 2^6 (64) down to about 12, reducing the problem space considerably. One would use categorical cross entropy combined with a softmax output function in this case. As I said, I had only so-so results with this, but wanted to try it again.
\n\nOf course I may be totally misrepresenting what @Guocong Song meant, but that's what I tried early on.
", "votes": null, "replies": [] }, { "id": 91252, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2015 18:43:40", "content": "Thank you Tim. You have a wonderful team : ) \nI hope to see your winning code soon. Best regards.
", "votes": null, "replies": [] }, { "id": 91265, "author_name": "rafalcycon", "author_url": "", "post_date": "09/01/2015 20:16:22", "content": "@Tim you rightfully feel good about your single model, it's very impressive. As well as rcarson's and Guocong's, very solid scores.
\n\nIt's hard to call them "single models" (I apologize for a small offtopic then), but in our case the true workhorses of the solution were Recurrent Neural Nets and XGBoosts trained on 1st level metafeatures (i.e. outputs of models trained on raw data). We had lots of diversity in metafeatures, so 2nd level models could truly shine here.
\n\nRecurrent Neural Nets naturally take into consideration time progression of data, which allowed the model to infer the sequence of events. Of course we trained them to output predictions of all events simultaneously, so they also capture dependencies between output classes. Our best not bagged RNN scores 0.97946 public/private (very stable). Unfortunately we didn't have much luck (and patience :-) ) in training/tuning a well-scoring RNN on raw data.
\n\nAs for XGBoosts, we added a time-unrolling preprocessing step - we used past samples along with the current time sample to let it also capture sequence of events, and they were trained per-event. Best not bagged XGB scores 0.9756 private / 0.9764 public.
\n\nOn a side note, probably inputting your powerful single models as metafeatures to our RNNs and XGBs could give excellent AUC :-)
", "votes": null, "replies": [] }, { "id": 91281, "author_name": "deepcnn", "author_url": "", "post_date": "09/01/2015 22:57:40", "content": "Congrats everyone! The results of everyone here are pretty impressive! The methods are even more stellar!
\n\n@rcarson, your effort of stacking many first level predictors and turn them in to such strong ensemble is quite impressive! Our 'ensembles' were the good old weighted averaging :) and we didn't even try and train per subject although with proper caution it could have helped much.
\n\n@blaine, your idea of using 3 layers of models is quite amazing! The use of RNNs and XGB in that manner seems quite intesresting and I cannot wait till I see your code :)
\n\nIf you are interested in using your two level stacking on some of our models and see what the results could have been, we could try that sometime in the future once the dusts settle and we all get our code up on github.
", "votes": null, "replies": [] }, { "id": 91289, "author_name": "bitsofbits", "author_url": "", "post_date": "09/02/2015 00:33:56", "content": "@blaine, I look forward to getting a look at your RNN. I really wanted to try that since it seems like a natural fit for this problem. However, I've never set one of those up before and so decided discretion was the better part of valor and went with what I knew.
\n\nAnyway, it was great fun and I enjoyed it a lot. It would indeed be interesting to see what would happen if our NN models got thrown into your machine. After the just settles a bit though, we have a bunch of clean up to do first!
", "votes": null, "replies": [] }, { "id": 91300, "author_name": "sudalairajkumar", "author_url": "", "post_date": "09/02/2015 04:17:49", "content": "Congrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot !
\n\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on "Kaggle Master" badge.
\n\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.!
", "votes": null, "replies": [] }, { "id": 91305, "author_name": "elenacuoco", "author_url": "", "post_date": "09/02/2015 07:47:00", "content": "[quote=SRK;91300]
\n\nCongrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot !
\n\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on "Kaggle Master" badge.
\n\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.!
\n\n[/quote]\nThanks a lot SRK! I'm happy to have been useful.
", "votes": null, "replies": [] }, { "id": 91306, "author_name": "elenacuoco", "author_url": "", "post_date": "09/02/2015 07:51:21", "content": "[quote=rcarson;91186]
\n\nCongratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.
\n\nOur best "single" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.
\n\n[/quote]\nYou are welcome rcarson! It is useful to read how you used my script, and I'm happy for your results. I did not succeed in obtaining very good results with my script. I was so concentrated in preprocessing the data, remove artifact, avoid aliasing and so on, that I did not explore other ways to work with those data until Tim posted his script and showed how NN worked so well with data with his idea of CNN.
", "votes": null, "replies": [] }, { "id": 91307, "author_name": "elenacuoco", "author_url": "", "post_date": "09/02/2015 07:55:11", "content": "[quote=Tim Hochberg;91244]
\n\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!
\n\n[/quote]\nThanks to you Tim! It was really a pleasure to be part of your team and learn so much from you. Thanks to all the team too!and congratulations.
", "votes": null, "replies": [] }, { "id": 91311, "author_name": "joseleiva", "author_url": "", "post_date": "09/02/2015 09:40:05", "content": "Congratulations to the winners,
\n\nI'm surprised at the fact that, apparently, all the top-performers used a pure ML approach.
\n\nI didn't have enough spare time to participate in the competition,but if I had, I would've used an approach based on a finite-state machine: since the sequences of events is known in advance, this fact imposes some constraints in the temporal sequence of labels. This could've been use to refine the initial predictions based on independent predictors for each labels, by running a forward-backward algorithm - or, since the "backward" thing is forbidden by the rules, just the "forward" step.
\n\nWhat do you think? Did anybody try this approach?
", "votes": null, "replies": [] }, { "id": 91321, "author_name": "rafalcycon", "author_url": "", "post_date": "09/02/2015 11:19:57", "content": "@Jose, I think that's a good idea, too bad you didn't have time to try it out. As for capturing the temporal sequence of labels - this is exactly the reasoning that led us to use Recurrent Neural Nets. They were extremely effective, but we had success with them rather on metafeatures than on raw data (they were very computationally intensive on raw data and required much tuning, so at some point we decided to focus our CPU/GPU power on other things).
\n\nWe chose RNNs over HMMs (where HMM is very close to being a FSM), because RNNs have a bigger learning capacity and can model complicated nonlinear dynamics, e.g. AFAIK they achieve better results in speech recognition tasks (e.g. last page of this article). However, we haven't done any experiments with FSMs/HMMs on this data, so they might have as well turned out to be superior than RNNs in this setting.
", "votes": null, "replies": [] }, { "id": 91358, "author_name": "jiweiliu", "author_url": "", "post_date": "09/02/2015 15:46:20", "content": "@deep @Elena Cuoco @SkyLibrary, congratulations! See you in the master competition :D
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "90198": "What's your single model performance so far? Let's hear it for those who don't mind.", "90221": "Currently at .94300. \r\n\r\nModification of Tim h.'s NN with some light feature engineering. Big issue for me seems to be performance. Trying to work on that now actually (after a vacation break)", "91079": "jamesdipadua, sorry to leave you hanging like, we've been buried under a pile neural nets and I haven't had a time to check the forum much. Anyway, our best single model scores a bit over 0.971. It shares a common ancestor with the code I posted, but the important parts are not very similar to that code any more.", "91080": "oh, no worries tim. i think what was needed was a bit of feature creation. not sure if that's what you mean by \"important parts.\" \r\n\r\nwas never really able to get the performance where i wanted (but went to croatia for a vacation and worked on some other projects).", "91099": "All will be revealed shortly, at least if we didn't dramatically overfit on public leaderboard, but I suspect you'll be surprised....", "91102": "You have my curiosity ! \r\n\r\nWill we be surprised by your single model ranking above 0.98 or by your thousands of weak models ?", "91121": "Actually, neither of the above, we had a few dozen intermediate (~0.97) models, although we would have trained more if we had time. But what I was referring to was James' suspicion that our secret was some sort of hybrid feature engineering / neural net scheme. \r\n\r\nWe *wanted* to try some fancy preprocessing, but we ran out of time, so the only preprocessing we did was some light filtering to take out the very low and very high frequencies. Other than that the models are just neural nets. So what's surprising is, in some sense, how boring our models are....\r\n\r\nI did figure out a couple of neat tricks on the neural net end that improved the results quite a bit. But most of the work was just tuning. And then more tuning...", "91123": "Congrats everyone. Not much LB movement (we didn't move at all) and congrats to Sky Library who has become a Master", "91168": "Hey,\r\n\r\nCongratulations! Thank you all very much for this awesome challenge :-)\r\n\r\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)\r\n\r\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.", "91178": "blaine: Congratulations! That's very gratifying that you ended up using some of my code; it makes me feel better about all the ideas I ended up borrowing from Alexandre's posts.\r\n\r\nSuddenly I'm feeling pretty good about our best single model. Now that I can see the private leaderboard scores I can see that the score for our best single model held up pretty well: 0.97117 / 0.97088 public/private leaderboard. It'll be interesting to see if there are any single models out there that beat that.", "91186": "Congratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.\r\n\r\nOur best \"single\" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.\r\n\r\nThe observation is that events are inter-dependent and temporally dependent, which is quite obvious. Please find our code on github, [here][1]\r\n\r\n\r\n [1]: https://github.com/daxiongshu/Grasp-and-Lift", "91202": "Congrats to the winners, We learned a lot from your top players script and discussion. I am also glad that several kagglers become kaggle master after this competition.\r\n\r\n@blaine Thank you, it's very nice of you to say that :)\r\n\r\n@Bluefool Thank you for being an awesome teammate~\r\n\r\n[quote=blaine;91168]\r\n\r\nCongratulations! Thank you all very much for this awesome challenge :-)\r\n\r\n@Skylibrary, congrats on a well deserved Master badge! I saw your disappointment in crowdflower, I'm really glad you made it :-)\r\n\r\nOur best single model was ~0.95 public/private LB, and it was Tim's Neural Net script with Bluefool's additions (thank you both very much for sharing!) and with lowpass filtering at 30Hz and 2D convolution across all electrodes simultaneously (i.e. filter size (NUM_ELECTRODES, X )). Very close to that was a Standard Scaler -> Linear Regression model trained on features obtained from Riemannian geometry along with raw data processed with a Filter Bank composed of many lowpass filters.\r\n\r\n[/quote]", "91221": "The best single model of our team was 0.964. \r\n\r\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC.", "91239": "[quote=Guocong Song;91221]\r\n\r\nThe best single model of our team was 0.964. \r\n\r\n@Tim, your ordinal regression objective function works very well for AUC. I used a multi-class objective function as well, which was very close to but statistically lower than the ordinal regression in terms of AUC. \r\n\r\n[/quote]\r\nCould you explain what you refer with \"ordinal regression objective \" and \"multi-class objective function\"? Are the Mean Squared Error and CrossEntropy?", "91244": "Guocong Song, that's interesting. I experimented a little bit with using a multi-class objective function early on, but it had the nasty habit of making my more \"interesting\" nets become unstable. So, I stopped using it. I always intended to come back and give it another try at some point, but there never enough time!\r\n\r\n@rcarson, Thanks for the compliment, but I don't think I qualify as a NN Master. I'm working on it, but there's still a ton I don't know! I'm glad a lot of people got some use out my code though; that's always gratifying. Also, I'm always impressed by schemes where one stacks a bunch of predictors up into something strong. You have my respect sir.\r\n\r\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!", "91247": "Incalo, FWIW, I interpreted that as binary cross entropy combined with sigmoid nonlinearity on the output in the first case (since that is what I did). This allows the net to predict the values of the different activities independently. However the activities were not independent, there were only 12 or so different combinations. So one could instead encode the 6 event values onto one of those twelve possible states. This has the effect of reducing the output space that one is trying to predict in from 2^6 (64) down to about 12, reducing the problem space considerably. One would use categorical cross entropy combined with a softmax output function in this case. As I said, I had only so-so results with this, but wanted to try it again.\r\n\r\nOf course I may be totally misrepresenting what @Guocong Song meant, but that's what I tried early on.", "91252": "Thank you Tim. You have a wonderful team : ) \r\nI hope to see your winning code soon. Best regards.", "91265": "Tim you rightfully feel good about your single model, it's very impressive. As well as rcarson's and Guocong's, very solid scores.\r\n\r\nIt's hard to call them \"single models\" (I apologize for a small offtopic then), but in our case the true workhorses of the solution were Recurrent Neural Nets and XGBoosts trained on 1st level metafeatures (i.e. outputs of models trained on raw data). We had lots of diversity in metafeatures, so 2nd level models could truly shine here. \r\n\r\nRecurrent Neural Nets naturally take into consideration time progression of data, which allowed the model to infer the sequence of events. Of course we trained them to output predictions of all events simultaneously, so they also capture dependencies between output classes. Our best not bagged RNN scores 0.97946 public/private (very stable). Unfortunately we didn't have much luck (and patience :-) ) in training/tuning a well-scoring RNN on raw data.\r\n\r\nAs for XGBoosts, we added a time-unrolling preprocessing step - we used past samples along with the current time sample to let it also capture sequence of events, and they were trained per-event. Best not bagged XGB scores 0.9756 private / 0.9764 public.\r\n\r\nOn a side note, probably inputting your powerful single models as metafeatures to our RNNs and XGBs could give excellent AUC :-)", "91281": "Congrats everyone! The results of everyone here are pretty impressive! The methods are even more stellar!\r\n\r\n@rcarson, your effort of stacking many first level predictors and turn them in to such strong ensemble is quite impressive! Our 'ensembles' were the good old weighted averaging :) and we didn't even try and train per subject although with proper caution it could have helped much.\r\n\r\n@blaine, your idea of using 3 layers of models is quite amazing! The use of RNNs and XGB in that manner seems quite intesresting and I cannot wait till I see your code :) \r\n\r\nIf you are interested in using your two level stacking on some of our models and see what the results could have been, we could try that sometime in the future once the dusts settle and we all get our code up on github.", "91289": "blaine, I look forward to getting a look at your RNN. I really wanted to try that since it seems like a natural fit for this problem. However, I've never set one of those up before and so decided discretion was the better part of valor and went with what I knew. \r\n\r\nAnyway, it was great fun and I enjoyed it a lot. It would indeed be interesting to see what would happen if our NN models got thrown into your machine. After the just settles a bit though, we have a bunch of clean up to do first!", "91300": "Congrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot ! \r\n\r\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on \"Kaggle Master\" badge. \r\n\r\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.!", "91305": "[quote=SRK;91300]\r\n\r\nCongrats @Alexandre and @blaine. You were really helpful throughout the competition. Thanks @Tim, @Bluefool and @Anil for the NN scripts. I finally started learning neural nets using your scripts. Thanks @Elena for the wonderful code using scikit learn. It helped a lot ! \r\n\r\nCongrats @SkyLibrary, @Tim, @Deep, @Elena, @Plankton, @Sungbin Choi on \"Kaggle Master\" badge. \r\n\r\nFinally, thanks to my wonderful team mate @rcarson for helping me learn a lot of new things during the competition and the fifth place finish.! \r\n\r\n[/quote]\r\nThanks a lot SRK! I'm happy to have been useful.", "91306": "[quote=rcarson;91186]\r\n\r\nCongratulations to everyone! Especially to the EEG master Alexandre Barachant & blaine and NN master Tim Hochberg.\r\n\r\nOur best \"single\" model is 0.970, which uses spatial and temporal stacking. For each subject, we trained 32*6*6 linear models (the benchmark code, thank you Elena!), where we used low pass filtering with 32 electrodes separately, 6 different classifiers, and for 6 events. Then we use a xgb to train these 32*6*6 predictions as meta features for each subject. We called this step spatial stacking. Next we built temporal meta features based on the result of spatial stacking. The predictions of 6 events in the past 100 frames, ... 1000 frames can be a good feature. And we use xgb to stack them again.\r\n\r\n \r\n\r\n[/quote]\r\nYou are welcome rcarson! It is useful to read how you used my script, and I'm happy for your results. I did not succeed in obtaining very good results with my script. I was so concentrated in preprocessing the data, remove artifact, avoid aliasing and so on, that I did not explore other ways to work with those data until Tim posted his script and showed how NN worked so well with data with his idea of CNN.", "91307": "[quote=Tim Hochberg;91244]\r\n\r\n \r\n\r\nI'd like to give a shout out to my team-mates @Deep, @Jing and @Elena Cuoco , they were great to work with. @Deep in particular worked tirelessly to help get us up near the top. Three of use made Master in this contest too! So: thanks and congratulations to them!\r\n\r\n\r\n\r\n[/quote]\r\nThanks to you Tim! It was really a pleasure to be part of your team and learn so much from you. Thanks to all the team too!and congratulations.", "91311": "Congratulations to the winners, \r\n\r\nI'm surprised at the fact that, apparently, all the top-performers used a pure ML approach. \r\n\r\nI didn't have enough spare time to participate in the competition,but if I had, I would've used an approach based on a finite-state machine: since the sequences of events is known in advance, this fact imposes some constraints in the temporal sequence of labels. This could've been use to refine the initial predictions based on independent predictors for each labels, by running a forward-backward algorithm - or, since the \"backward\" thing is forbidden by the rules, just the \"forward\" step. \r\n\r\nWhat do you think? Did anybody try this approach?", "91321": "Jose, I think that's a good idea, too bad you didn't have time to try it out. As for capturing the temporal sequence of labels - this is exactly the reasoning that led us to use Recurrent Neural Nets. They were extremely effective, but we had success with them rather on metafeatures than on raw data (they were very computationally intensive on raw data and required much tuning, so at some point we decided to focus our CPU/GPU power on other things).\r\n\r\nWe chose RNNs over HMMs (where HMM is very close to being a FSM), because RNNs have a bigger learning capacity and can model complicated nonlinear dynamics, e.g. AFAIK they achieve better results in speech recognition tasks (e.g. [last page of this article][1]). However, we haven't done any experiments with FSMs/HMMs on this data, so they might have as well turned out to be superior than RNNs in this setting.\r\n\r\n [1]: http://www.cs.toronto.edu/~asamir/papers/icassp10.pdf", "91358": "deep @Elena Cuoco @SkyLibrary, congratulations! See you in the master competition :D" }, "source": "meta" }