{ "id": 16479, "title": "Cat & Dog solution ", "url": "/competitions/grasp-and-lift-eeg-detection/writeups/cat-dog-cat-dog-solution", "author_name": "", "post_date": "2015-09-14T15:20:31.917Z", "votes": 36, "comment_count": 8, "views": 3171, "content": "

Hi everyone,

\n\n

You will find our code and documentation for our solution on this repo

\n\n

The key of this challenge was ensembling. We ended up with a 3-levels pipeline :

\n\n\n\n

With such a high level of ensembling, a main concern was overfitting. Our two final pick were 2 submissions we called "YOLO" and "Safe1". The YOLO submission was our submission with the highest CV and scored 0.98128 / 0.98109. The Safe1 submission was optimized to reduce variance on the CV and was composed of a subset of lvl2 models hand-picked for their stability. This submission scored 0.98108 / 0.98095

\n\n

The code is quite computationaly expensive, and it will take you up to one week to generate the final results. \nOur solution is well applicable in real world and works in real time, provided that the patient with prosthesis is constantly connected to 15 Amazon high-end instances via Google Fiber.

\n\n

All jokes apart, the final solution is a bit overkill. Judging by a visual inspection of lvl2 predictions, all of the events should be correctly detected, and the lvl3 ensembling only help to increase the sharpness of the predictions (reduce error on the edges of the 300ms frames)

\n\n

We did not focus too much on producing very high scoring lvl1 models. Instead, we tried to produce a maximum of diversity in the lvl1 predictions. We stopped working on lvl1 models a couple of weeks before the deadline to focus entirely on lvl2 models and later on lvl3 models.

\n\n

RNNs turns out to be very effective on lvl2 ensembling, they were able to catch sequential structure of the events. However, they were not very successful on lvl1 features.

\n\n

Finally, we also tried semi-supervised adaptation without any success. Seems that the signals were stable enough across series to not have to rely on such procedure.

\n\n

Best,

\n\n

Cat & Dog

", "messages": [ { "id": "92482", "postDate": "09/14/2015 14:26:44", "content": "

Hi everyone,

\n\n

You will find our code and documentation for our solution on this repo

\n\n

The key of this challenge was ensembling. We ended up with a 3-levels pipeline :

\n\n\n\n

With such a high level of ensembling, a main concern was overfitting. Our two final pick were 2 submissions we called "YOLO" and "Safe1". The YOLO submission was our submission with the highest CV and scored 0.98128 / 0.98109. The Safe1 submission was optimized to reduce variance on the CV and was composed of a subset of lvl2 models hand-picked for their stability. This submission scored 0.98108 / 0.98095

\n\n

The code is quite computationaly expensive, and it will take you up to one week to generate the final results. \nOur solution is well applicable in real world and works in real time, provided that the patient with prosthesis is constantly connected to 15 Amazon high-end instances via Google Fiber.

\n\n

All jokes apart, the final solution is a bit overkill. Judging by a visual inspection of lvl2 predictions, all of the events should be correctly detected, and the lvl3 ensembling only help to increase the sharpness of the predictions (reduce error on the edges of the 300ms frames)

\n\n

We did not focus too much on producing very high scoring lvl1 models. Instead, we tried to produce a maximum of diversity in the lvl1 predictions. We stopped working on lvl1 models a couple of weeks before the deadline to focus entirely on lvl2 models and later on lvl3 models.

\n\n

RNNs turns out to be very effective on lvl2 ensembling, they were able to catch sequential structure of the events. However, they were not very successful on lvl1 features.

\n\n

Finally, we also tried semi-supervised adaptation without any success. Seems that the signals were stable enough across series to not have to rely on such procedure.

\n\n

Best,

\n\n

Cat & Dog

", "rawMarkdown": "Hi everyone, \r\n\r\nYou will find our code and documentation for our solution on [this repo][1]\r\n\r\n\r\nThe key of this challenge was ensembling. We ended up with a 3-levels pipeline :\r\n\r\n- Level1 models are trained independently on each subjects, with features based on low-pass filtered signal and/or covariance matrices estimated in different frequency band. we have around 50 lvl1 models, with different combination of features and classifiers. Our best lvl1 model score around 0.95 LB and is an adaptation of Tim's CNN script. We had a couple of other models scoring around 0.94, and most of the other are in the 0.90-0.91 range.\r\n- Level2 models are trained globally (all subject put together) on predictions from lvl1 models. They are mainly based on XGB and Recurrent Neural Networks. We have around 30 of these models, and the highest scoring model is a RNN with 8s of history on the prediction and bagged with subset of lvl1 model. it score around 0.98.\r\n- Level3 models are just different weighted mean of lvl2 model where the weights are optimized to maximize AUC.\r\n\r\nWith such a high level of ensembling, a main concern was overfitting. Our two final pick were 2 submissions we called \"YOLO\" and \"Safe1\". The YOLO submission was our submission with the highest CV and scored 0.98128 / 0.98109. The Safe1 submission was optimized to reduce variance on the CV and was composed of a subset of lvl2 models hand-picked for their stability. This submission scored 0.98108 / 0.98095\r\n\r\nThe code is quite computationaly expensive, and it will take you up to one week to generate the final results. \r\nOur solution is well applicable in real world and works in real time, provided that the patient with prosthesis is constantly connected to 15 Amazon high-end instances via Google Fiber.\r\n\r\nAll jokes apart, the final solution is a bit overkill. Judging by a visual inspection of lvl2 predictions, all of the events should be correctly detected, and the lvl3 ensembling only help to increase the sharpness of the predictions (reduce error on the edges of the 300ms frames)\r\n\r\nWe did not focus too much on producing very high scoring lvl1 models. Instead, we tried to produce a maximum of diversity in the lvl1 predictions. We stopped working on lvl1 models a couple of weeks before the deadline to focus entirely on lvl2 models and later on lvl3 models.\r\n\r\nRNNs turns out to be very effective on lvl2 ensembling, they were able to catch sequential structure of the events. However, they were not very successful on lvl1 features.\r\n\r\nFinally, we also tried semi-supervised adaptation without any success. Seems that the signals were stable enough across series to not have to rely on such procedure.\r\n\r\nBest,\r\n\r\nCat & Dog\r\n\r\n [1]: https://github.com/alexandrebarachant/Grasp-and-lift-EEG-challenge", "votes": null }, { "id": "92515", "postDate": "09/14/2015 16:49:13", "content": "

Congratulations and thanks for sharing. Yours nice job help me much!

", "rawMarkdown": "Congratulations and thanks for sharing. Yours nice job help me much!", "votes": null }, { "id": "92757", "postDate": "09/16/2015 13:43:59", "content": "

Thanks for your kindness of sharing these most precious resources, I will try my best to comprehend these~~~

", "rawMarkdown": "Thanks for your kindness of sharing these most precious resources, I will try my best to comprehend these~~~", "votes": null }, { "id": "92893", "postDate": "09/18/2015 12:02:05", "content": "

Thank you Cat & Dog for posting your solution in such a well-documented and clean way!

\n\n

I've started reproducing your results. So far everything is working fine. I'll keep you posted.

\n\n

Best,\nKlaus

", "rawMarkdown": "Thank you Cat & Dog for posting your solution in such a well-documented and clean way!\r\n\r\nI've started reproducing your results. So far everything is working fine. I'll keep you posted.\r\n\r\nBest,\r\nKlaus", "votes": null }, { "id": "92900", "postDate": "09/18/2015 14:22:50", "content": "

Hi Klaus,\nYour feedback is appreciated :)\nWe spend a good amount of time to document the code and the solution, but considering the high number of model we developed, it wasn't possible to detail each of them. If you have any question, or parts of the code you think are difficult to understand, feel free to ask.

\n\n

Best, Alex

", "rawMarkdown": "Hi Klaus,\r\nYour feedback is appreciated :)\r\nWe spend a good amount of time to document the code and the solution, but considering the high number of model we developed, it wasn't possible to detail each of them. If you have any question, or parts of the code you think are difficult to understand, feel free to ask.\r\n\r\nBest, Alex", "votes": null }, { "id": "93075", "postDate": "09/21/2015 07:22:13", "content": "", "rawMarkdown": "", "votes": null }, { "id": "93689", "postDate": "09/30/2015 11:38:18", "content": "

Just to keep you updated: I've retrained most of the lvl1 models by now. Some of the convnets are still training, because Theano initially didn't use the GPU so they where really slow until I noticed and fixed this.

\n\n

So I hope to be able to reproduce your results by next week, and I will then run some further tests, and probably have some further questions.

", "rawMarkdown": "Just to keep you updated: I've retrained most of the lvl1 models by now. Some of the convnets are still training, because Theano initially didn't use the GPU so they where really slow until I noticed and fixed this.\r\n\r\nSo I hope to be able to reproduce your results by next week, and I will then run some further tests, and probably have some further questions.", "votes": null }, { "id": "96850", "postDate": "10/20/2015 18:01:32", "content": "

An excellent write-up on team Cat & Dog's background and approach is live here on no free hunch.

", "rawMarkdown": "An excellent write-up on team Cat & Dog's background and approach is live [here][1] on no free hunch.\r\n\r\n\r\n [1]: http://blog.kaggle.com/2015/10/12/grasp-and-lift-eeg-winners-interview-1st-place-cat-dog/", "votes": null }, { "id": "165271", "postDate": "03/04/2017 18:20:58", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing.", "votes": null } ], "comments": [ { "id": 92515, "author_name": "stepbystep", "author_url": "", "post_date": "09/14/2015 16:49:13", "content": "

Congratulations and thanks for sharing. Yours nice job help me much!

", "votes": null, "replies": [] }, { "id": 92757, "author_name": "columnchou", "author_url": "", "post_date": "09/16/2015 13:43:59", "content": "

Thanks for your kindness of sharing these most precious resources, I will try my best to comprehend these~~~

", "votes": null, "replies": [] }, { "id": 92893, "author_name": "qwlouse", "author_url": "", "post_date": "09/18/2015 12:02:05", "content": "

Thank you Cat & Dog for posting your solution in such a well-documented and clean way!

\n\n

I've started reproducing your results. So far everything is working fine. I'll keep you posted.

\n\n

Best,\nKlaus

", "votes": null, "replies": [] }, { "id": 92900, "author_name": "alexandrebarachant", "author_url": "", "post_date": "09/18/2015 14:22:50", "content": "

Hi Klaus,\nYour feedback is appreciated :)\nWe spend a good amount of time to document the code and the solution, but considering the high number of model we developed, it wasn't possible to detail each of them. If you have any question, or parts of the code you think are difficult to understand, feel free to ask.

\n\n

Best, Alex

", "votes": null, "replies": [] }, { "id": 93075, "author_name": "columnchou", "author_url": "", "post_date": "09/21/2015 07:22:13", "content": "", "votes": null, "replies": [] }, { "id": 93689, "author_name": "qwlouse", "author_url": "", "post_date": "09/30/2015 11:38:18", "content": "

Just to keep you updated: I've retrained most of the lvl1 models by now. Some of the convnets are still training, because Theano initially didn't use the GPU so they where really slow until I noticed and fixed this.

\n\n

So I hope to be able to reproduce your results by next week, and I will then run some further tests, and probably have some further questions.

", "votes": null, "replies": [] }, { "id": 96850, "author_name": "annavictoria", "author_url": "", "post_date": "10/20/2015 18:01:32", "content": "

An excellent write-up on team Cat & Dog's background and approach is live here on no free hunch.

", "votes": null, "replies": [] }, { "id": 165271, "author_name": "peacademy", "author_url": "", "post_date": "03/04/2017 18:20:58", "content": "

Thanks for sharing.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "92482": "Hi everyone, \r\n\r\nYou will find our code and documentation for our solution on [this repo][1]\r\n\r\n\r\nThe key of this challenge was ensembling. We ended up with a 3-levels pipeline :\r\n\r\n- Level1 models are trained independently on each subjects, with features based on low-pass filtered signal and/or covariance matrices estimated in different frequency band. we have around 50 lvl1 models, with different combination of features and classifiers. Our best lvl1 model score around 0.95 LB and is an adaptation of Tim's CNN script. We had a couple of other models scoring around 0.94, and most of the other are in the 0.90-0.91 range.\r\n- Level2 models are trained globally (all subject put together) on predictions from lvl1 models. They are mainly based on XGB and Recurrent Neural Networks. We have around 30 of these models, and the highest scoring model is a RNN with 8s of history on the prediction and bagged with subset of lvl1 model. it score around 0.98.\r\n- Level3 models are just different weighted mean of lvl2 model where the weights are optimized to maximize AUC.\r\n\r\nWith such a high level of ensembling, a main concern was overfitting. Our two final pick were 2 submissions we called \"YOLO\" and \"Safe1\". The YOLO submission was our submission with the highest CV and scored 0.98128 / 0.98109. The Safe1 submission was optimized to reduce variance on the CV and was composed of a subset of lvl2 models hand-picked for their stability. This submission scored 0.98108 / 0.98095\r\n\r\nThe code is quite computationaly expensive, and it will take you up to one week to generate the final results. \r\nOur solution is well applicable in real world and works in real time, provided that the patient with prosthesis is constantly connected to 15 Amazon high-end instances via Google Fiber.\r\n\r\nAll jokes apart, the final solution is a bit overkill. Judging by a visual inspection of lvl2 predictions, all of the events should be correctly detected, and the lvl3 ensembling only help to increase the sharpness of the predictions (reduce error on the edges of the 300ms frames)\r\n\r\nWe did not focus too much on producing very high scoring lvl1 models. Instead, we tried to produce a maximum of diversity in the lvl1 predictions. We stopped working on lvl1 models a couple of weeks before the deadline to focus entirely on lvl2 models and later on lvl3 models.\r\n\r\nRNNs turns out to be very effective on lvl2 ensembling, they were able to catch sequential structure of the events. However, they were not very successful on lvl1 features.\r\n\r\nFinally, we also tried semi-supervised adaptation without any success. Seems that the signals were stable enough across series to not have to rely on such procedure.\r\n\r\nBest,\r\n\r\nCat & Dog\r\n\r\n [1]: https://github.com/alexandrebarachant/Grasp-and-lift-EEG-challenge", "92515": "Congratulations and thanks for sharing. Yours nice job help me much!", "92757": "Thanks for your kindness of sharing these most precious resources, I will try my best to comprehend these~~~", "92893": "Thank you Cat & Dog for posting your solution in such a well-documented and clean way!\r\n\r\nI've started reproducing your results. So far everything is working fine. I'll keep you posted.\r\n\r\nBest,\r\nKlaus", "92900": "Hi Klaus,\r\nYour feedback is appreciated :)\r\nWe spend a good amount of time to document the code and the solution, but considering the high number of model we developed, it wasn't possible to detail each of them. If you have any question, or parts of the code you think are difficult to understand, feel free to ask.\r\n\r\nBest, Alex", "93075": "", "93689": "Just to keep you updated: I've retrained most of the lvl1 models by now. Some of the convnets are still training, because Theano initially didn't use the GPU so they where really slow until I noticed and fixed this.\r\n\r\nSo I hope to be able to reproduce your results by next week, and I will then run some further tests, and probably have some further questions.", "96850": "An excellent write-up on team Cat & Dog's background and approach is live [here][1] on no free hunch.\r\n\r\n\r\n [1]: http://blog.kaggle.com/2015/10/12/grasp-and-lift-eeg-winners-interview-1st-place-cat-dog/", "165271": "Thanks for sharing." }, "source": "existing_discussion", "complete": true, "competition_id": 4477, "competition_slug": "grasp-and-lift-eeg-detection", "team_id": 197978, "solution_url": "/competitions/grasp-and-lift-eeg-detection/writeups/cat-dog-cat-dog-solution" }