![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3147995%2F970876bd40cd74ade6b68a7ea415c349%2FCaptura%20desde%202023-12-06%2001-35-18.png?generation=1701823419282889&alt=media\")
Congratulations to kaggle and the host for this competition, it has been a very interesting and fun competition. Before we start, special thanks to @kyakovlev, it has been exceptional to be able to work with you and learn from you.
\n1. Preprocessing
\n2. Training (GRU + UNET + LGB)
\n3. Inference and postprocess
For final submission our GRU + UNET models work just with 7 features. We tried to add more features but for local validation we did not find any that would work.
\nWe decided to divide the series into one-day sequences and reduce the granularity from 5 secs to 30 secs. Therefore we had sequences of 2880 length in which there was normally an onset and a wakeup. Here is an standard input for our model.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3147995%2F4d5bb92e188eb1a4e751d18505ffa7ff%2Fstandard_series.png?generation=1701879825870419&alt=media\")
Make anglez absolute, this was giving +0.002 on local validation.
For the only two variables we had (anglez and enmo), we tried to find useful aggregations(diff, mean, median, skew, etc…), but the only thing that seemed to work was the standard deviation (anglez_abs_std and enmo_std).
Detecting noise was another of the key points of our solution. We realized that when exactly the same value is repeated in the same series at the same hour, minute and second, this was basically noise. Here our detected noise is the red line.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3147995%2F4926d54727c25100d547d5656cbc3667%2F04f547b8017d.png?generation=1701879870683731&alt=media\")
To incorporate temporal information into the model, we decided to add 2 frequency encoding variables (one for onsets and one for wakeups) at the hour-minute level.
A good augmentation trick was to reverse all the series during training, this allowed us to have more sequences and increased our local validation by 0.01
The public leaderboard has really been making us dizzy throughout the competition, we thought there was a bug in our code. So halfway through the competition we did a study of how each variable was working on the leaderboard and these are the results:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3147995%2F43c76444fe8e5fc6a312bced2d320ddb%2FCaptura%20desde%202023-12-06%2017-26-09.png?generation=1701880005278625&alt=media\")
Our best model was GRU, which alone gave us a local validation of 0.835
\nIt seems that after the competition is over, we can observe that the private is more correlated with our local validation than the public one. In the public there was an instability that was consuming us because we really thought we had a bug somewhere in our code.
\nOur best private submission is this ensemble: (GRU*0.68 + UNET * 0.2 + LGB*0.12)
\nCV: ~0.840 / public LB 0.784 / private LB 0.848
We've been trying to make lgb competitive since we formed a team. We finally managed to start getting him to contribute something to the ensemble the day before the competition ended. So I think that in this part we were a few days away from continuing to refine our LGB model.
\nAnyway, in our last submission we can see how it was giving us +0.002 in the private LB.
\nThe maximum we achieved with an LGB single was 0.757 public / 0.82 private.
For the best submission that we finally chose, we made an ensemble of several models (8 GRU and 2 UNET)
\nRunning time for inference was ~1,5 hours.
For postprocess we try several things, but basically what we did was keep the peaks every certain distance optimized for our predictions.
\nSurely there are things we are forgetting, so don't hesitate to ask anything and I'll update it in the post.
\nIt has been a type of problem that could be approached in many ways, as we are seeing in the different solutions. We would like to mention that it has been a really challenging competition where it was a constant battle with other teams that pushed us to the limit (everything was very tight). Apart from that, I hope we can help your mission and be useful to the host.
\nGithub code is here: https://github.com/FNoaGut/child-mind-institute-detect-sleep-states-3rd-place-solution
", "messages": [ { "id": 2550295, "postDate": "2023-12-06T00:46:40.577Z", "content": "Congratulations to kaggle and the host for this competition, it has been a very interesting and fun competition. Before we start, special thanks to @kyakovlev, it has been exceptional to be able to work with you and learn from you.
\n1. Preprocessing
\n2. Training (GRU + UNET + LGB)
\n3. Inference and postprocess
For final submission our GRU + UNET models work just with 7 features. We tried to add more features but for local validation we did not find any that would work.
\nWe decided to divide the series into one-day sequences and reduce the granularity from 5 secs to 30 secs. Therefore we had sequences of 2880 length in which there was normally an onset and a wakeup. Here is an standard input for our model.
\n
Make anglez absolute, this was giving +0.002 on local validation.
For the only two variables we had (anglez and enmo), we tried to find useful aggregations(diff, mean, median, skew, etc…), but the only thing that seemed to work was the standard deviation (anglez_abs_std and enmo_std).
Detecting noise was another of the key points of our solution. We realized that when exactly the same value is repeated in the same series at the same hour, minute and second, this was basically noise. Here our detected noise is the red line.
\n
To incorporate temporal information into the model, we decided to add 2 frequency encoding variables (one for onsets and one for wakeups) at the hour-minute level.
A good augmentation trick was to reverse all the series during training, this allowed us to have more sequences and increased our local validation by 0.01
The public leaderboard has really been making us dizzy throughout the competition, we thought there was a bug in our code. So halfway through the competition we did a study of how each variable was working on the leaderboard and these are the results:
\n
Our best model was GRU, which alone gave us a local validation of 0.835
\nIt seems that after the competition is over, we can observe that the private is more correlated with our local validation than the public one. In the public there was an instability that was consuming us because we really thought we had a bug somewhere in our code.
\nOur best private submission is this ensemble: (GRU*0.68 + UNET * 0.2 + LGB*0.12)
\nCV: ~0.840 / public LB 0.784 / private LB 0.848
We've been trying to make lgb competitive since we formed a team. We finally managed to start getting him to contribute something to the ensemble the day before the competition ended. So I think that in this part we were a few days away from continuing to refine our LGB model.
\nAnyway, in our last submission we can see how it was giving us +0.002 in the private LB.
\nThe maximum we achieved with an LGB single was 0.757 public / 0.82 private.
For the best submission that we finally chose, we made an ensemble of several models (8 GRU and 2 UNET)
\nRunning time for inference was ~1,5 hours.
For postprocess we try several things, but basically what we did was keep the peaks every certain distance optimized for our predictions.
\nSurely there are things we are forgetting, so don't hesitate to ask anything and I'll update it in the post.
\nIt has been a type of problem that could be approached in many ways, as we are seeing in the different solutions. We would like to mention that it has been a really challenging competition where it was a constant battle with other teams that pushed us to the limit (everything was very tight). Apart from that, I hope we can help your mission and be useful to the host.
\nGithub code is here: https://github.com/FNoaGut/child-mind-institute-detect-sleep-states-3rd-place-solution
", "rawMarkdown": "# 3rd place solution - GRU, UNET and LGB!\n\nCongratulations to kaggle and the host for this competition, it has been a very interesting and fun competition. Before we start, special thanks to @kyakovlev, it has been exceptional to be able to work with you and learn from you.\n\n## Structure of our approach\n\n**1. Preprocessing**\n**2. Training (GRU + UNET + LGB)**\n**3. Inference and postprocess**\n\n\n\n\n### 1. Preprocessing\nFor final submission our GRU + UNET models work just with 7 features. We tried to add more features but for local validation we did not find any that would work.\n\n#### 1.1. **How do we structure the series before introducing them into our model?**\nWe decided to divide the series into one-day sequences and reduce the granularity from 5 secs to 30 secs. Therefore we had sequences of 2880 length in which there was normally an onset and a wakeup. Here is an standard input for our model.\n\n\n\n\n#### 1.2. **Key points and features in preprocesing**\n\n- Make anglez absolute, this was giving +0.002 on local validation.\n\n- For the only two variables we had (anglez and enmo), we tried to find useful aggregations(diff, mean, median, skew, etc...), but the only thing that seemed to work was the standard deviation (**anglez_abs_std** and **enmo_std**).\n\n- Detecting noise was another of the key points of our solution. We realized that when exactly the same value is repeated in the same series at the same hour, minute and second, this was basically noise. Here our detected noise is the red line.\n\n\n- To incorporate temporal information into the model, we decided to add 2 frequency encoding variables (one for onsets and one for wakeups) at the hour-minute level.\t\n- A good augmentation trick was to reverse all the series during training, this allowed us to have more sequences and increased our local validation by 0.01\n\n#### 1.3. **Small ablation study of single model in the middle of the competition**\nThe public leaderboard has really been making us dizzy throughout the competition, we thought there was a bug in our code. So halfway through the competition we did a study of how each variable was working on the leaderboard and these are the results:\n\n\n\n\n### 2. Training\nOur best model was GRU, which alone gave us a local validation of **0.835**\n\n#### 2.1. **GRU and UNET -- training details** (we did the training similar)\n- **Augmentation**: just invert series\n- **Number of features**: 7 (anglez_std, enmo_std, noise_day_before, noise_day_after, noise, hour_min_onset, hour_min_wakeup)\n- **Target**: 2 outputs (one for onsets and other for wakeups)\n- **Target transformation**: Add two steps back and one forward. (0,0,0,0,1,0,0,0 -> 0,0,1,1,1,1,0,0)\n- **loss** : cross-entropy\n\n#### 2.2. **Leaderboard variability**\nIt seems that after the competition is over, we can observe that the private is more correlated with our local validation than the public one. In the public there was an instability that was consuming us because we really thought we had a bug somewhere in our code.\nOur best private submission is this ensemble: (GRU*0.68 + UNET * 0.2 + LGB*0.12)\nCV: ~**0.840** / public LB **0.784** / private LB **0.848**\n\n\n\n#### 2.3. **LGB part**\nWe've been trying to make lgb competitive since we formed a team. We finally managed to start getting him to contribute something to the ensemble the day before the competition ended. So I think that in this part we were a few days away from continuing to refine our LGB model.\n\nAnyway, in our last submission we can see how it was giving us **+0.002** in the private LB.\nThe maximum we achieved with an LGB single was **0.757** public / **0.82** private.\n\n\n### 3. Inference and posprocess\n\nFor the best submission that we finally chose, we made an ensemble of several models (**8** GRU and **2** UNET)\nRunning time for inference was ~**1,5** hours.\n\nFor postprocess we try several things, but basically what we did was keep the peaks every certain distance optimized for our predictions.\n\n\n\n### 4. Conclusions\n\nSurely there are things we are forgetting, so don't hesitate to ask anything and I'll update it in the post.\n\nIt has been a type of problem that could be approached in many ways, as we are seeing in the different solutions. We would like to mention that it has been a really challenging competition where it was a constant battle with other teams that pushed us to the limit (everything was very tight). Apart from that, I hope we can help your mission and be useful to the host.\n\nGithub code is here: https://github.com/FNoaGut/child-mind-institute-detect-sleep-states-3rd-place-solution\n", "votes": 77 }, { "id": 2550314, "postDate": "2023-12-06T01:09:50.077Z", "content": "Can’t wait to see this one that LGBM is ridiculous hahaha also great idea to invert series for more data!
", "rawMarkdown": "Can’t wait to see this one that LGBM is ridiculous hahaha also great idea to invert series for more data!", "votes": 3 }, { "id": 2554752, "postDate": "2023-12-09T11:20:18.873Z", "content": "Congrats! May I have a look at the code that does the noise detection job at 1.2? Thank you :)
", "rawMarkdown": "Congrats! May I have a look at the code that does the noise detection job at 1.2? Thank you :)", "votes": 1, "replies": [ { "id": 2557127, "postDate": "2023-12-11T08:18:28.323Z", "content": "` ### Detecting noise through same values in same st_hour and st_step for same series
\n ##############################################################
\n # repeat: number of times that the value appears in the series
\n # noise: binary feature is 1 when repeat>1
\n # group_len: number of consecutive values with noise
\n # noise_removal: binary feat is 1 when is noise==1 and group_len>360
temp_df = (series.group_by([,,,]).\n agg(pl.col().count().()))\n\nseries = series.(temp_df, =[,,,], how=)\n\nseries = series.with_columns([\n pl.(pl.col()>).().otherwise().cast(pl.Int32).(),\n])\n\nprint()`\nCongratulations on securing the 3rd position in this competition. Thanks for sharing very interesting approach with clever techniques.
", "rawMarkdown": "Congratulations on securing the 3rd position in this competition. Thanks for sharing very interesting approach with clever techniques. ", "votes": 1 }, { "id": 2551884, "postDate": "2023-12-07T03:22:15.890Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!", "votes": 1, "replies": [ { "id": 2551887, "postDate": "2023-12-07T03:30:31.960Z", "content": "How did you sample from the long series?
", "rawMarkdown": "How did you sample from the long series?", "replies": [ { "id": 2552093, "postDate": "2023-12-07T07:24:19.840Z", "content": "Thank you! Same as short series. We cut them into sequences of days.
", "rawMarkdown": "Thank you! Same as short series. We cut them into sequences of days.", "votes": 1 }, { "id": 2552585, "postDate": "2023-12-07T15:39:25.880Z", "content": "Thank you for answering.
\n\n\nwe had sequences of 2880 length in which there was normally an onset and a wakeup.
\n
This means that your input data which usually include both an onset and a wakeup within a day is sampled but data except for the data every 30 sec from the beginning is removed?
", "rawMarkdown": "Thank you for answering. \n>we had sequences of 2880 length in which there was normally an onset and a wakeup.\n\nThis means that your input data which usually include both an onset and a wakeup within a day is sampled but data except for the data every 30 sec from the beginning is removed?" }, { "id": 2553915, "postDate": "2023-12-08T16:02:15.087Z", "content": "We decided to create sequences of days starting at 17:00 local time. If one day it was not complete at the beginning or at the end, we added padding.
", "rawMarkdown": "We decided to create sequences of days starting at 17:00 local time. If one day it was not complete at the beginning or at the end, we added padding.", "votes": 1 }, { "id": 2556790, "postDate": "2023-12-11T01:40:34.507Z", "content": "Thank you. Fnoa.
", "rawMarkdown": "Thank you. Fnoa." } ] } ] }, { "id": 2551385, "postDate": "2023-12-06T17:02:43.220Z", "content": "Good luck everybody
", "rawMarkdown": "*Good luck everybody*", "votes": 1 }, { "id": 2550948, "postDate": "2023-12-06T12:23:24.317Z", "content": "Congratulations guys!
", "rawMarkdown": "Congratulations guys!", "votes": 1, "replies": [ { "id": 2551214, "postDate": "2023-12-06T15:03:42.433Z", "content": "Thanks Alejandro!
", "rawMarkdown": "Thanks Alejandro!" } ] }, { "id": 2550418, "postDate": "2023-12-06T02:55:48.703Z", "content": "Clean approach. Well deserved 👏
", "rawMarkdown": "Clean approach. Well deserved 👏 ", "votes": 1 }, { "id": 2550331, "postDate": "2023-12-06T01:28:35.023Z", "content": "Amazing result, congratulations!!
", "rawMarkdown": "Amazing result, congratulations!!", "votes": 1, "replies": [ { "id": 2550335, "postDate": "2023-12-06T01:31:36.457Z", "content": "Thanks Javier!!!
", "rawMarkdown": "Thanks Javier!!!" } ] }, { "id": 2556754, "postDate": "2023-12-11T00:55:26.997Z", "content": "Congratulations @trasibulo and @kyakovlev on fantastic cash gold 3rd place finish.
\nI love how simple and elegant your solution is. Your two deep learning models have few features and basically no post process and achieve top 5 by themselves. Your LGBM achieves an impressive CV LB and I'm excited to see that GBDT boosted the ensemble of deep learning.
\nHow did you ensemble the models? Specifically how did you pick the final prediction step value and how did you pick the final prediction score value?
Thank you Chris! Very nice read this from you.
\nOur way of ensembling results is very simple, since it is normally a pure average between the raw predictions of each model. Only at the end of the competition we started to perform some submissions with weighting ensemble.
\nOur submission that gives 0.846 private is a pure average of 8 GRU models and 2 UNET.
After making this ensemble we carry out post-processing, where we do:
\nI attach an image where you can see some raw predictions for one series_id and a specific night. And you can also see the result of our post-processing (the red dots).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3147995%2Fa2725411bf163205fbe316d4f98b5750%2FCaptura%20desde%202023-12-11%2009-37-14.png?generation=1702284315791504&alt=media\")
Using
\n\n\nRolling_mean(center=True) to smooth the predictions
\n
is smart. That simulates WBF. Without smoothing, then two peaks will not combine to give you the middle step (like WBF), but after smoothing, you get the middle step like WBF.
But in reality we only did the rolling_mean for the score, not for the steps. So it's a little different
", "rawMarkdown": "But in reality we only did the rolling_mean for the score, not for the steps. So it's a little different", "votes": 1 } ] } ] }, { "id": 2550299, "postDate": "2023-12-06T00:51:44.413Z", "content": "Great solution guys! Huge congrats!! Looking forward to read the details later on (would be awesome if you can attach some code as well 🥹)
", "rawMarkdown": "Great solution guys! Huge congrats!! Looking forward to read the details later on (would be awesome if you can attach some code as well 🥹)", "votes": 2, "replies": [ { "id": 2550305, "postDate": "2023-12-06T00:59:51.513Z", "content": "Many thanks @snnclsr!
", "rawMarkdown": "Many thanks @snnclsr! " } ] }, { "id": 2551798, "postDate": "2023-12-07T01:34:51.197Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 2552091, "postDate": "2023-12-07T07:23:29.813Z", "content": "Thank you! Yes, basically once we have made the predictions with GRU, UNET and LGB, we make an ensemble where we multiply each of them by a weight. We adjust these weights manually based on local validation.
", "rawMarkdown": "Thank you! Yes, basically once we have made the predictions with GRU, UNET and LGB, we make an ensemble where we multiply each of them by a weight. We adjust these weights manually based on local validation.", "replies": [ { "id": 2552100, "postDate": "2023-12-07T07:36:02.050Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2551745, "postDate": "2023-12-06T23:40:58.433Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2550314, "author_name": "Cody_Null", "author_url": "", "post_date": "2023-12-06T01:09:50.077000", "content": "Can’t wait to see this one that LGBM is ridiculous hahaha also great idea to invert series for more data!
", "votes": 3, "replies": [] }, { "id": 2554752, "author_name": "Feng Qilong", "author_url": "", "post_date": "2023-12-09T11:20:18.873000", "content": "Congrats! May I have a look at the code that does the noise detection job at 1.2? Thank you :)
", "votes": 1, "replies": [ { "id": 2557127, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-11T08:18:28.323000", "content": "` ### Detecting noise through same values in same st_hour and st_step for same series
\n ##############################################################
\n # repeat: number of times that the value appears in the series
\n # noise: binary feature is 1 when repeat>1
\n # group_len: number of consecutive values with noise
\n # noise_removal: binary feat is 1 when is noise==1 and group_len>360
temp_df = (series.group_by([,,,]).\n agg(pl.col().count().()))\n\nseries = series.(temp_df, =[,,,], how=)\n\nseries = series.with_columns([\n pl.(pl.col()>).().otherwise().cast(pl.Int32).(),\n])\n\nprint()`\nCongratulations on securing the 3rd position in this competition. Thanks for sharing very interesting approach with clever techniques.
", "votes": 1, "replies": [] }, { "id": 2551884, "author_name": "DongYK", "author_url": "", "post_date": "2023-12-07T03:22:15.890000", "content": "Congratulations!!
", "votes": 1, "replies": [ { "id": 2551887, "author_name": "DongYK", "author_url": "", "post_date": "2023-12-07T03:30:31.960000", "content": "How did you sample from the long series?
", "votes": 0, "replies": [ { "id": 2552093, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-07T07:24:19.840000", "content": "Thank you! Same as short series. We cut them into sequences of days.
", "votes": 1, "replies": [] }, { "id": 2552585, "author_name": "DongYK", "author_url": "", "post_date": "2023-12-07T15:39:25.880000", "content": "Thank you for answering.
\n\n\nwe had sequences of 2880 length in which there was normally an onset and a wakeup.
\n
This means that your input data which usually include both an onset and a wakeup within a day is sampled but data except for the data every 30 sec from the beginning is removed?
", "votes": 0, "replies": [] }, { "id": 2553915, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-08T16:02:15.087000", "content": "We decided to create sequences of days starting at 17:00 local time. If one day it was not complete at the beginning or at the end, we added padding.
", "votes": 1, "replies": [] }, { "id": 2556790, "author_name": "DongYK", "author_url": "", "post_date": "2023-12-11T01:40:34.507000", "content": "Thank you. Fnoa.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2551385, "author_name": "Md Nuruzzaman", "author_url": "", "post_date": "2023-12-06T17:02:43.220000", "content": "Good luck everybody
", "votes": 1, "replies": [] }, { "id": 2550948, "author_name": "delai50", "author_url": "", "post_date": "2023-12-06T12:23:24.317000", "content": "Congratulations guys!
", "votes": 1, "replies": [ { "id": 2551214, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-06T15:03:42.433000", "content": "Thanks Alejandro!
", "votes": 0, "replies": [] } ] }, { "id": 2550418, "author_name": "tomy4reel", "author_url": "", "post_date": "2023-12-06T02:55:48.703000", "content": "Clean approach. Well deserved 👏
", "votes": 1, "replies": [] }, { "id": 2550331, "author_name": "Javier Martín", "author_url": "", "post_date": "2023-12-06T01:28:35.023000", "content": "Amazing result, congratulations!!
", "votes": 1, "replies": [ { "id": 2550335, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-06T01:31:36.457000", "content": "Thanks Javier!!!
", "votes": 0, "replies": [] } ] }, { "id": 2556754, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-12-11T00:55:26.997000", "content": "Congratulations @trasibulo and @kyakovlev on fantastic cash gold 3rd place finish.
\nI love how simple and elegant your solution is. Your two deep learning models have few features and basically no post process and achieve top 5 by themselves. Your LGBM achieves an impressive CV LB and I'm excited to see that GBDT boosted the ensemble of deep learning.
\nHow did you ensemble the models? Specifically how did you pick the final prediction step value and how did you pick the final prediction score value?
Thank you Chris! Very nice read this from you.
\nOur way of ensembling results is very simple, since it is normally a pure average between the raw predictions of each model. Only at the end of the competition we started to perform some submissions with weighting ensemble.
\nOur submission that gives 0.846 private is a pure average of 8 GRU models and 2 UNET.
After making this ensemble we carry out post-processing, where we do:
\nI attach an image where you can see some raw predictions for one series_id and a specific night. And you can also see the result of our post-processing (the red dots).
\n
Using
\n\n\nRolling_mean(center=True) to smooth the predictions
\n
is smart. That simulates WBF. Without smoothing, then two peaks will not combine to give you the middle step (like WBF), but after smoothing, you get the middle step like WBF.
But in reality we only did the rolling_mean for the score, not for the steps. So it's a little different
", "votes": 1, "replies": [] } ] } ] }, { "id": 2550299, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-12-06T00:51:44.413000", "content": "Great solution guys! Huge congrats!! Looking forward to read the details later on (would be awesome if you can attach some code as well 🥹)
", "votes": 2, "replies": [ { "id": 2550305, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-06T00:59:51.513000", "content": "Many thanks @snnclsr!
", "votes": 0, "replies": [] } ] }, { "id": 2551798, "author_name": "", "author_url": "", "post_date": "2023-12-07T01:34:51.197000", "content": "", "votes": 1, "replies": [ { "id": 2552091, "author_name": "Fnoa", "author_url": "", "post_date": "2023-12-07T07:23:29.813000", "content": "Thank you! Yes, basically once we have made the predictions with GRU, UNET and LGB, we make an ensemble where we multiply each of them by a weight. We adjust these weights manually based on local validation.
", "votes": 0, "replies": [ { "id": 2552100, "author_name": "", "author_url": "", "post_date": "2023-12-07T07:36:02.050000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2551745, "author_name": "", "author_url": "", "post_date": "2023-12-06T23:40:58.433000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2550295": "# 3rd place solution - GRU, UNET and LGB!\n\nCongratulations to kaggle and the host for this competition, it has been a very interesting and fun competition. Before we start, special thanks to @kyakovlev, it has been exceptional to be able to work with you and learn from you.\n\n## Structure of our approach\n\n**1. Preprocessing**\n**2. Training (GRU + UNET + LGB)**\n**3. Inference and postprocess**\n\n\n\n\n### 1. Preprocessing\nFor final submission our GRU + UNET models work just with 7 features. We tried to add more features but for local validation we did not find any that would work.\n\n#### 1.1. **How do we structure the series before introducing them into our model?**\nWe decided to divide the series into one-day sequences and reduce the granularity from 5 secs to 30 secs. Therefore we had sequences of 2880 length in which there was normally an onset and a wakeup. Here is an standard input for our model.\n\n\n\n\n#### 1.2. **Key points and features in preprocesing**\n\n- Make anglez absolute, this was giving +0.002 on local validation.\n\n- For the only two variables we had (anglez and enmo), we tried to find useful aggregations(diff, mean, median, skew, etc...), but the only thing that seemed to work was the standard deviation (**anglez_abs_std** and **enmo_std**).\n\n- Detecting noise was another of the key points of our solution. We realized that when exactly the same value is repeated in the same series at the same hour, minute and second, this was basically noise. Here our detected noise is the red line.\n\n\n- To incorporate temporal information into the model, we decided to add 2 frequency encoding variables (one for onsets and one for wakeups) at the hour-minute level.\t\n- A good augmentation trick was to reverse all the series during training, this allowed us to have more sequences and increased our local validation by 0.01\n\n#### 1.3. **Small ablation study of single model in the middle of the competition**\nThe public leaderboard has really been making us dizzy throughout the competition, we thought there was a bug in our code. So halfway through the competition we did a study of how each variable was working on the leaderboard and these are the results:\n\n\n\n\n### 2. Training\nOur best model was GRU, which alone gave us a local validation of **0.835**\n\n#### 2.1. **GRU and UNET -- training details** (we did the training similar)\n- **Augmentation**: just invert series\n- **Number of features**: 7 (anglez_std, enmo_std, noise_day_before, noise_day_after, noise, hour_min_onset, hour_min_wakeup)\n- **Target**: 2 outputs (one for onsets and other for wakeups)\n- **Target transformation**: Add two steps back and one forward. (0,0,0,0,1,0,0,0 -> 0,0,1,1,1,1,0,0)\n- **loss** : cross-entropy\n\n#### 2.2. **Leaderboard variability**\nIt seems that after the competition is over, we can observe that the private is more correlated with our local validation than the public one. In the public there was an instability that was consuming us because we really thought we had a bug somewhere in our code.\nOur best private submission is this ensemble: (GRU*0.68 + UNET * 0.2 + LGB*0.12)\nCV: ~**0.840** / public LB **0.784** / private LB **0.848**\n\n\n\n#### 2.3. **LGB part**\nWe've been trying to make lgb competitive since we formed a team. We finally managed to start getting him to contribute something to the ensemble the day before the competition ended. So I think that in this part we were a few days away from continuing to refine our LGB model.\n\nAnyway, in our last submission we can see how it was giving us **+0.002** in the private LB.\nThe maximum we achieved with an LGB single was **0.757** public / **0.82** private.\n\n\n### 3. Inference and posprocess\n\nFor the best submission that we finally chose, we made an ensemble of several models (**8** GRU and **2** UNET)\nRunning time for inference was ~**1,5** hours.\n\nFor postprocess we try several things, but basically what we did was keep the peaks every certain distance optimized for our predictions.\n\n\n\n### 4. Conclusions\n\nSurely there are things we are forgetting, so don't hesitate to ask anything and I'll update it in the post.\n\nIt has been a type of problem that could be approached in many ways, as we are seeing in the different solutions. We would like to mention that it has been a really challenging competition where it was a constant battle with other teams that pushed us to the limit (everything was very tight). Apart from that, I hope we can help your mission and be useful to the host.\n\nGithub code is here: https://github.com/FNoaGut/child-mind-institute-detect-sleep-states-3rd-place-solution\n", "2550314": "Can’t wait to see this one that LGBM is ridiculous hahaha also great idea to invert series for more data!", "2554752": "Congrats! May I have a look at the code that does the noise detection job at 1.2? Thank you :)", "2552195": "Congratulations on securing the 3rd position in this competition. Thanks for sharing very interesting approach with clever techniques. ", "2551884": "Congratulations!!", "2551385": "*Good luck everybody*", "2550948": "Congratulations guys!", "2550418": "Clean approach. Well deserved 👏 ", "2550331": "Amazing result, congratulations!!", "2556754": "Congratulations @trasibulo and @kyakovlev on fantastic cash gold 3rd place finish.\n\nI love how simple and elegant your solution is. Your two deep learning models have few features and basically no post process and achieve top 5 by themselves. Your LGBM achieves an impressive CV LB and I'm excited to see that GBDT boosted the ensemble of deep learning.\n\nHow did you ensemble the models? Specifically how did you pick the final prediction `step` value and how did you pick the final prediction `score` value? ", "2550299": "Great solution guys! Huge congrats!! Looking forward to read the details later on (would be awesome if you can attach some code as well 🥹)", "2551798": "", "2551745": "" } }