![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F5bcbf85dc70115602ffe62032b00192c%2Fimage.png?generation=1706561255845081&alt=media\")
First, I would like to thank the competition hosts for organizing such a very interesting competition and the kaggle team for facilitating it so smoothly. Also, congratulations to all the top winners.
\nAnd of course, a big thank you to my teammate @nikhilmishradev for competeing with me for the last 3 weeks. Without him, we could not have achieved this great result.
\nIn this post I will mainly explain the model I developed. Check out @nikhilmishradev 's post for the main stream (link).
\nMy solution is an ensemble of 4 NN models and 2 GBDT models. In all models, target is a gaussian with variance 36 steps. I explain in detail below.
\nThe input is a series of 19200 steps patched every 12 steps. In other words, they have 1600 (=19200//12) length and the shape is (bs, 1600, 12*num_features). Predictions are made by minute. This patching is important for training models on long series in a short time without overfitting. During training and inference, input series is shifted by 1/8 length of input. That is, each position is predicted approximately 8 times and averaged at the end. The input is transformed by transformer, wavenet or 1dcnn and finally GRU is applied. Single model CV of theese models are 0.805~0.814. Finally, I seed averaged predictions with 1~2 seeds.
\nThe input features are as follows:
\nstandardized_anglez and enmo
perfectly matched counts in 24-hour cycles
\nn_days = ((df) // DAY_STEPS) + \ndf[] = \n day (-n_days, n_days + ):\n day == :\n \n df[] = df[].diff(DAY_STEPS * day)\n df[] = df[].fillna()\n df[] += (df[] == ).astype()\ndf[] = (df[].clip(, ) - ) / \nlarge anglez.diff() or not (i.e. anglez.diff() > 5)
is_longest_sleep_episode and is_sleep_block from @tatamikenn ‘s notebook.
\nlag features of the above features
GBDT also makes a prediction every minute. The features created were mainly those used in above NNs, but it was especially important to add rich lag features. I trained XGBoost and LightGBM. Their CV are 0.7941 and 0.7934, respectively. The richer features improved these CVs to 0.7998 and 0.7978, but were not employed because the final CV did not change after ensemble.
\nIt is based on @tubotubo ‘s excellent repo. I added described features and sliding window. CV is 0.789.
\nThis part is important. While the task of this competition is essentially ranking, we can submit a large number of low confidence forecasts with no risk. My post-process is similar to NMS, but allows for predictions of non-peaks as well. Specifically, the process is as follows:
\nleave only the 12n+6 (n>=0) steps to eliminate the steps that are too close.
repeat the following steps until there are more than 1000 predictions for each series. Also initialize the reduce_rate array with [1, 1, …, 1].
Select the triplet of (step, event, score) with the highest confidence (=score / reduce_rate) and add it to submission.
add ((RANGE - |r|) / RANGE) ** EXP * COEFF(|r| < RANGE) to reduce_rate. (RANGE, EXP and COEFF are hyper params to be tuned.)
This process was further refined by Nikhil so his post processing was used in the final team submissions.
\nFinally, averaged preds with NN:GBDT:tubo = 5:2:3 weights is the output of my part. The final CV of my part is 0.825, public LB is 0.784 and privete LB is 0.840 with my post-process.
\netc…
\nThank you for reading.
\nEdit:
\nI published my training repositories.
First, I would like to thank the competition hosts for organizing such a very interesting competition and the kaggle team for facilitating it so smoothly. Also, congratulations to all the top winners.
\nAnd of course, a big thank you to my teammate @nikhilmishradev for competeing with me for the last 3 weeks. Without him, we could not have achieved this great result.
\nIn this post I will mainly explain the model I developed. Check out @nikhilmishradev 's post for the main stream (link).
\nMy solution is an ensemble of 4 NN models and 2 GBDT models. In all models, target is a gaussian with variance 36 steps. I explain in detail below.
\nThe input is a series of 19200 steps patched every 12 steps. In other words, they have 1600 (=19200//12) length and the shape is (bs, 1600, 12*num_features). Predictions are made by minute. This patching is important for training models on long series in a short time without overfitting. During training and inference, input series is shifted by 1/8 length of input. That is, each position is predicted approximately 8 times and averaged at the end. The input is transformed by transformer, wavenet or 1dcnn and finally GRU is applied. Single model CV of theese models are 0.805~0.814. Finally, I seed averaged predictions with 1~2 seeds.
\nThe input features are as follows:
\nstandardized_anglez and enmo
perfectly matched counts in 24-hour cycles
\nn_days = ((df) // DAY_STEPS) + \ndf[] = \n day (-n_days, n_days + ):\n day == :\n \n df[] = df[].diff(DAY_STEPS * day)\n df[] = df[].fillna()\n df[] += (df[] == ).astype()\ndf[] = (df[].clip(, ) - ) / \nlarge anglez.diff() or not (i.e. anglez.diff() > 5)
is_longest_sleep_episode and is_sleep_block from @tatamikenn ‘s notebook.
\nlag features of the above features
GBDT also makes a prediction every minute. The features created were mainly those used in above NNs, but it was especially important to add rich lag features. I trained XGBoost and LightGBM. Their CV are 0.7941 and 0.7934, respectively. The richer features improved these CVs to 0.7998 and 0.7978, but were not employed because the final CV did not change after ensemble.
\nIt is based on @tubotubo ‘s excellent repo. I added described features and sliding window. CV is 0.789.
\nThis part is important. While the task of this competition is essentially ranking, we can submit a large number of low confidence forecasts with no risk. My post-process is similar to NMS, but allows for predictions of non-peaks as well. Specifically, the process is as follows:
\nleave only the 12n+6 (n>=0) steps to eliminate the steps that are too close.
repeat the following steps until there are more than 1000 predictions for each series. Also initialize the reduce_rate array with [1, 1, …, 1].
Select the triplet of (step, event, score) with the highest confidence (=score / reduce_rate) and add it to submission.
add ((RANGE - |r|) / RANGE) ** EXP * COEFF(|r| < RANGE) to reduce_rate. (RANGE, EXP and COEFF are hyper params to be tuned.)
This process was further refined by Nikhil so his post processing was used in the final team submissions.
\nFinally, averaged preds with NN:GBDT:tubo = 5:2:3 weights is the output of my part. The final CV of my part is 0.825, public LB is 0.784 and privete LB is 0.840 with my post-process.
\netc…
\nThank you for reading.
\nEdit:
\nI published my training repositories.
Wow, great pipeline! Congrats to both of you guys! Especially the tabular part, I lost my hope on those models way too early after seeing the tubo's repo.
", "rawMarkdown": "Wow, great pipeline! Congrats to both of you guys! Especially the tabular part, I lost my hope on those models way too early after seeing the tubo's repo.", "votes": null }, { "id": "2550296", "postDate": "12/06/2023 00:46:55", "content": "Thanks! His model is very powerful and we learned a lot also from him!
", "rawMarkdown": "Thanks! His model is very powerful and we learned a lot also from him!", "votes": null }, { "id": "2551087", "postDate": "12/06/2023 13:55:30", "content": "Congratulations @PENGUIN46 !
", "rawMarkdown": "Congratulations @PENGUIN46 !", "votes": null }, { "id": "2552196", "postDate": "12/07/2023 09:30:18", "content": "Congratulations on securing fourth position in this competition. Thanks for sharing useful insights into the ML part.
", "rawMarkdown": "Congratulations on securing fourth position in this competition. Thanks for sharing useful insights into the ML part.", "votes": null } ], "comments": [ { "id": 2550286, "author_name": "snnclsr", "author_url": "", "post_date": "12/06/2023 00:33:56", "content": "Wow, great pipeline! Congrats to both of you guys! Especially the tabular part, I lost my hope on those models way too early after seeing the tubo's repo.
", "votes": null, "replies": [ { "id": 2550296, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "12/06/2023 00:46:55", "content": "Thanks! His model is very powerful and we learned a lot also from him!
", "votes": null, "replies": [] } ] }, { "id": 2551087, "author_name": "abhasmalguri", "author_url": "", "post_date": "12/06/2023 13:55:30", "content": "Congratulations @PENGUIN46 !
", "votes": null, "replies": [] }, { "id": 2552196, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "12/07/2023 09:30:18", "content": "Congratulations on securing fourth position in this competition. Thanks for sharing useful insights into the ML part.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2550272": "First, I would like to thank the competition hosts for organizing such a very interesting competition and the kaggle team for facilitating it so smoothly. Also, congratulations to all the top winners.\n\nAnd of course, a big thank you to my teammate @nikhilmishradev for competeing with me for the last 3 weeks. Without him, we could not have achieved this great result.\n\nIn this post I will mainly explain the model I developed. Check out @nikhilmishradev 's post for the main stream ([link](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459637)).\n\nMy solution is an ensemble of 4 NN models and 2 GBDT models. In all models, target is a gaussian with variance 36 steps. I explain in detail below.\n\n## NN Models\n\nThe input is a series of 19200 steps patched every 12 steps. In other words, they have 1600 (=19200//12) length and the shape is (bs, 1600, 12*num_features). Predictions are made by minute. This patching is important for training models on long series in a short time without overfitting. During training and inference, input series is shifted by 1/8 length of input. That is, each position is predicted approximately 8 times and averaged at the end. The input is transformed by transformer, wavenet or 1dcnn and finally GRU is applied. Single model CV of theese models are 0.805~0.814. Finally, I seed averaged predictions with 1~2 seeds.\n\nThe input features are as follows:\n\n- standardized_anglez and enmo\n- perfectly matched counts in 24-hour cycles\n - This is useful for identifying deficiencies because the GGIR replaces deficiencies with the average of the same time on other days.\n \n ```python\n n_days = int(len(df) // DAY_STEPS) + 1\n df[\"same_count\"] = 0\n for day in range(-n_days, n_days + 1):\n if day == 0:\n continue\n df[\"_anglez_diff\"] = df[\"anglez\"].diff(DAY_STEPS * day)\n df[\"_anglez_diff\"] = df[\"_anglez_diff\"].fillna(1)\n df[\"same_count\"] += (df[\"_anglez_diff\"] == 0).astype(int)\n df[\"same_count\"] = (df[\"same_count\"].clip(0, 5) - 2.5) / 2.5\n ```\n \n- large anglez.diff() or not (i.e. `anglez.diff() > 5`)\n- is_longest_sleep_episode and is_sleep_block from @tatamikenn ‘s notebook.\n - https://www.kaggle.com/code/tatamikenn/sleep-hdcza-a-pure-heuristic-approach-lb-0-447\n- lag features of the above features\n\n\n\n## GBDT Models\n\nGBDT also makes a prediction every minute. The features created were mainly those used in above NNs, but it was especially important to add rich lag features. I trained XGBoost and LightGBM. Their CV are 0.7941 and 0.7934, respectively. The richer features improved these CVs to 0.7998 and 0.7978, but were not employed because the final CV did not change after ensemble.\n\n## tubo Models\n\nIt is based on @tubotubo ‘s excellent repo. I added described features and sliding window. CV is 0.789.\n\n## Post Processing\n\nThis part is important. While the task of this competition is essentially ranking, we can submit a large number of low confidence forecasts with no risk. My post-process is similar to NMS, but allows for predictions of non-peaks as well. Specifically, the process is as follows:\n\n1. leave only the 12n+6 (n>=0) steps to eliminate the steps that are too close.\n\n2. repeat the following steps until there are more than 1000 predictions for each series. Also initialize the reduce_rate array with [1, 1, ..., 1].\n\n3. Select the triplet of (step, event, score) with the highest confidence (=score / reduce_rate) and add it to submission.\n\n4. add `((RANGE - |r|) / RANGE) ** EXP * COEFF(|r| < RANGE)` to reduce_rate. (RANGE, EXP and COEFF are hyper params to be tuned.)\n\nThis process was further refined by Nikhil so his post processing was used in the final team submissions.\n\n## Ensemble\n\nFinally, averaged preds with NN:GBDT:tubo = 5:2:3 weights is the output of my part. The final CV of my part is 0.825, public LB is 0.784 and privete LB is 0.840 with my post-process.\n\n## Not Worked\n\n- longer series\n- BigBird-like sparse attention.\n- re-ranking by ranker\n - regresses on the number of matches with target directly, not on soft label targets.\n - I thought that close predictions could cooperatively modify their confidence, but the metric did not change.\n- data augmentation\n - copy&paste, time flip, etc…\n- hand labeling on unlabeled training data.\n- focal loss\n- pseudo labeling during submission\n\netc…\n\nThank you for reading.\n\n-----\n\nEdit:\nI published my training repositories.\n- https://github.com/penguin-prg/child-mind-institute-detect-sleep-states\n- https://github.com/penguin-prg/cmi-tubo-base", "2550286": "Wow, great pipeline! Congrats to both of you guys! Especially the tabular part, I lost my hope on those models way too early after seeing the tubo's repo.", "2550296": "Thanks! His model is very powerful and we learned a lot also from him!", "2551087": "Congratulations @PENGUIN46 !", "2552196": "Congratulations on securing fourth position in this competition. Thanks for sharing useful insights into the ML part." }, "source": "meta" }