![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1496178%2Fa090c848becdf28231300513b3bb6403%2Fenmo_vs_step_example.png?generation=1696529880697141&alt=media\")
Hello!
\nEDA of 'enmo' column in train_series shows that there are '1 077 877' rows with enmo values larger than 99th percentile. Such kind values look kike result ofdevice mistake.
\nWhat is the best way to deal with huge ENMO values? Change to a value of 99th percentile?
\n", "messages": [ { "id": "2468505", "postDate": "10/05/2023 14:54:44", "content": "Hello!
\nEDA of 'enmo' column in train_series shows that there are '1 077 877' rows with enmo values larger than 99th percentile. Such kind values look kike result ofdevice mistake.
\nWhat is the best way to deal with huge ENMO values? Change to a value of 99th percentile?
\n", "rawMarkdown": "Hello!\n\nEDA of 'enmo' column in train_series shows that there are '1 077 877' rows with enmo values larger than 99th percentile. Such kind values look kike result ofdevice mistake.\n\nWhat is the best way to deal with huge ENMO values? Change to a value of 99th percentile? \n\n[My EDA on 'enmo' values: ](https://www.kaggle.com/veronikasarikaya/it-s-a-wonderful-wonderful-sleep-eda/edit)", "votes": null }, { "id": "2468724", "postDate": "10/05/2023 18:29:23", "content": "Hi!
\nOf course by definition, we'd expect around 1 million enmos above 99th percentile since we have 100+ million valid enmo values. The table you have showing the g values for various activities is interesting. Very high g values don't have to be a mistake, they could happen for extreme activities like pounding on a table.
\nBut since we're interested in sleep times, it would seem OK to clip the values at some level, since enmo = 3 is not really twice as awake as enmo = 1.5. Based on this series here, I'd think clipping at 1.5 (or 1.0) would retain all the information needed for sleep prediction.
\n
@dan3dewey thank for your answer ! It helped me to choose the best way to deal with enmo outliers and to understand huge enmo values. I couldn't imagine what kind of activity may lead to enmo = 7 or 11 g. But now I see that these values are not the device mistakes.
", "rawMarkdown": "dan3dewey thank for your answer ! It helped me to choose the best way to deal with enmo outliers and to understand huge enmo values. I couldn't imagine what kind of activity may lead to enmo = 7 or 11 g. But now I see that these values are not the device mistakes.", "votes": null }, { "id": "2469561", "postDate": "10/06/2023 12:52:16", "content": "If your model doesn't like outliers, the QuantileTransformer in scikit-learn is an option.
", "rawMarkdown": "If your model doesn't like outliers, the QuantileTransformer in scikit-learn is an option.", "votes": null }, { "id": "2472232", "postDate": "10/07/2023 06:25:21", "content": "Thanks for the hint!
", "rawMarkdown": "Thanks for the hint!", "votes": null }, { "id": "2477695", "postDate": "10/11/2023 13:54:26", "content": "@dan3dewey , @veronikasarikaya : Veronika, Daniel, thank you for the suggested strategies to tackle the enmo outlier issues. I tried the recommendations you shared, and the best score improvement to at least two ML model architectures I tried were observed when enmo was upper-clipped at 1.0 (with the value of the 99th percentile to be the second best option).
\nThank you once again for the solid contributions to the body of knowledge for this contest!
", "rawMarkdown": "dan3dewey , @veronikasarikaya : Veronika, Daniel, thank you for the suggested strategies to tackle the *enmo* outlier issues. I tried the recommendations you shared, and the best score improvement to at least two ML model architectures I tried were observed when enmo was upper-clipped at 1.0 (with the value of the 99th percentile to be the second best option).\n\nThank you once again for the solid contributions to the body of knowledge for this contest!", "votes": null } ], "comments": [ { "id": 2468724, "author_name": "dan3dewey", "author_url": "", "post_date": "10/05/2023 18:29:23", "content": "Hi!
\nOf course by definition, we'd expect around 1 million enmos above 99th percentile since we have 100+ million valid enmo values. The table you have showing the g values for various activities is interesting. Very high g values don't have to be a mistake, they could happen for extreme activities like pounding on a table.
\nBut since we're interested in sleep times, it would seem OK to clip the values at some level, since enmo = 3 is not really twice as awake as enmo = 1.5. Based on this series here, I'd think clipping at 1.5 (or 1.0) would retain all the information needed for sleep prediction.
\n
@dan3dewey thank for your answer ! It helped me to choose the best way to deal with enmo outliers and to understand huge enmo values. I couldn't imagine what kind of activity may lead to enmo = 7 or 11 g. But now I see that these values are not the device mistakes.
", "votes": null, "replies": [ { "id": 2477695, "author_name": "gvyshnya", "author_url": "", "post_date": "10/11/2023 13:54:26", "content": "@dan3dewey , @veronikasarikaya : Veronika, Daniel, thank you for the suggested strategies to tackle the enmo outlier issues. I tried the recommendations you shared, and the best score improvement to at least two ML model architectures I tried were observed when enmo was upper-clipped at 1.0 (with the value of the 99th percentile to be the second best option).
\nThank you once again for the solid contributions to the body of knowledge for this contest!
", "votes": null, "replies": [] } ] } ] }, { "id": 2469561, "author_name": "tolgadincer", "author_url": "", "post_date": "10/06/2023 12:52:16", "content": "If your model doesn't like outliers, the QuantileTransformer in scikit-learn is an option.
", "votes": null, "replies": [ { "id": 2472232, "author_name": "veronikasarikaya", "author_url": "", "post_date": "10/07/2023 06:25:21", "content": "Thanks for the hint!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2468505": "Hello!\n\nEDA of 'enmo' column in train_series shows that there are '1 077 877' rows with enmo values larger than 99th percentile. Such kind values look kike result ofdevice mistake.\n\nWhat is the best way to deal with huge ENMO values? Change to a value of 99th percentile? \n\n[My EDA on 'enmo' values: ](https://www.kaggle.com/veronikasarikaya/it-s-a-wonderful-wonderful-sleep-eda/edit)", "2468724": "Hi!\n\nOf course by definition, we'd expect around 1 million enmos above 99th percentile since we have 100+ million valid enmo values. The table you have showing the g values for various activities is interesting. Very high g values don't have to be a mistake, they could happen for extreme activities like pounding on a table.\n\nBut since we're interested in sleep times, it would seem OK to clip the values at some level, since enmo = 3 is not really twice as awake as enmo = 1.5. Based on this series here, I'd think clipping at 1.5 (or 1.0) would retain all the information needed for sleep prediction.\n\n", "2469081": "dan3dewey thank for your answer ! It helped me to choose the best way to deal with enmo outliers and to understand huge enmo values. I couldn't imagine what kind of activity may lead to enmo = 7 or 11 g. But now I see that these values are not the device mistakes.", "2469561": "If your model doesn't like outliers, the QuantileTransformer in scikit-learn is an option.", "2472232": "Thanks for the hint!", "2477695": "dan3dewey , @veronikasarikaya : Veronika, Daniel, thank you for the suggested strategies to tackle the *enmo* outlier issues. I tried the recommendations you shared, and the best score improvement to at least two ML model architectures I tried were observed when enmo was upper-clipped at 1.0 (with the value of the 99th percentile to be the second best option).\n\nThank you once again for the solid contributions to the body of knowledge for this contest!" }, "source": "meta" }