![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fbf251597d934a89d8888f040af54001e%2FPlot_SeriesId_2f7504d0f426_Night_5.png?generation=1694530172397904&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2F511919b56fb455e847befabb2563deb0%2FPlot_SeriesId_4feda0596965_Night_8.png?generation=1694530184833126&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2F2fa383fdfb229666536c0644630ed871%2FPlot_SeriesId_6ca4f4fca6a2_Night_23.png?generation=1694530198200648&alt=media\")
I'm still in EDA. This is because the event definition file in the title had 3 times in 1 night and \"step\" column loss rate is as high as 34%, so I had some doubts about it, I checked its validity.
\nDataset of a diagram that visualizes all train data has been published here (4,795 IDxNight). As you can see, the event definitions tend to be slightly different for specific IDs.
\n\n\nIn conclusion, there were many outliers with inconsistent determination of event times (\"step\"/\"timestamps\").
\n
\n I plan to share the list to exclude in the future.
Although this work is very unremarkable, I believe that proper cleansing the training dataset has a significant impact on subsequent prediction accuracy. Everyone, let's stand up!
\n
\n
\n
I'm still in EDA. This is because the event definition file in the title had 3 times in 1 night and \"step\" column loss rate is as high as 34%, so I had some doubts about it, I checked its validity.
\nDataset of a diagram that visualizes all train data has been published here (4,795 IDxNight). As you can see, the event definitions tend to be slightly different for specific IDs.
\n\n\nIn conclusion, there were many outliers with inconsistent determination of event times (\"step\"/\"timestamps\").
\n
\n I plan to share the list to exclude in the future.
Although this work is very unremarkable, I believe that proper cleansing the training dataset has a significant impact on subsequent prediction accuracy. Everyone, let's stand up!
\n
\n
\n
Dear @mitsuyasuhoshino
\nNice analysis. Having incorrectly labelled event data in train_events.csvis not good but as you say, could be cleaned. What is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!
All the best,\ncarl
", "rawMarkdown": "Dear @mitsuyasuhoshino \n\nNice analysis. Having incorrectly labelled event data in `train_events.csv`is not good but as you say, could be cleaned. What is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!\n\nAll the best,\ncarl", "votes": null }, { "id": "2435373", "postDate": "09/12/2023 22:55:53", "content": "\n\nWhat is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!
\n
@carlmcbrideellis thank you for your comment. As you pointed out, my processing may not make sense if the unknown test data and train data have the same labeling quality. However, most of the labeling looks reasonable, so selecting them as the training data and manually labeling the 34% without ground-truth labels should have a large impact on model accuracy.
", "rawMarkdown": "> What is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!\n\n@carlmcbrideellis thank you for your comment. As you pointed out, my processing may not make sense if the unknown test data and train data have the same labeling quality. However, most of the labeling looks reasonable, so selecting them as the training data and manually labeling the 34% without ground-truth labels should have a large impact on model accuracy.", "votes": null }, { "id": "2438858", "postDate": "09/14/2023 14:25:03", "content": "Yes very interesting, I found the same here. I think the host should clarify this, and if this behavior could also be present in the hidden test data.
", "rawMarkdown": "Yes very interesting, I found the same [here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440290). I think the host should clarify this, and if this behavior could also be present in the hidden test data.", "votes": null }, { "id": "2438870", "postDate": "09/14/2023 14:30:33", "content": "Dear @ryanholbrook
\nI'm currently asking for an investigation into another post, and it seems that the labeling of events is inconsistent in this post. We would also appreciate an official opinion on the labeling of Train-Test data.
", "rawMarkdown": "Dear @ryanholbrook\n\nI'm currently asking for an investigation into [another post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437998), and **it seems that the labeling of events is inconsistent in this post**. We would also appreciate an official opinion on the labeling of Train-Test data.", "votes": null }, { "id": "2438972", "postDate": "09/14/2023 15:19:17", "content": "Hi @mitsuyasuhoshino , We are investigating currently. I'll have an update once we identify the source of the problem.
", "rawMarkdown": "Hi @mitsuyasuhoshino , We are investigating currently. I'll have an update once we identify the source of the problem.", "votes": null } ], "comments": [ { "id": 2434967, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/12/2023 16:10:47", "content": "Dear @mitsuyasuhoshino
\nNice analysis. Having incorrectly labelled event data in train_events.csvis not good but as you say, could be cleaned. What is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!
All the best,\ncarl
", "votes": null, "replies": [ { "id": 2435373, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/12/2023 22:55:53", "content": "\n\nWhat is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!
\n
@carlmcbrideellis thank you for your comment. As you pointed out, my processing may not make sense if the unknown test data and train data have the same labeling quality. However, most of the labeling looks reasonable, so selecting them as the training data and manually labeling the 34% without ground-truth labels should have a large impact on model accuracy.
", "votes": null, "replies": [] }, { "id": 2438858, "author_name": "hugodeheer", "author_url": "", "post_date": "09/14/2023 14:25:03", "content": "Yes very interesting, I found the same here. I think the host should clarify this, and if this behavior could also be present in the hidden test data.
", "votes": null, "replies": [] } ] }, { "id": 2438870, "author_name": "mitsuyasuhoshino", "author_url": "", "post_date": "09/14/2023 14:30:33", "content": "Dear @ryanholbrook
\nI'm currently asking for an investigation into another post, and it seems that the labeling of events is inconsistent in this post. We would also appreciate an official opinion on the labeling of Train-Test data.
", "votes": null, "replies": [ { "id": 2438972, "author_name": "ryanholbrook", "author_url": "", "post_date": "09/14/2023 15:19:17", "content": "Hi @mitsuyasuhoshino , We are investigating currently. I'll have an update once we identify the source of the problem.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2434848": "I'm still in EDA. This is because [the event definition file in the title had 3 times in 1 night](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437998) and \"step\" column loss rate is as high as 34%, so I had some doubts about it, I checked its validity.\n\n**Dataset of a diagram that visualizes all train data has been published [here](https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata) (4,795 IDxNight)**. As you can see, the event definitions tend to be slightly different for specific IDs.\n\n>In conclusion, there were many outliers with inconsistent determination of event times (\"step\"/\"timestamps\").\nI plan to share the list to exclude in the future.\n\n- This may be due to the event (onset/wakeup) being labeled by multiple people (or do you already have a model?)\n- I think it is forbidden to modify the ground truth labels, so I think they should be excluded from the training dataset\n- Also, missing values in \"step\"/\"timestamps\" can be filled in by looking at the diagram, so it can be done if you want to do it, although it may take some time\n\n***Although this work is very unremarkable, I believe that proper cleansing the training dataset has a significant impact on subsequent prediction accuracy. Everyone, let's stand up!***\n\n\n\n", "2434967": "Dear @mitsuyasuhoshino \n\nNice analysis. Having incorrectly labelled event data in `train_events.csv`is not good but as you say, could be cleaned. What is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!\n\nAll the best,\ncarl", "2435373": "> What is more concerning is if the events are incorrectly labeled event data for the hidden test data, which we have absolutely no control over!\n\n@carlmcbrideellis thank you for your comment. As you pointed out, my processing may not make sense if the unknown test data and train data have the same labeling quality. However, most of the labeling looks reasonable, so selecting them as the training data and manually labeling the 34% without ground-truth labels should have a large impact on model accuracy.", "2438858": "Yes very interesting, I found the same [here](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440290). I think the host should clarify this, and if this behavior could also be present in the hidden test data.", "2438870": "Dear @ryanholbrook\n\nI'm currently asking for an investigation into [another post](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/437998), and **it seems that the labeling of events is inconsistent in this post**. We would also appreciate an official opinion on the labeling of Train-Test data.", "2438972": "Hi @mitsuyasuhoshino , We are investigating currently. I'll have an update once we identify the source of the problem." }, "source": "meta" }