{
  "id": 438044,
  "title": "👀 [EDA] Visualization of Train Data",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/438044",
  "author_name": "",
  "post_date": "2023-09-09T07:37:01.820400800Z",
  "votes": 8,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am in the stage of visualizing data as EDA. In these diagram of series_id:038441c925bb, sleep(event=0-&gt;-1) can almost be detected at the timing when the oscillations of \"(normalized) anglez\" converge and there are no changes. However, can the opposite explanation be used for waking up(event=0-&gt;1)?<br>\nI'm planning to create a model using ML first, so I'm going to include the above as features. If you have any opinions, please leave a comment.</p>\n<p><strong>Notebook Link</strong> : <a href=\"https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda\" target=\"_blank\">https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2F75de93c39914cb1fddf6b4067f00352e%2FID_038441c925bb_Night6.bmp?generation=1694245016206765&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Ff02ec0e6d03807e1c7b0c147a8013410%2FID_038441c925bb_Night13.bmp?generation=1694245629311690&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "2430308",
      "postDate": "09/09/2023 07:37:01",
      "content": "<p>I am in the stage of visualizing data as EDA. In these diagram of series_id:038441c925bb, sleep(event=0-&gt;-1) can almost be detected at the timing when the oscillations of \"(normalized) anglez\" converge and there are no changes. However, can the opposite explanation be used for waking up(event=0-&gt;1)?<br>\nI'm planning to create a model using ML first, so I'm going to include the above as features. If you have any opinions, please leave a comment.</p>\n<p><strong>Notebook Link</strong> : <a href=\"https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda\" target=\"_blank\">https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2F75de93c39914cb1fddf6b4067f00352e%2FID_038441c925bb_Night6.bmp?generation=1694245016206765&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Ff02ec0e6d03807e1c7b0c147a8013410%2FID_038441c925bb_Night13.bmp?generation=1694245629311690&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "I am in the stage of visualizing data as EDA. In these diagram of series_id:038441c925bb, sleep(event=0->-1) can almost be detected at the timing when the oscillations of \"(normalized) anglez\" converge and there are no changes. However, can the opposite explanation be used for waking up(event=0->1)?\nI'm planning to create a model using ML first, so I'm going to include the above as features. If you have any opinions, please leave a comment.\n\n**Notebook Link** : [https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda](https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2F75de93c39914cb1fddf6b4067f00352e%2FID_038441c925bb_Night6.bmp?generation=1694245016206765&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Ff02ec0e6d03807e1c7b0c147a8013410%2FID_038441c925bb_Night13.bmp?generation=1694245629311690&alt=media)",
      "votes": null
    },
    {
      "id": "2435367",
      "postDate": "09/12/2023 22:42:05",
      "content": "<p>I have uploaded a plot of all train data (time series data of 4,795) to a dataset.</p>\n<p><strong>Dataset Link</strong> : <a href=\"https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata\" target=\"_blank\">https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata</a></p>",
      "rawMarkdown": "I have uploaded a plot of all train data (time series data of 4,795) to a dataset.\n\n**Dataset Link** : [https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata](https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata)",
      "votes": null
    },
    {
      "id": "2502071",
      "postDate": "10/27/2023 23:22:17",
      "content": "<p>All training data was visualized as Notebook Ver.6 using event data updated in September (※ It is not recommended to re-RUN the Notebook as it takes a long time to run).</p>\n<p><strong>Notebook Link</strong> : <a href=\"https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\" target=\"_blank\">https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa9617deb3d30dff371f2bd5e97e88eb2%2FID_fb223ed2278c_Night2.bmp?generation=1698447769182644&amp;alt=media\" alt=\"\"></p>\n<h1>There are two things I learned from this visualization.</h1>\n<ul>\n<li>There are still biases and obvious mistakes in event labeling (In the above diagram, both onset and wakeup seem to be defined slightly differently)</li>\n<li>There is <strong>good sleep and bad sleep</strong> (good or bad sleep depends on whether it is easy to visually distinguish between events and whether or not they wake up during sleep =&gt; This determination may be useful for CMI research ; ID 3452b878e596 may have a sleep disorder…)</li>\n</ul>\n<p>Regarding the former, I plan to exclude it from the training data and check its effect on LB.<br>\nHas anyone excluded event outliers??</p>",
      "rawMarkdown": "All training data was visualized as Notebook Ver.6 using event data updated in September (※ It is not recommended to re-RUN the Notebook as it takes a long time to run).\n\n**Notebook Link** : https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa9617deb3d30dff371f2bd5e97e88eb2%2FID_fb223ed2278c_Night2.bmp?generation=1698447769182644&alt=media)\n \n# There are two things I learned from this visualization.\n\n- There are still biases and obvious mistakes in event labeling (In the above diagram, both onset and wakeup seem to be defined slightly differently)\n- There is **good sleep and bad sleep** (good or bad sleep depends on whether it is easy to visually distinguish between events and whether or not they wake up during sleep => This determination may be useful for CMI research ; ID 3452b878e596 may have a sleep disorder...)\n\nRegarding the former, I plan to exclude it from the training data and check its effect on LB.\nHas anyone excluded event outliers??",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2435367,
      "author_name": "mitsuyasuhoshino",
      "author_url": "",
      "post_date": "09/12/2023 22:42:05",
      "content": "<p>I have uploaded a plot of all train data (time series data of 4,795) to a dataset.</p>\n<p><strong>Dataset Link</strong> : <a href=\"https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata\" target=\"_blank\">https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2502071,
      "author_name": "mitsuyasuhoshino",
      "author_url": "",
      "post_date": "10/27/2023 23:22:17",
      "content": "<p>All training data was visualized as Notebook Ver.6 using event data updated in September (※ It is not recommended to re-RUN the Notebook as it takes a long time to run).</p>\n<p><strong>Notebook Link</strong> : <a href=\"https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\" target=\"_blank\">https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa9617deb3d30dff371f2bd5e97e88eb2%2FID_fb223ed2278c_Night2.bmp?generation=1698447769182644&amp;alt=media\" alt=\"\"></p>\n<h1>There are two things I learned from this visualization.</h1>\n<ul>\n<li>There are still biases and obvious mistakes in event labeling (In the above diagram, both onset and wakeup seem to be defined slightly differently)</li>\n<li>There is <strong>good sleep and bad sleep</strong> (good or bad sleep depends on whether it is easy to visually distinguish between events and whether or not they wake up during sleep =&gt; This determination may be useful for CMI research ; ID 3452b878e596 may have a sleep disorder…)</li>\n</ul>\n<p>Regarding the former, I plan to exclude it from the training data and check its effect on LB.<br>\nHas anyone excluded event outliers??</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2430308": "I am in the stage of visualizing data as EDA. In these diagram of series_id:038441c925bb, sleep(event=0->-1) can almost be detected at the timing when the oscillations of \"(normalized) anglez\" converge and there are no changes. However, can the opposite explanation be used for waking up(event=0->1)?\nI'm planning to create a model using ML first, so I'm going to include the above as features. If you have any opinions, please leave a comment.\n\n**Notebook Link** : [https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda](https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2F75de93c39914cb1fddf6b4067f00352e%2FID_038441c925bb_Night6.bmp?generation=1694245016206765&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Ff02ec0e6d03807e1c7b0c147a8013410%2FID_038441c925bb_Night13.bmp?generation=1694245629311690&alt=media)",
    "2435367": "I have uploaded a plot of all train data (time series data of 4,795) to a dataset.\n\n**Dataset Link** : [https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata](https://www.kaggle.com/datasets/mitsuyasuhoshino/dss-visualizationofalltraindata)",
    "2502071": "All training data was visualized as Notebook Ver.6 using event data updated in September (※ It is not recommended to re-RUN the Notebook as it takes a long time to run).\n\n**Notebook Link** : https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14688007%2Fa9617deb3d30dff371f2bd5e97e88eb2%2FID_fb223ed2278c_Night2.bmp?generation=1698447769182644&alt=media)\n \n# There are two things I learned from this visualization.\n\n- There are still biases and obvious mistakes in event labeling (In the above diagram, both onset and wakeup seem to be defined slightly differently)\n- There is **good sleep and bad sleep** (good or bad sleep depends on whether it is easy to visually distinguish between events and whether or not they wake up during sleep => This determination may be useful for CMI research ; ID 3452b878e596 may have a sleep disorder...)\n\nRegarding the former, I plan to exclude it from the training data and check its effect on LB.\nHas anyone excluded event outliers??"
  },
  "source": "meta"
}