{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div style=\"padding:20px;color:#DfB078;margin:0;font-size:300%;text-align:center;display:fill;border-radius:5px;background-color:#010012;overflow:hidden;font-weight:800\"> 🧒Sleep States Child || Analysis and visualization </div>","metadata":{"papermill":{"duration":0.010264,"end_time":"2023-09-07T09:17:24.453672","exception":false,"start_time":"2023-09-07T09:17:24.443408","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## few concept you may need to konw...\n* **accele accelerometer**: An accelerometer is a sensor or device that measures the acceleration of an object it is attached to.\n* **segmentation problem**: a segmentation problem, rather than live timeseries analysis. which is aim to segment whole time into different perid by evenets","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport time\nimport seaborn as sns\nimport pyarrow.parquet as pq","metadata":{"papermill":{"duration":2.477636,"end_time":"2023-09-07T09:17:26.942523","exception":false,"start_time":"2023-09-07T09:17:24.464887","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T02:45:33.083233Z","iopub.execute_input":"2023-09-18T02:45:33.084558Z","iopub.status.idle":"2023-09-18T02:45:34.616288Z","shell.execute_reply.started":"2023-09-18T02:45:33.084515Z","shell.execute_reply":"2023-09-18T02:45:34.614784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\nsample_submission = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/sample_submission.csv')\ntrain_file = pq.read_table('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet').to_pandas()\ntest_file = pq.read_table('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet').to_pandas()","metadata":{"papermill":{"duration":0.065423,"end_time":"2023-09-07T09:17:27.01846","exception":false,"start_time":"2023-09-07T09:17:26.953037","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T03:08:18.423414Z","iopub.execute_input":"2023-09-18T03:08:18.424133Z","iopub.status.idle":"2023-09-18T03:09:16.652660Z","shell.execute_reply.started":"2023-09-18T03:08:18.424098Z","shell.execute_reply":"2023-09-18T03:09:16.649549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"train_events\")\ndisplay(train_events.head(5))\nprint(\"\\nsample_submission\")\ndisplay(sample_submission.head(5))\nprint(\"\\ntrain_file\")\ndisplay(train_file.head(5))\nprint(\"\\ntest_file\")\ndisplay(test_file.head(5))","metadata":{"execution":{"iopub.status.busy":"2023-09-18T02:46:26.038074Z","iopub.execute_input":"2023-09-18T02:46:26.038539Z","iopub.status.idle":"2023-09-18T02:46:26.117405Z","shell.execute_reply.started":"2023-09-18T02:46:26.038497Z","shell.execute_reply":"2023-09-18T02:46:26.116208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### data usage \nthere total 3 table of data to be used. except the sample_submission is an example of submission. ","metadata":{}},{"cell_type":"markdown","source":"## data properties discription \n* **series_id**: Each series is a continuous recording of accelerometer data for a single subject spanning many days.\n* **night**:  An enumeration of potential onset / wakeup event pairs. At most one pair of events can occur for each night.\n* **step**: An integer timestep for each observation within a series.\n* **timestamp**:A corresponding datetime with ISO 8601 format \n* **anglez**:  As calculated and described by the GGIR package, z-angle is a metric derived from individual accelerometer components that is commonly used in sleep detection, and refers to the angle of the arm relative to the vertical axis of the body\n* **enmo**: As calculated and described by the GGIR package, ENMO is the Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero. While no standard measure of acceleration exists in this space, this is one of the several commonly computed features. this value may need further consider and processs\n* **event**: The type of event, whether onset or wakeup.\n* **score**: confidence score for that event.","metadata":{}},{"cell_type":"markdown","source":"Q1:Is a series a observation?","metadata":{}},{"cell_type":"code","source":"print('Rows and Columns of Train Events:', train_events.shape,'|\\n')\nprint('Missing values in Train Events data:', train_events.isna().any().any(),'|\\n')\nprint('Missing Values in Train Events:', train_events[train_events.duplicated()].sum(),'|\\n')","metadata":{"papermill":{"duration":0.053923,"end_time":"2023-09-07T09:17:27.144518","exception":false,"start_time":"2023-09-07T09:17:27.090595","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T02:46:26.118994Z","iopub.execute_input":"2023-09-18T02:46:26.119327Z","iopub.status.idle":"2023-09-18T02:46:26.148705Z","shell.execute_reply.started":"2023-09-18T02:46:26.119298Z","shell.execute_reply":"2023-09-18T02:46:26.147504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('************************************************\\n',train_events['event'].value_counts(), '\\n************************************************')\nplt.figure(figsize=(4,4))\n\nax = sns.countplot(data=train_events, x='event')\nax.bar_label(ax.containers[0])\nplt.show()\nprint(\"the number of onset and wakeup events are basically equal\")","metadata":{"papermill":{"duration":0.322946,"end_time":"2023-09-07T09:17:27.479632","exception":false,"start_time":"2023-09-07T09:17:27.156686","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T02:46:37.795080Z","iopub.execute_input":"2023-09-18T02:46:37.795498Z","iopub.status.idle":"2023-09-18T02:46:38.113790Z","shell.execute_reply.started":"2023-09-18T02:46:37.795465Z","shell.execute_reply":"2023-09-18T02:46:38.111147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('************************************************\\n',train_events['night'].value_counts(), '\\n************************************************')\nplt.figure(figsize=(12,4))\n\nax = plt.plot(list(train_events['night']))\nplt.show()","metadata":{"papermill":{"duration":0.310919,"end_time":"2023-09-07T09:17:27.802026","exception":false,"start_time":"2023-09-07T09:17:27.491107","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T02:48:13.589694Z","iopub.execute_input":"2023-09-18T02:48:13.590086Z","iopub.status.idle":"2023-09-18T02:48:13.837811Z","shell.execute_reply.started":"2023-09-18T02:48:13.590055Z","shell.execute_reply":"2023-09-18T02:48:13.836477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(train_events['night'].unique())\nprint(\"total 80 days\")","metadata":{"papermill":{"duration":0.042081,"end_time":"2023-09-07T09:18:24.297717","exception":false,"start_time":"2023-09-07T09:18:24.255636","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T03:03:58.836613Z","iopub.execute_input":"2023-09-18T03:03:58.837984Z","iopub.status.idle":"2023-09-18T03:03:58.848256Z","shell.execute_reply.started":"2023-09-18T03:03:58.837935Z","shell.execute_reply":"2023-09-18T03:03:58.846862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events['date'] = train_events['timestamp'].str.split('T', expand=True)[0]\ntrain_events['time'] = train_events['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntrain_events['timestamp'] = pd.to_datetime(train_events['date']+' '+train_events['time'])\ntrain_events['hour'] = train_events['timestamp'].dt.hour","metadata":{"papermill":{"duration":0.27107,"end_time":"2023-09-07T09:18:24.582499","exception":false,"start_time":"2023-09-07T09:18:24.311429","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T03:09:16.659134Z","iopub.execute_input":"2023-09-18T03:09:16.659861Z","iopub.status.idle":"2023-09-18T03:09:17.242107Z","shell.execute_reply.started":"2023-09-18T03:09:16.659820Z","shell.execute_reply":"2023-09-18T03:09:17.239976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"train_events after process\")\ndisplay(train_events)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T03:11:00.014225Z","iopub.execute_input":"2023-09-18T03:11:00.015220Z","iopub.status.idle":"2023-09-18T03:11:00.043291Z","shell.execute_reply.started":"2023-09-18T03:11:00.015177Z","shell.execute_reply":"2023-09-18T03:11:00.041719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(6, 6))\nplt.title('Hours Distribution')\nsns.kdeplot(train_events['hour'],color='red');","metadata":{"papermill":{"duration":0.71912,"end_time":"2023-09-07T09:18:25.384864","exception":false,"start_time":"2023-09-07T09:18:24.665744","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-18T03:11:11.275352Z","iopub.execute_input":"2023-09-18T03:11:11.275768Z","iopub.status.idle":"2023-09-18T03:11:11.756028Z","shell.execute_reply.started":"2023-09-18T03:11:11.275733Z","shell.execute_reply":"2023-09-18T03:11:11.754701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_onset = train_events[train_events['event'] == \"onset\"]\ntrain_events_wakeup = train_events[train_events['event'] == \"wakeup\"]","metadata":{"execution":{"iopub.status.busy":"2023-09-18T03:20:36.255810Z","iopub.execute_input":"2023-09-18T03:20:36.256226Z","iopub.status.idle":"2023-09-18T03:20:36.273977Z","shell.execute_reply.started":"2023-09-18T03:20:36.256196Z","shell.execute_reply":"2023-09-18T03:20:36.272955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(6, 6))\nplt.title('Hours Distribution of train_events_onset')\nsns.kdeplot(train_events_onset['hour'],color='red');","metadata":{"execution":{"iopub.status.busy":"2023-09-18T03:21:15.392930Z","iopub.execute_input":"2023-09-18T03:21:15.393468Z","iopub.status.idle":"2023-09-18T03:21:15.874867Z","shell.execute_reply.started":"2023-09-18T03:21:15.393429Z","shell.execute_reply":"2023-09-18T03:21:15.873287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(6, 6))\nplt.title('Hours Distribution of train_events_wakeup')\nsns.kdeplot(train_events_wakeup['hour'],color='red');","metadata":{"execution":{"iopub.status.busy":"2023-09-18T03:21:53.548893Z","iopub.execute_input":"2023-09-18T03:21:53.549856Z","iopub.status.idle":"2023-09-18T03:21:53.958409Z","shell.execute_reply.started":"2023-09-18T03:21:53.549814Z","shell.execute_reply":"2023-09-18T03:21:53.957167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.jointplot(data=train_events, x='hour', y='event')\n# sns.jointplot(data=train_data, x='fixed acidity', y='event')\nsns.jointplot(data=train_events, x='step', y='event')\nsns.jointplot(data=train_events, x='night', y='event')","metadata":{"papermill":{"duration":3.141547,"end_time":"2023-09-07T09:18:28.552273","exception":false,"start_time":"2023-09-07T09:18:25.410726","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-08T12:22:48.353915Z","iopub.execute_input":"2023-09-08T12:22:48.354296Z","iopub.status.idle":"2023-09-08T12:22:50.786507Z","shell.execute_reply.started":"2023-09-08T12:22:48.354266Z","shell.execute_reply":"2023-09-08T12:22:50.785386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_events.columns)","metadata":{"papermill":{"duration":0.033484,"end_time":"2023-09-07T09:18:28.602678","exception":false,"start_time":"2023-09-07T09:18:28.569194","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-08T12:22:50.789421Z","iopub.execute_input":"2023-09-08T12:22:50.789749Z","iopub.status.idle":"2023-09-08T12:22:50.796235Z","shell.execute_reply.started":"2023-09-08T12:22:50.789721Z","shell.execute_reply":"2023-09-08T12:22:50.795020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events.dtypes.value_counts().plot.pie(autopct='%0.2f%%')","metadata":{"papermill":{"duration":0.213939,"end_time":"2023-09-07T09:18:28.835366","exception":false,"start_time":"2023-09-07T09:18:28.621427","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-08T12:22:50.797676Z","iopub.execute_input":"2023-09-08T12:22:50.798484Z","iopub.status.idle":"2023-09-08T12:22:50.959540Z","shell.execute_reply.started":"2023-09-08T12:22:50.798452Z","shell.execute_reply":"2023-09-08T12:22:50.958134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\ntrain_events_new = train_events.drop(['series_id', 'timestamp'], axis='columns')\nstring_col = train_events_new.select_dtypes(include=('object')).columns\nl_e = LabelEncoder()\nfor col in string_col:\n    train_events_new[col] = l_e.fit_transform(train_events_new[col].astype('str'))\n\ntrain_for_box_plot = train_events\nstring_col1 = train_for_box_plot.select_dtypes(include=('object')).columns\nl_e = LabelEncoder()\nfor col in string_col1:\n    train_for_box_plot[col] = l_e.fit_transform(train_for_box_plot[col].astype('str'))","metadata":{"papermill":{"duration":0.323251,"end_time":"2023-09-07T09:18:29.179118","exception":false,"start_time":"2023-09-07T09:18:28.855867","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-08T12:22:50.961235Z","iopub.execute_input":"2023-09-08T12:22:50.962210Z","iopub.status.idle":"2023-09-08T12:22:51.126479Z","shell.execute_reply.started":"2023-09-08T12:22:50.962163Z","shell.execute_reply":"2023-09-08T12:22:51.125295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axs = plt.subplots(3,2, figsize=(16,12))\n\nfor f, ax in zip(train_events_new,axs.ravel()):\n    sns.boxplot(data=train_for_box_plot, x='event',y= f, ax=ax)\nplt.suptitle(\"The Relationship between features and target\", y=1.02, size=18)\nplt.tight_layout()\nplt.show()","metadata":{"papermill":{"duration":1.384746,"end_time":"2023-09-07T09:18:30.584213","exception":false,"start_time":"2023-09-07T09:18:29.199467","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-08T12:22:51.128105Z","iopub.execute_input":"2023-09-08T12:22:51.128600Z","iopub.status.idle":"2023-09-08T12:22:52.166336Z","shell.execute_reply.started":"2023-09-08T12:22:51.128549Z","shell.execute_reply":"2023-09-08T12:22:52.164768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nsns.lineplot(data=train_events,x=\"event\",y=\"hour\",color=\"red\",label=\"onset\")\nsns.lineplot(data=train_events,x=\"event\",y=\"time\",color=\"blue\",label=\"wakeup\")","metadata":{"papermill":{"duration":0.862618,"end_time":"2023-09-07T09:18:31.464984","exception":false,"start_time":"2023-09-07T09:18:30.602366","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-09-08T12:22:52.167917Z","iopub.execute_input":"2023-09-08T12:22:52.168321Z","iopub.status.idle":"2023-09-08T12:22:52.870454Z","shell.execute_reply.started":"2023-09-08T12:22:52.168287Z","shell.execute_reply":"2023-09-08T12:22:52.869107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visit Related Health Datasets\n\n#### 1. Global COVID 19 Dataset \nhttps://www.kaggle.com/datasets/meeratif/full-grouped\n\n#### 2. Pakistan COVID 19 Dataset\nhttps://www.kaggle.com/datasets/meeratif/pakistan-covid-dataset","metadata":{"papermill":{"duration":0.022725,"end_time":"2023-09-07T09:18:31.557106","exception":false,"start_time":"2023-09-07T09:18:31.534381","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Wages \n#### 1. Countries by Average Wages Monthly and Yearly\nhttps://www.kaggle.com/datasets/meeratif/list-of-countries-by-average-wage-monthly-yearly","metadata":{"papermill":{"duration":0.020657,"end_time":"2023-09-07T09:18:31.601639","exception":false,"start_time":"2023-09-07T09:18:31.580982","status":"completed"},"tags":[]}}]}