{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 1 - Reading data files","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-09-09T04:49:20.947326Z","iopub.execute_input":"2023-09-09T04:49:20.947795Z","iopub.status.idle":"2023-09-09T04:49:21.416990Z","shell.execute_reply.started":"2023-09-09T04:49:20.947743Z","shell.execute_reply":"2023-09-09T04:49:21.415857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2 - Reading and Transforming Datasets","metadata":{}},{"cell_type":"markdown","source":"### 2-1 Reading datasets","metadata":{}},{"cell_type":"markdown","source":"In this competition, we have four files: Two of them are realted to the Train data (series and evenents), one of them related to test and the final one is a sample submission file","metadata":{}},{"cell_type":"markdown","source":"#### Training datasets","metadata":{}},{"cell_type":"code","source":"import pandas as pd","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:49:21.419398Z","iopub.execute_input":"2023-09-09T04:49:21.419998Z","iopub.status.idle":"2023-09-09T04:49:21.425395Z","shell.execute_reply.started":"2023-09-09T04:49:21.419956Z","shell.execute_reply":"2023-09-09T04:49:21.423950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Notice that we're working with a Parquet file. Parquet files are specifically designed for storing data in a columnar format, meaning that data belonging to each column is stored contiguously. This differs from traditional databases like SQL Server, where data is typically stored in a row-based format. Using columnar data stores like Parquet offers several advantages, particularly when handling large datasets. For instance, it significantly accelerates aggregation operations and enhances storage efficiency thanks to the utilization of compression algorithms.","metadata":{}},{"cell_type":"code","source":"train_series = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:49:21.427090Z","iopub.execute_input":"2023-09-09T04:49:21.427620Z","iopub.status.idle":"2023-09-09T04:50:17.617228Z","shell.execute_reply.started":"2023-09-09T04:49:21.427577Z","shell.execute_reply":"2023-09-09T04:50:17.616176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:50:17.618582Z","iopub.execute_input":"2023-09-09T04:50:17.618942Z","iopub.status.idle":"2023-09-09T04:50:17.663183Z","shell.execute_reply.started":"2023-09-09T04:50:17.618912Z","shell.execute_reply":"2023-09-09T04:50:17.661906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets first look at the train_series:","metadata":{}},{"cell_type":"code","source":"train_series.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:50:17.666928Z","iopub.execute_input":"2023-09-09T04:50:17.667492Z","iopub.status.idle":"2023-09-09T04:50:17.676452Z","shell.execute_reply.started":"2023-09-09T04:50:17.667447Z","shell.execute_reply":"2023-09-09T04:50:17.675036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:50:17.677919Z","iopub.execute_input":"2023-09-09T04:50:17.678300Z","iopub.status.idle":"2023-09-09T04:50:17.705081Z","shell.execute_reply.started":"2023-09-09T04:50:17.678269Z","shell.execute_reply":"2023-09-09T04:50:17.703932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:50:17.706787Z","iopub.execute_input":"2023-09-09T04:50:17.707166Z","iopub.status.idle":"2023-09-09T04:50:25.004011Z","shell.execute_reply.started":"2023-09-09T04:50:17.707133Z","shell.execute_reply":"2023-09-09T04:50:25.002899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:50:25.005517Z","iopub.execute_input":"2023-09-09T04:50:25.005994Z","iopub.status.idle":"2023-09-09T04:50:51.863919Z","shell.execute_reply.started":"2023-09-09T04:50:25.005953Z","shell.execute_reply":"2023-09-09T04:50:51.862621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of different series:","metadata":{"execution":{"iopub.status.busy":"2023-09-07T12:42:18.918264Z","iopub.execute_input":"2023-09-07T12:42:18.918659Z","iopub.status.idle":"2023-09-07T12:42:18.926630Z","shell.execute_reply.started":"2023-09-07T12:42:18.918627Z","shell.execute_reply":"2023-09-07T12:42:18.924712Z"}}},{"cell_type":"code","source":"series_id_groups = train_series['series_id'].nunique()\n\nprint(\"Number of unique groups (series_id):\", series_id_groups)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:50:51.865518Z","iopub.execute_input":"2023-09-09T04:50:51.866270Z","iopub.status.idle":"2023-09-09T04:51:03.571210Z","shell.execute_reply.started":"2023-09-09T04:50:51.866228Z","shell.execute_reply":"2023-09-09T04:51:03.569923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Working on a part of the dataset instead of all of it","metadata":{}},{"cell_type":"markdown","source":"Because of the lack of CPU and Memory, I have decided to select just a piece of data and work on it.","metadata":{}},{"cell_type":"markdown","source":"To do that, lets first see the minimum and maximum of the timestamp","metadata":{}},{"cell_type":"code","source":"train_series['timestamp'].min()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:03.573010Z","iopub.execute_input":"2023-09-09T04:51:03.573435Z","iopub.status.idle":"2023-09-09T04:51:24.161815Z","shell.execute_reply.started":"2023-09-09T04:51:03.573395Z","shell.execute_reply":"2023-09-09T04:51:24.160607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series['timestamp'].max()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:24.163271Z","iopub.execute_input":"2023-09-09T04:51:24.163620Z","iopub.status.idle":"2023-09-09T04:51:44.745576Z","shell.execute_reply.started":"2023-09-09T04:51:24.163588Z","shell.execute_reply":"2023-09-09T04:51:44.744740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we have 277 series between 2017-08 and 2019-07","metadata":{}},{"cell_type":"markdown","source":"Now lets see for which series the events have been recorded most. We need to do a group by here:","metadata":{}},{"cell_type":"code","source":"top_five_series = train_events.groupby('series_id')['event'].count().sort_values(ascending=False).head(5)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:44.747071Z","iopub.execute_input":"2023-09-09T04:51:44.747378Z","iopub.status.idle":"2023-09-09T04:51:44.757991Z","shell.execute_reply.started":"2023-09-09T04:51:44.747351Z","shell.execute_reply":"2023-09-09T04:51:44.756608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_five_series","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:44.759461Z","iopub.execute_input":"2023-09-09T04:51:44.759920Z","iopub.status.idle":"2023-09-09T04:51:44.773922Z","shell.execute_reply.started":"2023-09-09T04:51:44.759878Z","shell.execute_reply":"2023-09-09T04:51:44.772464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So lets focus on the first 5 series_id only which more events have been recorded for them than the others.","metadata":{}},{"cell_type":"code","source":"train_series_simplified = train_series[train_series['series_id'].isin(top_five_series.index)]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:44.805936Z","iopub.execute_input":"2023-09-09T04:51:44.808843Z","iopub.status.idle":"2023-09-09T04:51:57.357135Z","shell.execute_reply.started":"2023-09-09T04:51:44.808803Z","shell.execute_reply":"2023-09-09T04:51:57.355764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_simplified.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:57.358835Z","iopub.execute_input":"2023-09-09T04:51:57.359300Z","iopub.status.idle":"2023-09-09T04:51:57.367926Z","shell.execute_reply.started":"2023-09-09T04:51:57.359254Z","shell.execute_reply":"2023-09-09T04:51:57.366422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So instead of 127 million rows, we work on 5 million rows","metadata":{}},{"cell_type":"markdown","source":"Lets do the same thing on the train events:","metadata":{}},{"cell_type":"code","source":"train_events_simplified = train_events[train_events['series_id'].isin(top_five_series.index)]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:57.369537Z","iopub.execute_input":"2023-09-09T04:51:57.370006Z","iopub.status.idle":"2023-09-09T04:51:57.382380Z","shell.execute_reply.started":"2023-09-09T04:51:57.369963Z","shell.execute_reply":"2023-09-09T04:51:57.381444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:57.384336Z","iopub.execute_input":"2023-09-09T04:51:57.384795Z","iopub.status.idle":"2023-09-09T04:51:57.397334Z","shell.execute_reply.started":"2023-09-09T04:51:57.384752Z","shell.execute_reply":"2023-09-09T04:51:57.395996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Converting timestamp to datetime","metadata":{}},{"cell_type":"markdown","source":"It seems that we need to work on the timestamo column. We need to split it into some more columns like date, time and timezone","metadata":{}},{"cell_type":"markdown","source":"The timezone is based on the [ISO 86-01](https://en.wikipedia.org/wiki/ISO_8601). The 'T' character is used to separate the date and time part. The number after the '-' represents the timezone offset from Coordinated Universal Time (UTC).","metadata":{}},{"cell_type":"markdown","source":"First, lets change its type from object to datetime:","metadata":{}},{"cell_type":"code","source":"train_series_simplified.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:57.399484Z","iopub.execute_input":"2023-09-09T04:51:57.399857Z","iopub.status.idle":"2023-09-09T04:51:57.417568Z","shell.execute_reply.started":"2023-09-09T04:51:57.399826Z","shell.execute_reply":"2023-09-09T04:51:57.416234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_simplified['datetime'] = train_series_simplified['timestamp'].astype(str).str.replace('T', ' ').str[:-5]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:51:57.419077Z","iopub.execute_input":"2023-09-09T04:51:57.419504Z","iopub.status.idle":"2023-09-09T04:52:04.077343Z","shell.execute_reply.started":"2023-09-09T04:51:57.419461Z","shell.execute_reply":"2023-09-09T04:52:04.075858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_simplified.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:04.079353Z","iopub.execute_input":"2023-09-09T04:52:04.080071Z","iopub.status.idle":"2023-09-09T04:52:04.567676Z","shell.execute_reply.started":"2023-09-09T04:52:04.080032Z","shell.execute_reply":"2023-09-09T04:52:04.566330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_simplified['datetime'] = pd.to_datetime(train_series_simplified['datetime'])","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:04.569284Z","iopub.execute_input":"2023-09-09T04:52:04.569722Z","iopub.status.idle":"2023-09-09T04:52:06.877566Z","shell.execute_reply.started":"2023-09-09T04:52:04.569689Z","shell.execute_reply":"2023-09-09T04:52:06.876369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_simplified.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:06.878994Z","iopub.execute_input":"2023-09-09T04:52:06.879318Z","iopub.status.idle":"2023-09-09T04:52:06.891390Z","shell.execute_reply.started":"2023-09-09T04:52:06.879288Z","shell.execute_reply":"2023-09-09T04:52:06.890027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_simplified['datetime'].dt.year.unique()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:06.893161Z","iopub.execute_input":"2023-09-09T04:52:06.893595Z","iopub.status.idle":"2023-09-09T04:52:07.401040Z","shell.execute_reply.started":"2023-09-09T04:52:06.893563Z","shell.execute_reply":"2023-09-09T04:52:07.399893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets do the same thing fot the events:","metadata":{}},{"cell_type":"code","source":"train_events_simplified['datetime'] = train_events_simplified['timestamp'].astype(str).str.replace('T', ' ').str[:-5]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.402378Z","iopub.execute_input":"2023-09-09T04:52:07.402754Z","iopub.status.idle":"2023-09-09T04:52:07.411642Z","shell.execute_reply.started":"2023-09-09T04:52:07.402723Z","shell.execute_reply":"2023-09-09T04:52:07.410492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified['datetime'] = pd.to_datetime(train_events_simplified['datetime'])","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.413299Z","iopub.execute_input":"2023-09-09T04:52:07.414311Z","iopub.status.idle":"2023-09-09T04:52:07.429960Z","shell.execute_reply.started":"2023-09-09T04:52:07.414265Z","shell.execute_reply":"2023-09-09T04:52:07.428183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.431292Z","iopub.execute_input":"2023-09-09T04:52:07.432558Z","iopub.status.idle":"2023-09-09T04:52:07.453058Z","shell.execute_reply.started":"2023-09-09T04:52:07.432525Z","shell.execute_reply":"2023-09-09T04:52:07.451587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Some EDA!","metadata":{}},{"cell_type":"markdown","source":"Lets see what time has been the most common hour for going to sleep and waking up:","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.454412Z","iopub.execute_input":"2023-09-09T04:52:07.455183Z","iopub.status.idle":"2023-09-09T04:52:07.465421Z","shell.execute_reply.started":"2023-09-09T04:52:07.455146Z","shell.execute_reply":"2023-09-09T04:52:07.463899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.466963Z","iopub.execute_input":"2023-09-09T04:52:07.467302Z","iopub.status.idle":"2023-09-09T04:52:07.489742Z","shell.execute_reply.started":"2023-09-09T04:52:07.467274Z","shell.execute_reply":"2023-09-09T04:52:07.488630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.1 Onset and Wake-up times","metadata":{}},{"cell_type":"code","source":"train_events_simplified_onset = train_events_simplified[train_events_simplified['event']=='onset']\ntrain_events_simplified_wakeup = train_events_simplified[train_events_simplified['event']=='wakeup']","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.491147Z","iopub.execute_input":"2023-09-09T04:52:07.491919Z","iopub.status.idle":"2023-09-09T04:52:07.505270Z","shell.execute_reply.started":"2023-09-09T04:52:07.491882Z","shell.execute_reply":"2023-09-09T04:52:07.504050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified_onset.head(5)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.506806Z","iopub.execute_input":"2023-09-09T04:52:07.507819Z","iopub.status.idle":"2023-09-09T04:52:07.525961Z","shell.execute_reply.started":"2023-09-09T04:52:07.507775Z","shell.execute_reply":"2023-09-09T04:52:07.524822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified_wakeup.head(5)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T04:52:07.528128Z","iopub.execute_input":"2023-09-09T04:52:07.528657Z","iopub.status.idle":"2023-09-09T04:52:07.547981Z","shell.execute_reply.started":"2023-09-09T04:52:07.528611Z","shell.execute_reply":"2023-09-09T04:52:07.546745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most common sleep onset hours:","metadata":{}},{"cell_type":"code","source":"train_events_simplified_onset['datetime'].dt.hour.value_counts().plot(kind='barh')\nplt.title('Onset Times')\nplt.xlabel('Number of cases')\nplt.ylabel('Onset time')","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:05:20.204807Z","iopub.execute_input":"2023-09-09T05:05:20.205336Z","iopub.status.idle":"2023-09-09T05:05:20.541962Z","shell.execute_reply.started":"2023-09-09T05:05:20.205289Z","shell.execute_reply":"2023-09-09T05:05:20.540911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most common waking up hours:","metadata":{}},{"cell_type":"code","source":"train_events_simplified_wakeup['datetime'].dt.hour.value_counts().plot(kind='barh', color='green')\nplt.title('Wakeup Times')\nplt.xlabel('Number of cases')\nplt.ylabel('Wakeup time')","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:07:03.267102Z","iopub.execute_input":"2023-09-09T05:07:03.267605Z","iopub.status.idle":"2023-09-09T05:07:03.540288Z","shell.execute_reply.started":"2023-09-09T05:07:03.267570Z","shell.execute_reply":"2023-09-09T05:07:03.539004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, most of the case studies go to sleep at around 23 and wake up at around 7","metadata":{}},{"cell_type":"markdown","source":"## 3.2 Sleep duration","metadata":{}},{"cell_type":"markdown","source":"Lets see how many hours the case studies sleep","metadata":{}},{"cell_type":"code","source":"train_events_simplified.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:16:25.744917Z","iopub.execute_input":"2023-09-09T05:16:25.745344Z","iopub.status.idle":"2023-09-09T05:16:25.760899Z","shell.execute_reply.started":"2023-09-09T05:16:25.745314Z","shell.execute_reply":"2023-09-09T05:16:25.759363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets focus on non null rows only","metadata":{}},{"cell_type":"code","source":"mask_nonull= (~train_events_simplified['step'].isnull()) & (~train_events_simplified['datetime'].isnull()) \ntrain_events_simplified_nonull = train_events_simplified[mask_nonull]\ntrain_events_simplified_nonull.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:19:25.705764Z","iopub.execute_input":"2023-09-09T05:19:25.706208Z","iopub.status.idle":"2023-09-09T05:19:25.723981Z","shell.execute_reply.started":"2023-09-09T05:19:25.706173Z","shell.execute_reply":"2023-09-09T05:19:25.722735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified_nonull['series_id'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:21:49.705164Z","iopub.execute_input":"2023-09-09T05:21:49.706363Z","iopub.status.idle":"2023-09-09T05:21:49.717177Z","shell.execute_reply.started":"2023-09-09T05:21:49.706309Z","shell.execute_reply":"2023-09-09T05:21:49.715539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified_nonull.head(24)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:22:03.364100Z","iopub.execute_input":"2023-09-09T05:22:03.364483Z","iopub.status.idle":"2023-09-09T05:22:03.390109Z","shell.execute_reply.started":"2023-09-09T05:22:03.364454Z","shell.execute_reply":"2023-09-09T05:22:03.388934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Group by 'series_id' and calculate sleep duration for each night\nsleep_duration_df = train_events_simplified_nonull.groupby([train_events_simplified_nonull['series_id'], train_events_simplified_nonull['night']])['datetime'].agg(['min', 'max']).reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:32:43.987269Z","iopub.execute_input":"2023-09-09T05:32:43.987775Z","iopub.status.idle":"2023-09-09T05:32:43.999378Z","shell.execute_reply.started":"2023-09-09T05:32:43.987737Z","shell.execute_reply":"2023-09-09T05:32:43.998029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sleep_duration_df = sleep_duration_df.rename(columns={'min': 'onset', 'max': 'wakeup'})\nsleep_duration_df.head(6)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:38:39.607921Z","iopub.execute_input":"2023-09-09T05:38:39.608310Z","iopub.status.idle":"2023-09-09T05:38:39.624554Z","shell.execute_reply.started":"2023-09-09T05:38:39.608280Z","shell.execute_reply":"2023-09-09T05:38:39.622978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sleep_duration_df['sleep_duration'] = ((sleep_duration_df['wakeup'] - sleep_duration_df['onset']).dt.seconds / 3600).round(0)\nsleep_duration_df.head(6)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:44:51.365459Z","iopub.execute_input":"2023-09-09T05:44:51.365911Z","iopub.status.idle":"2023-09-09T05:44:51.383490Z","shell.execute_reply.started":"2023-09-09T05:44:51.365876Z","shell.execute_reply":"2023-09-09T05:44:51.382227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now it is time to plot the sleep_duration:","metadata":{}},{"cell_type":"code","source":"sleep_duration_df['sleep_duration'].value_counts().plot(kind='barh', color='red')\nplt.title('Sleep duration')\nplt.xlabel('Number of cases')\nplt.ylabel('Sleep duration by hours')","metadata":{"execution":{"iopub.status.busy":"2023-09-09T05:47:32.339775Z","iopub.execute_input":"2023-09-09T05:47:32.340178Z","iopub.status.idle":"2023-09-09T05:47:32.618993Z","shell.execute_reply.started":"2023-09-09T05:47:32.340146Z","shell.execute_reply":"2023-09-09T05:47:32.617777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, most people sleep between 9 and 10 hours. Lucky ones!","metadata":{}},{"cell_type":"markdown","source":"## 3.3 Anglez, Enmo, Onset and Wakeup","metadata":{}},{"cell_type":"markdown","source":"For our simplefied dataset, lets see if we can see any relation between anglez, enmo and the sleep events (onset and wakeup).\n\nAnglez is the angle of the arm relative to the vertical axis of the body. Also, ENMO is the Euclidean Norm Minus One of all accelerometer signals.","metadata":{}},{"cell_type":"markdown","source":"First, lets review our two datasets again:","metadata":{}},{"cell_type":"code","source":"train_series_simplified.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:03:49.746973Z","iopub.execute_input":"2023-09-09T06:03:49.747384Z","iopub.status.idle":"2023-09-09T06:03:49.763703Z","shell.execute_reply.started":"2023-09-09T06:03:49.747351Z","shell.execute_reply":"2023-09-09T06:03:49.761996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified_nonull.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:23:51.147307Z","iopub.execute_input":"2023-09-09T06:23:51.147741Z","iopub.status.idle":"2023-09-09T06:23:51.164507Z","shell.execute_reply.started":"2023-09-09T06:23:51.147708Z","shell.execute_reply":"2023-09-09T06:23:51.163118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For a single series, lets see how it changes in different nights.","metadata":{}},{"cell_type":"markdown","source":"First, lets concat the two train datasets","metadata":{}},{"cell_type":"code","source":"train_series_simplified = train_series_simplified.rename(columns={'timestamp': 'timestamp_series', 'datetime': 'datetime_series'})","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:42:42.287502Z","iopub.execute_input":"2023-09-09T06:42:42.288946Z","iopub.status.idle":"2023-09-09T06:42:42.538936Z","shell.execute_reply.started":"2023-09-09T06:42:42.288891Z","shell.execute_reply":"2023-09-09T06:42:42.537772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_simplified_nonull = train_events_simplified_nonull.rename(columns={'timestamp': 'timestamp_events', 'datetime': 'datetime_events'})","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:45:13.825247Z","iopub.execute_input":"2023-09-09T06:45:13.825710Z","iopub.status.idle":"2023-09-09T06:45:13.833348Z","shell.execute_reply.started":"2023-09-09T06:45:13.825642Z","shell.execute_reply":"2023-09-09T06:45:13.831810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events = pd.concat([train_series_simplified, train_events_simplified_nonull], axis=0)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:45:35.444984Z","iopub.execute_input":"2023-09-09T06:45:35.445384Z","iopub.status.idle":"2023-09-09T06:45:36.248336Z","shell.execute_reply.started":"2023-09-09T06:45:35.445355Z","shell.execute_reply":"2023-09-09T06:45:36.247019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets look at one specif case events","metadata":{}},{"cell_type":"code","source":"train_series_events[(train_series_events['series_id']=='78569a801a38') & (~train_series_events['event'].isnull())].head(4)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:51:45.084968Z","iopub.execute_input":"2023-09-09T06:51:45.085414Z","iopub.status.idle":"2023-09-09T06:51:46.255310Z","shell.execute_reply.started":"2023-09-09T06:51:45.085378Z","shell.execute_reply":"2023-09-09T06:51:46.254123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events['datetime'] = train_series_events['datetime_series'].fillna(train_series_events['datetime_events'])","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:54:59.465582Z","iopub.execute_input":"2023-09-09T06:54:59.466202Z","iopub.status.idle":"2023-09-09T06:54:59.525840Z","shell.execute_reply.started":"2023-09-09T06:54:59.466156Z","shell.execute_reply":"2023-09-09T06:54:59.524762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events[(train_series_events['series_id']=='78569a801a38') & (~train_series_events['event'].isnull())].head(4)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:55:14.005626Z","iopub.execute_input":"2023-09-09T06:55:14.006044Z","iopub.status.idle":"2023-09-09T06:55:15.215107Z","shell.execute_reply.started":"2023-09-09T06:55:14.006014Z","shell.execute_reply":"2023-09-09T06:55:15.213736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events[(train_series_events['series_id']=='78569a801a38')].head(4)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:55:43.621711Z","iopub.execute_input":"2023-09-09T06:55:43.622123Z","iopub.status.idle":"2023-09-09T06:55:44.773569Z","shell.execute_reply.started":"2023-09-09T06:55:43.622091Z","shell.execute_reply":"2023-09-09T06:55:44.772782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events['datetime'] = pd.to_datetime(train_series_events['datetime'])","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:56:37.765647Z","iopub.execute_input":"2023-09-09T06:56:37.766209Z","iopub.status.idle":"2023-09-09T06:56:37.810272Z","shell.execute_reply.started":"2023-09-09T06:56:37.766165Z","shell.execute_reply":"2023-09-09T06:56:37.809273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:57:00.425106Z","iopub.execute_input":"2023-09-09T06:57:00.425519Z","iopub.status.idle":"2023-09-09T06:57:00.437720Z","shell.execute_reply.started":"2023-09-09T06:57:00.425481Z","shell.execute_reply":"2023-09-09T06:57:00.436770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets get rid of unnecessary columns","metadata":{"execution":{"iopub.status.busy":"2023-09-09T06:57:37.966410Z","iopub.execute_input":"2023-09-09T06:57:37.966871Z","iopub.status.idle":"2023-09-09T06:57:37.983585Z","shell.execute_reply.started":"2023-09-09T06:57:37.966831Z","shell.execute_reply":"2023-09-09T06:57:37.982430Z"}}},{"cell_type":"code","source":"train_series_events.drop(columns=['timestamp_series', 'datetime_series', 'timestamp_events', 'datetime_events'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:01:09.316122Z","iopub.execute_input":"2023-09-09T07:01:09.316542Z","iopub.status.idle":"2023-09-09T07:01:09.619250Z","shell.execute_reply.started":"2023-09-09T07:01:09.316500Z","shell.execute_reply":"2023-09-09T07:01:09.617876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events = train_series_events.sort_values(by='datetime', ascending=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:03:01.126165Z","iopub.execute_input":"2023-09-09T07:03:01.126605Z","iopub.status.idle":"2023-09-09T07:03:01.865299Z","shell.execute_reply.started":"2023-09-09T07:03:01.126570Z","shell.execute_reply":"2023-09-09T07:03:01.864083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events = train_series_events.reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:05:01.026014Z","iopub.execute_input":"2023-09-09T07:05:01.026539Z","iopub.status.idle":"2023-09-09T07:05:01.702276Z","shell.execute_reply.started":"2023-09-09T07:05:01.026491Z","shell.execute_reply":"2023-09-09T07:05:01.700899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events[~train_series_events['event'].isnull()]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:05:43.105765Z","iopub.execute_input":"2023-09-09T07:05:43.106192Z","iopub.status.idle":"2023-09-09T07:05:43.346869Z","shell.execute_reply.started":"2023-09-09T07:05:43.106150Z","shell.execute_reply":"2023-09-09T07:05:43.345713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets see what happens before and after someone goes to sleep:","metadata":{}},{"cell_type":"code","source":"train_series_events.iloc[5480:5515]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:07:56.826543Z","iopub.execute_input":"2023-09-09T07:07:56.827041Z","iopub.status.idle":"2023-09-09T07:07:56.860074Z","shell.execute_reply.started":"2023-09-09T07:07:56.827007Z","shell.execute_reply":"2023-09-09T07:07:56.858717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For a specific day (2017-08-18) and for a specific series (78569a801a38), lets plot the Anglez, Enmo, Onset and Wakeup","metadata":{}},{"cell_type":"code","source":"specific_series_id = '78569a801a38'\nspecific_date = '2017-08-18'\ntrain_series_events_specific_case = train_series_events[(train_series_events['series_id'] == specific_series_id) & (train_series_events['datetime'].dt.date == pd.to_datetime(specific_date).date())]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:15:02.586858Z","iopub.execute_input":"2023-09-09T07:15:02.587299Z","iopub.status.idle":"2023-09-09T07:15:06.998620Z","shell.execute_reply.started":"2023-09-09T07:15:02.587266Z","shell.execute_reply":"2023-09-09T07:15:06.997435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events_specific_case","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:15:09.208289Z","iopub.execute_input":"2023-09-09T07:15:09.208870Z","iopub.status.idle":"2023-09-09T07:15:09.238519Z","shell.execute_reply.started":"2023-09-09T07:15:09.208822Z","shell.execute_reply":"2023-09-09T07:15:09.237606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"onset_events = train_series_events_specific_case[train_series_events_specific_case['event'] == 'onset']\nwakeup_events = train_series_events_specific_case[train_series_events_specific_case['event'] == 'wakeup']\nonset_events","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:43:04.905237Z","iopub.execute_input":"2023-09-09T07:43:04.906226Z","iopub.status.idle":"2023-09-09T07:43:04.926288Z","shell.execute_reply.started":"2023-09-09T07:43:04.906173Z","shell.execute_reply":"2023-09-09T07:43:04.925162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wakeup_events","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:43:39.387025Z","iopub.execute_input":"2023-09-09T07:43:39.387449Z","iopub.status.idle":"2023-09-09T07:43:39.403933Z","shell.execute_reply.started":"2023-09-09T07:43:39.387415Z","shell.execute_reply":"2023-09-09T07:43:39.402807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the line chart\nplt.figure(figsize=(20, 6))\nplt.plot(train_series_events_specific_case['datetime'], train_series_events_specific_case['anglez'], label='Anglez', linestyle='solid')\n\nonset_legend_added = False\nwakeup_legend_added = False\n\n# Add dashed lines for 'onset' and 'wakeup' events\nfor onset in onset_events['datetime']:\n    plt.axvline(x=onset, color='r', linestyle='--', linewidth=3, label='Onset' if not onset_legend_added else '_nolegend_')\n    onset_legend_added = True\n\nfor wakeup in wakeup_events['datetime']:\n    plt.axvline(x=wakeup, color='g', linestyle='--', linewidth=3, label='Wakeup' if not wakeup_legend_added else '_nolegend_')\n    wakeup_legend_added = True\n\n# Customize the plot\nplt.xlabel('Datetime')\nplt.ylabel('Anglez')\nplt.title(f'Anglez Plot for Series ID {specific_series_id} on {specific_date}')\nplt.legend()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:53:58.387678Z","iopub.execute_input":"2023-09-09T07:53:58.388717Z","iopub.status.idle":"2023-09-09T07:53:59.045838Z","shell.execute_reply.started":"2023-09-09T07:53:58.388649Z","shell.execute_reply":"2023-09-09T07:53:59.044826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now lets see the Enmo","metadata":{}},{"cell_type":"markdown","source":"As we can see for both Anglez and Enmo, the values pattern changes a lot.","metadata":{}},{"cell_type":"code","source":"# Create the line chart\nplt.figure(figsize=(20, 6))\nplt.plot(train_series_events_specific_case['datetime'], train_series_events_specific_case['enmo'], label='Enmo', linestyle='solid')\n\nonset_legend_added = False\nwakeup_legend_added = False\n\n# Add dashed lines for 'onset' and 'wakeup' events\nfor onset in onset_events['datetime']:\n    plt.axvline(x=onset, color='r', linestyle='--', linewidth=3, label='Onset' if not onset_legend_added else '_nolegend_')\n    onset_legend_added = True\n\nfor wakeup in wakeup_events['datetime']:\n    plt.axvline(x=wakeup, color='g', linestyle='--', linewidth=3, label='Wakeup' if not wakeup_legend_added else '_nolegend_')\n    wakeup_legend_added = True\n\n# Customize the plot\nplt.xlabel('Datetime')\nplt.ylabel('Enmo')\nplt.title(f'Enmo Plot for Series ID {specific_series_id} on {specific_date}')\nplt.legend()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-09T07:55:31.887561Z","iopub.execute_input":"2023-09-09T07:55:31.888054Z","iopub.status.idle":"2023-09-09T07:55:32.444939Z","shell.execute_reply.started":"2023-09-09T07:55:31.888018Z","shell.execute_reply":"2023-09-09T07:55:32.443404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the pattern changes a lot during sleep time for both Anglez and Enmo","metadata":{}},{"cell_type":"markdown","source":"Finally, lets plot the same plots for the same series in different days:","metadata":{}},{"cell_type":"code","source":"specific_series_id = '78569a801a38'\ntrain_series_events_specific_series = train_series_events[(train_series_events['series_id'] == specific_series_id)]","metadata":{"execution":{"iopub.status.busy":"2023-09-09T08:00:36.258386Z","iopub.execute_input":"2023-09-09T08:00:36.258890Z","iopub.status.idle":"2023-09-09T08:00:37.392788Z","shell.execute_reply.started":"2023-09-09T08:00:36.258854Z","shell.execute_reply":"2023-09-09T08:00:37.391415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series_events_specific_series.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-09T08:00:54.767410Z","iopub.execute_input":"2023-09-09T08:00:54.768153Z","iopub.status.idle":"2023-09-09T08:00:54.784306Z","shell.execute_reply.started":"2023-09-09T08:00:54.768114Z","shell.execute_reply":"2023-09-09T08:00:54.783008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for d in train_series_events_specific_series['datetime'].dt.date.unique()[0:6]:\n    train_series_events_specific_series_specific_date = train_series_events_specific_series[train_series_events_specific_series['datetime'].dt.date==d]\n    \n    # Create the line chart\n    plt.figure(figsize=(20, 3))\n    plt.plot(train_series_events_specific_series_specific_date['datetime'], train_series_events_specific_series_specific_date['anglez'], label='Anglez', linestyle='solid')\n    \n    onset_events_on_specific_date = train_series_events_specific_series_specific_date[train_series_events_specific_series_specific_date['event'] == 'onset']\n    wakeup_events_on_specific_date = train_series_events_specific_series_specific_date[train_series_events_specific_series_specific_date['event'] == 'wakeup']\n\n    onset_legend_added = False\n    wakeup_legend_added = False\n\n    # Add dashed lines for 'onset' and 'wakeup' events\n    for onset in onset_events_on_specific_date['datetime']:\n        plt.axvline(x=onset, color='r', linestyle='--', linewidth=3, label='Onset' if not onset_legend_added else '_nolegend_')\n        onset_legend_added = True\n\n    for wakeup in wakeup_events_on_specific_date['datetime']:\n        plt.axvline(x=wakeup, color='g', linestyle='--', linewidth=3, label='Wakeup' if not wakeup_legend_added else '_nolegend_')\n        wakeup_legend_added = True\n\n    # Customize the plot\n    plt.xlabel('Datetime')\n    plt.ylabel('Anglez')\n    plt.title(f'Anglez Plot for Series ID {specific_series_id} on {d}')\n    plt.legend()\n\n    plt.show()\n    \n    ","metadata":{"execution":{"iopub.status.busy":"2023-09-09T08:11:43.655871Z","iopub.execute_input":"2023-09-09T08:11:43.656311Z","iopub.status.idle":"2023-09-09T08:11:53.251265Z","shell.execute_reply.started":"2023-09-09T08:11:43.656276Z","shell.execute_reply":"2023-09-09T08:11:53.250134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And for the Enmo:","metadata":{}},{"cell_type":"code","source":"for d in train_series_events_specific_series['datetime'].dt.date.unique()[0:10]:\n    train_series_events_specific_series_specific_date = train_series_events_specific_series[train_series_events_specific_series['datetime'].dt.date==d]\n    \n    # Create the line chart\n    plt.figure(figsize=(20, 3))\n    plt.plot(train_series_events_specific_series_specific_date['datetime'], train_series_events_specific_series_specific_date['enmo'], label='Enmo', linestyle='solid')\n    \n    onset_events_on_specific_date = train_series_events_specific_series_specific_date[train_series_events_specific_series_specific_date['event'] == 'onset']\n    wakeup_events_on_specific_date = train_series_events_specific_series_specific_date[train_series_events_specific_series_specific_date['event'] == 'wakeup']\n\n    onset_legend_added = False\n    wakeup_legend_added = False\n\n    # Add dashed lines for 'onset' and 'wakeup' events\n    for onset in onset_events_on_specific_date['datetime']:\n        plt.axvline(x=onset, color='r', linestyle='--', linewidth=3, label='Onset' if not onset_legend_added else '_nolegend_')\n        onset_legend_added = True\n\n    for wakeup in wakeup_events_on_specific_date['datetime']:\n        plt.axvline(x=wakeup, color='g', linestyle='--', linewidth=3, label='Wakeup' if not wakeup_legend_added else '_nolegend_')\n        wakeup_legend_added = True\n\n    # Customize the plot\n    plt.xlabel('Datetime')\n    plt.ylabel('Enmo')\n    plt.title(f'Enmo Plot for Series ID {specific_series_id} on {d}')\n    plt.legend()\n\n    plt.show()\n    \n    ","metadata":{"execution":{"iopub.status.busy":"2023-09-09T08:14:31.416581Z","iopub.execute_input":"2023-09-09T08:14:31.417050Z","iopub.status.idle":"2023-09-09T08:14:45.984103Z","shell.execute_reply.started":"2023-09-09T08:14:31.417012Z","shell.execute_reply":"2023-09-09T08:14:45.982649Z"},"trusted":true},"execution_count":null,"outputs":[]}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}