{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"##### Files and Field Descriptions\n\n- train_series.parquet - Series to be used as training data. Each series is a continuous recording of accelerometer data for a single subject spanning many days.\n\n- series_id - Unique identifier for each accelerometer series.\n- step - An integer timestep for each observation within a series.\n- timestamp - A corresponding datetime with ISO 8601 format %Y-%m-%dT%H:%M:%S%z.\n- anglez - As calculated and described by the GGIR package, z-angle is a metric derived from individual accelerometer components that is commonly used in sleep detection, and refers to the angle of the arm relative to the vertical axis of the body\n- enmo - As calculated and described by the GGIR package, ENMO is the Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero. While no standard measure of acceleration exists in this space, this is one of the several commonly computed features\n- test_series.parquet - Series to be used as the test data, containing the same fields as above. You will predict event occurrences for series in this file.\n- train_events.csv - Sleep logs for series in the training set recording onset and wake events.\n- series_id - Unique identifier for each series of accelerometer data in train_series.parquet.\n- night - An enumeration of potential onset / wakeup event pairs. At most one pair of events can occur for each night.\n- event - The type of event, whether onset or wakeup.\n- step and timestamp - The recorded time of occurence of the event in the accelerometer series.\n\nsample_submission.csv - A sample submission file in the correct format. See the Evaluation page for more details.","metadata":{}},{"cell_type":"markdown","source":"# 1. DATA COLLECTION","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nplt.rcParams[\"figure.figsize\"] = (20,3)\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-30T22:02:00.349625Z","iopub.execute_input":"2023-10-30T22:02:00.350006Z","iopub.status.idle":"2023-10-30T22:02:00.710416Z","shell.execute_reply.started":"2023-10-30T22:02:00.349976Z","shell.execute_reply":"2023-10-30T22:02:00.709018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def translate_time(data: pd.DataFrame, col: str):\n    try:\n        data['datetime'] = data[col].astype(str).str.replace('T', ' ').str[:-5]\n        data['datetime'] = pd.to_datetime(data['datetime'] )\n    except Exception as e:\n        print(e)\n    return data","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.712779Z","iopub.execute_input":"2023-10-30T22:02:00.713237Z","iopub.status.idle":"2023-10-30T22:02:00.719554Z","shell.execute_reply.started":"2023-10-30T22:02:00.713208Z","shell.execute_reply":"2023-10-30T22:02:00.718446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. DATA UNDERSTANDING\n## Understanding Training Events and Series Data\n\n- Process through assessing data distributions\n- Identifying Missing Values\n- Outlier detection\n- Understanding Relationships between variables\n\nThe training events data has 14510 rows and 5 columns aka features.","metadata":{}},{"cell_type":"code","source":"training_data_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\ntraining_data_events.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.721991Z","iopub.execute_input":"2023-10-30T22:02:00.722554Z","iopub.status.idle":"2023-10-30T22:02:00.775484Z","shell.execute_reply.started":"2023-10-30T22:02:00.722525Z","shell.execute_reply":"2023-10-30T22:02:00.774454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_data_events.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.777947Z","iopub.execute_input":"2023-10-30T22:02:00.778624Z","iopub.status.idle":"2023-10-30T22:02:00.820401Z","shell.execute_reply.started":"2023-10-30T22:02:00.778591Z","shell.execute_reply":"2023-10-30T22:02:00.818987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Describing the numerical data types\n\n- ##### night :\n\n  - This is an Interval value that denotes the night number corresponding to the event. We can use this to find duration of sleep and awake.\n  - Night has integral range from [1 - 84]. Interval should be different for each series-id\n- Step has no signifance in this set but can act as a key for other train_series parquet file","metadata":{}},{"cell_type":"code","source":"training_data_events.describe()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.822098Z","iopub.execute_input":"2023-10-30T22:02:00.822746Z","iopub.status.idle":"2023-10-30T22:02:00.850823Z","shell.execute_reply.started":"2023-10-30T22:02:00.822672Z","shell.execute_reply":"2023-10-30T22:02:00.849882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Describing the categorical variables \n\n##### series_id :\n\nThis ID represents the id i.e Nominal Value of the device / denoting a child. There are 0 NULL values\n\n##### event :\n\nEvent is an Ordinal value that takes 2 values : onset and wakeup","metadata":{}},{"cell_type":"code","source":"training_data_events.describe(include=['O'])","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.852266Z","iopub.execute_input":"2023-10-30T22:02:00.852837Z","iopub.status.idle":"2023-10-30T22:02:00.879926Z","shell.execute_reply.started":"2023-10-30T22:02:00.852804Z","shell.execute_reply":"2023-10-30T22:02:00.879210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_data_events = translate_time(training_data_events, \"timestamp\")\ntraining_data_events.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.883345Z","iopub.execute_input":"2023-10-30T22:02:00.884362Z","iopub.status.idle":"2023-10-30T22:02:00.933545Z","shell.execute_reply.started":"2023-10-30T22:02:00.884325Z","shell.execute_reply":"2023-10-30T22:02:00.932151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Observation : Nights for each User-id available in the events data","metadata":{}},{"cell_type":"code","source":"seriesid_group_by_nights = training_data_events.groupby('series_id')['night'].nunique()\nseriesid_group_by_nights.describe()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.937236Z","iopub.execute_input":"2023-10-30T22:02:00.937573Z","iopub.status.idle":"2023-10-30T22:02:00.953121Z","shell.execute_reply.started":"2023-10-30T22:02:00.937544Z","shell.execute_reply":"2023-10-30T22:02:00.951332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seriesid_group_by_nights.sort_values(ascending=False).head(10).plot(kind='barh')\nplt.title('Nights per Series ID')\nplt.xlabel('Number of nights')\nplt.ylabel('Top 10 Series IDs')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:00.954498Z","iopub.execute_input":"2023-10-30T22:02:00.954979Z","iopub.status.idle":"2023-10-30T22:02:01.314598Z","shell.execute_reply.started":"2023-10-30T22:02:00.954950Z","shell.execute_reply":"2023-10-30T22:02:01.313542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check NaN values in the dataset\n\n4923 values in step, timestamp are NaN. This needs to be worked upon.","metadata":{}},{"cell_type":"code","source":"training_data_events.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.318542Z","iopub.execute_input":"2023-10-30T22:02:01.319048Z","iopub.status.idle":"2023-10-30T22:02:01.329937Z","shell.execute_reply.started":"2023-10-30T22:02:01.319023Z","shell.execute_reply":"2023-10-30T22:02:01.328000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check values which are NaNs or NaTs\nseries_id_having_all_nas = training_data_events[training_data_events.isna().any(axis=1)]\n\nprint(series_id_having_all_nas[\"series_id\"].nunique())","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.331356Z","iopub.execute_input":"2023-10-30T22:02:01.331751Z","iopub.status.idle":"2023-10-30T22:02:01.348773Z","shell.execute_reply.started":"2023-10-30T22:02:01.331686Z","shell.execute_reply":"2023-10-30T22:02:01.346883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Get the unique events in event data\ntraining_data_events[\"event\"].unique()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.350181Z","iopub.execute_input":"2023-10-30T22:02:01.350518Z","iopub.status.idle":"2023-10-30T22:02:01.369554Z","shell.execute_reply.started":"2023-10-30T22:02:01.350489Z","shell.execute_reply":"2023-10-30T22:02:01.364983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Get the unique series ids in event data\ntraining_data_events[\"series_id\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.372244Z","iopub.execute_input":"2023-10-30T22:02:01.372712Z","iopub.status.idle":"2023-10-30T22:02:01.383399Z","shell.execute_reply.started":"2023-10-30T22:02:01.372660Z","shell.execute_reply":"2023-10-30T22:02:01.381062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From 277 unique series ids, we end up with 269 series_ids. So 8 series_ids have no information.","metadata":{}},{"cell_type":"code","source":"training_data_events.dropna().nunique()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.386301Z","iopub.execute_input":"2023-10-30T22:02:01.387840Z","iopub.status.idle":"2023-10-30T22:02:01.410578Z","shell.execute_reply.started":"2023-10-30T22:02:01.387779Z","shell.execute_reply":"2023-10-30T22:02:01.409575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Observation: Wake Up and Onset data points","metadata":{}},{"cell_type":"code","source":"training_events_wake_up = training_data_events[training_data_events[\"event\"] == \"wakeup\"]\ntraining_events_onset = training_data_events[training_data_events[\"event\"] == \"onset\"]","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.413093Z","iopub.execute_input":"2023-10-30T22:02:01.414550Z","iopub.status.idle":"2023-10-30T22:02:01.424098Z","shell.execute_reply.started":"2023-10-30T22:02:01.414511Z","shell.execute_reply":"2023-10-30T22:02:01.423202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_events_onset['datetime'].dt.hour.value_counts().plot(kind='barh')\nplt.title('Onset Times')\nplt.xlabel('Number of cases')\nplt.ylabel('Onset time')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.425384Z","iopub.execute_input":"2023-10-30T22:02:01.426715Z","iopub.status.idle":"2023-10-30T22:02:01.799971Z","shell.execute_reply.started":"2023-10-30T22:02:01.426666Z","shell.execute_reply":"2023-10-30T22:02:01.798764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_events_wake_up['datetime'].dt.hour.value_counts().plot(kind='barh')\nplt.title('WakeUp Times')\nplt.xlabel('Number of cases')\nplt.ylabel('WakeUp time')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:01.801001Z","iopub.execute_input":"2023-10-30T22:02:01.801242Z","iopub.status.idle":"2023-10-30T22:02:02.143293Z","shell.execute_reply.started":"2023-10-30T22:02:01.801220Z","shell.execute_reply":"2023-10-30T22:02:02.142434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\ntrain_events.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:02.144604Z","iopub.execute_input":"2023-10-30T22:02:02.144916Z","iopub.status.idle":"2023-10-30T22:02:02.171121Z","shell.execute_reply.started":"2023-10-30T22:02:02.144889Z","shell.execute_reply":"2023-10-30T22:02:02.170091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:02:02.172276Z","iopub.execute_input":"2023-10-30T22:02:02.172866Z","iopub.status.idle":"2023-10-30T22:03:01.991301Z","shell.execute_reply.started":"2023-10-30T22:02:02.172836Z","shell.execute_reply":"2023-10-30T22:03:01.990475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series['timestamp'].min()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:01.995059Z","iopub.execute_input":"2023-10-30T22:03:01.996118Z","iopub.status.idle":"2023-10-30T22:03:08.107063Z","shell.execute_reply.started":"2023-10-30T22:03:01.996077Z","shell.execute_reply":"2023-10-30T22:03:08.105149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series['timestamp'].max()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:08.108932Z","iopub.execute_input":"2023-10-30T22:03:08.111246Z","iopub.status.idle":"2023-10-30T22:03:13.994121Z","shell.execute_reply.started":"2023-10-30T22:03:08.111214Z","shell.execute_reply":"2023-10-30T22:03:13.992108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_five_series = train_events.groupby('series_id')['event'].count().sort_values(ascending=False).head(5)\ntop_five_series","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:13.996250Z","iopub.execute_input":"2023-10-30T22:03:13.996852Z","iopub.status.idle":"2023-10-30T22:03:14.014260Z","shell.execute_reply.started":"2023-10-30T22:03:13.996804Z","shell.execute_reply":"2023-10-30T22:03:14.012263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Now get the top 5 series id data from train_series and work on the those data points \n\nSince the data has over 127 millions rows. By taking the top 5, we reduce it to 5 million rows.","metadata":{}},{"cell_type":"code","source":"train_series_subset = train_series[train_series['series_id'].isin(top_five_series.index)]\ntrain_series_subset = translate_time(train_series_subset, 'timestamp')\ntrain_series_subset.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:14.016214Z","iopub.execute_input":"2023-10-30T22:03:14.017727Z","iopub.status.idle":"2023-10-30T22:03:23.137072Z","shell.execute_reply.started":"2023-10-30T22:03:14.017636Z","shell.execute_reply":"2023-10-30T22:03:23.134329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The training events now has about 592 rows in the top 5 IDs","metadata":{}},{"cell_type":"code","source":"train_events_subset = train_events[train_events['series_id'].isin(top_five_series.index)]\ntrain_events_subset = translate_time(train_events_subset, 'timestamp')\ntrain_events_subset.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.138452Z","iopub.execute_input":"2023-10-30T22:03:23.138827Z","iopub.status.idle":"2023-10-30T22:03:23.155557Z","shell.execute_reply.started":"2023-10-30T22:03:23.138799Z","shell.execute_reply":"2023-10-30T22:03:23.153942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.158332Z","iopub.execute_input":"2023-10-30T22:03:23.158839Z","iopub.status.idle":"2023-10-30T22:03:23.171952Z","shell.execute_reply.started":"2023-10-30T22:03:23.158797Z","shell.execute_reply":"2023-10-30T22:03:23.169943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this step, we work on the NaN and NaT values for the event and series data points.","metadata":{}},{"cell_type":"code","source":"train_events_subset.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.173969Z","iopub.execute_input":"2023-10-30T22:03:23.174435Z","iopub.status.idle":"2023-10-30T22:03:23.200197Z","shell.execute_reply.started":"2023-10-30T22:03:23.174393Z","shell.execute_reply":"2023-10-30T22:03:23.198809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. DATA PREPARATION\n### Remove all NULL values\n\nThere is not a correlation that can be formed just from just the training events. So we removed all the NULL values in training events subset","metadata":{}},{"cell_type":"code","source":"mask_non_NULL = (~train_events_subset['step'].isnull()) | (~train_events_subset['timestamp'].isnull())\ntrain_events_subset_nonNULL = train_events_subset[mask_non_NULL]\n\ntrain_events_subset_nonNULL.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.201998Z","iopub.execute_input":"2023-10-30T22:03:23.202366Z","iopub.status.idle":"2023-10-30T22:03:23.326368Z","shell.execute_reply.started":"2023-10-30T22:03:23.202337Z","shell.execute_reply":"2023-10-30T22:03:23.324832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_subset_nonNULL['series_id'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.335458Z","iopub.execute_input":"2023-10-30T22:03:23.335822Z","iopub.status.idle":"2023-10-30T22:03:23.346180Z","shell.execute_reply.started":"2023-10-30T22:03:23.335794Z","shell.execute_reply":"2023-10-30T22:03:23.344400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_subset_nonNULL[train_events_subset_nonNULL['series_id'] == '78569a801a38']","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.347726Z","iopub.execute_input":"2023-10-30T22:03:23.348050Z","iopub.status.idle":"2023-10-30T22:03:23.370481Z","shell.execute_reply.started":"2023-10-30T22:03:23.348023Z","shell.execute_reply":"2023-10-30T22:03:23.369471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Looking at the sleep duration\n\n- We perform some exloratory analysis on the sleep events.\n- Duration of sleep gives us an idea of how long people\n- We can use this information to work out states beyond the onset and wakeup values.","metadata":{}},{"cell_type":"code","source":"sleep_duration = train_events_subset_nonNULL.groupby([\n    train_events_subset_nonNULL['series_id'], \n    train_events_subset_nonNULL['night']])['datetime'].agg(['min', 'max']).reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.371860Z","iopub.execute_input":"2023-10-30T22:03:23.372246Z","iopub.status.idle":"2023-10-30T22:03:23.393151Z","shell.execute_reply.started":"2023-10-30T22:03:23.372211Z","shell.execute_reply":"2023-10-30T22:03:23.391279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sleep_duration = sleep_duration.rename(columns={'min': 'onset', 'max': 'wakeup'})\nsleep_duration.head(6)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.394520Z","iopub.execute_input":"2023-10-30T22:03:23.394845Z","iopub.status.idle":"2023-10-30T22:03:23.410847Z","shell.execute_reply.started":"2023-10-30T22:03:23.394820Z","shell.execute_reply":"2023-10-30T22:03:23.409638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sleep_duration['duration'] = ((sleep_duration['wakeup'] - sleep_duration['onset']).dt.seconds / 3600).round(0)\nsleep_duration.head(6)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.412166Z","iopub.execute_input":"2023-10-30T22:03:23.412873Z","iopub.status.idle":"2023-10-30T22:03:23.428272Z","shell.execute_reply.started":"2023-10-30T22:03:23.412842Z","shell.execute_reply":"2023-10-30T22:03:23.427268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"sleep_duration['duration'].value_counts().plot(kind='barh')\n\nplt.title('Sleep duration')\nplt.xlabel('Frequency counts')\nplt.ylabel('Sleep duration in hours')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.429261Z","iopub.execute_input":"2023-10-30T22:03:23.430455Z","iopub.status.idle":"2023-10-30T22:03:23.665178Z","shell.execute_reply.started":"2023-10-30T22:03:23.430402Z","shell.execute_reply":"2023-10-30T22:03:23.663486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### So the people sleep for 9 - 10 hours mostly ","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"## Merge both the datsets based on the user_id\n\n- We can inner join on the timestamp and series_id\n- Merge the values so that we can perform EDA on the anglez and enmo","metadata":{}},{"cell_type":"code","source":"train_series_subset.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.666979Z","iopub.execute_input":"2023-10-30T22:03:23.667406Z","iopub.status.idle":"2023-10-30T22:03:23.682189Z","shell.execute_reply.started":"2023-10-30T22:03:23.667369Z","shell.execute_reply":"2023-10-30T22:03:23.679867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events_subset_nonNULL.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.684033Z","iopub.execute_input":"2023-10-30T22:03:23.684405Z","iopub.status.idle":"2023-10-30T22:03:23.705129Z","shell.execute_reply.started":"2023-10-30T22:03:23.684375Z","shell.execute_reply":"2023-10-30T22:03:23.702928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events = pd.merge(train_series_subset, train_events_subset_nonNULL, on=['series_id', 'timestamp', 'datetime'], how='left')\ntrain_data_events.head(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:23.706377Z","iopub.execute_input":"2023-10-30T22:03:23.706738Z","iopub.status.idle":"2023-10-30T22:03:30.566360Z","shell.execute_reply.started":"2023-10-30T22:03:23.706687Z","shell.execute_reply":"2023-10-30T22:03:30.564418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events = train_data_events.drop(['step_y'], axis=1)\ntrain_data_events = train_data_events.rename(columns = {'step_x' : 'step'})","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:30.568085Z","iopub.execute_input":"2023-10-30T22:03:30.568404Z","iopub.status.idle":"2023-10-30T22:03:31.342369Z","shell.execute_reply.started":"2023-10-30T22:03:30.568380Z","shell.execute_reply":"2023-10-30T22:03:31.340349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:31.343372Z","iopub.execute_input":"2023-10-30T22:03:31.343639Z","iopub.status.idle":"2023-10-30T22:03:31.355040Z","shell.execute_reply.started":"2023-10-30T22:03:31.343618Z","shell.execute_reply":"2023-10-30T22:03:31.353763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events[5496:13104]","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:31.356433Z","iopub.execute_input":"2023-10-30T22:03:31.356821Z","iopub.status.idle":"2023-10-30T22:03:31.384384Z","shell.execute_reply.started":"2023-10-30T22:03:31.356786Z","shell.execute_reply":"2023-10-30T22:03:31.382997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"onset_wakeup_events = train_data_events[~train_data_events['event'].isnull()]\nonset_wakeup_events = onset_wakeup_events.reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:31.385849Z","iopub.execute_input":"2023-10-30T22:03:31.386243Z","iopub.status.idle":"2023-10-30T22:03:31.509366Z","shell.execute_reply.started":"2023-10-30T22:03:31.386208Z","shell.execute_reply":"2023-10-30T22:03:31.507876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"onset_wakeup_events.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:31.510553Z","iopub.execute_input":"2023-10-30T22:03:31.510878Z","iopub.status.idle":"2023-10-30T22:03:31.521829Z","shell.execute_reply.started":"2023-10-30T22:03:31.510855Z","shell.execute_reply":"2023-10-30T22:03:31.520772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### USEFUL  METHODS FOR DATA PROCESSING\n\n- In this processing step we fill some of the NA values for nights and event\n- We create 2 new events sleep and awake to characterize the nights between the onset and wakeup time duration\n- ","metadata":{}},{"cell_type":"code","source":"def attach_nightID_and_sleep_state(\n    user_id: str,\n    train_series: pd.DataFrame, \n    wakeup_onset: pd.DataFrame,\n    init_last_index: int):\n    \n    train_series = train_series[train_series['series_id'] == user_id]\n    train_series = train_series.reset_index()\n    wakeup_onset = wakeup_onset[wakeup_onset['series_id'] == user_id]\n    wakeup_onset = wakeup_onset.reset_index()\n    \n    train_series.loc[0: init_last_index, 'night'] = 1.0\n    train_series.loc[0: init_last_index, 'event'] = 'awake'\n    \n    if user_id == 'f564985ab692':\n        wakeup_onset = wakeup_onset.drop([1, 3])\n    \n    for idx in np.arange(0, len(wakeup_onset), 2):\n        step_sleep = wakeup_onset.iloc[idx]['step']\n        step_wakeup = int(wakeup_onset.iloc[idx + 1]['step'])\n        night = wakeup_onset.iloc[idx]['night']\n        # print(step_sleep, step_wakeup, night)\n\n        train_series.loc[step_sleep+1: step_wakeup, 'night'] = night\n        train_series.loc[step_sleep+1: step_wakeup, 'event'] = 'sleep'\n\n    # print(\"======\")\n    for idx in np.arange(1, len(wakeup_onset) -1, 2):\n        # print(onset_wakeup_events.iloc[idx])\n        night_value = wakeup_onset.iloc[idx]['night']\n        step_sleep = wakeup_onset.iloc[idx]['step']\n        step_wakeup = int(wakeup_onset.iloc[idx + 1]['step'])\n        night = wakeup_onset.iloc[idx + 1]['night']\n        # print(step_sleep, step_wakeup, night)\n\n        train_series.loc[step_sleep+1: step_wakeup, 'night'] = night\n        train_series.loc[step_sleep+1: step_wakeup, 'event'] = 'awake'\n    return train_series, wakeup_onset","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:31.523314Z","iopub.execute_input":"2023-10-30T22:03:31.523970Z","iopub.status.idle":"2023-10-30T22:03:31.539577Z","shell.execute_reply.started":"2023-10-30T22:03:31.523932Z","shell.execute_reply":"2023-10-30T22:03:31.537883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Fix NULLs in the following IDs \n\n- A : 78569a801a38\n- B : f564985ab692\n- C : fb223ed2278c\n- D : f56824b503a0\n- E : cfeb11428dd7","metadata":{"execution":{"iopub.status.busy":"2023-09-24T23:05:37.326090Z","iopub.execute_input":"2023-09-24T23:05:37.327531Z","iopub.status.idle":"2023-09-24T23:05:37.335384Z","shell.execute_reply.started":"2023-09-24T23:05:37.327482Z","shell.execute_reply":"2023-09-24T23:05:37.333842Z"}}},{"cell_type":"markdown","source":"## SOME EDA\n\n## ID : 78569a801a38","metadata":{}},{"cell_type":"code","source":"# Set night from 0 to 5495 for ID A \n\ntrain_data_events_A, wakeup_onset_A = attach_nightID_and_sleep_state(\n    train_series=train_data_events.copy(),\n    wakeup_onset=onset_wakeup_events.copy(),\n    user_id=\"78569a801a38\",\n    init_last_index=5496\n)\n\ntrain_data_events_A.sample(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:31.543394Z","iopub.execute_input":"2023-10-30T22:03:31.543740Z","iopub.status.idle":"2023-10-30T22:03:32.458068Z","shell.execute_reply.started":"2023-10-30T22:03:31.543687Z","shell.execute_reply":"2023-10-30T22:03:32.457035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_unique_nights(data: pd.DataFrame):\n    return pd.Series(data['night'].unique()).dropna()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:32.460319Z","iopub.execute_input":"2023-10-30T22:03:32.460827Z","iopub.status.idle":"2023-10-30T22:03:32.466639Z","shell.execute_reply.started":"2023-10-30T22:03:32.460787Z","shell.execute_reply":"2023-10-30T22:03:32.465244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,60)\nnights = get_unique_nights(train_data_events_A)\nfor idx, night in enumerate(nights):\n    anglez_for_series_A = train_data_events_A[train_data_events_A['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(anglez_for_series_A['step'], anglez_for_series_A['anglez'])\n\n    anglez_for_onset_info_A = wakeup_onset_A[wakeup_onset_A['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(anglez_for_onset_info_A['step'], anglez_for_onset_info_A['anglez'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:32.468077Z","iopub.execute_input":"2023-10-30T22:03:32.468793Z","iopub.status.idle":"2023-10-30T22:03:36.356276Z","shell.execute_reply.started":"2023-10-30T22:03:32.468686Z","shell.execute_reply":"2023-10-30T22:03:36.353866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for \nnights = get_unique_nights(train_data_events_A)\nfor idx, night in enumerate(nights):\n    enmo_for_series_A = train_data_events_A[train_data_events_A['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(enmo_for_series_A['step'], enmo_for_series_A['enmo'])\n\n    enmo_for_onset_series_A = wakeup_onset_A[wakeup_onset_A['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(enmo_for_onset_series_A['step'], enmo_for_onset_series_A['enmo'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:36.357306Z","iopub.execute_input":"2023-10-30T22:03:36.357645Z","iopub.status.idle":"2023-10-30T22:03:39.688865Z","shell.execute_reply.started":"2023-10-30T22:03:36.357614Z","shell.execute_reply":"2023-10-30T22:03:39.687233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ID : f564985ab692","metadata":{}},{"cell_type":"code","source":"# Set night from 0 to 5495 for ID A \n\ntrain_data_events_B, wakeup_onset_B = attach_nightID_and_sleep_state(\n    train_series=train_data_events.copy(),\n    wakeup_onset=onset_wakeup_events.copy(),\n    user_id=\"f564985ab692\",\n    init_last_index=5640\n)\n\nwakeup_onset_B.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:39.690248Z","iopub.execute_input":"2023-10-30T22:03:39.690566Z","iopub.status.idle":"2023-10-30T22:03:40.387883Z","shell.execute_reply.started":"2023-10-30T22:03:39.690539Z","shell.execute_reply":"2023-10-30T22:03:40.386167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.rcParams[\"figure.figsize\"] = (20,20)\n# Plot for \nnights = get_unique_nights(train_data_events_B)\nfor idx, night in enumerate(nights):\n    anglez_for_series_B = train_data_events_B[train_data_events_B['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(anglez_for_series_B['step'], anglez_for_series_B['anglez'])\n\n    anglez_for_onset_series_B = wakeup_onset_B[wakeup_onset_B['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(anglez_for_onset_series_B['step'], anglez_for_onset_series_B['anglez'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:40.390073Z","iopub.execute_input":"2023-10-30T22:03:40.390567Z","iopub.status.idle":"2023-10-30T22:03:42.115884Z","shell.execute_reply.started":"2023-10-30T22:03:40.390526Z","shell.execute_reply":"2023-10-30T22:03:42.114807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for \nplt.rcParams[\"figure.figsize\"] = (20,20)\n# Plot for \nnights = get_unique_nights(train_data_events_B)\nfor idx, night in enumerate(nights):\n    enmo_for_series_B = train_data_events_B[train_data_events_B['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(enmo_for_series_B['step'], enmo_for_series_B['enmo'])\n\n    enmo_for_onset_series_B = wakeup_onset_B[wakeup_onset_B['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(enmo_for_onset_series_B['step'], enmo_for_onset_series_B['enmo'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:42.117108Z","iopub.execute_input":"2023-10-30T22:03:42.117404Z","iopub.status.idle":"2023-10-30T22:03:43.348480Z","shell.execute_reply.started":"2023-10-30T22:03:42.117378Z","shell.execute_reply":"2023-10-30T22:03:43.346855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ID : fb223ed2278c","metadata":{"execution":{"iopub.status.busy":"2023-09-24T23:11:12.365388Z","iopub.execute_input":"2023-09-24T23:11:12.366724Z","iopub.status.idle":"2023-09-24T23:11:12.375975Z","shell.execute_reply.started":"2023-09-24T23:11:12.366671Z","shell.execute_reply":"2023-09-24T23:11:12.374555Z"}}},{"cell_type":"code","source":"# Set night from 0 to 5495 for ID A \n\ntrain_data_events_C, wakeup_onset_C = attach_nightID_and_sleep_state(\n    train_series=train_data_events.copy(),\n    wakeup_onset=onset_wakeup_events.copy(),\n    user_id=\"fb223ed2278c\",\n    init_last_index=7536\n)\n\nwakeup_onset_C.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:43.350081Z","iopub.execute_input":"2023-10-30T22:03:43.350409Z","iopub.status.idle":"2023-10-30T22:03:44.020542Z","shell.execute_reply.started":"2023-10-30T22:03:43.350378Z","shell.execute_reply":"2023-10-30T22:03:44.018907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for \n# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,20)\n# Plot for \nnights = get_unique_nights(train_data_events_C)\nfor idx, night in enumerate(nights):\n    enmo_for_series_C = train_data_events_C[train_data_events_C['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(enmo_for_series_C['step'], enmo_for_series_C['enmo'])\n\n    enmo_for_onset_series_C = wakeup_onset_C[wakeup_onset_C['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(enmo_for_onset_series_C['step'], enmo_for_onset_series_C['enmo'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:44.022715Z","iopub.execute_input":"2023-10-30T22:03:44.023091Z","iopub.status.idle":"2023-10-30T22:03:45.373242Z","shell.execute_reply.started":"2023-10-30T22:03:44.023058Z","shell.execute_reply":"2023-10-30T22:03:45.372286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,20)\n# Plot for \nnights = get_unique_nights(train_data_events_C)\nfor idx, night in enumerate(nights):\n    anglez_for_series_C = train_data_events_C[train_data_events_C['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(anglez_for_series_C['step'], anglez_for_series_C['anglez'])\n\n    anglez_for_onset_series_C = wakeup_onset_C[wakeup_onset_C['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(anglez_for_onset_series_C['step'], anglez_for_onset_series_C['anglez'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:45.374509Z","iopub.execute_input":"2023-10-30T22:03:45.374842Z","iopub.status.idle":"2023-10-30T22:03:47.097456Z","shell.execute_reply.started":"2023-10-30T22:03:45.374806Z","shell.execute_reply":"2023-10-30T22:03:47.096062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ID : f56824b503a0","metadata":{}},{"cell_type":"code","source":"# Set night from 0 to 5495 for ID D\n\ntrain_data_events_D, wakeup_onset_D = attach_nightID_and_sleep_state(\n    train_series=train_data_events.copy(),\n    wakeup_onset=onset_wakeup_events.copy(),\n    user_id=\"f56824b503a0\",\n    init_last_index=24228\n)\n\nwakeup_onset_D.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:47.098739Z","iopub.execute_input":"2023-10-30T22:03:47.099075Z","iopub.status.idle":"2023-10-30T22:03:47.808797Z","shell.execute_reply.started":"2023-10-30T22:03:47.099033Z","shell.execute_reply":"2023-10-30T22:03:47.807647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,40)\n# Plot for \nnights = get_unique_nights(train_data_events_D)\nfor idx, night in enumerate(nights):\n    anglez_for_series_D = train_data_events_D[train_data_events_D['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(anglez_for_series_D['step'], anglez_for_series_D['anglez'])\n\n    anglez_for_onset_series_D = wakeup_onset_D[wakeup_onset_D['night'] == night]\n    # print(anglez_for_onset_info_A.shape)\n    plt.plot(anglez_for_onset_series_D['step'], anglez_for_onset_series_D['anglez'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:47.809920Z","iopub.execute_input":"2023-10-30T22:03:47.810233Z","iopub.status.idle":"2023-10-30T22:03:50.903616Z","shell.execute_reply.started":"2023-10-30T22:03:47.810207Z","shell.execute_reply":"2023-10-30T22:03:50.902754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for\n# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,45)\n# Plot for \nnights = get_unique_nights(train_data_events_D)\nfor idx, night in enumerate(nights):\n    enmo_for_series_D = train_data_events_D[train_data_events_D['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(enmo_for_series_D['step'], enmo_for_series_D['enmo'])\n\n    enmo_for_onset_series_D = wakeup_onset_D[wakeup_onset_D['night'] == night]\n    # print(enmo_for_onset_series_D.shape)\n    plt.plot(enmo_for_onset_series_D['step'], enmo_for_onset_series_D['enmo'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:50.904798Z","iopub.execute_input":"2023-10-30T22:03:50.905268Z","iopub.status.idle":"2023-10-30T22:03:54.102274Z","shell.execute_reply.started":"2023-10-30T22:03:50.905240Z","shell.execute_reply":"2023-10-30T22:03:54.101151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ID : cfeb11428dd7","metadata":{}},{"cell_type":"code","source":"# Set night from 0 to 5495 for ID D\n\ntrain_data_events_E, wakeup_onset_E = attach_nightID_and_sleep_state(\n    train_series=train_data_events.copy(),\n    wakeup_onset=onset_wakeup_events.copy(),\n    user_id=\"cfeb11428dd7\",\n    init_last_index=7200\n)\n\nwakeup_onset_E","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:54.103762Z","iopub.execute_input":"2023-10-30T22:03:54.104293Z","iopub.status.idle":"2023-10-30T22:03:54.781314Z","shell.execute_reply.started":"2023-10-30T22:03:54.104264Z","shell.execute_reply":"2023-10-30T22:03:54.779310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,45)\n# Plot for \nnights = get_unique_nights(train_data_events_E)\nfor idx, night in enumerate(nights):\n    anglez_for_series_E = train_data_events_E[train_data_events_E['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(anglez_for_series_E['step'], anglez_for_series_E['anglez'])\n\n    anglez_for_onset_series_E = wakeup_onset_E[wakeup_onset_E['night'] == night]\n    # print(enmo_for_onset_series_D.shape)\n    plt.plot(anglez_for_onset_series_E['step'], anglez_for_onset_series_E['anglez'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:54.783465Z","iopub.execute_input":"2023-10-30T22:03:54.783895Z","iopub.status.idle":"2023-10-30T22:03:58.057595Z","shell.execute_reply.started":"2023-10-30T22:03:54.783867Z","shell.execute_reply":"2023-10-30T22:03:58.056290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot for \n# Plot for\nplt.rcParams[\"figure.figsize\"] = (20,45)\n# Plot for \nnights = get_unique_nights(train_data_events_E)\nfor idx, night in enumerate(nights):\n    enmo_for_series_E = train_data_events_E[train_data_events_E['night'] == night]\n    plt.subplot(len(nights), 1, idx + 1)\n    plt.title(f'Night : {night}')\n    plt.plot(enmo_for_series_E['step'], enmo_for_series_E['enmo'])\n\n    enmo_for_onset_series_E = wakeup_onset_E[wakeup_onset_E['night'] == night]\n    # print(enmo_for_onset_series_D.shape)\n    plt.plot(enmo_for_onset_series_E['step'], enmo_for_onset_series_E['enmo'], color='red')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:03:58.059112Z","iopub.execute_input":"2023-10-30T22:03:58.059452Z","iopub.status.idle":"2023-10-30T22:04:00.840721Z","shell.execute_reply.started":"2023-10-30T22:03:58.059421Z","shell.execute_reply":"2023-10-30T22:04:00.839278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events.event.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:00.842727Z","iopub.execute_input":"2023-10-30T22:04:00.843125Z","iopub.status.idle":"2023-10-30T22:04:00.933440Z","shell.execute_reply.started":"2023-10-30T22:04:00.843092Z","shell.execute_reply":"2023-10-30T22:04:00.932203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:00.935047Z","iopub.execute_input":"2023-10-30T22:04:00.935399Z","iopub.status.idle":"2023-10-30T22:04:00.947118Z","shell.execute_reply.started":"2023-10-30T22:04:00.935368Z","shell.execute_reply":"2023-10-30T22:04:00.945495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events_E.event.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:00.948919Z","iopub.execute_input":"2023-10-30T22:04:00.949760Z","iopub.status.idle":"2023-10-30T22:04:00.987318Z","shell.execute_reply.started":"2023-10-30T22:04:00.949718Z","shell.execute_reply":"2023-10-30T22:04:00.985526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wakeup_onset_E.event.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:00.989730Z","iopub.execute_input":"2023-10-30T22:04:00.990575Z","iopub.status.idle":"2023-10-30T22:04:00.999999Z","shell.execute_reply.started":"2023-10-30T22:04:00.990536Z","shell.execute_reply":"2023-10-30T22:04:00.998016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events_E","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:01.001808Z","iopub.execute_input":"2023-10-30T22:04:01.003450Z","iopub.status.idle":"2023-10-30T22:04:01.027325Z","shell.execute_reply.started":"2023-10-30T22:04:01.003398Z","shell.execute_reply":"2023-10-30T22:04:01.026193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events[train_data_events['series_id'] == 'f564985ab692']['event'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:01.029275Z","iopub.execute_input":"2023-10-30T22:04:01.029952Z","iopub.status.idle":"2023-10-30T22:04:01.449402Z","shell.execute_reply.started":"2023-10-30T22:04:01.029915Z","shell.execute_reply":"2023-10-30T22:04:01.448253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events['series_id'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:01.451862Z","iopub.execute_input":"2023-10-30T22:04:01.452236Z","iopub.status.idle":"2023-10-30T22:04:01.631018Z","shell.execute_reply.started":"2023-10-30T22:04:01.452201Z","shell.execute_reply":"2023-10-30T22:04:01.629278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events['event'].fillna(\"NoChange\", inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:01.632725Z","iopub.execute_input":"2023-10-30T22:04:01.633071Z","iopub.status.idle":"2023-10-30T22:04:01.865247Z","shell.execute_reply.started":"2023-10-30T22:04:01.633039Z","shell.execute_reply":"2023-10-30T22:04:01.863326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def event_cat_mapper(x):\n    if x == \"NoChange\":\n        return 0\n    elif x == \"wakeup\":\n        return 1\n    return 2\ntrain_data_events[\"event_cat\"] = train_data_events['event'].apply(event_cat_mapper)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:01.866787Z","iopub.execute_input":"2023-10-30T22:04:01.867076Z","iopub.status.idle":"2023-10-30T22:04:03.382589Z","shell.execute_reply.started":"2023-10-30T22:04:01.867053Z","shell.execute_reply":"2023-10-30T22:04:03.381021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.plot(train_data_events[train_data_events['series_id'] == 'cfeb11428dd7']['event_cat'])\nplt.figure(figsize=(10, 1))\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:03.384580Z","iopub.execute_input":"2023-10-30T22:04:03.384985Z","iopub.status.idle":"2023-10-30T22:04:04.410514Z","shell.execute_reply.started":"2023-10-30T22:04:03.384954Z","shell.execute_reply":"2023-10-30T22:04:04.409556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events[\"y\"] = \"NULL\"","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:04.412450Z","iopub.execute_input":"2023-10-30T22:04:04.412863Z","iopub.status.idle":"2023-10-30T22:04:04.449599Z","shell.execute_reply.started":"2023-10-30T22:04:04.412829Z","shell.execute_reply":"2023-10-30T22:04:04.447866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for idx in train_data_events['series_id'].unique():\n    print(idx)\n    val = 0\n    for index, row in train_data_events[train_data_events['series_id'] == idx].iterrows():\n        if row['event'] == \"NoChange\":\n            train_data_events.loc[index, 'y'] = val\n        else:\n            if row['event'] == 'wakeup':\n                val = 0\n                train_data_events.loc[index, 'y'] = val\n            else:\n                val = 1\n                train_data_events.loc[index, 'y'] = val","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:04:04.451538Z","iopub.execute_input":"2023-10-30T22:04:04.452012Z","iopub.status.idle":"2023-10-30T22:15:57.482547Z","shell.execute_reply.started":"2023-10-30T22:04:04.451973Z","shell.execute_reply":"2023-10-30T22:15:57.480346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_data_events['series_id'].unique())","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:15:57.485031Z","iopub.execute_input":"2023-10-30T22:15:57.485406Z","iopub.status.idle":"2023-10-30T22:15:57.832784Z","shell.execute_reply.started":"2023-10-30T22:15:57.485381Z","shell.execute_reply":"2023-10-30T22:15:57.831515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events['y'].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:15:57.834413Z","iopub.execute_input":"2023-10-30T22:15:57.834891Z","iopub.status.idle":"2023-10-30T22:15:58.001409Z","shell.execute_reply.started":"2023-10-30T22:15:57.834858Z","shell.execute_reply":"2023-10-30T22:15:57.999657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.plot(train_data_events[train_data_events['series_id'] == 'f564985ab692']['event_cat'])\nplt.plot(train_data_events[train_data_events['series_id'] == 'f564985ab692']['y'])\nplt.figure(figsize=(10, 1))","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:15:58.003148Z","iopub.execute_input":"2023-10-30T22:15:58.003498Z","iopub.status.idle":"2023-10-30T22:15:59.658108Z","shell.execute_reply.started":"2023-10-30T22:15:58.003467Z","shell.execute_reply":"2023-10-30T22:15:59.656866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:15:59.660233Z","iopub.execute_input":"2023-10-30T22:15:59.660654Z","iopub.status.idle":"2023-10-30T22:15:59.687630Z","shell.execute_reply.started":"2023-10-30T22:15:59.660618Z","shell.execute_reply":"2023-10-30T22:15:59.685919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_events['y'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:15:59.689709Z","iopub.execute_input":"2023-10-30T22:15:59.690156Z","iopub.status.idle":"2023-10-30T22:15:59.879373Z","shell.execute_reply.started":"2023-10-30T22:15:59.690121Z","shell.execute_reply":"2023-10-30T22:15:59.878093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = train_data_events.drop(columns=['y', 'series_id', 'night', 'event', 'event_cat', 'timestamp', 'datetime'])\ny = train_data_events['y'].astype('int')","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:15:59.880912Z","iopub.execute_input":"2023-10-30T22:15:59.881354Z","iopub.status.idle":"2023-10-30T22:16:00.086809Z","shell.execute_reply.started":"2023-10-30T22:15:59.881326Z","shell.execute_reply":"2023-10-30T22:16:00.085329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\nX_train","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:00.089086Z","iopub.execute_input":"2023-10-30T22:16:00.089529Z","iopub.status.idle":"2023-10-30T22:16:01.145167Z","shell.execute_reply.started":"2023-10-30T22:16:00.089490Z","shell.execute_reply":"2023-10-30T22:16:01.143239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:01.147064Z","iopub.execute_input":"2023-10-30T22:16:01.147485Z","iopub.status.idle":"2023-10-30T22:16:01.197631Z","shell.execute_reply.started":"2023-10-30T22:16:01.147451Z","shell.execute_reply":"2023-10-30T22:16:01.195858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:01.199399Z","iopub.execute_input":"2023-10-30T22:16:01.199893Z","iopub.status.idle":"2023-10-30T22:16:01.228201Z","shell.execute_reply.started":"2023-10-30T22:16:01.199857Z","shell.execute_reply":"2023-10-30T22:16:01.226416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nlog_reg = LogisticRegression()\nlog_reg.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:01.230866Z","iopub.execute_input":"2023-10-30T22:16:01.231254Z","iopub.status.idle":"2023-10-30T22:16:04.975514Z","shell.execute_reply.started":"2023-10-30T22:16:01.231224Z","shell.execute_reply":"2023-10-30T22:16:04.974157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"log_reg_predictions = log_reg.predict(X_test)\nfrom sklearn.metrics import f1_score, accuracy_score\n# f1_score(y_test, log_reg_predictions)\naccuracy_score(y_test, log_reg_predictions)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:04.979248Z","iopub.execute_input":"2023-10-30T22:16:04.982378Z","iopub.status.idle":"2023-10-30T22:16:05.087490Z","shell.execute_reply.started":"2023-10-30T22:16:04.982339Z","shell.execute_reply":"2023-10-30T22:16:05.086720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sum(log_reg_predictions)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:05.100205Z","iopub.execute_input":"2023-10-30T22:16:05.103752Z","iopub.status.idle":"2023-10-30T22:16:05.185298Z","shell.execute_reply.started":"2023-10-30T22:16:05.103713Z","shell.execute_reply":"2023-10-30T22:16:05.184018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.linalg.norm(train_data_events[(train_data_events['y'] == 1) & (train_data_events['series_id'] == 'f564985ab692')]['anglez'].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:05.186335Z","iopub.execute_input":"2023-10-30T22:16:05.186643Z","iopub.status.idle":"2023-10-30T22:16:05.770818Z","shell.execute_reply.started":"2023-10-30T22:16:05.186612Z","shell.execute_reply":"2023-10-30T22:16:05.769769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.linalg.norm(train_data_events[(train_data_events['y'] == 0) & (train_data_events['series_id'] == 'f564985ab692')]['anglez'].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:05.772185Z","iopub.execute_input":"2023-10-30T22:16:05.772796Z","iopub.status.idle":"2023-10-30T22:16:06.424045Z","shell.execute_reply.started":"2023-10-30T22:16:05.772758Z","shell.execute_reply":"2023-10-30T22:16:06.422683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.linalg.norm(train_data_events[(train_data_events['y'] == 1) & (train_data_events['series_id'] == '78569a801a38')]['anglez'].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:06.426113Z","iopub.execute_input":"2023-10-30T22:16:06.426561Z","iopub.status.idle":"2023-10-30T22:16:07.012668Z","shell.execute_reply.started":"2023-10-30T22:16:06.426527Z","shell.execute_reply":"2023-10-30T22:16:07.011265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.linalg.norm(train_data_events[(train_data_events['y'] == 0) & (train_data_events['series_id'] == '78569a801a38')]['anglez'].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:07.014582Z","iopub.execute_input":"2023-10-30T22:16:07.015022Z","iopub.status.idle":"2023-10-30T22:16:07.711379Z","shell.execute_reply.started":"2023-10-30T22:16:07.014988Z","shell.execute_reply":"2023-10-30T22:16:07.709711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:07.713274Z","iopub.execute_input":"2023-10-30T22:16:07.713820Z","iopub.status.idle":"2023-10-30T22:16:07.720330Z","shell.execute_reply.started":"2023-10-30T22:16:07.713783Z","shell.execute_reply":"2023-10-30T22:16:07.718230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)  # You can adjust the number of estimators as needed\nrf_classifier.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:16:07.722445Z","iopub.execute_input":"2023-10-30T22:16:07.722927Z","iopub.status.idle":"2023-10-30T22:37:54.006219Z","shell.execute_reply.started":"2023-10-30T22:16:07.722888Z","shell.execute_reply":"2023-10-30T22:37:54.004794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = rf_classifier.predict(X_test)\nfrom sklearn.metrics import accuracy_score, confusion_matrix, classification_report\n","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:41:56.319240Z","iopub.execute_input":"2023-10-30T22:41:56.319712Z","iopub.status.idle":"2023-10-30T22:42:09.709494Z","shell.execute_reply.started":"2023-10-30T22:41:56.319657Z","shell.execute_reply":"2023-10-30T22:42:09.707951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy = accuracy_score(y_test, y_pred) * 100\nconf_matrix = confusion_matrix(y_test, y_pred)\nclassification_rep = classification_report(y_test, y_pred)\n\nprint(\"Accuracy:\", accuracy)\nprint(\"Confusion Matrix:\\n\", conf_matrix)\nprint(\"Classification Report:\\n\", classification_rep)\n","metadata":{"execution":{"iopub.status.busy":"2023-10-30T22:44:06.068230Z","iopub.execute_input":"2023-10-30T22:44:06.068649Z","iopub.status.idle":"2023-10-30T22:44:07.175835Z","shell.execute_reply.started":"2023-10-30T22:44:06.068618Z","shell.execute_reply":"2023-10-30T22:44:07.174650Z"},"trusted":true},"execution_count":null,"outputs":[]}]}