{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import HTML\nimport time\n\nhandle = display(HTML(\"\"\"👌\"\"\"), display_id='html_marquee1')\ntime.sleep(2)\nhandle = display(HTML(\"\"\"~ \"There is a time for many words, and there is also a time for sleep.\" – Homer, The Odyssey\"\"\"), display_id='html_marquee1', update=True)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:01.840222Z","iopub.execute_input":"2023-10-23T20:28:01.840891Z","iopub.status.idle":"2023-10-23T20:28:03.881333Z","shell.execute_reply.started":"2023-10-23T20:28:01.840857Z","shell.execute_reply":"2023-10-23T20:28:03.880205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Executive Summary
\n\n
📌 Summary: This notebook is intended to \n
\n\nBased on the insights derived in the EDA / data analytics drills across this notebook, the solid ML experiments can be set up down the road.","metadata":{}},{"cell_type":"code","source":"import gc\nimport pandas as pd\nimport numpy as np\nimport datetime as dt\n\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nimport seaborn as sns\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nimport plotly.offline\n\nfrom colorama import Fore, Style, init\nfrom pprint import pprint\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:03.884905Z","iopub.execute_input":"2023-10-23T20:28:03.885200Z","iopub.status.idle":"2023-10-23T20:28:06.446393Z","shell.execute_reply.started":"2023-10-23T20:28:03.885177Z","shell.execute_reply":"2023-10-23T20:28:06.445287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Color printing\n# inspired by https://www.kaggle.com/code/ravi20076/sleepstate-eda-baseline\ndef PrintColor(text:str, color = Fore.BLUE, style = Style.BRIGHT):\n \"Prints color outputs using colorama using a text F-string\";\n print(style + color + text + Style.RESET_ALL);","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:06.447562Z","iopub.execute_input":"2023-10-23T20:28:06.448034Z","iopub.status.idle":"2023-10-23T20:28:06.454088Z","shell.execute_reply.started":"2023-10-23T20:28:06.448008Z","shell.execute_reply":"2023-10-23T20:28:06.452733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
About the Competition and Its Dataset
\n\nThe **goal of the competition** is to detect sleep onset and wake. We are requested to develop a model trained on wrist-worn accelerometer data in order to determine a person's sleep state.\n\nThe training dataset is composed of\n- The Multi-day recordings of wrist-worn accelerometer (provided as a parquet file).\n- The labelling file with the data annotated with two event types: **onset**, the beginning of sleep, and **wakeup**, the end of the sleeping time (provided as a CSV file).\n\nBoth data files represent the essential time series-type data.\n\nFor the testing set, only the accelerometer-based data (in parquet format) are provided.\n\nEssential summary on the data for the contest is quoted below, as per the note from the contest hosts\n\n
\n
✅ Data description: The dataset comprises about 500 multi-day recordings of wrist-worn accelerometer data annotated with two event types: onset, the beginning of sleep, and wakeup, the end of sleep. Your task is to detect the occurrence of these two events in the accelerometer series.

\n\nWhile sleep logbooks remain the gold-standard, when working with accelerometer data we refer to sleep as the longest single period of inactivity while the watch is being worn. For this data, we have guided raters with several concrete instructions:\n\n
\n \nThough each series is a continuous recording, there may be periods in the series when the accelerometer device was removed. These period are determined as those where suspiciously little variation in the accelerometer signals occur over an extended period of time, which is unrealistic for typical human participants. Events are not annotated for these periods, and you should attempt to refrain from making event predictions during these periods: an event prediction will be scored as false positive.
\n\nEach data series represents this continuous (multi-day/event) recording for a unique experimental subject.\n
\n
\n\n##
train_series.parquet Details
\n\nThis file represents series to be used as training data. Each series is a continuous recording of accelerometer data for a single subject spanning many days.\n\nThe table below summarizes the structure of the initial accelerometer data provided:\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
AttributeDescriptionComments
series_idUnique identifier for each accelerometer series 
stepAn integer timestep for each observation within a series.It is unique within a series
timestampA corresponding datetime in ISO 8601 format: %Y-%m-%dT%H:%M:%S%z. 
anglezAs calculated and described by the GGIR package, z-angle is a metric derived from individual accelerometer components that is commonly used in sleep detection, and refers to the angle of the arm relative to the vertical axis of the bodyYou can refer to Accelerometer data processing with GGIR for more details
enmoAs calculated and described by the GGIR package, ENMO is the Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero. While no standard measure of acceleration exists in this space, this is one of the several commonly computed featuresYou can refer to Accelerometer data processing with GGIR for more details
","metadata":{"_kg_hide-output":true}},{"cell_type":"markdown","source":"##
test_series.parquet Details
\n\nThis file contains the details on the series to be used as the test data, containing the same fields as above. The goal is to predict event occurrences for series in this file.\n","metadata":{}},{"cell_type":"markdown","source":"##
train_events.csv Details
\n\nThis file represents series to be used as training data. Each series is a continuous recording of accelerometer data for a single subject spanning many days.\n\nThe table below summarizes the structure of the initial accelerometer data provided:\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
AttributeDescriptionComments
series_idUnique identifier for each accelerometer series, corresponding to the identifies for each series of accelerometer data in train_series.parquet. 
nightAn enumeration of potential onset / wakeup event pairs. At most one pair of events can occur for each night. 
eventThe type of event, whether onset or wakeup. 
stepAn integer timestep for each observation within a series.It is unique within a series
timestampThe recorded time of occurence of the event in the accelerometer series in ISO 8601 format: %Y-%m-%dT%H:%M:%S%z. 
","metadata":{}},{"cell_type":"markdown","source":"#
Reading Data
\n\n##
Initial Data Inspection
\n\nFirst of all, we are going to read the data in memory.\n\nSince the size of the dataset (especially, its parquet parts with accelerometer data feeds) is quite big, time- and memory-effective data loading should be implemented.\n\nTherefore, we are going to use Polars, the fast alternative to Pandas. So, out data loading strategy is going to be as follows\n\n- use Polars to read both the parquet and csv files in memory\n- convert the output to the Pandas dataframes to enable data analytics and EDA experiments as needed\n- as a part of such data loading, we will also extract the features from timestamp values in series and events data (namely, we will decompose the timestamps into year, month, day, and hour components to enable insight-reach EDA down the road)\n- compress the series data in memory by reducing the RAM usage by the respective Pandas dataframes\n\n
✅ Note: Pandas is an essential Python library for Data Science. But its biggest downside is that it can be slow for operations on large datasets. Polars is a Pandas alternative designed to process data faster. You can refer to the references provided in \"References\" chapter below to learn more about Polars.
\n","metadata":{}},{"cell_type":"code","source":"%%time\n# inspired by https://www.kaggle.com/code/enricomanosperti/detect-sleep-states-first-preprocessing-and-eda\nimport polars as pl\ntrain_series = (pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\n .with_columns(\n (\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.year().alias(\"year\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.month().alias(\"month\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.day().alias(\"day\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.hour().alias(\"hour\")),\n )\n )\n .collect()\n .to_pandas()\n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:06.458344Z","iopub.execute_input":"2023-10-23T20:28:06.458739Z","iopub.status.idle":"2023-10-23T20:28:53.789002Z","shell.execute_reply.started":"2023-10-23T20:28:06.458709Z","shell.execute_reply":"2023-10-23T20:28:53.787649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series.head().style.set_caption(\"Sample of data in the train series data file\"). \\\nset_properties(**{'border': '1.3px solid blue',\n 'color': 'grey'})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.790122Z","iopub.execute_input":"2023-10-23T20:28:53.790483Z","iopub.status.idle":"2023-10-23T20:28:53.808551Z","shell.execute_reply.started":"2023-10-23T20:28:53.790453Z","shell.execute_reply":"2023-10-23T20:28:53.807567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.809693Z","iopub.execute_input":"2023-10-23T20:28:53.810383Z","iopub.status.idle":"2023-10-23T20:28:53.824770Z","shell.execute_reply.started":"2023-10-23T20:28:53.810338Z","shell.execute_reply":"2023-10-23T20:28:53.824163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_events = (pl.scan_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\n .with_columns(\n (\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.year().alias(\"year\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.month().alias(\"month\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.day().alias(\"day\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.hour().alias(\"hour\")),\n )\n )\n .collect()\n .to_pandas()\n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.825909Z","iopub.execute_input":"2023-10-23T20:28:53.826185Z","iopub.status.idle":"2023-10-23T20:28:53.877767Z","shell.execute_reply.started":"2023-10-23T20:28:53.826164Z","shell.execute_reply":"2023-10-23T20:28:53.876586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events.head().style.set_caption(\"Sample of data in the train events data file\"). \\\nset_properties(**{'border': '1.3px solid blue',\n 'color': 'grey'})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.878721Z","iopub.execute_input":"2023-10-23T20:28:53.879549Z","iopub.status.idle":"2023-10-23T20:28:53.893708Z","shell.execute_reply.started":"2023-10-23T20:28:53.879522Z","shell.execute_reply":"2023-10-23T20:28:53.892480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.895325Z","iopub.execute_input":"2023-10-23T20:28:53.895693Z","iopub.status.idle":"2023-10-23T20:28:53.913531Z","shell.execute_reply.started":"2023-10-23T20:28:53.895666Z","shell.execute_reply":"2023-10-23T20:28:53.912384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest_series = (pl.scan_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet')\n .with_columns(\n (\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.year().alias(\"year\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.month().alias(\"month\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.day().alias(\"day\")),\n (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\").dt.hour().alias(\"hour\")),\n )\n )\n .collect()\n .to_pandas()\n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.918087Z","iopub.execute_input":"2023-10-23T20:28:53.918426Z","iopub.status.idle":"2023-10-23T20:28:53.937226Z","shell.execute_reply.started":"2023-10-23T20:28:53.918394Z","shell.execute_reply":"2023-10-23T20:28:53.936019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.938319Z","iopub.execute_input":"2023-10-23T20:28:53.938936Z","iopub.status.idle":"2023-10-23T20:28:53.952302Z","shell.execute_reply.started":"2023-10-23T20:28:53.938908Z","shell.execute_reply":"2023-10-23T20:28:53.951420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before we do the analytical drills, let's optimize the RAM consumption of the notebook, with compressing the series data in memorty.","metadata":{}},{"cell_type":"code","source":"%%time\n# inspired by https://www.kaggle.com/code/renatoreggiani/reduce-memory-usage-zzzs-cmi\n# with tweaks determined by the selected polars loading strategy\n# tweaks inspired by https://github.com/softhints/Pandas-Tutorials/blob/master/column/3.check-dtype-column-columns-pandas-dataframe.ipynb\nfrom pandas.api.types import is_datetime64_ns_dtype\nfrom pandas.api.types import is_datetime64_any_dtype\ndef reduce_mem_usage(df):\n \"\"\" iterate through all numeric columns of a dataframe and modify the data type\n to reduce memory usage. \n \"\"\"\n start_mem = df.memory_usage().sum() / 1024**2\n print(f'Memory usage of dataframe is {start_mem:.2f} MB')\n \n for col in df.columns:\n col_type = df[col].dtype\n\n # is_datetime64_any_dtype is more generic then is_datetime64_ns_dtype\n if col_type != object and not is_datetime64_any_dtype(df[col]):\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float16)\n\n end_mem = df.memory_usage().sum() / 1024**2\n print(f'Memory usage after optimization is: {end_mem:.2f} MB')\n decrease = 100 * (start_mem - end_mem) / start_mem\n print(f'Decreased by {decrease:.2f}%')\n \n return df\n\ntrain_series = reduce_mem_usage(train_series)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:53.953709Z","iopub.execute_input":"2023-10-23T20:28:53.954599Z","iopub.status.idle":"2023-10-23T20:28:57.701468Z","shell.execute_reply.started":"2023-10-23T20:28:53.954568Z","shell.execute_reply":"2023-10-23T20:28:57.700472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest_series = reduce_mem_usage(test_series)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-10-23T20:28:57.703005Z","iopub.execute_input":"2023-10-23T20:28:57.703325Z","iopub.status.idle":"2023-10-23T20:28:57.715884Z","shell.execute_reply.started":"2023-10-23T20:28:57.703297Z","shell.execute_reply":"2023-10-23T20:28:57.714792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Missing Values Review: Train Events Data
\n\nLet's review the spread of the missing values in the train events data.","metadata":{}},{"cell_type":"code","source":"for col in train_events.columns:\n msg = 'column: {:>10}\\t Percent of NaN value: {:.2f}%'.format(col,100*(train_events[col].isnull().sum()/train_events[col].shape[0]))\n PrintColor(f\"\\n---> {msg}\");\n #print(msg)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:57.717398Z","iopub.execute_input":"2023-10-23T20:28:57.717713Z","iopub.status.idle":"2023-10-23T20:28:57.734349Z","shell.execute_reply.started":"2023-10-23T20:28:57.717688Z","shell.execute_reply":"2023-10-23T20:28:57.732701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's visualize the missing values spread in the dataset. \n\n
✅ Tip: The good instrument to quickly visualize the missing values in a dataset is the specialized library - missingno. We are going to demonstrate it below.\n
\n\nFirst of all, we will look at the ratio of the missing values as a bar chart","metadata":{}},{"cell_type":"code","source":"import missingno as msno\n\nmsno.bar(train_events, color=(0.3,0.3,0.5))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:57.736137Z","iopub.execute_input":"2023-10-23T20:28:57.737012Z","iopub.status.idle":"2023-10-23T20:28:58.686666Z","shell.execute_reply.started":"2023-10-23T20:28:57.736975Z","shell.execute_reply":"2023-10-23T20:28:58.685616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we are going to look at the actual records inn the train events data affected in the matrix view","metadata":{}},{"cell_type":"code","source":"msno.matrix(train_events, color=(0.3,0.3,0.5))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:58.687668Z","iopub.execute_input":"2023-10-23T20:28:58.688673Z","iopub.status.idle":"2023-10-23T20:28:59.219055Z","shell.execute_reply.started":"2023-10-23T20:28:58.688646Z","shell.execute_reply":"2023-10-23T20:28:59.217978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, \n\n- There is a fraction of the series with incomplete labelling of sleeping/wakening events. \n- As we can see from the diagrams, the event labelling for such series is somewhat shrinked by certain dates.\n- Another insight is, the % of missing data in *step* and *timestamp* attributes is quite high (*33.93%*). \n\nIt makes it really unproductive to attempt to impute such missing values with any synthetical approach as it would introduce the type 2 noise in our dataset. If we do not find the logically determinant way of filling the missing values, it would be better to shrink the incompletely labelled series from the training set exposed to the ML experiments down the road.\n\nLet's look at the training events deeper to understand the impact of the shrinked labelling for some of the time series.","metadata":{}},{"cell_type":"markdown","source":"##
Missing Values Review: Train Series Data
\n\nLet's review the spread of the missing values in the train series data.","metadata":{}},{"cell_type":"code","source":"for col in train_series.columns:\n msg = 'column: {:>10}\\t Percent of NaN value: {:.2f}%'.format(col,100*(train_series[col].isnull().sum()/train_series[col].shape[0]))\n PrintColor(f\"\\n---> {msg}\");\n #print(msg)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:28:59.220384Z","iopub.execute_input":"2023-10-23T20:28:59.220774Z","iopub.status.idle":"2023-10-23T20:29:04.464271Z","shell.execute_reply.started":"2023-10-23T20:28:59.220740Z","shell.execute_reply":"2023-10-23T20:29:04.463273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that there are no missing values in the train series data (accelerometer-based time series).","metadata":{}},{"cell_type":"markdown","source":"##
Labelling Coverage for Train Series Data
\n\nLet's review how many seires are fully labelled with the events.","metadata":{}},{"cell_type":"code","source":"# base statistics\ntrain_events.describe(include='all').style.set_caption(\"Summary statistics for training events\"). \\\nset_properties(**{'border': '1.3px solid blue',\n 'color': 'grey'})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:04.465491Z","iopub.execute_input":"2023-10-23T20:29:04.465752Z","iopub.status.idle":"2023-10-23T20:29:04.504847Z","shell.execute_reply.started":"2023-10-23T20:29:04.465731Z","shell.execute_reply":"2023-10-23T20:29:04.503654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n- There are *277* unique series in the train events (labels) file\n- The length of the series ranges from *1 to 84* nights, with the median to be *14* nights and the average length to be around *15* nights\n- The observations that collected series data were between *'2017-08-05 22:44:00'* and *'2019-07-05 05:33:00'*, UTC \n- The series with the most labelled events has *id = '78569a801a38'*\n- There are NaN values in the *step* column, which means that we can not use those instances for generating the label\n- It may sound we will have to choose only those series that do not have any missing steps events\n\nNow, let's visualize how many training series have full coverage with event labelling","metadata":{}},{"cell_type":"code","source":"series_steps = train_events.groupby(['series_id'])['step'].apply(lambda x: x.isnull().any())\n\nframe = {'series_id': series_steps.index,\n 'has_missing_labels': series_steps}\n \n# Creating DataFrame by passing Dictionary\nagg_data = pd.DataFrame(frame)\n\ndfg = agg_data['has_missing_labels'].value_counts().reset_index()\ndfg.columns = ['has_missing_labels', 'number_of_series']\nfig = px.bar(dfg, x='has_missing_labels', y='number_of_series', \n title='Series with and without missing event labels',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:04.506078Z","iopub.execute_input":"2023-10-23T20:29:04.506387Z","iopub.status.idle":"2023-10-23T20:29:06.157475Z","shell.execute_reply.started":"2023-10-23T20:29:04.506342Z","shell.execute_reply":"2023-10-23T20:29:06.156812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that only *37* series out of *277* have fully labelled training events. Such series are going to be used in the model training down the road.\n\nLet's further drill into event labelling across the series. We are going to visualize the counts of series\n- with fully labelled events\n- with partially labelled events\n- without any events labelled at all","metadata":{}},{"cell_type":"code","source":"pivot = pd.pivot_table(train_events, index='series_id', columns='event', values='timestamp', aggfunc=['min', 'max'])\npivot.columns = [f'{level}_{col}_ts' for level, col in pivot.columns]\n\nevents_summary = train_events \\\n .groupby('series_id').agg({'series_id': 'count', 'step': 'count'}) \\\n .rename({'series_id': 'count_events', 'step': 'count_non_null_events'}, axis=1) \\\n .join(pivot, how='left')\n\nevents_summary.insert(\n 2, 'pct_non_null_events', events_summary.count_non_null_events / events_summary.count_events\n)\n\nevents_summary = events_summary.sort_values(['pct_non_null_events', 'count_non_null_events'], ascending=[False, True])\n\ncomplete_series = events_summary.query('pct_non_null_events == 1.0')\nempty_series = events_summary.query('pct_non_null_events == 0.0')\nincomplete_series = events_summary.query('(pct_non_null_events > 0.0) & (pct_non_null_events < 1.0)')\n\nsummary_evt = {\n 'event_coverage': ['Full', 'Partial', 'Missing'],\n 'series_count': [complete_series.shape[0], incomplete_series.shape[0], empty_series.shape[0]]\n}\n\nagg_data = pd.DataFrame(summary_evt)\nfig = px.bar(agg_data, x='event_coverage', y='series_count', \n title='Number of series with full, partial, and missing event label coverage',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:06.159001Z","iopub.execute_input":"2023-10-23T20:29:06.159480Z","iopub.status.idle":"2023-10-23T20:29:06.257642Z","shell.execute_reply.started":"2023-10-23T20:29:06.159454Z","shell.execute_reply":"2023-10-23T20:29:06.256636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now review the distribution of the training series by the percentage of labelled events (1 - fully labelled series; 0 - series without labelled events; 0..1 - series with parcially labelled events).","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-10-08T20:43:32.340289Z","iopub.execute_input":"2023-10-08T20:43:32.340691Z","iopub.status.idle":"2023-10-08T20:43:32.351265Z","shell.execute_reply.started":"2023-10-08T20:43:32.340662Z","shell.execute_reply":"2023-10-08T20:43:32.350203Z"}}},{"cell_type":"code","source":"fig = px.histogram(events_summary, x=\"pct_non_null_events\", marginal=\"rug\",\n title='Distribution of Training Series by % events labelled',\n color_discrete_sequence=px.colors.qualitative.Prism)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:06.258676Z","iopub.execute_input":"2023-10-23T20:29:06.258913Z","iopub.status.idle":"2023-10-23T20:29:06.392851Z","shell.execute_reply.started":"2023-10-23T20:29:06.258894Z","shell.execute_reply":"2023-10-23T20:29:06.391952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's display the ids of series with fully labelled events.","metadata":{}},{"cell_type":"code","source":"no_NaN_series = series_steps[~series_steps].index.tolist()\n\nPrintColor(f\"\\n---> Displaing the ids of series with fully labelled events:\\n\");\npprint(no_NaN_series, indent = 10, width = 50);\n\n# print(no_NaN_series)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:06.394070Z","iopub.execute_input":"2023-10-23T20:29:06.394382Z","iopub.status.idle":"2023-10-23T20:29:06.401221Z","shell.execute_reply.started":"2023-10-23T20:29:06.394335Z","shell.execute_reply":"2023-10-23T20:29:06.400042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Data Inspection Summary
\n\nBelow are the major takeaways from our data inspection drill\n\n- Train events csv fill shall be used to generate the label data for train series provided as a parquet file.\n- The envisioned ML approaches is to tackle either a binary (Awake/Onset) or multiclass (namely, 3-class) event classification problem (with target classess to be Awake, Onset, and Other).\n- The significant fraction of the series provided in the training set has incomplete labelling; therefore, one of the approaches to training the ML/DL models for this contest is to prepare a training subset with completely labelled series only.\n- We should have a look at individual series to get deeper-level insights about the data. The focus should be on *37* series with complete event labelling data we identified above.\n\n
✅ Note: It was Carl McBride Ellis who first paid attention to the issue with incomplete labelling for some of the series within this dataset. He put together a smaller dataset with 35 well-formed series to use in model training down the road.\n
","metadata":{}},{"cell_type":"markdown","source":"#
Deep Dive into Training Events
\n\nLet's dive deeper into training events data to mine more insights.\n\n##
General Summary for Train Events Data
","metadata":{}},{"cell_type":"code","source":"# inspired by https://www.kaggle.com/code/rishabh15virgo/cmi-dss-first-impression-data-understanding-eda\ndef summarize_dataframe(df):\n summary_df = pd.DataFrame(df.dtypes, columns=['dtypes'])\n summary_df['missing#'] = df.isna().sum().values*100\n summary_df['missing%'] = (df.isna().sum().values*100)/len(df)\n summary_df['uniques'] = df.nunique().values\n summary_df['count'] = df.count().values\n #sum['skew'] = df.skew().values\n desc = pd.DataFrame(df.describe().T)\n summary_df['min'] = desc['min']\n summary_df['max'] = desc['max']\n summary_df['mean'] = desc['mean']\n return summary_df\n\nsummarize_dataframe(train_events).style.background_gradient(cmap='Purples')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:06.402717Z","iopub.execute_input":"2023-10-23T20:29:06.403219Z","iopub.status.idle":"2023-10-23T20:29:06.499346Z","shell.execute_reply.started":"2023-10-23T20:29:06.403187Z","shell.execute_reply":"2023-10-23T20:29:06.498238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Training Events Density
\n\nLet's visualize the density of the train_series. In fact, it will draw the availability matrix for the events across the series and dates (1: available; 0: NaN).\n\nThis visualization helps to digest the periods during which events (*'onset'* and *'wakeup'*) have been recorded.\n\nWe are going to engage with clustermap for this visualization, where the y-axis represents individual time series clustered based on their data collection similarity.","metadata":{}},{"cell_type":"code","source":"# Pivot the DataFrame to create df_availability\ntr_ev = train_events.copy()\ntr_ev['dates'] = tr_ev['timestamp'].dt.date\ndf_availability = tr_ev.pivot_table(columns='dates', index='series_id', values='step', aggfunc='count')\n\n# Fill NA values with 0\ndf_availability = (~df_availability.isna()).fillna(0).astype('int')\n\n# Plotting the clustered heatmap\nsns.clustermap(df_availability, col_cluster=False, figsize=(12, 8))\n\n# TODO: re-implement for plotly per\n# https://stackoverflow.com/questions/66547583/plotly-clustered-heatmap-with-dendrogram-python\n# https://plotly.com/python/clustergram/#default-clustergram\n# https://plotly.com/python/dendrogram/","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:06.500483Z","iopub.execute_input":"2023-10-23T20:29:06.500742Z","iopub.status.idle":"2023-10-23T20:29:07.767604Z","shell.execute_reply.started":"2023-10-23T20:29:06.500713Z","shell.execute_reply":"2023-10-23T20:29:07.766952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Upon examining the clustermap, it is evident that the majority of the series tend to have a duration spanning 20 to 30 days (see also below for the night-based insights into the series durations).\n\nThe periods of time during which data have been collected are sparsely distributed across the years 2017 to 2019.\n\nWhenever timestamps are discussed, it is essential to note the relations between \n- *steps* and *timestamps*\n- impact of the sparsity of the series data on the usability of time-driven features\n\nAs per the description of the dataset, *step* is the basis for the evaluation of submitted predictions. *timestamps*, in turn, is generally used as an input for features (provided you tackle the prediction problem in the tabular classification setup).\n\nSince the training seria samples are very sparse, it could look like that only hourly/minute trends could make some sense for the model. Trends of higher scale (daily, weekly etc.) would be misleading for the model to train in turn.\n\n
✅ Note: More advanced time series insights on enmo/anglez are collected in a separate notebook. More specifically, it will contain the results of the clustering experiments to group series by the statistical trends displayed by enmo/anglez aggregated values.\n
","metadata":{}},{"cell_type":"code","source":"del tr_ev, df_availability\n\n# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:07.768704Z","iopub.execute_input":"2023-10-23T20:29:07.769452Z","iopub.status.idle":"2023-10-23T20:29:07.882761Z","shell.execute_reply.started":"2023-10-23T20:29:07.769427Z","shell.execute_reply":"2023-10-23T20:29:07.882124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Balance Between Onset and Wakeup Events
\n\nLet's now visualize the frequency of *onset* and *wakeup* events by hours in a day.\n\nIn the context, *onset* and *wakeup* events signify the times people go to sleep and wake up, respectively.\n\nUnderstanding the temporal patters in manifesting these events can draw critical insights into the sleeping behavior.","metadata":{}},{"cell_type":"code","source":"df_onset = train_events[train_events['event'] == 'onset']\ndf_wakeup = train_events[train_events['event'] == 'wakeup']\n\nfig = make_subplots(rows=1, cols=2, subplot_titles=('Hourly Distr. of Onset Events',\n 'Hourly Distr. of Wakeup Events',\n ))\n\nfig.add_trace(go.Histogram(x=df_onset['hour'].dropna(), nbinsx=24), row=1, col=1)\nfig.add_trace(go.Histogram(x=df_wakeup['hour'].dropna(), nbinsx=24), row=1, col=2)\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:07.884602Z","iopub.execute_input":"2023-10-23T20:29:07.884925Z","iopub.status.idle":"2023-10-23T20:29:07.984000Z","shell.execute_reply.started":"2023-10-23T20:29:07.884897Z","shell.execute_reply":"2023-10-23T20:29:07.983116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the resulting histograms, the x-axis denotes the 24-hour timeline, and the y-axis represents the frequency of *onset* and *wakeup* events.\n\nFrom the charts displayed, it is evident that the majority of individuals whose sleeping series are represented in the training dataset tend to \n- wake up between 6-8 am;\n- go to sleep around 8-10 pm.\n\nSuch a temporal pattern matches well with general intuition as well as provides a useful baseline for further analyses.\n\nLet's now review the balance between the labelled *onset* and *wakeup* events in the training dataset.","metadata":{}},{"cell_type":"code","source":"agg_data = train_events[[\"event\", \"hour\"]].groupby([\"event\", \"hour\"]).size().reset_index(name=\"count\")\nagg_data[\"hour\"] = agg_data[\"hour\"].astype(int)\n# define figure element\nfig = px.sunburst(\n agg_data,\n values='count',\n path=[\"event\", \"hour\"],\n title=\"Events by hours in a day\",\n color=\"event\",\n height=600,\n color_discrete_sequence=px.colors.qualitative.Prism,\n #color_continuous_scale=px.colors.sequential.Viridis\n)\n# display the figure\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:07.985128Z","iopub.execute_input":"2023-10-23T20:29:07.985417Z","iopub.status.idle":"2023-10-23T20:29:08.100991Z","shell.execute_reply.started":"2023-10-23T20:29:07.985389Z","shell.execute_reply":"2023-10-23T20:29:08.099813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the number of *onset* and *wakeup* events is quite balanced in the training dataset, which indicates no labelling errors on the series where labels were generated (either fully or parcially).\n\nHowever, as displayed in the sunburst chart above, the number of *wakeup* events is slightly higher then the number of *onset* events (the difference is 3 events, namely). It means the tiny fraction of the training series does have more *wakeup* events labeled than *onset* once. Let's locate the training series affected by such a mislabelling.","metadata":{}},{"cell_type":"code","source":"agg_data = train_events[[\"series_id\", \"event\"]].groupby([\"series_id\", \"event\"]).size().reset_index(name=\"count\")\n\ndf_wide=pd.pivot(agg_data, index=['series_id'], columns = ['event'],values = 'count') #Reshape from long to wide\n\n#Re-arange the new columns in the correct order\ncols = agg_data['event'].unique()\ndf_wide=df_wide[cols]\ndf_wide['diff'] = df_wide['wakeup'] - df_wide['onset']\ndf_wide.reset_index(inplace=True)\ndf_wide = df_wide.rename(columns = {'index':'series_id'})\n\ndf_unbalanced = df_wide[df_wide['diff'] !=0 ]\n\nunbalanced_series = df_unbalanced['series_id'].unique()\n\nif len(unbalanced_series) > 0:\n PrintColor(f\"\\n---> Displaing the ids of series where wakeups to do not equal to onset events:\\n\")\n pprint(unbalanced_series, indent = 10, width = 50)\nelse:\n PrintColor(f\"\\n---> No series with the unbalanced events detected\")\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.108194Z","iopub.execute_input":"2023-10-23T20:29:08.108519Z","iopub.status.idle":"2023-10-23T20:29:08.126065Z","shell.execute_reply.started":"2023-10-23T20:29:08.108490Z","shell.execute_reply":"2023-10-23T20:29:08.125097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Event Minute Patterns
\n\nLet's review the distribution of *onset/wakeup* events within the minutes of 15-min intervals (if we split the entire timeline of the training series into such 15-min intervals, it would be interesting to search for any regular pattern in the event distribution).","metadata":{}},{"cell_type":"code","source":"mn_train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\nmn_train_events = mn_train_events.loc[~mn_train_events['timestamp'].isna()] # drop NA\n\ndf_onset_mn = mn_train_events[mn_train_events['event'] == 'onset']\ndf_wakeup_mn = mn_train_events[mn_train_events['event'] == 'wakeup']\n\nmin15_onset = df_onset_mn['timestamp'].str[-10:-8].astype('int32')%15 # minute of event, in 15 min blocks\nmin15_wakeup = df_wakeup_mn['timestamp'].str[-10:-8].astype('int32')%15 # minute of event, in 15 min blocks\n\nframe1 = {'interval': min15_onset.index,\n 'onset_minute': min15_onset}\n \n# Creating DataFrame by passing Dictionary\nagg_data_onsets = pd.DataFrame(frame1)\n\nframe2 = {'interval': min15_wakeup.index,\n 'wakeup_minute': min15_wakeup}\n \n# Creating DataFrame by passing Dictionary\nagg_data_wakeups = pd.DataFrame(frame2)\n\nfig = make_subplots(rows=1, cols=2, subplot_titles=('Distr. of Onset Events in 15-min intervals',\n 'Distr. of Wakeup Events in 15-min intervals',\n ))\n\nfig.add_trace(go.Histogram(x=agg_data_onsets['onset_minute'].dropna(), nbinsx=24), row=1, col=1)\nfig.add_trace(go.Histogram(x=agg_data_wakeups['wakeup_minute'].dropna(), nbinsx=24), row=1, col=2)\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=700,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.127432Z","iopub.execute_input":"2023-10-23T20:29:08.127792Z","iopub.status.idle":"2023-10-23T20:29:08.210076Z","shell.execute_reply.started":"2023-10-23T20:29:08.127761Z","shell.execute_reply":"2023-10-23T20:29:08.208539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe a strange pattern in the event records, if they are looked up on 15-min intervals.\n\n- They are always on exact minutes, maybe due to the manual classification/labelling leading to limited precision\n- They demonstrate repeatable patterns in 15 minutes cycle\n- The most of events (both *onset* and *wakeup*) happen on 0th, 3rd, 7th, and 11th minutes of 15-min intervals (0 stands for the first minute of the interval, and 14 stands for the last one)\n- The distribution and period of minute differ by event type, if looking at the more granular level \n\nAdditional note for *wakeups*, we can see almost regularly equal number of events in every minute category within the 4 event frequency buckets there\n- the most frequent event minutes (0, 3, 7, 11 minutes)\n- second the most frequent event minutes (2, 6, 10, 14 minutes) - they stay 1 min behind the minutes in the most frequent bucket, btw\n- modereate events bucket minutes (4, 8, 12 minutes)\n- least frequent event bucket (1, 5, 9, 13 minutes)\n\nIt is hard to expain the rationale behind the patterns observed. Nor is it clear if it is possible to use such a discovery in the ML solutions to predict the sleeping events down the road.\n\n
✅ Note: Such a strange phenomenon has been first mentioned in one of the discussion threads.
","metadata":{}},{"cell_type":"markdown","source":"As mentioned by [Daniel Dewey](https://www.kaggle.com/dan3dewey) in the comments to the above-mentioned discussion thread, the observed phenomenon indicates that\n- The \"ground truth\" in the Events file is not exactly correct within 2 minutes or so. \n- A model that correctly identifies the real onsets/wakeups scored against this would not get an AP of 1.0, maybe a max of 0.95\n- It is a function of the human rater (a person who labelled the events in the training set) and the tools they used\n- The event minutes pattern is likely to follow Binomial distribution\n\nAs Daniel outliened,\n
\n \"It's actually a classic Binomial distribution if we take n = number of events in the series and x = sum of events in 0,3,7,11 bins. Based on about 9600 events in all and 4 bins of 1450 in @ymatioun 's plot, the probability of an event being in the summed-bins is p ~ 41450/9600 ~ 0.60 (0,3,7,11-to-total ratio 0.60). The output below is from Matt Bognar's nice Binomial applet. It shows that if all the series follow this pattern then the probability of a series with 18 nights, n=36 events, having 15 or fewer (ratio 0.42 or less) in the combined 0,3,7,11 bins is unlikely, only 2 %. The evenly-distributed case would peak at about (4/15)36 ~ 9 or 10 events (ratio 0.26). So, if some series are \"0,3,7,11-quantized\" and some are evenly-distributed, the 0,3,7,11-to-total ratio histogram might show two peaks\".\n
\n\nLet's try to identify clusters of series with similar raters-tools. There aren't many events per series so comparing the number in 0, 3, 7, 11 vs the number of others could be a measure of the effect with some amount of signal to noise.","metadata":{}},{"cell_type":"markdown","source":"### Onset Event Patterns\n\nLet's look at *Onset* event patterns. Initially, we are going to plot the chart to visualize relation between the number of onset events in [0,3,7,11] minute bin vs. the events in other bin.","metadata":{}},{"cell_type":"code","source":"def is_in_frequent_event_minute_slot(x):\n value = 0\n if x in [0,3,7,11]:\n value = 1\n return value\n\ndef is_in_other_event_minute_slot(x):\n value = 0\n if x not in [0,3,7,11]:\n value = 1\n return value\n\ndf_onset_mn['timestamp'] = pd.to_datetime(df_onset_mn['timestamp']).apply(lambda t: t.tz_localize(None))\ndf_onset_mn['hour'] = df_onset_mn[\"timestamp\"].dt.hour\ndf_onset_mn['minute'] = df_onset_mn[\"timestamp\"].dt.minute\ndf_onset_mn['15min'] = df_onset_mn['minute'] % 15\n\ndf_onset_mn['in_0_3_7_11_slot'] = df_onset_mn['15min'].apply(lambda x: is_in_frequent_event_minute_slot(x))\ndf_onset_mn['in_other_slot'] = df_onset_mn['15min'].apply(lambda x: is_in_other_event_minute_slot(x))\n\ndf_series_agg = df_onset_mn.groupby(['series_id'])[['in_0_3_7_11_slot', 'in_other_slot']].sum().reset_index()\n\nfig = px.scatter(df_series_agg, x=\"in_0_3_7_11_slot\", y=\"in_other_slot\",\n size='in_0_3_7_11_slot', \n title='Onset events in minute buckets: [0,3,7,11] vs. others',\n color_discrete_sequence=px.colors.qualitative.Prism,\n hover_data=['series_id'])\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.211564Z","iopub.execute_input":"2023-10-23T20:29:08.211900Z","iopub.status.idle":"2023-10-23T20:29:08.371391Z","shell.execute_reply.started":"2023-10-23T20:29:08.211872Z","shell.execute_reply":"2023-10-23T20:29:08.370433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that \n- there is a clear separation between the series in terms of the event distribution (some series are \"0,3,7,11-quantized\" and some are evenly-distributed)\n- the majority of series tend to have more onset events recorded in [0,3,7,11] minute bin.\n\nLet's now calculate the ratios of events in [0,3,7,11] minute bin and in other bin for each series as well as plot the histograms of them.","metadata":{}},{"cell_type":"code","source":"df_series_agg['ratio_0_3_7_11'] = df_series_agg['in_0_3_7_11_slot'] / (df_series_agg['in_0_3_7_11_slot'] + df_series_agg['in_other_slot'])\ndf_series_agg['ratio_other'] = 1 - df_series_agg['ratio_0_3_7_11']\n\nfig = make_subplots(rows=1, cols=2, subplot_titles=('Distr. of Onsets ratio in [0,3,7,11] minute bucket',\n 'Distr. of Onsets ratio in other minute bucket',\n ))\n\nfig.add_trace(go.Histogram(x=df_series_agg['ratio_0_3_7_11'].dropna(), nbinsx=24), row=1, col=1)\nfig.add_trace(go.Histogram(x=df_series_agg['ratio_other'].dropna(), nbinsx=24), row=1, col=2)\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.372608Z","iopub.execute_input":"2023-10-23T20:29:08.372894Z","iopub.status.idle":"2023-10-23T20:29:08.430885Z","shell.execute_reply.started":"2023-10-23T20:29:08.372872Z","shell.execute_reply":"2023-10-23T20:29:08.429876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n- there are two closely located peaks at 0.525-05749 and 0.625-06749, respectively; so, we can conclude it fits into the binomial distribution\n- there is a large cluster of series in terms of the spread of `ratio_0_3_7_11` values (between 0.525 and 0.6749, 114 out of 277 series inclusive) \n- the overall distribution of `ratio_0_3_7_11` is skewed to the right, indicating more series to have events recorded in the *[0,3,7,11]* minute bin","metadata":{}},{"cell_type":"markdown","source":"### Wakeup Event Patterns\n\nLet's look at *Wakeup* event patterns now. Initially, we are going to plot the chart to visualize relation between the number of wakeup events in [0,3,7,11] minute bin vs. the events in other bin.","metadata":{}},{"cell_type":"code","source":"df_wakeup_mn['timestamp'] = pd.to_datetime(df_wakeup_mn['timestamp']).apply(lambda t: t.tz_localize(None))\ndf_wakeup_mn['hour'] = df_wakeup_mn[\"timestamp\"].dt.hour\ndf_wakeup_mn['minute'] = df_wakeup_mn[\"timestamp\"].dt.minute\ndf_wakeup_mn['15min'] = df_wakeup_mn['minute'] % 15\n\ndf_wakeup_mn['in_0_3_7_11_slot'] = df_wakeup_mn['15min'].apply(lambda x: is_in_frequent_event_minute_slot(x))\ndf_wakeup_mn['in_other_slot'] = df_wakeup_mn['15min'].apply(lambda x: is_in_other_event_minute_slot(x))\n\ndf_series_agg = df_wakeup_mn.groupby(['series_id'])[['in_0_3_7_11_slot', 'in_other_slot']].sum().reset_index()\n\nfig = px.scatter(df_series_agg, x=\"in_0_3_7_11_slot\", y=\"in_other_slot\",\n size='in_0_3_7_11_slot', \n title='Wakeup events in minute buckets: [0,3,7,11] vs. others',\n color_discrete_sequence=px.colors.qualitative.Prism,\n hover_data=['series_id'])\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.432219Z","iopub.execute_input":"2023-10-23T20:29:08.432541Z","iopub.status.idle":"2023-10-23T20:29:08.553970Z","shell.execute_reply.started":"2023-10-23T20:29:08.432515Z","shell.execute_reply":"2023-10-23T20:29:08.552669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that \n- there is a clear separation between the series in terms of the event distribution (some series are \"0,3,7,11-quantized\" and some are evenly-distributed)\n- the majority of series tend to have more wakeup events recorded in [0,3,7,11] minute bin.\n\nLet's now calculate the ratios of events in [0,3,7,11] minute bin and in other bin for each series as well as plot the histograms of them.","metadata":{}},{"cell_type":"code","source":"df_series_agg['ratio_0_3_7_11'] = df_series_agg['in_0_3_7_11_slot'] / (df_series_agg['in_0_3_7_11_slot'] + df_series_agg['in_other_slot'])\ndf_series_agg['ratio_other'] = 1 - df_series_agg['ratio_0_3_7_11']\n\nfig = make_subplots(rows=1, cols=2, subplot_titles=('Distr. of Wakeups ratio in [0,3,7,11] minute bucket',\n 'Distr. of Wakeups ratio in other minute bucket',\n ))\n\nfig.add_trace(go.Histogram(x=df_series_agg['ratio_0_3_7_11'].dropna(), nbinsx=24), row=1, col=1)\nfig.add_trace(go.Histogram(x=df_series_agg['ratio_other'].dropna(), nbinsx=24), row=1, col=2)\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=800,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.555299Z","iopub.execute_input":"2023-10-23T20:29:08.555643Z","iopub.status.idle":"2023-10-23T20:29:08.614619Z","shell.execute_reply.started":"2023-10-23T20:29:08.555617Z","shell.execute_reply":"2023-10-23T20:29:08.613825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n- there are two peaks at 0.625-06749 and 0.975-1, respectively; so, we can conclude it fits into the binomial distribution\n- there is a large cluster of series in terms of the spread of `ratio_0_3_7_11` values (between 0.525 and 0.8249, 171 out of 277 series inclusive) \n- the overall distribution of `ratio_0_3_7_11` is skewed to the right, indicating more series to have events recorded in the *[0,3,7,11]* minute bin","metadata":{}},{"cell_type":"code","source":"del mn_train_events, df_onset_mn, df_wakeup_mn, agg_data_onsets, agg_data_wakeups, df_series_agg, frame1, frame2\n\n# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.615961Z","iopub.execute_input":"2023-10-23T20:29:08.616245Z","iopub.status.idle":"2023-10-23T20:29:08.740077Z","shell.execute_reply.started":"2023-10-23T20:29:08.616221Z","shell.execute_reply":"2023-10-23T20:29:08.739073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Sleep Duration
\n\nLet's review how many hours the people observed in the training set sleep in average.","metadata":{}},{"cell_type":"code","source":"# inspired by https://www.kaggle.com/code/alinasri/sleep-data-analysis-some-eda\n\n# Lets focus on non null rows of the training events only\nmask_nonull= (~train_events['step'].isnull()) & (~train_events['timestamp'].isnull()) \ntrain_events_nonull = train_events[mask_nonull]\n\n# Group by 'series_id' and calculate sleep duration for each night\nsleep_duration_df = train_events_nonull.groupby([train_events_nonull['series_id'], train_events_nonull['night']])['timestamp'].agg(['min', 'max']).reset_index()\nsleep_duration_df = sleep_duration_df.rename(columns={'min': 'onset', 'max': 'wakeup'})\nsleep_duration_df['sleep_duration'] = ((sleep_duration_df['wakeup'] - sleep_duration_df['onset']).dt.seconds / 3600).round(0)\n\ndfg = sleep_duration_df['sleep_duration'].value_counts().reset_index()\ndfg.columns = ['sleep_duration', 'number_of_observations']\nfig = px.bar(dfg, x='sleep_duration', y='number_of_observations', \n title='Sleep Duration, Training seria (hours)',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.741185Z","iopub.execute_input":"2023-10-23T20:29:08.741521Z","iopub.status.idle":"2023-10-23T20:29:08.807082Z","shell.execute_reply.started":"2023-10-23T20:29:08.741493Z","shell.execute_reply":"2023-10-23T20:29:08.806178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the most of the training observations (within the series-to-nights dimensions) demonstrated the sleep duration to be between 8 and 10 h per night (day), with the sleeping time of 9 h per day to be the most frequent value. ","metadata":{}},{"cell_type":"markdown","source":"##
Nights Data Insights
","metadata":{}},{"cell_type":"code","source":"nights_in_series = train_events.groupby('series_id')['night'].max()\n\nframe = {'series_id': nights_in_series.index,\n 'number_of_nights_in_seria': nights_in_series}\n \n# Creating DataFrame by passing Dictionary\nagg_data = pd.DataFrame(frame)\n\nfig = px.histogram(agg_data, x=\"number_of_nights_in_seria\", marginal=\"rug\",\n title='Distribution of Nights per Series in Training Events',\n color_discrete_sequence=px.colors.qualitative.Prism)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.808093Z","iopub.execute_input":"2023-10-23T20:29:08.808332Z","iopub.status.idle":"2023-10-23T20:29:08.874099Z","shell.execute_reply.started":"2023-10-23T20:29:08.808313Z","shell.execute_reply":"2023-10-23T20:29:08.873179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n\n- the majority of the series has *20-29* nights in it\n- the second 'cluster' contains the series with *30-39* nights in it\n- the tails of the distribution are minor; \n- we can indicate there are exceptionally long ('outlier') series with *84* and *62* nights, respectively","metadata":{}},{"cell_type":"markdown","source":"##
Steps Data Insights
","metadata":{}},{"cell_type":"code","source":"min_steps_in_series = train_series.groupby('series_id')['step'].min()\nmax_steps_in_series = train_series.groupby('series_id')['step'].max()\n\nframe1 = {'series_id': min_steps_in_series.index,\n 'min_number_of_steps_in_seria': min_steps_in_series}\n\nframe2 = {'series_id': max_steps_in_series.index,\n 'max_number_of_steps_in_seria': max_steps_in_series}\n \n# Creating DataFrames by passing Dictionary\nmin_steps_df = pd.DataFrame(frame1)\nmax_steps_df = pd.DataFrame(frame2)\n\nmin_steps_df.reset_index(drop = True, inplace = True)\nmax_steps_df.reset_index(drop = True, inplace = True)\n\nagg_data = pd.merge(min_steps_df, max_steps_df, how='inner', on=['series_id'])\nagg_data['steps_number'] = agg_data['max_number_of_steps_in_seria'] - agg_data['min_number_of_steps_in_seria']\n\nfig = px.histogram(agg_data, x=\"steps_number\", marginal=\"rug\",\n title='Distribution of Steps per Series in Training Series',\n color_discrete_sequence=px.colors.qualitative.Prism)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:08.875478Z","iopub.execute_input":"2023-10-23T20:29:08.875800Z","iopub.status.idle":"2023-10-23T20:29:27.200656Z","shell.execute_reply.started":"2023-10-23T20:29:08.875772Z","shell.execute_reply":"2023-10-23T20:29:27.199521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the distribution of steps follows the general distributions by nights, more or less. Every step corresponds to one instance of the accelerometer measurements that are recorded every 5 s (if no errors/breaks happen).","metadata":{}},{"cell_type":"markdown","source":"#
Timestamp-Related Insights
\n\nIn this chapter, we are going to investigate several time-related issues as follows\n\n- Shall we keep the events and *enmo/anglez* series timestamps in local time zone time or in UTC?\n- What could be the effect of daylight time saving shifts on the series data?\n\n##
To UTC or not to UTC?
\n\nSince the timestamps in both the series and events data are presented with the local time zone offsets, the real question is whether it makes sense to keep the event time in the local time zones or cast them all to UTC. This section is going to visualize some TZ-related patterns. \n\n
✅ Note: It was Carl McBride Ellis who first paid attention to the UTC vs. local time zone issues. He put together a quick notebook to spot the problem. We are going to elevate it to the next level by visualizing the patterns.\n
","metadata":{}},{"cell_type":"code","source":"ground_truth = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")\n\n# get event counts in UTC\ndf = ground_truth.copy()\ndf[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"],utc=True)\ndf[\"hour\"] = df[\"timestamp\"].dt.hour\n# filter for mornings and afternons\ndf_morning = df.loc[df[\"hour\"] <= 12].copy()\ndf_afternoon = df.loc[df[\"hour\"] > 12].copy()\n\nutc_stat = {\n \"label\": \"UTC\",\n \"morning_wakeups\": df_morning.loc[df_morning[\"event\"] == \"wakeup\"].shape[0],\n \"morning_onsets\": df_morning.loc[df_morning[\"event\"] == \"onset\"].shape[0],\n \"afternoon_onsets\": df_afternoon.loc[df_afternoon[\"event\"] == \"onset\"].shape[0],\n \"afternoon_wakeups\": df_afternoon.loc[df_afternoon[\"event\"] == \"wakeup\"].shape[0]\n}\n\n# get event counts in the local time zone time\ndf = ground_truth.copy()\ndf['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\ndf[\"hour\"] = df[\"timestamp\"].dt.hour\n# filter for mornings and afternons\ndf_morning = df.loc[df[\"hour\"] <= 12].copy()\ndf_afternoon = df.loc[df[\"hour\"] > 12].copy()\n\nlocal_tz_stat = {\n \"label\": \"Local time\",\n \"morning_wakeups\": df_morning.loc[df_morning[\"event\"] == \"wakeup\"].shape[0],\n \"morning_onsets\": df_morning.loc[df_morning[\"event\"] == \"onset\"].shape[0],\n \"afternoon_onsets\": df_afternoon.loc[df_afternoon[\"event\"] == \"onset\"].shape[0],\n \"afternoon_wakeups\": df_afternoon.loc[df_afternoon[\"event\"] == \"wakeup\"].shape[0]\n}\n\ndata = [utc_stat, local_tz_stat]\ndf_stat = pd.DataFrame.from_dict(data)\n\nfig = make_subplots(rows=2, cols=2, subplot_titles=('Morning Wakeups: UTC vs. Local time',\n 'Morning Onsets: UTC vs. Local time',\n 'Afternoon Onsets: UTC vs. Local time',\n 'Afternoon Wakeups: UTC vs. Local time'\n ))\n\nfig.add_trace(go.Bar(x=df_stat['label'], y=df_stat['morning_wakeups']), row=1, col=1)\nfig.add_trace(go.Bar(x=df_stat['label'], y=df_stat['morning_onsets']), row=1, col=2)\nfig.add_trace(go.Bar(x=df_stat['label'], y=df_stat['afternoon_onsets']), row=2, col=1)\nfig.add_trace(go.Bar(x=df_stat['label'], y=df_stat['afternoon_wakeups']), row=2, col=2)\n \n# Update visual layout\nfig.update_layout(\n showlegend=False,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n)\n\n# update font size at the axes\nfig.update_coloraxes(colorbar_tickfont_size=10)\n# Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\nfig.update_annotations(font_size=12)\n# Reduce opacity\nfig.update_traces(opacity=0.75)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:27.202172Z","iopub.execute_input":"2023-10-23T20:29:27.202508Z","iopub.status.idle":"2023-10-23T20:29:27.441547Z","shell.execute_reply.started":"2023-10-23T20:29:27.202480Z","shell.execute_reply":"2023-10-23T20:29:27.440544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that operating the events / series timestamps in the local time makes more sense from the logical standpoint as the number of wakeup/onset events in the morning and afternoon looks more intuitive.\n\n
🚩 Note: Using the timestamps in UTC vs. the local time zone's time makes a difference when it comes to the visualizations and other EDA activities. However, using either UTC or local time stamps does not make a big difference for ML models to be trained. There is a DST change date-related glitch in the raw training series data provided by the contest host. Its assessment, along with the impact on the data loading strategies and workarounds, is extensively covered in a separate notebook of mine.
","metadata":{}},{"cell_type":"code","source":"del ground_truth, df, df_morning, df_afternoon, df_stat\n\n\n# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:27.443570Z","iopub.execute_input":"2023-10-23T20:29:27.443931Z","iopub.status.idle":"2023-10-23T20:29:27.575861Z","shell.execute_reply.started":"2023-10-23T20:29:27.443904Z","shell.execute_reply":"2023-10-23T20:29:27.574459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Deep Dive into Series Data
\n\n##
Initial Data Inspection
\n\nLet's look at the series data both in the training and testing sets. Before jumping deeply inside the series insights, let's shrink the training series data to only contain 37 series fully covered with the labelled events data.","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_series = train_series[train_series['series_id'].isin(no_NaN_series)]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:27.577062Z","iopub.execute_input":"2023-10-23T20:29:27.577392Z","iopub.status.idle":"2023-10-23T20:29:33.582373Z","shell.execute_reply.started":"2023-10-23T20:29:27.577348Z","shell.execute_reply":"2023-10-23T20:29:33.581526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"summarize_dataframe(train_series).style.background_gradient(cmap='Purples')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:33.583423Z","iopub.execute_input":"2023-10-23T20:29:33.584237Z","iopub.status.idle":"2023-10-23T20:29:45.070639Z","shell.execute_reply.started":"2023-10-23T20:29:33.584214Z","shell.execute_reply":"2023-10-23T20:29:45.069435Z"}}},{"cell_type":"code","source":"summarize_dataframe(test_series).style.background_gradient(cmap='Purples')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:45.071988Z","iopub.execute_input":"2023-10-23T20:29:45.072332Z","iopub.status.idle":"2023-10-23T20:29:45.115897Z","shell.execute_reply.started":"2023-10-23T20:29:45.072305Z","shell.execute_reply":"2023-10-23T20:29:45.115012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nWe find that\n- the series data in the shrinked training set contains the information for *37* series\n- the series data in the testing set contains the information for *3* series to be used for prediction\n- we can note statistically significant differences in the spread of *angelz* and *enmo* in the training and testing series","metadata":{}},{"cell_type":"markdown","source":"##
Dailight Saving Time Impact on Traing Series
\n\nThis section will draw intelligence on how the training series are affected by DST change dates.\n\nSince the competion host (Child Mind Insitute) is located in New York, USA, its patient sleep records are provided on East time (East Cost, USA).\n\nThis area is affected by Dailight Saving Time (DST) changes twice a year. Its timezone is either -0400 (EST) or -0500 (EDT), depending on daylight saving.\n\n### USA Daylight Change Dates\n\n

On August 8, 2005, the Energy Policy Act of 2005 was signed into law, which extended Daylight Saving Time (DST) in the United States by approximately four weeks. As a result, beginning in 2007, Daylight Saving Time starts three weeks earlier on the second Sunday in March and ends one week later on the first Sunday of November, resulting in a new Daylight Saving Time period that is four weeks longer than previously observed.

\n\n

Daylight Saving Time dates are listed specifically by year, as per the table below.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
YearDST BeginsDST EndsYearDST BeginsDST Ends
2014March 9November 22024March 10November 3
2015March 8November 12025March 9November 2
2016March 13November 62026March 8November 1
2017March 12November 52027March 14November 7
2018March 11November 42028March 12November 5
2019March 10November 32029March 11November 4
2020March 8November 12030March 10November 3
2021March 14November 72031March 9November 2
2022March 13November 62032March 14November 7
2023March 12November 52033March 13November 6
\n

\n

\n\n
✅ Note:The information above is cited based on the respective Knowledge Base article.
","metadata":{}},{"cell_type":"code","source":"!pip install duckdb --quiet","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:45.116793Z","iopub.execute_input":"2023-10-23T20:29:45.117329Z","iopub.status.idle":"2023-10-23T20:29:57.301166Z","shell.execute_reply.started":"2023-10-23T20:29:45.117304Z","shell.execute_reply":"2023-10-23T20:29:57.300207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import duckdb\n\nparquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet'\ntesting_parquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet'\ntraining_events_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv'\n\ncon = duckdb.connect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:57.302910Z","iopub.execute_input":"2023-10-23T20:29:57.303312Z","iopub.status.idle":"2023-10-23T20:29:57.341604Z","shell.execute_reply.started":"2023-10-23T20:29:57.303281Z","shell.execute_reply":"2023-10-23T20:29:57.340855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n CREATE TABLE train_series AS\n SELECT * \n FROM read_parquet('{parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:29:57.343246Z","iopub.execute_input":"2023-10-23T20:29:57.343597Z","iopub.status.idle":"2023-10-23T20:30:14.808210Z","shell.execute_reply.started":"2023-10-23T20:29:57.343568Z","shell.execute_reply":"2023-10-23T20:30:14.807270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n CREATE TABLE test_series AS\n SELECT * \n FROM read_parquet('{testing_parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:14.809464Z","iopub.execute_input":"2023-10-23T20:30:14.809772Z","iopub.status.idle":"2023-10-23T20:30:14.821303Z","shell.execute_reply.started":"2023-10-23T20:30:14.809746Z","shell.execute_reply":"2023-10-23T20:30:14.820203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we intentionally load every col as VARCHAR; we need to load timestamp as a varchar, to prevent the UTC time casting at load\ncon.execute(f\"\"\"\n CREATE TABLE train_events AS\n SELECT * \n FROM read_csv('{training_events_table}', ALL_VARCHAR=1, AUTO_DETECT=TRUE);\n\"\"\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:14.822468Z","iopub.execute_input":"2023-10-23T20:30:14.823219Z","iopub.status.idle":"2023-10-23T20:30:14.939210Z","shell.execute_reply.started":"2023-10-23T20:30:14.823199Z","shell.execute_reply":"2023-10-23T20:30:14.937938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# inspired by \n# - https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\n# - https://www.kaggle.com/code/carlmcbrideellis/zzzs-eda-utc-or-not-utc\ndef get_series_data_with_events(series_id):\n \n train_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM train_series WHERE series_id = '{series_id}'\n \"\"\"\n df = con.execute(train_series_sql).fetchdf()\n \n #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n df = df.drop('timestamp', axis=1)\n \n train_events_sql = f\"\"\"\n SELECT \n series_id,\n CAST (night as INTEGER) as night,\n event,\n CAST (step as INTEGER) as step,\n timestamp\n FROM train_events \n WHERE series_id = '{series_id}'\n \"\"\"\n \n ev = con.execute(train_events_sql).fetchdf()\n #ev['date_time'] = pd.to_datetime(ev['timestamp'], utc=True)\n ev['date_time'] = pd.to_datetime(ev['timestamp']).apply(lambda t: t.tz_localize(None))\n ev = ev.drop('timestamp', axis=1)\n \n df2 = pd.merge(df, ev, on='date_time', how='outer')\n df2 = df2.drop(['series_id_x','series_id_y','step_x','step_y'], axis=1)\n df2['event'] = df2['event'].fillna(0)\n \n # fill values for night feature\n df2['night'][0] = 1\n df2['night'] = df2['night'].interpolate()\n df2['night'] = round(df2['night'])\n # map numeric values to event feature\n df2.loc[df2['event']=='onset','event'] = -1\n df2.loc[df2['event']=='wakeup','event'] = 1\n \n return df2\n\ndef get_train_series_data(series_id):\n train_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM train_series WHERE series_id = '{series_id}'\n \"\"\"\n train_df = con.execute(train_series_sql).fetchdf()\n return train_df\n\ndef get_test_series_data(series_id):\n test_series_sql = f\"\"\"\n SELECT\n series_id,\n CAST (step AS INTEGER) AS step,\n timestamp,\n CAST (anglez AS FLOAT) AS anglez,\n CAST (enmo AS FLOAT) AS enmo\n FROM test_series WHERE series_id = '{series_id}'\n \"\"\"\n df = con.execute(test_series_sql).fetchdf()\n \n return df\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:14.940909Z","iopub.execute_input":"2023-10-23T20:30:14.941305Z","iopub.status.idle":"2023-10-23T20:30:14.953317Z","shell.execute_reply.started":"2023-10-23T20:30:14.941271Z","shell.execute_reply":"2023-10-23T20:30:14.952654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Checking Training Series Boundaries\n\nSince the training series data spreads on 2017-2019 sparsly, we need to check which series overlap with the dailight saving events.","metadata":{}},{"cell_type":"code","source":"def get_train_series_date_boundaries():\n sql = f\"\"\"\n WITH start_end AS (\n SELECT\n series_id,\n MIN (timestamp) as series_start,\n MAX (timestamp) as series_end\n FROM train_series\n GROUP BY 1\n ),\n over_dst_start AS (\n SELECT DISTINCT\n series_id,\n '1' AS is_over_dst_start\n FROM train_series\n WHERE \n timestamp LIKE '2017-03-12%' OR\n timestamp LIKE '2018-03-11%' OR\n timestamp LIKE '2019-03-10%'\n ),\n over_dst_end AS (\n SELECT DISTINCT\n series_id,\n '1' AS is_over_dst_end\n FROM train_series\n WHERE \n timestamp LIKE '2017-11-05%' OR\n timestamp LIKE '2018-11-04%' OR\n timestamp LIKE '2019-11-03%'\n )\n SELECT\n start_end.series_id,\n start_end.series_start,\n start_end.series_end,\n over_dst_start.is_over_dst_start,\n over_dst_end.is_over_dst_end\n FROM start_end\n LEFT JOIN over_dst_start ON\n (over_dst_start.series_id = start_end.series_id)\n LEFT JOIN over_dst_end ON\n (over_dst_end.series_id = start_end.series_id);\n \n \"\"\"\n df = con.execute(sql).fetchdf()\n \n return df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:14.955507Z","iopub.execute_input":"2023-10-23T20:30:14.955856Z","iopub.status.idle":"2023-10-23T20:30:14.974625Z","shell.execute_reply.started":"2023-10-23T20:30:14.955830Z","shell.execute_reply":"2023-10-23T20:30:14.973454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's review the start and end timestamps of each of the training series as well as check if any of DST start/end dates for *2017-2019* falls into them.\n\nFirst of all, we are going to load train series data (start of their timestamps interval, end of their timestamps interval, and a set of binary *0/1* variables to indicate how each series is affected by DTS).","metadata":{}},{"cell_type":"code","source":"train_series_data = get_train_series_date_boundaries()\ntrain_series_data = train_series_data.fillna('0')\ntrain_series_data['is_over_dst_start'] = train_series_data['is_over_dst_start'].astype(int)\ntrain_series_data['is_over_dst_end'] = train_series_data['is_over_dst_end'].astype(int)\n\ndef sum_row(x):\n return x['is_over_dst_start'] + x['is_over_dst_end']\n\ntrain_series_data['affected_by_dst'] = train_series_data.apply(sum_row, axis=1)\ntrain_series_data.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:14.976461Z","iopub.execute_input":"2023-10-23T20:30:14.977104Z","iopub.status.idle":"2023-10-23T20:30:24.137861Z","shell.execute_reply.started":"2023-10-23T20:30:14.977075Z","shell.execute_reply":"2023-10-23T20:30:24.136345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### DST Change Effects on Training Series Data\n\nLet's check if any series goes through either the beginning and/or the end of DST period.","metadata":{}},{"cell_type":"code","source":"dst_df = train_series_data[ train_series_data['affected_by_dst'] > 1]\n\nmsg = f\"Number of training series going through both the DST start and DST end dates: {dst_df.shape[0]}\"\nPrintColor(f\"\\n---> {msg}\")\nif dst_df.shape[0] == 0:\n PrintColor(f\"\\n---> As we can see, none of the training series is long enough to go through both thee DST start and DST end dates of the same year.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:24.139170Z","iopub.execute_input":"2023-10-23T20:30:24.139503Z","iopub.status.idle":"2023-10-23T20:30:24.149132Z","shell.execute_reply.started":"2023-10-23T20:30:24.139480Z","shell.execute_reply":"2023-10-23T20:30:24.147673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's visualize the distribution of series that are\n\n- affeced by DTS change start\n- affected by DTS change end\n- not affected by DTS change","metadata":{}},{"cell_type":"code","source":"summmary_stat = {\n 'series_with': [\"DST start date\", \"DST end date\", \"None of DST dates\"],\n 'series_count': [\n train_series_data['is_over_dst_start'].sum(),\n train_series_data['is_over_dst_end'].sum(),\n train_series_data.shape[0] - train_series_data['is_over_dst_start'].sum() - train_series_data['is_over_dst_end'].sum()\n ]\n}\nsummary_df = pd.DataFrame(summmary_stat)\nfig = px.bar(summary_df, x='series_with', y='series_count', \n title='Number of series overlapping with any of DST dates',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:30:24.150590Z","iopub.execute_input":"2023-10-23T20:30:24.150909Z","iopub.status.idle":"2023-10-23T20:30:24.215509Z","shell.execute_reply.started":"2023-10-23T20:30:24.150883Z","shell.execute_reply":"2023-10-23T20:30:24.214425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that\n\n- the majority of the series in the training dataset (`234`) are not affected by DST change\n- `22` series overlap with DST change start dates\n- `21` series overlap with DST change end dates\n- none of the training series lasts through both DST change start and end dates","metadata":{}},{"cell_type":"markdown","source":"
🚩 Note: There is a DST change date-related glitch in the raw training series data provided by the contest host. Its assessment, along with the impact on the data loading strategies and workarounds, is extensively covered in a separate notebook of mine.
","metadata":{}},{"cell_type":"markdown","source":"##
Series Start and End: Minute patterns
\n\nLet's look at the series start minutes.","metadata":{}},{"cell_type":"code","source":"def get_train_series_date_intervals():\n sql = f\"\"\"\n SELECT\n series_id,\n MIN (timestamp) as series_start,\n MAX (timestamp) as series_end\n FROM train_series\n GROUP BY 1\n \"\"\"\n df = con.execute(sql).fetchdf()\n \n return df\n\ndf_train_series = get_train_series_date_intervals()\n\ndf_train_series['series_start'] = pd.to_datetime(df_train_series['series_start']).apply(lambda t: t.tz_localize(None))\ndf_train_series['series_end'] = pd.to_datetime(df_train_series['series_end']).apply(lambda t: t.tz_localize(None))\ndf_train_series['start_minute'] = df_train_series[\"series_start\"].dt.minute\ndf_train_series['end_minute'] = df_train_series[\"series_end\"].dt.minute","metadata":{"execution":{"iopub.status.busy":"2023-10-23T20:30:24.216731Z","iopub.execute_input":"2023-10-23T20:30:24.217023Z","iopub.status.idle":"2023-10-23T20:30:28.408131Z","shell.execute_reply.started":"2023-10-23T20:30:24.216998Z","shell.execute_reply":"2023-10-23T20:30:28.407500Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfg = df_train_series['start_minute'].value_counts().reset_index()\ndfg.columns = ['start_minute', 'number_of_series']\nfig = px.bar(dfg, x='start_minute', y='number_of_series', \n title='Training Series: Start minutes',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:33:17.995488Z","iopub.execute_input":"2023-10-23T20:33:17.995792Z","iopub.status.idle":"2023-10-23T20:33:18.055672Z","shell.execute_reply.started":"2023-10-23T20:33:17.995771Z","shell.execute_reply":"2023-10-23T20:33:18.054424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's look at the series end minutes.","metadata":{}},{"cell_type":"code","source":"dfg = df_train_series['end_minute'].value_counts().reset_index()\ndfg.columns = ['end_minute', 'number_of_series']\nfig = px.bar(dfg, x='end_minute', y='number_of_series', \n title='Training Series: End minutes',\n color_discrete_sequence=px.colors.qualitative.Prism) \nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-23T20:35:55.536035Z","iopub.execute_input":"2023-10-23T20:35:55.536399Z","iopub.status.idle":"2023-10-23T20:35:55.593102Z","shell.execute_reply.started":"2023-10-23T20:35:55.536354Z","shell.execute_reply":"2023-10-23T20:35:55.592114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we review the diagrams above, we can see the clear patterns as for\n- the training series to start exactly in one of the four minute slots (that is, either in 0th, 15th, 30th, or 45th minute of an hour)\n- the training series to end exactly in ne of the four minute slots (that is, either in 14th, 29th, 44th, or 59th minute of an hour)\n\nSuch a 15-min step in the series start/finish is probably the artifact to the measuring system at the competition host end as well as the artifact to the method of shrinking the series data for the purpose of formating the training set of this competition.\n\n
✅ Note: Such a pattern extends the set of phenomena related to the methods of the data collection/preparation on the contest host side. Earlier, we have been discussing the event minute pattern.
","metadata":{}},{"cell_type":"markdown","source":"##
Data Analytics: enmo (training series)
","metadata":{}},{"cell_type":"code","source":"def get_all_series_boxplot_data(table, col_name, where_clause = None):\n min_max_sql = \"\"\n quantile_sql = \"\"\n if where_clause:\n min_max_sql = f\"\"\"\n SELECT series_id, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n GROUP BY series_id\n WHERE {where_clause}\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n series_id,\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n GROUP BY series_id\n WHERE {where_clause}\n )\n SELECT\n series_id,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n else:\n min_max_sql = f\"\"\"\n SELECT series_id, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n GROUP BY series_id\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n series_id,\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n GROUP BY series_id\n )\n SELECT\n series_id,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n df_mean_max = con.execute(min_max_sql).fetchdf()\n df_quantile = con.execute(quantile_sql).fetchdf()\n \n df = pd.merge(\n df_mean_max,\n df_quantile,\n how=\"inner\",\n on='series_id')\n \n #rearrange the order of cols to make it a canonical boxplot style\n df = df[['series_id', 'min_value', 'q1', 'median', 'q3', 'max_value', 'mean_value', 'stddev']]\n \n return df\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_boxplot_data(table, col_name, where_clause = None):\n min_max_sql = \"\"\n quantile_sql = \"\"\n if where_clause:\n min_max_sql = f\"\"\"\n SELECT '{col_name}' AS attribute, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n WHERE {where_clause}\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n WHERE {where_clause}\n )\n SELECT\n '{col_name}' AS attribute,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n else:\n min_max_sql = f\"\"\"\n SELECT '{col_name}' AS attribute, \n MIN({col_name}) AS min_value, \n AVG({col_name}) AS mean_value, \n MAX({col_name}) AS max_value,\n stddev_pop({col_name}) AS stddev\n FROM {table}\n \"\"\"\n quantile_sql = f\"\"\"\n WITH stats as (\n SELECT\n percentile_disc([0.25, 0.50, 0.75]) WITHIN GROUP \n (ORDER BY {col_name}) AS percentiles\n FROM {table}\n )\n SELECT\n '{col_name}' AS attribute,\n percentiles[1] as q1,\n percentiles[2] as median,\n percentiles[3] as q3\n FROM stats;\n \"\"\"\n df_mean_max = con.execute(min_max_sql).fetchdf()\n df_quantile = con.execute(quantile_sql).fetchdf()\n \n df = pd.merge(\n df_mean_max,\n df_quantile,\n how=\"inner\",\n on='attribute')\n \n #rearrange the order of cols to make it a canonical boxplot style\n df = df[['attribute', 'min_value', 'q1', 'median', 'q3', 'max_value', 'mean_value', 'stddev']]\n \n return df\n\ndef build_boxplot(table, col_name, where_clause = None):\n df = get_boxplot_data(table, col_name, where_clause)\n fig_title = \"\"\n \n bar_name = \"\".join([table, \".\", col_name])\n \n if where_clause:\n fig_title = \"\".join([table, \".\", col_name, \": \", where_clause])\n else:\n fig_title = \"\".join([table, \".\", col_name, \": Entire table population\"])\n \n fig = go.Figure()\n\n fig.add_trace(go.Box(\n y= [ df['min_value'].iloc[0], df['max_value'].iloc[0] ],\n name=bar_name, boxpoints=False,)\n )\n\n fig.update_traces(q1=[ df['q1'].iloc[0] ], median=[ df['median'].iloc[0] ],\n q3=[ df['q3'].iloc[0] ], lowerfence=[df['min_value'].iloc[0]],\n upperfence=[df['max_value'].iloc[0]], mean=[ df['mean_value'].iloc[0] ],\n sd=[ df['stddev'].iloc[0]]\n )\n # Update visual layout\n fig.update_layout(\n showlegend=False,\n width=600,\n height=400,\n autosize=False,\n margin=dict(t=15, b=0, l=5, r=5),\n title=dict(text=fig_title, font=dict(size=16), automargin=True, yref='paper'),\n title_x=0.5, # center the title\n template=\"plotly_white\",\n colorway=px.colors.qualitative.Prism ,\n )\n \n # update font size at the axes\n fig.update_coloraxes(colorbar_tickfont_size=10)\n # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n fig.update_annotations(font_size=12)\n # Reduce opacity\n fig.update_traces(opacity=0.75)\n \n return fig\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://duckdb.org/docs/sql/aggregates\ndef get_histogram_data(table, col_name, where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n SELECT histogram({col_name}) FROM {table}\n WHERE {where_clause};\n \"\"\"\n else:\n query = f\"\"\"\n SELECT histogram({col_name}) FROM {table};\n \"\"\"\n df = con.execute(query).fetchdf()\n \n hist_col_name = f\"histogram({col_name})\"\n\n histogram_dict = df[hist_col_name].iloc[0] # dictionary: dict_keys(['key', 'value'])\n\n frame = {\n col_name: histogram_dict.get('key'),\n 'record_count': histogram_dict.get('value')}\n \n # Creating DataFrame by passing Dictionary\n agg_data = pd.DataFrame(frame)\n return agg_data\n \ndef build_histogram(table, col_name, where_clause = None):\n figure_title = \"\"\n if where_clause:\n figure_title = f\"Distribution of {col_name} in {table}; filter: {where_clause}\"\n else:\n figure_title = f\"Distribution of {col_name} in {table}\"\n \n agg_data = get_histogram_data(table, col_name, where_clause)\n \n fig = px.histogram(agg_data, x=col_name, marginal=\"rug\",\n title=figure_title,\n color_discrete_sequence=px.colors.qualitative.Prism)\n return fig\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_histogram_data_2(table, col_name, where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n SELECT FLOOR({col_name}/1.00)*1 AS {col_name}, \n COUNT(*) AS record_count\n FROM {table}\n WHERE {where_clause}\n GROUP BY FLOOR({col_name}/1.00)*1\n ORDER BY 1\n \"\"\"\n else:\n query = f\"\"\"\n SELECT FLOOR({col_name}/1.00)*1 AS {col_name}, \n COUNT(*) AS record_count\n FROM {table}\n GROUP BY FLOOR({col_name}/1.00)*1\n ORDER BY 1\n \"\"\"\n df_agg = con.execute(query).fetchdf()\n return df_agg\n\ndef build_histogram_2(table, col_name, where_clause = None):\n fig_title = \"\"\n if where_clause:\n fig_title = f\"Distribution of {col_name}: {table}; filtered: {where_clause}\"\n else:\n fig_title = f\"Distribution of {col_name}: {table}\"\n df_agg = get_histogram_data_2(table, col_name, where_clause)\n fig = px.bar(df_agg, x=col_name, y=\"record_count\",\n title=\"Distribution of anglez: training series\",\n color_discrete_sequence=px.colors.qualitative.Prism)\n return fig\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_hourly_averages(table, col_name, where_clause = None):\n query = \"\"\n if where_clause:\n query = f\"\"\"\n WITH hours_of_day AS (\n SELECT extract('hour' FROM \n strptime(substring(replace(timestamp, 'T', ' '), 1, length(timestamp)-5), '%Y-%m-%d %H:%M:%S')) AS hour_of_day,\n {col_name}\n FROM {table}\n WHERE {where_clause}\n )\n SELECT\n hour_of_day, \n AVG({col_name}) AS avg_{col_name}\n FROM hours_of_day\n GROUP BY 1\n ORDER BY 1\n \"\"\"\n else:\n query = f\"\"\"\n WITH hours_of_day AS (\n SELECT extract('hour' FROM \n strptime(substring(replace(timestamp, 'T', ' '), 1, length(timestamp)-5), '%Y-%m-%d %H:%M:%S')) AS hour_of_day,\n {col_name}\n FROM {table}\n )\n SELECT\n hour_of_day, \n AVG({col_name}) AS avg_{col_name}\n FROM hours_of_day\n GROUP BY 1\n ORDER BY 1\n \"\"\"\n df_agg = con.execute(query).fetchdf()\n return df_agg\n\n\ndef build_hourly_averages_bar(table, col_name, where_clause = None):\n y_variable = f\"avg_{col_name}\"\n fig_title = \"\"\n if where_clause:\n fig_title = f\"Average {col_name} by hours in a day (local time); filtered: {where_clause}\"\n else:\n fig_title = f\"Average {col_name} by hours in a day (local time)\"\n df_agg = get_hourly_averages(table, col_name, where_clause)\n fig = px.bar(df_agg, x='hour_of_day', y=y_variable,\n title=fig_title,\n color_discrete_sequence=px.colors.qualitative.Prism)\n return fig\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the summary statistics on *enmo* values in the training series","metadata":{}},{"cell_type":"code","source":"fig = build_boxplot('train_series', 'enmo', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that training-set enmo values are described as follows\n- enmo max is 11.433699607849121\n- enmo min is 0.0\n- enmo mean is 0.04131503403186798\n- enmo standard deviation is 0.101828932762146\n- enmo first quantile is 0.0013000000035390258\n- enmo median is 0.01720000058412552\n- enmo third quantile is 0.043699998408555984\n- enmo 99 percentile is 0.39430001378059387","metadata":{}},{"cell_type":"markdown","source":"Let's now look at the distribution of the *enmo* values in the training series","metadata":{}},{"cell_type":"code","source":"fig = build_histogram(table='train_series', col_name='enmo')\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is manifested the majority of *enmo* values to be between 0 and 1. \n\nWe can also see there are definite outliers in the values of *enmo* (while 99% of the data are under *0.39430001378059387*). \n\nLet's also look at the hourly averages of *enmo* acorss the training series data.","metadata":{}},{"cell_type":"code","source":"fig = build_hourly_averages_bar('train_series', 'enmo')\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The key takeaways from the *enmo* data analaysis in the training series are listed below\n\n- According to emno value ranges described in some of the scientific papers, we can suppose that mean emon values should be about 0.0093 gravity unites for sedentary behaviours; and around 0.1075 gravity unites for active periods. These data is consistent with train_series data where enmo mean = 0.041315 and 99% of enmo values are up to 0.39430001378059387.\n- Extreme enmo values can be explained by extreme activities (e.g. pounding on a table) and they don't last for very long time.\n- enmo outliers should be imputed with the certain upper-clipping value in ML experiments (with either 99th percentile value or 1.0, or 1.5 proposed by different contributors as an adequate upper cap value).\n- enmo values are stationary.\n\n
✅ Note: You can delve in the notebooks referred below for more detailed analysis of enmo patterns in the training data\n\n\n
","metadata":{}},{"cell_type":"markdown","source":"##
Data Analytics: enmo (testing series)
\n\nLet's look at the summary statistisc on *enmo* on the testing series data.","metadata":{}},{"cell_type":"code","source":"df_test_enmo_summary = get_boxplot_data(\"test_series\", 'enmo')\n\ndf_test_enmo_summary.style.set_caption(\"Summary statistics on enmo: test series\"). \\\nset_properties(**{'border': '1.3px solid blue',\n 'color': 'grey'})","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_boxplot('test_series', 'enmo', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's look at the distribution of *enmo* on the testing series data.","metadata":{}},{"cell_type":"code","source":"fig = build_histogram(table='test_series', col_name='enmo')\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that\n\n- the distribution of *enmo* on training and testing series is quite different (although both distributions are severely skewed to the left)\n- the most of the *enmo* values in the training series belong to the interval of 0.025..2.175 (within this interval, evey bucket with 0.05 incremental step contains either 500 or 501 observations)\n- the most of the *enmo* values in the testing series belong to the interval of -0.025..0.0749","metadata":{}},{"cell_type":"markdown","source":"##
Data Analytics: anglez (training series)
\n\nLet's look at the summary statistics on *anglez* values in the training series","metadata":{}},{"cell_type":"code","source":"df_train_anglez_summary = get_boxplot_data(\"train_series\", 'anglez')\n\ndf_test_enmo_summary.style.set_caption(\"Summary statistics on anglez: train series\"). \\\nset_properties(**{'border': '1.3px solid blue',\n 'color': 'grey'})","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_boxplot('train_series', 'anglez', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's look at the histogram of *anglez* to review its distribution on the train series data.","metadata":{}},{"cell_type":"code","source":"fig = build_histogram_2('train_series', 'anglez')\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's review the distribution of hourly averages of *anglez* acorss the hours in a day.","metadata":{}},{"cell_type":"code","source":"fig = build_hourly_averages_bar('train_series', 'anglez')\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The key takeaways from the *anglez* data analaysis in the training series are listed below\n\n- *'anglez'* metric refers to the angle of the arm relative to the vertical axis of the body.\n- *anglez* values have normal distribution with max = 90.0, min = -90.0, mean = -8.81 and standard deviation = 35.52.\n- *anglez* values changes with following pattern: periods with more frequent changes in values are switched to periods with less frequent changes in values.\n- periods with less frequent changes in *anglez* values correspond to lower *enmo* values.\n- *anglez* values are stationary.\n- during sleeping periods, *anglez* changes resemble a pure jump process.\n- during wakeful periods, *anglez* changes resemble resemble a diffusion process,\n\n\n
✅ Note: You can delve in the notebooks referred below for more detailed analysis of anglez patterns in the training data\n\n\n
","metadata":{}},{"cell_type":"markdown","source":"##
Data Analytics: anglez (testing series)
\n\nLet's look at the summary statistisc on *anglez* on the testing series data.","metadata":{}},{"cell_type":"code","source":"df_test_anglez_summary = get_boxplot_data(\"test_series\", 'anglez')\n\ndf_test_anglez_summary.style.set_caption(\"Summary statistics on anglez: test series\"). \\\nset_properties(**{'border': '1.3px solid blue',\n 'color': 'grey'})","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = build_boxplot('test_series', 'anglez', where_clause = None)\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's look at the distribution of *anglez* on the testing series data.","metadata":{}},{"cell_type":"code","source":"fig = build_histogram(table='test_series', col_name='anglez')\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Additional Insights from Series Data
\n\nThe table below summarizes the key takeaways for the BigData-scale EDA drills on training and testing seria:\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
 Description
❗There is a lack of the systematic patterns in the population-level and series-level distribution and aggregated statistics for enmo and angllez in both training and test seria sets
✅Distribution and summary statistics of enmo and angllez are significantly different in training seria vs. test seria
🚩Test series data contains the enmo and angllez feeds for just three seria, 1 h long each; such a structure of the test set may impose some complications/road blocks to the ML solutions attempting to tackle the contest problem in the tabular classification setup
📝The average values of enmo are typically lower for the hours where onset events are more frequent (in the training set)
📝The average values of enmo are typically lower for the hours where wakeup events are more frequent (in the training set)
📝The average values of anglez are typically higher for the hours where wakeup events are more frequent (although we can see some hours without major number of wakeup events to have the similar average values of anglez)
📝The average values of anglez are typically typically lower for the hours where onset events are more frequent
\n\n\nRegarding the intervals of sleeping and being awake, we also noticed the facts below\n- the average values of *anglez* are even more high in the time intervals where a person is awake (it can be even higher then the average values for the hours with the bigger number of *wakeup* events registered)\n- higher values of *enmo* are observed for the awake state of the observable humans\n\n
💥 Highlights: The above-mentioned finding about the hourly patterns in matching onset/wakeup events to the average hourly values of enmo and angllez, combined with the earlier detected findings on the lack of the systematic patterns in the population-level and series-level distribution and aggregated statistics for enmo and angllez, may highlight an interesting forecasting strategy.

\n \nThe ML solution can focus on scanning the time series breakpoints where simultaneoursly observed are\n\n
\n\n
✅ Note: In order to perform the analytics operations to get the insights on training and testing series data (BigData-scale data, indeed), we leveraged DuckDB, the fast, lightweight, and easy-to-use analytics database with a good API/wrappers for Python. You can find the detailed flow of the respective analysis in a separate notebook per Plotting Big Data: Sleeping Series.\n
\n","metadata":{}},{"cell_type":"markdown","source":"#
Additional EDA Insights
\n\nThis notebook is a part of my EDA contributions to this contest. You can find more EDA-centric insights in other notebooks of mine, as per the links below\n\n- [Plotting Big Data: Sleeping Series](https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series) - it drills down into the individual series data insights\n- [TS Windows Around Events](https://www.kaggle.com/code/gvyshnya/ts-windows-around-events) - it focuses on the investigations of the time-series windows around the labelled events in training series, using OCHL resampling methodology\n- [Sleeping Series Integrity Check](https://www.kaggle.com/code/gvyshnya/sleeping-series-integrity-check) - it provides a comprehensive check of the quality/integrity of the data provided in the training series as well as assesses the impact of the Dailight Saving Time (DTS) changes (it happens in the US/Eastern time zone twice a year) on the timestamps in the training series","metadata":{}},{"cell_type":"markdown","source":"#
References
\n\n- Polars (2023): User Guide https://pola-rs.github.io/polars-book/user-guide/ (accessed Sep 28, 2023)\n- “Stackoverflow”, “What are the differences between feather and parquet?”. stackoverflow.com. https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet (accessed Sep 28, 2023)\n- Leonie Monegatti, \"Pandas vs. Polars: A Syntax and Speed Comparison\" https://towardsdatascience.com/pandas-vs-polars-a-syntax-and-speed-comparison-5aa54e27497e#454c (accessed Sep 28, 2023)\n- Priyanshu Chaudhary, \"Pandas 2.0 vs Polars: The Ultimate Battle\" https://medium.com/cuenex/pandas-2-0-vs-polars-the-ultimate-battle-a378eb75d6d1 (accessed Sep 28, 2023)\n- Boriharn K, \"7 Visualizations with Python to Handle Multivariate Categorical Data\" https://towardsdatascience.com/7-visualizations-with-python-to-handle-multivariate-categorical-data-63158db0911d (accessed Sep 28, 2023)\n- \"CRAN R-Project\", \"Accelerometer data processing with GGIR\" https://cran.r-project.org/web/packages/GGIR/vignettes/GGIR.html#4_Inspecting_the_results (accessed Sep 28, 2023)\n- Kalaivani Sundararajan, Sonja Georgievska, Bart H. W. te Lindert, Philip R. Gehrman, Jennifer Ramautar, Diego R. Mazzotti, Séverine Sabia, Michael N. Weedon, Eus J. W. van Someren, Lars Ridder, Jian Wang & Vincent T. van Hees, \"Sleep classification from wrist‑worn accelerometer data using random forests\", \"Nature\" - https://www.nature.com/articles/s41598-020-79217-x.pdf (accessed Sep 28, 2023)\n- van Hees, V.T., Sabia, S., Jones, S.E. et al. Estimating sleep parameters using an accelerometer without sleep diary. Sci Rep 8, 12975 (2018). https://doi.org/10.1038/s41598-018-31266-z (accessed Sep 28, 2023)","metadata":{}},{"cell_type":"markdown","source":"#
Kudos
\n\nThis notebook is inspired by some other peer Kaggler contributions as per the list below\n- Carl Ellis [@carlmcbrideellis](https://www.kaggle.com/carlmcbrideellis) and his contributions as for *(1)* detecting incompletely labelled training series as well as building the [lightweight training dataset + binary target (awake)](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target) for the contest, and *(2)* highlighting UTC vs. local time zone issues in the series and events data (see it in the [respective notebook](https://www.kaggle.com/code/carlmcbrideellis/zzzs-eda-utc-or-not-utc))\n- Enrico Manosperti [@enricomanosperti](https://www.kaggle.com/enricomanosperti) and his swift time-effective data loading strategy with Polars (as conveyed by [his notebook](https://www.kaggle.com/code/enricomanosperti/detect-sleep-states-first-preprocessing-and-eda))\n- Renato Regiani [@renatoreggiani](https://www.kaggle.com/renatoreggiani) and his implementation of Pandas DF memory reduction (as per the [notebook](https://www.kaggle.com/code/renatoreggiani/reduce-memory-usage-zzzs-cmi)) \n- Rishabh Jain [@rishabh15virgo](https://www.kaggle.com/rishabh15virgo) and his ideas on the express automated visualized summary of the data in Pandas dataframes (as presented by [his notebook](https://www.kaggle.com/code/rishabh15virgo/cmi-dss-first-impression-data-understanding-eda))\n- Chun Fu [@patrick0302](https://www.kaggle.com/patrick0302) and his idea to visualize time-based density of the events across all of the training series (as per the respective [notebook](https://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series))\n- Ali Nasri Nazif [@alinasri](https://www.kaggle.com/alinasri) and his [idea](https://www.kaggle.com/code/alinasri/sleep-data-analysis-some-eda) to calculate the summary stats for sleeping duration\n- Ravi Ramakrishnan [@ravi20076](https://www.kaggle.com/ravi20076) and his swift use of *colorama* and *pprint* to decorate and nicely format the console print output (as demonstrated in his [notebook](https://www.kaggle.com/code/ravi20076/sleepstate-eda-baseline))\n- Marília Prata [@mpwolke](https://www.kaggle.com/mpwolke) and her profound and medically enriched knowledge sharing about z-angle (*anglez* in our datasets) as well as using its variance in accelerometer observations to predict sleeping interrupts (see the [respective notebook](https://www.kaggle.com/code/mpwolke/the-zzzs-anglez-aka-z-angle) for more info)\n- Daniel Dewey [@dan3dewey](https://www.kaggle.com/dan3dewey) and his excellent statistically and mathematically justified insights on various contest-related matters (comments on [unexpected event minute patterns](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/444374#2475187); suggestions on the optimal strategy to [impute outliers for enmo feature](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/445166#2468724); etc.)\n- Jun Koda [@junkoda](https://www.kaggle.com/junkoda) who paid attention to local TZ inconsistency due to DTS change dates recorded in the data incorrectly, as per [Checking Daylight Saving Times](https://www.kaggle.com/code/junkoda/checking-daylight-saving-times)\n- Veronika Sarikaya [@veronikasarikaya](https://www.kaggle.com/veronikasarikaya) who contributed to the body of knowledge for this contest with the bunch of nice insights (as per [It's a wonderful, wonderful sleep (EDA)](https://www.kaggle.com/code/veronikasarikaya/it-s-a-wonderful-wonderful-sleep-eda))\n- Lucas Burke [@lccburk](https://www.kaggle.com/lccburk) and his brilliant insights on *anglez* changes to resemble a pure jump process during sleeping periods as well as to resemble a diffusion process during wakeful periods (as manifested by the [respective notebook](https://www.kaggle.com/code/lccburk/sleep-data-exploration))\n- Sergii Kryvenko [@krivenkozz](https://www.kaggle.com/krivenkozz) and his suggestions to generalize the code of *reduce_mem_usage* function","metadata":{}}]}