{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## This notebook is to visualize the time series and get some insights from vizs 😎","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.graph_objs as go\nimport plotly\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport cufflinks as cf\ncf.set_config_file(offline=True)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-07T08:58:28.527507Z","iopub.execute_input":"2023-09-07T08:58:28.527851Z","iopub.status.idle":"2023-09-07T08:58:32.987772Z","shell.execute_reply.started":"2023-09-07T08:58:28.527822Z","shell.execute_reply":"2023-09-07T08:58:32.986690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')\ntest_series = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet')\ntrain_series","metadata":{"execution":{"iopub.status.busy":"2023-09-07T08:58:32.989598Z","iopub.execute_input":"2023-09-07T08:58:32.990110Z","iopub.status.idle":"2023-09-07T08:59:29.422812Z","shell.execute_reply.started":"2023-09-07T08:58:32.990077Z","shell.execute_reply":"2023-09-07T08:59:29.421574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-09-07T08:59:29.425629Z","iopub.execute_input":"2023-09-07T08:59:29.426023Z","iopub.status.idle":"2023-09-07T08:59:29.441654Z","shell.execute_reply.started":"2023-09-07T08:59:29.425991Z","shell.execute_reply":"2023-09-07T08:59:29.440342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\n\ntrain_events['date'] = train_events['timestamp'].str.split('T', expand=True)[0]\ntrain_events['time'] = train_events['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntrain_events['timestamp'] = pd.to_datetime(train_events['date']+' '+train_events['time'])\ntrain_events['hour'] = train_events['timestamp'].dt.hour\ntrain_events","metadata":{"execution":{"iopub.status.busy":"2023-09-07T08:59:29.443651Z","iopub.execute_input":"2023-09-07T08:59:29.444501Z","iopub.status.idle":"2023-09-07T08:59:29.661563Z","shell.execute_reply.started":"2023-09-07T08:59:29.444464Z","shell.execute_reply":"2023-09-07T08:59:29.659783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let's visualize the availability of the train_series (1: available; 0: NaN)\n\nThis visualization aids in discerning the periods during which events ('onset' and 'wakeup') have been recorded. \n\nA clustermap is employed for this visualization, where the y-axis represents individual time series clustered based on their data collection similarity. \n\nUpon examining the clustermap, it is evident that the majority of the series tend to have a duration spanning 20 to 30 days. \n\nInterestingly, the periods during which data have been collected are sparsely distributed across the years 2017 to 2019.","metadata":{}},{"cell_type":"code","source":"# Pivot the DataFrame to create df_availability\ndf_availability = train_events.pivot_table(columns='date', index='series_id', values='step', aggfunc='count')\n\n# Fill NA values with 0\ndf_availability = (~df_availability.isna()).fillna(0).astype('int')\n\n# Plotting the clustered heatmap\nsns.clustermap(df_availability, col_cluster=False, figsize=(12, 8))","metadata":{"execution":{"iopub.status.busy":"2023-09-07T08:59:29.663158Z","iopub.execute_input":"2023-09-07T08:59:29.663585Z","iopub.status.idle":"2023-09-07T08:59:31.033644Z","shell.execute_reply.started":"2023-09-07T08:59:29.663547Z","shell.execute_reply":"2023-09-07T08:59:31.032570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let's visualize the frequency of `onset` and `wakeup` events along the hour axis in a histogram plot.\n\nIn the competition, `Onset` and `Wakeup` events signify the times people go to sleep and wake up, respectively. \n\nUnderstanding the temporal distribution of these events can offer critical insights into sleep patterns and behavior. \n\nTo that end, we can employ histograms to visualize the frequency of `Onset` and `Wakeup` events along the hour axis. \n\nIn the resulting histogram, the x-axis denotes the 24-hour timestamp, while the y-axis represents the frequency of `Onset` and `Wakeup` events. \n\nFrom the data, it is evident that the majority of individuals tend to wake up between 6-8 am and go to sleep around 8-10 pm. \n\nThis temporal pattern aligns well with general expectations and offers a useful baseline for further analyses.","metadata":{}},{"cell_type":"code","source":"# Set figure size\nplt.figure(figsize=(10, 5))\n\n# Assuming your DataFrame is called train_events\ndf_onset = train_events[train_events['event'] == 'onset']\ndf_wakeup = train_events[train_events['event'] == 'wakeup']\n\n# Plotting the distribution plot for onset events\nsns.histplot(df_onset['hour'].dropna(), kde=False, bins=24, label='Onset')  # You can adjust bins as needed\n\n# Plotting the distribution plot for wakeup events\nsns.histplot(df_wakeup['hour'].dropna(), kde=False, bins=24, label='Wakeup')  # You can adjust bins as needed\n\n# Adding labels and title\nplt.xlabel('Hour of Event')\nplt.ylabel('Frequency')\nplt.title('Distribution of Hour of Onset and Wakeup')\nplt.legend()\n\n# Show the plot\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-07T08:59:31.034817Z","iopub.execute_input":"2023-09-07T08:59:31.035342Z","iopub.status.idle":"2023-09-07T08:59:31.402880Z","shell.execute_reply.started":"2023-09-07T08:59:31.035299Z","shell.execute_reply":"2023-09-07T08:59:31.401720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let's visualize the time series of `enmo` and `anglez` with the flag of `onset` and `wakeup` for 10 samples in **train_series**\n\nTo advance our understanding of sleep behavior, it is essential to delve deeper into the time series of `enmo` and `anglez`, particularly in relation to the `onset` (Go-to-sleep) and `wakeup` (Wake-up) events. \n\nWe employ time-series plots to flag the events of 'Onset' and 'Wakeup' within the `enmo` and `anglez` data streams for series in `train_series`.\n\nUpon examining the time-series plots, several observations can be made:\n\n1. `enmo` levels decrease during sleep periods—defined as the time between `Onset` and `Wakeup` events.\n2. For `anglez`, the intensity appears to differ between sleep and wake times, but not very significant.\n3. Interestingly, not all profiles are consistently labeled, and some appear to be repetitive over multiple days. ","metadata":{}},{"cell_type":"code","source":"def plot_series_with_events(series_id, train_series, train_events):\n    # Filter the DataFrame based on the series_id\n    sample_serie = train_series[train_series['series_id'] == series_id]\n    \n    # Convert timestamp to Pandas datetime object\n    sample_serie['date'] = sample_serie['timestamp'].str.split('T', expand=True)[0]\n    sample_serie['time'] = sample_serie['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\n    sample_serie['timestamp'] = pd.to_datetime(sample_serie['date'] + ' ' + sample_serie['time'])\n    \n    # Filter event data based on the series_id\n    sample_events = train_events[train_events['series_id'] == series_id]\n    sample_onset = sample_events.loc[sample_events['event'] == 'onset', 'timestamp'].dropna()\n    sample_wakeup = sample_events.loc[sample_events['event'] == 'wakeup', 'timestamp'].dropna()\n    \n    # Helper function to plot data and events\n    def plot_data_and_events(data, ylabel):\n        plt.figure(figsize=(20, 3))\n        plt.plot(sample_serie['timestamp'], sample_serie[data], label=data, linewidth=1)\n        \n        for onset in sample_onset:\n            plt.axvline(x=onset, color='r', linestyle='--', label='onset')\n            \n        for wakeup in sample_wakeup:\n            plt.axvline(x=wakeup, color='g', linestyle='--', label='wakeup')\n            \n        handles, labels = plt.gca().get_legend_handles_labels()\n        new_labels, new_handles = [], []\n        for handle, label in zip(handles, labels):\n            if label not in new_labels:\n                new_handles.append(handle)\n                new_labels.append(label)\n        \n        plt.legend(new_handles, new_labels)\n        plt.xlabel('Timestamp')\n        plt.ylabel(ylabel)\n        plt.title(f'{ylabel} over Time with Event Flags - '+series_id)\n        plt.show()\n    \n    # Plot enmo and anglez\n    plot_data_and_events('enmo', 'ENMO Value')\n    plot_data_and_events('anglez', 'anglez Value')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-07T08:59:31.404432Z","iopub.execute_input":"2023-09-07T08:59:31.404814Z","iopub.status.idle":"2023-09-07T08:59:31.417118Z","shell.execute_reply.started":"2023-09-07T08:59:31.404780Z","shell.execute_reply":"2023-09-07T08:59:31.415306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for series_id in train_events['series_id'].unique()[:10]:\n    plot_series_with_events(series_id, train_series, train_events)","metadata":{"execution":{"iopub.status.busy":"2023-09-07T08:59:31.418829Z","iopub.execute_input":"2023-09-07T08:59:31.419147Z","iopub.status.idle":"2023-09-07T09:01:59.623020Z","shell.execute_reply.started":"2023-09-07T08:59:31.419121Z","shell.execute_reply":"2023-09-07T09:01:59.622107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let's visualize the time series of `enmo` and `anglez` with the flag of `onset` and `wakeup` for samples in **test_series** with flags in **sample_submission**","metadata":{}},{"cell_type":"code","source":"test_events = sample_submission.merge(test_series, on=['series_id','step'],how='left')\n\ntest_events['date'] = test_events['timestamp'].str.split('T', expand=True)[0]\ntest_events['time'] = test_events['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntest_events['timestamp'] = pd.to_datetime(test_events['date']+' '+test_events['time'])\ntest_events['hour'] = test_events['timestamp'].dt.hour\n\nfor series_id in test_events['series_id'].unique():\n    plot_series_with_events(series_id, test_series, test_events)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-07T09:01:59.624231Z","iopub.execute_input":"2023-09-07T09:01:59.625209Z","iopub.status.idle":"2023-09-07T09:02:01.122119Z","shell.execute_reply.started":"2023-09-07T09:01:59.625160Z","shell.execute_reply":"2023-09-07T09:02:01.121226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}