{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import HTML, Markdown\nimport time\n\nhandle = display(HTML(\"\"\"<marquee>👌</marquee>\"\"\"), display_id='html_marquee1')\ntime.sleep(2)\nhandle = display(HTML(\"\"\"<marquee>~ \"I have seen the future and it's like the present, only longer.\" – Dan Quisenberry</marquee>\"\"\"), display_id='html_marquee1', update=True)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:41:42.980958Z","iopub.execute_input":"2023-10-24T17:41:42.981410Z","iopub.status.idle":"2023-10-24T17:41:45.027570Z","shell.execute_reply.started":"2023-10-24T17:41:42.981373Z","shell.execute_reply":"2023-10-24T17:41:45.025070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;background-color:#1d1545;padding:3%;border-radius:50px 50px;font-size:1em;text-align:center\">Executive Summary</div>\n\n## <div style=\"font-size:20px;text-align:center;color:black;border-bottom:5px #0026d6 solid;padding-bottom:3%\">Preface</div>\n\n<div class=\"alert alert-block alert-info\"> 📌 <b>Summary</b>: This notebook is intended to \n<ul>\n    <li>Extend the exhaustive EDA on the training and test series in the dataset for <a href=\"https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states\" target='_blannk'>Child Mind Institute - Detect Sleep States contest</a></li>\n    <li>Extend/compliment the  findings and insights already collected in/manifested by another notebook per <a href=\"https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series\" target=\"_blank\">Plotting Big Data: Sleeping Series</a> by finding the repeatable strong patterns in behaviour of <i>anglez/enmo</i> around <i>onset/wakeup</i> events</li>\n</ul></div>\n\n## <div style=\"font-size:20px;text-align:center;color:black;border-bottom:5px #0026d6 solid;padding-bottom:3%\">Major takeaways</div>\n\nBased on the visual inspections of the OHLC charts for the testing series, matched with the 30-min window patterns for *anglez/enmo* around the events in the selective training series, we can conclude that \n\n- series **'038441c925bb'**: there is a high chance of the *onset* event to happen on *Aug 14, 2018 15:33 (local time)*\n- series **'03d92c9f6f8a'**: there is a high chance of the *onset* event to happen on *May 31, 2018 12:03 (local time)*\n- series **'0402a003dae9'**: there is a high chance of the *wakeup* event to happen on *Dec 18, 2018 12:47 (local time)*\n\nThe sections below manifest clear *anglez/enmo* curve behaviour patterns around both the *onset* and *wakeup* events in the training series data.","metadata":{}},{"cell_type":"markdown","source":"# <div style=\"color:white;background-color:#1d1545;padding:3%;border-radius:50px 50px;font-size:1em;text-align:center\">Prerequisites</div>\n\nBefore doing the BigData-scale analytics itself, we are going to complete the preparational steps below\n\n- import additional Python packages needed in our work (namely, duckdb)\n- start the instance of the in-memory database engine managed by DuckDB\n- mount the dataset files (train_series.parquet, test_series.parquet, and train_events.csv) as tables in the in-memory DuckDB database\n- create and publish the custom Python function to automate certain routines as for the data retrieval from DuckDB as well as building the visualizations","metadata":{}},{"cell_type":"code","source":"!pip install duckdb --quiet","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-24T17:41:45.032450Z","iopub.execute_input":"2023-10-24T17:41:45.033050Z","iopub.status.idle":"2023-10-24T17:42:00.404196Z","shell.execute_reply.started":"2023-10-24T17:41:45.032996Z","shell.execute_reply":"2023-10-24T17:42:00.402164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Importing gc module\nimport gc\nimport warnings\n\nimport pandas as pd\nimport numpy as np\nimport datetime as dt\n\n\nfrom scipy.signal import spectrogram\n\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nimport matplotlib.dates as mdates\nimport seaborn as sns\n\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nimport plotly.offline","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:00.406630Z","iopub.execute_input":"2023-10-24T17:42:00.408729Z","iopub.status.idle":"2023-10-24T17:42:02.198706Z","shell.execute_reply.started":"2023-10-24T17:42:00.408678Z","shell.execute_reply":"2023-10-24T17:42:02.197554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import duckdb\n\nparquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet'\ntesting_parquet_table = '/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet'\ntraining_events_table = '/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv'\n\ncon = duckdb.connect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:02.201610Z","iopub.execute_input":"2023-10-24T17:42:02.202319Z","iopub.status.idle":"2023-10-24T17:42:02.260749Z","shell.execute_reply.started":"2023-10-24T17:42:02.202277Z","shell.execute_reply":"2023-10-24T17:42:02.259636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n    CREATE TABLE train_series AS\n    SELECT * \n    FROM read_parquet('{parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:02.262606Z","iopub.execute_input":"2023-10-24T17:42:02.263033Z","iopub.status.idle":"2023-10-24T17:42:29.180215Z","shell.execute_reply.started":"2023-10-24T17:42:02.262996Z","shell.execute_reply":"2023-10-24T17:42:29.178760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"con.execute(f\"\"\"\n    CREATE TABLE test_series AS\n    SELECT * \n    FROM read_parquet('{testing_parquet_table}');\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.181735Z","iopub.execute_input":"2023-10-24T17:42:29.182309Z","iopub.status.idle":"2023-10-24T17:42:29.197156Z","shell.execute_reply.started":"2023-10-24T17:42:29.182270Z","shell.execute_reply":"2023-10-24T17:42:29.195987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we intentionally load every col as VARCHAR; we need to load timestamp as a varchar, to prevent the UTC time casting at load\ncon.execute(f\"\"\"\n    CREATE TABLE train_events AS\n    SELECT * \n    FROM read_csv('{training_events_table}', ALL_VARCHAR=1, AUTO_DETECT=TRUE);\n\"\"\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.199797Z","iopub.execute_input":"2023-10-24T17:42:29.200347Z","iopub.status.idle":"2023-10-24T17:42:29.286632Z","shell.execute_reply.started":"2023-10-24T17:42:29.200297Z","shell.execute_reply":"2023-10-24T17:42:29.285624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# inspired by \n# - https://www.kaggle.com/code/mitsuyasuhoshino/cmi-detectsleepstates-eda-visualization\n# - https://www.kaggle.com/code/carlmcbrideellis/zzzs-eda-utc-or-not-utc\ndef get_series_data(series_id):\n    \n    train_series_sql = f\"\"\"\n        SELECT\n        series_id,\n        CAST (step AS INTEGER) AS step,\n        timestamp,\n        CAST (anglez AS FLOAT) AS anglez,\n        CAST (enmo AS FLOAT) AS enmo\n        FROM train_series WHERE series_id = '{series_id}'\n    \"\"\"\n    df = con.execute(train_series_sql).fetchdf()\n    \n    #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n    df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n    df = df.drop('timestamp', axis=1)\n    \n    train_events_sql = f\"\"\"\n        SELECT \n            series_id,\n            CAST (night as INTEGER) as night,\n            event,\n            CAST (step as INTEGER) as step,\n            timestamp\n        FROM train_events \n        WHERE series_id = '{series_id}'\n    \"\"\"\n    \n    ev = con.execute(train_events_sql).fetchdf()\n    #ev['date_time'] = pd.to_datetime(ev['timestamp'], utc=True)\n    ev['date_time'] = pd.to_datetime(ev['timestamp']).apply(lambda t: t.tz_localize(None))\n    ev = ev.drop('timestamp', axis=1)\n    \n    df2 = pd.merge(df, ev, on='date_time', how='outer')\n    df2 = df2.drop(['series_id_x','series_id_y','step_x','step_y'], axis=1)\n    df2['event'] = df2['event'].fillna(0)\n    \n    # fill values for night feature\n    df2['night'][0] = 1\n    df2['night'] = df2['night'].interpolate()\n    df2['night'] = round(df2['night'])\n    # map numeric values to event feature\n    df2.loc[df2['event']=='onset','event'] = -1\n    df2.loc[df2['event']=='wakeup','event'] = 1\n    \n    return df2\n\ndef get_test_series_data(series_id):\n    test_series_sql = f\"\"\"\n        SELECT\n        series_id,\n        CAST (step AS INTEGER) AS step,\n        timestamp,\n        CAST (anglez AS FLOAT) AS anglez,\n        CAST (enmo AS FLOAT) AS enmo\n        FROM test_series WHERE series_id = '{series_id}'\n    \"\"\"\n    df = con.execute(test_series_sql).fetchdf()\n    \n    #df['date_time'] = pd.to_datetime(df['timestamp'], utc=True)\n    df['date_time'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n    df = df.drop('timestamp', axis=1)\n    \n    return df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.290505Z","iopub.execute_input":"2023-10-24T17:42:29.290942Z","iopub.status.idle":"2023-10-24T17:42:29.307227Z","shell.execute_reply.started":"2023-10-24T17:42:29.290903Z","shell.execute_reply":"2023-10-24T17:42:29.306053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# reused from https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series#Temporal-View:-OHLC-Resampling-of-Training-Series-TS\ndef plot_series_ohlc(series_id, freq='1min', nights=2):\n    \"\"\"\n    freq = '1min' : Sampling frequency, Optional 5s, 1min, 5min, 1H\n    \"\"\"\n    series_df  = get_series_data(series_id)\n    series = series_df[series_df['night'] <= nights] # Only display data for a specified number of nights (days)\n    series = series.set_index('date_time')\n    \n    tss_anglez = series['anglez'].resample(freq).ohlc()\n    tss_enmo = series['enmo'].resample(freq).ohlc()\n        \n    fig = make_subplots(rows=3, cols=1, subplot_titles=('<b>OHLC: anglez</b>',\n                                                    '<b>OHLC: enmo</b>',\n                                                    '<b>Event timeline</b>',\n                                                   ))\n\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['open'], name='anglez: opening'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['high'], name='anglez: high'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['low'], name='anglez: low'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['close'], name='anglez: close'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['open'], name='enmo: open'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['high'], name='enmo: high'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['low'], name='enmo: low'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['close'], name='enmo: close'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=series.index, y=series['event'], name='events'), row=3, col=1)\n                                \n    # Update visual layout\n    fig.update_layout(\n        showlegend=True,\n        width=600,\n        height=400,\n        autosize=False,\n        margin=dict(t=15, b=0, l=5, r=5),\n        template=\"plotly_white\",\n        colorway=px.colors.qualitative.Prism ,\n    )\n\n    # update font size at the axes\n    fig.update_coloraxes(colorbar_tickfont_size=10)\n    # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n    fig.update_annotations(font_size=12)\n    # Reduce opacity\n    fig.update_traces(opacity=0.75)\n    \n    return fig\n\ndef plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min'):\n    start_date = window_median_date_time - dt.timedelta(minutes=mins_to_look_up)\n    end_date = window_median_date_time + dt.timedelta(minutes=mins_to_look_up)\n    series = get_series_data(series_id)\n    series = series[(series['date_time'] >= start_date) & (series['date_time'] <= end_date)]\n    \n    series = series.set_index('date_time')\n    \n    tss_anglez = series['anglez'].resample(freq).ohlc()\n    tss_enmo = series['enmo'].resample(freq).ohlc()\n        \n    fig = make_subplots(rows=2, cols=1, subplot_titles=(\n                            '<b>OHLC: anglez</b>',\n                            '<b>OHLC: enmo</b>',\n    ))\n\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['open'], name='anglez: opening'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['high'], name='anglez: high'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['low'], name='anglez: low'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['close'], name='anglez: close'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['open'], name='enmo: open'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['high'], name='enmo: high'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['low'], name='enmo: low'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['close'], name='enmo: close'), row=2, col=1)\n                                \n    # Update visual layout\n    fig.update_layout(\n        showlegend=True,\n        width=600,\n        height=400,\n        autosize=False,\n        margin=dict(t=15, b=0, l=5, r=5),\n        template=\"plotly_white\",\n        colorway=px.colors.qualitative.Prism ,\n    )\n\n    # update font size at the axes\n    fig.update_coloraxes(colorbar_tickfont_size=10)\n    # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n    fig.update_annotations(font_size=12)\n    # Reduce opacity\n    fig.update_traces(opacity=0.75)\n    \n    return fig\n\n# reused from https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series#OHLC-Resampling-of-Test-Series-Data\ndef plot_test_series_ohlc(series_id, freq='1min'):\n    \"\"\"\n    freq = '1min' : Sampling frequency, Optional 5s, 1min, 5min, 1H\n    \"\"\"\n    series  = get_test_series_data(series_id)\n\n    series = series.set_index('date_time')\n    \n    tss_anglez = series['anglez'].resample(freq).ohlc()\n    tss_enmo = series['enmo'].resample(freq).ohlc()\n        \n    fig = make_subplots(rows=2, cols=1, subplot_titles=(\n                            '<b>OHLC: anglez</b>',\n                            '<b>OHLC: enmo</b>',\n    ))\n\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['open'], name='anglez: opening'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['high'], name='anglez: high'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['low'], name='anglez: low'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_anglez.index, y=tss_anglez['close'], name='anglez: close'), row=1, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['open'], name='enmo: open'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['high'], name='enmo: high'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['low'], name='enmo: low'), row=2, col=1)\n    fig.add_trace(go.Scatter(x=tss_enmo.index, y=tss_enmo['close'], name='enmo: close'), row=2, col=1)\n                                \n    # Update visual layout\n    fig.update_layout(\n        showlegend=True,\n        width=600,\n        height=400,\n        autosize=False,\n        margin=dict(t=15, b=0, l=5, r=5),\n        template=\"plotly_white\",\n        colorway=px.colors.qualitative.Prism ,\n    )\n\n    # update font size at the axes\n    fig.update_coloraxes(colorbar_tickfont_size=10)\n    # Update font in the titles: Apparently subplot titles are annotations (Subplot font size is hardcoded to 16pt · Issue #985)\n    fig.update_annotations(font_size=12)\n    # Reduce opacity\n    fig.update_traces(opacity=0.75)\n    \n    return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.310375Z","iopub.execute_input":"2023-10-24T17:42:29.311348Z","iopub.status.idle":"2023-10-24T17:42:29.352656Z","shell.execute_reply.started":"2023-10-24T17:42:29.311287Z","shell.execute_reply":"2023-10-24T17:42:29.351421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = \"SELECT DISTINCT series_id FROM test_series\"\ndf_testing_series_id = con.execute(query).fetchdf()\n\ntest_series_id_list = df_testing_series_id[\"series_id\"].tolist()\n\n# list of series_id for training series with the fully labelled events\n# see https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive/ for the data analysis flow resulted in locating these seria\nfully_labelled_training_series_id_list = ['08db4255286f', '0a96f4993bd7', '0cfc06c129cc', '1087d7b0ff2e', '10f8bc1f7b07', '18b61dd5aae8', \n                                '29c75c018220', '31011ade7c0a', '3452b878e596', '349c5562ee2c', '3664fe9233f9', '483d6545417f', \n                                '55a47ff9dc8a', '5acc9d63b5fd', '5f94bb3e1bed', '655f19eabf1e', '67f5fc60e494', '72bbd1ac3edf', \n                                '76237b9406d5', '7822ee8fe3ec', '89bd631d1769', '8e32047cbc1f', '939932f1822d', '9ee455e4770d', \n                                'a596ad0b82aa', 'a9a2f7fac455', 'a9e5f5314bcb', 'af91d9a50547', 'b364205aba43', 'c535634d7dcd', \n                                'c6788e579967', 'c68260cc9e8f', 'ca730dbf521d', 'd150801f3145', 'd25e479ecbb7', 'd515236bdeec', \n                                'd5e47b94477e']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.359770Z","iopub.execute_input":"2023-10-24T17:42:29.361266Z","iopub.status.idle":"2023-10-24T17:42:29.377113Z","shell.execute_reply.started":"2023-10-24T17:42:29.361211Z","shell.execute_reply":"2023-10-24T17:42:29.375808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;background-color:#1d1545;padding:3%;border-radius:50px 50px;font-size:1em;text-align:center\">Visualizing Training Series Data: TS Charts</div>\n\nLet's build the *anglez* and *enmo* charts for one of the nights in the selective training series. As an example, let's choose the series with series_id = *'08db4255286f'*, night *1*.","metadata":{}},{"cell_type":"code","source":"# inspired by https://www.kaggle.com/code/gauravk123/interactive-eda-per-subject-train-series-ts\ndef visualize_ts_data(series_id, night, attribute_name):\n    series  = get_series_data(series_id)\n    \n    night1_series = series[series['night'] == night]\n    \n    chart_title = f\"Time Series Data: series '{series_id}'; night {night}; attribute: {attribute_name}\"\n    \n    fig = px.line(night1_series, x='date_time', y=attribute_name, title=chart_title)\n    fig.update_xaxes(title_text='timestamp')\n    fig.update_yaxes(title_text=attribute_name)\n    \n    event_onset = night1_series[night1_series['event'] == -1]\n    state = \"Sleep\" \n    \n    fig.add_vrect(x0=event_onset['date_time'].iloc[0], \n                  x1=event_onset['date_time'].iloc[0], \n                  fillcolor='rgba(255, 0, 0, 0.3)', \n                  annotation_text=f'{state}')\n    \n    event_wakeup = night1_series[night1_series['event'] == 1]\n    state = \"Wakeup\" \n    \n    fig.add_vrect(x0=event_wakeup['date_time'].iloc[0], \n                  x1=event_wakeup['date_time'].iloc[0], \n                  fillcolor='rgba(255, 0, 0, 0.3)', \n                  annotation_text=f'{state}')\n    \n    fig.add_shape(type=\"rect\",\n                  x0=event_onset[\"date_time\"].iloc[0],\n                  x1=event_wakeup[\"date_time\"].iloc[0],\n                  y0=-500,  # Set this to a large negative number\n                  y1=500,   # Set this to a large positive number\n                  fillcolor=\"LightSalmon\",\n                  opacity=0.5,\n                  layer=\"below\",\n                  line_width=0,\n    )\n    \n    return fig","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.378769Z","iopub.execute_input":"2023-10-24T17:42:29.379165Z","iopub.status.idle":"2023-10-24T17:42:29.392165Z","shell.execute_reply.started":"2023-10-24T17:42:29.379133Z","shell.execute_reply":"2023-10-24T17:42:29.391070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings('ignore')\nseries_id = '08db4255286f'\nnight = 1\nattribute_name = 'anglez'\nfig = visualize_ts_data(series_id, night, attribute_name)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:29.393991Z","iopub.execute_input":"2023-10-24T17:42:29.394671Z","iopub.status.idle":"2023-10-24T17:42:42.506541Z","shell.execute_reply.started":"2023-10-24T17:42:29.394636Z","shell.execute_reply":"2023-10-24T17:42:42.502897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings('ignore')\nseries_id = '08db4255286f'\nnight = 1\nattribute_name = 'enmo'\nfig = visualize_ts_data(series_id, night, attribute_name)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:42.508868Z","iopub.execute_input":"2023-10-24T17:42:42.510490Z","iopub.status.idle":"2023-10-24T17:42:55.265252Z","shell.execute_reply.started":"2023-10-24T17:42:42.510439Z","shell.execute_reply":"2023-10-24T17:42:55.263938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;background-color:#1d1545;padding:3%;border-radius:50px 50px;font-size:1em;text-align:center\">Temporal View: OHLC Resampling of Training Series TS</div>\n\nIn finance (and some other industries as well), a popular way to aggregate a time series is to implement **Open-high-low-close(OHLC) resampling**.\n\nThe essence of this technique is to to compute four values for each bucket (time interval) as follows\n\n- the *first (open)* value,\n- the *last (close)* value,\n- the *maximum (high)* value, and\n- the *minimal (low)* values\n\nLet's apply OHLC resampling to visualize the correlated *anglez, enmo, and event trends* on a single diagram, for selected training series.","metadata":{}},{"cell_type":"markdown","source":"## <div style=\"font-size:20px;text-align:center;color:black;border-bottom:5px #0026d6 solid;padding-bottom:3%\">Series '08db4255286f': OHLC View</div>\n\nLet's build OHLC charts for initial *6* nights in series *'08db4255286f'*, resampling by the *15-min* intervals.","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nfig = plot_series_ohlc(series_id, freq='15min', nights=6)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:42:55.266865Z","iopub.execute_input":"2023-10-24T17:42:55.267274Z","iopub.status.idle":"2023-10-24T17:43:10.227075Z","shell.execute_reply.started":"2023-10-24T17:42:55.267239Z","shell.execute_reply":"2023-10-24T17:43:10.224433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n          \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-24T17:43:10.228947Z","iopub.execute_input":"2023-10-24T17:43:10.229368Z","iopub.status.idle":"2023-10-24T17:43:10.359520Z","shell.execute_reply.started":"2023-10-24T17:43:10.229333Z","shell.execute_reply":"2023-10-24T17:43:10.358074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see certain patterns in *anglez/enmo* behaviour (as described in [another notebook](https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series) in more details).\n\nNow, let's try to see local *anglez/enmo* curve changes around the events. To achieve it, we are going to do OHLC resampling of a selected TS windows around selective *wakeup events* in the series (30 min windows around the event time, with resampling frequency to be *1 min*)\n- Nov 6, 2018  5:59\n- Nov 7, 2018  5:30\n- Nov 8, 2018  6:14\n- Nov 9, 2018  6:07\n- Nov 10, 2018 8:28\n- Nov 11, 2018 9:33\n\n### 30-min Window around the wakeup event: Nov 6, 2018  5:59","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-06 05:59:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:43:10.361739Z","iopub.execute_input":"2023-10-24T17:43:10.362251Z","iopub.status.idle":"2023-10-24T17:43:22.808021Z","shell.execute_reply.started":"2023-10-24T17:43:10.362203Z","shell.execute_reply":"2023-10-24T17:43:22.806942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that, around the wakeup event (*5:59*), \n- *anglez high* and *anglez low* curves display going up at the momement of wakeup\n- just prior to the wakeup event (on the short time interval of *5:51 - 5:58*), we can see the start of the drastic growth of *anglez high* and *anglez low* curves - it could indicate the certain boundary state of a person to be about to wake up\n- *anglez high* and *anglez low* curves go to the levels of the boundary state of *5:51 - 5:58* after wakeup (see the dymanics of these curves after *6:00*)\n- *enmo low* starts to grow in a few mins after the wakeup (see the dynamics of the respective curve since *6:03*)\n- *enmo high* displays the minor spike at the event spot (*5:59*), and then it starts to grow significantly a few min after the wakeup, in synch with *enmo low* curve (see the dynamics of the respective curve since *6:03*)","metadata":{}},{"cell_type":"markdown","source":"### 30-min Window around the wakeup event: Nov 7, 2018  5:30","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-07 05:30:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:43:22.809742Z","iopub.execute_input":"2023-10-24T17:43:22.810213Z","iopub.status.idle":"2023-10-24T17:43:34.775718Z","shell.execute_reply.started":"2023-10-24T17:43:22.810150Z","shell.execute_reply":"2023-10-24T17:43:34.774471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that, around the wakeup event (*5:30*), \n- *anglez high* and *anglez low* curves display significant going up at the momement of wakeup, after a long time of the low values \n- there is no pre-wakeup  spikes in *anglez high* and *anglez low*, in contrary to the wakeup for the previous night\n- *enmo low* starts to grow in a few mins after the wakeup (see the dynamics of the respective curve since *5:34*), and then it oscellates around the value-at-wakeup\n- *enmo high* display significant going up at the momement of wakeup, after a long time of the low values\n- *enmo high* goes down a little shortly after wakeup (*5:37*), and then it starts growing again (until *5:53*); after the growth, it drops down to the level registered around *5:37*","metadata":{}},{"cell_type":"markdown","source":"### 30-min Window around the wakeup event: Nov 8, 2018 6:14","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-08 06:14:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:43:34.777682Z","iopub.execute_input":"2023-10-24T17:43:34.778979Z","iopub.status.idle":"2023-10-24T17:43:46.918907Z","shell.execute_reply.started":"2023-10-24T17:43:34.778923Z","shell.execute_reply":"2023-10-24T17:43:46.917426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the wakeup event: Nov 9, 2018 6:07","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-09 06:07:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:43:46.920562Z","iopub.execute_input":"2023-10-24T17:43:46.921105Z","iopub.status.idle":"2023-10-24T17:43:59.091647Z","shell.execute_reply.started":"2023-10-24T17:43:46.921065Z","shell.execute_reply":"2023-10-24T17:43:59.090466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the wakeup event: Nov 10, 2018 8:28","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-10 08:28:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-24T17:43:59.093543Z","iopub.execute_input":"2023-10-24T17:43:59.094066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the wakeup event: Nov 11, 2018 9:33","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-11 09:33:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n          \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's do OHLC resampling of a selected TS windows around selective *onset events* in the series\n- Nov 6, 2018  2:39\n- Nov 7, 2018  1:50\n- Nov 7, 2018  23:22\n- Nov 8, 2018  21:48\n- Nov 9, 2018  23:56\n- Nov 11, 2018 2:15","metadata":{}},{"cell_type":"markdown","source":"### 30-min Window around the onset event: Nov 6, 2018 2:39","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-06 02:39:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the onset event: Nov 7, 2018 1:50","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-07 01:50:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the onset event: Nov 7, 2018 23:22","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-07 23:22:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the onset event: Nov 8, 2018 21:48","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-08 21:48:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the onset event: Nov 9, 2018 23:56","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-09 23:56:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 30-min Window around the onset event: Nov 11, 2018 2:15","metadata":{}},{"cell_type":"code","source":"series_id = '08db4255286f'\nwindow_median_date_time = dt.datetime.strptime('2018-11-11 02:15:00', '%Y-%m-%d %H:%M:%S')\nfig = plot_train_series_ts_window_ohlc(series_id, window_median_date_time, mins_to_look_up=30, freq='1min')\nfig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n          \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;background-color:#1d1545;padding:3%;border-radius:50px 50px;font-size:1em;text-align:center\">OHLC Resampling of Test Series Data</div>\n\n","metadata":{}},{"cell_type":"code","source":"for series_id in test_series_id_list:\n    display(Markdown(\"## <div style='font-size:20px;text-align:center;color:black;border-bottom:5px #0026d6 solid;padding-bottom:3%'>OHLC Resampling: Series '\" + series_id + \"'</div>\"  ))\n    fig = plot_test_series_ohlc(series_id, freq='1min')\n    fig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Returns the number of\n# objects it has collected\n# and deallocated\ncollected = gc.collect()\n \n# Prints Garbage collector\n# as 0 object\nprint(\"Garbage collector: collected\",\n          \"%d objects.\" % collected)# Importing gc module","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <div style=\"font-size:20px;text-align:center;color:black;border-bottom:5px #0026d6 solid;padding-bottom:3%\">Test Series Event Predictions</div>\n\nBased on the visual inspections of the OHLC charts for the testing series above, matched with the 30-min window patterns for *anglez/enmo* around the events in the selective training series, we can conclude that \n\n- series **'038441c925bb'**: there is a high chance of the *onset* event to happen on *Aug 14, 2018 15:33 (local time)*\n- series **'03d92c9f6f8a'**: there is a high chance of the *onset* event to happen on *May 31, 2018 12:03 (local time)*\n- series **'0402a003dae9'**: there is a high chance of the *wakeup* event to happen on *Dec 18, 2018 12:47 (local time)*","metadata":{}},{"cell_type":"markdown","source":"# <div style=\"color:white;background-color:#1d1545;padding:3%;border-radius:50px 50px;font-size:1em;text-align:center\">References</div>\n\nThis notebook extends the set of EDA insights/findings for **Child Mind Institute - Detect Sleep States** competition. You can find other notebooks of mine with interesting EDA insights as per the links below\n\n- [To Sleep or Not to Sleep? Deep EDA Dive](https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive)\n- [Plotting Big Data: Sleeping Series](https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series)","metadata":{}}]}