{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"}],"dockerImageVersionId":30558,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### INTRO\n---\n\nThis notebook consists of several parts:\n\n- competition data checks and clean up\n- interactive data inspector widgets with Ipython (**YOU NEED TO RUN THE NOTEBOOK TO SEE THE WIDGETS AND THEIR OUTPUT**)\n- preparation of data and creation of neural network for prediction of asleep state (target)\n- test data pre- and post- processing, prediction and preparation for submission (**)\n\n(**) If you try to make a submission from this notebook, it will fail with a \"Notebook Threw Exception\" error. I have been able to make succesful submissions with other notebooks based on this one, after splitting the data pre-processing, the model training and the test data predictions three separate notebooks, but remember this is just an EDA analysis and proof-of-concept notebook, and as such you may find errors, omissions or sub-optimal and unfinished bits.\n\n---\n\n**WARNING: Be aware that training the neural network with a large number of series_id and EPOCH may take several hours**  \n\n---","metadata":{}},{"cell_type":"markdown","source":"### Library imports and data loading\n---","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport numpy as np\nimport pandas as pd\nimport random\nfrom datetime import datetime\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport ipywidgets as widgets\nfrom IPython.display import display\nimport warnings\nfrom sklearn.preprocessing import MinMaxScaler, StandardScaler\nfrom keras.models import Sequential, Functional, Model\nfrom keras.layers import Input, Conv1D, Dense, LSTM, Dropout, Bidirectional\nfrom keras.optimizers import Adam, Adamax\nfrom keras.losses import BinaryCrossentropy\nfrom keras.callbacks import EarlyStopping\nfrom keras.utils import plot_model\n\n# Set precision to two decimals\npd.set_option(\"display.precision\", 2)\n\n# Charts inline\n%matplotlib inline\n\n# Ignore warning\nwarnings.filterwarnings(\"ignore\")\n\n# List files \nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:30:47.068112Z","iopub.execute_input":"2023-11-21T21:30:47.068591Z","iopub.status.idle":"2023-11-21T21:30:47.096284Z","shell.execute_reply.started":"2023-11-21T21:30:47.068549Z","shell.execute_reply":"2023-11-21T21:30:47.094776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Reading files.....')\npath = '/kaggle/input/child-mind-institute-detect-sleep-states/'\ntrain_events = pd.read_csv(path + 'train_events.csv')\ntrain_series = pd.read_parquet(path + 'train_series.parquet')\ntest_series = pd.read_parquet(path + 'test_series.parquet')\nsample_submission = pd.read_csv(path + 'sample_submission.csv')\nprint('..... done!')","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:30:47.099755Z","iopub.execute_input":"2023-11-21T21:30:47.100280Z","iopub.status.idle":"2023-11-21T21:32:05.929867Z","shell.execute_reply.started":"2023-11-21T21:30:47.100229Z","shell.execute_reply":"2023-11-21T21:32:05.928226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training data inspection and clean up\n---\n\nAny event records without time information is of no use, so I´ll start removing those.\n\nAlso, each night can have only two events, one onset and one wakeup, so any nights breaking the rule will be discarded too.","metadata":{}},{"cell_type":"code","source":"train_events.info()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:05.931544Z","iopub.execute_input":"2023-11-21T21:32:05.931934Z","iopub.status.idle":"2023-11-21T21:32:05.954669Z","shell.execute_reply.started":"2023-11-21T21:32:05.931902Z","shell.execute_reply":"2023-11-21T21:32:05.953022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let´s start removing any event records without time information.","metadata":{}},{"cell_type":"code","source":"train_events = train_events.dropna(axis=0, ignore_index=True)\ntrain_events.info()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:05.959383Z","iopub.execute_input":"2023-11-21T21:32:05.959967Z","iopub.status.idle":"2023-11-21T21:32:05.989694Z","shell.execute_reply.started":"2023-11-21T21:32:05.959916Z","shell.execute_reply":"2023-11-21T21:32:05.988238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let´s look for any instances of \"bad nights\" meaning any nights with anything different to a single onset / wakeup pair.","metadata":{}},{"cell_type":"code","source":"bad_nights = train_events[['series_id', 'night', 'event']].groupby(by=['series_id', 'night']).count()\nbad_nights[bad_nights['event']!=2]","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:05.991299Z","iopub.execute_input":"2023-11-21T21:32:05.992302Z","iopub.status.idle":"2023-11-21T21:32:06.025072Z","shell.execute_reply.started":"2023-11-21T21:32:05.992265Z","shell.execute_reply":"2023-11-21T21:32:06.023294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So here we have a few of such cases, with just one event.","metadata":{}},{"cell_type":"code","source":" train_events[(((train_events['series_id']=='0ce74d6d2106') & (train_events['night']==20)) |\n               ((train_events['series_id']=='154fe824ed87') & (train_events['night']==30)) |\n               ((train_events['series_id']=='44a41bba1ee7') & (train_events['night']==10)) |                \n               ((train_events['series_id']=='efbfc4526d58') & (train_events['night']==7)) |\n               ((train_events['series_id']=='f8a8da8bdd00') & (train_events['night']==17)))]","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:06.027221Z","iopub.execute_input":"2023-11-21T21:32:06.027795Z","iopub.status.idle":"2023-11-21T21:32:06.070890Z","shell.execute_reply.started":"2023-11-21T21:32:06.027735Z","shell.execute_reply":"2023-11-21T21:32:06.069553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This data is problematic, so I'll remove these records.","metadata":{}},{"cell_type":"code","source":"train_events = train_events[~(((train_events['series_id']=='0ce74d6d2106') & (train_events['night']==20)) |\n                              ((train_events['series_id']=='154fe824ed87') & (train_events['night']==30)) |\n                              ((train_events['series_id']=='44a41bba1ee7') & (train_events['night']==10)) |                \n                              ((train_events['series_id']=='efbfc4526d58') & (train_events['night']==7)) |\n                              ((train_events['series_id']=='f8a8da8bdd00') & (train_events['night']==17)))].reset_index(drop=True)\n\ntrain_events.info()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:06.072646Z","iopub.execute_input":"2023-11-21T21:32:06.072984Z","iopub.status.idle":"2023-11-21T21:32:06.110945Z","shell.execute_reply.started":"2023-11-21T21:32:06.072955Z","shell.execute_reply":"2023-11-21T21:32:06.109619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_list = list(train_events['series_id'].unique())\nprint('Total number of valid series in training dataset: '+ str(len(series_list)))","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:06.113336Z","iopub.execute_input":"2023-11-21T21:32:06.113856Z","iopub.status.idle":"2023-11-21T21:32:06.122410Z","shell.execute_reply.started":"2023-11-21T21:32:06.113812Z","shell.execute_reply":"2023-11-21T21:32:06.120920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Update train_series dataset with final set of series\ntrain_series = train_series[train_series['series_id'].isin(series_list)].reset_index(drop=True)\n# Check for nulls\ntrain_series.isnull().any()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:32:06.124630Z","iopub.execute_input":"2023-11-21T21:32:06.125027Z","iopub.status.idle":"2023-11-21T21:33:16.180566Z","shell.execute_reply.started":"2023-11-21T21:32:06.124994Z","shell.execute_reply":"2023-11-21T21:33:16.179095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A look at the test series and submission sample\n---\n\nThe test series data gives us a very important requirement: we have to try and detect events (onset, wakeup) in sequences of 150 steps (x 5 sec/step = 750 seconds = 12.5 minutes).","metadata":{}},{"cell_type":"code","source":"def plot_test_series():\n    fig, ax = plt.subplots(3,1, figsize=(10,8), sharex=True)\n    ax2 = {}\n    i = 0\n    test_series_list = list(test_series['series_id'].unique())\n    for id in test_series_list:\n        ax[i].set_title('series_id: ' + id, loc='left', fontsize=10)\n        x = test_series[test_series['series_id']==id]['step']\n        y_enmo = test_series[test_series['series_id']==id]['enmo']\n        y_anglez = test_series[test_series['series_id']==id]['anglez']\n        ax[i].plot(x, y_enmo, label='enmo', color='magenta', lw=1, alpha=0.9)\n        ax[i].set_ylabel('enmo', fontsize=9)\n        ax[i].tick_params(labelsize=9)\n        ax2[i] = ax[i].twinx()\n        ax2[i].plot(x, y_anglez, label='anglez', color='lime', lw=1, alpha=0.9)\n        ax2[i].set_ylabel('anglez', fontsize=9)\n        i = i + 1\n    h1, l1 = ax[2].get_legend_handles_labels()\n    h2, l2 = ax2[2].get_legend_handles_labels()\n    plt.figlegend(h1+h2, l1+l2, loc='upper left', bbox_to_anchor=(0.12, 0.97, 0.0, 0.0), ncols=2)\n    ax[2].set_xlabel('step')\n    plt.show()\n\n# Plot it\nplot_test_series()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:33:16.187584Z","iopub.execute_input":"2023-11-21T21:33:16.188073Z","iopub.status.idle":"2023-11-21T21:33:17.487805Z","shell.execute_reply.started":"2023-11-21T21:33:16.188028Z","shell.execute_reply":"2023-11-21T21:33:17.486494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series.isnull().any()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:33:17.489592Z","iopub.execute_input":"2023-11-21T21:33:17.490002Z","iopub.status.idle":"2023-11-21T21:33:17.501880Z","shell.execute_reply.started":"2023-11-21T21:33:17.489965Z","shell.execute_reply":"2023-11-21T21:33:17.500498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:33:17.503909Z","iopub.execute_input":"2023-11-21T21:33:17.504348Z","iopub.status.idle":"2023-11-21T21:33:17.521186Z","shell.execute_reply.started":"2023-11-21T21:33:17.504310Z","shell.execute_reply":"2023-11-21T21:33:17.519778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Subsampling (ONLY FOR DEVELOPMENT) and merge of train series and events datasets\n---\n\nThe training_series data file is too hefty so I just sampled it down whilst developing the solution.\n\nThe actual number of series is set in the first line of the following code cell.","metadata":{}},{"cell_type":"code","source":"%%time\nNBR_TRAINING_SERIES = 200\nfull_series = series_list\nselected_series = random.sample(full_series, NBR_TRAINING_SERIES)\n\n# Subsample data\ntrain_series = train_series[train_series['series_id'].isin(selected_series)]\ntrain_events = train_events[train_events['series_id'].isin(selected_series)]\n\n# Merge train_series and train_events into a single fat dataframe\ntrain_merged = pd.merge(train_series, train_events, on=['series_id', 'step', 'timestamp'], how='outer').reset_index(drop=True)\n\nprint('Nbr. of selected training series: ' + str(NBR_TRAINING_SERIES))\nprint('Nbr. of data points (timesteps): ' + str(len(train_merged)))\nprint()","metadata":{"execution":{"iopub.status.busy":"2023-11-21T21:33:17.522949Z","iopub.execute_input":"2023-11-21T21:33:17.523553Z","iopub.status.idle":"2023-11-21T21:36:49.073780Z","shell.execute_reply.started":"2023-11-21T21:33:17.523520Z","shell.execute_reply":"2023-11-21T21:36:49.072713Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Full series inspector widget\n---\n\nThis widget allows you to browse through the selected series of accelerometer readings. In each case, all the nights and events registered for that series are displayed.","metadata":{}},{"cell_type":"code","source":"def plot_series_and_events(series_id, train_merged):\n    # Make local copy of datasets\n    data = train_merged.copy()\n    # Filter the DataFrame based on the series_id\n    sample_series = data[data['series_id'] == series_id].copy()\n    # Convert timestamp to Pandas datetime object\n    sample_series['Date'] = sample_series['timestamp'].str.split('T', expand=True)[0]\n    sample_series['Time'] = sample_series['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\n    sample_series['timestamp'] = pd.to_datetime(sample_series['Date'] + ' ' + sample_series['Time'])\n    # Filter event data based on the series_id\n    sample_events = data[data['series_id'] == series_id].copy()\n    sample_events['Date'] = sample_events['timestamp'].str.split('T', expand=True)[0]\n    sample_events['Time'] = sample_events['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\n    sample_events['timestamp'] = pd.to_datetime(sample_events['Date'] + ' ' + sample_events['Time'])\n    # Separate events\n    sample_onset = sample_events[sample_events['event'] == 'onset'] #.dropna()\n    sample_wakeup = sample_events[sample_events['event'] == 'wakeup'] #.dropna()\n    ####################################\n    # Plot anglez and enmo along with onset and wakeup events\n    fig, ax = plt.subplots(2, 1, figsize=(16, 4), sharex=True)\n    fig.suptitle('Accelerometer readings (enmo, anglez) and annotated events for series_id = ' + series_id,\n                 fontsize=13, y=0.95)\n    ####################################    \n    for onset in sample_onset['timestamp']:\n        ax[0].axvline(x=onset, color='Navy', linestyle='--', label='onset', alpha=0.8)\n    for wakeup in sample_wakeup['timestamp']:\n        ax[0].axvline(x=wakeup, color='Red', linestyle='--', label='wakeup', alpha=0.8)    \n    ax[0].plot(sample_series['timestamp'], sample_series['enmo'], label='enmo',\n               linewidth=1, color='magenta', alpha=0.4)    \n    ax[0].set_ylabel('enmo', fontsize=11)    \n    handles, labels = ax[0].get_legend_handles_labels()\n    new_labels, new_handles = [], []\n    for handle, label in zip(handles, labels):\n        if label not in new_labels:\n            new_handles.append(handle)\n            new_labels.append(label)\n    ax[0].legend(new_handles, new_labels, loc='best', fontsize=10)            \n    ####################################    \n    for onset in sample_onset['timestamp']:\n        ax[1].axvline(x=onset, color='Navy', ls='--', label='onset', alpha=0.8)        \n    for wakeup in sample_wakeup['timestamp']:\n        ax[1].axvline(x=wakeup, color='Red', ls='--', label='wakeup', alpha=0.8)        \n    ax[1].plot(sample_series['timestamp'], sample_series['anglez'], label='anglez',\n               linewidth=1, color='lime', alpha=0.4)    \n    ax[1].set_ylabel('anglez',fontsize=11)    \n    handles, labels = ax[1].get_legend_handles_labels()\n    new_labels, new_handles = [], []\n    for handle, label in zip(handles, labels):\n        if label not in new_labels:\n            new_handles.append(handle)\n            new_labels.append(label)\n    ax[1].legend(new_handles, new_labels, loc='best', fontsize=10)        \n    ####################################    \n    plt.xlabel('timestamp')\n    plt.show()\n\n# Selector box handlenr function \ndef series_selector_eventhandler(change):\n    selection = series_selector.value\n    output.clear_output(wait=True)\n    with output:\n        plot_series_and_events(selection, train_merged)   \n\n#################################\n# Build and display widget\nseries_selector = widgets.Select(description='Series_id', options=selected_series, value=selected_series[0], rows=8)\nseries_selector.observe(series_selector_eventhandler, names='value')\noutput = widgets.Output()\ndisplay(series_selector, output)\n#################################\n\nwith output:\n    plot_series_and_events(series_selector.value, train_merged)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:36:49.077407Z","iopub.execute_input":"2023-11-21T21:36:49.077845Z","iopub.status.idle":"2023-11-21T21:38:14.699399Z","shell.execute_reply.started":"2023-11-21T21:36:49.077809Z","shell.execute_reply":"2023-11-21T21:38:14.698162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Single events inspector widget\n---\n\nThis widget allows you to browse through individual nights, showing enmo and anglez readings, along with the onset and wakeup events, in a window including the sleep period plus / minus six hours.\n\nSometimes, an error may occur if one specific series does not have a pair onset / wakeup registered for one night. Just click on another night to continue.","metadata":{}},{"cell_type":"code","source":"window = 4320 # full sleep period plus six hours before and after\n# Function to plot one pair onset - wakeup of events\ndef plot_event_pair(series_id, night, train_merged, w=window):\n    # Extract onset and wakeup events on the night\n    onset_id = train_merged[(train_merged['series_id']==series_id) &\n                            (train_merged['night']==night) &\n                            (train_merged['event']=='onset')].index[0]\n    wakeup_id = train_merged[(train_merged['series_id']==series_id) &\n                             (train_merged['night']==night) &\n                             (train_merged['event']=='wakeup')].index[0]\n    sample_pair = train_merged.iloc[onset_id - w : wakeup_id + w]\n    \n    # Plot charts of enmo and anglez, with onset and wakeup events for one night\n    fig, ax = plt.subplots(2,1, figsize=(20,4), sharex=True)\n    fig.suptitle('Full sleep period plus / minus six hours for series_id = ' + str(series_id) + ', night = ' + str(night),\n                 fontsize=13, y=0.95)\n    #############\n    ax[0].plot(sample_pair['step'], sample_pair['enmo'], label='enmo',\n               color='magenta', lw=1, alpha=0.6)\n    ax[0].axvline(train_merged.iloc[onset_id]['step'], label='onset', ls='--', color='Navy', alpha=0.9)\n    ax[0].axvline(train_merged.iloc[wakeup_id]['step'], label='wakeup', ls='--', color='Red', alpha=0.9)\n    ax[0].axvspan(train_merged.iloc[onset_id]['step'], train_merged.iloc[wakeup_id]['step'], ymin=0, ymax=1,\n                  color='coral', linewidth=0.1, alpha=0.1, label='asleep')\n    ax[0].legend(loc='best', fontsize=10)\n    ##########################\n    ax[1].plot(sample_pair['step'], sample_pair['anglez'], label='anglez',\n                   color='lime', lw=1, alpha=0.6)\n    ax[1].axvline(train_merged.iloc[onset_id]['step'], label='onset', ls='--', color='Navy', alpha=0.9)\n    ax[1].axvline(train_merged.iloc[wakeup_id]['step'], label='wakeup', ls='--', color='Red', alpha=0.9)\n    ax[1].axvspan(train_merged.iloc[onset_id]['step'], train_merged.iloc[wakeup_id]['step'], ymin=0, ymax=1,\n                  color='coral', linewidth=0.1, alpha=0.1, label='asleep')\n    ax[1].legend(loc='best', fontsize=10)\n    ##########################\n    plt.xlabel('step')\n    plt.show()\n    \n    ##########################\n    # Plot charts of single events (150 timestep windows)\n    fig, ax = plt.subplots(1,2, figsize=(20,4), sharex=False)\n    fig.suptitle('Single onset and wakeup events and emno / anglez readings in 150 timesteps window (12.5 minutes) for series_id = ' + str(series_id) + ', night = ' + str(night), \n                 fontsize=13, y=0.95)\n    ##########################\n    single_onset = train_merged.iloc[onset_id - 75 : onset_id + 75]\n    ax[0].plot(single_onset['step'], single_onset['enmo'], color='magenta', lw=1, alpha=0.75) \n    ax[0].axvline(train_merged.iloc[onset_id]['step'], ls='--', color='Navy', alpha=0.9, label='onset')\n    ax[0].axvspan(train_merged.iloc[onset_id]['step'], train_merged.iloc[onset_id + 75]['step'], ymin=0, ymax=1,\n                  color='coral', linewidth=0.1, alpha=0.1, label='asleep')\n    ax[0].legend(loc='upper right', fontsize=10)\n    ax[0].set_xlabel('step')\n    ax[0].set_ylabel('enmo')\n    ax0b = ax[0].twinx()\n    ax0b.plot(single_onset['step'], single_onset['anglez'], color='lime', lw=1, alpha=0.75) \n    ax0b.set_ylabel('anglez')\n    ax0b.legend(loc='best', fontsize=10)\n    ##########################\n    single_wakeup = train_merged.iloc[wakeup_id - 75 : wakeup_id + 75]\n    ax[1].plot(single_wakeup['step'], single_wakeup['enmo'], color='magenta', lw=1, alpha=0.75)\n    ax[1].axvline(train_merged.iloc[wakeup_id]['step'], ls='--', color='Red', alpha=0.9, label='wakeup')\n    ax[1].axvspan(train_merged.iloc[wakeup_id - 75]['step'], train_merged.iloc[wakeup_id]['step'], ymin=0, ymax=1,\n                  color='coral', linewidth=0.1, alpha=0.1, label='asleep')\n    ax[1].legend(loc='upper left', fontsize=10)\n    ax[1].set_xlabel('step')\n    ax[1].set_ylabel('enmo')\n    ax1b = ax[1].twinx()\n    ax1b.plot(single_wakeup['step'], single_wakeup['anglez'], color='lime', lw=1, alpha=0.75) \n    ax1b.set_ylabel('anglez')\n    ax1b.legend(loc='best', fontsize=10)\n    ##########################\n    plt.show()\n    \n# Selector box handlenr function \ndef series_selector_2_eventhandler(change):\n    series_id = series_selector_2.value\n    night = night_selector.value\n    output_2.clear_output(wait=True)\n    with output_2:\n        plot_event_pair(series_id, night, train_merged)   \n        \ndef night_selector_eventhandler(change):\n    series_id = series_selector_2.value\n    night = night_selector.value\n    output_2.clear_output(wait=True)\n    with output_2:\n        plot_event_pair(series_id, night, train_merged)  \n\n####################################################\n# Build and display widget\nseries_selector_2 = widgets.Select(description='Series_id', value=selected_series[0], options=selected_series, rows=8)\nseries_selector_2.observe(series_selector_2_eventhandler, names='value')\n\nnights = 25\n\nnight_selector = widgets.Dropdown(description='Night', value=1, options=range(1, nights+1))\nnight_selector.observe(night_selector_eventhandler, names='value')\n\noutput_2 = widgets.Output()\n\ndisplay(series_selector_2, night_selector, output_2)\n####################################################\n\nwith output_2:\n    plot_event_pair(selected_series[0], 1, train_merged)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:38:14.701670Z","iopub.execute_input":"2023-11-21T21:38:14.702428Z","iopub.status.idle":"2023-11-21T21:38:57.308572Z","shell.execute_reply.started":"2023-11-21T21:38:14.702384Z","shell.execute_reply":"2023-11-21T21:38:57.307183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Creation of target column 'asleep' and visualization against accelerometer readings (predictors) and annotated events\n---\n\nThe charts show two training sequences, one for an onset event and one for a wakeup, selected randomly from the full sample.\n\nEach sequence (or event sample) consists of 150 timesteps centered around the annotated event.","metadata":{}},{"cell_type":"code","source":"onset_records = list(train_merged[train_merged['event']=='onset'].index)\nonset_samples = {}\n\nfor id in onset_records:\n    onset_samples[id] = train_merged.iloc[id - 75 : id + 75].copy()\n    onset_samples[id]['asleep'] = 0\n    onset_samples[id]['asleep'] = onset_samples[id]['event'].replace({\"onset\":1})\n    onset_samples[id]['asleep'] = onset_samples[id]['asleep'].ffill(axis ='rows')\n    onset_samples[id]['asleep'].fillna(value=0, inplace=True)\n    \nnbr_onset_samples = len(onset_samples.keys())\n# print(onset_records)\nprint(\"Nbr. onset samples for training: \" + str(nbr_onset_samples))","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:38:57.310612Z","iopub.execute_input":"2023-11-21T21:38:57.311746Z","iopub.status.idle":"2023-11-21T21:39:05.645351Z","shell.execute_reply.started":"2023-11-21T21:38:57.311696Z","shell.execute_reply":"2023-11-21T21:39:05.644010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Pick one onset_id randomly\nonset_id = random.choice(onset_records)\n# Plot enmo, anglez, event and asleep\nfig, ax = plt.subplots(2,1, figsize=(16, 5), sharex=True)\nfig.suptitle('Single onset', fontsize=13, y=0.95)\nax[0].plot(onset_samples[onset_id].step, onset_samples[onset_id].enmo, label='enmo', color='magenta', alpha=0.9)\nax[0].set_ylabel('enmo')\n#####################\nax0b = ax[0].twinx()\nax0b.plot(onset_samples[onset_id].step, onset_samples[onset_id].anglez, label='anglez', color='lime', alpha=0.9)\nax0b.set_ylabel('anglez')\n#####################\nax[0].axvline(train_merged.iloc[onset_id]['step'], label='onset', ls='--', lw=2, color='Navy', alpha=0.9) \n#####################\nax[1].bar(onset_samples[onset_id].step, onset_samples[onset_id].asleep, label='asleep', alpha=0.75)\nax[1].set_ylabel('asleep = 1')\nax[1].set_xlabel('step')\n#####################\nfig.legend(loc='center left', fontsize=11)\nplt.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:05.647301Z","iopub.execute_input":"2023-11-21T21:39:05.648736Z","iopub.status.idle":"2023-11-21T21:39:06.643780Z","shell.execute_reply.started":"2023-11-21T21:39:05.648688Z","shell.execute_reply":"2023-11-21T21:39:06.642347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wakeup_records = list(train_merged[train_merged['event']=='wakeup'].index)\nwakeup_samples = {}\n\nfor id in wakeup_records:\n    wakeup_samples[id] = train_merged.iloc[id - 75 : id + 75].copy()\n    wakeup_samples[id]['asleep'] = 0\n    wakeup_samples[id]['asleep'] = wakeup_samples[id]['event'].replace({\"wakeup\":1})\n    wakeup_samples[id]['asleep'] = wakeup_samples[id]['asleep'].bfill(axis ='rows')\n    wakeup_samples[id]['asleep'].fillna(value=0, inplace=True)\n\nnbr_wakeup_samples = len(wakeup_samples.keys())\n# print(wakeup_samples)\nprint(\"Nbr. wakeup samples for training: \" + str(nbr_wakeup_samples))","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:06.645167Z","iopub.execute_input":"2023-11-21T21:39:06.645499Z","iopub.status.idle":"2023-11-21T21:39:15.715264Z","shell.execute_reply.started":"2023-11-21T21:39:06.645469Z","shell.execute_reply":"2023-11-21T21:39:15.713769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Pick one wakeup_id randomly\nwakeup_id = random.choice(wakeup_records)\n# Plot enmo, anglez, event and asleep\nfig, ax = plt.subplots(2,1, figsize=(16, 5), sharex=True)\nfig.suptitle('Single wakeup', fontsize=13, y=0.95)\nax[0].plot(wakeup_samples[wakeup_id].step, wakeup_samples[wakeup_id].enmo, label='enmo', color='magenta', alpha=0.9)\nax[0].set_ylabel('enmo')\n#####################\nax0b = ax[0].twinx()\nax0b.plot(wakeup_samples[wakeup_id].step, wakeup_samples[wakeup_id].anglez, label='anglez', color='lime', alpha=0.9)\nax0b.set_ylabel('anglez')\n#####################\nax[0].axvline(train_merged.iloc[wakeup_id]['step'], label='wakeup', ls='--',  lw=2, color='Red', alpha=0.9) \n#####################\nax[1].bar(wakeup_samples[wakeup_id].step, wakeup_samples[wakeup_id].asleep, label='asleep', alpha=0.75)\nax[1].set_ylabel('asleep = 1')\nax[1].set_xlabel('step')\n#####################\nfig.legend(loc='center left', fontsize=11)\nplt.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:15.717350Z","iopub.execute_input":"2023-11-21T21:39:15.717807Z","iopub.status.idle":"2023-11-21T21:39:16.758698Z","shell.execute_reply.started":"2023-11-21T21:39:15.717771Z","shell.execute_reply":"2023-11-21T21:39:16.757571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# build dataframes of onsets and wakeups from samples dictionaries\nonset_samples_df = pd.concat(onset_samples, axis=0)\nwakeup_samples_df = pd.concat(wakeup_samples, axis=0)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:16.760016Z","iopub.execute_input":"2023-11-21T21:39:16.760400Z","iopub.status.idle":"2023-11-21T21:39:20.225777Z","shell.execute_reply.started":"2023-11-21T21:39:16.760363Z","shell.execute_reply":"2023-11-21T21:39:20.224132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have two dataframes (one for onsets and another for wakeups) with a number of sequences of 150 steps centered around the specific event annotation. These will be the sequences fitted to the neural network for training.","metadata":{}},{"cell_type":"markdown","source":"### Creation of derived features and preparation of data for RNN training\n---\n\nHere I will create new features with the rolling averages (1 minute window) of enmo and anglez, and with their first derivative.\n\nThen I will extract X and y and convert them into tensor flow format.\n\nNote that given the huge size of the training dataset and the relative independence of the the different samples (surely different), I am using the full dataset for training and evaluation (in-sample testing) instead of adding additional complexity with held-out data.","metadata":{}},{"cell_type":"code","source":"df1 = onset_samples_df[['series_id', 'asleep', 'anglez', 'enmo']]\ndf2 = wakeup_samples_df[['series_id', 'asleep', 'anglez', 'enmo']]\n\ndf1['enmo_diff'] = onset_samples_df['enmo'].diff(1)\ndf1['anglez_diff'] = onset_samples_df['anglez'].diff(1)\n\ndf1['enmo_ma'] = onset_samples_df['enmo'].rolling(12).mean() # 12 steps = 1 minute\ndf1['anglez_ma'] = onset_samples_df['anglez'].rolling(12).mean()\n\ndf2['enmo_diff'] = wakeup_samples_df['enmo'].diff(1)\ndf2['anglez_diff'] = wakeup_samples_df['anglez'].diff(1)\n\ndf2['enmo_ma'] = wakeup_samples_df['enmo'].rolling(12).mean()\ndf2['anglez_ma'] = wakeup_samples_df['anglez'].rolling(12).mean()\n\ndf = pd.concat([df1, df2], axis=0)\n\ndf = df.fillna(method='ffill').fillna(method='bfill')\n \n# Each sample iss a 150 steps sequence with a single onset or wakeup event\nNBR_TRAINING_SAMPLES = nbr_onset_samples + nbr_wakeup_samples\nprint(\"Nbr. samples for training: \" + str(NBR_TRAINING_SAMPLES))","metadata":{"execution":{"iopub.status.busy":"2023-11-21T21:39:20.227757Z","iopub.execute_input":"2023-11-21T21:39:20.228144Z","iopub.status.idle":"2023-11-21T21:39:21.303117Z","shell.execute_reply.started":"2023-11-21T21:39:20.228111Z","shell.execute_reply":"2023-11-21T21:39:21.301612Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TIMESTEPS = 150\nfeatures = ['anglez', 'enmo', 'anglez_diff', 'enmo_diff',  'anglez_ma', 'enmo_ma']\nNBR_FEATURES = len(features)\n\nX = df[features].to_numpy().reshape(NBR_TRAINING_SAMPLES, TIMESTEPS, NBR_FEATURES)\ny = df['asleep'].to_numpy().reshape(NBR_TRAINING_SAMPLES, TIMESTEPS)\n\nprint('X shape:' + str(X.shape))\nprint('y shape:' + str(y.shape))","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:21.304549Z","iopub.execute_input":"2023-11-21T21:39:21.304901Z","iopub.status.idle":"2023-11-21T21:39:21.343118Z","shell.execute_reply.started":"2023-11-21T21:39:21.304871Z","shell.execute_reply":"2023-11-21T21:39:21.341959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = X\ny_train = y","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:21.344914Z","iopub.execute_input":"2023-11-21T21:39:21.345276Z","iopub.status.idle":"2023-11-21T21:39:21.350422Z","shell.execute_reply.started":"2023-11-21T21:39:21.345246Z","shell.execute_reply":"2023-11-21T21:39:21.349328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Neural network model\n---\n\nMy proposed neural network model includes a 1D convolutional layer followed by a bidirectional LSTM layer or two, along with the input and output (dense) layers.\n\nFor each test sample, the model predicts the probability of each timestep belonging to the positive class (that is, the probability of the asleep state).\n\nPretty simple eh??? Well, may be not so much so!","metadata":{}},{"cell_type":"code","source":"#####################################################\n# RNN training parameters\nBATCH_SIZE = 32\nEPOCHS = 40\n#####################################################\n\n# Input layer\ninput_layer = Input(shape=(TIMESTEPS, NBR_FEATURES)) \n\n# Convolutional layer\nconv_layer = Conv1D(filters=32, \n                    kernel_size=8,\n                    strides=1,\n                    activation='relu',\n                    padding='same')(input_layer)\n\n# Bidirectional LSTM layer\nlstm_layer = Bidirectional(LSTM(TIMESTEPS, return_sequences=True))(conv_layer)\n\n# Bidirectional LSTM layer\n# lstm_layer_2 = Bidirectional(LSTM(TIMESTEPS, return_sequences=True))(lstm_layer)\n\n# Output layer\noutput_layer = Dense(1, activation='sigmoid')(lstm_layer)\n\n# Build the RNN model\nRNN = Model(inputs=input_layer, outputs=output_layer)\n\nRNN.summary()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:21.352141Z","iopub.execute_input":"2023-11-21T21:39:21.352570Z","iopub.status.idle":"2023-11-21T21:39:22.124038Z","shell.execute_reply.started":"2023-11-21T21:39:21.352535Z","shell.execute_reply":"2023-11-21T21:39:22.122713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print()\nprint('TRAINING DATASET DIMENSIONS')\nprint('-'*80)\nprint('  NUMBER OF TRAINING SERIES : ' + str(NBR_TRAINING_SERIES))\nprint('  NUMBER OF TRAINING SAMPLES : ' + str(NBR_TRAINING_SAMPLES))\nprint('  TIMESTEPS PER SAMPLE : ' + str(TIMESTEPS))\nprint('  NUMBER OF FEATURES : ' + str(NBR_FEATURES ))\nprint()\nprint('NEURAL NETWORK TRAINING PARAMETERS')\nprint('-'*80)\nprint('  BATCH SIZE : ' + str(BATCH_SIZE))\nprint('  NUMBER OF EPOCHS : ' + str(EPOCHS))\nprint()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:22.126863Z","iopub.execute_input":"2023-11-21T21:39:22.127400Z","iopub.status.idle":"2023-11-21T21:39:22.137313Z","shell.execute_reply.started":"2023-11-21T21:39:22.127354Z","shell.execute_reply":"2023-11-21T21:39:22.135872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_model(RNN, \"deleteme.png\", show_shapes=True)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:22.139801Z","iopub.execute_input":"2023-11-21T21:39:22.140196Z","iopub.status.idle":"2023-11-21T21:39:22.279332Z","shell.execute_reply.started":"2023-11-21T21:39:22.140165Z","shell.execute_reply":"2023-11-21T21:39:22.277975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Compile model\nRNN.compile(optimizer=Adamax(learning_rate=1e-3),\n            loss=BinaryCrossentropy())\n\n# Define EarlyStopping callback\ncallback = EarlyStopping(monitor='loss', patience=3)\n\n# Fit the model with the data\nRNN_history = RNN.fit(X_train, y_train, \n                      batch_size=BATCH_SIZE, \n                      epochs=EPOCHS, \n                      callbacks=[callback])","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T21:39:22.281079Z","iopub.execute_input":"2023-11-21T21:39:22.281492Z","iopub.status.idle":"2023-11-21T22:26:48.017148Z","shell.execute_reply.started":"2023-11-21T21:39:22.281441Z","shell.execute_reply":"2023-11-21T22:26:48.015643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Convert the training history to a dataframe\nRNN_history_df = pd.DataFrame(RNN_history.history)\n\n# Plot RNN training loss vs. epochs\nRNN_history_df['loss'].plot(figsize=(8,4), title='Neural network training loss', color='brown');","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:26:48.029975Z","iopub.execute_input":"2023-11-21T22:26:48.030514Z","iopub.status.idle":"2023-11-21T22:26:48.459572Z","shell.execute_reply.started":"2023-11-21T22:26:48.030477Z","shell.execute_reply":"2023-11-21T22:26:48.457847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make predictions and compare to true values\nX_test = X_train\ny_test = y_train\npredicted_values = RNN.predict(X_test)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:26:48.461916Z","iopub.execute_input":"2023-11-21T22:26:48.462474Z","iopub.status.idle":"2023-11-21T22:27:30.599288Z","shell.execute_reply.started":"2023-11-21T22:26:48.462404Z","shell.execute_reply":"2023-11-21T22:27:30.597926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The predicted values are the probabilities of a step belonging to the asleep state. ","metadata":{}},{"cell_type":"code","source":"# Onset predictions\nfig, ax = plt.subplots(1,1, figsize=(10,4))\nfor i in range(15):\n    ax.set_title('Onset events (target) and asleep probability predictions')\n    ax.plot(y_test[i], linestyle='--', color='Navy')\n    ax.plot(predicted_values[i], lw=1)\n    ax.set_xlabel('step')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-21T22:28:11.729743Z","iopub.execute_input":"2023-11-21T22:28:11.730384Z","iopub.status.idle":"2023-11-21T22:28:12.117956Z","shell.execute_reply.started":"2023-11-21T22:28:11.730349Z","shell.execute_reply":"2023-11-21T22:28:12.116743Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Wakeup predictions\nfig, ax = plt.subplots(1,1, figsize=(10,4))\nfor i in range(15):\n    ax.set_title('Wakeup events (target) and asleep probability predictions')\n    ax.plot(y_test[i + nbr_onset_samples], linestyle='--', color='Red')\n    ax.plot(predicted_values[i + nbr_onset_samples], lw=1)\n    ax.set_xlabel('step')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-21T22:28:20.346428Z","iopub.execute_input":"2023-11-21T22:28:20.347130Z","iopub.status.idle":"2023-11-21T22:28:20.705589Z","shell.execute_reply.started":"2023-11-21T22:28:20.347097Z","shell.execute_reply":"2023-11-21T22:28:20.704335Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems to be predicting the asleep state reasonably well, even taking into account the testing is done over the same samples the model was trained with (in-sample test), and also that the train and test sequencies are just 150 steps (12.5 minutes). But it does not look bad for a start.","metadata":{}},{"cell_type":"code","source":"predicted_values.shape","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:31.414182Z","iopub.execute_input":"2023-11-21T22:27:31.414638Z","iopub.status.idle":"2023-11-21T22:27:31.423536Z","shell.execute_reply.started":"2023-11-21T22:27:31.414599Z","shell.execute_reply":"2023-11-21T22:27:31.422107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert predicted asleep probabilities to binary state\npredicted_asleep_state = np.where(predicted_values > 0.5, 1, 0 )\npredicted_asleep_state.T","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:31.425031Z","iopub.execute_input":"2023-11-21T22:27:31.426056Z","iopub.status.idle":"2023-11-21T22:27:31.444280Z","shell.execute_reply.started":"2023-11-21T22:27:31.426018Z","shell.execute_reply":"2023-11-21T22:27:31.442747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_asleep_state.shape","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:31.446904Z","iopub.execute_input":"2023-11-21T22:27:31.447588Z","iopub.status.idle":"2023-11-21T22:27:31.455949Z","shell.execute_reply.started":"2023-11-21T22:27:31.447546Z","shell.execute_reply":"2023-11-21T22:27:31.454985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only left is to identify the step at which the wakeup event occurs. To do this, I will calculate the diff of the asleep column.","metadata":{}},{"cell_type":"code","source":"predictions_df = pd.DataFrame(index=df.index)\npredictions_df['asleep'] = predicted_asleep_state.reshape(NBR_TRAINING_SAMPLES*TIMESTEPS,1)\npredictions_df['asleep_diff'] = predictions_df['asleep'].diff(1).fillna(method='bfill')\npredictions_df['event'] = predictions_df['asleep_diff'].replace({1 : 'onset', -1 : 'wakeup'})\n\npredictions_df","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:31.457150Z","iopub.execute_input":"2023-11-21T22:27:31.457512Z","iopub.status.idle":"2023-11-21T22:27:31.600193Z","shell.execute_reply.started":"2023-11-21T22:27:31.457482Z","shell.execute_reply":"2023-11-21T22:27:31.598758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the RNN model\nRNN_name = 'RNN_' + str(NBR_TRAINING_SERIES) + '_SERIES_' + str(EPOCHS)+ '_EPOCHS'\nRNN.save(RNN_name)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:31.602748Z","iopub.execute_input":"2023-11-21T22:27:31.603101Z","iopub.status.idle":"2023-11-21T22:27:48.516321Z","shell.execute_reply.started":"2023-11-21T22:27:31.603071Z","shell.execute_reply":"2023-11-21T22:27:48.514791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Recovering memory\ngc.collect()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:48.519596Z","iopub.execute_input":"2023-11-21T22:27:48.520642Z","iopub.status.idle":"2023-11-21T22:27:49.785468Z","shell.execute_reply.started":"2023-11-21T22:27:48.520605Z","shell.execute_reply":"2023-11-21T22:27:49.784194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make predictions for submission\n---\n\nThere is something wrong somwwhere that will result in an \"exception\" error\" if a submission is attempted. Never mind, I fixed it somewhere else.\n\nJust in case you find your self in such situation, the question you have to ask yourself is: *\"What is it that could be so beautifully working in this interactive session, and could however fail in the hidden set evaluation?\"*\n\nBased on my experience, usual suspects are:\n- hardcoded values (remember hidden data is different to the small test_series set provided)\n- addition of probabilities from different predictions (use clip(0,1))\n- manipulating files, changing directory and then saving the submission file in the wrong place (use pwd)\n- ....\n\nJust persevere.... and you will find it.\n","metadata":{}},{"cell_type":"code","source":"test_series","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:49.790500Z","iopub.execute_input":"2023-11-21T22:27:49.791559Z","iopub.status.idle":"2023-11-21T22:27:49.811366Z","shell.execute_reply.started":"2023-11-21T22:27:49.791511Z","shell.execute_reply":"2023-11-21T22:27:49.809834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series['step'] = test_series['step'].astype(np.int64) \ntest_series.info()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:49.813305Z","iopub.execute_input":"2023-11-21T22:27:49.813770Z","iopub.status.idle":"2023-11-21T22:27:49.833768Z","shell.execute_reply.started":"2023-11-21T22:27:49.813736Z","shell.execute_reply":"2023-11-21T22:27:49.832745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series_ids = list(test_series['series_id'].unique())\nNBR_TEST_SERIES = len(test_series_ids)\nTIMESTEPS_PER_SAMPLE = 150\n\nprint('Nbr. test series for submission: ' + str(NBR_TEST_SERIES))\nprint(test_series_ids)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:49.835921Z","iopub.execute_input":"2023-11-21T22:27:49.836494Z","iopub.status.idle":"2023-11-21T22:27:49.847024Z","shell.execute_reply.started":"2023-11-21T22:27:49.836428Z","shell.execute_reply":"2023-11-21T22:27:49.845968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def sample_test_series(df):\n    # Create dictionary of test sequences\n    series = {}\n    # Iterate through the samples\n    for id in test_series_ids:\n        series[id] = df[df['series_id']==id].reset_index(drop=True)\n        n_steps = len(series[id])\n        n_samples = n_steps // TIMESTEPS_PER_SAMPLE\n        l_samples = list(range(n_samples))\n        samples = {}\n        for s in l_samples:\n            first_step = TIMESTEPS_PER_SAMPLE * s\n            samples[s] = series[id].loc[first_step : first_step + TIMESTEPS_PER_SAMPLE - 1]\n    series[id] = pd.concat([samples[s] for s in l_samples], axis='rows')\n\n    # Concatenate samples into single test dataframe\n    df = pd.concat([series[id] for id in test_series_ids], axis='rows').reset_index(drop=True)\n    # Return updated dataframe\n    return df\n        \n# Resample test_series dataset into set of sequences of 150 steps\ntest_series = sample_test_series(test_series)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:49.848776Z","iopub.execute_input":"2023-11-21T22:27:49.849375Z","iopub.status.idle":"2023-11-21T22:27:49.871135Z","shell.execute_reply.started":"2023-11-21T22:27:49.849341Z","shell.execute_reply":"2023-11-21T22:27:49.869679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_features(df):\n    # Create dictionary of test sequences\n    series = {}\n    # Iterate through the samples to create derived features\n    for id in test_series_ids:\n        series[id] = df[df['series_id']==id]\n        series[id]['anglez_diff'] = series[id]['anglez'].diff().fillna(method='bfill').fillna(method=\"ffill\")   \n        series[id]['enmo_diff'] = series[id]['enmo'].diff().fillna(method='bfill').fillna(method=\"ffill\")   \n        series[id]['anglez_ma'] = series[id]['anglez'].rolling(12).mean().fillna(method='bfill').fillna(method=\"ffill\")    \n        series[id]['enmo_ma'] = series[id]['enmo'].rolling(12).mean().fillna(method='bfill').fillna(method=\"ffill\")   \n    # Concatenate samples into single test dataframe\n    df = pd.concat([series[id] for id in test_series_ids], axis=0)\n    # Return updated dataframe\n    return df\n\n# Creature derived features\ntest_series = create_features(test_series)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:49.873179Z","iopub.execute_input":"2023-11-21T22:27:49.873669Z","iopub.status.idle":"2023-11-21T22:27:49.906925Z","shell.execute_reply.started":"2023-11-21T22:27:49.873632Z","shell.execute_reply":"2023-11-21T22:27:49.905597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_predictions(df):\n    # Prepare input sequences for RNN\n    features = ['anglez', 'enmo', 'anglez_diff', 'enmo_diff',  'anglez_ma', 'enmo_ma']\n    NBR_FEATURES = len(features)\n    TEST_SERIES_STEPS = len(df)\n\n    # Isolate test set predictors\n    X_test = df[features].to_numpy().reshape(NBR_TEST_SERIES, TIMESTEPS_PER_SAMPLE, NBR_FEATURES)\n    # Make submission predictions (probabilities of asleep)\n    test_predictions = RNN.predict(X_test)\n    # Make binary asleep predictions (1=asleep)\n    test_classes = np.where(test_predictions > 0.5, 1,0)\n    # Reshape\n    test_predictions = test_predictions.reshape(TEST_SERIES_STEPS, 1)\n    test_classes = test_classes.reshape(TEST_SERIES_STEPS, 1) \n    \n    # Create asleep and event predictions columns\n    df['score'] = test_predictions\n    df ['asleep'] =  test_classes\n    df['asleep_diff'] = df['asleep'].diff().fillna(method='bfill').fillna(method=\"ffill\")\n    df['event'] = df[df['score']>0.4]['asleep_diff'].replace({1 : 'onset', -1 : 'wakeup'})\n    \n    return df\n\n# Make predictions\ntest_series = make_predictions(test_series)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:49.908884Z","iopub.execute_input":"2023-11-21T22:27:49.910352Z","iopub.status.idle":"2023-11-21T22:27:50.048149Z","shell.execute_reply.started":"2023-11-21T22:27:49.910303Z","shell.execute_reply":"2023-11-21T22:27:50.046855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_series.sample(5)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:50.049789Z","iopub.execute_input":"2023-11-21T22:27:50.050169Z","iopub.status.idle":"2023-11-21T22:27:50.079082Z","shell.execute_reply.started":"2023-11-21T22:27:50.050137Z","shell.execute_reply":"2023-11-21T22:27:50.077719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(3,1, figsize=(10,6), sharex=True)\nfig.suptitle('Test predictions', y=0.93)\ni = 0\nfor id in test_series_ids:\n    ax[i].plot(test_series[test_series['series_id']==id]['step'],\n               test_series[test_series['series_id']==id]['asleep'],\n               lw=1.5, label=id)\n    ax[i].legend(loc='lower left')\n    i = i + 1\nplt.xlabel('step')\nplt.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:50.080868Z","iopub.execute_input":"2023-11-21T22:27:50.081299Z","iopub.status.idle":"2023-11-21T22:27:50.768657Z","shell.execute_reply.started":"2023-11-21T22:27:50.081262Z","shell.execute_reply":"2023-11-21T22:27:50.767654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Isolate events and add row_id\nsubmission_df = test_series.loc[((test_series[\"event\"]=='onset') |\n                                 (test_series[\"event\"]=='wakeup'))\n                               ][[\"series_id\",\"step\",\"event\",\"score\"]].copy().reset_index(drop=True).reset_index(names=\"row_id\")\n\nsubmission_df","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:50.770084Z","iopub.execute_input":"2023-11-21T22:27:50.770664Z","iopub.status.idle":"2023-11-21T22:27:50.789515Z","shell.execute_reply.started":"2023-11-21T22:27:50.770630Z","shell.execute_reply":"2023-11-21T22:27:50.788610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.info()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:50.790973Z","iopub.execute_input":"2023-11-21T22:27:50.791549Z","iopub.status.idle":"2023-11-21T22:27:50.804850Z","shell.execute_reply.started":"2023-11-21T22:27:50.791513Z","shell.execute_reply":"2023-11-21T22:27:50.803710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.info()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:50.806682Z","iopub.execute_input":"2023-11-21T22:27:50.807029Z","iopub.status.idle":"2023-11-21T22:27:50.824511Z","shell.execute_reply.started":"2023-11-21T22:27:50.806997Z","shell.execute_reply":"2023-11-21T22:27:50.823630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save submission data to csv file\nsubmission_df.to_csv('submission.csv', index=False)\nprint('finished')","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-21T22:27:50.826494Z","iopub.execute_input":"2023-11-21T22:27:50.827176Z","iopub.status.idle":"2023-11-21T22:27:50.834149Z","shell.execute_reply.started":"2023-11-21T22:27:50.827132Z","shell.execute_reply":"2023-11-21T22:27:50.833275Z"},"trusted":true},"execution_count":null,"outputs":[]}]}