{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Create a smaller \"starter\" datasets + a target column called `awake`\n# ~~**Updated**: with the corrected [Offset Events](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697) (18 September 2023)~~\n# **Updated**: with the corrected [Offset Events](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697) (20 September 2023)\n\n\nIn this competition we are given 277 series. However, for 8 of these series we have absolutely no event data (`onset` or `wakeup` timestamps). Indeed, there are only 37 series of events data for which there are no missing events at all, and two of these seem to be truncated. \n\nWithout the corresponding event data it is particularly hard to untangle the sleeping from the awake data contained in the `train_series` data. With this in mind here we create a smaller training dataset consisting of a selection of just 35 of the series. We also add a binary target column called `awake`.\n\n(For plots of all 277 series see the notebook [\"Zzzs: Creating a \"target\" column + viz of all data\"](https://www.kaggle.com/code/carlmcbrideellis/zzzs-creating-a-target-column-viz-of-all-data))\n\n# `Zzzs_train.parquet`\nBinary dataset (`awake = 1, sleeping = 0`,'onset'=6,'wakeup'=7)","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import Markdown\nimport gc\nimport numpy as np","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-10T11:52:57.320124Z","iopub.execute_input":"2023-10-10T11:52:57.320518Z","iopub.status.idle":"2023-10-10T11:52:59.235310Z","shell.execute_reply.started":"2023-10-10T11:52:57.320486Z","shell.execute_reply":"2023-10-10T11:52:59.234382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.237232Z","iopub.execute_input":"2023-10-10T11:52:59.237750Z","iopub.status.idle":"2023-10-10T11:52:59.280348Z","shell.execute_reply.started":"2023-10-10T11:52:59.237713Z","shell.execute_reply":"2023-10-10T11:52:59.279336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# take a quick look\ntrain_events","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.281402Z","iopub.execute_input":"2023-10-10T11:52:59.282110Z","iopub.status.idle":"2023-10-10T11:52:59.312413Z","shell.execute_reply.started":"2023-10-10T11:52:59.282080Z","shell.execute_reply":"2023-10-10T11:52:59.311249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check that all of the series start off in the `awake` state (*i.e.* the very first `event` is the `onset` of sleep):","metadata":{}},{"cell_type":"code","source":"train_events.groupby('series_id').head(1)[\"event\"].unique()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.314284Z","iopub.execute_input":"2023-10-10T11:52:59.314746Z","iopub.status.idle":"2023-10-10T11:52:59.332053Z","shell.execute_reply.started":"2023-10-10T11:52:59.314719Z","shell.execute_reply":"2023-10-10T11:52:59.330939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check that all of the series end in the `awake` state (*i.e.* the very last `event` is a `wakeup`):","metadata":{}},{"cell_type":"code","source":"train_events.groupby('series_id').tail(1)[\"event\"].unique()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.333342Z","iopub.execute_input":"2023-10-10T11:52:59.333892Z","iopub.status.idle":"2023-10-10T11:52:59.348449Z","shell.execute_reply.started":"2023-10-10T11:52:59.333864Z","shell.execute_reply":"2023-10-10T11:52:59.347328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How many series have `NaN` values?","metadata":{}},{"cell_type":"code","source":"series_has_NaN = train_events.groupby('series_id')['step'].apply(lambda x: x.isnull().any())\nseries_has_NaN.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.349843Z","iopub.execute_input":"2023-10-10T11:52:59.350234Z","iopub.status.idle":"2023-10-10T11:52:59.397857Z","shell.execute_reply.started":"2023-10-10T11:52:59.350197Z","shell.execute_reply":"2023-10-10T11:52:59.396825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see the following 37 series have no `NaN` values whatsoever:","metadata":{}},{"cell_type":"code","source":"no_NaN_series = series_has_NaN[~series_has_NaN].index.tolist()\nno_NaN_series","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:34:31.945915Z","iopub.execute_input":"2023-10-10T13:34:31.946302Z","iopub.status.idle":"2023-10-10T13:34:31.954762Z","shell.execute_reply.started":"2023-10-10T13:34:31.946272Z","shell.execute_reply":"2023-10-10T13:34:31.953425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# also drop these two \"truncated\" events series seen in EDA:\nno_NaN_series.remove('31011ade7c0a') # incomplete events data\nno_NaN_series.remove('a596ad0b82aa') # incomplete events data","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.410014Z","iopub.execute_input":"2023-10-10T11:52:59.410409Z","iopub.status.idle":"2023-10-10T11:52:59.419397Z","shell.execute_reply.started":"2023-10-10T11:52:59.410373Z","shell.execute_reply":"2023-10-10T11:52:59.418548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_series(series):\n    train_series = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", filters=[('series_id','=',series)])\n    train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\").query('series_id == @series')\n    \n    train_events = train_events.dropna()\n    train_events[\"step\"]  = train_events[\"step\"].astype(\"int\")\n    train_events[\"event_interval\"]=train_events[\"event\"].replace({\"onset\":5,\"wakeup\":7})\n    \n    #train_events[\"event_interval\"] = pd.to_numeric(train_events[\"event_interval\"], errors='coerce').fillna(0, inplace=True)\n#     train_events[\"event_interval\"]=train_events[\"event_interval\"].fillna(0, inplace=True)\n    train_events[\"awake\"] = train_events[\"event\"].replace({\"onset\":1,\"wakeup\":0})\n    \n    train = pd.merge(train_series, train_events[['step','awake',\"event_interval\"]], on='step', how='left')\n    train[\"event_interval\"] = pd.to_numeric(train[\"event_interval\"], errors='coerce').fillna(0)\n    train[\"event_interval\"] = train[\"event_interval\"].astype(\"int\")\n    train[\"awake\"] = train[\"awake\"].bfill(axis ='rows')\n    train['awake'] = train['awake'].fillna(1) # awake\n    train[\"awake\"] = train[\"awake\"].astype(\"int\")\n#     train_events[\"event_interval\"] = pd.to_numeric(train_events[\"event_interval\"], errors='coerce').fillna(0)\n#     train_events[\"event_interval\"]=train_events[\"event_interval\"].fillna(0)\n#     train_events[\"event_interval\"]=train_events[\"event_interval\"].astype(\"int\")\n    train.loc[:,[\"event_interval\"]]=train[\"event_interval\"]+train[\"awake\"]\n    onset_list=train.index[(train['event_interval'] >= 3) & (train['event_interval'] <7)].tolist()\n    wakeup_list=train.index[train['event_interval'] >= 7].tolist()\n    for row_index in onset_list:\n        train.iloc[row_index:row_index+120, train.columns.get_loc('event_interval')] = 5\n    for row_index in wakeup_list:\n        train.iloc[row_index:row_index+120, train.columns.get_loc('event_interval')] = 7\n    train[\"awake\"]=train[\"event_interval\"]\n    train.drop('event_interval', axis=1, inplace=True)\n    train['convolve']=np.convolve(train[\"anglez\"].values,train[\"enmo\"].values, 'same')\n    \n    # final section:\n    # train_events.groupby('series_id').tail(1)[\"event\"].unique()\n    # Result: the last event is always a \"wakeup\"\n    return(train , onset_list,wakeup_list)","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.420473Z","iopub.execute_input":"2023-10-10T11:52:59.421417Z","iopub.status.idle":"2023-10-10T11:52:59.433918Z","shell.execute_reply.started":"2023-10-10T11:52:59.421390Z","shell.execute_reply":"2023-10-10T11:52:59.432903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train , onset_list,wakeup_list = get_train_series('08db4255286f')","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:52:59.437660Z","iopub.execute_input":"2023-10-10T11:52:59.437963Z","iopub.status.idle":"2023-10-10T11:53:43.087396Z","shell.execute_reply.started":"2023-10-10T11:52:59.437937Z","shell.execute_reply":"2023-10-10T11:53:43.086379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:53:43.088602Z","iopub.execute_input":"2023-10-10T11:53:43.089197Z","iopub.status.idle":"2023-10-10T11:53:43.107491Z","shell.execute_reply.started":"2023-10-10T11:53:43.089152Z","shell.execute_reply":"2023-10-10T11:53:43.105987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_id='08db4255286f'\ntrain, onset_list,wakeup_list= get_train_series(series_id)\n# onset_list=train.index[(train['awake'] >= 3) & (train['awake'] <7)].tolist()\n# wakeup_list=train.index[train['awake'] >= 7].tolist()\nsns.color_palette(\"husl\", 9)\nfor something in range(5):\n#     print(something)\n#     print(onset_list)\n#     print(onset_list[something])\n#     print(wakeup_list[something])\n    train_xp=train[onset_list[something]-100:wakeup_list[something]+100]\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train_xp, x=\"step\", y=\"anglez\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train_xp, x=\"step\", y=\"enmo\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  convolve for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train_xp, x=\"step\", y=\"convolve\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    del train_xp\n    gc.collect();","metadata":{"execution":{"iopub.status.busy":"2023-10-10T11:53:43.108785Z","iopub.execute_input":"2023-10-10T11:53:43.109159Z","iopub.status.idle":"2023-10-10T11:54:25.940357Z","shell.execute_reply.started":"2023-10-10T11:53:43.109129Z","shell.execute_reply":"2023-10-10T11:54:25.939396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"def get_train_series_compressed(series):\n    train_series = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", filters=[('series_id','=',series)])\n    train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\").query('series_id == @series')\n    \n    train_events = train_events.dropna()\n    train_events[\"step\"]  = train_events[\"step\"].astype(\"int\")\n    train_events[\"event_interval\"]=train_events[\"event\"].replace({\"onset\":5,\"wakeup\":7})\n    \n    #train_events[\"event_interval\"] = pd.to_numeric(train_events[\"event_interval\"], errors='coerce').fillna(0, inplace=True)\n#     train_events[\"event_interval\"]=train_events[\"event_interval\"].fillna(0, inplace=True)\n    train_events[\"awake\"] = train_events[\"event\"].replace({\"onset\":1,\"wakeup\":0})\n    \n    train = pd.merge(train_series, train_events[['step','awake',\"event_interval\"]], on='step', how='left')\n    train[\"event_interval\"] = pd.to_numeric(train[\"event_interval\"], errors='coerce').fillna(0)\n    train[\"event_interval\"] = train[\"event_interval\"].astype(\"int\")\n    train[\"awake\"] = train[\"awake\"].bfill(axis ='rows')\n    train['awake'] = train['awake'].fillna(1) # awake\n    train[\"awake\"] = train[\"awake\"].astype(\"int\")\n#     train_events[\"event_interval\"] = pd.to_numeric(train_events[\"event_interval\"], errors='coerce').fillna(0)\n#     train_events[\"event_interval\"]=train_events[\"event_interval\"].fillna(0)\n#     train_events[\"event_interval\"]=train_events[\"event_interval\"].astype(\"int\")\n    train.loc[:,[\"event_interval\"]]=train[\"event_interval\"]+train[\"awake\"]\n    onset_list=train.index[(train['event_interval'] >= 3) & (train['event_interval'] <7)].tolist()\n    wakeup_list=train.index[train['event_interval'] >= 7].tolist()\n    for row_index in onset_list:\n        train.iloc[row_index:row_index+120, train.columns.get_loc('event_interval')] = 5\n    for row_index in wakeup_list:\n        train.iloc[row_index:row_index+120, train.columns.get_loc('event_interval')] = 7\n    train[\"awake\"]=train[\"event_interval\"]\n    train.drop('event_interval', axis=1, inplace=True)\n    train['convolve']=np.convolve(train[\"anglez\"].values,train[\"enmo\"].values, 'same')\n    train[\"anglez_rolling\"]=train[\"anglez\"].rolling(60).median()\n    \n    train[\"enmo_rolling\"]=train[\"enmo\"].rolling(60).median()\n    train[\"anglez_rolling\"]=abs(train[\"anglez_rolling\"]-train[\"anglez\"])\n    train[\"enmo_rolling\"]=abs(train[\"enmo_rolling\"]-train[\"enmo\"])\n    train = train.dropna()\n    compressed=train[(train[\"anglez_rolling\"] >= 0.02) & (train[\"enmo_rolling\"]  >= 0.009)]\n    # final section:\n    # train_events.groupby('series_id').tail(1)[\"event\"].unique()\n    # Result: the last event is always a \"wakeup\"\n    return(compressed , onset_list,wakeup_list)","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:27:56.355021Z","iopub.execute_input":"2023-10-10T13:27:56.355384Z","iopub.status.idle":"2023-10-10T13:27:56.371521Z","shell.execute_reply.started":"2023-10-10T13:27:56.355357Z","shell.execute_reply":"2023-10-10T13:27:56.370196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_id='08db4255286f'\ntrain, onset_list,wakeup_list= get_train_series_compressed(series_id)\n# onset_list=train.index[(train['awake'] >= 3) & (train['awake'] <7)].tolist()\n# wakeup_list=train.index[train['awake'] >= 7].tolist()\nsns.color_palette(\"husl\", 9)\nfor something in range(5):\n#     print(something)\n#     print(onset_list)\n    print(onset_list[something])\n    print(wakeup_list[something])\n    train_xp=train[onset_list[something]-100:wakeup_list[something]+100]\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train_xp, x=\"step\", y=\"anglez\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train_xp, x=\"step\", y=\"enmo\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  convolve for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train_xp, x=\"step\", y=\"convolve\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    del train_xp\n    gc.collect();","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:27:57.253622Z","iopub.execute_input":"2023-10-10T13:27:57.254438Z","iopub.status.idle":"2023-10-10T13:28:44.447539Z","shell.execute_reply.started":"2023-10-10T13:27:57.254399Z","shell.execute_reply":"2023-10-10T13:28:44.446377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"https://hal.science/hal-01917529/document\n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC6354999/\n\nhttps://www.nature.com/articles/s41598-021-86432-7\n\nhttps://www.tejwin.com/en/insight/lstm-trading-signal-detection/\n\nhttps://www.kaggle.com/code/afajohn/cnn-lstm-for-signal-classification-lb-0-513","metadata":{}},{"cell_type":"code","source":"from scipy import signal\nfrom scipy.fft import fftshift","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:13.225875Z","iopub.execute_input":"2023-10-10T13:32:13.227150Z","iopub.status.idle":"2023-10-10T13:32:13.298065Z","shell.execute_reply.started":"2023-10-10T13:32:13.227101Z","shell.execute_reply":"2023-10-10T13:32:13.296836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fs=10e-33\ntrain_xp=train[onset_list[0]-255:onset_list[0]+255]\ntrain_xp=train_xp['enmo']\nf, t, Sxx = signal.spectrogram(train_xp, fs)\nplt.pcolormesh(t, f, Sxx, shading='gouraud')\nplt.ylabel('Frequency [Hz]')\nplt.xlabel('Time [sec]')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:13.540022Z","iopub.execute_input":"2023-10-10T13:32:13.540407Z","iopub.status.idle":"2023-10-10T13:32:13.803655Z","shell.execute_reply.started":"2023-10-10T13:32:13.540371Z","shell.execute_reply":"2023-10-10T13:32:13.802941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fs=10e-35\ntrain_xp=train[onset_list[0]-600:onset_list[0]+600]\ntrain_xp=train_xp['anglez']\nf, t, Sxx = signal.spectrogram(train_xp, fs)\nplt.pcolormesh(t, f, Sxx, shading='gouraud')\nplt.ylabel('Frequency [Hz]')\nplt.xlabel('Time [sec]')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:13.805002Z","iopub.execute_input":"2023-10-10T13:32:13.805686Z","iopub.status.idle":"2023-10-10T13:32:14.085213Z","shell.execute_reply.started":"2023-10-10T13:32:13.805656Z","shell.execute_reply":"2023-10-10T13:32:14.084066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Compressing**","metadata":{}},{"cell_type":"markdown","source":"imbalanced","metadata":{"execution":{"iopub.status.busy":"2023-10-09T19:27:10.225481Z","iopub.execute_input":"2023-10-09T19:27:10.225982Z","iopub.status.idle":"2023-10-09T19:27:10.240710Z","shell.execute_reply.started":"2023-10-09T19:27:10.225938Z","shell.execute_reply":"2023-10-09T19:27:10.239859Z"}}},{"cell_type":"code","source":"# train.event_interval=train.event_interval[onset_list[0]:onset_list[0]+60]=6","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:14.284815Z","iopub.execute_input":"2023-10-10T13:32:14.285819Z","iopub.status.idle":"2023-10-10T13:32:14.289580Z","shell.execute_reply.started":"2023-10-10T13:32:14.285783Z","shell.execute_reply":"2023-10-10T13:32:14.288901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train.iloc[onset_list[0]:onset_list[0]+60, train.columns.get_loc('event_interval')] = 999","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:14.393737Z","iopub.execute_input":"2023-10-10T13:32:14.394456Z","iopub.status.idle":"2023-10-10T13:32:14.398093Z","shell.execute_reply.started":"2023-10-10T13:32:14.394424Z","shell.execute_reply":"2023-10-10T13:32:14.397268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train.event_interval.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:14.618850Z","iopub.execute_input":"2023-10-10T13:32:14.620037Z","iopub.status.idle":"2023-10-10T13:32:14.624649Z","shell.execute_reply.started":"2023-10-10T13:32:14.619962Z","shell.execute_reply":"2023-10-10T13:32:14.623617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 313872/3000","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:14.834796Z","iopub.execute_input":"2023-10-10T13:32:14.835181Z","iopub.status.idle":"2023-10-10T13:32:14.839731Z","shell.execute_reply.started":"2023-10-10T13:32:14.835149Z","shell.execute_reply":"2023-10-10T13:32:14.838553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # train.index[[train['event_interval'] <7 & [train['event_interval']>2]]].tolist()\n# # train.loc[([train['event_interval'] >= 3) & ([train['event_interval'] = 7)].tolist()\n# onset_list=train.index[(train['event_interval'] >= 3) & (train['event_interval'] <7)].tolist()\n# wakeup_list=train.index[train['event_interval'] >= 7].tolist()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:14.993761Z","iopub.execute_input":"2023-10-10T13:32:14.994169Z","iopub.status.idle":"2023-10-10T13:32:14.998365Z","shell.execute_reply.started":"2023-10-10T13:32:14.994136Z","shell.execute_reply":"2023-10-10T13:32:14.997108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# len(onset_list)==len(wakeup_list)","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:15.144481Z","iopub.execute_input":"2023-10-10T13:32:15.145390Z","iopub.status.idle":"2023-10-10T13:32:15.149620Z","shell.execute_reply.started":"2023-10-10T13:32:15.145352Z","shell.execute_reply":"2023-10-10T13:32:15.148661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"smaller_train_data = []\n\nfor series_id in no_NaN_series:\n    train, onset_list,wakeup_list = get_train_series(series_id)\n    smaller_train_data.append(train)\n    \n    # vizualize these series\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"anglez\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"enmo\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n\n    del train\n    gc.collect(); ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"smaller_train_data_compressed = []\n\nfor series_id in no_NaN_series:\n    train, onset_list,wakeup_list = get_train_series(series_id)\n    smaller_train_data_compressed .append(train)\n    \n    # vizualize these series\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"anglez\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"enmo\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n\n    del train\n    gc.collect();","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:48:31.336516Z","iopub.execute_input":"2023-10-10T13:48:31.336879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train = pd.concat(smaller_train_data).reset_index(drop=True)\nZzzs_train[\"series_id\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:55.141969Z","iopub.status.idle":"2023-10-10T13:32:55.142515Z","shell.execute_reply.started":"2023-10-10T13:32:55.142261Z","shell.execute_reply":"2023-10-10T13:32:55.142288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train_compressed = pd.concat(smaller_train_data_compressed).reset_index(drop=True)\nZzzs_train[\"series_id\"].nunique()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:55.144115Z","iopub.status.idle":"2023-10-10T13:32:55.144668Z","shell.execute_reply.started":"2023-10-10T13:32:55.144413Z","shell.execute_reply":"2023-10-10T13:32:55.144439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Write out the new lightweight dataset with these 35 selected series","metadata":{}},{"cell_type":"code","source":"Zzzs_train.to_parquet('Zzzs_train.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:55.147084Z","iopub.status.idle":"2023-10-10T13:32:55.147493Z","shell.execute_reply.started":"2023-10-10T13:32:55.147293Z","shell.execute_reply":"2023-10-10T13:32:55.147312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train_compressed.to_parquet('Zzzs_train_compressed.parquet')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# `Zzzs_train_multi.parquet`\n## Now also add on the 8 datasets that have no event data whatsoever (`class=2`)","metadata":{}},{"cell_type":"code","source":"train_series_names = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", columns=['series_id'])\nseries_id_list = train_series_names['series_id'].unique().tolist()\n\ntrain_events = train_events.dropna()\nevents_series_id_list = train_events['series_id'].unique().tolist()\n\nbase_series = list(set(series_id_list) - set(events_series_id_list))\n\n# list out the series:\nbase_series","metadata":{"execution":{"iopub.status.busy":"2023-10-10T13:32:55.149115Z","iopub.status.idle":"2023-10-10T13:32:55.149513Z","shell.execute_reply.started":"2023-10-10T13:32:55.149329Z","shell.execute_reply":"2023-10-10T13:32:55.149348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"add the target `awake=2`","metadata":{}},{"cell_type":"code","source":"def get_train_series_base(series):\n    train = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", filters=[('series_id','=',series)])\n    train[\"awake\"] = 2\n    # just to make sure\n    train[\"awake\"] = train[\"awake\"].astype(\"int\")\n    return(train)","metadata":{"execution":{"iopub.status.busy":"2023-10-09T20:43:19.738289Z","iopub.status.idle":"2023-10-09T20:43:19.738838Z","shell.execute_reply.started":"2023-10-09T20:43:19.738626Z","shell.execute_reply":"2023-10-09T20:43:19.738654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_train_data = []\n\nfor series_id in base_series:\n    train= get_train_series_base(series_id)\n    base_train_data.append(train)\n    \n    # vizualize these series\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"anglez\", color=\"purple\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"enmo\", color=\"olive\", linewidth = 0.5)\n    plt.show();\n    del train\n    gc.collect();","metadata":{"execution":{"iopub.status.busy":"2023-10-09T20:43:19.740590Z","iopub.status.idle":"2023-10-09T20:43:19.740978Z","shell.execute_reply.started":"2023-10-09T20:43:19.740786Z","shell.execute_reply":"2023-10-09T20:43:19.740804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_base = pd.concat(base_train_data).reset_index(drop=True)\n# combine the 35 series from before with the 8 series that have no event data\nZzzs_train_multi = pd.concat([Zzzs_train,Zzzs_base]).reset_index(drop=True)\n# check number of series\nZzzs_train_multi[\"series_id\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-10-09T20:43:19.745641Z","iopub.status.idle":"2023-10-09T20:43:19.746066Z","shell.execute_reply.started":"2023-10-09T20:43:19.745880Z","shell.execute_reply":"2023-10-09T20:43:19.745898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train_multi.to_parquet('Zzzs_train_multi.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-10-09T20:43:19.747829Z","iopub.status.idle":"2023-10-09T20:43:19.748537Z","shell.execute_reply.started":"2023-10-09T20:43:19.748295Z","shell.execute_reply":"2023-10-09T20:43:19.748315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The datasets:\nThe two datasets have been saved to the Kaggle dataset: [\"Zzzs: Lightweight training dataset + target\"](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target)\n\n* `Zzzs_train.parquet` 180M\n* `Zzzs_train_multi.parquet` 220M\n\nI hope they are useful to get started with!","metadata":{}}]}