{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Create a smaller \"starter\" datasets + a target column called `awake`\n# ~~**Updated**: with the corrected [Offset Events](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697) (18 September 2023)~~\n# **Updated**: with the corrected [Offset Events](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/440697) (20 September 2023)\n\n\nIn this competition we are given 277 series. However, for 8 of these series we have absolutely no event data (`onset` or `wakeup` timestamps). Indeed, there are only 37 series of events data for which there are no missing events at all, and two of these seem to be truncated. \n\nWithout the corresponding event data it is particularly hard to untangle the sleeping from the awake data contained in the `train_series` data. With this in mind here we create a smaller training dataset consisting of a selection of just 35 of the series. We also add a binary target column called `awake`.\n\n(For plots of all 277 series see the notebook [\"Zzzs: Creating a \"target\" column + viz of all data\"](https://www.kaggle.com/code/carlmcbrideellis/zzzs-creating-a-target-column-viz-of-all-data))\n\n# `Zzzs_train.parquet`\nBinary dataset (`awake = 1, sleeping = 0`)","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import Markdown\nimport gc","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-18T17:49:30.962936Z","iopub.execute_input":"2023-09-18T17:49:30.963359Z","iopub.status.idle":"2023-09-18T17:49:30.970495Z","shell.execute_reply.started":"2023-09-18T17:49:30.963327Z","shell.execute_reply":"2023-09-18T17:49:30.969235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:30.976740Z","iopub.execute_input":"2023-09-18T17:49:30.977888Z","iopub.status.idle":"2023-09-18T17:49:31.007753Z","shell.execute_reply.started":"2023-09-18T17:49:30.977851Z","shell.execute_reply":"2023-09-18T17:49:31.006297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# take a quick look\ntrain_events","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.009923Z","iopub.execute_input":"2023-09-18T17:49:31.010298Z","iopub.status.idle":"2023-09-18T17:49:31.029997Z","shell.execute_reply.started":"2023-09-18T17:49:31.010266Z","shell.execute_reply":"2023-09-18T17:49:31.028678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check that all of the series start off in the `awake` state (*i.e.* the very first `event` is the `onset` of sleep):","metadata":{}},{"cell_type":"code","source":"train_events.groupby('series_id').head(1)[\"event\"].unique()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.031692Z","iopub.execute_input":"2023-09-18T17:49:31.032149Z","iopub.status.idle":"2023-09-18T17:49:31.050075Z","shell.execute_reply.started":"2023-09-18T17:49:31.032115Z","shell.execute_reply":"2023-09-18T17:49:31.048734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check that all of the series end in the `awake` state (*i.e.* the very last `event` is a `wakeup`):","metadata":{}},{"cell_type":"code","source":"train_events.groupby('series_id').tail(1)[\"event\"].unique()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.054859Z","iopub.execute_input":"2023-09-18T17:49:31.055766Z","iopub.status.idle":"2023-09-18T17:49:31.067966Z","shell.execute_reply.started":"2023-09-18T17:49:31.055729Z","shell.execute_reply":"2023-09-18T17:49:31.066513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How many series have `NaN` values?","metadata":{}},{"cell_type":"code","source":"series_has_NaN = train_events.groupby('series_id')['step'].apply(lambda x: x.isnull().any())\nseries_has_NaN.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.069941Z","iopub.execute_input":"2023-09-18T17:49:31.070380Z","iopub.status.idle":"2023-09-18T17:49:31.126302Z","shell.execute_reply.started":"2023-09-18T17:49:31.070339Z","shell.execute_reply":"2023-09-18T17:49:31.125198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see the following 37 series have no `NaN` values whatsoever:","metadata":{}},{"cell_type":"code","source":"no_NaN_series = series_has_NaN[~series_has_NaN].index.tolist()\nno_NaN_series","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.127808Z","iopub.execute_input":"2023-09-18T17:49:31.128156Z","iopub.status.idle":"2023-09-18T17:49:31.137051Z","shell.execute_reply.started":"2023-09-18T17:49:31.128125Z","shell.execute_reply":"2023-09-18T17:49:31.135822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# also drop these two \"truncated\" events series seen in EDA:\nno_NaN_series.remove('31011ade7c0a') # incomplete events data\nno_NaN_series.remove('a596ad0b82aa') # incomplete events data","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.138681Z","iopub.execute_input":"2023-09-18T17:49:31.139468Z","iopub.status.idle":"2023-09-18T17:49:31.149604Z","shell.execute_reply.started":"2023-09-18T17:49:31.139431Z","shell.execute_reply":"2023-09-18T17:49:31.148215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_series(series):\n    train_series = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", filters=[('series_id','=',series)])\n    train_events = pd.read_csv(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\").query('series_id == @series')\n    \n    train_events = train_events.dropna()\n    train_events[\"step\"]  = train_events[\"step\"].astype(\"int\")\n    train_events[\"awake\"] = train_events[\"event\"].replace({\"onset\":1,\"wakeup\":0})\n\n    train = pd.merge(train_series, train_events[['step','awake']], on='step', how='left')\n    train[\"awake\"] = train[\"awake\"].bfill(axis ='rows')\n    # final section:\n    # train_events.groupby('series_id').tail(1)[\"event\"].unique()\n    # Result: the last event is always a \"wakeup\"\n    train['awake'] = train['awake'].fillna(1) # awake\n    train[\"awake\"] = train[\"awake\"].astype(\"int\")\n    return(train)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.151276Z","iopub.execute_input":"2023-09-18T17:49:31.152298Z","iopub.status.idle":"2023-09-18T17:49:31.162605Z","shell.execute_reply.started":"2023-09-18T17:49:31.152265Z","shell.execute_reply":"2023-09-18T17:49:31.161444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"smaller_train_data = []\n\nfor series_id in no_NaN_series:\n    train = get_train_series(series_id)\n    smaller_train_data.append(train)\n    \n    # vizualize these series\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"anglez\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"enmo\",hue=\"awake\", linewidth = 0.5)\n    plt.show();\n    del train\n    gc.collect();\n    ","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:49:31.164601Z","iopub.execute_input":"2023-09-18T17:49:31.164974Z","iopub.status.idle":"2023-09-18T17:57:16.276824Z","shell.execute_reply.started":"2023-09-18T17:49:31.164943Z","shell.execute_reply":"2023-09-18T17:57:16.275474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train = pd.concat(smaller_train_data).reset_index(drop=True)\nZzzs_train[\"series_id\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:57:16.281943Z","iopub.execute_input":"2023-09-18T17:57:16.282413Z","iopub.status.idle":"2023-09-18T17:57:22.565540Z","shell.execute_reply.started":"2023-09-18T17:57:16.282363Z","shell.execute_reply":"2023-09-18T17:57:22.564378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:57:22.566848Z","iopub.execute_input":"2023-09-18T17:57:22.567170Z","iopub.status.idle":"2023-09-18T17:57:22.587347Z","shell.execute_reply.started":"2023-09-18T17:57:22.567143Z","shell.execute_reply":"2023-09-18T17:57:22.585971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Write out the new lightweight dataset with these 35 selected series","metadata":{}},{"cell_type":"code","source":"Zzzs_train.to_parquet('Zzzs_train.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:57:22.589057Z","iopub.execute_input":"2023-09-18T17:57:22.589582Z","iopub.status.idle":"2023-09-18T17:57:27.411358Z","shell.execute_reply.started":"2023-09-18T17:57:22.589551Z","shell.execute_reply":"2023-09-18T17:57:27.410311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# `Zzzs_train_multi.parquet`\n## Now also add on the 8 datasets that have no event data whatsoever (`class=2`)","metadata":{}},{"cell_type":"code","source":"train_series_names = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", columns=['series_id'])\nseries_id_list = train_series_names['series_id'].unique().tolist()\n\ntrain_events = train_events.dropna()\nevents_series_id_list = train_events['series_id'].unique().tolist()\n\nbase_series = list(set(series_id_list) - set(events_series_id_list))\n\n# list out the series:\nbase_series","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:57:27.413519Z","iopub.execute_input":"2023-09-18T17:57:27.414410Z","iopub.status.idle":"2023-09-18T17:57:45.664615Z","shell.execute_reply.started":"2023-09-18T17:57:27.414344Z","shell.execute_reply":"2023-09-18T17:57:45.663248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"add the target `awake=2`","metadata":{}},{"cell_type":"code","source":"def get_train_series_base(series):\n    train = pd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\", filters=[('series_id','=',series)])\n    train[\"awake\"] = 2\n    # just to make sure\n    train[\"awake\"] = train[\"awake\"].astype(\"int\")\n    return(train)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:57:45.666376Z","iopub.execute_input":"2023-09-18T17:57:45.667090Z","iopub.status.idle":"2023-09-18T17:57:45.674730Z","shell.execute_reply.started":"2023-09-18T17:57:45.667047Z","shell.execute_reply":"2023-09-18T17:57:45.673196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_train_data = []\n\nfor series_id in base_series:\n    train = get_train_series_base(series_id)\n    base_train_data.append(train)\n    \n    # vizualize these series\n    display(Markdown('###  anglez for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"anglez\", color=\"purple\", linewidth = 0.5)\n    plt.show();\n    display(Markdown('###  enmo for series ' + series_id))\n    fig, ax = plt.subplots(figsize=(20, 3))\n    sns.lineplot(data=train, x=\"step\", y=\"enmo\", color=\"olive\", linewidth = 0.5)\n    plt.show();\n    del train\n    gc.collect();","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:57:45.676216Z","iopub.execute_input":"2023-09-18T17:57:45.676607Z","iopub.status.idle":"2023-09-18T17:59:25.482436Z","shell.execute_reply.started":"2023-09-18T17:57:45.676574Z","shell.execute_reply":"2023-09-18T17:59:25.481119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_base = pd.concat(base_train_data).reset_index(drop=True)\n# combine the 35 series from before with the 8 series that have no event data\nZzzs_train_multi = pd.concat([Zzzs_train,Zzzs_base]).reset_index(drop=True)\n# check number of series\nZzzs_train_multi[\"series_id\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:59:25.483988Z","iopub.execute_input":"2023-09-18T17:59:25.484314Z","iopub.status.idle":"2023-09-18T17:59:31.870640Z","shell.execute_reply.started":"2023-09-18T17:59:25.484286Z","shell.execute_reply":"2023-09-18T17:59:31.869325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Zzzs_train_multi.to_parquet('Zzzs_train_multi.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-09-18T17:59:31.872221Z","iopub.execute_input":"2023-09-18T17:59:31.872679Z","iopub.status.idle":"2023-09-18T17:59:37.886740Z","shell.execute_reply.started":"2023-09-18T17:59:31.872637Z","shell.execute_reply":"2023-09-18T17:59:37.885636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The datasets:\nThe two datasets have been saved to the Kaggle dataset: [\"Zzzs: Lightweight training dataset + target\"](https://www.kaggle.com/datasets/carlmcbrideellis/zzzs-lightweight-training-dataset-target)\n\n* `Zzzs_train.parquet` 180M\n* `Zzzs_train_multi.parquet` 220M\n\nI hope they are useful to get started with!","metadata":{}}]}