{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](\"https://i.imgur.com/de3OrxO.png\")
\n\n Detect Sleep States
\n- time patterns explore & spectrograms -
\n\n> ๐ **Competition Scope**: detect the occurrence of *onset* (the beginning of sleep) and *wakeup* (the end of sleep) in the accelerometer series.\n\n### Data\n\nAs the dataframes are very large (especially `train_series.parquet`) I will be using [this customized dataset](https://www.kaggle.com/datasets/andradaolteanu/detect-sleep-states-memory-decrease) I made that occupies less memory than the original ones.\n\nThe notebook that contains the steps made to create the datasets: [Zzz๐ค: good night sleep with 80% Memory Reduction](https://www.kaggle.com/code/andradaolteanu/zzz-good-night-sleep-with-80-memory-reduction)\n\n> ๐ **Disclaimer**: in this notebook `series_id` has been remapped to `id_map`; as there are only 277 unique ids and the original id *took a lot of space in terms of memory* (object containing numbers and letters) I chose to remap it using simple integers for **development purposes**.\n\n### โ Libraries","metadata":{}},{"cell_type":"code","source":"# libraries\nimport os\nimport re\nimport gc\nimport wandb\nimport random\nimport math\nfrom glob import glob\nfrom tqdm import tqdm\nfrom time import time\nimport warnings\nimport pandas as pd\nfrom pandas.api.types import is_string_dtype\nfrom pandas.api.types import is_numeric_dtype\nimport numpy as np\nfrom scipy.signal import spectrogram\n\n# visuals\nimport seaborn as sns\nimport matplotlib as mpl\nfrom matplotlib import cm\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport matplotlib.dates as mdates\nfrom matplotlib.ticker import MaxNLocator\nfrom matplotlib.offsetbox import AnnotationBbox, OffsetImage\nplt.rcParams.update({'font.size': 18})\n\n# env check\nwarnings.filterwarnings('ignore')\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCONFIG = {'competition': '2023_sleep', '_wandb_kernel': 'aot', \"source_type\": \"artifact\"}\n\n# color\nclass clr:\n S = '\\033[1m' + '\\033[90m'\n E = '\\033[0m'\n \nmy_colors = [\"#f79256\", \"#fbd1a2\", \"#7dcfb6\", \"#00b2ca\"]\n\nprint(clr.S+\"Notebook Color Schemes:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:24:57.385861Z","iopub.execute_input":"2023-10-13T13:24:57.386838Z","iopub.status.idle":"2023-10-13T13:25:00.312296Z","shell.execute_reply.started":"2023-10-13T13:24:57.386787Z","shell.execute_reply":"2023-10-13T13:25:00.311218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐ W&B Fork & Run\n\nIn order to run this notebook you will need to input your own **secret API key** within the `! wandb login $secret_value_0` line. \n\n๐**How do you get your own API key?**\n\nSuper simple! Go to **https://wandb.ai/site** -> Login -> Click on your profile in the top right corner -> Settings -> Scroll down to API keys -> copy your very own key (for more info check [this amazing notebook for ML Experiment Tracking on Kaggle](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases)).\n\n![](\"https://i.imgur.com/fFccmoS.png\")
","metadata":{}},{"cell_type":"code","source":"# ๐ secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"wandb\")\n\n! wandb login $secret_value_0","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:25:00.314690Z","iopub.execute_input":"2023-10-13T13:25:00.315690Z","iopub.status.idle":"2023-10-13T13:25:04.933067Z","shell.execute_reply.started":"2023-10-13T13:25:00.315649Z","shell.execute_reply":"2023-10-13T13:25:04.931509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ Helper Functions Below","metadata":{}},{"cell_type":"code","source":"# === data discover ===\n\ndef jitter(values,j):\n return values + np.random.normal(j,0.05,values.shape)\n\n\ndef find_rectangles(arr):\n '''\n return indices where the rectangle starts and ends\n '''\n rectangles = []\n start = None\n for i, val in enumerate(arr):\n if val == 3:\n if start is None:\n start = i\n elif start is not None:\n rectangles.append((start, i - 1))\n start = None\n if start is not None:\n rectangles.append((start, len(arr) - 1))\n return rectangles\n\n\ndef get_general_info(df, desc=None):\n \n # ๐ new exp\n run = wandb.init(project='2023_sleep', name=f'{desc}_data_summary', config=CONFIG)\n\n print(clr.S+\"--- General Info ---\"+clr.E)\n print(clr.S+\"Data Shape:\"+clr.E, df.shape)\n print(clr.S+\"Data Cols:\"+clr.E, df.columns.tolist())\n print(clr.S+\"Total No. of Cols:\"+clr.E, len(df.columns.tolist()))\n print(clr.S+\"No. Missing Values:\"+clr.E, df.isna().sum().sum())\n print(clr.S+\"Columns with missing data:\"+clr.E, \"\\n\",\n df.isna().sum()[df.isna().sum() != 0], \"\\n\")\n\n for col in df.columns:\n if is_string_dtype(df[col]):\n print(clr.S+f\"--- {col} --- is type string\"+clr.E)\n print(clr.S+f\"[nunique] {col}:\"+clr.E, \n df[col].nunique())\n \n elif is_numeric_dtype(df[col]):\n print(clr.S+f\"--- {col} --- is type numeric\"+clr.E)\n print(clr.S+f\"[describe] {col}:\"+clr.E, \"\\n\",\n df[col].describe())\n \n # log data\n wandb.log\n (\n {\"data_shape\": len(df),\n \"missing_values\": df.isna().sum().sum()\n }\n )\n wandb.finish()\n print(\"๐ Info saved to dashboard.\")\n \n\ndef get_missing_values_plot(df):\n '''\n Plots missing values barchart for a given dataframe.\n '''\n \n # count missing values\n missing_counts = df.isnull().sum().reset_index()\\\n .sort_values(0, ascending=False)\\\n .reset_index(drop=True)\n missing_counts.columns = [\"col_name\", \"missing_count\"]\n\n # plot\n plt.figure(figsize=(24, 16))\n axs = sns.barplot(y=missing_counts.col_name, x=missing_counts.missing_count, \n color=my_colors[0])\n show_values_on_bars(axs, h_v=\"h\", space=0.4)\n plt.xlabel('no. missing values', size=20, weight=\"bold\")\n plt.ylabel('column name', size=20, weight=\"bold\")\n plt.title('Missing Values', size=22, weight=\"bold\")\n plt.show();\n \n \n# === plots ===\ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)\n \n \n# === ๐ w&b ===\ndef save_dataset_artifact(run_name, artifact_name, path, data_type=\"dataset\"):\n '''Saves dataset to W&B Artifactory.\n run_name: name of the experiment\n artifact_name: under what name should the dataset be stored\n path: path to the dataset'''\n \n run = wandb.init(project='2023_sleep', \n name=run_name, \n config=CONFIG)\n artifact = wandb.Artifact(name=artifact_name, \n type=data_type)\n artifact.add_file(path)\n\n wandb.log_artifact(artifact)\n wandb.finish()\n print(f\"๐Artifact {artifact_name} has been saved successfully.\")\n \n \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n '''Create and save lineplot/barplot in W&B Environment.\n x_data & y_data: Pandas Series containing x & y data\n x_name & y_name: strings containing axis names\n title: title of the graph\n log: string containing name of log'''\n \n data = [[label, val] for (label, val) in zip(x_data, y_data)]\n table = wandb.Table(data=data, columns = [x_name, y_name])\n \n if plot == \"line\":\n wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n elif plot == \"bar\":\n wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n elif plot == \"scatter\":\n wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n \n \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n '''Create and save histogram in W&B Environment.\n x_data: Pandas Series containing x values\n x_name: strings containing axis name\n title: title of the graph\n log: string containing name of log'''\n \n data = [[x] for x in x_data]\n table = wandb.Table(data=data, columns=[x_name])\n wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-10-13T13:25:04.935251Z","iopub.execute_input":"2023-10-13T13:25:04.935605Z","iopub.status.idle":"2023-10-13T13:25:04.961548Z","shell.execute_reply.started":"2023-10-13T13:25:04.935574Z","shell.execute_reply":"2023-10-13T13:25:04.960825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐ log cover\nrun = wandb.init(project='2023_sleep', name='cover', config=CONFIG)\ncover = plt.imread(\"/kaggle/input/detect-sleep-states-memory-decrease/de3OrxO.png\")\nwandb.log({\"cover\": wandb.Image(cover)})\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:25:04.964817Z","iopub.execute_input":"2023-10-13T13:25:04.965380Z","iopub.status.idle":"2023-10-13T13:25:46.692510Z","shell.execute_reply.started":"2023-10-13T13:25:04.965351Z","shell.execute_reply":"2023-10-13T13:25:46.691631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Datasets first look","metadata":{}},{"cell_type":"code","source":"# events\nevents = pd.read_parquet(\"/kaggle/input/detect-sleep-states-memory-decrease/train_events.parquet\")\nevents.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:25:46.693420Z","iopub.execute_input":"2023-10-13T13:25:46.693744Z","iopub.status.idle":"2023-10-13T13:25:46.993269Z","shell.execute_reply.started":"2023-10-13T13:25:46.693718Z","shell.execute_reply":"2023-10-13T13:25:46.992295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_general_info(df=events, desc=\"events\")","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:25:46.994562Z","iopub.execute_input":"2023-10-13T13:25:46.994927Z","iopub.status.idle":"2023-10-13T13:26:24.964518Z","shell.execute_reply.started":"2023-10-13T13:25:46.994893Z","shell.execute_reply":"2023-10-13T13:26:24.963477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# series\nseries = pd.read_parquet(\"/kaggle/input/detect-sleep-states-memory-decrease/train_series.parquet\")\nseries.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:26:24.965891Z","iopub.execute_input":"2023-10-13T13:26:24.966206Z","iopub.status.idle":"2023-10-13T13:26:46.318081Z","shell.execute_reply.started":"2023-10-13T13:26:24.966180Z","shell.execute_reply":"2023-10-13T13:26:46.316907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_general_info(df=series, desc=\"series\")","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:26:46.319710Z","iopub.execute_input":"2023-10-13T13:26:46.320129Z","iopub.status.idle":"2023-10-13T13:27:45.146738Z","shell.execute_reply.started":"2023-10-13T13:26:46.320089Z","shell.execute_reply":"2023-10-13T13:27:45.145336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Exploratory Analysis\n\n## 2.1 Are people consistent when they wake up/ fall asleep\n\n**Question**: I was curious to see if people are consistent on falling asleep every night at around the same hour. Or is the deviation high?\n\n**Answer**: it looks like the *onset* has a much higher deviation (the hour when one person falls asleep can vary by a lot for more than half of the patients), while the *wakeup* hour is quite consistent (meaning that regularly a person wakes up at around the same hour).\n\nThis would align with the fact that people can go to sleep at any time (depending on the day), but daily activities/routines make us be more diciplined on the hour when we wake up.\n\n*TODO: Check weekends*","metadata":{}},{"cell_type":"code","source":"# ๐\nrun = wandb.init(project='2023_sleep', name='eda_consistency', config=CONFIG)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:27:45.148156Z","iopub.execute_input":"2023-10-13T13:27:45.148716Z","iopub.status.idle":"2023-10-13T13:28:19.086085Z","shell.execute_reply.started":"2023-10-13T13:27:45.148684Z","shell.execute_reply":"2023-10-13T13:28:19.084857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get the hour\nevents[\"hour\"] = events.timestamp.dt.hour\n# compute the std of the hour each night per id\nonset = events[events.event==1].groupby(by=[\"id_map\"]).hour.std().reset_index()\nwakeup = events[events.event==2].groupby(by=[\"id_map\"]).hour.std().reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:19.089380Z","iopub.execute_input":"2023-10-13T13:28:19.089932Z","iopub.status.idle":"2023-10-13T13:28:19.761999Z","shell.execute_reply.started":"2023-10-13T13:28:19.089892Z","shell.execute_reply":"2023-10-13T13:28:19.760523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 12))\nfig.suptitle('Falling asleep/ waking up consistency', \n weight=\"bold\", size=25)\n\nsns.distplot(onset.hour, rug=True, hist=False, \n rug_kws={\"color\": my_colors[2]},\n kde_kws={\"color\": my_colors[2], \"lw\": 5, \"alpha\": 1},\n ax=ax1)\nax1.set_title('Onset Deviation', weight=\"bold\", size=20)\n\nsns.distplot(wakeup.hour, rug=True, hist=False, \n rug_kws={\"color\": my_colors[0]},\n kde_kws={\"color\": my_colors[0], \"lw\": 5, \"alpha\": 1},\n ax=ax2)\nax2.set_title('Wakeup Deviation', weight=\"bold\", size=20)\n\nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:19.763830Z","iopub.execute_input":"2023-10-13T13:28:19.764232Z","iopub.status.idle":"2023-10-13T13:28:21.173145Z","shell.execute_reply.started":"2023-10-13T13:28:19.764193Z","shell.execute_reply":"2023-10-13T13:28:21.171750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"create_wandb_hist(x_data=onset.hour, \n x_name=\"hour\",\n title=\"Onset Deviation\",\n log=\"onset_deviation\")\n\ncreate_wandb_hist(x_data=wakeup.hour, \n x_name=\"hour\",\n title=\"Wakeup Deviation\",\n log=\"wakeup_deviation\")","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:21.174444Z","iopub.execute_input":"2023-10-13T13:28:21.174784Z","iopub.status.idle":"2023-10-13T13:28:22.326016Z","shell.execute_reply.started":"2023-10-13T13:28:21.174755Z","shell.execute_reply":"2023-10-13T13:28:22.324994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:22.327556Z","iopub.execute_input":"2023-10-13T13:28:22.327845Z","iopub.status.idle":"2023-10-13T13:28:34.219502Z","shell.execute_reply.started":"2023-10-13T13:28:22.327821Z","shell.execute_reply":"2023-10-13T13:28:34.218378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Regular times for onset & wakeup\n\n**Question**: what are the hours when people usually go to sleep or wake up?\n\n**Answer**: I really thought this would not look like this - so observants can:\n* wake up any time between 00:00 and 14:00 (with the vast majority between 04:00 -> 10:00)\n* fall asleep between 00:00 -> 04:00 and between 19:00 -> 00:00.","metadata":{}},{"cell_type":"code","source":"events[\"time\"] = events.timestamp.dt.time\ndf = events.copy()\ndf.time = pd.to_datetime(df.time, format='%H:%M:%S')\ndf.set_index(['time'],inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:34.221034Z","iopub.execute_input":"2023-10-13T13:28:34.221558Z","iopub.status.idle":"2023-10-13T13:28:34.243818Z","shell.execute_reply.started":"2023-10-13T13:28:34.221518Z","shell.execute_reply":"2023-10-13T13:28:34.242481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 12))\nax = sns.scatterplot(x=df.index, y=jitter(df[\"event\"], 0.01), hue=df[\"event\"], \n palette=[my_colors[2], my_colors[0]],\n size=df[\"event\"], alpha=0.7, sizes=(500, 500),\n )\n\nplt.title('Times for onset & wakeup', weight=\"bold\", size=25)\nax.set(xlabel=\"time\", ylabel=\"event type\")\nax.xaxis.set_major_locator(mdates.HourLocator())\nax.xaxis.set_major_formatter(mdates.DateFormatter(\"%H:%M\"))\nax.tick_params(axis=\"x\", rotation=45)\nax.set_ylim(0, 3)\nax.legend(labels=[\"wakeup\", \"onset\"]);","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:34.245056Z","iopub.execute_input":"2023-10-13T13:28:34.245456Z","iopub.status.idle":"2023-10-13T13:28:36.285796Z","shell.execute_reply.started":"2023-10-13T13:28:34.245426Z","shell.execute_reply":"2023-10-13T13:28:36.284929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:36.287100Z","iopub.execute_input":"2023-10-13T13:28:36.287721Z","iopub.status.idle":"2023-10-13T13:28:36.417965Z","shell.execute_reply.started":"2023-10-13T13:28:36.287690Z","shell.execute_reply":"2023-10-13T13:28:36.416733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Length of sleep\n\n**Question**: Now that we know the variations in sleep and what are usually the hours, I want to see how long is usually the *longest* session of sleep (as per competition description, only the longest period has been recorded).\n\n**Answer**: On average *between 7 and 10 hours*. We have a few quite extreme outliers when some nights they slept only ~2 hrs and some other outliers with nights when they slept 15+ hours.","metadata":{}},{"cell_type":"code","source":"# ๐\nrun = wandb.init(project='2023_sleep', name='eda_sleep_duration', config=CONFIG)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:28:36.419503Z","iopub.execute_input":"2023-10-13T13:28:36.419922Z","iopub.status.idle":"2023-10-13T13:29:47.409160Z","shell.execute_reply.started":"2023-10-13T13:28:36.419884Z","shell.execute_reply":"2023-10-13T13:29:47.407809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compute the time diff\nevents = events.sort_values(by=[\"id_map\", \"night\"]).reset_index(drop=True)\nevents[\"time_diff\"] = events.timestamp.diff()\n# drop value at first night as it's not correct\nevents.loc[(events.night == 1) & (events.event==1), 'time_diff'] = np.nan\n# select only length of sleep\ntime_diffs = events[events.event==2].dropna()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:29:47.410371Z","iopub.execute_input":"2023-10-13T13:29:47.411132Z","iopub.status.idle":"2023-10-13T13:29:48.161327Z","shell.execute_reply.started":"2023-10-13T13:29:47.411103Z","shell.execute_reply":"2023-10-13T13:29:48.160148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot\nf, (a0, a1) = plt.subplots(2, 1, gridspec_kw={'height_ratios': [3, 1]}, figsize=(24, 15))\nsns.distplot(time_diffs.time_diff.dt.components['hours'],\n rug=True, hist=False,\n rug_kws={\"color\": my_colors[2]},\n kde_kws={\"color\": my_colors[2], \"lw\": 5, \"alpha\": 0.7},\n ax=a0)\n\nsns.boxplot(x=time_diffs.time_diff.dt.components['hours'],\n ax=a1, notch=False, showcaps=True, \n flierprops={\"marker\": \"x\"},\n boxprops={\"facecolor\": my_colors[2]},\n medianprops={\"color\": my_colors[0]},)\n\nplt.suptitle(\"How long does a sleep session take?\", weight=\"bold\", size=25)\nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:29:48.162638Z","iopub.execute_input":"2023-10-13T13:29:48.162935Z","iopub.status.idle":"2023-10-13T13:29:50.397943Z","shell.execute_reply.started":"2023-10-13T13:29:48.162902Z","shell.execute_reply":"2023-10-13T13:29:50.396896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"create_wandb_hist(x_data=time_diffs.time_diff.dt.components['hours'], \n x_name=\"hour\",\n title=\"How long does a sleep session take?\",\n log=\"sleep_duration\")","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:29:50.399349Z","iopub.execute_input":"2023-10-13T13:29:50.399659Z","iopub.status.idle":"2023-10-13T13:29:51.405259Z","shell.execute_reply.started":"2023-10-13T13:29:50.399632Z","shell.execute_reply":"2023-10-13T13:29:51.403937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:29:51.406602Z","iopub.execute_input":"2023-10-13T13:29:51.406904Z","iopub.status.idle":"2023-10-13T13:29:59.146404Z","shell.execute_reply.started":"2023-10-13T13:29:51.406878Z","shell.execute_reply":"2023-10-13T13:29:59.145383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4 enmo&anglez vs onset&waking time\n\n**Question**: I also wanted to see how the waking and onset times look for an individual when they sleep, in regards to the features that we get from the accelerometer data (`angles` and `enmo`).\n\n**Answer**: This is actually SO COOL! The plot below show anglez and enmo on a window in time for a few ids. The vertical shade showcases when the observant is asleep and we can see with the naked eye the difference in data during this period vs the rest of the day:\n* angles - more sparse\n* enmo - lower values","metadata":{}},{"cell_type":"code","source":"def get_accelerometer_plot_data(id_map, night_max=5):\n \n # --- events ---\n # get one id_map and the dates for 5 nights\n df_e = events[(events[\"id_map\"]==id_map) & \n (events[\"night\"]>=1) & \n (events[\"night\"]<=night_max)]\\\n .reset_index(drop=True)\n df_e[\"date\"] = df_e.timestamp.dt.date.astype(str)\n\n # --- series ---\n # filter on id_map\n df_s = series[series[\"id_map\"]==id_map].reset_index(drop=True)\n df_s[\"date\"] = df_s.timestamp.dt.date.astype(str)\n df_s[\"time\"] = df_s.timestamp.dt.time\n # now retrieve only 1 day\n df_s = df_s[df_s[\"date\"].isin(df_e.date.unique())]\n\n # grab event (target) info\n df_s = df_s.merge(right=events[[\"event\", \"step\", \"id_map\"]],\n on=[\"id_map\", \"step\"], how=\"left\")\n # fill between 1 and 2 with value 3\n # 3 meaning that the person is sleeping\n fill_value = 3\n where_value = 1\n df_s['event'] = df_s['event'].fillna(fill_value)\\\n .where(df_s['event']\\\n .fillna(method='ffill').eq(where_value) | df_s['event'].notna())\n \n return df_s","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:29:59.147979Z","iopub.execute_input":"2023-10-13T13:29:59.148278Z","iopub.status.idle":"2023-10-13T13:29:59.156229Z","shell.execute_reply.started":"2023-10-13T13:29:59.148253Z","shell.execute_reply":"2023-10-13T13:29:59.155085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"id_maps = [26, 32, 100, 105, 133, 150, 190, 195]\n\nfor id_map in id_maps:\n df_s = get_accelerometer_plot_data(id_map)\n features = [\"anglez\", \"enmo\"]\n\n for feat in features:\n # plot\n plt.figure(figsize=(24, 12))\n plt.title(f\"id [{id_map}]: {feat}\", weight=\"bold\", size=25)\n sns.lineplot(data=df_s, x=\"timestamp\", y=feat,\n color=\"#00B2CA\", lw=2)\n plt.xlabel(\"Time\", size = 18, weight=\"bold\")\n plt.ylabel(f\"{feat}\", size = 18, weight=\"bold\")\n sns.despine(right=True, top=True, left=True)\n\n # sleeping windows\n rectangles = find_rectangles(df_s['event'])\n for start, end in rectangles:\n plt.axvspan(df_s.loc[start, \"timestamp\"], \n df_s.loc[end, \"timestamp\"], \n color=my_colors[2], alpha=0.5)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:29:59.157662Z","iopub.execute_input":"2023-10-13T13:29:59.158219Z","iopub.status.idle":"2023-10-13T13:30:25.623610Z","shell.execute_reply.started":"2023-10-13T13:29:59.158188Z","shell.execute_reply":"2023-10-13T13:30:25.622395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Spectrogram analysis\n\nThis competition can be tackled from multiple angles:\n* it can be a tabular classification problem\n* it can be an LSTM-RNN classifier using the sequential (time) data\n* it can be a computer vision problem! if we convert the time data to spectrograms :)\n\nI always wanted to do that and this competition's data looks perfect to try.\n\n*TODO: this area is still WIP*","metadata":{}},{"cell_type":"code","source":"def create_spectrogram(value):\n \n # defines the width of each chunk in terms of samples\n # large window size -> better frequency resolution -> poor time localization\n window_size = 256 \n # ensures that each chunk has a certain number of samples \n # that are overlapping for each chunk\n # default is 0.5*window\n overlap = 128\n # how many FFT points are desired to be computed per chunk\n # how fine-grained the frequency resolution will be\n nfft = 256 \n # sampling frequency of your signal\n # default is 1\n fs = 1.0 \n\n frequencies, times, Sxx = spectrogram(value, fs=fs, nperseg=window_size, \n noverlap=overlap, nfft=nfft)\n return frequencies, times, Sxx","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:30:25.625194Z","iopub.execute_input":"2023-10-13T13:30:25.625536Z","iopub.status.idle":"2023-10-13T13:30:25.631354Z","shell.execute_reply.started":"2023-10-13T13:30:25.625505Z","shell.execute_reply":"2023-10-13T13:30:25.630310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### What is a Spectrogram\n\n[A spectrogram is](https://stackoverflow.com/questions/29321696/what-is-a-spectrogram-and-how-do-i-set-its-parameters) a visual representation of the Short-Time Fourier Transform. Think of this as taking chunks of an input signal and applying a local Fourier Transform on each chunk. Each chunk has a specified width and you apply a Fourier Transform to this chunk. You should take note that each chunk has an associated frequency distribution. For each chunk that is centred at a specific time point in your time signal, you get a bunch of frequency components. The collection of all of these frequency components at each chunk and plotted all together is what is essentially a spectrogram.\n\nThe spectrogram is a 2D visual heat map where the horizontal axis represents the time of the signal and the vertical axis represents the frequency axis. What is visualized is an image where darker colours means that for a particular time point and a particular frequency, the lower in magnitude the frequency component is, the darker the colour. Similarly, the higher in magnitude the frequency component is, the lighter the colour.","metadata":{}},{"cell_type":"code","source":"id_maps = [26, 32, 100, 105, 133, 150, 190, 195]\n\nfor id_map in id_maps:\n spec_data = get_accelerometer_plot_data(id_map, night_max=1).drop(columns=\"event\")\n \n features = [\"anglez\", \"enmo\"]\n fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 10))\n axs = [ax1, ax2]\n fig.suptitle(f'A night for id: {id_map}')\n\n for k, feat in enumerate(features):\n f, t, s = create_spectrogram(spec_data[feat].values)\n axs[k].pcolormesh(t, f, 10 * np.log10(s), cmap=\"RdYlBu\")\n axs[k].set_xlabel('Time (s)')\n axs[k].set_ylabel('Frequency (Hz)')\n axs[k].set_title(f'Spectrogram of {feat}')","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:30:25.632699Z","iopub.execute_input":"2023-10-13T13:30:25.633375Z","iopub.status.idle":"2023-10-13T13:30:34.987414Z","shell.execute_reply.started":"2023-10-13T13:30:25.633334Z","shell.execute_reply":"2023-10-13T13:30:34.986700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Advanced feature engineering for time\n\nFound this amazing module called `add_datepart` from `fastai` that enables me to create in an instant time features from my `timestamp` columns (without me having to compute them).\n\nThe new features added:\n* **date features**: 'Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear', 'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start'\n* **time features**: 'Hour', 'Minute', 'Second'\n\n*TODO: add more time features like day/nigh or dusk/morning/noon/afternoon/evening/night.*","metadata":{}},{"cell_type":"code","source":"! pip install fastai\n\nfrom fastai.tabular.core import add_datepart","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-10-13T13:30:34.988530Z","iopub.execute_input":"2023-10-13T13:30:34.989438Z","iopub.status.idle":"2023-10-13T13:30:49.235267Z","shell.execute_reply.started":"2023-10-13T13:30:34.989408Z","shell.execute_reply":"2023-10-13T13:30:49.234361Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# first we need a dataframe with one single column\n# the timestamp\n# ! it is necessary to be THIS format for it to work\nseries_timestamp = pd.DataFrame(series[\"timestamp\"])\n\n# now we overwrite this dataframe\n# and explode the timestamp into multiple features\nseries_timestamp = add_datepart(series_timestamp, \n field_name='timestamp', \n prefix='fast_', \n drop=True, \n time=True)\n\nseries_timestamp.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:30:49.236431Z","iopub.execute_input":"2023-10-13T13:30:49.236919Z","iopub.status.idle":"2023-10-13T13:34:09.431781Z","shell.execute_reply.started":"2023-10-13T13:30:49.236893Z","shell.execute_reply":"2023-10-13T13:34:09.430653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# repeat for events dataset\nevents_timestamp = pd.DataFrame(events[\"timestamp\"])\n\n# now we overwrite this dataframe\n# and explode the timestamp into multiple features\nevents_timestamp = add_datepart(events_timestamp, \n field_name='timestamp', \n prefix='fast_', \n drop=True, \n time=True)\n\nevents_timestamp.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:34:09.958679Z","iopub.execute_input":"2023-10-13T13:34:09.959430Z","iopub.status.idle":"2023-10-13T13:34:10.011997Z","shell.execute_reply.started":"2023-10-13T13:34:09.959390Z","shell.execute_reply":"2023-10-13T13:34:10.011201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save datasets\nseries_timestamp.to_parquet(\"train_series_time_fe.parquet\", index=False)\nevents_timestamp.to_parquet(\"train_events_time_fe.parquet\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:34:10.013455Z","iopub.execute_input":"2023-10-13T13:34:10.014110Z","iopub.status.idle":"2023-10-13T13:34:36.736380Z","shell.execute_reply.started":"2023-10-13T13:34:10.014073Z","shell.execute_reply":"2023-10-13T13:34:36.734929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐ save to W&B\nsave_dataset_artifact(run_name=\"train_series_time_fe\",\n artifact_name=\"train_series_time_fe\",\n path=\"/kaggle/input/detect-sleep-states-memory-decrease/feature_engineering/train_series_time_fe.parquet\", \n data_type=\"dataset\")\n\nsave_dataset_artifact(run_name=\"train_events_time_fe\",\n artifact_name=\"train_events_time_fe\",\n path=\"/kaggle/input/detect-sleep-states-memory-decrease/feature_engineering/train_events_time_fe.parquet\", \n data_type=\"dataset\")","metadata":{"execution":{"iopub.status.busy":"2023-10-13T13:42:40.602502Z","iopub.execute_input":"2023-10-13T13:42:40.602890Z","iopub.status.idle":"2023-10-13T13:44:16.739931Z","shell.execute_reply.started":"2023-10-13T13:42:40.602858Z","shell.execute_reply":"2023-10-13T13:44:16.738784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐ [my W&B dash](https://wandb.ai/andrada/2023_sleep?workspace=user-andrada)\n \n![](\"https://i.imgur.com/SDicyoE.png\")
\n\n------\n\n![](\"https://i.imgur.com/knxTRkO.png\")
\n\n### My Specs\n\n* ๐ฅ Z8 G4 Workstation\n* ๐พ 2 CPUs & 96GB Memory\n* ๐ฎ 2x NVIDIA A6000\n* ๐ป Zbook Studio G9 on the go","metadata":{}}]}