{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on September 05, 2023. By Marília Prata, mpwolke.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom matplotlib import pyplot as plt\nfrom tqdm import tqdm\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-06T04:46:10.500917Z","iopub.execute_input":"2023-09-06T04:46:10.501714Z","iopub.status.idle":"2023-09-06T04:46:12.770109Z","shell.execute_reply.started":"2023-09-06T04:46:10.501672Z","shell.execute_reply":"2023-09-06T04:46:12.769006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition Citation\n\n@misc{child-mind-institute-detect-sleep-states,\n\n    author = {Greg Kiar, Maggie, Nathalia Esper, Ryan Holbrook, Yuki Kotani},\n    \n    title = {Child Mind Institute - Detect Sleep States},\n    \n    publisher = {Kaggle},\n    \n    year = {2023},\n    \n    url = {https://kaggle.com/competitions/child-mind-institute-detect-sleep-states}\n}","metadata":{}},{"cell_type":"markdown","source":"#Wake up Pandas. Time to code!\n\n\"Zzz is an onomatopoeic representation of snoring. Zzz is commonly used in comic books to indicate that someone is sleeping. It has also become a part of the idiom catch some Zs, which means to sleep.\"\nhttps://www.grammarly.com/blog/zzz-meaning/#:~:text=Zzz%20is%20an%20onomatopoeic%20representation,Zs%2C%20which%20means%20to%20sleep.\n\n![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSHO5a4RATTFpDAKrpmvdb8BAqj4ZnRodqBzg&usqp=CAU)\nSoundCloud","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\nsubmission = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-09-06T04:46:24.570827Z","iopub.execute_input":"2023-09-06T04:46:24.571473Z","iopub.status.idle":"2023-09-06T04:46:24.632453Z","shell.execute_reply.started":"2023-09-06T04:46:24.571420Z","shell.execute_reply":"2023-09-06T04:46:24.631200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Sedentary Behaviour Importance\n\nCitation: Bakrania K, Yates T, Rowlands AV, Esliger DW, Bunnewell S, Sanders J, et al. (2016) Intensity Thresholds on Raw Acceleration Data: Euclidean Norm Minus One (ENMO) and Mean Amplitude Deviation (MAD) Approaches. PLoS ONE 11(10): e0164045. https://doi.org/10.1371/journal.pone.0164045\n\n\"There is cumulative evidence that sedentary behaviour, characterised as any waking behaviour with low energy expenditure (≤1.5 metabolic equivalents) while in a sitting or reclining posture, is detrimentally associated with a number of health outcomes including cardiovascular disease, type 2 diabetes mellitus and all-cause mortality.\"\n\n\"This has important implications given that adults spend the majority of their waking hours (~55% to ~70%) sedentary. Correspondingly, engaging in light-intensity physical activities (e.g. standing and light walking) has been shown to have beneficial effects on health. Therefore, accurately identifying and distinguishing between sedentary behaviour and light-intensity physical activity is extremely important.\"\n\nhttps://journals.plos.org/plosone/article?id=10.1371/journal.pone.0164045","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T03:41:09.505607Z","iopub.execute_input":"2023-09-06T03:41:09.507324Z","iopub.status.idle":"2023-09-06T03:41:09.534889Z","shell.execute_reply.started":"2023-09-06T03:41:09.507276Z","shell.execute_reply":"2023-09-06T03:41:09.533748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#thanks to Chun Fu to teach me How to parse those timestamps with WakeUp/Onset","metadata":{}},{"cell_type":"code","source":"#By Chun Fu https://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series\n\ntrain['date'] = train['timestamp'].str.split('T', expand=True)[0]\ntrain['time'] = train['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ntrain['timestamp'] = pd.to_datetime(train['date']+' '+train['time'])\ntrain['hour'] = train['timestamp'].dt.hour\ntrain","metadata":{"execution":{"iopub.status.busy":"2023-09-06T04:46:43.939667Z","iopub.execute_input":"2023-09-06T04:46:43.940156Z","iopub.status.idle":"2023-09-06T04:46:44.112379Z","shell.execute_reply.started":"2023-09-06T04:46:43.940115Z","shell.execute_reply":"2023-09-06T04:46:44.111127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Didn't work as expected\n\n#train['timestamp'] = pd.to_datetime(train['timestamp'], format='%y%m%d_%H%M%S%f_%H%M%S')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-06T04:37:09.883424Z","iopub.execute_input":"2023-09-06T04:37:09.884261Z","iopub.status.idle":"2023-09-06T04:37:09.896081Z","shell.execute_reply.started":"2023-09-06T04:37:09.884215Z","shell.execute_reply":"2023-09-06T04:37:09.895161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"event\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:46:11.176682Z","iopub.execute_input":"2023-09-06T02:46:11.177130Z","iopub.status.idle":"2023-09-06T02:46:11.191676Z","shell.execute_reply.started":"2023-09-06T02:46:11.177097Z","shell.execute_reply":"2023-09-06T02:46:11.190273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nfig, ax = plt.subplots(figsize=(4, 4))\ntrain[\"event\"].value_counts().head(3).sort_values(ascending=True).plot(\n    kind=\"barh\", color='g', ax=ax, title=\"Zzzzleep onset and wake\"\n)\nax.set_xlabel(\"Number of Training Examples\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:54:55.563657Z","iopub.execute_input":"2023-09-06T02:54:55.564093Z","iopub.status.idle":"2023-09-06T02:54:55.796496Z","shell.execute_reply.started":"2023-09-06T02:54:55.564062Z","shell.execute_reply":"2023-09-06T02:54:55.795039Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:30:06.848947Z","iopub.execute_input":"2023-09-06T02:30:06.849354Z","iopub.status.idle":"2023-09-06T02:30:06.863596Z","shell.execute_reply.started":"2023-09-06T02:30:06.849321Z","shell.execute_reply":"2023-09-06T02:30:06.862400Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nBASE_DIR = '../input/child-mind-institute-detect-sleep-states/'\ntrain = pd.read_csv(f'{BASE_DIR}/train_events.csv')","metadata":{"execution":{"iopub.status.busy":"2023-09-06T03:00:47.060780Z","iopub.execute_input":"2023-09-06T03:00:47.061196Z","iopub.status.idle":"2023-09-06T03:00:47.092548Z","shell.execute_reply.started":"2023-09-06T03:00:47.061165Z","shell.execute_reply":"2023-09-06T03:00:47.091321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Parquet code by Rob Mulla\n\nThat below will take a little bit","metadata":{}},{"cell_type":"code","source":"#By Rob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream\n\nexample_fn = train.query('event == \"wakeup\"')[\"series_id\"].values[0]\nexample_landmark = pd.read_parquet(f\"{BASE_DIR}/train_series.parquet\")\nexample_landmark.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T03:07:53.842064Z","iopub.execute_input":"2023-09-06T03:07:53.842467Z","iopub.status.idle":"2023-09-06T03:08:43.678087Z","shell.execute_reply.started":"2023-09-06T03:07:53.842436Z","shell.execute_reply":"2023-09-06T03:08:43.677090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.express as px\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2023-09-06T04:49:17.299847Z","iopub.execute_input":"2023-09-06T04:49:17.300414Z","iopub.status.idle":"2023-09-06T04:49:18.352158Z","shell.execute_reply.started":"2023-09-06T04:49:17.300359Z","shell.execute_reply":"2023-09-06T04:49:18.351012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-09-06T05:03:35.272179Z","iopub.execute_input":"2023-09-06T05:03:35.272584Z","iopub.status.idle":"2023-09-06T05:03:35.281298Z","shell.execute_reply.started":"2023-09-06T05:03:35.272552Z","shell.execute_reply":"2023-09-06T05:03:35.280457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#Code by Firat Gonen https://www.kaggle.com/frtgnn/elo-eda-lgbm/notebook \n\nplt.figure(figsize=(10, 6))\nplt.title('Hours Distribution')\nsns.despine()\nsns.set_context(\"notebook\", font_scale=1.5, rc={\"lines.linewidth\": 2.5})\n\nsns.distplot(train['hour'], hist=True, rug=False,norm_hist=True, color='brown');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-06T05:17:47.342046Z","iopub.execute_input":"2023-09-06T05:17:47.342447Z","iopub.status.idle":"2023-09-06T05:17:47.811417Z","shell.execute_reply.started":"2023-09-06T05:17:47.342414Z","shell.execute_reply":"2023-09-06T05:17:47.810085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_df(train, x, y, title=\"Sleeping Hours\", xlabel='hour', ylabel='event', dpi=100):\n    plt.figure(figsize=(15,4), dpi=dpi)\n    plt.plot(x, y, color='tab:red')\n    plt.gca().set(title=title, xlabel=xlabel, ylabel=ylabel)\n    plt.show()\n    \nplot_df(train, x=train['hour'], y=train['event'], title='Sleeping Wakeup and Onset')","metadata":{"execution":{"iopub.status.busy":"2023-09-06T05:04:28.394899Z","iopub.execute_input":"2023-09-06T05:04:28.395438Z","iopub.status.idle":"2023-09-06T05:04:30.223964Z","shell.execute_reply.started":"2023-09-06T05:04:28.395395Z","shell.execute_reply":"2023-09-06T05:04:30.222575Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_df(train, x, y, title=\"Timesteps for each observation\", xlabel='step', ylabel='event', dpi=100):\n    plt.figure(figsize=(15,4), dpi=dpi)\n    plt.plot(x, y, color='tab:green')\n    plt.gca().set(title=title, xlabel=xlabel, ylabel=ylabel)\n    plt.show()\n    \nplot_df(train, x=train['step'], y=train['event'], title='Timesteps for each observation') ","metadata":{"execution":{"iopub.status.busy":"2023-09-06T05:08:49.518497Z","iopub.execute_input":"2023-09-06T05:08:49.519011Z","iopub.status.idle":"2023-09-06T05:08:50.555253Z","shell.execute_reply.started":"2023-09-06T05:08:49.518973Z","shell.execute_reply":"2023-09-06T05:08:50.553855Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nimport matplotlib\n\nmatplotlib.style.use('classic')\n\ndef vis_dist(train, col):\n    variable = train[col].values\n    ax = sns.displot(variable, color='mistyrose')\n    fig = matplotlib.pyplot.gcf()\n    fig.set_size_inches(12.5, 4.5)\n    plt.title(f'Distribution of {col}')\n    plt.xlabel(f'{col}')\n    return plt.show()\n\nvis_dist(train, 'step')","metadata":{"execution":{"iopub.status.busy":"2023-09-06T05:13:18.638103Z","iopub.execute_input":"2023-09-06T05:13:18.638580Z","iopub.status.idle":"2023-09-06T05:13:19.118709Z","shell.execute_reply.started":"2023-09-06T05:13:18.638545Z","shell.execute_reply":"2023-09-06T05:13:19.117400Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Read Parquet another way","metadata":{}},{"cell_type":"code","source":"import dask.dataframe as dd\nimport dask.array as da\nfrom dask.diagnostics import ProgressBar","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:35:10.466819Z","iopub.execute_input":"2023-09-06T02:35:10.467193Z","iopub.status.idle":"2023-09-06T02:35:11.293501Z","shell.execute_reply.started":"2023-09-06T02:35:10.467163Z","shell.execute_reply":"2023-09-06T02:35:11.292283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf = dd.read_parquet(\"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\")","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:36:15.699618Z","iopub.execute_input":"2023-09-06T02:36:15.700059Z","iopub.status.idle":"2023-09-06T02:36:15.789348Z","shell.execute_reply.started":"2023-09-06T02:36:15.700024Z","shell.execute_reply":"2023-09-06T02:36:15.788364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nlen(ddf)","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:36:30.583019Z","iopub.execute_input":"2023-09-06T02:36:30.584354Z","iopub.status.idle":"2023-09-06T02:36:42.296880Z","shell.execute_reply.started":"2023-09-06T02:36:30.584312Z","shell.execute_reply":"2023-09-06T02:36:42.295972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:36:48.868942Z","iopub.execute_input":"2023-09-06T02:36:48.869353Z","iopub.status.idle":"2023-09-06T02:36:50.838919Z","shell.execute_reply.started":"2023-09-06T02:36:48.869317Z","shell.execute_reply":"2023-09-06T02:36:50.837868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Anglez - Z Angle\n\n\"Anglez - As calculated and described by the GGIR package, z-angle is a metric derived from individual accelerometer components that is commonly used in sleep detection, and refers to the angle of the arm relative to the vertical axis of the body.\"\n\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data","metadata":{}},{"cell_type":"code","source":"ddf.anglez.describe().compute()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:38:53.956419Z","iopub.execute_input":"2023-09-06T02:38:53.956856Z","iopub.status.idle":"2023-09-06T02:38:59.608172Z","shell.execute_reply.started":"2023-09-06T02:38:53.956821Z","shell.execute_reply":"2023-09-06T02:38:59.607017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Enmo - Euclidean Norm Minus One\n\n\"enmo - As calculated and described by the GGIR package, ENMO is the Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero. While no standard measure of acceleration exists in this space, this is one of the several commonly computed features.\"\n\nhttps://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/data","metadata":{}},{"cell_type":"code","source":"ddf.enmo.describe().compute()","metadata":{"execution":{"iopub.status.busy":"2023-09-06T02:39:30.458415Z","iopub.execute_input":"2023-09-06T02:39:30.458842Z","iopub.status.idle":"2023-09-06T02:39:34.530966Z","shell.execute_reply.started":"2023-09-06T02:39:30.458808Z","shell.execute_reply":"2023-09-06T02:39:34.529797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#That was the expected to be plot. Better not compare with my charts above.\n\n![](https://media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-018-31266-z/MediaObjects/41598_2018_31266_Fig2_HTML.png)https://www.nature.com/articles/s41598-018-31266-z","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nChun Fu https://www.kaggle.com/code/patrick0302/viz-of-sleeping-time-series\n\nLuxcem: https://www.kaggle.com/code/luxcem/dask-on-large-parquet-dataset/notebook\n\nRob Mulla https://www.kaggle.com/code/robikscube/sign-language-recognition-eda-twitch-stream","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}