{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"},{"sourceId":7034161,"sourceType":"datasetVersion","datasetId":4043445}],"dockerImageVersionId":30587,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"*This notebook only contains the description. For the corresponding code, please refer to https://github.com/HKEdwin22/CMI_SleepWindowPrediction*","metadata":{}},{"cell_type":"markdown","source":"# Overview\nThe project is completed with the following procedures in which functions are designed and implemented for generating files that help lower the time cost and improve performance:\n1. Exploratory data analysis I - train_events.csv\n2. Exploratory data analysis II - train_series.parquet\n3. Model training\n\nReferences are quoted in the end of this notebook and the development flowchart is shown below:\n
\n
\n \n
Figure 1 Development flowchart of the model
\n
","metadata":{}},{"cell_type":"markdown","source":"# Exploratory Data Analaysis I - train_events.csv\nAccording to the descriptive statistics, this data set has a balanced target classes (i.e. onset and wakeup) and contains data from 277 accelerometers. Some of which have odd number of timesteps which indicates that some events don’t appear in pair. We implement the function *IdentifyContradictions()* to investigate this finding, the results are shown below. The identified nights are excluded from now on.\n\n| Series_id | Timstep | Redundant record |\n| :--------: | :-----: | :-------------: |\n| 0ce74d6d2106 | 20 | onset |\n| 154fe824ed87 | 30 | wakeup |\n| 44a41bba1ee7 | 10 | wakeup |\n| efbfc4526d58 | 7 | wakeup |\n| f8a8da8bdd00 | 17 | wakeup|\n
Table 1 Accelerometers with and the corresponding timestep with incomplete onset-wakeup pair

\n\nWe also notice that there are accelerometers having missing entries (i.e. absence of particular nights). We have developed the function *CheckMissingVal()* to identify them and printed the results as below:\n\n
\n
\n \n
Figure 2 Accelerometers with missing nights
\n
\n
\n\nThe five accelerometers have missing entry at night 1. We add them to the data set to make it more systematic. Then we implement the functions *ComputeStepSleep()* and *DecomposeTimeDelta()* for data mining.\n \nReferencing published research papers, most of the study required subjects to wear the accelerometer for nine consecutive days. According to our analysis, if the criteria is set to be seven consecutive days, 67.5% (186) of the accelerometers will be involved in the model training. If nine consecutive days is chosen, 57.5% (159) will be involved. We will try to involve as many as samples as possible, therefore seven consecutive days will be our selection criteria of the samples.","metadata":{}},{"cell_type":"markdown","source":"# Exploratory Data Analysis II - train_series.parquet\nWe implement *CheckId()* and confirm that all the accelerometers in the time series file exist in the sleep log. Now, we randomly draw an onset-wakeup time series of an accelerometer and plot its anglez and enmo as below:\n
\n
\n \n
Figure 3a Angle of a sampled accelerometer
\n \n
Figure 3b Enmo of a sampled accelerometer
\n
\n
\n\nThe red rectangle colors the sleep period while the dashed lines indicate the time of onset and wakeup. The change of signal frequency during or out of the sleep period was more obvious in anglez than enmo. A pair of sudden drop or rise in the signal magnitude was observed only in anglez before and after the reported sleep. These suggested the limited information provided by enmo, and so we decide not to include enmo in our model training. \n\nWe also notice that the time of sleep is vary between or within the same individuals and there is no proof that the sleep is associated with time, so we make an assumption that the onset/wakeup activities are not time dependent, and will deploy a supervised machine learning algorithm for the prediction.\n\nTo avoid biased model, we extract only a pair of onset/wakeup activity that includes the buffer time +/- 2hrs before and after the sleep per accelerometer that contains seven days/nights of data regardless of the number of sleep activities within the 24hrs. It is done by the *BuildDataSet()* and a column named “state” that represents if the subject was asleep or awake is included in the new dataset. The file is named as **ExtactTimeSeries_2hrs_labeled.parquet**. The extracted time series is checked by *CheckOverlapSeries()* to avoid overlapped buffer hours, so that each selected accelerometer contains eight hours of data.","metadata":{}},{"cell_type":"markdown","source":"# Model training\nAccording to signal theory, the awake period is a fractional Gaussian noise could be normalized to a standard Gaussian distribution, while the sleep period could be perceived as a step function. The signals vary significantly to a standard Gaussian mean and variance, these parameters allow us to identify which period a signal refers to, therefore we calculate the rolling average and standard deviation of the periods with the function *RollingAvg()*. Our chosen window size and step size are 540 and 360 correspondingly, which refers to 45mins and 30mins interval. We choose this as the state is identified to be changed if it lasts for 30mins or more. The file generated is named as training set.parquet.\n\nWe use cross validation to train our baseline model. The model runs for 0.06s and has an accuracy of 0.7062.","metadata":{}},{"cell_type":"markdown","source":"# References\n[1]\tV.T. van Hees, S. Sabia, S.E. Jones, A.R. Wood, K.N. Anderson, M. Kivimaki, T.M. Frayling, A.I. Pack, M. Bucan, M.I. Trenell, D.R. Mazzotti, P.R. Gehrman, B.A. Singh-Manoux & M.N. Weedon, “Estimating sleep parameters using an accelerometer without sleep diary,” *Sci. Rep.*, vol. 8, Aug. 2018. Accessed: Nov. 23, 2023. doi:10.1038/s41598-018-31266-z. [Online]. Available: https://www.nature.com/articles/s41598-018-31266-z#citeas.
\n[2]\tS.E. Jones, V.T. van Hees, D.R. Mazzotti, P. Marques-Vidal, S. Sabia, A. van der Spek, H.S. Dashti, J. E, D. Kocevska, J. Tyrrell, R.N. Beaumont, M. Hillsdon, K.S. Ruth, M.A. Tuke, H. Yaghootkar, S.A. Sharp, Y. Ji, J.W. Harrison, R.M. Freathy, A. Murray, A.I. Luik, N. Amin, J.M. Lane, R. Saxena, M.K. Rutter, H. Tiemeier, Z. Kutalik, M. Kumari, T.M. Frayling, M.N. Weedon, P.R. Gehrman & A.R. Wood, “Genetic studies of accelerometer-based sleep measures yield new insights into human sleep behaviour,” *Nat, Commun.*, vol. 10, Apr. 2019. Accessed: Nov. 23, 2023. doi:10.1038/s41467-019-09576-1. [Online]. Available: https://www.nature.com/articles/s41467-019-09576-1#citeas.
\n[3]\tV.T. van Hees, S. Sabia, K.N. Anderson, S.J. Denton, J. Oliver, M. Catt, J.G. Abell, M. Kivimaki, M.I. Trenell & A. Singh-Manoux, “A novel, open access method to assess sleep duration using a wrist-worn accelerometer,” *PLoS One*, Nov. 2015. Accessed: Nov. 23, 2023. doi:10.1371/journal.pone.0142533. [Online]. Available: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0142533.\n","metadata":{}}]}