{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <p style=\"font-family:JetBrains Mono; font-weight:bold; letter-spacing: 2px; font-size:120%; text-align:left;padding: 0px; border-bottom: 3px solid \">Child Mind Institute - Detect Sleep States</p>","metadata":{}},{"cell_type":"markdown","source":"## Competiton Understanding\nAt first glance this competition has a lot of moving parts, making it hard to understand how to approach. In this notebook I'm going through some of my thought processes to understand the competition and it's data.\nInitial thoughts : \n- **Goal of the Competition** - To detect sleep onset and wake. You will develop a model trained on wrist-worn accelerometer data in order to determine a person's sleep state.\n- **What does the data look like** -  Multi-day recordings of wrist-worn accelerometer data annotated with two event types: onset, the beginning of sleep, and wakeup, the end of sleep.\n- **How are we evaluated?** - Submissions are evaluated on the average precision of detected events, averaged over timestamp error tolerance thresholds, averaged over event classes.","metadata":{}},{"cell_type":"code","source":"# packages\n\n# standard\nimport numpy as np\nimport pandas as pd\nimport time\nfrom tabulate import tabulate\n# plots\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:58:28.466808Z","iopub.execute_input":"2023-09-08T10:58:28.467441Z","iopub.status.idle":"2023-09-08T10:58:28.477650Z","shell.execute_reply.started":"2023-09-08T10:58:28.467369Z","shell.execute_reply":"2023-09-08T10:58:28.475626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load all the datsets","metadata":{}},{"cell_type":"code","source":"# load csv data\ndf_train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv') \n\n# Converting the timestamp to datetime format\ndf_train_events['date'] = df_train_events['timestamp'].str.split('T', expand=True)[0]\ndf_train_events['time'] = df_train_events['timestamp'].str.split('T', expand=True)[1].str.split('-', expand=True)[0]\ndf_train_events['timestamp'] = pd.to_datetime(df_train_events['date']+' '+df_train_events['time'])\ndf_train_events['hour'] = df_train_events['timestamp'].dt.hour\n\n# load parquet data\ndf_train_pq = pd.read_parquet('../input/child-mind-institute-detect-sleep-states/train_series.parquet')\n\ndf_test_pq = pd.read_parquet('../input/child-mind-institute-detect-sleep-states/test_series.parquet')\n\n# loading the sample submission\nsample_submission = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/sample_submission.csv') ","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:58:28.853045Z","iopub.execute_input":"2023-09-08T10:58:28.853493Z","iopub.status.idle":"2023-09-08T10:59:58.143746Z","shell.execute_reply.started":"2023-09-08T10:58:28.853455Z","shell.execute_reply":"2023-09-08T10:59:58.141730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_events.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.147197Z","iopub.execute_input":"2023-09-08T10:59:58.147705Z","iopub.status.idle":"2023-09-08T10:59:58.157630Z","shell.execute_reply.started":"2023-09-08T10:59:58.147666Z","shell.execute_reply":"2023-09-08T10:59:58.155746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_events.head() ","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.159811Z","iopub.execute_input":"2023-09-08T10:59:58.160380Z","iopub.status.idle":"2023-09-08T10:59:58.188240Z","shell.execute_reply.started":"2023-09-08T10:59:58.160329Z","shell.execute_reply":"2023-09-08T10:59:58.186628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_pq.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.192334Z","iopub.execute_input":"2023-09-08T10:59:58.194995Z","iopub.status.idle":"2023-09-08T10:59:58.221983Z","shell.execute_reply.started":"2023-09-08T10:59:58.194910Z","shell.execute_reply":"2023-09-08T10:59:58.220452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test_pq.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.223802Z","iopub.execute_input":"2023-09-08T10:59:58.225815Z","iopub.status.idle":"2023-09-08T10:59:58.236620Z","shell.execute_reply.started":"2023-09-08T10:59:58.225758Z","shell.execute_reply":"2023-09-08T10:59:58.234525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test_pq.tail()","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.239011Z","iopub.execute_input":"2023-09-08T10:59:58.239740Z","iopub.status.idle":"2023-09-08T10:59:58.260796Z","shell.execute_reply.started":"2023-09-08T10:59:58.239695Z","shell.execute_reply":"2023-09-08T10:59:58.258908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### First level insights\n- Train event csv will be used to generate the label data for train parquet.\n- It will be a binary(Awake, onset) or 3 event classification problem(awake, onset,other)\n- We will have to look at individual series for detailed understanding about the data.\n","metadata":{}},{"cell_type":"markdown","source":"**The below function will provide the initial understanding about an individual series**","metadata":{}},{"cell_type":"code","source":"def get_ind_serties_info(series_id, event_train_data, pq_train_data, pq_test_data):\n    print(\"Data Analysis for series : \",series_id)\n    print(\"\")\n    \n    ind_event_data = event_train_data[event_train_data[\"series_id\"] == series_id]\n    ind_train_pq_data = pq_train_data[pq_train_data[\"series_id\"] == series_id]\n    ind_test_pq_data = pq_test_data[pq_test_data[\"series_id\"] == series_id]\n    \n    print(\"Event data\")\n    print(tabulate(ind_event_data.head(), headers = list(ind_event_data.columns)))\n\n    print(\"\")\n    print(\"Pq train data\")\n    print(tabulate(ind_train_pq_data.head(), headers = list(ind_train_pq_data.columns)))\n    print(\"\")\n    \n    print(\"Pq test data\")\n    print(tabulate(ind_test_pq_data.head(), headers = list(ind_test_pq_data.columns)))\n    print(\"\")\n    \n    print(\"Event data shape for series {0} : {1}\".format(series_id,ind_event_data.shape))\n    print(\"Pq train data shape for series {0} : {1}\".format(series_id,ind_train_pq_data.shape))\n    print(\"Pq test data shape for series {0} : {1}\".format(series_id,ind_test_pq_data.shape))\n    print(\"\")\n    print(\"Start date time of event data for series {0} : {1}\".format(series_id,ind_event_data[\"timestamp\"].min()))\n    print(\"End date time of event  data for series {0} : {1}\".format(series_id,ind_event_data[\"timestamp\"].max()))\n    print(\"\")\n    print(\"Start date time of pq train data for series {0} : {1}\".format(series_id,ind_train_pq_data[\"timestamp\"].min()))\n    print(\"End date time of pq train data for series {0} : {1}\".format(series_id,ind_train_pq_data[\"timestamp\"].max()))\n    print(\"\")\n    print(\"Start date time of pq test data for series {0} : {1}\".format(series_id,ind_test_pq_data[\"timestamp\"].min()))\n    print(\"End date time of pq test data for series {0} : {1}\".format(series_id,ind_test_pq_data[\"timestamp\"].max()))\n    return ind_event_data, ind_train_pq_data\n","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.262969Z","iopub.execute_input":"2023-09-08T10:59:58.264384Z","iopub.status.idle":"2023-09-08T10:59:58.282489Z","shell.execute_reply.started":"2023-09-08T10:59:58.264325Z","shell.execute_reply":"2023-09-08T10:59:58.280672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ind_event_data,ind_train_pq_data = get_ind_serties_info('038441c925bb', df_train_events, df_train_pq, df_test_pq)","metadata":{"execution":{"iopub.status.busy":"2023-09-08T10:59:58.284530Z","iopub.execute_input":"2023-09-08T10:59:58.286206Z","iopub.status.idle":"2023-09-08T11:00:23.025154Z","shell.execute_reply.started":"2023-09-08T10:59:58.286151Z","shell.execute_reply":"2023-09-08T11:00:23.023292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Insights\n- We have around 46 records in the event dataset which states that we have around 23 days of data \n    - We have one event for ofset and one event for wakeup so in that way we have 2 events each day\n- We have data at an interval of 5 seconds so we have 86400 seconds each day. \n    - for 23 days we have 86400*23 = 1987200\n    - As we are taking data on every 5 seconds so total records in train pq = 1987200/5 = 397440 (which is approximately equal to the number of records in train pq for the series","metadata":{}},{"cell_type":"code","source":"# ind_event_data,ind_train_pq_data = get_ind_serties_info('fe90110788d2', df_train_events, df_train_pq, df_test_pq)","metadata":{"execution":{"iopub.status.busy":"2023-09-08T11:06:42.730634Z","iopub.execute_input":"2023-09-08T11:06:42.731809Z","iopub.status.idle":"2023-09-08T11:06:42.738859Z","shell.execute_reply.started":"2023-09-08T11:06:42.731759Z","shell.execute_reply":"2023-09-08T11:06:42.737147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Now lets check for how many series do we have the label data in Train","metadata":{"execution":{"iopub.status.busy":"2023-09-08T09:52:11.143964Z","iopub.execute_input":"2023-09-08T09:52:11.145276Z","iopub.status.idle":"2023-09-08T09:52:11.151339Z","shell.execute_reply.started":"2023-09-08T09:52:11.145223Z","shell.execute_reply":"2023-09-08T09:52:11.149993Z"}}},{"cell_type":"code","source":"# base statistics\ndf_train_events.describe(include='all')","metadata":{"execution":{"iopub.status.busy":"2023-09-08T11:28:03.591761Z","iopub.execute_input":"2023-09-08T11:28:03.592389Z","iopub.status.idle":"2023-09-08T11:28:03.669095Z","shell.execute_reply.started":"2023-09-08T11:28:03.592347Z","shell.execute_reply":"2023-09-08T11:28:03.668002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Initial insights\n- As we can see we have nan values in the step column which means that we can not use those instances for generating the label.\n- So for initial understanding we will choose only those series which does not have any missing steps events","metadata":{}},{"cell_type":"code","source":"series_has_NaN = df_train_events.groupby('series_id')['step'].apply(lambda x: x.isnull().any())\nseries_has_NaN.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-09-08T11:31:53.690997Z","iopub.execute_input":"2023-09-08T11:31:53.692253Z","iopub.status.idle":"2023-09-08T11:31:53.756840Z","shell.execute_reply.started":"2023-09-08T11:31:53.692199Z","shell.execute_reply":"2023-09-08T11:31:53.755465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see we have 37 series which do not have any missing steps","metadata":{}},{"cell_type":"code","source":"no_NaN_series = series_has_NaN[~series_has_NaN].index.tolist()\nprint(no_NaN_series)","metadata":{"execution":{"iopub.status.busy":"2023-09-08T11:32:32.992350Z","iopub.execute_input":"2023-09-08T11:32:32.992801Z","iopub.status.idle":"2023-09-08T11:32:33.000717Z","shell.execute_reply.started":"2023-09-08T11:32:32.992763Z","shell.execute_reply":"2023-09-08T11:32:32.999096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## For detailed EDA and data understanding you can refer to my other notebook\n[https://www.kaggle.com/code/rishabh15virgo/cmi-dss-first-impression-data-understanding-eda](http://)\n\n***If you like this notebook do upvote.***","metadata":{}},{"cell_type":"markdown","source":"### In the coming notebooks I will generate the data both for 37 non null series as well as all non null events and will build the baseline model.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}