{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Initial Data Exploration\n\nThis notebooks loads all available datasets and checks their structure. It also introduces basic initial visualisations of the train data.","metadata":{}},{"cell_type":"code","source":"import os\nimport psutil\n\nimport polars as pl\nimport pandas as pd\n\npd.options.plotting.backend = 'plotly'","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-07T23:16:35.471128Z","iopub.execute_input":"2023-10-07T23:16:35.471501Z","iopub.status.idle":"2023-10-07T23:16:35.971274Z","shell.execute_reply.started":"2023-10-07T23:16:35.471474Z","shell.execute_reply":"2023-10-07T23:16:35.970027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Loading","metadata":{}},{"cell_type":"code","source":"INPUT_DIR = '/kaggle/input'\n# Directory with 20GB of available space, is preserved between sessions\nWORKING_DIR = 'kaggle/working'\n# Not preserved between sessions\nTEMP_DIR = 'kaggle/temp/'\n\nDATASET = 'child-mind-institute-detect-sleep-states'","metadata":{"execution":{"iopub.status.busy":"2023-10-07T23:16:41.871660Z","iopub.execute_input":"2023-10-07T23:16:41.872218Z","iopub.status.idle":"2023-10-07T23:16:41.877983Z","shell.execute_reply.started":"2023-10-07T23:16:41.872186Z","shell.execute_reply":"2023-10-07T23:16:41.876582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a LazyFrame using Polars\ntrain_series_lazy = (\n    pl.scan_parquet(os.path.join(INPUT_DIR, DATASET, 'train_series.parquet'))\n)\n\n# Process the timestamp into new columns\ntrain_series_lazy = train_series_lazy.with_columns(\n                    (\n    (pl.col(\"timestamp\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%S%Z\")))\n                )\n# Create date and time coluns\ntrain_series_lazy = train_series_lazy.with_columns(\n    (pl.col(\"timestamp\").dt.time().alias(\"time\")),\n    (pl.col(\"timestamp\").dt.date().alias(\"date\"))\n).sort(pl.col(\"timestamp\"), descending=False)\n\ntrain_series = train_series.collect().to_pandas()\n\nprint(\"Successfully loaded training data\")\n\n# test_series = pd.read_parquet(\n#    os.path.join(INPUT_DIR, DATASET, 'test_series.parquet')\n# )\n# print(\"Successfully loaded test data\")\n\ntrain_events = pd.read_csv(\n   os.path.join(INPUT_DIR, DATASET, 'train_events.csv')\n)\nprint(\"Successfully loaded annotations\")\n\n# sample_submission = pd.read_csv(\n#    os.path.join(INPUT_DIR, DATASET, 'sample_submission.csv')\n# )\n# print(\"Successfully loaded sample submission\")\n\nprint(\"Successfully loaded all datasets!\")","metadata":{"execution":{"iopub.status.busy":"2023-10-07T23:37:51.130569Z","iopub.execute_input":"2023-10-07T23:37:51.131003Z","iopub.status.idle":"2023-10-07T23:39:25.233175Z","shell.execute_reply.started":"2023-10-07T23:37:51.130965Z","shell.execute_reply":"2023-10-07T23:39:25.231714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"memory_usage = psutil.virtual_memory()\nprint(f'After loading all data, memory usage is at {memory_usage.total >> 30} GB')","metadata":{"execution":{"iopub.status.busy":"2023-10-07T23:41:33.030582Z","iopub.execute_input":"2023-10-07T23:41:33.031177Z","iopub.status.idle":"2023-10-07T23:41:33.039996Z","shell.execute_reply.started":"2023-10-07T23:41:33.031133Z","shell.execute_reply":"2023-10-07T23:41:33.038379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training Data Analysis","metadata":{}},{"cell_type":"code","source":"print(\"Train data shape:\")\nprint(train_series.shape)\ntrain_series.head()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-07T23:33:56.189264Z","iopub.execute_input":"2023-10-07T23:33:56.189654Z","iopub.status.idle":"2023-10-07T23:35:43.705637Z","shell.execute_reply.started":"2023-10-07T23:33:56.189624Z","shell.execute_reply":"2023-10-07T23:35:43.703960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have around 130 M rows, with 5 columns. The sampling time seems to be 5 seconds.\n\nFrom the documentation, we know that the columns can be interpreted as:\n\n\n* `series_id` - Unique identifier for each accelerometer series.\n* `step` - An integer timestep for each observation within a series.\n* `timestamp` - A corresponding datetime with ISO 8601 format %Y-%m-%dT%H:%M:%S%z.\n* `anglez` - As calculated and described by the GGIR package, z-angle is a metric derived from individual accelerometer components that is commonly used in sleep detection, and refers to the angle of the arm relative to the vertical axis of the body\n* `enmo` - As calculated and described by the GGIR package, ENMO is the Euclidean Norm Minus One of all accelerometer signals, with negative values rounded to zero. While no standard measure of acceleration exists in this space, this is one of the several commonly computed features\n\nNow, we investigate how many different series we have.","metadata":{}},{"cell_type":"code","source":"n_unique_series_train = train_series['series_id'].unique().shape[0]\nprint(f'We have {n_unique_series_train} series for training')","metadata":{"execution":{"iopub.status.busy":"2023-09-24T21:10:57.010178Z","iopub.execute_input":"2023-09-24T21:10:57.010624Z","iopub.status.idle":"2023-09-24T21:11:05.563969Z","shell.execute_reply.started":"2023-09-24T21:10:57.010569Z","shell.execute_reply":"2023-09-24T21:11:05.562796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What is the distribution of the lengths - or number of steps - of these series?","metadata":{}},{"cell_type":"code","source":"series_lengths = train_series[['series_id', 'step']].groupby('series_id').count()\n\nfig = series_lengths.hist()\nfig.update_layout(\n    showlegend=False,\n    title='Series Lenghts Distribution',\n    xaxis_title='Steps',\n    yaxis_title='Count'\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-24T21:11:05.565328Z","iopub.execute_input":"2023-09-24T21:11:05.565677Z","iopub.status.idle":"2023-09-24T21:11:23.450846Z","shell.execute_reply.started":"2023-09-24T21:11:05.565639Z","shell.execute_reply":"2023-09-24T21:11:23.449539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_lengths.describe()","metadata":{"execution":{"iopub.status.busy":"2023-09-24T21:11:23.452633Z","iopub.execute_input":"2023-09-24T21:11:23.453249Z","iopub.status.idle":"2023-09-24T21:11:23.469027Z","shell.execute_reply.started":"2023-09-24T21:11:23.453216Z","shell.execute_reply":"2023-09-24T21:11:23.467723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The statistics and sitribution of the series lengths show significant variation in the duration of each series. At this point, what delimits the start and end of a series is not clear yet.\n\nNow, the annotations are investigated.","metadata":{}},{"cell_type":"markdown","source":"### Annotations","metadata":{}},{"cell_type":"code","source":"train_events.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-24T21:11:23.470875Z","iopub.execute_input":"2023-09-24T21:11:23.471310Z","iopub.status.idle":"2023-09-24T21:11:23.496409Z","shell.execute_reply.started":"2023-09-24T21:11:23.471269Z","shell.execute_reply":"2023-09-24T21:11:23.495077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, the step associated with each event are annotated. Additionally, timestamps and night indices are provided.","metadata":{}},{"cell_type":"code","source":"# sample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-24T21:11:23.498966Z","iopub.execute_input":"2023-09-24T21:11:23.499344Z","iopub.status.idle":"2023-09-24T21:11:23.508299Z","shell.execute_reply.started":"2023-09-24T21:11:23.499313Z","shell.execute_reply":"2023-09-24T21:11:23.507424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test_series.head()\n# print(test_series.shape)","metadata":{"execution":{"iopub.status.busy":"2023-09-24T21:11:23.510319Z","iopub.execute_input":"2023-09-24T21:11:23.510755Z","iopub.status.idle":"2023-09-24T21:11:23.519334Z","shell.execute_reply.started":"2023-09-24T21:11:23.510724Z","shell.execute_reply":"2023-09-24T21:11:23.518441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, both train and test datasets have the same structure.","metadata":{}}]}