{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"As you may notive when loading the data provided, you are kind of stuck when loading it, as they are very large. The idea of this notebook is to create a dataframe and reducing the data provided.\n\nLet's see how to manage that together.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nTRAIN_EVENT = \"/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv\"\nTRAIN_SERIES = \"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\"\n\nTRAIN_EVENT_MEMORY  = \"/kaggle/working/train_events.csv\"\nTRAIN_SERIES_MEMORY = \"/kaggle/working/train_series.csv\"","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-09-16T07:59:56.475711Z","iopub.execute_input":"2023-09-16T07:59:56.476091Z","iopub.status.idle":"2023-09-16T07:59:56.926470Z","shell.execute_reply.started":"2023-09-16T07:59:56.476061Z","shell.execute_reply":"2023-09-16T07:59:56.925239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first dataframe is easy to simpify as we can simply load it in memory and to the memory reduction simply by applying dataframe function.","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(TRAIN_EVENT)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:56.928915Z","iopub.execute_input":"2023-09-16T07:59:56.929421Z","iopub.status.idle":"2023-09-16T07:59:56.996306Z","shell.execute_reply.started":"2023-09-16T07:59:56.929389Z","shell.execute_reply":"2023-09-16T07:59:56.994920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One optimization possible here is to reduce the `series_id` memory. We do not need to store for each data a large id, we simply need a unique id. We can also notice that we have 277 unique id in our dataset, so we can reduce the memory taken by using `uint16` (value max 65535). \n\nTo do that we will need to realize a mapping between the previous id and the new one.","metadata":{}},{"cell_type":"code","source":"unique_series_id = df_train.series_id.unique()\n\nprint(f\"Number of unique id: {len(unique_series_id)}\")\n\nmapping_series_id = {unique_series_id[i]: i for i in range(len(unique_series_id))}","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:56.998148Z","iopub.execute_input":"2023-09-16T07:59:56.998608Z","iopub.status.idle":"2023-09-16T07:59:57.013911Z","shell.execute_reply.started":"2023-09-16T07:59:56.998573Z","shell.execute_reply":"2023-09-16T07:59:57.012345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also optimize the storage for the night as it is a number and the values are between [1 ; 84]","metadata":{}},{"cell_type":"code","source":"print(f\"Value range for the nights: [{df_train.night.min()} ; {df_train.night.max()}]\")","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:57.017926Z","iopub.execute_input":"2023-09-16T07:59:57.018424Z","iopub.status.idle":"2023-09-16T07:59:57.030507Z","shell.execute_reply.started":"2023-09-16T07:59:57.018378Z","shell.execute_reply":"2023-09-16T07:59:57.028895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, regarding the event, we only have two events available: `onset`, `wakeup`.\nInstead of storing them as a string, we can store them as an int or even a boolean in this case.","metadata":{}},{"cell_type":"code","source":"from enum import Enum\n\nclass Event(Enum):\n    ONSET  = 0\n    WAKEUP = 1\n\nevent_mapping = {i.name.lower(): i.value for i in Event}","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:57.032850Z","iopub.execute_input":"2023-09-16T07:59:57.033750Z","iopub.status.idle":"2023-09-16T07:59:57.043957Z","shell.execute_reply.started":"2023-09-16T07:59:57.033704Z","shell.execute_reply":"2023-09-16T07:59:57.042621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.step.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:57.045774Z","iopub.execute_input":"2023-09-16T07:59:57.046921Z","iopub.status.idle":"2023-09-16T07:59:57.062002Z","shell.execute_reply.started":"2023-09-16T07:59:57.046877Z","shell.execute_reply":"2023-09-16T07:59:57.060735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, we have undefined values. This concerns the `step` and the `timestamp` columns. As we want to reduce momery space and we do not want to prepare the data, we have decided to not optimized these two columns, as we could maybe lose information if we wanted to reconstruct them later.","metadata":{}},{"cell_type":"code","source":"df_train.series_id = df_train.series_id.map(mapping_series_id).astype(np.uint16)\ndf_train.night     = df_train.night.astype(np.uint8)\ndf_train.event     = df_train.event.map(event_mapping).astype(np.uint8)\n# df_train.step      = df_train.step.astype(np.uint32)\n# df_train.timestamp = df_train.to_datetime(df.timestamp, utc=True)\n\ndf_train.to_csv(TRAIN_EVENT_MEMORY, index=False)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:57.063989Z","iopub.execute_input":"2023-09-16T07:59:57.064856Z","iopub.status.idle":"2023-09-16T07:59:57.171515Z","shell.execute_reply.started":"2023-09-16T07:59:57.064810Z","shell.execute_reply":"2023-09-16T07:59:57.170352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train series parquet\n\nThis one is a bit more complicated to optimized, as it is a large file. A solution for that is to load it chunk by chunk. And on each chunck, we are going to optimize the data memory and save it to another dataframe. This technique allows us to do treatment on our data without having to load all the dataset. \n","metadata":{}},{"cell_type":"code","source":"import pyarrow.parquet as pq\n\nparquet_file = pq.ParquetFile(TRAIN_SERIES)\n\nwith open(TRAIN_SERIES_MEMORY, 'a') as f:\n    \n    # Load chunk by chunk our data\n    for data in parquet_file.iter_batches(batch_size=100_000):\n        df = data.to_pandas()\n        \n        df.series_id = df.series_id.map(mapping_series_id).astype(np.uint16)\n        df.step      = df.step.astype(np.uint32)\n        df.timestamp = pd.to_datetime(df.timestamp, utc=True)\n        \n        df.to_csv(f, mode='a', header=f.tell()==0, index=False)","metadata":{"execution":{"iopub.status.busy":"2023-09-16T07:59:57.172687Z","iopub.execute_input":"2023-09-16T07:59:57.173001Z","iopub.status.idle":"2023-09-16T08:00:28.209043Z","shell.execute_reply.started":"2023-09-16T07:59:57.172963Z","shell.execute_reply":"2023-09-16T08:00:28.207051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the data\n# df_train_series = pd.read_csv(TRAIN_SERIES_MEMORY, dtype={\n#     \"series_id\": np.uint16,\n#     \"step\": np.uint32,\n# #     \"anglez\": \n# #     \"enmo\"\n# }, parse_dates=[\"timestamp\"])\n\n# df_train_series.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-16T08:01:43.639191Z","iopub.execute_input":"2023-09-16T08:01:43.639621Z","iopub.status.idle":"2023-09-16T08:02:00.369781Z","shell.execute_reply.started":"2023-09-16T08:01:43.639586Z","shell.execute_reply":"2023-09-16T08:02:00.368702Z"},"trusted":true},"execution_count":null,"outputs":[]}]}