{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **FOREWORD**\n\nThis kernel aims to store the large training .csv files into parquet for effective data storage and quick retrieval. Hope this is helpful for one and all! \n\nThe training device data is 1.2GB large and one could find it slightly cumbersome to work with such a large .csv file regularly. Parquets offer lazy execution in conjunction with polars and hive based data storage with partitions as well, making them an obvious choice for such large dataset assignments. \n\nLet's get started with polars 1.29.0 and parquet storage!\n\nI have partitioned the data here on **subject ID** to obtain 81 partitions for a hive-storage!\n\n\n## **PACKAGES**\n\nPlease peruse my adjutant kernel [here](https://www.kaggle.com/code/ravi20076/cmi2025-public-imports-v1) for a comprehensive list of downloaded packages and starter scripts\n\n## **USING THE PARTITIONED DATASET**\n\n- Polars has an inbuilt mechanism to interact with hive storage and can work seamlessly with such partitioned data\n- Just use polars.read_parquet() / polars.scan_parquet() and you are good to go!\n- You can also tweek the parameters in these methods to read in low memory mode and in chunks as well! \n","metadata":{}},{"cell_type":"markdown","source":"# **IMPORTS AND INSTALLATIONS**","metadata":{}},{"cell_type":"code","source":"!pip install -q polars==1.29.0\n\nimport numpy as np, pandas as pd, polars as pl\nfrom gc import collect\nfrom colorama import Fore, Back, Style\nfrom tqdm.notebook import tqdm\n\ndef PrintColor(text: str, color = Fore.BLUE, style = Style.BRIGHT) :\n    print( color + style + text + Style.RESET_ALL)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **DATA LOADING AND PROCESSING**","metadata":{}},{"cell_type":"code","source":"%%time \n\ntrain      = \\\n(\n    pl.read_csv(f\"/kaggle/input/cmi-detect-behavior-with-sensor-data/train.csv\").\n    select(pl.all().shrink_dtype())\n)\ntrain_demo = pl.read_csv(f\"/kaggle/input/cmi-detect-behavior-with-sensor-data/train_demographics.csv\")\n\nPrintColor(f\"---> Columns in device data-  train\\n\")\nwith np.printoptions(linewidth = 100, threshold = 10000) :\n    print(np.array(train.collect_schema().names()))\n\ntrain.write_parquet(\"train.parquet\", partition_by = \"subject\")\n\nprint(\n    f\"\\n\\n---> Estimated memory for the training data = {train.estimated_size() / 10**6} Mb\\n\"\n)\n\ntrain_demo.write_csv(f\"train_demo.csv\")\n\ntest = \\\n(\n    pl.read_csv(f\"/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv\").\n    select(pl.all().shrink_dtype())\n)\ntest_demo = pl.read_csv(f\"/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv\")\n\ntest.write_parquet(\"test.parquet\", partition_by = \"subject\")\ntest_demo.write_csv(f\"test_demo.csv\")\n\ncollect()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **BASIC NULLS ANALYSIS**","metadata":{}},{"cell_type":"code","source":"nulls_ = train.null_count() / train.height\nnulls_ = nulls_.to_pandas().transpose().squeeze()\n\nPrintColor(f\"---> Top 25 columns by null values\")\ndisplay(\n    nulls_.sort_values(ascending = False).loc[nulls_ > 0].\n    head(25)\n)","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}