{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🚀 Speeding Up Sensor Data Analysis with Parquet & Polars\nWelcome! In this notebook, I will show how to convert the large `train.csv` file from the [CMI 2025 Competition](https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data) into an efficient, partitioned Parquet format.\n\nWe will be using the blazing-fast [Polars](https://pola-rs.github.io/polars-book/) library to:\n- Read the original CSV\n- Partition the data by `subject`\n- Save as Parquet in Hive-style format\n- Reload it efficiently for analysis or feature extraction\n\n📌 Inspired by [Ravi Ramakrishnan's informative notebook!](https://www.kaggle.com/code/ravi20076/cmi2025-data-v1) \n","metadata":{}},{"cell_type":"code","source":"import polars as pl\n\n# Load train.csv\ncsv_path = \"/kaggle/input/cmi-detect-behavior-with-sensor-data/train.csv\"\ndf = pl.read_csv(csv_path)\n\n# Quick shape and schema\nprint(f\"Rows: {df.shape[0]}, Columns: {df.shape[1]}\")\ndf.head(1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-08T08:13:00.907341Z","iopub.execute_input":"2025-06-08T08:13:00.907768Z","iopub.status.idle":"2025-06-08T08:13:06.608118Z","shell.execute_reply.started":"2025-06-08T08:13:00.907741Z","shell.execute_reply":"2025-06-08T08:13:06.607343Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This CSV has hundreds of thousands of rows, representing raw time-series sensor data collected from wearable devices.\n","metadata":{}},{"cell_type":"markdown","source":"## 🔍 3. Why Partitioning?\nCSV files are not optimized for repeated reads or partial access.\n\n📉 Problems with large CSVs:\n- Slow to read\n- High memory usage\n- Can't access subsets efficiently\n\n✅ **Solution**: Convert to [Parquet](https://parquet.apache.org/) — a compressed, columnar format, and partition it **by `subject`** using Hive-style storage.\n\nThis means:\n```\ntrain_parquet/\n├── subject=SUBJ_000206/\n│ └── 00000000.parquet\n├── subject=SUBJ_002923/\n│ └── 00000000.parquet\n...\n```\n\nTools like Polars will automatically detect this structure and let us filter/load just what we need.","metadata":{}},{"cell_type":"markdown","source":"## 🛠️ 4. Convert CSV → Partitioned Parquet","metadata":{}},{"cell_type":"code","source":"# Output path\noutput_path = \"train_parquet/\"\n\n# Save as Parquet, partitioned by \"subject\" column values\ndf.write_parquet(\n    output_path, \n    partition_by=[\"subject\"]\n)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-08T07:44:02.917522Z","iopub.execute_input":"2025-06-08T07:44:02.917868Z","iopub.status.idle":"2025-06-08T07:44:10.374216Z","shell.execute_reply.started":"2025-06-08T07:44:02.917844Z","shell.execute_reply":"2025-06-08T07:44:10.373219Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"✅ Done! We now have 81 folders (one per subject), each with its own compact Parquet file.\n","metadata":{}},{"cell_type":"markdown","source":"## ⚡ 5. Lazy Loading Data with Polars\nIn Polars, we have two main ways to read Parquet files:\n\n- `pl.read_parquet()`  Eager Loading: Reads the entire data into memory \n- `pl.scan_parquet()`  Lazily Loading: Loads metadata first, defers actual reading\n\n### 🔍 Why Lazy Loading?\n- Fast metadata access (no full read)\n\n- Lazy pipelines (filter, group, select)\n\n- Loads only needed data — great for large, partitioned datasets\n","metadata":{}},{"cell_type":"code","source":"%%time\ndf_lazy = pl.scan_parquet(\"train_parquet/\")\n\n\n# Example: load one subject's data\ndf_subject_0 = pl.read_parquet(\"train_parquet/subject=SUBJ_000206/*.parquet\")\ndf_subject_0.head()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-08T07:59:03.262851Z","iopub.execute_input":"2025-06-08T07:59:03.263137Z","iopub.status.idle":"2025-06-08T07:59:03.343538Z","shell.execute_reply.started":"2025-06-08T07:59:03.263119Z","shell.execute_reply":"2025-06-08T07:59:03.342235Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This is faster, uses less RAM, and is perfect for looping over subjects or extracting features on demand.\n### 🔁 Example: Filter + Aggregate\n","metadata":{}},{"cell_type":"code","source":"# Lazy load + compute mean acceleration per sequence\nfeatures = (\n    df_lazy\n    .group_by(\"sequence_id\")\n    .agg([\n        pl.col(\"acc_x\").mean().alias(\"acc_x_mean\"),\n        pl.col(\"acc_y\").mean().alias(\"acc_y_mean\"),\n        pl.col(\"acc_z\").mean().alias(\"acc_z_mean\"),\n    ])\n)\n\nfeatures_df = features.collect()\nfeatures_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-08T08:03:22.723909Z","iopub.execute_input":"2025-06-08T08:03:22.724228Z","iopub.status.idle":"2025-06-08T08:03:22.856685Z","shell.execute_reply.started":"2025-06-08T08:03:22.724205Z","shell.execute_reply":"2025-06-08T08:03:22.855797Z"}},"outputs":[],"execution_count":null}]}