{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.13.15","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"84cb545c-4f02-40a1-908a-ef740a0dd81a","cell_type":"markdown","source":"# Energy Consumption Analysis in Smart Buildings\n\n**Goal:** Identify energy usage patterns across buildings and group them by consumption behavior to support energy-optimization decisions.\n\n**Data:** ASHRAE Great Energy Predictor III — hourly meter readings, building metadata, and site-level weather data.\n\n**Pipeline:**\n1. Data loading & merging (meter + building + weather)\n2. Preprocessing (missing values, timestamps)\n3. Feature engineering (time features, per-building behavioral features)\n4. Exploratory data analysis\n5. Weather impact analysis\n6. Dimensionality reduction with PCA\n7. K-Means clustering of buildings\n8. Cluster profiling & energy-optimization insights","metadata":{}},{"id":"cbeddc52-ec5e-4a03-aa70-d689e0ff9ad4","cell_type":"markdown","source":"## 1. Data Loading","metadata":{}},{"id":"7910e037-c637-4441-ab7d-46a514b6a4db","cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:04.186691Z","iopub.execute_input":"2026-10-06T16:34:04.187337Z","iopub.status.idle":"2026-10-06T16:34:04.192225Z","shell.execute_reply.started":"2026-10-06T16:34:04.187255Z","shell.execute_reply":"2026-10-06T16:34:04.191166Z"}},"outputs":[],"execution_count":null},{"id":"fec84902-bfe2-4253-9eb5-c4254f0c389d","cell_type":"code","source":"path = \"/kaggle/input/competitions/ashrae-energy-prediction/\"\n\nbuilding = pd.read_csv(path + \"building_metadata.csv\")\ntest     = pd.read_csv(path + \"test.csv\")\ntrain    = pd.read_csv(path + \"train.csv\")\nwtrain   = pd.read_csv(path + \"weather_train.csv\")\nwtest    = pd.read_csv(path + \"weather_test.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:04.193936Z","iopub.execute_input":"2026-10-06T16:34:04.194187Z","iopub.status.idle":"2026-10-06T16:34:53.183328Z","shell.execute_reply.started":"2026-10-06T16:34:04.194145Z","shell.execute_reply":"2026-10-06T16:34:53.182418Z"}},"outputs":[],"execution_count":null},{"id":"2466b013-c7e1-4617-a365-e7a5a1915bee","cell_type":"code","source":"building","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.185044Z","iopub.execute_input":"2026-10-06T16:34:53.185416Z","iopub.status.idle":"2026-10-06T16:34:53.233536Z","shell.execute_reply.started":"2026-10-06T16:34:53.185389Z","shell.execute_reply":"2026-10-06T16:34:53.232586Z"}},"outputs":[],"execution_count":null},{"id":"106c5748-d651-41f8-8e83-803b8f13579a","cell_type":"code","source":"test","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.234916Z","iopub.execute_input":"2026-10-06T16:34:53.235319Z","iopub.status.idle":"2026-10-06T16:34:53.246379Z","shell.execute_reply.started":"2026-10-06T16:34:53.235293Z","shell.execute_reply":"2026-10-06T16:34:53.245587Z"}},"outputs":[],"execution_count":null},{"id":"80edbb6c-be26-498b-9889-81cf182e854a","cell_type":"code","source":"train","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.248247Z","iopub.execute_input":"2026-10-06T16:34:53.248631Z","iopub.status.idle":"2026-10-06T16:34:53.270467Z","shell.execute_reply.started":"2026-10-06T16:34:53.248604Z","shell.execute_reply":"2026-10-06T16:34:53.269559Z"}},"outputs":[],"execution_count":null},{"id":"ba5cc6cf-0320-4b74-901a-1655ef6ea85a","cell_type":"code","source":"wtrain","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.271786Z","iopub.execute_input":"2026-10-06T16:34:53.272144Z","iopub.status.idle":"2026-10-06T16:34:53.304023Z","shell.execute_reply.started":"2026-10-06T16:34:53.272087Z","shell.execute_reply":"2026-10-06T16:34:53.303014Z"}},"outputs":[],"execution_count":null},{"id":"bd305e90-e57c-4247-ae68-800d3adf632c","cell_type":"code","source":"wtest","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.306181Z","iopub.execute_input":"2026-10-06T16:34:53.306609Z","iopub.status.idle":"2026-10-06T16:34:53.344201Z","shell.execute_reply.started":"2026-10-06T16:34:53.306569Z","shell.execute_reply":"2026-10-06T16:34:53.343283Z"}},"outputs":[],"execution_count":null},{"id":"1155a296-cae9-4dce-868a-812faf3aa24c","cell_type":"markdown","source":"## 2. Preprocessing\n\nWeather gaps are interpolated **per site, in time order, before merging**, so values are never filled using readings from a different site.","metadata":{}},{"id":"4c4a585e-8feb-4b1e-9f2e-0c83faf1e0de","cell_type":"code","source":"weather_cols = [\n    'air_temperature', 'cloud_coverage', 'dew_temperature',\n    'precip_depth_1_hr', 'sea_level_pressure',\n    'wind_direction', 'wind_speed'\n]\n\nfor w in (wtrain, wtest):\n    w.sort_values(['site_id', 'timestamp'], inplace=True)\n    w[weather_cols] = (\n        w.groupby('site_id')[weather_cols]\n         .transform(lambda s: s.interpolate(limit_direction='both'))\n    )\n\nwtrain.isna().mean()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.345477Z","iopub.execute_input":"2026-10-06T16:34:53.346039Z","iopub.status.idle":"2026-10-06T16:34:53.730439Z","shell.execute_reply.started":"2026-10-06T16:34:53.345996Z","shell.execute_reply":"2026-10-06T16:34:53.729611Z"}},"outputs":[],"execution_count":null},{"id":"4fb39e26-542a-4868-9c9d-e928c1ff57b7","cell_type":"code","source":"train2 = train.merge(building, on='building_id', how='left')\ntest2 = test.merge(building, on='building_id', how='left')\n\ntrain2 = train2.merge(wtrain, on=['site_id', 'timestamp'], how='left')\ntest2 = test2.merge(wtest, on=['site_id', 'timestamp'], how='left')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:34:53.731525Z","iopub.execute_input":"2026-10-06T16:34:53.731897Z","iopub.status.idle":"2026-10-06T16:35:29.831132Z","shell.execute_reply.started":"2026-10-06T16:34:53.731863Z","shell.execute_reply":"2026-10-06T16:35:29.830404Z"}},"outputs":[],"execution_count":null},{"id":"4a213bc5-4a2b-49fe-8129-ff3934890e04","cell_type":"code","source":"train2[\"timestamp\"] = pd.to_datetime(train2[\"timestamp\"])\ntest2[\"timestamp\"] = pd.to_datetime(test2[\"timestamp\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:29.832223Z","iopub.execute_input":"2026-10-06T16:35:29.832598Z","iopub.status.idle":"2026-10-06T16:35:38.497465Z","shell.execute_reply.started":"2026-10-06T16:35:29.832549Z","shell.execute_reply":"2026-10-06T16:35:38.496434Z"}},"outputs":[],"execution_count":null},{"id":"cdc159e6-697c-451a-a34b-e6d377f55384","cell_type":"code","source":"train2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:38.500404Z","iopub.execute_input":"2026-10-06T16:35:38.501205Z","iopub.status.idle":"2026-10-06T16:35:38.524648Z","shell.execute_reply.started":"2026-10-06T16:35:38.501177Z","shell.execute_reply":"2026-10-06T16:35:38.523984Z"}},"outputs":[],"execution_count":null},{"id":"1b73c989-3e37-487e-a1bb-9af5cc8227c0","cell_type":"markdown","source":"### Data availability (train vs test)","metadata":{}},{"id":"3882f83e-3eca-4417-ac91-5bc1c1325091","cell_type":"code","source":"train_data = (train2.count() / len(train2)).drop('meter_reading').sort_values().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='pink')\n\ntest_data = (test2.count() / len(test2)).drop('row_id').sort_values().values\ntt = axes.bar(ind+width, test_data, width, color='black')\n\naxes.set_ylabel('Amount of data available');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels((train2.count() / len(train2)).drop('meter_reading').sort_values().index, rotation=40)\naxes.legend([tr, tt], ['Train', 'Test']);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:38.526022Z","iopub.execute_input":"2026-10-06T16:35:38.52638Z","iopub.status.idle":"2026-10-06T16:35:45.35161Z","shell.execute_reply.started":"2026-10-06T16:35:38.526345Z","shell.execute_reply":"2026-10-06T16:35:45.350564Z"}},"outputs":[],"execution_count":null},{"id":"259a2e01-0638-4303-b61c-0261bf8471ff","cell_type":"code","source":"train2['square_feet'] = train2['square_feet'].fillna(\n    train2['square_feet'].median()\n)\n\ntrain2 = train2.dropna(subset=['meter_reading'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:45.352937Z","iopub.execute_input":"2026-10-06T16:35:45.353294Z","iopub.status.idle":"2026-10-06T16:35:48.789976Z","shell.execute_reply.started":"2026-10-06T16:35:45.35326Z","shell.execute_reply":"2026-10-06T16:35:48.789038Z"}},"outputs":[],"execution_count":null},{"id":"a4befa35-caf8-429f-86b2-ccaf650b8461","cell_type":"markdown","source":"## 3. Feature Engineering\n\n### Time features","metadata":{}},{"id":"73282306-7e14-4703-b820-8c0451ffe830","cell_type":"code","source":"train2['hour'] = train2['timestamp'].dt.hour\ntrain2['dayofweek'] = train2['timestamp'].dt.dayofweek\ntrain2['is_weekend'] = train2['dayofweek'].isin([5,6]).astype(int)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:48.791234Z","iopub.execute_input":"2026-10-06T16:35:48.791953Z","iopub.status.idle":"2026-10-06T16:35:50.18626Z","shell.execute_reply.started":"2026-10-06T16:35:48.791923Z","shell.execute_reply":"2026-10-06T16:35:50.185486Z"}},"outputs":[],"execution_count":null},{"id":"31ac17de-bdd5-4f11-b6c4-3019d63a02d8","cell_type":"code","source":"train2['is_day'] = train2['hour'].between(8,18).astype(int)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:50.187226Z","iopub.execute_input":"2026-10-06T16:35:50.187471Z","iopub.status.idle":"2026-10-06T16:35:50.387327Z","shell.execute_reply.started":"2026-10-06T16:35:50.187438Z","shell.execute_reply":"2026-10-06T16:35:50.386601Z"}},"outputs":[],"execution_count":null},{"id":"1e4f5f98-d5a5-4cab-ab44-6d0f1e1ec2b5","cell_type":"markdown","source":"### Per-building behavioral features\n\n- **mean / peak / std load** – overall consumption level and variability\n- **load_factor** = mean / peak – how flat vs. peaky the demand is (low = peaky)\n- **day_night_ratio** – dependence on working hours\n- **weekend_weekday_ratio** – dependence on the weekly schedule","metadata":{}},{"id":"d36653c0-1997-478b-9636-d9975ab1f40f","cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nbuilding_features = train2.groupby('building_id').apply(\n    lambda x: pd.Series({\n\n        'mean_load': x['meter_reading'].mean(),\n        'peak_load': x['meter_reading'].max(),\n        'load_std': x['meter_reading'].std(),\n\n        'square_feet': x['square_feet'].iloc[0],\n\n        'load_factor': x['meter_reading'].mean() / x['meter_reading'].max(),\n\n        'day_mean': x.loc[x['is_day']==1, 'meter_reading'].mean(),\n        'night_mean': x.loc[x['is_day']==0, 'meter_reading'].mean(),\n\n        'weekend_mean': x.loc[x['is_weekend']==1, 'meter_reading'].mean(),\n        'weekday_mean': x.loc[x['is_weekend']==0, 'meter_reading'].mean(),\n    })\n)\n\nbuilding_features['day_night_ratio'] = (\n    building_features['day_mean'] /\n    building_features['night_mean']\n)\n\nbuilding_features['weekend_weekday_ratio'] = (\n    building_features['weekend_mean'] /\n    building_features['weekday_mean']\n)\n\nbuilding_features = building_features.dropna()\nload_cols = [\n    'mean_load',\n    'peak_load',\n    'load_std',\n    'day_mean',\n    'night_mean',\n    'weekend_mean',\n    'weekday_mean'\n]\n\nbuilding_features.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:50.388543Z","iopub.execute_input":"2026-10-06T16:35:50.388903Z","iopub.status.idle":"2026-10-06T16:35:59.000269Z","shell.execute_reply.started":"2026-10-06T16:35:50.388867Z","shell.execute_reply":"2026-10-06T16:35:58.999463Z"}},"outputs":[],"execution_count":null},{"id":"6c9a47c5-db4f-43d3-97de-d576654d4113","cell_type":"markdown","source":"### Log transform\nLoad values are heavily right-skewed, so a log transform is applied before scaling.","metadata":{}},{"id":"03887852-3326-4fe1-b895-7743c09cde7e","cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\nfeature = building_features['peak_load']\n\nplt.figure(figsize=(12,5))\n\nplt.subplot(1,2,1)\nplt.hist(feature, bins=50)\nplt.title(\"Peak Load (Before Log Transform)\")\nplt.xlabel(\"Peak Load\")\nplt.ylabel(\"Frequency\")\n\nplt.subplot(1,2,2)\nplt.hist(np.log1p(feature), bins=50)\nplt.title(\"Peak Load (After Log Transform)\")\nplt.xlabel(\"Log(Peak Load)\")\nplt.ylabel(\"Frequency\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:59.001372Z","iopub.execute_input":"2026-10-06T16:35:59.001609Z","iopub.status.idle":"2026-10-06T16:35:59.414011Z","shell.execute_reply.started":"2026-10-06T16:35:59.001589Z","shell.execute_reply":"2026-10-06T16:35:59.413059Z"}},"outputs":[],"execution_count":null},{"id":"903c60c3-6679-4f0c-a195-73f1cbce5e8e","cell_type":"code","source":"for col in load_cols:\n    building_features[col] = np.log1p(building_features[col])\n\nbuilding_features","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:59.415252Z","iopub.execute_input":"2026-10-06T16:35:59.415839Z","iopub.status.idle":"2026-10-06T16:35:59.439346Z","shell.execute_reply.started":"2026-10-06T16:35:59.415809Z","shell.execute_reply":"2026-10-06T16:35:59.438499Z"}},"outputs":[],"execution_count":null},{"id":"6e173eb7-2a16-426d-b484-b593d96c6c5b","cell_type":"code","source":"building_features.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:59.440806Z","iopub.execute_input":"2026-10-06T16:35:59.44124Z","iopub.status.idle":"2026-10-06T16:35:59.474522Z","shell.execute_reply.started":"2026-10-06T16:35:59.441203Z","shell.execute_reply":"2026-10-06T16:35:59.473798Z"}},"outputs":[],"execution_count":null},{"id":"50f83ed5-df98-46cf-91c1-45914823f447","cell_type":"code","source":"building_features.sort_values('day_mean', ascending=False).head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:59.47565Z","iopub.execute_input":"2026-10-06T16:35:59.476072Z","iopub.status.idle":"2026-10-06T16:35:59.493245Z","shell.execute_reply.started":"2026-10-06T16:35:59.476024Z","shell.execute_reply":"2026-10-06T16:35:59.492316Z"}},"outputs":[],"execution_count":null},{"id":"9ffcb4fc-ff52-42a6-a06d-66f595174241","cell_type":"markdown","source":"## 4. Exploratory Data Analysis","metadata":{}},{"id":"61178ee0-c467-498c-8ffd-c2087e9b4359","cell_type":"code","source":"train2['timestamp'] = pd.to_datetime(train2['timestamp'])\n\ntrain2['hour'] = train2['timestamp'].dt.hour\ntrain2['dayofweek'] = train2['timestamp'].dt.dayofweek\ntrain2['is_weekend'] = train2['dayofweek'].isin([5,6])\nhourly_pattern = (\n    train2.groupby('hour')['meter_reading']\n    .mean()\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:35:59.494405Z","iopub.execute_input":"2026-10-06T16:35:59.494842Z","iopub.status.idle":"2026-10-06T16:36:02.127012Z","shell.execute_reply.started":"2026-10-06T16:35:59.494809Z","shell.execute_reply":"2026-10-06T16:36:02.123675Z"}},"outputs":[],"execution_count":null},{"id":"55c1f669-5325-4388-a8c5-26f482c07d12","cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(10,5))\nplt.plot(hourly_pattern.index, hourly_pattern.values)\nplt.xlabel(\"Hour of Day\")\nplt.ylabel(\"Average Meter Reading\")\nplt.title(\"Average Hourly Energy Usage Pattern\")\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:02.131593Z","iopub.execute_input":"2026-10-06T16:36:02.13295Z","iopub.status.idle":"2026-10-06T16:36:02.590914Z","shell.execute_reply.started":"2026-10-06T16:36:02.132913Z","shell.execute_reply":"2026-10-06T16:36:02.587225Z"}},"outputs":[],"execution_count":null},{"id":"c8017d1d-729a-4e08-9ce4-218d8502aa95","cell_type":"code","source":"weekday_pattern = (\n    train2[~train2['is_weekend']]\n    .groupby('hour')['meter_reading']\n    .mean()\n)\n\nweekend_pattern = (\n    train2[train2['is_weekend']]\n    .groupby('hour')['meter_reading']\n    .mean()\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:02.595338Z","iopub.execute_input":"2026-10-06T16:36:02.596769Z","iopub.status.idle":"2026-10-06T16:36:05.554533Z","shell.execute_reply.started":"2026-10-06T16:36:02.596492Z","shell.execute_reply":"2026-10-06T16:36:05.553649Z"}},"outputs":[],"execution_count":null},{"id":"51e8064d-9e11-4410-8f77-410a7e7a59d7","cell_type":"code","source":"plt.figure(figsize=(10,5))\n\nplt.plot(weekday_pattern.index, weekday_pattern.values, label=\"Weekday\")\nplt.plot(weekend_pattern.index, weekend_pattern.values, label=\"Weekend\")\n\nplt.xlabel(\"Hour of Day\")\nplt.ylabel(\"Average Meter Reading\")\nplt.title(\"Weekday vs Weekend Energy Pattern\")\nplt.legend()\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:05.555618Z","iopub.execute_input":"2026-10-06T16:36:05.556099Z","iopub.status.idle":"2026-10-06T16:36:05.733814Z","shell.execute_reply.started":"2026-10-06T16:36:05.556072Z","shell.execute_reply":"2026-10-06T16:36:05.733074Z"}},"outputs":[],"execution_count":null},{"id":"7ba3e1d8-ad68-42e4-932c-8612b1f365e2","cell_type":"code","source":"weekday_avg = train2[~train2['is_weekend']]['meter_reading'].mean()\nweekend_avg = train2[train2['is_weekend']]['meter_reading'].mean()\n\nprint(\"Weekday Average:\", weekday_avg)\nprint(\"Weekend Average:\", weekend_avg)\nprint(\"Ratio (Weekend/Weekday):\", weekend_avg / weekday_avg)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:05.734881Z","iopub.execute_input":"2026-10-06T16:36:05.735201Z","iopub.status.idle":"2026-10-06T16:36:08.034203Z","shell.execute_reply.started":"2026-10-06T16:36:05.735179Z","shell.execute_reply":"2026-10-06T16:36:08.033416Z"}},"outputs":[],"execution_count":null},{"id":"6c15ab4d-f435-4fd3-9df7-9303d0bb6063","cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nplt.figure(figsize=(10,8))\nsns.heatmap(building_features.corr(),\n            cmap='coolwarm',\n            center=0,\n            annot=False)\n\nplt.title(\"Correlation Between Engineered Features\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:08.035408Z","iopub.execute_input":"2026-10-06T16:36:08.035832Z","iopub.status.idle":"2026-10-06T16:36:08.335551Z","shell.execute_reply.started":"2026-10-06T16:36:08.035807Z","shell.execute_reply":"2026-10-06T16:36:08.334867Z"}},"outputs":[],"execution_count":null},{"id":"afbf147f-e3c0-4a7e-8dfd-efebe62970ce","cell_type":"code","source":"sample_building = train2[train2['building_id'] == train2['building_id'].iloc[0]]\n\nplt.figure(figsize=(8,4))\nplt.plot(sample_building['timestamp'], sample_building['meter_reading'])\nplt.title(\"Raw Hourly Meter Reading (Single Building)\")\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:08.336619Z","iopub.execute_input":"2026-10-06T16:36:08.337048Z","iopub.status.idle":"2026-10-06T16:36:08.513682Z","shell.execute_reply.started":"2026-10-06T16:36:08.337023Z","shell.execute_reply":"2026-10-06T16:36:08.512583Z"}},"outputs":[],"execution_count":null},{"id":"070da0e4-03d6-420c-a5f1-200c307380bf","cell_type":"markdown","source":"## 5. Weather Impact Analysis\n\nHow does outdoor weather relate to energy consumption? Readings are aggregated to **daily** level to reduce hourly noise.","metadata":{}},{"id":"5c2cb1e5-3b3f-4d95-82ed-6f4cb075baea","cell_type":"code","source":"daily = (\n    train2\n    .groupby(['building_id', pd.Grouper(key='timestamp', freq='D')])\n    .agg(load=('meter_reading', 'mean'),\n         air_temperature=('air_temperature', 'mean'),\n         dew_temperature=('dew_temperature', 'mean'),\n         cloud_coverage=('cloud_coverage', 'mean'),\n         wind_speed=('wind_speed', 'mean'),\n         precip_depth_1_hr=('precip_depth_1_hr', 'mean'),\n         sea_level_pressure=('sea_level_pressure', 'mean'))\n    .reset_index()\n)\n\n# Spearman is rank-based, so it is robust to the very different load scales across buildings\nweather_corr = (\n    daily.drop(columns=['building_id', 'timestamp'])\n         .corr(method='spearman')['load']\n         .drop('load')\n         .sort_values()\n)\nprint(weather_corr)\n\nplt.figure(figsize=(8,4))\nweather_corr.plot(kind='barh')\nplt.title(\"Correlation of Weather Variables with Daily Energy Load\")\nplt.xlabel(\"Spearman correlation\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:08.515043Z","iopub.execute_input":"2026-10-06T16:36:08.515379Z","iopub.status.idle":"2026-10-06T16:36:14.631387Z","shell.execute_reply.started":"2026-10-06T16:36:08.515355Z","shell.execute_reply":"2026-10-06T16:36:14.630604Z"}},"outputs":[],"execution_count":null},{"id":"6f941b7e-572f-4b06-9bbc-13e99606ba58","cell_type":"code","source":"# Average load by outdoor temperature band\ntrain2['temp_bin'] = pd.cut(train2['air_temperature'], bins=range(-30, 50, 5))\ntemp_load = train2.groupby('temp_bin', observed=True)['meter_reading'].median()\n\nplt.figure(figsize=(10,5))\ntemp_load.plot(marker='o')\nplt.title(\"Median Energy Load vs Outdoor Temperature\")\nplt.xlabel(\"Air temperature band (°C)\")\nplt.ylabel(\"Median meter reading\")\nplt.xticks(rotation=45)\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:14.63542Z","iopub.execute_input":"2026-10-06T16:36:14.635854Z","iopub.status.idle":"2026-10-06T16:36:16.16708Z","shell.execute_reply.started":"2026-10-06T16:36:14.635828Z","shell.execute_reply":"2026-10-06T16:36:16.166154Z"}},"outputs":[],"execution_count":null},{"id":"9369202d-810c-44af-96ff-10a0656ea697","cell_type":"code","source":"# Temperature sensitivity of each building = correlation of its daily load with daily temperature\nbuilding_temp_sens = (\n    daily.groupby('building_id')\n         .apply(lambda d: d['load'].corr(d['air_temperature'], method='spearman'))\n         .rename('temp_sensitivity')\n)\n\nplt.figure(figsize=(8,4))\nplt.hist(building_temp_sens.dropna(), bins=40)\nplt.title(\"Distribution of Building Temperature Sensitivity\")\nplt.xlabel(\"Correlation of daily load with temperature\")\nplt.ylabel(\"Number of buildings\")\nplt.show()\n\nprint(\"Cooling-driven buildings (corr > 0.5):\", (building_temp_sens > 0.5).sum())\nprint(\"Heating-driven buildings (corr < -0.5):\", (building_temp_sens < -0.5).sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:16.168335Z","iopub.execute_input":"2026-10-06T16:36:16.16868Z","iopub.status.idle":"2026-10-06T16:36:17.82711Z","shell.execute_reply.started":"2026-10-06T16:36:16.168646Z","shell.execute_reply":"2026-10-06T16:36:17.826307Z"}},"outputs":[],"execution_count":null},{"id":"56251ea5-dc2c-495c-b9ec-5933ba333d8a","cell_type":"markdown","source":"## 6. Dimensionality Reduction (PCA)\n\nClustering uses **behavioral features only**. `square_feet` is excluded so buildings are grouped by *how* they consume energy, not just how big they are. It is kept aside for profiling the clusters afterwards.","metadata":{}},{"id":"97b6918e-bc0d-4cb7-bf30-3b8087aa159d","cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\n\nbehavior_features = building_features.drop(columns=['square_feet'])\n\nscaler = StandardScaler()\nX_scaled = scaler.fit_transform(behavior_features)\n\npca = PCA(n_components=0.90)\nX_pca = pca.fit_transform(X_scaled)\n\nprint(\"Number of input features:\", X_scaled.shape[1])\nprint(\"Components selected:\", pca.n_components_)\nprint(\"Explained variance:\", pca.explained_variance_ratio_)\nprint(\"Total variance retained:\", pca.explained_variance_ratio_.sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:17.82828Z","iopub.execute_input":"2026-10-06T16:36:17.828645Z","iopub.status.idle":"2026-10-06T16:36:18.469537Z","shell.execute_reply.started":"2026-10-06T16:36:17.828608Z","shell.execute_reply":"2026-10-06T16:36:18.468691Z"}},"outputs":[],"execution_count":null},{"id":"52e69d59-65c1-4859-a4bf-1f5d09ba81e7","cell_type":"code","source":"pca_full = PCA().fit(X_scaled)\n\nplt.plot(range(1, len(pca_full.explained_variance_ratio_) + 1),\n         np.cumsum(pca_full.explained_variance_ratio_), marker='o')\nplt.axhline(0.90, color='red', linestyle='--', label='90% threshold')\nplt.xlabel(\"Number of Components\")\nplt.ylabel(\"Cumulative Explained Variance\")\nplt.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:18.470791Z","iopub.execute_input":"2026-10-06T16:36:18.471792Z","iopub.status.idle":"2026-10-06T16:36:18.598546Z","shell.execute_reply.started":"2026-10-06T16:36:18.471764Z","shell.execute_reply":"2026-10-06T16:36:18.597792Z"}},"outputs":[],"execution_count":null},{"id":"66f1554d-27a7-4a92-9a7d-0d08d88f051e","cell_type":"code","source":"pca_loadings = pd.DataFrame(\n    pca.components_.T,\n    index=behavior_features.columns,\n    columns=[f'PC{i+1}' for i in range(pca.n_components_)]\n)\npca_loadings.sort_values('PC1', key=abs, ascending=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:18.599831Z","iopub.execute_input":"2026-10-06T16:36:18.60028Z","iopub.status.idle":"2026-10-06T16:36:18.612404Z","shell.execute_reply.started":"2026-10-06T16:36:18.600242Z","shell.execute_reply":"2026-10-06T16:36:18.611737Z"}},"outputs":[],"execution_count":null},{"id":"9d5d63ab-1dba-4167-9e0f-665fe6808333","cell_type":"markdown","source":"## 7. K-Means Clustering\n\nThe number of clusters is chosen using both the elbow method and the silhouette score.","metadata":{}},{"id":"d137a128-0d3b-4043-8bdc-7041fceec91d","cell_type":"code","source":"from sklearn.cluster import KMeans\nfrom sklearn.metrics import silhouette_score\n\ninertia, sil = [], []\nK = range(2, 9)\n\nfor k in K:\n    km = KMeans(n_clusters=k, random_state=42, n_init=10)\n    labels = km.fit_predict(X_pca)\n    inertia.append(km.inertia_)\n    sil.append(silhouette_score(X_pca, labels))\n\nfig, ax = plt.subplots(1, 2, figsize=(12,4))\nax[0].plot(K, inertia, marker='o'); ax[0].set_title(\"Elbow Method\"); ax[0].set_xlabel(\"k\"); ax[0].set_ylabel(\"Inertia\")\nax[1].plot(K, sil, marker='o'); ax[1].set_title(\"Silhouette Score\"); ax[1].set_xlabel(\"k\"); ax[1].set_ylabel(\"Score\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:18.613599Z","iopub.execute_input":"2026-10-06T16:36:18.614036Z","iopub.status.idle":"2026-10-06T16:36:20.161236Z","shell.execute_reply.started":"2026-10-06T16:36:18.614012Z","shell.execute_reply":"2026-10-06T16:36:20.160392Z"}},"outputs":[],"execution_count":null},{"id":"7b1b93a7-e2a6-4c5e-8e22-860678c4b70e","cell_type":"code","source":"kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)\nclusters = kmeans.fit_predict(X_pca)\n\nbuilding_features['cluster'] = clusters\nprint(\"Silhouette score (k=4):\", silhouette_score(X_pca, clusters))\nbuilding_features['cluster'].value_counts().sort_index()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:20.162566Z","iopub.execute_input":"2026-10-06T16:36:20.163102Z","iopub.status.idle":"2026-10-06T16:36:20.256856Z","shell.execute_reply.started":"2026-10-06T16:36:20.163075Z","shell.execute_reply":"2026-10-06T16:36:20.256226Z"}},"outputs":[],"execution_count":null},{"id":"0ab6298a-1331-48f6-8d23-176bef0b7646","cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.scatter(X_pca[:,0], X_pca[:,1], c=clusters, cmap='viridis')\nplt.xlabel(\"PC1\")\nplt.ylabel(\"PC2\")\nplt.title(\"Behavior-Based Building Clusters\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:20.257633Z","iopub.execute_input":"2026-10-06T16:36:20.258027Z","iopub.status.idle":"2026-10-06T16:36:20.470294Z","shell.execute_reply.started":"2026-10-06T16:36:20.257999Z","shell.execute_reply":"2026-10-06T16:36:20.469485Z"}},"outputs":[],"execution_count":null},{"id":"341311f3-f69e-4979-9c7e-df6c5fa92d89","cell_type":"markdown","source":"## 8. Cluster Profiling","metadata":{}},{"id":"2d4895ff-0b06-4736-a2f5-c1aab6df1828","cell_type":"code","source":"building_features = building_features.join(building_temp_sens)\ncluster_profile = building_features.groupby('cluster').mean()\ncluster_profile","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:20.471343Z","iopub.execute_input":"2026-10-06T16:36:20.472015Z","iopub.status.idle":"2026-10-06T16:36:20.492935Z","shell.execute_reply.started":"2026-10-06T16:36:20.471988Z","shell.execute_reply":"2026-10-06T16:36:20.492192Z"}},"outputs":[],"execution_count":null},{"id":"f4386bd4-f812-42c0-a410-ab21aca2d5be","cell_type":"code","source":"scaled_profile = pd.DataFrame(\n    StandardScaler().fit_transform(cluster_profile),\n    columns=cluster_profile.columns,\n    index=cluster_profile.index\n)\n\nscaled_profile.T.plot(kind='bar', figsize=(12,6))\nplt.title(\"Standardized Cluster Comparison\")\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:20.49399Z","iopub.execute_input":"2026-10-06T16:36:20.494223Z","iopub.status.idle":"2026-10-06T16:36:20.919535Z","shell.execute_reply.started":"2026-10-06T16:36:20.494203Z","shell.execute_reply":"2026-10-06T16:36:20.918718Z"}},"outputs":[],"execution_count":null},{"id":"1be905b5-69c5-437e-a3ed-0ce753d4b1b2","cell_type":"code","source":"train_clustered = train2.merge(\n    building_features[['cluster']],\n    left_on='building_id',\n    right_index=True,\n    how='left'\n)\nhourly_cluster = (\n    train_clustered\n    .groupby(['cluster', 'hour'])['meter_reading']\n    .mean()\n    .reset_index()\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:20.920853Z","iopub.execute_input":"2026-10-06T16:36:20.921199Z","iopub.status.idle":"2026-10-06T16:36:25.680118Z","shell.execute_reply.started":"2026-10-06T16:36:20.921166Z","shell.execute_reply":"2026-10-06T16:36:25.67939Z"}},"outputs":[],"execution_count":null},{"id":"df02cece-3531-4fbd-8b1d-782eaeba028b","cell_type":"code","source":"for c in sorted(hourly_cluster['cluster'].unique()):\n    plt.figure(figsize=(8,5))\n    \n    cluster_data = hourly_cluster[hourly_cluster['cluster'] == c]\n    plt.plot(cluster_data['hour'], cluster_data['meter_reading'])\n    \n    plt.xlabel(\"Hour\")\n    plt.ylabel(\"Average Load\")\n    plt.title(f\"Cluster {c} - Hourly Pattern\")\n    plt.grid(True)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:25.68115Z","iopub.execute_input":"2026-10-06T16:36:25.681449Z","iopub.status.idle":"2026-10-06T16:36:26.186883Z","shell.execute_reply.started":"2026-10-06T16:36:25.681421Z","shell.execute_reply":"2026-10-06T16:36:26.186069Z"}},"outputs":[],"execution_count":null},{"id":"c4114ddd-0bdb-47ed-9920-636d2c57ef1a","cell_type":"code","source":"peak_cluster = (\n    train_clustered\n    .groupby('cluster')['meter_reading']\n    .max()\n    .reset_index()\n)\n\npeak_cluster","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:26.188036Z","iopub.execute_input":"2026-10-06T16:36:26.188351Z","iopub.status.idle":"2026-10-06T16:36:26.549408Z","shell.execute_reply.started":"2026-10-06T16:36:26.188327Z","shell.execute_reply":"2026-10-06T16:36:26.548432Z"}},"outputs":[],"execution_count":null},{"id":"408b71f5-640b-4f1b-8f18-0db3dbd6473f","cell_type":"code","source":"plt.figure(figsize=(6,4))\n\nplt.bar(peak_cluster['cluster'],\n        peak_cluster['meter_reading'])\n\nplt.xlabel(\"Cluster\")\nplt.ylabel(\"Peak Demand\")\nplt.title(\"Peak Demand per Cluster\")\nplt.xticks(peak_cluster['cluster'])\nplt.grid(axis='y')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:26.550817Z","iopub.execute_input":"2026-10-06T16:36:26.551198Z","iopub.status.idle":"2026-10-06T16:36:26.685886Z","shell.execute_reply.started":"2026-10-06T16:36:26.551173Z","shell.execute_reply":"2026-10-06T16:36:26.685133Z"}},"outputs":[],"execution_count":null},{"id":"88837d55-bb88-4c20-b9a2-5128d8aad711","cell_type":"code","source":"import numpy as np\n\npeak_cluster['log_peak'] = np.log1p(peak_cluster['meter_reading'])\n\nplt.figure(figsize=(6,4))\n\nplt.bar(peak_cluster['cluster'],\n        peak_cluster['log_peak'])\n\nplt.xlabel(\"Cluster\")\nplt.ylabel(\"Log(Peak Demand)\")\nplt.title(\"Log-Transformed Peak Demand per Cluster\")\nplt.xticks(peak_cluster['cluster'])\nplt.grid(axis='y')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:26.686974Z","iopub.execute_input":"2026-10-06T16:36:26.687295Z","iopub.status.idle":"2026-10-06T16:36:26.817821Z","shell.execute_reply.started":"2026-10-06T16:36:26.687271Z","shell.execute_reply":"2026-10-06T16:36:26.816917Z"}},"outputs":[],"execution_count":null},{"id":"b6e19df0-e66c-4308-983d-525822197050","cell_type":"code","source":"weekend_weekday = (\n    train_clustered\n    .groupby(['cluster', 'is_weekend'])['meter_reading']\n    .mean()\n    .unstack()\n)\n\nweekend_weekday.columns = ['Weekday', 'Weekend']\nweekend_weekday","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:26.819056Z","iopub.execute_input":"2026-10-06T16:36:26.819427Z","iopub.status.idle":"2026-10-06T16:36:27.916182Z","shell.execute_reply.started":"2026-10-06T16:36:26.819393Z","shell.execute_reply":"2026-10-06T16:36:27.915299Z"}},"outputs":[],"execution_count":null},{"id":"26d07de5-e4e5-49a3-bc9a-ea7c6e3d334a","cell_type":"code","source":"hourly_cluster_avg = (\n    train_clustered\n    .groupby(['cluster', 'hour'])['meter_reading']\n    .mean()\n    .reset_index()\n)\npeak_hour_cluster = (\n    hourly_cluster_avg\n    .loc[hourly_cluster_avg.groupby('cluster')['meter_reading'].idxmax()]\n    .reset_index(drop=True)\n)\npeak_hour_cluster = peak_hour_cluster.rename(columns={\n    'hour': 'Peak Hour',\n    'meter_reading': 'Average Load at Peak Hour'\n})\n\npeak_hour_cluster\n\nplt.figure(figsize=(6,4))\n\nplt.bar(peak_hour_cluster['cluster'],\n        peak_hour_cluster['Peak Hour'])\n\nplt.xlabel(\"Cluster\")\nplt.ylabel(\"Peak Hour (0–23)\")\nplt.title(\"Peak Hour per Cluster\")\n\nplt.ylim(0, 18) \nplt.xticks(peak_hour_cluster['cluster'])\n\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:27.917606Z","iopub.execute_input":"2026-10-06T16:36:27.918428Z","iopub.status.idle":"2026-10-06T16:36:29.156425Z","shell.execute_reply.started":"2026-10-06T16:36:27.918403Z","shell.execute_reply":"2026-10-06T16:36:29.155628Z"}},"outputs":[],"execution_count":null},{"id":"0ed07518-3a9e-4ac5-89b3-7a30cc671932","cell_type":"code","source":"# Temperature sensitivity by cluster\nplt.figure(figsize=(7,4))\nsns.boxplot(data=building_features.reset_index(), x='cluster', y='temp_sensitivity')\nplt.axhline(0, color='grey', linestyle='--')\nplt.title(\"Temperature Sensitivity by Cluster\")\nplt.ylabel(\"Correlation of daily load with temperature\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:36:29.15756Z","iopub.execute_input":"2026-10-06T16:36:29.157919Z","iopub.status.idle":"2026-10-06T16:36:29.354514Z","shell.execute_reply.started":"2026-10-06T16:36:29.15789Z","shell.execute_reply":"2026-10-06T16:36:29.35347Z"}},"outputs":[],"execution_count":null},{"id":"9ef717eb-011e-42f3-a4ca-e165541813e1","cell_type":"markdown","source":"### Average sensitivity hides heating vs cooling buildings cancelling out, so absolute sensitivity and counts per cluster are used instead.","metadata":{}},{"id":"1051021a-2633-4ff2-81cb-fb00c8eb636e","cell_type":"code","source":"building_features['abs_temp_sens'] = building_features['temp_sensitivity'].abs()\nbuilding_features['weather_type'] = pd.cut(building_features['temp_sensitivity'],\n                                           bins=[-1, -0.5, 0.5, 1],\n                                           labels=['Heating-driven', 'Weak/none', 'Cooling-driven'])\nprint(building_features.groupby('cluster')['abs_temp_sens'].mean().round(3))\nprint()\nprint(pd.crosstab(building_features['cluster'], building_features['weather_type']))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-06T16:45:13.498857Z","iopub.execute_input":"2026-10-06T16:45:13.499671Z","iopub.status.idle":"2026-10-06T16:45:13.524736Z","shell.execute_reply.started":"2026-10-06T16:45:13.499637Z","shell.execute_reply":"2026-10-06T16:45:13.523579Z"}},"outputs":[],"execution_count":null},{"id":"6320e8b3-403e-408f-9359-29c0f479a748","cell_type":"markdown","source":"## 9. Insights for Energy Optimization\n\n**Overall pattern:** Average weekend consumption is **~17% lower** than weekday consumption across all 1,449 buildings, confirming that occupancy schedules strongly influence energy use.\n\n**Weather:** Outdoor temperature is a major driver of consumption. **532 buildings (37%)** are strongly cooling-driven (ρ > 0.5) and **226 (16%)** are strongly heating-driven (ρ < −0.5), so over half of all buildings are highly temperature-dependent. Because heating- and cooling-driven buildings cancel each other out when averaged, temperature sensitivity is compared across clusters using its **absolute value**.\n\n**Model summary:** PCA reduced 10 behavioral features to 3 components retaining **96% of the variance**. K-Means with k = 4 produced a silhouette score of **0.34**, indicating moderate but meaningful cluster separation.\n\n**Cluster 0 – Large, peaky, highly weather-dependent (345 buildings)**\nLargest buildings (~184k sq ft avg) with the highest consumption and the **lowest load factor (0.16)**. They are also the **most temperature-sensitive group (|ρ| = 0.63)**, with a near-even split of heating-driven (108) and cooling-driven (128) buildings, consistent with large facilities running both heating and cooling systems.\n**Opportunity:** Peak shaving, load shifting, and weather-based HVAC control; the largest absolute savings potential.\n\n**Cluster 1 – Mid-size, cooling-dominant (640 buildings)**\nMid-sized buildings (~86k sq ft) with high temperature sensitivity (|ρ| = 0.51). Of the strongly temperature-dependent buildings in this group, **cooling-driven ones outnumber heating-driven ones roughly 3.5 to 1 (268 vs 75)**.\n**Opportunity:** Pre-cooling strategies, cooling-system efficiency upgrades, and weather-based demand forecasting.\n\n**Cluster 2 – Small, steady base-load buildings (316 buildings)**\nSmallest buildings (~24k sq ft) with the **highest load factor (0.37)** and only an ~8% weekend drop, i.e. flat, continuous consumption. Most (169 of 316) show weak temperature dependence.\n**Opportunity:** Scheduling offers little gain; savings come from efficient equipment and reducing always-on base load.\n\n**Cluster 3 – Schedule-driven buildings (148 buildings)**\nDaytime consumption is **2.35× nighttime** and weekend load is **~48% lower** than weekday, a typical office or school pattern. This is the **least weather-sensitive group (|ρ| = 0.37)**, with 116 of 148 buildings showing weak temperature dependence: occupancy, not weather, drives their consumption.\n**Opportunity:** Automated HVAC and lighting setback during nights and weekends; the strongest candidate for occupancy-based control.\n\n## Limitations\n- All meter types (electricity, chilled water, steam, hot water) are combined into one reading; analysing each meter type separately would give cleaner profiles.\n- PC1 loads almost equally on all load-level features, so part of the clustering reflects overall consumption magnitude (visible in Cluster 0 vs Cluster 2).\n- A silhouette score of 0.34 indicates overlapping clusters; building behavior is a continuum rather than sharply separated groups.\n\n","metadata":{}}]}