{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":9994,"databundleVersionId":752467}],"dockerImageVersionId":31400,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **ASHRAE - Great Energy Predictor III**\n### **Background**\n\nQ: How much does it cost to cool a skyscraper in the summer? \n\nA: A lot! And not just in dollars, but in environmental impact.\n\nThankfully, significant investments are being made to improve building efficiencies to reduce costs and emissions. The question is, are the improvements working? That’s where you come in.\n\n\nIn this competition, you’ll develop accurate models of metered building energy usage in the following areas: chilled water, electric, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe. With better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n\n### **Evaluation metric**\nThe evaluation metric for this competition is Root Mean Squared Logarithmic Error.\n\nThe RMSLE is calculated as\n\n$$\\epsilon = \\sqrt{\\frac{1}{n} \\sum_{i=1}^{n} (\\log(p_i + 1) - \\log(a_i + 1))^2}$$\n\n\n* $\\epsilon$ is the RMSLE value (score)\n* $n$ is the total number of observations in the (public/private) data set\n* $p_i$ is your prediction of the target\n* $a_i$ is the actual target for $i$\n* $\\log(x)$ is the natural logarithm of $x$\n\n<div style=\"background-color: rgba(35, 162, 217, 0.18); padding: 10px; border-radius: 5px; border-left: 5px solid #23a2d9;\">\n  <h2 style=\"color: #1d8ebf; margin: 0; padding: 0; line-height: 1.2;\">Energy Usage Prediction (ML Model)</h2>\n</div>\n<br>\n\nUnder pay-for-performance financing, the building owner makes payments based on the difference between their real energy consumption and what they would have used without any retrofits. The latter values have to come from a model. Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nThe objective is to minimize the **Root Mean Squared Logarithmic Error (RMSLE)**\n\nTo capture the intricate relationships between building specifications, temporal cycles, and volatile weather patterns, we implement a **Stratified Gradient Boosting Framework**. Rather than training a single massive model, our architectural strategy splits the problem space by Meter Type or Site ID. Energy consumption behaviors differ fundamentally—a hot water meter's relationship with ambient temperature is inversely non-linear, whereas electricity consumption is heavily dictated by temporal schedules (weekday vs. weekend business hours).\n\n1. LightGBM / XGBoost Regressors: Highly optimized, leaf-wise tree growth models capable of handling massive tabular data and capturing non-linear feature splits efficiently.\n2. K-Fold Time-Series / Stratified Cross-Validation: An out-of-fold (OOF) cross-validation scheme to safeguard against temporal data leakage and ensure robust generalization on unseen future data.\n3. Target Transformation: Since energy data is heavily right-skewed and evaluated via RMSLE, we optimize our models by converting the target variable using a log transformation: $y_{train} = \\log(meter\\_reading + 1)$. This aligns the loss function directly with Root Mean Squared Error (RMSE) optimizations in Gradient Boosting Trees.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n\ndataFiles = []\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        dataFiles.append(os.path.join(dirname, filename))\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n# Use the kagglehub client library to attach Kaggle resources like competitions, datasets, and models to your session\n# Learn more about kagglehub: https://github.com/Kaggle/kagglehub/blob/main/README.md\n\nimport kagglehub\n# kagglehub.dataset_download('<owner>/<dataset-slug>')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-06-02T06:58:49.125929Z","iopub.execute_input":"2026-06-02T06:58:49.127040Z","iopub.status.idle":"2026-06-02T06:58:50.994458Z","shell.execute_reply.started":"2026-06-02T06:58:49.127000Z","shell.execute_reply":"2026-06-02T06:58:50.993410Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for file in dataFiles:\n    baseName = os.path.basename(file)\n    dfName = os.path.splitext(baseName)[0]\n    globals()[dfName] = pd.read_csv(file)\n\n    print(f\"Successfully created DataFrame: {dfName} with shape {globals()[dfName].shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T06:58:50.995801Z","iopub.execute_input":"2026-06-02T06:58:50.996285Z","iopub.status.idle":"2026-06-02T06:59:57.432746Z","shell.execute_reply.started":"2026-06-02T06:58:50.996252Z","shell.execute_reply":"2026-06-02T06:59:57.431637Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Dataset Exploration & Goal Definition**\n\n1. `train.csv`\n\n* **The Building ID:** Which specific building are we looking at? (e.g., Building #1).\n* **The Meter Type:** Buildings don't just use electricity. A single building might have separate meters tracking different energy types :\n    * `0`: Electricity (lights, computers, appliances)\n    * `1`: Chilled Water (running through central A/C units to cool the building)\n    * `2`: Steam (often used in large complexes for heavy heating)\n    * `3`: Hot Water (for taps, showers, and radiators)\n\n\n* **The Timestamp:** The exact hour and day the reading was taken.\n* **The Meter Reading (The Target):** This is the actual value we want to learn to predict. It's the number on the utility bill for that specific hour.\n\n2. `building_meta.csv`\n\n* **`site_id` (0 to 15):** The location index. Buildings are clustered into 16 different locations/cities around the world.\n* **`primary_use`:** What happens inside. Data showcases, **38%** are Education buildings (like schools or university labs) and **19%** are Offices. This matters because a school empties out on weekends, but an office might have a completely different routine.\n* **`square_feet`:** How massive is the property? Data Overview shows a highly skewed distribution—most buildings are on the smaller side, but some massive ones reach up to 875,000 square feet. Bigger buildings naturally swallow more energy.\n* **`year_built` & `floor_count`:** Age and height. Older buildings might have poor insulation; taller buildings might have different airflow and heating demands.\n\n3. `weather_train.csv`/`weather_test.csv`\n\n* **Temperatures (`air_temperature`, `dew_temperature`):** If it’s freezing outside, the hot water and steam usage will skyrocket. If it's boiling, the chilled water loop screams to life.\n* **The Sky & Air (`cloud_coverage`, `sea_level_pressure`, `precip_depth_1_hr`):** Cloud coverage is measured in *oktas* (0 means a completely clear blue sky; 8 means completely overcast). Heavy cloud cover means less solar heat radiating through windows, which alters how hard an HVAC system has to work.\n* **Wind (`wind_speed`, `wind_direction`):** High winds can strip heat away from a building's exterior walls rapidly, driving up heating needs.","metadata":{}},{"cell_type":"code","source":"building_metadata.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T06:59:57.434096Z","iopub.execute_input":"2026-06-02T06:59:57.434397Z","iopub.status.idle":"2026-06-02T06:59:57.464796Z","shell.execute_reply.started":"2026-06-02T06:59:57.434363Z","shell.execute_reply":"2026-06-02T06:59:57.464056Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"building_metadata.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T06:59:57.466454Z","iopub.execute_input":"2026-06-02T06:59:57.466978Z","iopub.status.idle":"2026-06-02T06:59:57.492180Z","shell.execute_reply.started":"2026-06-02T06:59:57.466952Z","shell.execute_reply":"2026-06-02T06:59:57.491107Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Data Cleaning and Preprocessing","metadata":{}},{"cell_type":"code","source":"weather_train['timestamp'] = pd.to_datetime(weather_train['timestamp'])\n\n# Fill missing gaps forward and backward within each isolated city site index\nweather_cols = ['air_temperature', 'dew_temperature', 'cloud_coverage', 'wind_speed', 'sea_level_pressure']\nfor col in weather_cols:\n    if col in weather_train.columns:\n        weather_train[col] = weather_train.groupby('site_id')[col].ffill().bfill()\n\ntrain['timestamp'] = pd.to_datetime(train['timestamp'])\n\n# Calculate the first true timestamp per building-meter group where reading > 0\nstartup_dates = (\n    train[train['meter_reading'] > 0]\n    .groupby(['building_id', 'meter'])['timestamp']\n    .min()\n    .reset_index()\n    .rename(columns={'timestamp': 'first_operational_timestamp'})\n)\n\n# Merge the startup timestamps back onto the main dataframe safely\ntrain_clean = train.merge(startup_dates, on=['building_id', 'meter'], how='left')\n\n# Discard only the records that sit BEFORE that building's startup timestamp\ntrain_clean = train_clean[train_clean['timestamp'] >= train_clean['first_operational_timestamp']].copy()\ntrain_clean.drop(columns=['first_operational_timestamp'], inplace=True)\n\n\n# Compute the 99.9th percentile threshold per building and meter asset\nq_high = train_clean.groupby(['building_id', 'meter'])['meter_reading'].transform(lambda x: x.quantile(0.999))\ntrain_clean = train_clean[train_clean['meter_reading'] <= q_high]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T06:59:57.493104Z","iopub.execute_input":"2026-06-02T06:59:57.493323Z","iopub.status.idle":"2026-06-02T07:00:07.491757Z","shell.execute_reply.started":"2026-06-02T06:59:57.493300Z","shell.execute_reply":"2026-06-02T07:00:07.490947Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Force pandas to create a completely isolated, independent copy\ndf_clean = train_clean.merge(building_metadata[['building_id', 'site_id', 'primary_use']], on='building_id', how='left')\ndf_clean = df_clean.merge(weather_train, on=['site_id', 'timestamp'], how='left').copy()\n\n# Temporal transformations\ndf_clean['hour'] = df_clean['timestamp'].dt.hour\ndf_clean['dayofweek'] = df_clean['timestamp'].dt.dayofweek\ndf_clean['month'] = df_clean['timestamp'].dt.month\ndf_clean['is_weekend'] = df_clean['dayofweek'].isin([5, 6]).astype(int)\n\n# Structural and thermodynamic interactions\ndf_clean['temp_humidity_spread'] = df_clean['air_temperature'] - df_clean['dew_temperature']\ndf_clean['target'] = np.log1p(df_clean['meter_reading'])\n\ndf_clean.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:57:08.622359Z","iopub.execute_input":"2026-06-02T08:57:08.622677Z","iopub.status.idle":"2026-06-02T08:57:13.838061Z","shell.execute_reply.started":"2026-06-02T08:57:08.622656Z","shell.execute_reply":"2026-06-02T08:57:13.837398Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_clean.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:47:20.879264Z","iopub.execute_input":"2026-06-02T08:47:20.879691Z","iopub.status.idle":"2026-06-02T08:47:20.939504Z","shell.execute_reply.started":"2026-06-02T08:47:20.879594Z","shell.execute_reply":"2026-06-02T08:47:20.938815Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(20, 7))\n\ntop_categories = building_metadata[\"primary_use\"].value_counts().nlargest(6).index\ngrouped_use = building_metadata[\"primary_use\"].apply(\n    lambda x: x if x in top_categories else \"Other\"\n)\n\ncounts = grouped_use.value_counts()\nbuilding_order = counts.index\n\nsns.countplot(\n    data=building_metadata,\n    x=\"primary_use\",\n    order=building_order,\n    ax=axes[0]\n)\n\nplt.setp(axes[0].get_xticklabels(), rotation=45, ha=\"right\")\naxes[0].set_xlabel(\"Building Type (Primary Use)\", fontsize=12)\naxes[0].set_ylabel(\"Count of Buildings\", fontsize=12)\naxes[0].set_title(\"No of Building (By Type)\", fontsize=14, fontweight=\"bold\", pad=15)\n\n\nwedges, texts, autotexts = axes[1].pie(\n    counts,\n    autopct=\"%1.1f%%\",\n    pctdistance=0.75,\n    startangle=140,\n    colors=sns.color_palette(\"Blues\", len(building_order)),\n    textprops=dict(color=\"w\", weight=\"bold\", size=11),\n)\n\naxes[1].legend(\n    wedges,\n    building_order,\n    title=\"Building Types\",\n    loc=\"center left\",\n    bbox_to_anchor=(1, 0, 0.5, 1),\n    fontsize=11,\n)\n\naxes[1].set_title(\n    \"Proportion of Building Types\", fontsize=16, fontweight=\"bold\", pad=20\n)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:12.249084Z","iopub.execute_input":"2026-06-02T07:00:12.249315Z","iopub.status.idle":"2026-06-02T07:00:12.644810Z","shell.execute_reply.started":"2026-06-02T07:00:12.249290Z","shell.execute_reply":"2026-06-02T07:00:12.643939Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* Over 37% of the buildings are educational institutes\n* Offices come at 19.3% just ahead of Entertainment and public assembly\n* Public Services, Lodging, residential and other buildings make up the rest 30% roughly\n\nNow I would like to study the correlation and relation to the energy usage by the different building types. I would draw relations and connections between the energy use types per building type as well to see if a certain building type uses more of a certain energy than other.","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:12.645608Z","iopub.execute_input":"2026-06-02T07:00:12.645839Z","iopub.status.idle":"2026-06-02T07:00:12.654704Z","shell.execute_reply.started":"2026-06-02T07:00:12.645810Z","shell.execute_reply":"2026-06-02T07:00:12.654087Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:12.655568Z","iopub.execute_input":"2026-06-02T07:00:12.656004Z","iopub.status.idle":"2026-06-02T07:00:12.678323Z","shell.execute_reply.started":"2026-06-02T07:00:12.655974Z","shell.execute_reply":"2026-06-02T07:00:12.677169Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"merged_df = train.merge(building_metadata, on=\"building_id\", how=\"left\")\n\nmeter_mapping = {\n    0: \"Electricity\",\n    1: \"Chilled Water\",\n    2: \"Steam\",\n    3: \"Hot Water\",\n}\n\nmerged_df[\"meter_name\"] = merged_df[\"meter\"].map(meter_mapping)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:12.681344Z","iopub.execute_input":"2026-06-02T07:00:12.681637Z","iopub.status.idle":"2026-06-02T07:00:14.318873Z","shell.execute_reply.started":"2026-06-02T07:00:12.681613Z","shell.execute_reply":"2026-06-02T07:00:14.317697Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create a cross-tabulation of building types vs meter types\nenergy_mix = pd.crosstab(merged_df[\"primary_use\"], merged_df[\"meter_name\"])\n\n# Plot a bar chart to see the distribution of energy streams\nenergy_mix.plot(kind=\"bar\", stacked=False, figsize=(13, 5), cmap=\"coolwarm\")\n\nplt.yscale(\"log\")\n\nplt.title(\"Distribution of Energy Types Across Building Uses\", fontsize=14, fontweight=\"bold\")\nplt.xlabel(\"Building Primary Use\", fontsize=12)\nplt.ylabel(\"Number of Records (Log Scale)\", fontsize=12)\nplt.xticks(rotation=45, ha=\"right\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:14.319845Z","iopub.execute_input":"2026-06-02T07:00:14.320152Z","iopub.status.idle":"2026-06-02T07:00:17.613098Z","shell.execute_reply.started":"2026-06-02T07:00:14.320129Z","shell.execute_reply":"2026-06-02T07:00:17.612224Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* Education, Office, Entertainment, Lodging, Public Services. These profiles use all 4 energy types heavily. They are likely large campus-style complexes with centralized HVAC plants.\n* Parking, Retail, Warehouse/Storage, Religious Worship. These profiles almost exclusively register electricity records.\n* Manufacturing/Industrial, Technology/Science. These show a unique balance, often needing unexpected spikes in steam or electricity for machinery rather than ambient comfort.","metadata":{}},{"cell_type":"code","source":"# Calculate the average energy consumption for each building type and energy type\nconsumption_df = (\n    merged_df.groupby([\"primary_use\", \"meter_name\"])[\"meter_reading\"]\n    .mean()\n    .reset_index()\n)\n\n# Pivot the data for clean plotting\nconsumption_pivot = consumption_df.pivot(\n    index=\"primary_use\", columns=\"meter_name\", values=\"meter_reading\"\n)\n\n# Plot the consumption breakdown\nconsumption_pivot.plot(kind=\"bar\", stacked=False, figsize=(14, 7), colormap=\"coolwarm\")\n\nplt.yscale(\"log\")\n\nplt.title(\"Average Energy Consumption Volume by Building Type\", fontsize=14, fontweight=\"bold\")\nplt.xlabel(\"Building Primary Use\", fontsize=12)\nplt.ylabel(\"Average Meter Reading Volume (Log Scale)\", fontsize=12)\nplt.xticks(rotation=45, ha=\"right\")\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:17.613966Z","iopub.execute_input":"2026-06-02T07:00:17.614225Z","iopub.status.idle":"2026-06-02T07:00:20.160796Z","shell.execute_reply.started":"2026-06-02T07:00:17.614205Z","shell.execute_reply":"2026-06-02T07:00:20.160054Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* **Electricity is highly stable across categories**: It almost always floats tightly between $50$ and $300$ units on average. It has a high baseline but lower structural variance across building types.\n* **Thermal Energies (Steam/Chilled Water) fluctuate wildly**: Look at Utility or Entertainment. Chilled water spikes immensely compared to their electricity footprints.\n* **How to use this for the model**: Trees split on features that reduce variance. Because thermal energy consumption is highly dependent on building architecture and weather (thermodynamics), adding interaction features like `building_id + meter` or `primary_use + meter` will be crucial to helping my model map these massive swings.","metadata":{}},{"cell_type":"code","source":"weather_train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:20.162394Z","iopub.execute_input":"2026-06-02T07:00:20.162673Z","iopub.status.idle":"2026-06-02T07:00:20.173839Z","shell.execute_reply.started":"2026-06-02T07:00:20.162650Z","shell.execute_reply":"2026-06-02T07:00:20.173007Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"weather_train.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:20.174695Z","iopub.execute_input":"2026-06-02T07:00:20.174930Z","iopub.status.idle":"2026-06-02T07:00:20.202447Z","shell.execute_reply.started":"2026-06-02T07:00:20.174886Z","shell.execute_reply":"2026-06-02T07:00:20.201574Z"},"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train['timestamp'] = pd.to_datetime(train['timestamp'])\nweather_train['timestamp'] = pd.to_datetime(weather_train['timestamp'])\nbuilding_metadata['building_id'] = building_metadata['building_id'].astype(np.int16)\nbuilding_metadata['site_id'] = building_metadata['site_id'].astype(np.int8)\n\n# 1. Bridge train_df to get the site_id for each building\nweather_energy = train.merge(building_metadata[['building_id', 'site_id', 'primary_use']], on='building_id', how='left')\n\n# 2. Merge the weather data using BOTH site_id and timestamp\nweather_energy = weather_energy.merge(weather_train, on=['site_id', 'timestamp'], how='left')\n\n# Memory Optimization: Target transformation to log1p to handle the Steam scale anomaly\nweather_energy['log_meter_reading'] = np.log1p(weather_energy['meter_reading'])\n\nprint(\"Unified Dataframe Shape:\", weather_energy.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:20.203469Z","iopub.execute_input":"2026-06-02T07:00:20.203705Z","iopub.status.idle":"2026-06-02T07:00:24.005979Z","shell.execute_reply.started":"2026-06-02T07:00:20.203684Z","shell.execute_reply":"2026-06-02T07:00:24.005159Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"weather_energy.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:24.006845Z","iopub.execute_input":"2026-06-02T07:00:24.007145Z","iopub.status.idle":"2026-06-02T07:00:24.019313Z","shell.execute_reply.started":"2026-06-02T07:00:24.007123Z","shell.execute_reply":"2026-06-02T07:00:24.018604Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"A building's relationship with ambient temperature is non-linear. It forms a **\"V-Curve\"** (or U-curve). When it's freezing, heating energy (Steam/Hot Water) goes up. When it's blistering hot, cooling energy (Chilled Water) goes up. In the middle sits the \"Comfort Zone\" where energy usage bottoms out.","metadata":{}},{"cell_type":"code","source":"# Map meter integer codes for readability\nmeter_mapping = {0: 'Electricity', 1: 'Chilled Water', 2: 'Steam', 3: 'Hot Water'}\nweather_energy['meter_name'] = weather_energy['meter'].map(meter_mapping)\n\n# Round air temperature to the nearest whole degree to group data points\nweather_energy['temp_bin'] = weather_energy['air_temperature'].round()\n\n# Calculate mean log consumption per temperature bin and meter type\nv_curve_data = weather_energy.groupby(['temp_bin', 'meter_name'])['log_meter_reading'].mean().reset_index()\n\n# Plot the V-Curves\nplt.figure(figsize=(8, 7))\nsns.lineplot(\n    data=v_curve_data, \n    x='temp_bin', \n    y='log_meter_reading', \n    hue='meter_name', \n    linewidth=2.5,\n    palette='Set1'\n)\n\nplt.title(\"The Thermodynamic V-Curve: Energy Consumption vs Air Temperature\", fontsize=14, fontweight='bold')\nplt.xlabel(\"Outside Air Temperature (°C)\", fontsize=12)\nplt.ylabel(\"Mean Log Energy Consumption\", fontsize=12)\nplt.grid(True, linestyle=\"--\", alpha=0.5)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:24.020246Z","iopub.execute_input":"2026-06-02T07:00:24.020480Z","iopub.status.idle":"2026-06-02T07:00:26.005189Z","shell.execute_reply.started":"2026-06-02T07:00:24.020458Z","shell.execute_reply":"2026-06-02T07:00:26.004473Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* As the outside air temp reachs around 12-30°C, the hot water usage and steam usage fall while the chilled water rises sharply\n* The electricity usage remains relatively flat with slight fall and rise around 0°C and rises around 15-20°C","metadata":{}},{"cell_type":"markdown","source":"Air temperature tells only half the story. Air conditioning units don't just cool the air; they have to pull moisture out of it (dehumidification). High humidity forces chiller plants to work drastically harder. **Dew point** is a direct metric for atmospheric moisture","metadata":{}},{"cell_type":"code","source":"# Filter for Chilled Water records only, where humidity matters most\nchiller_df = weather_energy[weather_energy['meter_name'] == 'Chilled Water'].dropna(subset=['air_temperature', 'dew_temperature'])\n\n# Sample a fraction of the data to avoid an over-cluttered scatter plot matrix\nsample_df = chiller_df.sample(n=50000, random_state=42)\n\nplt.figure(figsize=(10, 8))\nscatter = plt.scatter(\n    x=sample_df['air_temperature'],\n    y=sample_df['dew_temperature'],\n    c=sample_df['log_meter_reading'],\n    cmap='Blues',\n    alpha=0.4,\n    s=15\n)\n\ncbar = plt.colorbar(scatter)\ncbar.set_label('Chilled Water Consumption Volume (Log Scale)', fontsize=12)\nplt.title(\"Chilled Water Demand Driven by Heat vs Humidity Interaction\", fontsize=14, fontweight='bold')\nplt.xlabel(\"Air Temperature (°C)\", fontsize=12)\nplt.ylabel(\"Dew Point Temperature (°C)\", fontsize=12)\nplt.grid(True, alpha=0.3)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:26.006123Z","iopub.execute_input":"2026-06-02T07:00:26.006355Z","iopub.status.idle":"2026-06-02T07:00:28.694033Z","shell.execute_reply.started":"2026-06-02T07:00:26.006334Z","shell.execute_reply":"2026-06-02T07:00:28.693115Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1. The Diagonal Hard-Ceiling (The Saturation Line)\nNotice how the data points form a perfectly sharp, diagonal wall running along the top left. There are absolutely zero data points above that line.\n\n* **Science Reason**: Dew point temperature can never be higher than the actual air temperature. When dew point equals air temperature, relative humidity hits 100%.\n* **What that boundary is**: That diagonal edge represents rainy, foggy, or completely saturated weather conditions. The data is literally showing you the physical threshold of Earth's atmosphere\n\n### 2. The Multi-Dimensional Target behavior (The Color Gradient)\n\nLook at how the colors shift from bottom-left to top-right.\n\n* **The Freezing Zone (Bottom Left):** When air temp is $-20^\\circ\\text{C}$ and dew point is $-30^\\circ\\text{C}$, the points are completely white ($0$ to $2$ on the log scale). This makes complete physical sense: you don't run an air conditioning chiller plant to cool a building when it is freezing outside.\n* **The High-Load Zone (Top Right):** As you move toward the top right—where it is both hot (Air Temp $> 25^\\circ\\text{C}$) and highly humid (Dew Point $> 15^\\circ\\text{C}$)—the dots turn into a much denser, darker blue. The chiller plants are working at maximum capacity.\n\n### 3. The \"Hidden Information\" Your Model Will Struggle With\n\nLook at the cluster of points between **$20^\\circ\\text{C}$ and $40^\\circ\\text{C}$ Air Temperature**, but stretching vertically across different dew points.\n\nIf you look horizontally at $30^\\circ\\text{C}$ air temperature:\n\n* At a low dew point ($-10^\\circ\\text{C}$, hot and bone-dry desert air), the color is pale blue.\n* At a high dew point ($20^\\circ\\text{C}$, hot and extremely humid swampy air), the color is noticeably darker blue.\n\n**The Crucial Modeling Takeaway:** If you only give your machine learning model `air_temperature`, it will see \"$30^\\circ\\text{C}$\" and have to guess blindly between low energy usage and massive energy usage. The *vertical spread* proves that humidity is acting as a massive modifier.","metadata":{}},{"cell_type":"code","source":"weather_energy[\"timestamp\"] = pd.to_datetime(weather_energy[\"timestamp\"])\nweather_energy[\"hour\"] = weather_energy[\"timestamp\"].dt.hour\n\n# Map the integer keys to their descriptive string labels\nmeter_mapping = {\n    0: \"Electricity\",\n    1: \"Chilled Water\",\n    2: \"Steam\",\n    3: \"Hot Water\",\n}\nweather_energy[\"meter_name\"] = weather_energy[\"meter\"].map(meter_mapping)\n\nhourly_means = (\n    weather_energy.groupby([\"hour\", \"meter_name\"])[\"meter_reading\"].mean().reset_index()\n)\n\n\nfig, axes = plt.subplots(2, 2, figsize=(18, 8), sharex=True)\naxes = axes.flatten()  # Flatten the 2D array of axes to easily loop over them\n\ncolors = [\"#4e79a7\", \"#f28e2b\", \"#e15759\", \"#76b7b2\"]\n\nfor i, meter_name in enumerate(meter_mapping.values()):\n    meter_data = hourly_means[hourly_means[\"meter_name\"] == meter_name]\n\n    sns.lineplot(\n        data=meter_data,\n        x=\"hour\",\n        y=\"meter_reading\",\n        ax=axes[i],\n        color=colors[i],\n        linewidth=2.5,\n    )\n\n    axes[i].set_title(f\"Mean Consumption: {meter_name}\", fontsize=13, weight=\"bold\")\n    axes[i].set_ylabel(\"Mean Reading Volume\", fontsize=11)\n    axes[i].set_xlabel(\"Hour of Day\", fontsize=11)\n    axes[i].grid(True, linestyle=\"--\", alpha=0.5)\n    axes[i].set_xticks(range(0, 24, 2))  # Tick marks every two hours\n\nplt.suptitle(\n    \"24-Hour Energy Profile Across Different Meter Types\",\n    fontsize=16,\n    weight=\"bold\",\n    y=0.98,\n)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:00:28.694959Z","iopub.execute_input":"2026-06-02T07:00:28.695623Z","iopub.status.idle":"2026-06-02T07:00:31.501941Z","shell.execute_reply.started":"2026-06-02T07:00:28.695575Z","shell.execute_reply":"2026-06-02T07:00:31.501019Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"* The average consumption of electricity rises from the start of the day and starts to decline as night falls when most people may be asleep\n* The mean consumption of steam rises at the start of the dat and stays very high throughout the remainder of the day\n* The mean consumption of chilled water is inversely related to the consumption of hot water\n* The consumption of hot water rises from 4th hour to 8th hour and then falls for the rest of the day\n* Meanwhile chilled water rises at the start of the day and stays high till end of the day","metadata":{}},{"cell_type":"code","source":"import matplotlib.dates as mdates\n\nmeter_mapping = {0: \"Electricity\", 1: \"Chilled Water\", 2: \"Steam\", 3: \"Hot Water\"}\ntrend_colors  = [\"#4e79a7\", \"#f28e2b\", \"#e15759\", \"#76b7b2\"]\n\nplot_df = df_clean[df_clean[\"meter_reading\"] >= 1.0].copy()\nplot_df[\"meter_name\"] = plot_df[\"meter\"].map(meter_mapping)\n\n# Normalise per building-meter\nbuilding_medians = (\n    plot_df.groupby([\"building_id\", \"meter\"])[\"meter_reading\"]\n    .median()\n    .rename(\"building_median\")\n    .reset_index()\n)\nplot_df = plot_df.merge(building_medians, on=[\"building_id\", \"meter\"], how=\"left\")\nplot_df[\"normalised_reading\"] = plot_df[\"meter_reading\"] / plot_df[\"building_median\"]\n\n# Aggregate across buildings at each timestamp\ntimeline_means = (\n    plot_df.groupby([\"timestamp\", \"meter_name\"])[\"normalised_reading\"]\n    .median()\n    .reset_index()\n)\n\nfig, axes = plt.subplots(2, 2, figsize=(18, 9))\naxes = axes.flatten()\n\nfor i, meter_name in enumerate(meter_mapping.values()):\n    ax = axes[i]\n\n    trend_subset = (\n        timeline_means[timeline_means[\"meter_name\"] == meter_name]\n        .sort_values(\"timestamp\")\n        .copy()\n    )\n\n    # 7-day rolling median\n    trend_subset[\"smoothed\"] = (\n        trend_subset[\"normalised_reading\"]\n        .rolling(window=168, min_periods=12, center=True)\n        .median()\n    )\n\n    ax.plot(\n        trend_subset[\"timestamp\"],\n        trend_subset[\"smoothed\"],\n        color=trend_colors[i],\n        linewidth=2.5,\n        label=\"7-day Rolling Median\",\n    )\n\n    ax.axhline(y=1.0, color=\"black\", linestyle=\":\", linewidth=1.0, alpha=0.5, label=\"Annual Median Baseline\")\n\n    ax.set_title(f\"{meter_name}: Full-Year Timeline Performance\", fontsize=13, weight=\"bold\")\n    ax.set_ylabel(\"Normalised Meter Reading (× Median)\", fontsize=11)\n    ax.set_xlabel(\"Date\", fontsize=11)\n    ax.grid(True, linestyle=\"--\", alpha=0.4)\n    ax.legend(loc=\"upper left\", fontsize=9)\n\n    ax.xaxis.set_major_locator(mdates.MonthLocator())\n    ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%b\\n%Y\"))\n    ax.tick_params(axis=\"x\", labelsize=9)\n\nplt.suptitle(\n    \"Continuous Historical Energy Performance & Seasonal Distributions\",\n    fontsize=16, weight=\"bold\", y=1.01,\n)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T07:03:25.104937Z","iopub.execute_input":"2026-06-02T07:03:25.105503Z","iopub.status.idle":"2026-06-02T07:03:33.165800Z","shell.execute_reply.started":"2026-06-02T07:03:25.105476Z","shell.execute_reply":"2026-06-02T07:03:33.164844Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"1. Strong Seasonal Inversion (Chilled Water vs. Steam/Hot Water) : **The Insight:** Chilled water usage peaks dramatically in summer (July/August, reaching $\\sim 1.9\\times$ baseline) and bottoms out in winter. Conversely, Steam and Hot Water exhibit an inverse bell curve, peaking in winter ($\\sim 2.25\\times$ and $\\sim 1.7\\times$ respectively) and dropping significantly in summer.\n2. Electricity is Non-Linear & Flat-Bottomed : **The Insight:** Unlike the thermal utilities, Electricity stays relatively high year-round ($0.85\\times$ to $1.05\\times$), dipping briefly in late Spring (May/June) and early Winter (December). This indicates a constant baseload (servers, lighting, appliances) supplemented by minor cooling peaks in late summer.\n3. High-Frequency Volatility & Micro-Seasons : **The Insight:** Even with a 7-day rolling median, all four utilities show distinct, sharp micro-dips and spikes (e.g., the sharp drop in Electricity and Hot Water right at the end of December, likely due to holiday building closures).\n4. Non-Symmetric Shoulder Seasons : **The Insight:** The transition periods (Spring vs. Autumn) are asymmetric. For example, Hot Water drops sharply from Jan to April, but rises much more gradually from August to December. Steam behaves similarly.","metadata":{}},{"cell_type":"markdown","source":"# **Model Construction + EDA Conclusions**","metadata":{}},{"cell_type":"markdown","source":"* If you throw all this data into an AI model at once, the AI will get really smart at predicting schools and offices, but it will suck at predicting stores or churches\n* We might need to split the data and train a separate model just for the rare building types so they don't get ignored\n\n* Because the steam numbers are so huge, they can confuse the AI and make it ignore small changes in electricity\n* We need to use math to squish these giant steam numbers down (using a log scale) before feeding them to the AI, so every energy type gets treated fairly\n\n* **When it's freezing cold**: Steam and Hot Water lines spike all the way up because buildings are blasting their heaters. Chilled water (AC) drops to zero.\n* **When it's boiling hot**: The red Chilled Water line shoots up because everyone is cranking the AC. The heating lines drop and flatten out.\n* **The Fix**: We should create simple helper features for the AI called Heating Days and Cooling Days to explicitly tell it how far the temperature is from a comfortable room temperature\n\n* The AC has to work twice as hard when it is hot and humid (high dew point) because it has to literally squeeze the water out of the air to make the room feel cool. If it's hot but dry, the AC doesn't work quite as hard\n* We should multiply Temperature by Humidity to create a \"Sticky Weather Index\" feature for the model\n\n* Electricity and AC peak in the middle of the afternoon (when people are working and school is in session) and drop to a baseline at 4:00 AM when everyone is asleep. Heating (Steam), however, stays pretty high all night during winter to keep the pipes from freezing\n* AC peaks in July/August; heating peaks in January/February\n* Giant concrete buildings act like rocks left in the sun—they take a long time to heat up and a long time to cool down. If a heatwave hits at 12:00 PM, the building might not get boiling hot inside until 3:00 PM\n\n## Final Dataset Schema\n### Master Engineered Dataset Feature List\n\n* **Identifiers & Target Variables**\n    * `row_id`: Unique row identifier (essential for mapping test predictions back to the sample submission format).\n    * `meter_reading_log1p`: The transformed target variable calculated as $\\log1p(\\text{meter\\_reading})$ to handle the heavy-tailed steam data fairly.\n\n* **Categorical Stratification Features**\n    * `primary_use_grouped`: A consolidated building type feature where rare classes (e.g., *Religious worship, Retail, Services*) are grouped together or flagged for localized modeling to prevent institutional bias.\n\n* **Thermodynamic Weather Features**\n    * `heating_degree_days (HDD)`: Calculated as $\\max(0, T_{\\text{base}} - \\text{air\\_temperature})$ to map cold-weather heating spikes.\n    * `cooling_degree_days (CDD)`: Calculated as $\\max(0, \\text{air\\_temperature} - T_{\\text{base}})$ to map hot-weather cooling spikes.\n    * `sticky_weather_index`: An interaction term capturing latent heat load, calculated by multiplying `air_temperature` by a calculated relative humidity or directly using the `dew_temperature` interaction.\n\n* **Temporal Profile Features**\n    * `hour_of_day`: Cyclical integer (0–23) to capture diurnal occupancy patterns.\n    * `day_of_week`: Categorical indicator (0–6) to capture lower weekend energy profiles.\n    * `month_of_year`: Categorical indicator (1–12) to capture seasonal peaks (July vs. January).\n\n* **Thermal Mass Lag Features**\n    * `air_temp_lag_3h`: Historical temperature from 3 hours prior to account for structural thermal delay.\n    * `air_temp_rolling_mean_24h`: Smooth 24-hour temperature moving average to capture overall heatwave or cold-snap duration.","metadata":{}},{"cell_type":"code","source":"# 1. Convert text categories to native categorical types\ndf_clean['primary_use'] = df_clean['primary_use'].astype('category')\n\n# 2. Downcast numeric columns to save massive amounts of memory\ndf_clean['building_id'] = df_clean['building_id'].astype('int16')\ndf_clean['meter'] = df_clean['meter'].astype('int8')\ndf_clean['site_id'] = df_clean['site_id'].astype('int8')\n\nfloat_cols = ['air_temperature', 'cloud_coverage', 'dew_temperature', \n              'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'wind_speed']\nfor col in float_cols:\n    df_clean[col] = df_clean[col].astype('float32')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:57:24.646667Z","iopub.execute_input":"2026-06-02T08:57:24.646976Z","iopub.status.idle":"2026-06-02T08:57:26.202750Z","shell.execute_reply.started":"2026-06-02T08:57:24.646931Z","shell.execute_reply":"2026-06-02T08:57:26.202008Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Heating Degree Days (Blasting heaters when it's cold)\ndf_clean['heating_degree_days'] = (18.0 - df_clean['air_temperature']).clip(lower=0).astype('float32')\n\n# Cooling Degree Days (Cranking the AC when it's hot)\ndf_clean['cooling_degree_days'] = (df_clean['air_temperature'] - 18.0).clip(lower=0).astype('float32')\n\n# Interactive Sticky Weather (Air Temp & Dew Temp interaction)\ndf_clean['sticky_weather_index'] = (df_clean['air_temperature'] * df_clean['dew_temperature']).astype('float32')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:57:31.542534Z","iopub.execute_input":"2026-06-02T08:57:31.542845Z","iopub.status.idle":"2026-06-02T08:57:31.789589Z","shell.execute_reply.started":"2026-06-02T08:57:31.542822Z","shell.execute_reply":"2026-06-02T08:57:31.788940Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Sort data chronologically to ensure rolling windows align perfectly\ndf_clean = df_clean.sort_values(by=['site_id', 'timestamp']).reset_index(drop=True)\n\n# Create a 3-hour lag feature for air temperature per site\ndf_clean['air_temp_lag_3h'] = df_clean.groupby('site_id')['air_temperature'].shift(3).astype('float32')\n\n# Create a 24-hour rolling mean feature to capture lingering heatwaves\ndf_clean['air_temp_rolling_mean_24h'] = df_clean.groupby('site_id')['air_temperature'].transform(\n    lambda x: x.rolling(window=24, min_periods=1).mean()\n).astype('float32')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:57:33.646707Z","iopub.execute_input":"2026-06-02T08:57:33.647054Z","iopub.status.idle":"2026-06-02T08:57:37.790604Z","shell.execute_reply.started":"2026-06-02T08:57:33.647026Z","shell.execute_reply":"2026-06-02T08:57:37.789665Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 1. Sort chronologically by site so time moves forward cleanly\ndf_clean = df_clean.sort_values(by=['site_id', 'timestamp']).reset_index(drop=True)\n\n# 2. Fill weather gaps smoothly based on the hours before and after\nweather_cols = ['air_temperature', 'cloud_coverage', 'dew_temperature', \n                'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'wind_speed']\n\nfor col in weather_cols:\n    # Interpolate linearly per site, then fill any remaining edge gaps\n    df_clean[col] = df_clean.groupby('site_id')[col].transform(\n        lambda x: x.interpolate(method='linear').ffill().bfill()\n    )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:57:44.469924Z","iopub.execute_input":"2026-06-02T08:57:44.470229Z","iopub.status.idle":"2026-06-02T08:57:51.504966Z","shell.execute_reply.started":"2026-06-02T08:57:44.470205Z","shell.execute_reply":"2026-06-02T08:57:51.503930Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Double check we are working on a clean copy\ndf_clean = df_clean.copy()\n\n# --- 1. Downcast Types (Saves massive RAM) ---\ndf_clean['primary_use'] = df_clean['primary_use'].astype('category')\ndf_clean['building_id'] = df_clean['building_id'].astype('int16')\ndf_clean['meter'] = df_clean['meter'].astype('int8')\ndf_clean['site_id'] = df_clean['site_id'].astype('int8')\n\nfloat_cols = ['air_temperature', 'cloud_coverage', 'dew_temperature', \n              'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'wind_speed']\nfor col in float_cols:\n    df_clean[col] = df_clean[col].astype('float32')\n\n# --- 2. Advanced Thermodynamic Interactions ---\ndf_clean['heating_degree_days'] = (18.0 - df_clean['air_temperature']).clip(lower=0).astype('float32')\ndf_clean['cooling_degree_days'] = (df_clean['air_temperature'] - 18.0).clip(lower=0).astype('float32')\ndf_clean['sticky_weather_index'] = (df_clean['air_temperature'] * df_clean['dew_temperature']).astype('float32')\n\n# --- 3. Thermal Mass Lags ---\n# 3-Hour Delay\ndf_clean['air_temp_lag_3h'] = df_clean.groupby('site_id')['air_temperature'].shift(3).astype('float32')\n# 24-Hour Rolling Average\ndf_clean['air_temp_rolling_mean_24h'] = df_clean.groupby('site_id')['air_temperature'].transform(\n    lambda x: x.rolling(window=24, min_periods=1).mean()\n).astype('float32')\n\n# Fill any new NaNs created at the very beginning of the rows due to the 3-hour shift\ndf_clean['air_temp_lag_3h'] = df_clean['air_temp_lag_3h'].fillna(df_clean['air_temperature'])\n\nprint(\"Pipeline complete! Current RAM footprint:\")\ndf_clean.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T08:58:02.401934Z","iopub.execute_input":"2026-06-02T08:58:02.402226Z","iopub.status.idle":"2026-06-02T08:58:04.305843Z","shell.execute_reply.started":"2026-06-02T08:58:02.402204Z","shell.execute_reply":"2026-06-02T08:58:04.305008Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Machine Learning Prediction Model**","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\nimport numpy as np\nimport pandas as pd\n\n# Define mapping for tracking\nmeter_mapping = {0: 'Electricity', 1: 'Chilled_Water', 2: 'Steam', 3: 'Hot_Water'}\n\n# Define exactly which features the model should look at\nfeatures = [\n    'building_id', 'site_id', 'primary_use', \n    'air_temperature', 'cloud_coverage', 'dew_temperature',\n    'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'wind_speed',\n    'hour', 'dayofweek', 'month', 'is_weekend',\n    'heating_degree_days', 'cooling_degree_days', 'sticky_weather_index',\n    'air_temp_lag_3h', 'air_temp_rolling_mean_24h'\n]\n\n# Dictionary to hold our 4 trained models\nmodels = {}\n\n# Time-based validation split threshold (e.g., last 2 months of 2016)\nsplit_date = '2016-11-01'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T09:02:40.598018Z","iopub.execute_input":"2026-06-02T09:02:40.598321Z","iopub.status.idle":"2026-06-02T09:02:44.892655Z","shell.execute_reply.started":"2026-06-02T09:02:40.598296Z","shell.execute_reply":"2026-06-02T09:02:44.891775Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for meter_id, meter_name in meter_mapping.items():\n    print(f\"\\n--- Training Model for {meter_name} (Meter {meter_id}) ---\")\n    \n    # Filter for the current meter type\n    df_meter = df_clean[df_clean['meter'] == meter_id]\n    \n    if df_meter.empty:\n        print(f\"No data found for {meter_name}. Skipping.\")\n        continue\n        \n    # Chronological Split\n    train_mask = df_meter['timestamp'] < split_date\n    val_mask = df_meter['timestamp'] >= split_date\n    \n    X_train, y_train = df_meter[train_mask][features], df_meter[train_mask]['target']\n    X_val, y_val = df_meter[val_mask][features], df_meter[val_mask]['target']\n    \n    print(f\"Train shapes: {X_train.shape}, Val shapes: {X_val.shape}\")\n    \n    # Set up LightGBM Datasets\n    train_data = lgb.Dataset(X_train, label=y_train)\n    val_data = lgb.Dataset(X_val, label=y_val, reference=train_data)\n    \n    # Baseline hyperparameters optimized for speed and stability\n    params = {\n        'objective': 'regression',\n        'metric': 'rmse',\n        'learning_rate': 0.1,\n        'num_leaves': 31,\n        'feature_fraction': 0.8,\n        'bagging_fraction': 0.8,\n        'bagging_freq': 5,\n        'seed': 42,\n        'verbose': -1,\n        'n_jobs': -1\n    }\n    \n    # Train the model with early stopping to prevent overfitting\n    model = lgb.train(\n        params,\n        train_data,\n        num_boost_round=1000,\n        valid_sets=[train_data, val_data],\n        callbacks=[\n            lgb.early_stopping(stopping_rounds=50, verbose=False),\n            lgb.log_evaluation(period=100)\n        ]\n    )\n    \n    # Save model instance to our dictionary\n    models[meter_id] = model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T09:02:52.758838Z","iopub.execute_input":"2026-06-02T09:02:52.759975Z","iopub.status.idle":"2026-06-02T09:10:26.327476Z","shell.execute_reply.started":"2026-06-02T09:02:52.759879Z","shell.execute_reply":"2026-06-02T09:10:26.326971Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\n\n# 1. Setup paths and submission base\ntest_path = '/kaggle/input/competitions/ashrae-energy-prediction/test.csv'  # Adjust path if files are in a specific directory\nweather_test_path = '/kaggle/input/competitions/ashrae-energy-prediction/weather_test.csv'\nbuilding_meta_path = '/kaggle/input/competitions/ashrae-energy-prediction/building_metadata.csv'\n\nprint(\"Loading metadata tables...\")\nbuilding_metadata = pd.read_csv(building_meta_path)\nweather_test = pd.read_csv(weather_test_path)\n\n# Ensure types match training data exactly\nbuilding_metadata['building_id'] = building_metadata['building_id'].astype('int16')\nbuilding_metadata['site_id'] = building_metadata['site_id'].astype('int8')\nbuilding_metadata['primary_use'] = building_metadata['primary_use'].astype('category')\n\nweather_test['site_id'] = weather_test['site_id'].astype('int8')\nweather_test['timestamp'] = pd.to_datetime(weather_test['timestamp'])\n\n# Fix weather test NaNs cleanly before chunking\nprint(\"Cleaning weather test anomalies...\")\nweather_cols = ['air_temperature', 'cloud_coverage', 'dew_temperature', \n                'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'wind_speed']\nfor col in weather_cols:\n    weather_test[col] = weather_test.groupby('site_id')[col].transform(\n        lambda x: x.interpolate(method='linear').ffill().bfill()\n    ).astype('float32')\n\n# Create rolling weather metrics directly on the full weather dataframe\nweather_test = weather_test.sort_values(by=['site_id', 'timestamp']).reset_index(drop=True)\nweather_test['air_temp_lag_3h'] = weather_test.groupby('site_id')['air_temperature'].shift(3)\nweather_test['air_temp_lag_3h'] = weather_test['air_temp_lag_3h'].fillna(weather_test['air_temperature']).astype('float32')\nweather_test['air_temp_rolling_mean_24h'] = weather_test.groupby('site_id')['air_temperature'].transform(\n    lambda x: x.rolling(window=24, min_periods=1).mean()\n).astype('float32')\n\n# 2. Initialize the submission file structure\nsubmission_file = 'submission.csv'\nif os.path.exists(submission_file):\n    os.remove(submission_file)\n\n# Process in blocks of 5 million rows to stay safely within RAM limits\nchunk_size = 5_000_000 \nprint(\"Beginning chunked inference engine...\")\n\nfor chunk_idx, chunk in enumerate(pd.read_csv(test_path, chunksize=chunk_size)):\n    print(f\"Processing chunk {chunk_idx + 1}...\")\n    \n    # Cast base chunk types\n    chunk['building_id'] = chunk['building_id'].astype('int16')\n    chunk['meter'] = chunk['meter'].astype('int8')\n    chunk['timestamp'] = pd.to_datetime(chunk['timestamp'])\n    \n    # Merge structural and environmental features\n    chunk = chunk.merge(building_metadata[['building_id', 'site_id', 'primary_use']], on='building_id', how='left')\n    chunk = chunk.merge(weather_test, on=['site_id', 'timestamp'], how='left')\n    \n    # Generate on-the-fly temporal features\n    chunk['hour'] = chunk['timestamp'].dt.hour.astype('int32')\n    chunk['dayofweek'] = chunk['timestamp'].dt.dayofweek.astype('int32')\n    chunk['month'] = chunk['timestamp'].dt.month.astype('int32')\n    chunk['is_weekend'] = chunk['dayofweek'].isin([5, 6]).astype('int64')\n    \n    # Generate on-the-fly thermodynamic interactions\n    chunk['heating_degree_days'] = (18.0 - chunk['air_temperature']).clip(lower=0).astype('float32')\n    chunk['cooling_degree_days'] = (chunk['air_temperature'] - 18.0).clip(lower=0).astype('float32')\n    chunk['sticky_weather_index'] = (chunk['air_temperature'] * chunk['dew_temperature']).astype('float32')\n    \n    # Initialize array to store predictions for this specific block\n    chunk_preds = np.zeros(len(chunk))\n    \n    # Route row subsets to their specific model\n    for meter_id, model in models.items():\n        meter_mask = chunk['meter'] == meter_id\n        if meter_mask.any():\n            X_test = chunk[meter_mask][features]\n            # Predict log values and immediately flip them back to real kWh values via expm1\n            chunk_preds[meter_mask] = np.expm1(model.predict(X_test))\n            \n    # Clip any accidental minor negative values back to absolute zero\n    chunk_preds = np.clip(chunk_preds, a_min=0, a_max=None)\n    \n    # Create clean submission piece\n    submission_part = pd.DataFrame({\n        'row_id': chunk['row_id'],\n        'meter_reading': chunk_preds\n    })\n    \n    # Stream directly to disk to keep RAM usage totally flat\n    if chunk_idx == 0:\n        submission_part.to_csv(submission_file, index=False)\n    else:\n        submission_part.to_csv(submission_file, mode='a', header=False, index=False)\n\nprint(\"Inference successfully complete! submission.csv is ready for export.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T09:23:53.157366Z","iopub.execute_input":"2026-06-02T09:23:53.157720Z","iopub.status.idle":"2026-06-02T09:34:17.775850Z","shell.execute_reply.started":"2026-06-02T09:23:53.157687Z","shell.execute_reply":"2026-06-02T09:34:17.775066Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submissions = pd.read_csv('/kaggle/working/submission.csv')\nsubmissions.head(20)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-02T09:36:54.454477Z","iopub.execute_input":"2026-06-02T09:36:54.454793Z","iopub.status.idle":"2026-06-02T09:37:02.116810Z","shell.execute_reply.started":"2026-06-02T09:36:54.454768Z","shell.execute_reply":"2026-06-02T09:37:02.116221Z"}},"outputs":[],"execution_count":null}]}