{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceType":"competition","sourceId":9994,"databundleVersionId":752467}],"dockerImageVersionId":31329,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Predicting Building Energy Consumption with Machine Learning\n\n## 1. Introduction and Data Loading\nIn this section, we import the necessary libraries and load the ASHRAE dataset. Since the data is massive (over 20 million rows), we will apply memory optimization techniques to prevent Out-of-Memory (OOM) errors and ensure smooth execution.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nimport gc\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Standard Kaggle paths for ASHRAE dataset\npath_train = '/kaggle/input/competitions/ashrae-energy-prediction/train.csv'\npath_building = '/kaggle/input/competitions/ashrae-energy-prediction/building_metadata.csv'\npath_weather = '/kaggle/input/competitions/ashrae-energy-prediction/weather_train.csv'\n\n# Memory Optimization Function\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: \n        print('Memory usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n\nprint(\"Loading and optimizing data... This might take a while.\")\n\ntrain_df = pd.read_csv(path_train)\nbuilding_df = pd.read_csv(path_building)\nweather_train_df = pd.read_csv(path_weather)\n\ntrain_df = reduce_mem_usage(train_df)\nbuilding_df = reduce_mem_usage(building_df)\nweather_train_df = reduce_mem_usage(weather_train_df)\n\nprint(\"\\nData shapes after loading:\")\nprint(\"Train Data:\", train_df.shape)\nprint(\"Building Data:\", building_df.shape)\nprint(\"Weather Data:\", weather_train_df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:04:53.536021Z","iopub.execute_input":"2026-03-27T23:04:53.536778Z","iopub.status.idle":"2026-03-27T23:05:08.104363Z","shell.execute_reply.started":"2026-03-27T23:04:53.536743Z","shell.execute_reply":"2026-03-27T23:05:08.103470Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Data Merging and Exploratory Data Analysis (EDA)\nTo build a robust model, we need to combine our target variable (`meter_reading`) with the physical attributes of the buildings and the weather conditions at the time of the reading. \n\n**Steps in this section:**\n1. **Merge Data:** Left merge `train_df` with `building_meta` and `weather_train`.\n2. **Re-optimize Memory:** Merging expands the dataframe, so we must compress it again.\n3. **Missing Values Analysis:** Identify which columns have missing data.\n4. **Visualizations:** - Observe the distribution of the target variable (using log transformation due to extreme skewness).\n   - Analyze monthly consumption trends by meter type to spot seasonality.\n   - Plot a correlation heatmap to see which features directly impact energy usage.","metadata":{}},{"cell_type":"code","source":"# 1. Convert timestamps to datetime objects\ntrain_df['timestamp'] = pd.to_datetime(train_df['timestamp'])\nweather_train_df['timestamp'] = pd.to_datetime(weather_train_df['timestamp'])\n\n# 2. Merge building metadata\nprint(\"Merging building metadata...\")\ntrain_df = train_df.merge(building_df, on='building_id', how='left')\n\n# 3. Merge weather data\nprint(\"Merging weather data...\")\ntrain_df = train_df.merge(weather_train_df, on=['site_id', 'timestamp'], how='left')\n\n# Free up RAM by deleting original unmerged dataframes\ndel building_df\ndel weather_train_df\ngc.collect()\n\n# 4. Re-apply memory optimization on the massive merged dataframe\nprint(\"Re-optimizing memory after merging...\")\ntrain_df = reduce_mem_usage(train_df)\n\n# 5. Missing Values Analysis\nprint(\"\\n--- Missing Values Percentage ---\")\nmissing_stats = (train_df.isnull().sum() / len(train_df)) * 100\nprint(missing_stats[missing_stats > 0].sort_values(ascending=False).round(2).astype(str) + ' %')\n\n# Extract 'month' temporarily for EDA time-series plotting\ntrain_df['month'] = train_df['timestamp'].dt.month\n\n# 6. Data Visualization\nprint(\"\\nGenerating EDA plots... (This might take a moment on 20M rows)\")\nfig, axes = plt.subplots(3, 1, figsize=(14, 20))\nfig.subplots_adjust(hspace=0.4)\n\n# Plot A: Target Variable Distribution\n# We use log1p because energy consumption data is heavily right-skewed (lots of zeros and a few massive peaks)\nsns.histplot(np.log1p(train_df['meter_reading']), bins=50, ax=axes[0], color='royalblue')\naxes[0].set_title('Log Transformed Target Distribution: log1p(meter_reading)', fontsize=14)\naxes[0].set_xlabel('log1p(Meter Reading)')\naxes[0].set_ylabel('Frequency')\n\n# Plot B: Time Series Seasonality by Meter Type\n# 0: electricity, 1: chilledwater, 2: steam, 3: hotwater\nmeter_dict = {0: 'Electricity', 1: 'Chilled Water', 2: 'Steam', 3: 'Hot Water'}\nmonthly_consumption = train_df.groupby(['month', 'meter'])['meter_reading'].mean().reset_index()\nmonthly_consumption['meter'] = monthly_consumption['meter'].map(meter_dict)\n\nsns.lineplot(data=monthly_consumption, x='month', y='meter_reading', hue='meter', ax=axes[1], marker='o', linewidth=2)\naxes[1].set_title('Average Monthly Energy Consumption by Meter Type', fontsize=14)\naxes[1].set_xlabel('Month (1 to 12)')\naxes[1].set_ylabel('Average Meter Reading (kWh or equivalent)')\naxes[1].set_xticks(range(1, 13))\n\n# Plot C: Correlation Heatmap\n# Select only numeric columns for correlation to avoid errors\nnumeric_cols = train_df.select_dtypes(include=[np.number]).columns\ncorr_matrix = train_df[numeric_cols].corr()\n\n# Mask the upper triangle for a cleaner look\nmask = np.triu(np.ones_like(corr_matrix, dtype=bool))\nsns.heatmap(corr_matrix, mask=mask, cmap='coolwarm', vmin=-1, vmax=1, annot=False, ax=axes[2])\naxes[2].set_title('Feature Correlation Heatmap', fontsize=14)\n\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:06:29.093822Z","iopub.execute_input":"2026-03-27T23:06:29.094171Z","iopub.status.idle":"2026-03-27T23:07:09.294397Z","shell.execute_reply.started":"2026-03-27T23:06:29.094144Z","shell.execute_reply":"2026-03-27T23:07:09.293655Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Feature Engineering & Data Cleaning\nBased on our EDA, we will now prepare our features for the machine learning models. \n\n**Transformations applied:**\n1. **Drop Low-Value Columns:** `cloud_coverage` (>50% missing), `precip_depth_1_hr`, `sea_level_pressure`, and `wind_direction` are removed to save memory and reduce noise. `year_built` and `floor_count` are also dropped due to massive missing rates in metadata.\n2. **Time Features:** Extract `hour`, `dayofweek`, and `is_weekend` from the timestamp to capture human behavior patterns.\n3. **Imputation:** Fill the remaining minor missing values in `air_temperature` and `dew_temperature` with their median values.\n4. **Log Transformation:** Apply `log1p` to `square_feet` to normalize its distribution, and to the target `meter_reading` as the competition evaluates on RMSLE.\n5. **Categorical Encoding:** Label encode `primary_use`.","metadata":{}},{"cell_type":"code","source":"# 1. Drop highly missing / low correlation columns\ndrop_cols = ['cloud_coverage', 'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction', 'year_built', 'floor_count']\ntrain_df.drop(columns=[col for col in drop_cols if col in train_df.columns], inplace=True)\n\n# 2. Extract Time Features\nprint(\"Extracting time features...\")\ntrain_df['hour'] = train_df['timestamp'].dt.hour\ntrain_df['dayofweek'] = train_df['timestamp'].dt.dayofweek\ntrain_df['is_weekend'] = (train_df['dayofweek'] >= 5).astype(np.int8)\n\n# Drop timestamp as we extracted what we need (month was extracted in EDA)\ntrain_df.drop(columns=['timestamp'], inplace=True)\n\n# 3. Handle remaining missing weather data (Median Imputation)\nprint(\"Imputing missing weather data...\")\nweather_cols = ['air_temperature', 'dew_temperature', 'wind_speed']\nfor col in weather_cols:\n    if col in train_df.columns:\n        train_df[col].fillna(train_df[col].median(), inplace=True)\n\n# 4. Log Transformations\nprint(\"Applying log transformations...\")\ntrain_df['log_square_feet'] = np.log1p(train_df['square_feet'])\ntrain_df.drop(columns=['square_feet'], inplace=True)\n\n# Target variable log transformation\ntrain_df['target'] = np.log1p(train_df['meter_reading'])\ntrain_df.drop(columns=['meter_reading'], inplace=True)\n\n# 5. Encode Categorical Variables (primary_use)\nfrom sklearn.preprocessing import LabelEncoder\nle = LabelEncoder()\ntrain_df['primary_use'] = le.fit_transform(train_df['primary_use']).astype(np.int8)\n\n# Re-optimize memory one last time before modeling\ntrain_df = reduce_mem_usage(train_df, verbose=False)\n\nprint(\"\\nFeature Engineering Complete!\")\nprint(\"New Data Shape:\", train_df.shape)\nprint(\"Remaining Missing Values Check:\\n\", train_df.isnull().sum())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:11:17.966121Z","iopub.execute_input":"2026-03-27T23:11:17.966710Z","iopub.status.idle":"2026-03-27T23:11:28.973712Z","shell.execute_reply.started":"2026-03-27T23:11:17.966678Z","shell.execute_reply":"2026-03-27T23:11:28.972701Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Data Splitting (Half & Half) and Scaling\nIn time-series forecasting, a random `train_test_split` causes data leakage. To evaluate our models robustly, we split the training data chronologically:\n* **Half 1:** January to June (Months 1-6)\n* **Half 2:** July to December (Months 7-12)\n\nFurthermore, models like Linear Regression and CNNs are highly sensitive to the scale of input features. We will apply `StandardScaler` to our continuous numerical variables before modeling.","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n\nprint(\"Splitting data chronologically into Half 1 and Half 2...\")\n\n# Define target and features\ntarget = 'target'\nfeatures = [col for col in train_df.columns if col != target]\n\n# Continuous numerical columns that need scaling\nnum_cols = ['air_temperature', 'dew_temperature', 'wind_speed', 'log_square_feet']\n\n# Split the data\nX_half_1 = train_df[train_df['month'] <= 6][features].copy()\ny_half_1 = train_df[train_df['month'] <= 6][target].copy()\n\nX_half_2 = train_df[train_df['month'] > 6][features].copy()\ny_half_2 = train_df[train_df['month'] > 6][target].copy()\n\nprint(\"Half 1 shape:\", X_half_1.shape)\nprint(\"Half 2 shape:\", X_half_2.shape)\n\nprint(\"\\nApplying StandardScaler to numerical features...\")\nscaler = StandardScaler()\n\n# Fit on Half 1, transform both (to avoid data leakage in real scenarios, though here we cross-validate)\nX_half_1[num_cols] = scaler.fit_transform(X_half_1[num_cols])\nX_half_2[num_cols] = scaler.transform(X_half_2[num_cols])\n\n# Also create a scaler fitted on Half 2 for the reverse validation\nscaler_rev = StandardScaler()\nX_half_2_rev = X_half_2.copy()\nX_half_1_rev = X_half_1.copy()\n\nX_half_2_rev[num_cols] = scaler_rev.fit_transform(X_half_2[num_cols])\nX_half_1_rev[num_cols] = scaler_rev.transform(X_half_1[num_cols])\n\nprint(\"Data is ready for modeling!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:14:52.908743Z","iopub.execute_input":"2026-03-27T23:14:52.909485Z","iopub.status.idle":"2026-03-27T23:15:03.551842Z","shell.execute_reply.started":"2026-03-27T23:14:52.909454Z","shell.execute_reply":"2026-03-27T23:15:03.551047Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Machine Learning Models\n\n### 5.1 Baseline Model: Linear Regression\nWe start with a traditional Linear Regression model to establish a baseline performance. Because building energy consumption is highly non-linear and involves complex interactions between categorical variables (like meter type and building usage), we expect this model to underperform compared to tree-based and deep learning methods. However, it provides a crucial mathematical anchor for our academic comparison.","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_squared_error\n\nprint(\"--- Training Linear Regression (Baseline) ---\")\n\n# Initialize models\nlr_model_1 = LinearRegression()\nlr_model_2 = LinearRegression()\n\n# ----------------- TRAIN MODEL 1 -----------------\n# Train on Half 1, Predict on Half 2\nprint(\"Training Model 1 (Train: Half 1, Val: Half 2)...\")\nlr_model_1.fit(X_half_1, y_half_1)\npred_lr_half_2 = lr_model_1.predict(X_half_2)\n\n# ----------------- TRAIN MODEL 2 -----------------\n# Train on Half 2, Predict on Half 1\nprint(\"Training Model 2 (Train: Half 2, Val: Half 1)...\")\nlr_model_2.fit(X_half_2_rev, y_half_2)\npred_lr_half_1 = lr_model_2.predict(X_half_1_rev)\n\n# ----------------- EVALUATION -----------------\n# Clip predictions to 0 because energy consumption cannot be negative\npred_lr_half_2 = np.clip(pred_lr_half_2, 0, None)\npred_lr_half_1 = np.clip(pred_lr_half_1, 0, None)\n\n# Calculate RMSE (which equates to RMSLE because target is log1p)\nrmse_lr_1 = np.sqrt(mean_squared_error(y_half_2, pred_lr_half_2))\nrmse_lr_2 = np.sqrt(mean_squared_error(y_half_1, pred_lr_half_1))\nlr_avg_rmse = (rmse_lr_1 + rmse_lr_2) / 2\n\nprint(f\"\\nLinear Regression -> Half 2 Validation RMSLE: {rmse_lr_1:.4f}\")\nprint(f\"Linear Regression -> Half 1 Validation RMSLE: {rmse_lr_2:.4f}\")\nprint(f\"**Linear Regression Combined RMSLE: {lr_avg_rmse:.4f}**\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:15:38.645594Z","iopub.execute_input":"2026-03-27T23:15:38.645859Z","iopub.status.idle":"2026-03-27T23:15:44.390485Z","shell.execute_reply.started":"2026-03-27T23:15:38.645838Z","shell.execute_reply":"2026-03-27T23:15:44.389766Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 5.2 Tree-Based Model: Random Forest Regressor\nNext, we apply a Random Forest Regressor. Unlike Linear Regression, tree-based models can capture non-linear relationships and complex interactions between features (e.g., how the hour of the day interacts with the building's primary use). \n\n**Memory Constraint Strategy:** Because Random Forest builds multiple decision trees simultaneously, training it on 20 million rows requires hundreds of gigabytes of RAM. To prevent Out-of-Memory (OOM) crashes, we will train the model on a representative **10% random subsample** of the training data, while still validating it on the entire validation set. We also limit `max_depth` to 10 and `n_estimators` to 30.","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_squared_error\nimport numpy as np\n\nprint(\"--- Training Random Forest Regressor ---\")\nprint(\"Extracting a 10% random subsample for training to prevent RAM crash...\")\n\n# Subsample 10% of the training data\nsample_fraction = 0.10\n\n# Half 1 Sample\nX_half_1_sample = X_half_1.sample(frac=sample_fraction, random_state=42)\ny_half_1_sample = y_half_1.loc[X_half_1_sample.index]\n\n# Half 2 Sample (Using the scaled versions)\nX_half_2_sample = X_half_2_rev.sample(frac=sample_fraction, random_state=42)\ny_half_2_sample = y_half_2.loc[X_half_2_sample.index]\n\n# Initialize models with constrained parameters to save memory and time\nrf_model_1 = RandomForestRegressor(n_estimators=30, max_depth=10, random_state=42, n_jobs=-1)\nrf_model_2 = RandomForestRegressor(n_estimators=30, max_depth=10, random_state=42, n_jobs=-1)\n\n# ----------------- TRAIN MODEL 1 -----------------\nprint(\"Training Model 1 (Train: 10% of Half 1, Val: All of Half 2)...\")\nrf_model_1.fit(X_half_1_sample, y_half_1_sample)\npred_rf_half_2 = rf_model_1.predict(X_half_2)\n\n# ----------------- TRAIN MODEL 2 -----------------\nprint(\"Training Model 2 (Train: 10% of Half 2, Val: All of Half 1)...\")\nrf_model_2.fit(X_half_2_sample, y_half_2_sample)\npred_rf_half_1 = rf_model_2.predict(X_half_1_rev)\n\n# ----------------- EVALUATION -----------------\n# Clip predictions to 0\npred_rf_half_2 = np.clip(pred_rf_half_2, 0, None)\npred_rf_half_1 = np.clip(pred_rf_half_1, 0, None)\n\nrmse_rf_1 = np.sqrt(mean_squared_error(y_half_2, pred_rf_half_2))\nrmse_rf_2 = np.sqrt(mean_squared_error(y_half_1, pred_rf_half_1))\nrf_avg_rmse = (rmse_rf_1 + rmse_rf_2) / 2\n\nprint(f\"\\nRandom Forest -> Half 2 Validation RMSLE: {rmse_rf_1:.4f}\")\nprint(f\"Random Forest -> Half 1 Validation RMSLE: {rmse_rf_2:.4f}\")\nprint(f\"**Random Forest Combined RMSLE: {rf_avg_rmse:.4f}**\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:17:01.694006Z","iopub.execute_input":"2026-03-27T23:17:01.694524Z","iopub.status.idle":"2026-03-27T23:18:36.861724Z","shell.execute_reply.started":"2026-03-27T23:17:01.694496Z","shell.execute_reply":"2026-03-27T23:18:36.860846Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 5.3 Gradient Boosting Model: LightGBM\nLightGBM is highly optimized for tabular data and handles large-scale datasets efficiently. Unlike Random Forest, which builds trees independently, LightGBM builds trees sequentially, where each new tree corrects the errors of the previous ones. \n\nCrucially, LightGBM has native support for categorical features. We don't need to perform memory-intensive One-Hot Encoding; we simply pass a list of categorical columns to the model. Because of its memory efficiency, we can train LightGBM on **100% of the training data** for each half.","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\nfrom sklearn.metrics import mean_squared_error\n\nprint(\"--- Training LightGBM ---\")\n\n# We use the unscaled features for LightGBM, as tree-based models don't require scaling\n# and handling raw categorical values is better for its internal algorithm.\nfeatures_lgb = [col for col in train_df.columns if col != 'target']\ntarget = 'target'\ncategorical_features = ['building_id', 'site_id', 'meter', 'primary_use', 'month', 'hour', 'dayofweek', 'is_weekend']\n\n# Split unscaled data chronologically\nX_half_1_lgb = train_df[train_df['month'] <= 6][features_lgb]\ny_half_1_lgb = train_df[train_df['month'] <= 6][target]\n\nX_half_2_lgb = train_df[train_df['month'] > 6][features_lgb]\ny_half_2_lgb = train_df[train_df['month'] > 6][target]\n\n# Create LightGBM Datasets\nprint(\"Creating LightGBM Datasets...\")\nd_half_1 = lgb.Dataset(X_half_1_lgb, label=y_half_1_lgb, categorical_feature=categorical_features, free_raw_data=False)\nd_half_2 = lgb.Dataset(X_half_2_lgb, label=y_half_2_lgb, categorical_feature=categorical_features, free_raw_data=False)\n\n# LightGBM Parameters\nparams = {\n    \"objective\": \"regression\",\n    \"boosting\": \"gbdt\",\n    \"num_leaves\": 40,\n    \"learning_rate\": 0.05,\n    \"feature_fraction\": 0.85,\n    \"reg_lambda\": 2,\n    \"metric\": \"rmse\",\n    \"random_state\": 42\n}\n\n# ----------------- TRAIN MODEL 1 -----------------\nprint(\"\\nTraining Model 1 (Train: All Half 1, Val: All Half 2)...\")\nmodel_lgb_1 = lgb.train(\n    params, \n    train_set=d_half_1, \n    num_boost_round=1000, \n    valid_sets=[d_half_1, d_half_2], \n    callbacks=[lgb.early_stopping(stopping_rounds=200), lgb.log_evaluation(period=200)]\n)\n\n# ----------------- TRAIN MODEL 2 -----------------\nprint(\"\\nTraining Model 2 (Train: All Half 2, Val: All Half 1)...\")\nmodel_lgb_2 = lgb.train(\n    params, \n    train_set=d_half_2, \n    num_boost_round=1000, \n    valid_sets=[d_half_2, d_half_1], \n    callbacks=[lgb.early_stopping(stopping_rounds=200), lgb.log_evaluation(period=200)]\n)\n\n# ----------------- EVALUATION -----------------\nprint(\"\\nCalculating Predictions...\")\npred_lgb_half_2 = model_lgb_1.predict(X_half_2_lgb, num_iteration=model_lgb_1.best_iteration)\npred_lgb_half_1 = model_lgb_2.predict(X_half_1_lgb, num_iteration=model_lgb_2.best_iteration)\n\n# Clip predictions to 0\npred_lgb_half_2 = np.clip(pred_lgb_half_2, 0, None)\npred_lgb_half_1 = np.clip(pred_lgb_half_1, 0, None)\n\nrmse_lgb_1 = np.sqrt(mean_squared_error(y_half_2_lgb, pred_lgb_half_2))\nrmse_lgb_2 = np.sqrt(mean_squared_error(y_half_1_lgb, pred_lgb_half_1))\nlgb_avg_rmse = (rmse_lgb_1 + rmse_lgb_2) / 2\n\nprint(f\"\\nLightGBM -> Half 2 Validation RMSLE: {rmse_lgb_1:.4f}\")\nprint(f\"LightGBM -> Half 1 Validation RMSLE: {rmse_lgb_2:.4f}\")\nprint(f\"**LightGBM Combined RMSLE: {lgb_avg_rmse:.4f}**\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:19:54.999030Z","iopub.execute_input":"2026-03-27T23:19:54.999777Z","iopub.status.idle":"2026-03-27T23:28:10.130438Z","shell.execute_reply.started":"2026-03-27T23:19:54.999743Z","shell.execute_reply":"2026-03-27T23:28:10.129546Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 5.4 Advanced 1D CNN (100% Data & Deep Architecture)\nTo significantly improve our neural network's performance, we will now train it on **100% of the dataset**. We have also upgraded the architecture specifically for tabular data by introducing:\n* **Batch Normalization:** To stabilize and accelerate training.\n* **Dropout Layers (20%):** To prevent the model from overfitting on the massive training set.\n* **Deeper Convolutional Blocks:** Using 64 filters and `padding='same'` to capture more intricate patterns before flattening.\n* **Large Batch Size (8192):** To fully utilize Kaggle's GPU capabilities and keep training times efficient across 10 million rows per fold.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv1D, MaxPooling1D, Flatten, Dense, Dropout, BatchNormalization\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau\nfrom sklearn.metrics import mean_squared_error\nimport numpy as np\n\nprint(\"--- Training Advanced 1D CNN (100% Data) ---\")\n\n# Reshaping 100% of the scaled tabular data into 3D shape (samples, features, 1)\ninput_shape = (X_half_1.shape[1], 1)\n\nprint(\"Reshaping massive datasets for CNN... (This takes a few seconds)\")\nX_train_1_3d = X_half_1.values.reshape(X_half_1.shape[0], X_half_1.shape[1], 1)\nX_val_2_3d = X_half_2.values.reshape(X_half_2.shape[0], X_half_2.shape[1], 1)\n\nX_train_2_3d = X_half_2_rev.values.reshape(X_half_2_rev.shape[0], X_half_2_rev.shape[1], 1)\nX_val_1_3d = X_half_1_rev.values.reshape(X_half_1_rev.shape[0], X_half_1_rev.shape[1], 1)\n\n# Advanced 1D CNN Architecture\ndef build_advanced_cnn():\n    model = Sequential([\n        # First Convolutional Block\n        Conv1D(filters=64, kernel_size=3, padding='same', activation='relu', input_shape=input_shape),\n        BatchNormalization(),\n        Conv1D(filters=64, kernel_size=3, padding='same', activation='relu'),\n        BatchNormalization(),\n        MaxPooling1D(pool_size=2),\n        Dropout(0.2),\n\n        # Fully Connected Block\n        Flatten(),\n        Dense(128, activation='relu'),\n        BatchNormalization(),\n        Dropout(0.2),\n        Dense(32, activation='relu'),\n        Dense(1) # Linear output for regression\n    ])\n    \n    # Using Adam optimizer with a slightly higher initial learning rate\n    opt = tf.keras.optimizers.Adam(learning_rate=0.005)\n    model.compile(optimizer=opt, loss='mse')\n    return model\n\n# Callbacks: Stop early if no improvement, reduce learning rate if plateauing\nearly_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=1, min_lr=0.0001)\n\n# ----------------- TRAIN MODEL 1 -----------------\nprint(\"\\nTraining Advanced CNN Model 1 (Train: 100% Half 1, Val: 100% Half 2)...\")\ncnn_model_1 = build_advanced_cnn()\ncnn_model_1.fit(\n    X_train_1_3d, y_half_1, \n    validation_data=(X_val_2_3d, y_half_2), \n    epochs=10, \n    batch_size=8192, \n    callbacks=[early_stop, reduce_lr],\n    verbose=1\n)\n\n# ----------------- TRAIN MODEL 2 -----------------\nprint(\"\\nTraining Advanced CNN Model 2 (Train: 100% Half 2, Val: 100% Half 1)...\")\ncnn_model_2 = build_advanced_cnn()\ncnn_model_2.fit(\n    X_train_2_3d, y_half_2, \n    validation_data=(X_val_1_3d, y_half_1), \n    epochs=10, \n    batch_size=8192, \n    callbacks=[early_stop, reduce_lr],\n    verbose=1\n)\n\n# ----------------- EVALUATION -----------------\nprint(\"\\nCalculating Final CNN Predictions...\")\npred_cnn_half_2 = cnn_model_1.predict(X_val_2_3d, batch_size=8192).flatten()\npred_cnn_half_1 = cnn_model_2.predict(X_val_1_3d, batch_size=8192).flatten()\n\n# Clip predictions to 0\npred_cnn_half_2 = np.clip(pred_cnn_half_2, 0, None)\npred_cnn_half_1 = np.clip(pred_cnn_half_1, 0, None)\n\nrmse_cnn_1 = np.sqrt(mean_squared_error(y_half_2, pred_cnn_half_2))\nrmse_cnn_2 = np.sqrt(mean_squared_error(y_half_1, pred_cnn_half_1))\ncnn_avg_rmse = (rmse_cnn_1 + rmse_cnn_2) / 2\n\nprint(f\"\\nAdvanced 1D CNN -> Half 2 Validation RMSLE: {rmse_cnn_1:.4f}\")\nprint(f\"Advanced 1D CNN -> Half 1 Validation RMSLE: {rmse_cnn_2:.4f}\")\nprint(f\"**Advanced 1D CNN Combined RMSLE: {cnn_avg_rmse:.4f}**\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:38:37.334417Z","iopub.execute_input":"2026-03-27T23:38:37.335652Z","iopub.status.idle":"2026-03-27T23:44:30.312130Z","shell.execute_reply.started":"2026-03-27T23:38:37.335620Z","shell.execute_reply":"2026-03-27T23:44:30.311140Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 6. Feature Importance (LightGBM)\nTo interpret our best-performing model (LightGBM), we visualize the Feature Importance. This helps us understand which variables (e.g., building size, outside temperature, hour of the day) most heavily influence energy consumption.","metadata":{}},{"cell_type":"code","source":"print(\"Generating Feature Importance Plot from LightGBM...\")\n\n# Get feature importances from both models\nimportance_1 = model_lgb_1.feature_importance(importance_type='split')\nimportance_2 = model_lgb_2.feature_importance(importance_type='split')\n\n# Calculate average importance\navg_importance = (importance_1 + importance_2) / 2\n\n# Create a DataFrame for visualization\nfeature_names = X_half_1_lgb.columns\nimportance_df = pd.DataFrame({\n    'Feature': feature_names,\n    'Importance': avg_importance\n}).sort_values(by='Importance', ascending=False)\n\n# Plotting\nplt.figure(figsize=(12, 8))\nsns.barplot(x='Importance', y='Feature', data=importance_df, palette='viridis')\nplt.title('LightGBM Average Feature Importance (Split Type)', fontsize=16)\nplt.xlabel('Average Number of Splits', fontsize=12)\nplt.ylabel('Features', fontsize=12)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-27T23:46:03.576761Z","iopub.execute_input":"2026-03-27T23:46:03.577678Z","iopub.status.idle":"2026-03-27T23:46:03.857289Z","shell.execute_reply.started":"2026-03-27T23:46:03.577648Z","shell.execute_reply":"2026-03-27T23:46:03.856449Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Comprehensive Project Overview: Predicting Building Energy Consumption using Machine Learning\n\n## 1. Introduction and Objectives\nThe primary objective of this project is to accurately predict the energy consumption (`meter_reading`) of 1,449 buildings across various sites over a one-year period, utilizing the **ASHRAE Great Energy Predictor III** dataset. Building energy consumption is highly dynamic, influenced by complex, non-linear interactions between weather conditions, time of day, and specific building characteristics.\n\nThis notebook demonstrates a complete end-to-end data science pipeline: from aggressive memory optimization and exploratory data analysis to advanced feature engineering and the comparative evaluation of four distinct machine learning architectures (Linear Regression, Random Forest, LightGBM, and a 1D Convolutional Neural Network).\n\n---\n\n## 2. Data Processing and Memory Optimization\nThe dataset comprises over 20 million rows, presenting a significant computational challenge. To prevent Out-of-Memory (OOM) failures:\n* **Memory Reduction (`reduce_mem_usage`):** A custom function was utilized to downcast numerical data types (e.g., converting `float64` to `float16` or `int8` where appropriate), reducing the dataframe's memory footprint by over 50%.\n* **Data Merging:** The primary `train` dataset was sequentially merged with `building_metadata` and `weather_train` datasets using `building_id` and `site_id` + `timestamp` as keys. \n\n---\n\n## 3. Exploratory Data Analysis (EDA) & Data Cleaning\n### 3.1 Handling Missing Values\n* **Dropped Columns:** Features with extremely high missing ratios (e.g., `cloud_coverage` > 50%, `year_built`, `floor_count`) or near-zero correlation to the target (`sea_level_pressure`, `wind_direction`) were dropped to reduce noise and dimensionality.\n* **Median Imputation:** Minor missing values in continuous weather variables (`air_temperature`, `dew_temperature`, `wind_speed`) were imputed using their respective median values to maintain statistical integrity without introducing extreme outliers.\n\n### 3.2 Target Variable Transformation\nThe target variable (`meter_reading`) and the `square_feet` feature exhibited extreme right-skewness. To stabilize the variance, normalize the distribution, and align with the official competition evaluation metric (RMSLE), a **logarithmic transformation (`np.log1p`)** was applied to both.\n\n---\n\n## 4. Feature Engineering\nTo enable the models to capture human behavior and seasonal trends, time-based features were extracted from the `timestamp`:\n* `hour` (0-23)\n* `dayofweek` (0-6)\n* `is_weekend` (Binary indicator)\n* `month` (1-12)\n\nCategorical variables (such as `primary_use`) were processed using **Label Encoding**, preparing them for tree-based models that natively support categorical splits without the explosive memory cost of One-Hot Encoding.\n\n---\n\n## 5. Validation Strategy: The \"Half and Half\" Split\nIn time-series forecasting, a random `train_test_split` leads to severe **Data Leakage**, as the model would effectively \"look into the future\" to predict the present, ignoring seasonality. \n\nTo ensure robust evaluation, a **Chronological Split** was implemented:\n* **Half 1:** Months 1 through 6.\n* **Half 2:** Months 7 through 12.\n* **Methodology:** Model 1 is trained on Half 1 and validates on Half 2. Model 2 is trained on Half 2 and validates on Half 1. The final evaluation metric is the average of both validation scores. Continuous numerical features were scaled using `StandardScaler` specifically for the Linear and CNN models.\n\n---\n\n## 6. Modeling Pipeline and Comparative Results\nFour different algorithms were trained and evaluated based on the **Combined Average RMSLE** score.\n\n1. **Linear Regression (Baseline):** * **Score:** `1.9804`\n   * **Takeaway:** Served as our mathematical baseline. As expected, it struggled to capture the complex, non-linear relationships in energy consumption.\n2. **Random Forest Regressor:** * **Score:** `1.6533`\n   * **Takeaway:** To prevent RAM exhaustion, it was trained on a random 10% subsample. Despite the reduced data, its tree-based architecture easily outperformed the baseline.\n3. **LightGBM (Gradient Boosting):** * **Score:** `1.5526` **(Best Performing)**\n   * **Takeaway:** Our champion model. Trained on 100% of the data. Its native handling of categorical features and sequential boosting allowed it to effectively parse the massive tabular dataset, achieving the lowest error rate.\n4. **Advanced 1D CNN (Deep Learning):** * **Score:** `1.6593`\n   * **Takeaway:** A deep neural network utilizing `Conv1D`, `BatchNormalization`, and `Dropout` layers, trained with large batch sizes (8192). While it demonstrated massive improvement over a shallow network and proved neural networks can learn tabular structures, it slightly trailed behind the highly optimized LightGBM.\n\n---\n\n## 7. Crucial Discussion: Feature Importance and \"Memorization vs. Generalization\"\nAnalysis of the LightGBM Feature Importance (using the `split` metric) revealed a massive dominance of the `building_id` feature, vastly outnumbering physical parameters like `air_temperature` or `square_feet`. \n\n**This highlights a fundamental philosophical dilemma in Machine Learning:**\n* **High Cardinality Bias:** The model heavily relied on `building_id` to reduce error. Instead of learning universal laws of physics (e.g., \"higher temperature = more cooling energy\"), the model essentially **memorized the baseline consumption profile of each specific building**. \n* **The Kaggle Context (Valid Use):** Because the test dataset contains the exact same buildings, using `building_id` to \"memorize\" their quirks is the optimal strategy to win the competition. It acts as a proxy for unknown variables (insulation quality, HVAC system age, etc.).\n* **Real-World Application (Dangerous Flaw):** If this model were deployed in the real world to predict the energy consumption of a *newly constructed* building (an unseen `building_id`), it would fail entirely. For true real-world generalization, the model would require extensive physical metadata rather than relying on unique identifiers.","metadata":{}}]}