{"metadata":{"colab":{"machine_shape":"hm","provenance":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.12.5"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":9994,"databundleVersionId":752467,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **ASHRAE - Great Energy Predictor III**","metadata":{"id":"-TAEcN6u3RaM"}},{"cell_type":"markdown","source":"Here, we will try to build end-to-end regression pipeline for great energy predictor competition (https://www.kaggle.com/competitions/ashrae-energy-prediction), where we asked to predict the energy consumed by a building given some features.","metadata":{"id":"GrEoii1f3VhK"}},{"cell_type":"markdown","source":"## Get  the Data and Quick Look at Data Structure\n","metadata":{"id":"5ubXKGnsowqS"}},{"cell_type":"markdown","source":"First, we begin by reading the data from the CSV. Since the CSV files are too large, instead of trying to read them in one shot, which may be computationally heavy, we read the CSV in chunks, process each chunk individually, and then concatenate them at the end.","metadata":{"id":"Oh03ik0r4VPU"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt","metadata":{"id":"Pdj3zKReo6xc"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"ashrae-energy-prediction/train.csv\", chunksize=10**6)\nbuilding_meta = pd.read_csv(\"ashrae-energy-prediction/building_metadata.csv\")\nweather_train = pd.read_csv(\"ashrae-energy-prediction/weather_train.csv\")\n\ntrain_chunks = []\nfor chunk in train:\n    merged_chunk = pd.merge(chunk, building_meta, on='building_id', how='left')\n    merged_chunk = pd.merge(merged_chunk, weather_train, on=['site_id', 'timestamp'], how='left')\n    train_chunks.append(merged_chunk)\n\ntrain_df = pd.concat(train_chunks)\n","metadata":{"id":"CGt9ImPho2DS"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will have a look on our data columns and check for how many null values.","metadata":{"id":"XRezBoGZ56mg"}},{"cell_type":"code","source":"train_df.head()","metadata":{"id":"0ihW73xxrRlq","outputId":"7d51e402-b843-42fc-a4cb-dba6264dc1d5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info(show_counts=True)","metadata":{"id":"0i4t5FyDvGDt","outputId":"2223a406-7306-42c0-b56f-177dbba04a16"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see above, we have two non-numeric column, which are *timestamp* (we will take a look later) and *primary_use*. Now we will see what categories inside the *primary_use*:\n\n","metadata":{"id":"6n06CAvq6Qba"}},{"cell_type":"code","source":"(\n    train_df.loc[:, \"primary_use\"]\n              .value_counts()\n)","metadata":{"id":"8EZ96A9Nvcr0","outputId":"213b5d1c-bdf7-4c68-e28b-b729ff0e4e25"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We also want to examine the composition of our data:","metadata":{"id":"04dH6XlP-XRL"}},{"cell_type":"code","source":"train_df.describe()","metadata":{"id":"5eFd_FYBvjcm","outputId":"71f0b4c4-1247-476c-d9bc-3dd84b8d1aab"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we found invalid valid values in *precip_depth_1_hr*, by definition it is : \"The precipitation depth indicates to what depth liquid precipitation would cover a horizontal surface in an observation period if nothing could drain, evaporate or percolate from this surface\" and the unit is in mm. This means negative values are impossible, so we will need to process this data further. Now, let's visualize the distribution of our attributes using histograms:","metadata":{"id":"eH0dneEF-0j5"}},{"cell_type":"code","source":"_ = train_df.hist(bins=50, figsize=(16, 12))","metadata":{"id":"U5eClUADvsOO","outputId":"83a96a78-c475-49da-da9d-bee0b6776e40"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the histograms above, we can observe that some columns have skewed distributions:*meter_reading*, *site_id*, *square_feet*, *floor_count*, *cloud_coverage*, *precip_depth_1_hr*, and *wind_speed*. It's also worth noting that there are potential outliers in *precip_depth_1_hr* and *meter_reading*, as the graphs show a single peak but the graph span across a large range of values, indicating the presence of data in those regions. To make sure if there are outliers, we will have a look of the log scale on those graphs:","metadata":{"id":"7qRjS_FTeprA"}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n\naxes[0].hist(train_df['meter_reading'], bins=50)\naxes[0].set_title('Meter Reading')\naxes[0].set_yscale('log')\naxes[0].set_xlabel('Meter Reading')\naxes[0].set_ylabel('Frequency (log scale)')\n\naxes[1].hist(train_df['precip_depth_1_hr'], bins=50)\naxes[1].set_title('Precipitation Depth (1 hr)')\naxes[1].set_yscale('log')\naxes[1].set_xlabel('Precip Depth (1 hr)')\naxes[1].set_ylabel('Frequency (log scale)')\n\nplt.tight_layout()\nplt.show()","metadata":{"id":"gZsgNRmv0xk6","outputId":"86012ef8-be98-4172-f697-7223e9079f1d"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the graphs above, it's clear that there are outliers, particularly in the lower ranges of meter reading and precipitation depth (1 hr).","metadata":{"id":"9EVSUXtsw7Tp"}},{"cell_type":"markdown","source":"## Create a Test Set","metadata":{"id":"KjAgdqXrmTAD"}},{"cell_type":"markdown","source":"Now, we will separate train set and test set using stratified sampling by using *primary_use* category:","metadata":{"id":"JOCYbUY3CdYc"}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nstrat_train_set, strat_test_set = train_test_split(train_df, test_size=0.2, stratify=train_df[\"primary_use\"], random_state=42)","metadata":{"id":"DWrKGSIBmVqT"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\njoblib.dump(strat_train_set, \"drive/MyDrive/Colab Notebooks/strat_train_set.pkl\")\njoblib.dump(strat_test_set, \"drive/MyDrive/Colab Notebooks/strat_test_set.pkl\")","metadata":{"id":"Jr8aSMoU5fr9","outputId":"c9f93a67-76e9-43e7-c050-7f816fb7bb1a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploratory Data Analysis","metadata":{"id":"qoENRskne3X1"}},{"cell_type":"markdown","source":"We will copy the previous data set to do exploratory data analysis.","metadata":{"id":"euq7IUtMFJxX"}},{"cell_type":"code","source":"from google.colab import drive\ndrive.mount('/content/drive')\n","metadata":{"id":"8ztfX11q2_H0","outputId":"e1e1080e-b58d-4a8f-961e-55b3ffd5f100"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"explore_df = strat_train_set.copy()","metadata":{"id":"LFOXVUUDtd4d"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Handling Timestamp","metadata":{"id":"9-3fruggC31r"}},{"cell_type":"markdown","source":"Directly feeding the timestamp to the regressor might not be useful (despite the datatype is not numeric too), so we need to process the *timestamp* (https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.html). To understand how we should further process the *timestamp*, we want to know how often the data is acquired for each *building_id*. Additionally, we want to determine the oldest and newest timestamps, so we can understand the span of the data.","metadata":{"id":"BciuyqXzUTFx"}},{"cell_type":"code","source":"explore_df['timestamp'] = pd.to_datetime(explore_df['timestamp'])\nexplore_df['time_diff'] = explore_df.groupby(['building_id', 'meter'])['timestamp'].diff()\n\noldest_timestamp = explore_df['timestamp'].min()\nnewest_timestamp = explore_df['timestamp'].max()\nexplore_df['month'] = explore_df['timestamp'].dt.to_period('M')\nmonthly_data = explore_df.groupby('month').size()\n\ntime_diff_counts = explore_df['time_diff'].value_counts()\nsorted_counts = pd.concat([time_diff_counts[time_diff_counts.index == '1 days 01:00:00'],\n                           time_diff_counts[time_diff_counts.index != '1 days 01:00:00']])\n\n\nprint(sorted_counts)\nprint(\"=\"*91)\nprint(f\"Oldest Timestamp: {oldest_timestamp}\")\nprint(f\"Newest Timestamp: {newest_timestamp}\")\nprint(monthly_data)","metadata":{"id":"CBHQCt8Atd4e","outputId":"e1179e55-6f18-40f2-a229-dcd2b3461fe5"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the data spans from the beginning of January until the end of December. Therefore, it makes sense to create a *month* category and then encode it. Additionally, we observe that within a single day, data is often acquired on an hourly basis, so creating an *hour* category is also logical. Furthermore, given that there are different types of *primary_use* for the buildings, some may only operate during weekdays and not weekends. This suggests that creating a *weekend* category could be useful. Finally, since the data spans an entire year, dividing it into *quarters* may help capture seasonal variations.","metadata":{"id":"rzgecgDfWMOF"}},{"cell_type":"code","source":"explore_df['weekend'] = explore_df['timestamp'].dt.dayofweek >= 5\nexplore_df['weekend'] = explore_df['weekend'].astype(int)\nexplore_df['quarter'] = explore_df['timestamp'].dt.quarter\nexplore_df['hour'] = explore_df['timestamp'].dt.hour","metadata":{"id":"xLQxVZQntd4f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"explore_df.info(show_counts=True)","metadata":{"id":"JjeJu128td4f","outputId":"4ccbbb33-9f98-4bca-c3b5-6004bb52b3e7"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we want to visualize the *meter_reading* with respect to time categories that we just made","metadata":{"id":"o43gGxCrngkG"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(5, 2))\nmonthly_readings = explore_df.groupby(explore_df['timestamp'].dt.to_period('M'))['meter_reading'].sum()\nmonthly_readings.plot(color='blue')\nplt.title('Total Meter Readings Aggregated by Month')\nplt.ylabel('Total Meter Reading')\nplt.xlabel('Month')\nplt.show()\n\nplt.figure(figsize=(5, 2))\nhourly_readings = explore_df.groupby(explore_df['timestamp'].dt.hour)['meter_reading'].sum()\nhourly_readings.plot(color='green')\nplt.title('Total Meter Readings Aggregated by Hour of Day')\nplt.ylabel('Total Meter Reading')\nplt.xlabel('Hour of Day (0-23)')\nplt.show()\n\nplt.figure(figsize=(5, 2))\nquarterly_readings = explore_df.groupby(explore_df['timestamp'].dt.quarter)['meter_reading'].sum()\nquarterly_readings.plot(color='red')\nplt.title('Total Meter Readings Aggregated by Quarter')\nplt.ylabel('Total Meter Reading')\nplt.xlabel('Quarter')\nplt.show()\n\n# Plot for Weekend Aggregation\nplt.figure(figsize=(5, 2))\nweekend_readings = explore_df.groupby(explore_df['timestamp'].dt.weekday >= 5)['meter_reading'].sum()  # Weekend is Saturday (5) and Sunday (6)\nweekend_readings.plot(kind='bar', color='purple')\nplt.title('Total Meter Readings Aggregated by Weekend')\nplt.ylabel('Total Meter Reading')\nplt.xlabel('Is Weekend (False=Weekday, True=Weekend)')\nplt.show()\n","metadata":{"id":"1MXtwK4Ftd4g","outputId":"bfe3b828-84ab-459e-d795-6a9b0bf0b170"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we want to know the *meter_reading* with respect to *primary_use* and *site_id*.","metadata":{"id":"V4gYGI9Lpduq"}},{"cell_type":"code","source":"explore_df.groupby('primary_use')['meter_reading'].sum().plot(kind='bar', figsize=(8, 3))\nplt.title('Total Energy Usage by Primary Use')\nplt.ylabel('Total Meter Reading')\nplt.show()\n","metadata":{"id":"MMpHa2dThYzy","outputId":"bddd3f1e-969e-491b-99ad-7442c8ab158a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"explore_df.groupby('site_id')['meter_reading'].sum().plot(kind='bar', figsize=(8, 3))\nplt.title('Total Energy Usage by Site')\nplt.ylabel('Total Meter Reading')\nplt.xlabel('Site ID')\nplt.show()\n","metadata":{"id":"7-fASXUEkghl","outputId":"31fbe4ac-fadd-4134-b2f7-f62ff3afdf98"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, *meter_reading* varies significantly across different *primary_use* categories and *site_id*.","metadata":{"id":"VwVfOBmH95it"}},{"cell_type":"markdown","source":"## Correlation","metadata":{"id":"lQrRDkPAkmS-"}},{"cell_type":"markdown","source":"Now, we want to examine the correlation between each feature and meter_reading. However, we should exclude weekend, quarter, and hour from the correlation analysis, as it is more appropriate to treat these features as categorical rather than numerical.","metadata":{"id":"fB6E1FdQq79q"}},{"cell_type":"code","source":"explore_df = explore_df.drop(columns=['timestamp', 'primary_use', 'time_diff', 'month'])\ncorr_matrix = explore_df.corr()","metadata":{"id":"N8SOUlGvsFYE"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_matrix[\"meter_reading\"]","metadata":{"id":"xb-2sBWhz-Xk","outputId":"389557f6-5145-4352-f9ba-a7c32192ac74"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pandas.plotting import scatter_matrix\n\nattributes = [\"meter_reading\", \"year_built\", \"floor_count\",\n              \"square_feet\"]\nsampled_df = explore_df[attributes].sample(frac=0.1, random_state=42)\nscatter_matrix(sampled_df, figsize=(12, 8))\nplt.show()","metadata":{"id":"pZ892q2RKABj","outputId":"05c5196b-7fa5-482e-858f-28b3062c4404"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the graph, we can see that there is no promising correlation. Therefore, we may need to create new attribute combinations.","metadata":{"id":"juLmvgQ6RS8X"}},{"cell_type":"markdown","source":"## Experimenting with Attribute Combination","metadata":{"id":"mBnMstsuK-zF"}},{"cell_type":"markdown","source":"We want to make new attributes:\n1. The ratio between square footage and the number of floors.\n2. The age of the building.\n3. The interaction between the building's area and the air temperature, which could indicate energy usage based on the size and temperature conditions.","metadata":{"id":"K462W0zaug5l"}},{"cell_type":"code","source":"explore_df['sqft_per_floor'] = explore_df['square_feet'] / explore_df['floor_count']\nexplore_df['building_age'] = explore_df['year_built'].max() - explore_df['year_built']\nexplore_df['temp_sqft_interaction'] = explore_df['air_temperature'] * explore_df['square_feet']\n\ncorr_matrix = explore_df.corr()\ncorr_matrix[\"meter_reading\"]. sort_values(ascending=False)","metadata":{"id":"0raVmRULTuEP","outputId":"06773fbc-9511-43dd-bfdd-1d4486460819"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, our new categories show good correlations with *meter_reading* (despite *building_age* which essentially is the negative of the* year_buit* since they basically represent the same data)","metadata":{"id":"J0oHI3WO-gPA"}},{"cell_type":"markdown","source":"# Prepare the Data","metadata":{"id":"bGWpC3jytd4k"}},{"cell_type":"code","source":"energy_train = strat_train_set.drop(\"meter_reading\", axis=1)\nenergy_labels = strat_train_set[\"meter_reading\"].copy()","metadata":{"id":"-JQ9nd6McCqQ"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Downcasting Data","metadata":{"id":"UJdissvU2o3H"}},{"cell_type":"markdown","source":"Since we have large size of datasets, we will try to downcast the data to save memory.","metadata":{"id":"x5TWskoRtd4l"}},{"cell_type":"code","source":"print(\"Memory usage before downcasting:\")\nprint(energy_train.memory_usage(deep=True))\n\ndef downcast_dtypes(df):\n    int_cols = df.select_dtypes(include=['int64']).columns\n    df[int_cols] = df[int_cols].apply(pd.to_numeric, downcast='integer')\n\n    float_cols = df.select_dtypes(include=['float64']).columns\n    df[float_cols] = df[float_cols].apply(pd.to_numeric, downcast='float')\n\n    return df\n\nenergy_downcasted = downcast_dtypes(energy_train)\n\n# Check the memory usage after downcasting\nprint(\"\\nMemory usage after downcasting:\")\nprint(energy_train.memory_usage(deep=True))\n\n# Verify the new data types\nprint(\"\\nNew data types:\")\nprint(energy_train.dtypes)\n","metadata":{"id":"EJ3WoEkLtd4l","outputId":"cc4d1719-e4f6-44ca-f2f1-a088f1b4ea52"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"energy_train.info()","metadata":{"id":"qvoT4aZRtd4m","outputId":"0c821530-a2d1-4c07-862d-11c49646fab7"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be seen we saved 1.5 GB (from 2.5+ GB to 1.0+GB) from downcasting the data.","metadata":{"id":"cUDxO3ayz9vz"}},{"cell_type":"markdown","source":"## Handling Invalid Values","metadata":{"id":"UaLLshVo2xLx"}},{"cell_type":"markdown","source":"We observed that *precip_depth_1_hr* contains some invalid (negative) values. We need to decide whether to remove the entire attribute, remove the rows with invalid values, or impute them. First, we will check what percentage of the values are invalid (negative) compared to the non-null values.\n","metadata":{"id":"2DFKtUaFtd4m"}},{"cell_type":"code","source":"non_null_precip_count = strat_train_set['precip_depth_1_hr'].notnull().sum()\nnegative_precip_count = strat_train_set[strat_train_set['precip_depth_1_hr'] < 0].shape[0]\n\nnegative_precip_percentage = (negative_precip_count / non_null_precip_count) * 100\n\nprint(f\"Percentage of negative values in 'precip_depth_1_hr': {negative_precip_percentage:.2f}%\")\n","metadata":{"id":"lbacKUzNtd4m","outputId":"b9829491-9ae0-455e-cd59-3d244f6d9ec1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since the percentage is quite low, we will try to impute the invalid values.","metadata":{"id":"MlSIAW253amo"}},{"cell_type":"markdown","source":"## Handling *building_id* and *site_id*","metadata":{"id":"42iOzJOo3q_a"}},{"cell_type":"markdown","source":"Although *building_id* and *site_id* are numerical data, passing them directly to a model may not be meaningful. Therefore, we will apply clustering to group similar buildings and sites using KMeans. To determine the optimal number of clusters, we will use the elbow method.","metadata":{"id":"sNPjZa_s4ANw"}},{"cell_type":"markdown","source":"## Making The Pipeline","metadata":{"id":"yNTsCU6ktd4n"}},{"cell_type":"markdown","source":"Now, it's time to build the pipeline. We have discussed how to handle most of the data, except for the categorical features. Additionally, we will create new time-based categories (as discussed earlier) and drop the timestamp column, as it is no longer needed.\n\nThe features we will treat as categorical are: *primary_use, meter, weekend, quarter, hour, year_built, and floor_count*. For most of these features (*excluding year_built* and *floor_count*), we will use One-Hot Encoding since there is no ordinal relationship between the categories. Meanwhile, year_built and floor_count will be encoded using an Ordinal Encoder, as it makes sense to treat them as ordered variables (i.e., newer buildings or buildings with more floors can be considered 'larger' in some sense).\n\nFor the numerical features (not listed above), the standard numerical pipeline will be applied: imputing missing values using the median and then standardizing the data","metadata":{"id":"ORAYwPEE6IrK"}},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.preprocessing import OneHotEncoder, StandardScaler, MinMaxScaler, OrdinalEncoder, FunctionTransformer\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics.pairwise import rbf_kernel\nimport numpy as np\nimport pandas as pd\n\nclass TimestampTransformer(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        pass\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        X['timestamp'] = pd.to_datetime(X['timestamp'])\n        X['weekend'] = (X['timestamp'].dt.dayofweek >= 5).astype(int)\n        X['quarter'] = X['timestamp'].dt.quarter\n        X['hour'] = X['timestamp'].dt.hour\n        return X.drop(columns=['timestamp'])\n\nclass ClusterSimilarity(BaseEstimator, TransformerMixin):\n    def __init__(self, n_clusters=10, gamma=1.0, random_state=None):\n        self.n_clusters = n_clusters\n        self.gamma = gamma\n        self.random_state = random_state\n\n    def fit(self, X, y=None, sample_weight=None):\n        self.kmeans_ = KMeans(self.n_clusters, random_state=self.random_state)\n        self.kmeans_.fit(X, sample_weight=sample_weight)\n        return self\n\n    def transform(self, X):\n        return rbf_kernel(X, self.kmeans_.cluster_centers_, gamma=self.gamma)\n\n    def get_feature_names_out(self, names=None):\n        return [f\"Cluster {i} similarity\" for i in range(self.n_clusters)]\n\ndef handle_negative_precip(df):\n    df = df.copy()\n    df[df['precip_depth_1_hr'] < 0] = np.nan\n    return df\n\ndef bucketize_wind_direction(df):\n    bins = [0, 45, 90, 135, 180, 225, 270, 315, 360]\n    labels = ['N', 'NE', 'E', 'SE', 'S', 'SW', 'W', 'NW']\n    return pd.cut(df['wind_direction'], bins=bins, labels=labels, right=False)\n\ndef column_ratio(df):\n    return df.iloc[:, 0] / df.iloc[:, 1]\n\ndef column_product(df):\n    return df.iloc[:, 0] * df.iloc[:, 1]\n\ndef ratio_name(function_transformer, feature_names_in):\n    return [\"ratio\"]\n\ndef multiplier_name(function_transformer, feature_names_in):\n    return [\"multiplier\"]\n\ndef make_ratio_pipeline():\n    ratio_pipeline = (\n        make_pipeline(\n            SimpleImputer(strategy=\"median\"),\n            FunctionTransformer(column_ratio, feature_names_out=ratio_name),\n            StandardScaler(),\n            verbose=True\n        ).set_output(\n            transform=\"pandas\"\n        )\n    )\n    return ratio_pipeline\n\ndef make_multiplier_pipeline():\n    product_pipeline = (\n        make_pipeline(\n            SimpleImputer(strategy=\"median\"),\n            FunctionTransformer(column_product, feature_names_out=multiplier_name),\n            StandardScaler(),\n            verbose=True\n        ).set_output(\n            transform=\"pandas\"\n        )\n    )\n    return product_pipeline\n\n\ndef log_transform_offset(X):\n    return np.log(X + 1e-6)\n\nwind_direction_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"most_frequent\"),\n    FunctionTransformer(bucketize_wind_direction, feature_names_out=\"one-to-one\"),\n    OneHotEncoder(sparse_output=False, handle_unknown=\"ignore\")\n).set_output(transform=\"pandas\")\n\n\nprecip_pipeline = make_pipeline(\n    FunctionTransformer(handle_negative_precip, feature_names_out=\"one-to-one\"),\n    SimpleImputer(strategy=\"median\"),\n    StandardScaler()\n).set_output(transform=\"pandas\")\n\nlog_transform_pipeline = (\n    make_pipeline(\n        SimpleImputer(strategy=\"median\"),\n        FunctionTransformer(log_transform_offset, np.exp, feature_names_out=\"one-to-one\"),\n        StandardScaler()\n    ).set_output(\n        transform=\"pandas\"\n    )\n)\n\nordinal_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"most_frequent\"),\n    OrdinalEncoder()\n).set_output(transform=\"pandas\")\n\ncategorical_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"most_frequent\"),\n    OneHotEncoder(sparse_output=False, handle_unknown=\"ignore\")\n).set_output(transform=\"pandas\")\n\ndefault_numerical_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"median\"),\n    StandardScaler()\n).set_output(transform=\"pandas\")\n\npreprocessing_pipeline = make_pipeline(\n    TimestampTransformer(),\n    ColumnTransformer(\n        transformers=[\n            (\"sqft_per_floor\", make_ratio_pipeline(), [\"square_feet\", \"floor_count\"]),\n            (\"interaction_temp_sqft\", make_multiplier_pipeline(), [\"air_temperature\", \"square_feet\"]),\n            (\"negative_precip\", precip_pipeline, [\"precip_depth_1_hr\"]),\n            (\"bucketizing_wind\", wind_direction_pipeline, [\"wind_direction\"]),\n            (\"log\", log_transform_pipeline, [\"square_feet\", \"cloud_coverage\", \"wind_speed\"]),\n            (\"categorical\", categorical_pipeline, [\"primary_use\", \"meter\", \"weekend\", \"quarter\", \"hour\"]),\n            (\"ordinal\", ordinal_pipeline, [\"year_built\", \"floor_count\"]),\n            (\"building_cluster\", ClusterSimilarity(n_clusters=3, gamma=1., random_state=42), [\"building_id\"]),\n            (\"site_cluster\", ClusterSimilarity(n_clusters=4, gamma=1., random_state=42), [\"site_id\"]),\n        ],\n        remainder=default_numerical_pipeline,\n        verbose=True\n    )\n)\n\nprocessed_data = preprocessing_pipeline.fit_transform(energy_train)","metadata":{"id":"MMZMHzuotd4n","outputId":"34317903-f7fa-4d48-f0fd-f2a5c8a2afcd"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\njoblib.dump(preprocessing_pipeline, \"drive/MyDrive/Colab Notebooks/preprocessing_pipeline.pkl\")\njoblib.dump(energy_train, \"drive/MyDrive/Colab Notebooks/energy_train.pkl\")\njoblib.dump(energy_labels, \"drive/MyDrive/Colab Notebooks/energy_labels.pkl\")","metadata":{"id":"EX5kZrHwiGlP"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is the overview of the pipeline that we created and sampled of processed data.","metadata":{"id":"aW1enkIapJR7"}},{"cell_type":"code","source":"preprocessing_pipeline","metadata":{"id":"tFJsPrDnE73s","outputId":"8cff4274-b38e-424d-c011-f6e0c6cf507f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_data","metadata":{"id":"KZ-M4qorE1JG","outputId":"ffa254b8-d46c-4ee5-fd31-9a05ed8da018"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save/Load Checkpoint","metadata":{"id":"iU-vHopCfhU4"}},{"cell_type":"code","source":"import joblib\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nenergy_train = joblib.load(\"drive/MyDrive/Colab Notebooks/energy_train.pkl\")\nenergy_labels = joblib.load(\"drive/MyDrive/Colab Notebooks/energy_labels.pkl\")\npreprocessing_pipeline = joblib.load(\"drive/MyDrive/Colab Notebooks/preprocessing_pipeline.pkl\")","metadata":{"id":"eN4VW7Cwfpom"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Select and Train a Model","metadata":{"id":"Hm4coZ3_pDCQ"}},{"cell_type":"markdown","source":"## Linear Regression","metadata":{"id":"C5U4QPsfqV94"}},{"cell_type":"markdown","source":"First, we will try to use linear regression with a pipeline that includes preprocessing for the input features and a transformation of the target variable. Since the target variable is heavily skewed, we apply a log transformation followed by standardization. After fitting the model, we use the inverse transformation to bring predictions back to their original scale.\n\n","metadata":{"id":"9M55AORpqZop"}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.compose import TransformedTargetRegressor\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler, FunctionTransformer\nimport numpy as np\n\nlog_labels_transformer = make_pipeline(\n    FunctionTransformer(log_transform_offset, np.exp),\n    StandardScaler()\n)\n\nlin_reg_pipeline = make_pipeline(\n    preprocessing_pipeline,\n    TransformedTargetRegressor(\n        regressor=LinearRegression(),\n        transformer=log_labels_transformer\n    )\n)\n\nlin_reg_pipeline.fit(energy_train, energy_labels)\nlin_energy_predictions = lin_reg_pipeline.predict(energy_train)\n","metadata":{"id":"yINIt_FkE_cq","outputId":"e19d2b8d-579a-48f8-cbc0-0cb092b754b5"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will see the performance of the trained model on first ten datas:","metadata":{"id":"Q6bAfR_ir_Nh"}},{"cell_type":"code","source":"lin_energy_predictions[:10]","metadata":{"id":"u9TH1e97IpYb","outputId":"77ab8882-9f54-42a2-8327-ee9696ebffa2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"energy_labels.iloc[:10].values","metadata":{"id":"EVY1VuxtIw8X","outputId":"0f740f82-5855-473a-fe52-5501937385de"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Based on the output, for several data points, the model's predictions are quite poor. Next, we will evaluate the model's performance using the root mean square log error (as requested by the competition).","metadata":{"id":"dKNn-jMYsGG5"}},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_log_error\n\nlin_rmsle = mean_squared_log_error(energy_labels, lin_energy_predictions, squared=False)\nlin_rmsle","metadata":{"id":"SsAzo3tSxW55","outputId":"b6466946-6f1f-46ce-b09f-73832c5a389e"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\n\njoblib.dump(lin_reg_pipeline, \"drive/MyDrive/Colab Notebooks/lin_reg_pipeline.pkl\")","metadata":{"id":"iN3k5r5vH77K","outputId":"0dfcda61-1d5e-4977-c5f3-8ef438e97425"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The RMSLE value is quite large, which suggests that the model is underfitting the data. Next, we will try a different method to improve the model's performance.","metadata":{"id":"OPENmbHas9cw"}},{"cell_type":"markdown","source":"## Decision Tree Regressor","metadata":{"id":"YmzvZ5T7sXsy"}},{"cell_type":"markdown","source":"We will use a decision tree regressor, with the pipeline being quite similar to before (except for the regressor itself, of course).","metadata":{"id":"320EJh8ltT6X"}},{"cell_type":"code","source":"from sklearn.tree import DecisionTreeRegressor\n\nlog_labels_transformer = make_pipeline(\n    FunctionTransformer(log_transform_offset, np.exp),\n    StandardScaler()\n)\n\ntree_reg = make_pipeline(\n    preprocessing_pipeline,\n    TransformedTargetRegressor(\n        regressor=DecisionTreeRegressor(random_state=42),\n        transformer=log_labels_transformer\n    )\n)\n\ntree_reg.fit(energy_train, energy_labels)\ntree_energy_predictions = tree_reg.predict(energy_train)","metadata":{"id":"jZB_fIeDNtMP","outputId":"1d44d69f-2014-45f3-84cf-0ae81fb8f649"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"joblib.dump(tree_reg, \"drive/MyDrive/Colab Notebooks/tree_reg.pkl\")","metadata":{"id":"TWugHFtRScWG"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will evaluate the RMSLE of the model:","metadata":{"id":"BpNuOksktvBv"}},{"cell_type":"code","source":"tree_rmsle = mean_squared_log_error(energy_labels, tree_energy_predictions, squared=False)\ntree_rmsle","metadata":{"id":"xYKhDTV4N6x3","outputId":"6f927ac7-e795-434d-e2ab-71cfdaa65b67"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tree_energy_predictions[:10]","metadata":{"id":"bykWo8g_es0s","outputId":"36dd156e-7f65-4efb-d94c-1c39d2c9516d"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"energy_labels.iloc[:10].values","metadata":{"id":"8ELVDzBseu-x","outputId":"206b0246-4ce6-4df2-e12b-93740d62785f"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the model performs much better than the previous regressor. Now, we will apply cross-validation to ensure that these results are not due to overfitting.","metadata":{"id":"AChiZ97CffCn"}},{"cell_type":"code","source":"from sklearn.model_selection import cross_val_score\nfrom sklearn.metrics import make_scorer\n\ndef custom_rmsle(y_true, y_pred):\n    return np.sqrt(mean_squared_log_error(y_true, y_pred))\n\nrmsle_scorer = make_scorer(custom_rmsle, greater_is_better=False)\n\ntree_rmsles = -cross_val_score(tree_reg, energy_train, energy_labels,\n                               scoring=rmsle_scorer, cv=10)","metadata":{"id":"8C_Ly3N_gaOU","outputId":"838175ba-51ac-43ef-f7a6-38cc1d31f8be"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"joblib.dump(tree_rmsles, \"drive/MyDrive/Colab Notebooks/tree_rmsles.pkl\")","metadata":{"id":"IVGrEIcOZq2b"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(tree_rmsles).describe()","metadata":{"id":"aRooK78q9GFd","outputId":"78193a1e-3ec8-462e-e4fc-31b1e93ef1db"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the data above, the mean RMSLE from cross-validation is not significantly different from the training set, and the standard deviation is quite low, indicating consistent performance across the folds","metadata":{"id":"AzNV1FcvupFH"}},{"cell_type":"markdown","source":"## Random Forest Regressor","metadata":{"id":"jCbRgOBAvMI-"}},{"cell_type":"markdown","source":"Sorry, I cant do this due to computational limit.","metadata":{"id":"0Uqgw6CavP2O"}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\n\nforest_reg = make_pipeline(\n    preprocessing_pipeline,\n    TransformedTargetRegressor(\n        regressor=RandomForestRegressor(random_state=42),\n        transformer=log_labels_transformer\n    )\n)\n\nforest_reg.fit(energy_train, energy_labels)\nforest_energy_predictions = forest_reg.predict(energy_train)","metadata":{"id":"U1YZcOaEWCba"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"joblib.dump(tree_reg, \"drive/MyDrive/Colab Notebooks/forest_reg.pkl\")","metadata":{"id":"KLJ80NzGWbRl"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save/Load Checkpoint\n","metadata":{"id":"pGUd49Bag7g6"}},{"cell_type":"code","source":"from google.colab import drive\ndrive.mount('/content/drive')","metadata":{"id":"El9tlIxqgThV","outputId":"0ad102eb-d058-424a-c383-f9db990a8091"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.compose import TransformedTargetRegressor\nfrom sklearn.preprocessing import OneHotEncoder, StandardScaler, MinMaxScaler, OrdinalEncoder, FunctionTransformer\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics.pairwise import rbf_kernel\nfrom sklearn.metrics import mean_squared_log_error\nimport numpy as np\nimport pandas as pd\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom scipy.stats import uniform, randint\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.metrics import make_scorer\n\ndef custom_rmsle(y_true, y_pred):\n    return np.sqrt(mean_squared_log_error(y_true, y_pred))\n\nclass TimestampTransformer(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        pass\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        X['timestamp'] = pd.to_datetime(X['timestamp'])\n        X['weekend'] = (X['timestamp'].dt.dayofweek >= 5).astype(int)\n        X['quarter'] = X['timestamp'].dt.quarter\n        X['hour'] = X['timestamp'].dt.hour\n        return X.drop(columns=['timestamp'])\n\nclass ClusterSimilarity(BaseEstimator, TransformerMixin):\n    def __init__(self, n_clusters=10, gamma=1.0, random_state=None):\n        self.n_clusters = n_clusters\n        self.gamma = gamma\n        self.random_state = random_state\n\n    def fit(self, X, y=None, sample_weight=None):\n        self.kmeans_ = KMeans(self.n_clusters, random_state=self.random_state)\n        self.kmeans_.fit(X, sample_weight=sample_weight)\n        return self\n\n    def transform(self, X):\n        return rbf_kernel(X, self.kmeans_.cluster_centers_, gamma=self.gamma)\n\n    def get_feature_names_out(self, names=None):\n        return [f\"Cluster {i} similarity\" for i in range(self.n_clusters)]\n\ndef handle_negative_precip(df):\n    df = df.copy()\n    df[df['precip_depth_1_hr'] < 0] = np.nan\n    return df\n\ndef bucketize_wind_direction(df):\n    bins = [0, 45, 90, 135, 180, 225, 270, 315, 360]\n    labels = ['N', 'NE', 'E', 'SE', 'S', 'SW', 'W', 'NW']\n    return pd.cut(df['wind_direction'], bins=bins, labels=labels, right=False)\n\ndef column_ratio(df):\n    return df.iloc[:, 0] / df.iloc[:, 1]\n\ndef column_product(df):\n    return df.iloc[:, 0] * df.iloc[:, 1]\n\ndef ratio_name(function_transformer, feature_names_in):\n    return [\"ratio\"]\n\ndef multiplier_name(function_transformer, feature_names_in):\n    return [\"multiplier\"]\n\ndef make_ratio_pipeline():\n    ratio_pipeline = (\n        make_pipeline(\n            SimpleImputer(strategy=\"median\"),\n            FunctionTransformer(column_ratio, feature_names_out=ratio_name),\n            StandardScaler(),\n            verbose=True\n        ).set_output(\n            transform=\"pandas\"\n        )\n    )\n    return ratio_pipeline\n\ndef make_multiplier_pipeline():\n    product_pipeline = (\n        make_pipeline(\n            SimpleImputer(strategy=\"median\"),\n            FunctionTransformer(column_product, feature_names_out=multiplier_name),\n            StandardScaler(),\n            verbose=True\n        ).set_output(\n            transform=\"pandas\"\n        )\n    )\n    return product_pipeline\n\n\ndef log_transform_offset(X):\n    return np.log(X + 1e-6)\n\ndef downcast_dtypes(df):\n    int_cols = df.select_dtypes(include=['int64']).columns\n    df[int_cols] = df[int_cols].apply(pd.to_numeric, downcast='integer')\n\n    float_cols = df.select_dtypes(include=['float64']).columns\n    df[float_cols] = df[float_cols].apply(pd.to_numeric, downcast='float')\n\n    return df\n\nrmsle_scorer = make_scorer(custom_rmsle, greater_is_better=False)\n\n\nparam_distribs = {\n    'pipeline__columntransformer__building_cluster__n_clusters': randint(low=3, high=20),\n    'pipeline__columntransformer__site_cluster__n_clusters': randint(low=3, high=20),\n    'transformedtargetregressor__regressor__ccp_alpha': uniform(0, 0.02),\n    'transformedtargetregressor__regressor__max_depth': randint(5, 15),\n    'transformedtargetregressor__regressor__min_samples_split': randint(2, 6),\n    'transformedtargetregressor__regressor__min_samples_leaf': randint(1, 6),\n    'transformedtargetregressor__regressor__criterion': ['squared_error', 'friedman_mse'],\n    'transformedtargetregressor__regressor__splitter': ['best'],\n    'transformedtargetregressor__regressor__max_features': ['sqrt', None]\n}\n\nlog_labels_transformer = make_pipeline(\n    FunctionTransformer(log_transform_offset, np.exp),\n    StandardScaler()\n)\n\n\n\n\nwind_direction_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"most_frequent\"),\n    FunctionTransformer(bucketize_wind_direction, feature_names_out=\"one-to-one\"),\n    OneHotEncoder(sparse_output=False, handle_unknown=\"ignore\")\n).set_output(transform=\"pandas\")\n\n\nprecip_pipeline = make_pipeline(\n    FunctionTransformer(handle_negative_precip, feature_names_out=\"one-to-one\"),\n    SimpleImputer(strategy=\"median\"),\n    StandardScaler()\n).set_output(transform=\"pandas\")\n\nlog_transform_pipeline = (\n    make_pipeline(\n        SimpleImputer(strategy=\"median\"),\n        FunctionTransformer(log_transform_offset, np.exp, feature_names_out=\"one-to-one\"),\n        StandardScaler()\n    ).set_output(\n        transform=\"pandas\"\n    )\n)\n\nordinal_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"most_frequent\"),\n    OrdinalEncoder()\n).set_output(transform=\"pandas\")\n\ncategorical_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"most_frequent\"),\n    OneHotEncoder(sparse_output=False, handle_unknown=\"ignore\")\n).set_output(transform=\"pandas\")\n\ndefault_numerical_pipeline = make_pipeline(\n    SimpleImputer(strategy=\"median\"),\n    StandardScaler()\n).set_output(transform=\"pandas\")\n\n\npreprocessing_pipeline = make_pipeline(\n    TimestampTransformer(),\n    ColumnTransformer(\n        transformers=[\n            (\"sqft_per_floor\", make_ratio_pipeline(), [\"square_feet\", \"floor_count\"]),\n            (\"interaction_temp_sqft\", make_multiplier_pipeline(), [\"air_temperature\", \"square_feet\"]),\n            (\"negative_precip\", precip_pipeline, [\"precip_depth_1_hr\"]),\n            (\"bucketizing_wind\", wind_direction_pipeline, [\"wind_direction\"]),\n            (\"log\", log_transform_pipeline, [\"square_feet\", \"cloud_coverage\", \"wind_speed\"]),\n            (\"categorical\", categorical_pipeline, [\"primary_use\", \"meter\", \"weekend\", \"quarter\", \"hour\"]),\n            (\"ordinal\", ordinal_pipeline, [\"year_built\", \"floor_count\"]),\n            (\"building_cluster\", ClusterSimilarity(n_clusters=3, gamma=1., random_state=42), [\"building_id\"]),\n            (\"site_cluster\", ClusterSimilarity(n_clusters=4, gamma=1., random_state=42), [\"site_id\"]),\n        ],\n        remainder=default_numerical_pipeline,\n        verbose=True\n    )\n)\n\nfull_pipeline = make_pipeline(\n    preprocessing_pipeline,\n    TransformedTargetRegressor(\n        regressor=DecisionTreeRegressor(random_state=42),\n        transformer=log_labels_transformer\n    )\n)\n\nrnd_search = RandomizedSearchCV(\n    full_pipeline, param_distributions=param_distribs, n_iter=3, cv=10,\n    scoring=rmsle_scorer, random_state=42\n)\n\n","metadata":{"id":"aVyLcUCdo3O4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nenergy_train = joblib.load(\"drive/MyDrive/Colab Notebooks/energy_train.pkl\")\nenergy_labels = joblib.load(\"drive/MyDrive/Colab Notebooks/energy_labels.pkl\")\ntree_reg = joblib.load(\"drive/MyDrive/Colab Notebooks/tree_reg.pkl\")\npreprocessing_pipeline = joblib.load(\"drive/MyDrive/Colab Notebooks/preprocessing_pipeline.pkl\")\n","metadata":{"id":"361u9IUShGvh"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fine Tuning Model","metadata":{"id":"dAUCUF-IveJB"}},{"cell_type":"markdown","source":"Before doing fine tuning, let's have a look to our current decision tree regressor's parameters:","metadata":{"id":"EAZUv5HOvj93"}},{"cell_type":"code","source":"tree_reg.get_params()","metadata":{"id":"D3rpOTgExahw","outputId":"623f1291-879f-420c-c298-03f06a992513"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will perform a randomized search since our dataset is large and we want to tune multiple parameters efficiently.","metadata":{"id":"o1uvOjfi1dd0"}},{"cell_type":"code","source":"import numpy as np\nfrom scipy.stats import uniform, randint\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.compose import TransformedTargetRegressor\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.metrics import make_scorer\nfrom sklearn.metrics import mean_squared_log_error\n\ndef custom_rmsle(y_true, y_pred):\n    return np.sqrt(mean_squared_log_error(y_true, y_pred))\n\nlog_labels_transformer = make_pipeline(\n    FunctionTransformer(log_transform_offset, np.exp),\n    StandardScaler()\n)\n\nrmsle_scorer = make_scorer(custom_rmsle, greater_is_better=False)\n\nparam_distribs = {\n    'pipeline__columntransformer__building_cluster__n_clusters': randint(low=3, high=20),\n    'pipeline__columntransformer__site_cluster__n_clusters': randint(low=3, high=20),\n    'transformedtargetregressor__regressor__ccp_alpha': uniform(0, 0.02),\n    'transformedtargetregressor__regressor__max_depth': randint(5, 15),\n    'transformedtargetregressor__regressor__min_samples_split': randint(2, 6),\n    'transformedtargetregressor__regressor__min_samples_leaf': randint(1, 6),\n    'transformedtargetregressor__regressor__criterion': ['squared_error', 'friedman_mse'],\n    'transformedtargetregressor__regressor__splitter': ['best'],\n    'transformedtargetregressor__regressor__max_features': ['sqrt', None]\n}\n\nfull_pipeline = make_pipeline(\n    preprocessing_pipeline,\n    TransformedTargetRegressor(\n        regressor=DecisionTreeRegressor(random_state=42),\n        transformer=log_labels_transformer\n    )\n)\n\nrnd_search = RandomizedSearchCV(\n    full_pipeline, param_distributions=param_distribs, n_iter=3, cv=10,\n    scoring=rmsle_scorer, random_state=42\n)\n\nrnd_search.fit(energy_train, energy_labels)","metadata":{"id":"xpCblfzLyDOd","outputId":"03b31869-46e8-428b-dad7-8c9290baa47e"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"joblib.dump(rnd_search, 'drive/MyDrive/Colab Notebooks/rnd_search_model_10.pkl')","metadata":{"id":"hGLUjlwMZpow"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We want to see the RMSLE of a new model:","metadata":{"id":"NF-p2Ko3GqS_"}},{"cell_type":"code","source":"rnd_search_predictions = rnd_search.predict(energy_train)\nfinal_rmsle = mean_squared_log_error(energy_labels, rnd_search_predictions, squared=False)\n\nfinal_rmsle","metadata":{"id":"FA55M_HKiWVM","outputId":"0d08b196-c139-410b-9362-ec5991526b38"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rnd_search_predictions[:10]","metadata":{"id":"2YSjDWYgjEYA","outputId":"269ae692-9e08-4ca5-9066-815fec71fa4d"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"energy_labels.iloc[:10].values","metadata":{"id":"vhydw4wpjDs4","outputId":"e2d23e94-74d8-40fb-d723-af4fbc665d35"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, it performs worse than the previous model (tree_reg). So, we will use the previous model","metadata":{"id":"YCQsQ2WRG2On"}},{"cell_type":"markdown","source":"# Evaluating on Our Own Test Set","metadata":{"id":"FkVo18D5VRkk"}},{"cell_type":"markdown","source":"Now, we will evaluate our best model (initial model of decision tree regressor) on our own test set.","metadata":{"id":"kojv4n62c7RT"}},{"cell_type":"code","source":"strat_test_set = joblib.load(\"drive/MyDrive/Colab Notebooks/strat_test_set.pkl\")","metadata":{"id":"Aho98fFS9G1N"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = strat_test_set.drop(\"meter_reading\", axis=1)\ny_test = strat_test_set[\"meter_reading\"].copy()","metadata":{"id":"0OOYo41mVVP0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_predictions = tree_reg.predict(X_test)\n\nfinal_rmsle = mean_squared_log_error(y_test, final_predictions, squared=False)\nprint(final_rmsle)","metadata":{"id":"Bt31dk5NVprN","outputId":"23fc8021-fdb7-4619-acde-7052905d366a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the RMSLE is nearly the same as when we performed cross-validation on the test set. Now, we want to compare the predictions with the true values from the training set we created.","metadata":{"id":"9Vi9o-gbHtaR"}},{"cell_type":"code","source":"random_indices = np.random.choice(len(final_predictions), size=10, replace=False)\n\nfor idx in random_indices:\n    print(f\"True value: {y_test.iloc[idx]:.2f}, Prediction: {final_predictions[idx]:.2f}\")\n","metadata":{"id":"SNAVjmLRHhES","outputId":"0fca16a9-ce55-4bfc-b024-0dd447fd6720"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluating on Real Test Set","metadata":{"id":"z7VrZWpRJXM9"}},{"cell_type":"markdown","source":"Now, we want to get the test data:","metadata":{"id":"8_aP_KvsU2jb"}},{"cell_type":"code","source":"test = pd.read_csv(\"ashrae-energy-prediction/test.csv\")\nbuilding_meta = pd.read_csv(\"ashrae-energy-prediction/building_metadata.csv\")\nweather_test = pd.read_csv(\"ashrae-energy-prediction/weather_test.csv\")\n\n\nreal_test_df = pd.merge(test, building_meta, on='building_id', how='left')\nreal_test_df = pd.merge(real_test_df, weather_test, on=['site_id', 'timestamp'], how='left')\n","metadata":{"id":"a-_wmVGKMSr2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"real_test_df.head()","metadata":{"id":"C_pR8cqNN-Qd","outputId":"71adeafd-9605-4eed-ef5f-d6b43dd96257"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"real_test_df.info(show_counts=True)","metadata":{"id":"m339B4WqNZjW","outputId":"bc23e364-71e3-46ee-c39b-5b1667d802d6"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since the data is too big, we will try to do downcasting:","metadata":{"id":"wNYX_x4Ja3jo"}},{"cell_type":"code","source":"print(\"Memory usage before downcasting:\")\nprint(real_test_df.memory_usage(deep=True))\n\n\nreal_test_df_downcasted = downcast_dtypes(real_test_df)\n\n# Check the memory usage after downcasting\nprint(\"\\nMemory usage after downcasting:\")\nprint(real_test_df.memory_usage(deep=True))\n\n# Verify the new data types\nprint(\"\\nNew data types:\")\nprint(real_test_df.dtypes)\n\n","metadata":{"id":"DEQDcaBNaN-u","outputId":"a96676a9-7ddf-4d50-ec65-d9d34d6047a3"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will do prediction, in order to make the processing lighter (again), we will do batch prediction and save the prediction in the csv.","metadata":{"id":"zXESDyJscgZP"}},{"cell_type":"code","source":"def batch_predict(model, data, batch_size):\n    predictions = []\n    for i in range(0, len(data), batch_size):\n        batch_data = data.iloc[i:i+batch_size]\n        batch_pred = model.predict(batch_data)\n        predictions.extend(batch_pred)\n    return np.array(predictions)\n\nfinal_test_predictions = batch_predict(tree_reg, real_test_df, batch_size=10**7)","metadata":{"id":"UnXuWXuFUujs"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n    'row_id': test['row_id'],\n    'meter_reading': final_test_predictions\n})\n\nsubmission['meter_reading'] = submission['meter_reading'].round(4)\nsubmission.to_csv('submission.csv', index=False)","metadata":{"id":"4Frezl8UcVZ4"},"execution_count":null,"outputs":[]}]}