{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":35297,"databundleVersionId":3589144,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### **Notebook summary**\n\n**Notebook type:** Study log based on original notebooks by other authors<br>\n**Model type:** Classification <br>\n**Dataset:** Large-scale Energy Anomaly Detection (LEAD) competition dataset<br>\n\n******","metadata":{}},{"cell_type":"markdown","source":"# LEAD Competition Study Log","metadata":{}},{"cell_type":"markdown","source":"**TABLE OF CONTENTS**<br>\n\n* [1. Introduction](#chapter_1) \n* [2. Train Data Preprocessing and Feature Engineering](#chapter_2)\n* [3. Test Data Preprocessing and Feature Engineering](#chapter_3)   \n* [4. Predictive Model](#chapter_4)\n* [5. Creating Submission](#chapter_5)\n* [6. Afterword](#chapter_6)","metadata":{}},{"cell_type":"markdown","source":"## 1. Introduction <a class=\"anchor\" id=\"chapter_1\"></a>","metadata":{}},{"cell_type":"markdown","source":"As mentioned in the summary, I can hardly call this notebook my own original work. The basis for this edition is the notebook\n(*link:* __[Kaggle notebook](https://www.kaggle.com/code/patrick0302/2-anomaly-detection-features-csv-revised-ensem)__) published by **Chun Fu** (*link:* __[Chun Fu's profile on Kaggle](https://www.kaggle.com/patrick0302)__). Continuing the collaborative theme, this source notebook mentioned was actually a complementary notebook based on another earlier work \n(*link:* __[Kaggle notebook](https://www.kaggle.com/code/abhishek123maurya/2-anomaly-detection-features-csv)__) by **Abishek Maurya** (*link:* __[Abishek Maurya's profile on Kaggle](https://www.kaggle.com/abhishek123maurya)__).  Also, more information about the original dataset etc. can be found on the official LEAD competition homepage (*link:* __[LEAD competition on Kaggle](https://www.kaggle.com/competitions/energy-anomaly-detection)__). \n\nWhile studying the two original notebooks, I noticed that they contained a plethora of useful and versatile approaches and methods to be applied also to other similar tasks beyond the actual LEAD competition. However, I found some parts of the original code hard to grasp, as I am not a seasoned coder myself. Thus my original intention was to study the two source notebooks in private, but in the end I thought it would maybe be useful to publish the results as a kind of private study log made public. \n\nAs a part of this study process, I went through the existing code basically cell by cell, trying to follow and understand better what was going on at a certain point. In this, I applied the help of **ChatGPT** in researching the code in a more detailed manner. I have added some of these explanations as comments in this notebook, as I thought they might come of use also to other people. Whenever ChatGPT explanations appear in this notebook, there will be a mention or a separate comment about it included.\n\nIf there are any changes or adaptations to the original work I have made, this does not mean that my version would somehow be better or more effective. In all this I am like an apprentice painter, adding a splash of complementary coating here and there to a house designed, built and decorated by other people. Moreover, as mentioned, I tried to follow the original works to the best of my ability, and if any variations were made, they were based on my personal knowledge, which undoubtedly is limited. In other words, I only used code in this notebook I could in the end understand myself.\n\nThat was somewhat lengthier an introduction than intended, so while we still have energy, let's proceed to the actual subject of this notebook. The mission is the same as in the original competition: to build a model to identify energy usage anomalies in hourly smart electricity meter readings.\n\n\n**August 19th, 2024<br>\nJari Peltola**\n\n******\n","metadata":{}},{"cell_type":"code","source":"# import modules\nimport pandas as pd\nimport numpy as np\nimport datetime\n\nimport matplotlib.pyplot as plt\nimport seaborn as sb\n\nfrom tqdm import tqdm\nfrom sklearn import metrics\nfrom sklearn.preprocessing import StandardScaler\n\nfrom xgboost import XGBClassifier\nimport lightgbm as lgb\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# show all columns\npd.set_option('display.max_columns', None)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:31:40.968204Z","iopub.execute_input":"2024-08-19T09:31:40.970437Z","iopub.status.idle":"2024-08-19T09:31:40.986923Z","shell.execute_reply.started":"2024-08-19T09:31:40.970358Z","shell.execute_reply":"2024-08-19T09:31:40.984897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.   Train Data Preprocessing and Feature Engineering <a class=\"anchor\" id=\"chapter_2\"></a>","metadata":{}},{"cell_type":"markdown","source":"We start by uploading the training data to a dataframe **train**.","metadata":{}},{"cell_type":"code","source":"# upload data to dataframe\ntrain = pd.read_csv('/kaggle/input/energy-anomaly-detection/train_features.csv')\n\n# show dataframe head\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:31:41.045356Z","iopub.execute_input":"2024-08-19T09:31:41.046277Z","iopub.status.idle":"2024-08-19T09:32:01.550449Z","shell.execute_reply.started":"2024-08-19T09:31:41.046239Z","shell.execute_reply":"2024-08-19T09:32:01.548573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By printing the dataframe shape, we can see that there are 57 columns and over 1.7 million datapoints (rows) in the dataframe.","metadata":{}},{"cell_type":"code","source":"# show dataframe shape\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:01.552778Z","iopub.execute_input":"2024-08-19T09:32:01.553184Z","iopub.status.idle":"2024-08-19T09:32:01.567462Z","shell.execute_reply.started":"2024-08-19T09:32:01.553151Z","shell.execute_reply":"2024-08-19T09:32:01.565922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The code below loops through every column in the dataframe and prints the number of null values (NaN) per column in the dataframe. The result shows that only the column **meter_reading** has null values.","metadata":{}},{"cell_type":"code","source":"# print the sum of null values in the dataframe per column\nfor col in train.columns:\n    k = train[col].isnull().sum()\n    if k > 0:\n        print(f'{col} : {k}')","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:01.569812Z","iopub.execute_input":"2024-08-19T09:32:01.570299Z","iopub.status.idle":"2024-08-19T09:32:03.198715Z","shell.execute_reply.started":"2024-08-19T09:32:01.570259Z","shell.execute_reply":"2024-08-19T09:32:03.197357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As for anomalies, we can see that only some 37 thousand datapoints in the whole dataframe are labeled as such.","metadata":{}},{"cell_type":"code","source":"# show the number of anomalies\ntrain.anomaly.value_counts()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.202107Z","iopub.execute_input":"2024-08-19T09:32:03.202653Z","iopub.status.idle":"2024-08-19T09:32:03.231939Z","shell.execute_reply.started":"2024-08-19T09:32:03.202610Z","shell.execute_reply":"2024-08-19T09:32:03.230207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we take a look at different column datatypes, we can see that the dataframe contains integer (int) values, decimal (float64) values and objects (in most cases text string values).","metadata":{}},{"cell_type":"code","source":"# show all datatypes\ntrain.dtypes","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.233491Z","iopub.execute_input":"2024-08-19T09:32:03.234015Z","iopub.status.idle":"2024-08-19T09:32:03.254437Z","shell.execute_reply.started":"2024-08-19T09:32:03.233971Z","shell.execute_reply":"2024-08-19T09:32:03.252561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The code below creates three lists (integers, floats, objects) of all columns, so they can be further analyzed more closely. One caveat in this method is that the same code must by run again were any changes made to the columns. For example, if the column **anomaly** were removed from the dataframe in the future, it would still show up in the integer column list (ints), as this list was created before removing the aforementioned column from the dataframe. ","metadata":{}},{"cell_type":"code","source":"# create three empty lists\nints = []\nobjects = []\nfloats = []\n\n# divide the columns into three list categories\nfor col in train.columns:\n    if train[col].dtype == int:\n        ints.append(col)\n    elif train[col].dtype == object:\n        objects.append(col)\n    else:\n        floats.append(col)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.256788Z","iopub.execute_input":"2024-08-19T09:32:03.257339Z","iopub.status.idle":"2024-08-19T09:32:03.270002Z","shell.execute_reply.started":"2024-08-19T09:32:03.257298Z","shell.execute_reply":"2024-08-19T09:32:03.268088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show integer columns\nints","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.271999Z","iopub.execute_input":"2024-08-19T09:32:03.272983Z","iopub.status.idle":"2024-08-19T09:32:03.290447Z","shell.execute_reply.started":"2024-08-19T09:32:03.272919Z","shell.execute_reply":"2024-08-19T09:32:03.288782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the original dataframe, the column **timestamp** contains all the necessary time and date information as object. Therefore the columns **hour**, **weekday**, **month** and **year** are unnecessary for our purposes. As mentioned in the source code notebooks, the **is_holiday** actually an expansion of the **timestamp** column, so it can be removed as well.\n\nThe removal is done by referring to the integer column list order. As this list is not zero indexed, we remove all columns starting from the ninth column (**hour**) in the list.","metadata":{}},{"cell_type":"code","source":"# drop columns\ntrain = train.drop(ints[9:], axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.292413Z","iopub.execute_input":"2024-08-19T09:32:03.292925Z","iopub.status.idle":"2024-08-19T09:32:03.700192Z","shell.execute_reply.started":"2024-08-19T09:32:03.292881Z","shell.execute_reply":"2024-08-19T09:32:03.698910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As mentioned, we have to create the three lists again to see any changes in them. In this case only the integer columns in the list have been changed, but for clarity all three column lists are recreated. ","metadata":{}},{"cell_type":"code","source":"ints = []\nobjects = []\nfloats = []\n\nfor col in train.columns:\n    if train[col].dtype == int:\n        ints.append(col)\n    elif train[col].dtype == object:\n        objects.append(col)\n    else:\n        floats.append(col)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.702025Z","iopub.execute_input":"2024-08-19T09:32:03.702406Z","iopub.status.idle":"2024-08-19T09:32:03.714101Z","shell.execute_reply.started":"2024-08-19T09:32:03.702376Z","shell.execute_reply":"2024-08-19T09:32:03.712786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can see that the aforementioned columns are not included in the list.","metadata":{}},{"cell_type":"code","source":"ints","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.720656Z","iopub.execute_input":"2024-08-19T09:32:03.721949Z","iopub.status.idle":"2024-08-19T09:32:03.735939Z","shell.execute_reply.started":"2024-08-19T09:32:03.721695Z","shell.execute_reply":"2024-08-19T09:32:03.734478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the original code, the number of unique values are next taken into consideration. The purpose of this to see if any of the columns contain the same exact value on all rows. This would make the column unnecessary, since it would not be able to give any additional input for the machine learning model to be created.","metadata":{}},{"cell_type":"code","source":"# show the number of unique values per column\ntrain[ints].nunique()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.737365Z","iopub.execute_input":"2024-08-19T09:32:03.737779Z","iopub.status.idle":"2024-08-19T09:32:03.896404Z","shell.execute_reply.started":"2024-08-19T09:32:03.737748Z","shell.execute_reply":"2024-08-19T09:32:03.894799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the source code, it is noted that the value 255 in both **year_built** and **cloud_coverage** columns actually refers to null values. In the source code, the **cloud_coverage** datapoints containing value 255 were changed to 10. In this notebook, the values are left unchanged, as the two aforementioned columns might in my opinion be able to learn from each other whenever their shared value of null cases is the same (255).\n\nWhen the whole dataframe is concerned, one can see that the column **gte_meter** contains the same value on each row. This means the column can be removed, as the machine learning model cannot learn anything new from these fixed values.","metadata":{}},{"cell_type":"code","source":"# show the number of unique values in dataframe\nfor col in train.columns:\n    if train[col].nunique()==1:\n        print(col, train[col].dtype)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:03.898388Z","iopub.execute_input":"2024-08-19T09:32:03.898924Z","iopub.status.idle":"2024-08-19T09:32:06.076252Z","shell.execute_reply.started":"2024-08-19T09:32:03.898878Z","shell.execute_reply":"2024-08-19T09:32:06.073611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# drop column\ntrain = train.drop(['gte_meter'], axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:06.082968Z","iopub.execute_input":"2024-08-19T09:32:06.084213Z","iopub.status.idle":"2024-08-19T09:32:06.632168Z","shell.execute_reply.started":"2024-08-19T09:32:06.084166Z","shell.execute_reply":"2024-08-19T09:32:06.630579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we take a closer look at the objects column list.","metadata":{}},{"cell_type":"code","source":"objects","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:06.633978Z","iopub.execute_input":"2024-08-19T09:32:06.634364Z","iopub.status.idle":"2024-08-19T09:32:06.652317Z","shell.execute_reply.started":"2024-08-19T09:32:06.634334Z","shell.execute_reply":"2024-08-19T09:32:06.650641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As the two original notebooks describe, the columns with name starting with \"building_\" except **building_id** contain no original data, so they are unnecessary for our purposes. Before removing them, we separate the date value from the **timestamp** column to its own **date** column. After that, the standard deviation of the **meter_reading** column values are created based on **building_id** and **date** columns. \n\nThe code below included both of these operations as well as ChatGPT commentary are included in the code cell. As a general rule of thumb, my own comments will appear above a specific piece of code, and the ChatGPT-generated comments are placed under that same code.","metadata":{}},{"cell_type":"code","source":"# ChatGPT was applied in analyzing the code \n# separate the dates from timestamp column\ntrain['date'] = pd.to_datetime(train['timestamp']).dt.date\n# Purpose: Convert the timestamp column in the train DataFrame to a date format, then extract just the date part (ignoring the time)#\n# Result: A new column date is created in train, containing only the date (year-month-day) for each timestamp#\n\n\n# create standard deviation values\ntrain['meterReadings_daily_std'] = train.groupby(['building_id','date'])['meter_reading'].transform('std')\n# Purpose: Calculate the standard deviation of the meter_reading values for each combination of building_id and date#\n# How It Works:#\n# Grouping: The data is grouped by building_id and date, meaning that for each building on each day, the calculations will be done separately.#\n# Standard Deviation: The standard deviation (std) of the meter_reading values within each group (i.e., for each building on each specific day) \n# is computed.#\n# Transform: The transform function ensures that the calculated standard deviation is returned for every row in the original DataFrame, \n# rather than just a summary for each group.#\n# Result: A new column meterReadings_daily_std is created, where each value represents the standard deviation of the meter_reading values \n# for the corresponding building_id on that specific date.#","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:06.654364Z","iopub.execute_input":"2024-08-19T09:32:06.654865Z","iopub.status.idle":"2024-08-19T09:32:08.272221Z","shell.execute_reply.started":"2024-08-19T09:32:06.654822Z","shell.execute_reply":"2024-08-19T09:32:08.270625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The next line of code should basically be unnecessary, but at least in this notebook the **date** column values are still in string (object) format after running the code cell above. This is not the first occasion the same thing has happened to me, and so far I have just transformed the values again into datetime format with the following line of code.","metadata":{}},{"cell_type":"code","source":"# transform values to datetime format\ntrain['date'] = pd.to_datetime(train['date'])","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:08.274023Z","iopub.execute_input":"2024-08-19T09:32:08.274644Z","iopub.status.idle":"2024-08-19T09:32:08.826098Z","shell.execute_reply.started":"2024-08-19T09:32:08.274455Z","shell.execute_reply":"2024-08-19T09:32:08.824469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next a dataframe copy **train_two** is created with only the **timestamp** column in it. After this, all object / string columns (among them the **timestamp** column) will be removed from the **train** dataframe, as we want it to contain only numerical data as for now.","metadata":{}},{"cell_type":"code","source":"# create a dataframe copy with only the timestamp column\ntrain_two = train.loc[:, 'timestamp']\n\n# drop all object columns from the train dataframe\ntrain = train.drop(objects, axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:08.827584Z","iopub.execute_input":"2024-08-19T09:32:08.827938Z","iopub.status.idle":"2024-08-19T09:32:09.232233Z","shell.execute_reply.started":"2024-08-19T09:32:08.827906Z","shell.execute_reply":"2024-08-19T09:32:09.230854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now only numerical data (integers, decimals) and the **date** column remain in the **train** dataframe. This enables the imputation (replacing missing values by creating simulated values) of all null values in **meter_reading** column. This is done by applying the **impute_nulls** function below to the whole **train** dataframe. This is possible as we know from our earlier work that only the **meter_reading** column in the **train** dataframe contained null values.\n\nWithin the function, comments created by ChatGPT are included.","metadata":{}},{"cell_type":"code","source":"# ChatGPT was applied in analyzing the function\n\ndef impute_nulls(data):\n    mean_reading = data.groupby('building_id').mean()['meter_reading']\n    \n# Purpose: Calculate the mean (mean) of the meter_reading for each building_id in the data DataFrame.\n# Result: mean_reading is a Pandas Series where the index is building_id and the value is the average meter reading for that \n# building across all timestamps.\n# For example, if building_id 1 has meter readings [10, 20, 30], the mean for building_id 1 would be (10+20+30)/3 = 20\n\n    building_id = mean_reading.index\n    values = mean_reading.values\n    \n    # Purpose: Extract the building_id and corresponding mean values from the mean_reading Series.\n    # Result:building_id: An index containing the unique building_id values.\n    # values: An array of the corresponding mean meter readings.\n    \n    for i, idx in tqdm(enumerate(building_id)):\n        data[data['building_id']==idx] = data[data['building_id']==idx].fillna(values[i]) \n        \n        # Purpose: Iterate over each building_id and fill in the missing (NaN) values in the meter_reading column with the mean \n        # value for that particular building.\n        # How It Works: The loop iterates over each building_id (idx) and its corresponding mean value (values[i]).\n        # For each building_id, it selects all rows in data where the building_id matches idx.\n        # The fillna(values[i]) function is used to replace all NaN values in the selected rows' meter_reading column with the \n        # mean value for that building_id.\n        # Result: After the loop finishes, all NaN values in the meter_reading column for each building have been replaced with \n        # that building's mean meter reading.\n    \n    return data\n    \n    # Purpose: Return the modified DataFrame with imputed values.\n    # Result: The train DataFrame, after being passed through the impute_nulls function, will have no NaN values in the meter_reading column. \n    # Any missing values will have been replaced by the mean meter reading for the corresponding building_id.\n\n\n# apply function to whole dataframe\ntrain = impute_nulls(train)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:09.234294Z","iopub.execute_input":"2024-08-19T09:32:09.234920Z","iopub.status.idle":"2024-08-19T09:32:19.584399Z","shell.execute_reply.started":"2024-08-19T09:32:09.234737Z","shell.execute_reply":"2024-08-19T09:32:19.582743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show the number of null values in meter_reading column\ntrain[\"meter_reading\"].isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:19.586176Z","iopub.execute_input":"2024-08-19T09:32:19.586570Z","iopub.status.idle":"2024-08-19T09:32:19.599235Z","shell.execute_reply.started":"2024-08-19T09:32:19.586529Z","shell.execute_reply":"2024-08-19T09:32:19.597518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We no longer have any null values in the **meter_reading** column. Next we join (method: left join) the dataframes **train** and **train_two**. After this, we will have a dataframe **train** with no null values in **meter_reading** column and also the original **timestamp** column included. As there will still be use for the version without any timestamp data, a dataframe copy **train_numeric** will be created before joining the two dataframes.","metadata":{}},{"cell_type":"code","source":"# create copy\ntrain_numeric = train.copy()\n\n# join the dataframes\ntrain = train.join(train_two)\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:19.600777Z","iopub.execute_input":"2024-08-19T09:32:19.601133Z","iopub.status.idle":"2024-08-19T09:32:21.112161Z","shell.execute_reply.started":"2024-08-19T09:32:19.601104Z","shell.execute_reply":"2024-08-19T09:32:21.110090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next the **train_numeric** dataframe will be used to create a heatmap of correlating features (columns) in the dataframe. According to original two source notebooks, this approach is used to discover highly correlating features, which would distort the machine learning model.\n\nIn this sense, the ideal heatmap would be a straight line of values of 1 ranging from top left corner to bottom right, as these points are crossroads where the column meets itself i.e. correlates with its own value. Anything else may be considered unwanted correlation, set here to threshold 0.75 or higher (the original notebooks used a threshold of 0.95 at the same stage).","metadata":{}},{"cell_type":"code","source":"# set figure size\nplt.figure(figsize=(15,15))\n\n# set parameters\nsb.heatmap(train_numeric.corr() > 0.75, annot=True, cbar=False)\n\n# show plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:21.114204Z","iopub.execute_input":"2024-08-19T09:32:21.114817Z","iopub.status.idle":"2024-08-19T09:32:37.326009Z","shell.execute_reply.started":"2024-08-19T09:32:21.114766Z","shell.execute_reply":"2024-08-19T09:32:37.324344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interpreting such heatmap information is in the end both subjective and case sensitive. All datasets as well as the tasks related to those datasets differ from each other, so there is no one rule of interpretation that would apply to all individual tasks. \n\nNext a list **hcorr** containing highly correlated columns will be compiled manually, based on the heatmap above. After that, the columns in the list will be removed from the **train** dataframe.","metadata":{}},{"cell_type":"code","source":"# create column list of highly correlating columns (hcorr)\n# the list contents may vary depending on subjective interpretation\nhcorr = [\n 'gte_meter_hour',\n 'gte_meter_weekday',\n 'gte_meter_building_id',\n 'gte_meter_primary_use',\n 'gte_meter_site_id',\n 'gte_meter_building_id_hour',\n 'gte_meter_building_id_weekday',\n 'gte_meter_building_id_month',\n 'air_temperature_mean_lag73',\n 'air_temperature_max_lag73',\n 'air_temperature_min_lag73',\n 'air_temperature_mean_lag7',\n 'air_temperature_max_lag7',\n 'air_temperature_min_lag7']    \n\n# drop the hcorr columns from train dataframe\ntrain = train.drop(hcorr, axis=1)\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:37.328221Z","iopub.execute_input":"2024-08-19T09:32:37.328719Z","iopub.status.idle":"2024-08-19T09:32:37.538064Z","shell.execute_reply.started":"2024-08-19T09:32:37.328674Z","shell.execute_reply":"2024-08-19T09:32:37.536282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After removing the highly correlated features (columns) from the dataframe, the next step is to start compiling the anomaly detection model. First it is good to check that the datatypes in the **train** dataframe are still intact.","metadata":{}},{"cell_type":"code","source":"train.dtypes","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:37.540756Z","iopub.execute_input":"2024-08-19T09:32:37.541346Z","iopub.status.idle":"2024-08-19T09:32:37.555367Z","shell.execute_reply.started":"2024-08-19T09:32:37.541300Z","shell.execute_reply":"2024-08-19T09:32:37.553542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All columns except **date** and **timestamp** are in numeric format as expected, so we're good to move on. In the original source code notebooks, **Chun Fu** added lag features to **Abishek Maurya's** earlier work, which eventually resulted as slightly more accurate predictions albeit NaN values are also included in the lag values. Based on this, I inserted some new time lag hour shifts in this notebook. This will make the notebook a bit slower, but the runtime difference is not significant.\n\nBelow is the original code for creating the time lags with explanation created with the help of ChatGPT. Also, at least in my notebook the code line:\n\nmeter_reading_shift = train_feature[['building_id', 'timestamp', 'meter_reading']]\n\ncaused an error message about a \"value trying to be set on a copy of a slice from a dataframe\". The error message stopped appearing when the line was changed into:\n\nmeter_reading_shift = train_feature[['building_id', 'timestamp', 'meter_reading']].copy()\n","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:37.557618Z","iopub.execute_input":"2024-08-19T09:32:37.558260Z","iopub.status.idle":"2024-08-19T09:32:37.604280Z","shell.execute_reply.started":"2024-08-19T09:32:37.558084Z","shell.execute_reply":"2024-08-19T09:32:37.602662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Overall Process: The code creates lagged versions of the meter_reading column for different time shifts (±6 hours etc.)\n# The code merges these lagged values back into the original DataFrame. \n# Each iteration of the loop adds a new column to train_feature with the shifted readings.\n# Output: The final train_feature DataFrame will have additional columns (lag_value_-6, lag_value_6 etc.) representing \n# the meter readings shifted by the respective hours.\n\n\n# create a copy of the dataframe\n# this is not necessary but more like a safety measure to retain the original dataframe as such\ntrain_feature = train.copy()\n\nfor shift_hours in tqdm([-1,1,-6,6,-12,12,-24,24,-48,48,-72,72,-96,96,-7*24,7*24]):\n# Purpose: This loop iterates through a list of specified hour shifts: [-1, 1, -24, 24 etc.]. \n# These represent time shifts of -6 hours, +6 hours etc. \n# Result: The loop will perform its body for each of these shifts.\n\n    meter_reading_shift = train_feature[['building_id', 'timestamp', 'meter_reading']].copy()\n    # Purpose: Create a new DataFrame meter_reading_shift that contains only the columns building_id, timestamp, \n    # and meter_reading from train_feature.\n    # Result: meter_reading_shift is a subset of train_feature with only the relevant columns for creating lag values.\n    \n    meter_reading_shift['timestamp'] = pd.to_datetime(meter_reading_shift['timestamp']) + datetime.timedelta(hours=shift_hours)\n    # Purpose: Convert the timestamp column to datetime format and then shift it by the specified number of hours (shift_hours).\n    # Result: The timestamp in meter_reading_shift is adjusted forward or backward by the number of hours specified in shift_hours.\n    \n    meter_reading_shift['timestamp'] = meter_reading_shift['timestamp'].astype('str')\n    # Purpose: Convert the adjusted timestamp back to string format.\n    # Result: The timestamp column in meter_reading_shift is now a string again, which is necessary for merging later.\n    \n    meter_reading_shift = meter_reading_shift.rename(columns={'meter_reading':'lag_value_'+str(shift_hours)})\n    # Purpose: Rename the meter_reading column to lag_value_<shift_hours> where <shift_hours> is the current shift value \n    # (e.g., lag_value_-6, lag_value_6 etc.).\n    # Result: The meter_reading column is now labeled according to the shift applied.\n    \n    train_feature = train_feature.merge(meter_reading_shift, on=['building_id', 'timestamp'], how='left')\n    # Purpose: Merge the original train_feature DataFrame with the shifted meter_reading_shift DataFrame on the building_id and timestamp columns.\n    # Result: The train_feature DataFrame now includes new columns for each lagged meter reading value, filled with the corresponding shifted readings. \n    # If no corresponding shifted reading exists for a particular row, the new column will have NaN for that row.","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:32:37.606406Z","iopub.execute_input":"2024-08-19T09:32:37.606948Z","iopub.status.idle":"2024-08-19T09:34:17.705275Z","shell.execute_reply.started":"2024-08-19T09:32:37.606910Z","shell.execute_reply":"2024-08-19T09:34:17.704044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train_feature.copy()\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:34:17.706866Z","iopub.execute_input":"2024-08-19T09:34:17.707303Z","iopub.status.idle":"2024-08-19T09:34:18.379983Z","shell.execute_reply.started":"2024-08-19T09:34:17.707263Z","shell.execute_reply":"2024-08-19T09:34:18.378869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in train.columns:\n    k = train[col].isnull().sum()\n    if k > 0:\n        print(f'{col} : {k}')","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:34:18.381973Z","iopub.execute_input":"2024-08-19T09:34:18.382405Z","iopub.status.idle":"2024-08-19T09:34:18.688140Z","shell.execute_reply.started":"2024-08-19T09:34:18.382366Z","shell.execute_reply":"2024-08-19T09:34:18.686909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The most number of NaN values are included in the lag value columns where time lag time is set as one week (168 hours). However, as those roughly 37 thousand datapoints (rows) are a relatively small portion (about two percent) of the 1.7 million overall datapoints, in this case the tradeoff seems acceptable.","metadata":{}},{"cell_type":"code","source":"train.dtypes","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:34:18.695212Z","iopub.execute_input":"2024-08-19T09:34:18.695595Z","iopub.status.idle":"2024-08-19T09:34:18.705799Z","shell.execute_reply.started":"2024-08-19T09:34:18.695566Z","shell.execute_reply":"2024-08-19T09:34:18.704482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:34:18.706909Z","iopub.execute_input":"2024-08-19T09:34:18.707230Z","iopub.status.idle":"2024-08-19T09:34:18.718152Z","shell.execute_reply.started":"2024-08-19T09:34:18.707202Z","shell.execute_reply":"2024-08-19T09:34:18.717006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We now have 46 columns to work with in the **train** dataframe, and the new lag values are in correct decimal format (float64). Next a similar process will be done to the test dataset. The only real format difference between the two datasets are the missing **anomaly** values in the test dataset, as these values are the ones the model should be able to predict after training.\n\nIn the following code cells, the test dataset will be transformed to **test** dataframe by using the same procedures and methods already explained while working with the **train** data earlier. Hence the more condensed code structure with less comments. \n\n******","metadata":{}},{"cell_type":"markdown","source":"## 3.  Test Data Preprocessing and Feature Engineering <a class=\"anchor\" id=\"chapter_3\"></a>","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('../input/energy-anomaly-detection/test_features.csv', index_col=0)\n\n# for clarity, these lists will be named differently than when working with the train dataset\ntest_ints = []\ntest_objects = []\ntest_floats = []\n\nfor col in test.columns:\n    if test[col].dtype == int:\n        test_ints.append(col)\n    elif test[col].dtype == object:\n        test_objects.append(col)\n    else:\n        test_floats.append(col)\n\n# in the train dataframe, dropped integers started from the 9th integer\n# the test dataframe does not contain the anomaly column\n# thus dropping the columns now starts from the 8th integer in the list\ntest = test.drop(test_ints[8:], axis=1)\n\ntest['date'] = pd.to_datetime(test['timestamp']).dt.date\ntest['meterReadings_daily_std'] = test.groupby(['building_id','date'])['meter_reading'].transform('std')\n\ntest['date'] = pd.to_datetime(test['date'])\n\ntest_two = test.loc[:, 'timestamp']\n\n# the hcorr column list is exactly the same as when processing the train dataframe\ntest = test.drop(test_objects + hcorr + ['gte_meter'], axis=1)\n\n# the function impute_nulls is exactly the same as when processing the train dataframe\ntest = impute_nulls(test)\n\ntest = test.join(test_two)\n\ntest_feature = test.copy()\n\nfor shift_hours in tqdm([-1,1,-6,6,-12,12,-24,24,-48,48,-72,72,-96,96,-7*24,7*24]):\n    meter_reading_shift = test_feature[['building_id', 'timestamp', 'meter_reading']].copy()\n    meter_reading_shift['timestamp'] = pd.to_datetime(meter_reading_shift['timestamp']) + datetime.timedelta(hours=shift_hours)\n    meter_reading_shift['timestamp'] = meter_reading_shift['timestamp'].astype('str')\n    meter_reading_shift = meter_reading_shift.rename(columns={'meter_reading':'lag_value_'+str(shift_hours)})\n    test_feature = test_feature.merge(meter_reading_shift, on=['building_id', 'timestamp'], how='left')\n    \ntest = test_feature.copy()\n\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:34:18.719917Z","iopub.execute_input":"2024-08-19T09:34:18.720263Z","iopub.status.idle":"2024-08-19T09:36:22.456467Z","shell.execute_reply.started":"2024-08-19T09:34:18.720235Z","shell.execute_reply":"2024-08-19T09:36:22.455352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the sum of null values in the dataframe per column\nfor col in test.columns:\n    k = test[col].isnull().sum()\n    if k > 0:\n        print(f'{col} : {k}')","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:22.457792Z","iopub.execute_input":"2024-08-19T09:36:22.458132Z","iopub.status.idle":"2024-08-19T09:36:22.779735Z","shell.execute_reply.started":"2024-08-19T09:36:22.458104Z","shell.execute_reply":"2024-08-19T09:36:22.778397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.dtypes","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:22.781296Z","iopub.execute_input":"2024-08-19T09:36:22.781692Z","iopub.status.idle":"2024-08-19T09:36:22.792890Z","shell.execute_reply.started":"2024-08-19T09:36:22.781659Z","shell.execute_reply":"2024-08-19T09:36:22.791622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:22.794411Z","iopub.execute_input":"2024-08-19T09:36:22.794805Z","iopub.status.idle":"2024-08-19T09:36:22.809772Z","shell.execute_reply.started":"2024-08-19T09:36:22.794773Z","shell.execute_reply":"2024-08-19T09:36:22.808517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, the **test** dataframe contains one column (**anomaly**) less than the **train** dataframe. Also, the overall number of NaN values in the time lag columns are on similar level than in the **train** dataframe.\n\nIn the final section of this notebook, the predictive model will finally be created.\n\n******","metadata":{}},{"cell_type":"markdown","source":"## 4.  Predictive Model <a class=\"anchor\" id=\"chapter_4\"></a>","metadata":{}},{"cell_type":"markdown","source":"As the time-related columns **timestamp** and **date** are not needed at this stage, they will be removed from both **train** and **test** dataframes.","metadata":{}},{"cell_type":"code","source":"[df.drop(['date', 'timestamp'], axis=1, inplace=True) for df in [train, test]]","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:22.812028Z","iopub.execute_input":"2024-08-19T09:36:22.812376Z","iopub.status.idle":"2024-08-19T09:36:23.317132Z","shell.execute_reply.started":"2024-08-19T09:36:22.812348Z","shell.execute_reply":"2024-08-19T09:36:23.316112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As mentioned in the beginning, there are relatively few datapoints (rows) in the dataset where the anomaly value is 1, meaning that datapoint was labeled as an anomaly. To address this, in the original code a new dataframe **df_eq** is created. This dataframe will balance the amount of datapoints based on the value in the **anomaly** column (either 0 or 1). This is done in the code cell below, and all comments included are created by ChatGPT.","metadata":{}},{"cell_type":"code","source":"# Summary of What the Code Does\n# The code first separates the train DataFrame into rows with anomaly equal to 0 (neg) and 1 (pos).\n# It then takes two random samples of 37,296 rows each from the neg DataFrame.\n# The df_eq DataFrame is created by concatenating these two samples of normal data along with the entire set of anomalous data twice.\n# Finally, the shape of df_eq is printed, showing how many rows and columns the resulting DataFrame has after this operation.\n\nneg = train[train['anomaly'] == 0]\npos = train[train['anomaly'] == 1]\n\n# Purpose: Split the train DataFrame into two separate DataFrames based on the values in the anomaly column.\n# neg: Contains all rows from train where the anomaly value is 0.\n# pos: Contains all rows from train where the anomaly value is 1.\n# Result:\n# neg will be a DataFrame with all the “normal” (non-anomalous) data points.\n# pos will be a DataFrame with all the “anomalous” data points. \n\nprint(neg.shape, pos.shape)\n\nnegs1 = neg.sample(n = 37296, random_state=10)\nnegs2 = neg.sample(n = 37296, random_state=20)\n\n# Purpose: Randomly sample 37,296 rows from the neg DataFrame twice.\n# negs1: A sample of 37,296 rows from neg, with a fixed random seed (random_state=10).\n# negs2: Another sample of 37,296 rows from neg, but with a different random seed (random_state=20).\n# Random State: The random_state parameter ensures that the sampling is reproducible. \n# If you run the code multiple times with the same random_state, you’ll get the same sample.\n# Result:\n# negs1 and negs2 are two different subsets of 37,296 “normal” rows each, sampled from neg.\n\ndf_eq = pd.concat([negs1, pos, negs2, pos], axis=0)\n# Purpose: Combine the negs1, pos, negs2, and pos DataFrames into a single DataFrame df_eq.\n# axis=0: Indicates that the concatenation is performed row-wise, stacking the DataFrames on top of each other.\n# Order of Concatenation:\n# negs1 (sample of normal data)\n# pos (all anomalous data)\n# negs2 (another sample of normal data)\n# pos (all anomalous data again)\n# Result:\n# df_eq will have twice the number of rows in pos and two samples of 37,296 rows from neg.\n# The order of rows in df_eq will first contain negs1, then pos, followed by negs2, and finally pos.\n\nprint(df_eq.shape)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:23.318621Z","iopub.execute_input":"2024-08-19T09:36:23.319000Z","iopub.status.idle":"2024-08-19T09:36:23.819925Z","shell.execute_reply.started":"2024-08-19T09:36:23.318971Z","shell.execute_reply":"2024-08-19T09:36:23.818744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As the first printout above shows, there are a total of 37296 **pos** datapoints (labeled as anomaly). This figure is used as a \"measuring stick\" for defining the complete size (4 x 37296 datapoints) of the balanced dataframe **df_eq**.\n\nAs the second printout proves, we are left with less than 150 thousand datapoints, which to me is an important point, as it is less than 10 percent of the 1.7 million datapoints in the original **train** dataframe. As one may argue, *in machine learning model training, the quality of data is more important than the quantity of data*. \n\nThe dataframe **df_eq** will next be divided between **target** (the value to be predicted) and **features** (columns used in predicting the target values). This will leave us with one target and 45 different features.","metadata":{}},{"cell_type":"code","source":"features = df_eq.drop(['anomaly'], axis=1)\ntarget = df_eq['anomaly']","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:23.821264Z","iopub.execute_input":"2024-08-19T09:36:23.821637Z","iopub.status.idle":"2024-08-19T09:36:23.846311Z","shell.execute_reply.started":"2024-08-19T09:36:23.821607Z","shell.execute_reply":"2024-08-19T09:36:23.845281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(features.shape, target.shape)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:23.847818Z","iopub.execute_input":"2024-08-19T09:36:23.848194Z","iopub.status.idle":"2024-08-19T09:36:23.854986Z","shell.execute_reply.started":"2024-08-19T09:36:23.848164Z","shell.execute_reply":"2024-08-19T09:36:23.853566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the two source notebooks, the following code was originally added by **Chun Fu**. The code applies the modulo operation on the **building_id** column to split the **features** data into training and validation sets. Here the **building_id** value from the earlier **train** dataframe are used a separator. A more detailed explanation created with the help of ChatGPT is included in the code comments.","metadata":{}},{"cell_type":"code","source":"# Summary\n# Training Set (X_train and Y_train): Contains data where building_id % 5 results in 0, 1, 2, or 3.\n# This means buildings with building_id ending in 0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, etc.\n# Essentially, 80% of the data (4 out of every 5 buildings).\n# Validation Set (X_val and Y_val): Contains data where building_id % 5 results in 4.\n# This means buildings with building_id ending in 4, 9, 14, 19, etc.\n# Essentially, 20% of the data (1 out of every 5 buildings).\n\nX_train = features[train['building_id']%5<4]\n# Purpose: Select rows from features where the building_id modulo 5 is less than 4 (i.e., 0, 1, 2, or 3).\n# Result: X_train will contain the rows of features corresponding to buildings with IDs that, when divided by 5, leave a remainder of 0, 1, 2, or 3.\n\nX_val = features[train['building_id']%5==4]\n# Purpose: Select rows from features where the building_id modulo 5 equals 4.\n# Result: X_val will contain the rows of features corresponding to buildings with IDs that, when divided by 5, leave a remainder of 4.\n\nY_train = target[train['building_id']%5<4]\n# Purpose: Select rows from target where the building_id modulo 5 is less than 4 (i.e., 0, 1, 2, or 3).\n# Result: Y_train will contain the rows of target corresponding to buildings with IDs that, when divided by 5, leave a remainder of 0, 1, 2, or 3.\n\nY_val = target[train['building_id']%5==4]\n# Purpose: Select rows from target where the building_id modulo 5 equals 4.\n# Result: Y_val will contain the rows of target corresponding to buildings with IDs that, when divided by 5, leave a remainder of 4.\n\nprint(X_train.shape, X_val.shape)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:23.856609Z","iopub.execute_input":"2024-08-19T09:36:23.857005Z","iopub.status.idle":"2024-08-19T09:36:24.016961Z","shell.execute_reply.started":"2024-08-19T09:36:23.856974Z","shell.execute_reply":"2024-08-19T09:36:24.015751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the final processed training data **X_train** consists of roughly 120 thousand datapoints (rows), and the validation data **X_val** has some 30 thousand datapoints. In the following stage, this data is standardized by scaling it. Again, a more detailed ChatGPT-powered explanation is included as comments.","metadata":{}},{"cell_type":"code","source":"# Summary\n# Initialization: An instance of StandardScaler is created to standardize the data.\n# Fit and Transform on Training Data: The scaler is fitted to the training data (X_train), calculating the mean and standard deviation for each feature, \n# and then transforming X_train to have standardized features.\n# Transform Validation and Test Data: The same scaling transformation is applied to the validation data (X_val) and the test data (test), ensuring all datasets \n# are scaled in the same way as the training data.\n# This process is crucial because it ensures that all datasets (training, validation, and test) are on the same scale, which is especially important \n# for many machine learning models that are sensitive to the scale of input features.\n\n# Summary of the Entire Process:\n# Initialize the Scaler: Create an instance of StandardScaler.\n# Fit and Transform Training Data: Compute the mean and standard deviation of X_train, then standardize it.\n# Transform Validation Data: Standardize X_val using the same statistics derived from X_train.\n# Transform Test Data: Standardize the test dataset (resulting in test1) using the same statistics derived from X_train.\n# This approach ensures that the training, validation, and test data are all on the same scale, which is crucial for many machine learning models to perform well. \n# The consistency in scaling also prevents data leakage, as the validation and test data are not influencing the scaling parameters.\n\n\nscaler = StandardScaler()\n# Purpose: Create an instance of StandardScaler from the sklearn.preprocessing module.\n# StandardScaler standardizes features by removing the mean and scaling to unit variance.\n# Specifically, it transforms the data so that each feature has a mean of 0 and a standard deviation of 1.\n# This is done because many machine learning models perform better or converge faster when the features are standardized.\n\nX_train = scaler.fit_transform(X_train)\n# Purpose: This line standardizes the X_train data.\n# What It Does: The fit_transform() method does two things:\n# Fit: It computes the mean and standard deviation for each feature in X_train.\n# Transform: It then standardizes the data using the computed mean and standard deviation.\n# Result: X_train is transformed so that each feature in the training data has a mean of 0 and a standard deviation of 1.\n\nX_val = scaler.transform(X_val)\n# Purpose: This line standardizes the X_val (validation) data.\n# What It Does: The transform() method uses the mean and standard deviation computed from the training data (in the previous step) to standardize the validation data.\n# Why Only transform()? It’s important to only apply transform() (and not fit_transform()) to the validation data to ensure that the model is not influenced \n# by the validation data during the scaling process. The idea is to apply the same transformation (using the statistics from the training data) \n# to the validation data.\n# Result: X_val is transformed based on the scaling parameters (mean and standard deviation) learned from X_train, ensuring consistency between training \n# and validation data.\n\ntest_trans = scaler.transform(test)\n# Purpose: This line standardizes the test data, assigning the result to test_trans.\n# What It Does: Similar to the validation data, the transform() method standardizes the test data using the mean and standard deviation computed from \n# the training data.\n# Result: test_trans is now the standardized version of the test dataset, where each feature is scaled according to the statistics (mean and standard deviation) \n# computed from X_train.","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:24.018348Z","iopub.execute_input":"2024-08-19T09:36:24.018712Z","iopub.status.idle":"2024-08-19T09:36:25.076585Z","shell.execute_reply.started":"2024-08-19T09:36:24.018683Z","shell.execute_reply":"2024-08-19T09:36:25.074457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to original source notebooks, **Chun Fu** used an ensemble of two algorithms (XGBClassifier, LGBMClassifier) to create the actual models for predicting the anomaly values. The two models are run here with their default parameters, but for reassurance, below are those default parameters from both models.\n\nThe **xgb_model** has \"None\" as it default value for more or less everything, whereas the **lgb_model** contains actual chosen values for some of the parameters. By fine-tuning these parameters, the models' performance can be changed, but here this is not done as it was not a part of the two original notebooks either.","metadata":{}},{"cell_type":"code","source":"# XGBoost classifier parameters\nxgb_model = XGBClassifier()\n\nprint(xgb_model.get_params())","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:25.078430Z","iopub.execute_input":"2024-08-19T09:36:25.078848Z","iopub.status.idle":"2024-08-19T09:36:25.086316Z","shell.execute_reply.started":"2024-08-19T09:36:25.078797Z","shell.execute_reply":"2024-08-19T09:36:25.084149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LightGBM classifier parameters\nlgb_model = lgb.LGBMClassifier()\n\nprint(lgb_model.get_params())","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:25.087687Z","iopub.execute_input":"2024-08-19T09:36:25.088134Z","iopub.status.idle":"2024-08-19T09:36:25.100100Z","shell.execute_reply.started":"2024-08-19T09:36:25.088091Z","shell.execute_reply":"2024-08-19T09:36:25.098350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next the two models are fitted with the training data. See the code commentary section for ChatGPT-created notes.","metadata":{}},{"cell_type":"code","source":"# Summary of the Entire Process:\n# Model Initialization: Two models (XGBoost and LightGBM classifiers) are created and stored in a list.\n# Model Training: Each model is trained on the training data (X_train, Y_train).\n# Performance Evaluation:\n# Training Accuracy: The ROC AUC score is calculated and printed for the training data to see how well the model has fit the training data.\n# Validation Accuracy: The ROC AUC score is calculated and printed for the validation data to evaluate how well the model generalizes to unseen data.\n# Output: The results are printed in a readable format for each model.\n\n# combine the two classifiers as \"models\"\nmodels = [xgb_model,lgb_model]\n\n# Models:\n# XGBClassifier(): This is an instance of the XGBoost classifier, which is an implementation of gradient boosting decision trees \n# designed for speed and performance.\n# LGBMClassifier(): This is an instance of the LightGBM classifier, another gradient boosting framework that uses tree-based \n# learning algorithms, known for its efficiency and scalability.\n\nfor i in range(2):\n    # Purpose: This line sets up a loop that will iterate over the two models in the models list.\n    # range(2): Generates a sequence of numbers from 0 to 1, corresponding to the indices of the models in the models list.\n    # What It Does: The loop will run twice, once for each model in the list.\n    \n    models[i].fit(X_train, Y_train)\n    # Purpose: This line trains the current model on the training data.\n    # What It Does:\n    # models[i]: Refers to the current model in the iteration (models[0] for the first iteration, models[1] for the second).\n    # .fit(X_train, Y_train): Trains the model using the X_train features and Y_train labels.\n    # Result: The model learns the relationship between the features in X_train and the target variable in Y_train.\n\n    print(f'{models[i]} : ')\n    # Purpose: This line prints the name of the current model.\n    # What It Does: Outputs the model’s class name (e.g., XGBClassifier or LGBMClassifier) to indicate which model’s results are being shown.\n    \n    print('Training Accuracy : ', metrics.roc_auc_score(Y_train, models[i].predict_proba(X_train)[:,1]))  \n    # Purpose: This line calculates and prints the training accuracy of the current model using the ROC AUC score.\n    # What It Does:\n    # models[i].predict_proba(X_train)[:,1]:\n    # The predict_proba() method returns the predicted probabilities for each class.\n    # [:,1] extracts the probability of the positive class (usually class 1).\n    # metrics.roc_auc_score(Y_train, ...): Computes the Area Under the Receiver Operating Characteristic Curve (ROC AUC) score, \n    # which is a performance measurement for classification problems at various threshold settings.\n    # Result: The ROC AUC score for the training data is printed, indicating how well the model fits the training data.\n     \n    print('Validation Accuracy : ', metrics.roc_auc_score(Y_val, models[i].predict_proba(X_val)[:,1]))\n    # Purpose: This line calculates and prints the validation accuracy of the current model using the ROC AUC score.\n    # What It Does:\n    # models[i].predict_proba(X_val)[:,1]: Similar to the training data, this line gets the predicted probabilities for the positive class \n    # on the validation data.\n    # metrics.roc_auc_score(Y_val, ...): Computes the ROC AUC score for the validation data.\n    # Result: The ROC AUC score for the validation data is printed, indicating how well the model generalizes to unseen data.\n    \n    print()\n    # Purpose: This line prints an empty line to separate the output for each model.","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:25.101943Z","iopub.execute_input":"2024-08-19T09:36:25.102330Z","iopub.status.idle":"2024-08-19T09:36:40.015729Z","shell.execute_reply.started":"2024-08-19T09:36:25.102301Z","shell.execute_reply":"2024-08-19T09:36:40.014487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Both training and validation accuracies for the two models are on the same level. Next all features in the test data will be scaled, and the result will be named **X_all**.","metadata":{}},{"cell_type":"code","source":"X_all = scaler.transform(features)\n# Purpose: This line standardizes the features data, assigning the result to X_all.","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:40.017185Z","iopub.execute_input":"2024-08-19T09:36:40.017552Z","iopub.status.idle":"2024-08-19T09:36:40.072023Z","shell.execute_reply.started":"2024-08-19T09:36:40.017520Z","shell.execute_reply":"2024-08-19T09:36:40.071039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now both of the models will be fitted with the scaled features data. As a result, both models will create their own probability predictions regarding the target (anomaly value), which was not a part of the test data.","metadata":{}},{"cell_type":"code","source":"# fit xgb_model\nmodels[0].fit(X_all, target)\n\n# create predictions\npredictions_xgb = models[0].predict_proba(test_trans)[:,1]\n\n# fit lgb_model\nmodels[1].fit(X_all, target)\n\n# create predictions\npredictions_lgb = models[1].predict_proba(test_trans)[:,1]","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:36:40.073387Z","iopub.execute_input":"2024-08-19T09:36:40.073738Z","iopub.status.idle":"2024-08-19T09:37:13.627736Z","shell.execute_reply.started":"2024-08-19T09:36:40.073709Z","shell.execute_reply":"2024-08-19T09:37:13.626332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions_xgb","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:13.629631Z","iopub.execute_input":"2024-08-19T09:37:13.630069Z","iopub.status.idle":"2024-08-19T09:37:13.638970Z","shell.execute_reply.started":"2024-08-19T09:37:13.630020Z","shell.execute_reply":"2024-08-19T09:37:13.637583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions_lgb","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:13.640423Z","iopub.execute_input":"2024-08-19T09:37:13.641240Z","iopub.status.idle":"2024-08-19T09:37:13.661567Z","shell.execute_reply.started":"2024-08-19T09:37:13.641202Z","shell.execute_reply":"2024-08-19T09:37:13.659947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The predictions are arrays containing the probabilities regarding whether or not the two models consider each case as an anomaly or not. But how did different features in our data affect these predictions? \n\nBelow are the different features and their respective importances as raw numerical data.","metadata":{}},{"cell_type":"code","source":"# feature importances for XGBoost\nprint(models[0].feature_importances_)\n\n# feature importances for LightGBM\nprint(models[1].feature_importances_)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:13.663276Z","iopub.execute_input":"2024-08-19T09:37:13.663902Z","iopub.status.idle":"2024-08-19T09:37:13.678936Z","shell.execute_reply.started":"2024-08-19T09:37:13.663844Z","shell.execute_reply":"2024-08-19T09:37:13.676927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To transform this data into more readable format, below two new datadrames are created. The **feature_importance_xgb** dataframe contains the **xgb_model** feature importance values along with column names taken from **features**. The **feature_importance_lgb** dataframe has the same data related to the **lgb_model** results.","metadata":{}},{"cell_type":"code","source":"# ChatGPT was used in creating the code\n# Define model\nmodel = xgb_model\n\n# Extract feature importances\nfeature_importances = model.feature_importances_\n\n# Create a DataFrame with feature names and their importances\nfeature_importance_xgb = pd.DataFrame({\n    'Feature': features.columns,  \n    # Feature names from the train dataframe\n    'Importance': feature_importances\n})\n\n# Sort the DataFrame by importance (optional)\nfeature_importance_xgb = feature_importance_xgb.sort_values(by='Importance', ascending=False)\n\n# Display the DataFrame\nprint(feature_importance_xgb)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:13.680668Z","iopub.execute_input":"2024-08-19T09:37:13.681122Z","iopub.status.idle":"2024-08-19T09:37:13.706647Z","shell.execute_reply.started":"2024-08-19T09:37:13.681078Z","shell.execute_reply":"2024-08-19T09:37:13.705049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = lgb_model\n\nfeature_importances = model.feature_importances_\n\nfeature_importance_lgb = pd.DataFrame({\n    'Feature': features.columns,  \n    'Importance': feature_importances\n})\n\nfeature_importance_lgb = feature_importance_lgb.sort_values(by='Importance', ascending=False)\n\nprint(feature_importance_lgb)","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:13.709116Z","iopub.execute_input":"2024-08-19T09:37:13.709473Z","iopub.status.idle":"2024-08-19T09:37:13.725286Z","shell.execute_reply.started":"2024-08-19T09:37:13.709444Z","shell.execute_reply":"2024-08-19T09:37:13.723892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is interesting to see that the **lgb_model** considered the **meterReadings_daily_std** feature as clearly the most significant factor in predictions. Also, the lag values appear relatively high in both results, so they seem to make the accuracy of both models better. \n\nAt this point, our predicted results are ready to be compiled and submitted.\n\n******","metadata":{}},{"cell_type":"markdown","source":"## 5.  Creating Submission <a class=\"anchor\" id=\"chapter_5\"></a>","metadata":{}},{"cell_type":"markdown","source":"Submitting a contest entry on Kaggle has no uniform procedure, and there may indeed be different methods in the same contest for people to send in their entry submission data. In this case, there was a separate sample submission file provided by the organizers. Let's see first what it looks like.","metadata":{}},{"cell_type":"code","source":"sample_sub = pd.read_csv('../input/energy-anomaly-detection/sample_submission.csv')\n\nsample_sub.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:13.727157Z","iopub.execute_input":"2024-08-19T09:37:13.727653Z","iopub.status.idle":"2024-08-19T09:37:14.154271Z","shell.execute_reply.started":"2024-08-19T09:37:13.727553Z","shell.execute_reply":"2024-08-19T09:37:14.152744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The sample submission consists of two columns. The first column is a simple row id column containing an integer, and the second column contains the anomaly values for each predicted case in the test data.\n\nAs there were actually two different models and algorithms predicting the same anomaly values, next these results will be added together and divided by two.","metadata":{}},{"cell_type":"code","source":"# combine the predictions and divide the values by two\nsample_sub['anomaly'] = (predictions_xgb + predictions_lgb)/2\n\nsample_sub.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:14.155758Z","iopub.execute_input":"2024-08-19T09:37:14.156127Z","iopub.status.idle":"2024-08-19T09:37:14.181232Z","shell.execute_reply.started":"2024-08-19T09:37:14.156096Z","shell.execute_reply":"2024-08-19T09:37:14.179337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Personally this was something I would not have realized on my own. In my mind, the sample submission would have contained either 0 or 1 as the predicted anomaly value, similar to the training data. *I had no idea that the anomaly prediction could be something between 0 and 1 in the submission file.*\n\nIn the original notebooks, there was one more detail **Chun Fu** brought up in his notebook. It concerns cases where the **meter_reading** value is 1 (or 1.0) in the test data. The code below shows the predicted anomaly values in these particular cases.","metadata":{}},{"cell_type":"code","source":"sample_sub.loc[test['meter_reading']==1.0, 'anomaly'].describe()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:14.183206Z","iopub.execute_input":"2024-08-19T09:37:14.183690Z","iopub.status.idle":"2024-08-19T09:37:14.207287Z","shell.execute_reply.started":"2024-08-19T09:37:14.183650Z","shell.execute_reply":"2024-08-19T09:37:14.206006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One can see from the percentiles above that practically all of these cases (where the **meter_reading** in test data is 1) were predicted as anomalies by the model ensemble. Based on this, **Chun Fu** edited the submission file afterwards so that the anomaly value was actually changed to 1 in each case where the **meter_reading** value in the test data was 1. Below is the code performing this change. After that, the results are saved in csv file format. For practical reasons, this line of code is commented out in this notebook.","metadata":{}},{"cell_type":"code","source":"# change the anomaly values based on meter_reading in test data\nsample_sub.loc[test['meter_reading']==1.0, 'anomaly'] = 1\n\n# write the results to csv file and show the file head (both lines are commented out)\n# sample_sub.to_csv('final_submission.csv', index=False)\n# final_submission.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-19T09:37:14.209353Z","iopub.execute_input":"2024-08-19T09:37:14.209828Z","iopub.status.idle":"2024-08-19T09:37:14.222656Z","shell.execute_reply.started":"2024-08-19T09:37:14.209785Z","shell.execute_reply":"2024-08-19T09:37:14.221107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.  Afterword <a class=\"anchor\" id=\"chapter_6\"></a>","metadata":{}},{"cell_type":"markdown","source":"When I submitted the results created in this notebook (which is based on earlier work by others as mentioned) a couple of times as a late submission with minor tweaks, the highest private score was 95.3, which is indeed very good. Thus one can argue that the methods and approaches in this notebook are both valid and likely applicable also in contexts outside the LEAD competition. For example, in various tasks related to time series analysis, the contents of the two original notebooks are in my opinion invaluable.\n\nTo conclude, one final humble thank you to both **Abishek Maurya** and **Chun Fu** for their skillful and innovative original work. Without their earlier efforts this notebook would simply would not exist. I also hope that this notebook - originally a personal learning project for private eyes only - may prove to be useful also to other people developing their personal data science and data analysis skills and knowledge further.\n\n******","metadata":{}}]}