{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHRAE: Automatic FE: Featuretools & Selection FE with SelectFromModel"},{"metadata":{},"cell_type":"markdown","source":"**I took my kernel as a basis: [Titanic - Featuretools (automatic FE)](https://www.kaggle.com/vbmokin/titanic-featuretools-automatic-fe)**\nBut classifiers were used there, and in our case regressors were needed."},{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0.1\"></a>\n\n## Table of Contents\n\n1. [Import libraries](#1)\n1. [Download datasets](#2)\n1. [Clearning data and basic FE](#3)\n1. [Automatic FE with Featuretools](#4)\n1. [Automatic feature selection (FS)](#5)\n -  [FS with the Pearson correlation](#5.1)\n -  [FS with SelectFromModel and LinearSVR](#5.2)\n -  [FS with SelectFromModel and RandomForestRegressor](#5.3)\n1. [Comparison of all options of selected features](#6)"},{"metadata":{},"cell_type":"markdown","source":"## 1. Import libraries <a class=\"anchor\" id=\"1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Thanks to:\n\n* Automatic FE: https://www.kaggle.com/vbmokin/titanic-featuretools-automatic-fe\n\n* The main code for basic FE: https://www.kaggle.com/isaienkov/lightgbm-fe-1-19"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport featuretools as ft\nfrom featuretools.primitives import *\nfrom featuretools.variable_types import Numeric\n\nfrom sklearn.preprocessing import LabelEncoder, MinMaxScaler\nfrom sklearn.svm import LinearSVR\nfrom sklearn.feature_selection import SelectFromModel\n# from sklearn.linear_model import LinearRegression, LogisticRegression, LassoCV\nfrom sklearn.ensemble import RandomForestRegressor\n# from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n# from sklearn.metrics import explained_variance_score\n# from sklearn.svm import LinearSVC\n# from sklearn.feature_selection import SelectFromModel, SelectKBest, RFE, chi2\n\n# model tuning\n# from hyperopt import STATUS_OK, Trials, fmin, hp, tpe, space_eval\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Download datasets <a class=\"anchor\" id=\"2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to https://www.kaggle.com/isaienkov/lightgbm-fe-1-19\nbuilding_df = pd.read_csv(\"../input/ashrae-energy-prediction/building_metadata.csv\")\nweather_train = pd.read_csv(\"../input/ashrae-energy-prediction/weather_train.csv\")\ntrain = pd.read_csv(\"../input/ashrae-energy-prediction/train.csv\")\ntrain = train.merge(building_df, left_on = \"building_id\", right_on = \"building_id\", how = \"left\")\ntrain = train.merge(weather_train, left_on = [\"site_id\", \"timestamp\"], right_on = [\"site_id\", \"timestamp\"])\ndel weather_train\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Clearning data and basic FE <a class=\"anchor\" id=\"3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Thanks to\n# basic code: https://www.kaggle.com/isaienkov/lightgbm-fe-1-19\n# optimization of format: https://www.kaggle.com/vbmokin/very-significant-safe-memory-lightgbm\ntrain[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\ntrain[\"weekday\"] = train[\"timestamp\"].dt.weekday\ntrain[\"hour\"] = train[\"timestamp\"].dt.hour\ntrain[\"weekday\"] = train['weekday'].astype(np.uint8)\ntrain[\"hour\"] = train['hour'].astype(np.uint8)\ntrain['year_built'] = train['year_built']-1900\ntrain['square_feet'] = (10*np.log(train['square_feet'])).astype(np.uint8)\n\ndef average_imputation(df, column_name):\n    imputation = df.groupby(['timestamp'])[column_name].mean()\n    \n    df.loc[df[column_name].isnull(), column_name] = df[df[column_name].isnull()][[column_name]].apply(lambda x: imputation[df['timestamp'][x.index]].values)\n    del imputation\n    return df\n\ntrain = average_imputation(train, 'wind_speed')\ntrain = average_imputation(train, 'wind_direction')\n\ndel train[\"timestamp\"]\n\nbeaufort = [(0, 0, 0.3), (1, 0.3, 1.6), (2, 1.6, 3.4), (3, 3.4, 5.5), (4, 5.5, 8), (5, 8, 10.8), (6, 10.8, 13.9), \n          (7, 13.9, 17.2), (8, 17.2, 20.8), (9, 20.8, 24.5), (10, 24.5, 28.5), (11, 28.5, 33), (12, 33, 200)]\n\nfor item in beaufort:\n    train.loc[(train['wind_speed']>=item[1]) & (train['wind_speed']<item[2]), 'beaufort_scale'] = item[0]\n\ndef degToCompass(num):\n    val=int((num/22.5)+.5)\n    arr=[i for i in range(0,16)]\n    return arr[(val % 16)]\n\ntrain['wind_direction'] = train['wind_direction'].apply(degToCompass)\ntrain['beaufort_scale'] = train['beaufort_scale'].astype(np.uint8)\ntrain[\"wind_direction\"] = train['wind_direction'].astype(np.uint8)\ntrain[\"meter\"] = train['meter'].astype(np.uint8)\ntrain[\"site_id\"] = train['site_id'].astype(np.uint8)\n\n# Thanks to https://www.kaggle.com/vbmokin/very-significant-safe-memory-lightgbm\ntrain[\"building_id\"] = train['building_id'].astype(np.uint16)\ntrain['air_temperature'] = np.int8(round(2*train['air_temperature'],0)) # store values with precision 0.5\ntrain['cloud_coverage'] = np.uint8(round(10*train['cloud_coverage'],0)) # store values with precision 0.1\ntrain['dew_temperature'] = np.int8(round(5*train['dew_temperature'],0)) # store values with precision 0.2\ntrain['wind_speed'] = np.int8(round(5*train['wind_speed'],0)) # store values with precision 0.2\ntrain['precip_depth_1_hr'] = np.uint8(np.clip(round(train['precip_depth_1_hr'],0),0,255)) # transform [-1,343] to [0,255]\ntrain = train.fillna(0)\ntrain['year_built'] = train['year_built'].astype(np.uint8)\ntrain['floor_count'] = train['floor_count'].astype(np.uint8)\n\n# Thanks to https://www.kaggle.com/isaienkov/lightgbm-fe-1-19\nle = LabelEncoder()\ntrain[\"primary_use\"] = le.fit_transform(train[\"primary_use\"])\ncategoricals = [\"site_id\", \"building_id\", \"primary_use\", \"hour\", \"weekday\", \"meter\",  \"wind_direction\"]\ndrop_cols = [\"sea_level_pressure\", \"site_id\"]\nnumericals = [\"square_feet\", \"year_built\", \"air_temperature\", \"cloud_coverage\",\n              \"dew_temperature\", 'precip_depth_1_hr', 'floor_count', 'beaufort_scale']\nfeat_cols = categoricals + numericals\ntarget = np.log1p(train[\"meter_reading\"]).astype(np.float16) # float16 - from https://www.kaggle.com/vbmokin/very-significant-safe-memory-lightgbm\ntrain = train.drop(drop_cols, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Selection part of data for automatic FE - 10000 meters and it's preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"dfe = train.sample(n=10000, replace=True, random_state=1)\ndfe = dfe.drop_duplicates(keep=False);\ndfe['index'] = dfe.index.tolist()\ndfe = dfe.fillna(0); #for cloud and precipitation\ndfe.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Set all features in Not-negative status\ndfe['air_temperature'] = dfe['air_temperature'] - dfe['air_temperature'].min()\ndfe['dew_temperature'] = dfe['dew_temperature'] - dfe['dew_temperature'].min()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_fe = dfe['meter_reading']\ndfe = dfe.drop(['meter_reading', 'floor_count','year_built'], axis = 1)\ndel train[\"meter_reading\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Automatic FE with Featuretools <a class=\"anchor\" id=\"4\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Thanks to:\n* https://www.kaggle.com/vbmokin/titanic-featuretools-automatic-fe\n* https://www.kaggle.com/liananapalkova/automated-feature-engineering-for-titanic-dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"es = ft.EntitySet(id = 'ashrae_energy_data')\nes = es.entity_from_dataframe(entity_id = 'dfe', dataframe = dfe, \n                              variable_types = \n                              {\n                                  'air_temperature': ft.variable_types.Numeric,\n                                  'cloud_coverage': ft.variable_types.Numeric,\n                                  'dew_temperature': ft.variable_types.Numeric,\n                                  'precip_depth_1_hr': ft.variable_types.Numeric\n                              },\n                              index = 'index')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"es = es.normalize_entity(base_entity_id='dfe', new_entity_id='air_temperature', index='air_temperature')\nes = es.normalize_entity(base_entity_id='dfe', new_entity_id='dew_temperature', index='dew_temperature')\nes = es.normalize_entity(base_entity_id='dfe', new_entity_id='cloud_coverage', index='cloud_coverage')\nes = es.normalize_entity(base_entity_id='dfe', new_entity_id='precip_depth_1_hr', index='precip_depth_1_hr')\nes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"primitives = ft.list_primitives()\nprimitives[primitives['type'] == 'aggregation'].head(primitives[primitives['type'] == 'aggregation'].shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('max_columns',500)\npd.set_option('max_rows',500)\npd.options.display.max_rows = 300","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features, feature_names = ft.dfs(entityset = es, \n                                 target_entity = 'dfe', \n                                 max_depth = 2)\nfeatures = features.fillna(0)\nlen(feature_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_names","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Automatic feature selection (FS)<a class=\"anchor\" id=\"5\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_norm = MinMaxScaler().fit_transform(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.1. FS with the Pearson correlation <a class=\"anchor\" id=\"5.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Absolute value correlation matrix\ncorr_matrix = features.corr().abs()\nupper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool));\n\n# Threshold for removing correlated variables\nthreshold = 0.9\n\ndef highlight(value):\n    if value > threshold:\n        style = 'background-color: pink'\n    else:\n        style = 'background-color: palegreen'\n    return style\n\n# Select columns with correlations above threshold\ncollinear_features = [column for column in upper.columns if any(upper[column] > threshold)]\nupper.style.applymap(highlight)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features_filtered = features.drop(columns = collinear_features)\nfeatures_positive = features_filtered.loc[:, features_filtered.ge(0).all()]\nprint('The number of features that passed the collinearity threshold: ', features_filtered.shape[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_option1 = features_positive.columns\nFE_option1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2. FS with SelectFromModel and LinearSVR <a class=\"anchor\" id=\"5.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"lsvr = LinearSVR(C=0.05, max_iter = 1000).fit(dfe, target_fe)\nmodel = SelectFromModel(lsvr, prefit=True)\nX_new = model.transform(dfe)\nX_selected_df = pd.DataFrame(X_new, columns=[dfe.columns[i] for i in range(len(dfe.columns)) if model.get_support()[i]])\nX_selected_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_option2 = features_positive.columns\nFE_option2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.3. FS with SelectFromModel and RandomForestRegressor <a class=\"anchor\" id=\"5.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = RandomForestRegressor(max_depth=2, random_state=0, n_estimators=100)\nembeded_rf_selector = SelectFromModel(model, threshold='1.25*median')\nembeded_rf_selector.fit(dfe, target_fe)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embeded_rf_support = embeded_rf_selector.get_support()\nembeded_rf_feature = dfe.loc[:,embeded_rf_support].columns.tolist()\nprint(str(len(embeded_rf_feature)), 'selected features')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_option3 = embeded_rf_feature\nFE_option3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Comparison of all options of selected features<a class=\"anchor\" id=\"6\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_all = set(FE_option1).union(set(FE_option2), set(FE_option3))\nlen(FE_all)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_all","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_general = set.intersection(set(FE_option1), set(FE_option2), set(FE_option3))\nlen(FE_all)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FE_general","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}