{"cells":[{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# ASHRAE Exploration"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"hidden":true},"cell_type":"markdown","source":"Kernel by [chmaxx](https://www.kaggle.com/chmaxx) – Oktober 2019"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"hidden":true},"cell_type":"markdown","source":"\nPhoto by NASA on unsplash.com"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"heading_collapsed":true,"hidden":true},"cell_type":"markdown","source":"## Improving energy efficiency with data science"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"hidden":true},"cell_type":"markdown","source":"**Improving energy efficiency is a very essential goal in times of massive climate change.**\n\n**Humans spend 80-90% of their living time in buildings** and we **consume substantial amounts of energy** e.g. for heating. Therefore we want buildings to be as efficient as possible. This often goes along with retrofits of existing buildings to make them consume less energy. \n\n> **This competition seeks to improve predictions about how efficient building retrofits are in terms of energy savings.**\n\nAssessing the value of energy efficiency improvements is challenging as **there's no way to truly know how much energy a building would have used without the improvements.** \n\nThe challenge in this competition is to build **counterfactual models:** Once a building is overhauled the new (lower) energy consumption is **compared against modeled values for the original building** to **calculate the savings from the retrofit**. \n\nMore accurate models could ease decision making, incentivize investments and make financing less expensive.\n\n- In this competition, we are asked to develop accurate models of metered building energy usage in four areas: chilled water, electric, hot water, and steam meters. \n- We are challenged to build **counterfactual models** across based on historic usage rates and observed weather. \n- The dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.\n\nThe initiator of this competition is **ASHRAE**, the **American Society of Heating, Refrigerating and Air-Conditioning Engineers**. This industry organization represents \"building system design and industrial processes professionals around the world\". It was founded 1904, has more than 57k members from 132 countries and is based in Georgia, USA."},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T20:12:25.030903Z","start_time":"2019-09-16T20:12:24.896099Z"},"hidden":true},"cell_type":"markdown","source":">We are asked to build a model and **predict energy consumption in kWh of various buildings** from describing features. This **a supervised regression machine learning task**."},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"heading_collapsed":true,"hidden":true},"cell_type":"markdown","source":"## Dataset description"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"hidden":true},"cell_type":"markdown","source":"In a nutshell we have **3 datasets** and one helper file at our disposal (`submission.csv` not counting): \n\n- **`train.csv`:** IDs of buildings, ID of meter type, meter readings and timestamps. **`meter_readings` is our target variable.**\n- **`building_meta.csv`:** Metadata of buildings\n- **`weather_train/test.csv`:** Weather data from closeby meteorological stations\n- `test.csv` is a sample file to enable submissions. It has no feature data but unique IDs for row of submission sample, building, meter and timestamp."},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"hidden":true},"cell_type":"markdown","source":"Detailed description:\n\n**`train.csv`**\n\n building_id - Foreign key for the building metadata.\n meter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. \n Not every building has all meter types.\n timestamp - When the measurement was taken\n meter_reading - The target variable. Energy consumption in kWh (or equivalent). \n Note that this is real data with measurement error, which we expect \n will impose a baseline level of modeling error.\n\n**`building_meta.csv`**\n\n site_id - Foreign key for the weather files.\n building_id - Foreign key for training.csv\n primary_use - Indicator of the primary category of activities for the building \n based on EnergyStar property type definitions\n square_feet - Gross floor area of the building\n year_built - Year building was opened\n floor_count - Number of floors of the building\n\n\n**`weather_[train/test].csv`**\n\nWeather data from a meteorological station as close as possible to the site.\n\n site_id\n air_temperature - Degrees Celsius\n cloud_coverage - Portion of the sky covered in clouds, in oktas\n dew_temperature - Degrees Celsius\n precip_depth_1_hr - Millimeters\n sea_level_pressure - Millibar/hectopascals\n wind_direction - Compass direction (0-360)\n wind_speed - Meters per second\n\n\n**`test.csv`**\n\nThe submission files use row numbers for ID codes in order to save space on the file uploads. test.csv has no feature data; it exists so you can get your predictions into the correct order.\n\n row_id - Row id for your submission file\n building_id - Building id code\n meter - The meter id code\n timestamp - Timestamps for the test data period\n\n\n**`sample_submission.csv`**\n\nA valid sample submission.\n\n All floats in the solution file were truncated to four decimal places; \n we recommend you do the same to save space on your file upload.\n There are gaps in some of the meter readings for both the train and test sets. \n Gaps in the test set are not revealed or scored."},{"metadata":{"ExecuteTime":{"end_time":"2019-09-19T18:40:06.764410Z","start_time":"2019-09-19T18:40:06.762417Z"},"heading_collapsed":true},"cell_type":"markdown","source":"# 1. Imports and globals"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:05:23.234648Z","start_time":"2019-10-22T13:05:14.168762Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\nimport matplotlib.pyplot as plt\n\nplt.style.use('ggplot')\nimport matplotlib.cm as cm\nimport seaborn as sns\n\nimport plotly.express as px\n\nimport pandas as pd\nimport pandas_profiling\nimport numpy as np\nfrom numpy import percentile\nfrom scipy import stats\nfrom scipy.stats import skew\nfrom scipy.special import boxcox1p\nimport random\n\nimport os, sys\nimport re\nfrom tabulate import tabulate\nimport missingno\n\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import MaxAbsScaler\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.preprocessing import Normalizer\nfrom sklearn.preprocessing import QuantileTransformer\nfrom sklearn.preprocessing import PowerTransformer\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.preprocessing import OrdinalEncoder\nfrom sklearn.preprocessing import OneHotEncoder\n\nfrom sklearn.model_selection import train_test_split\n\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.pipeline import Pipeline, make_pipeline\nfrom sklearn.model_selection import cross_val_score, cross_validate\nfrom sklearn.model_selection import GridSearchCV, KFold\nfrom sklearn.decomposition import PCA\nfrom sklearn.manifold import TSNE\n\nfrom sklearn.metrics import explained_variance_score\nfrom sklearn.metrics import max_error\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import mean_squared_log_error\nfrom sklearn.metrics import median_absolute_error\nfrom sklearn.metrics import r2_score\n\nfrom sklearn.dummy import DummyRegressor\nfrom sklearn.linear_model import LinearRegression, ElasticNet, Lasso, Ridge, RidgeCV\nfrom sklearn.svm import SVR\nfrom sklearn.kernel_ridge import KernelRidge\nfrom sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n\nimport xgboost as xgb\nimport lightgbm as lgb\n\nfrom umap import UMAP\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\ndef seed_everything(seed=0):\n random.seed(seed)\n np.random.seed(seed)\n \nseed_everything()\n\nplt.rc('font', size=18) \nplt.rc('axes', titlesize=22) \nplt.rc('axes', labelsize=18) \nplt.rc('xtick', labelsize=12) \nplt.rc('ytick', labelsize=12) \nplt.rc('legend', fontsize=12) \n\nplt.rcParams['font.sans-serif'] = ['Verdana']\n\npd.options.mode.chained_assignment = None\npd.options.display.max_seq_items = 500\npd.options.display.max_rows = 500\npd.set_option('display.float_format', lambda x: '%.5f' % x)\n\nBASE_PATH = \"../input/ashrae-energy-prediction/\"","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:05:23.246765Z","start_time":"2019-10-22T13:05:23.236217Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# reducing memory for now, we check later if this affects our models\ndef reduce_memory(df_):\n for col in df_.columns:\n if df_[col].dtype =='float64': df_[col] = df_[col].astype('float32')\n if df_[col].dtype =='int64': df_[col] = df_[col].astype('int32')\n return df_\n\n\n# This functions is based on this cool script: \n# https://www.kaggle.com/bwilsonkg/column-statistics\ndef show_stats(data_frame):\n stats_column_names = ('column', 'dtype', 'nan_cts', 'nan_perc', 'val_cts',\n 'min', 'max', 'mean', 'median', 'stdev', 'skew', 'kurtosis')\n stats_array = []\n length_df = len(data_frame)\n for column_name in sorted(data_frame.columns):\n col = data_frame[column_name]\n if is_numeric_column(col):\n nan_perc = 100 / length_df * col.isna().sum()\n stats_array.append(\n [column_name, col.dtype, col.isna().sum(), nan_perc, len(col.value_counts()),\n col.min(), col.max(), col.mean(), col.median(), col.std(), col.skew(),\n col.kurtosis()])\n else:\n nan_perc = 100 / length_df * col.isna().sum()\n stats_array.append(\n [column_name, col.dtype, col.isna().sum(), nan_perc, len(col.value_counts()),\n 0, 0, 0, 0, 0, 0, 0])\n stats_df = pd.DataFrame(data=stats_array, columns=stats_column_names)\n with pd.option_context('display.float_format', lambda x: '%.1f' % x):\n display(stats_df)\n \ndef of_type(stats_data_frame, column_dtype):\n return stats_data_frame[stats_data_frame['dtype'] == column_dtype]\n\ndef sort(data_frame, column_name, ascending=False):\n return data_frame.sort_values(column_name, ascending=ascending)\n\ndef is_numeric_column(df_column):\n numeric_types = (np.int16, np.float16, np.int32, np.float32,\n np.int64, np.float64)\n return df_column.dtype in numeric_types","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-20T20:32:53.713288Z","start_time":"2019-10-20T20:32:53.709249Z"},"heading_collapsed":true},"cell_type":"markdown","source":"# 1.a. Load data"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:02.285261Z","start_time":"2019-10-22T13:07:18.854760Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"bldg_df = reduce_memory(pd.read_csv(f\"{BASE_PATH}building_metadata.csv\"))\n\nwth_train = reduce_memory(pd.read_csv(f\"{BASE_PATH}weather_train.csv\"))\nwth_test = reduce_memory(pd.read_csv(f\"{BASE_PATH}weather_test.csv\"))\nwth_train.timestamp = pd.to_datetime(wth_train.timestamp)\nwth_test.timestamp = pd.to_datetime(wth_test.timestamp)\n\nweather_df = pd.concat([wth_train, wth_test])\nweather_df.timestamp = pd.to_datetime(weather_df.timestamp)\n\ntrain = reduce_memory(pd.read_csv(f\"{BASE_PATH}train.csv\"))\ntest = reduce_memory(pd.read_csv(f\"{BASE_PATH}test.csv\"))\ntrain.timestamp = pd.to_datetime(train.timestamp)\ntest.timestamp = pd.to_datetime(test.timestamp)\n\n# columns that can be set to dtype category\ncategory_cols = [\"site_id\", \"building_id\", \"primary_use\", \"year_built\", \"meter\"]\n\n# merge building with training data for EDA\n#df = pd.merge(train, bldg_df, how=\"left\")\n#df.timestamp = pd.to_datetime(df.timestamp)\n#df[category_cols] = df[category_cols].astype(\"category\")\n\n# merge train and test sets for modelling\ntrain_df = pd.merge(pd.merge(train, bldg_df, how=\"left\"), wth_train, how=\"left\")\ntest_df = pd.merge(pd.merge(test, bldg_df, how=\"left\"), wth_test, how=\"left\")\n\ntrain_df[category_cols] = train_df[category_cols].astype(\"category\")\ntest_df[category_cols] = test_df[category_cols].astype(\"category\")\n\ntrain_df.timestamp = pd.to_datetime(train_df.timestamp)\ntest_df.timestamp = pd.to_datetime(test_df.timestamp)\n\ndel wth_train, wth_test, train, test","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:02.301524Z","start_time":"2019-10-22T13:08:02.286598Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for frame_name, frame in zip([\"bldg_df \", \n \"weather_df \", \n \"train_df \", \n \"test_df \", \n \"df \"], \n [bldg_df, weather_df, train_df, test_df]):\n print(f'{frame_name}: {sys.getsizeof(frame)/(1024.0**3) :.2f} GB')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T20:12:25.030903Z","start_time":"2019-09-16T20:12:24.896099Z"},"heading_collapsed":true},"cell_type":"markdown","source":"# 2. Exploratory data analysis"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-19T19:12:15.345421Z","start_time":"2019-10-19T19:12:15.323034Z"},"hidden":true},"cell_type":"markdown","source":"### First checks"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:14.793898Z","start_time":"2019-10-22T13:08:02.303520Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_stats(train_df)\ndisplay(train_df.sample(5).head())","execution_count":null,"outputs":[]},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- Training data consists of **more than 20 Mio. meter readings**\n- **No obvious errors or missing values** noticeable yet\n- **Target variable highly skewed**"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:15.419235Z","start_time":"2019-10-22T13:08:14.795347Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_stats(bldg_df)\ndisplay(bldg_df.sample(5).head())\nmissingno.matrix(bldg_df, figsize=(16,5), fontsize=12);","execution_count":null,"outputs":[]},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- **1449 samples**\n- Buildings **built between 1900 and 2017**\n- **16 different sites**\n- **Many missing values** for `floor_count` and `building_year`"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:17.008844Z","start_time":"2019-10-22T13:08:15.420592Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_stats(weather_df)\ndisplay(weather_df.sample(5).head())\nmissingno.matrix(weather_df, figsize=(16,5), fontsize=12);","execution_count":null,"outputs":[]},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- **417k samples** (train and test) of **weather data**\n- **Missing values for 7 features** of 9"},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# 2.a. Exploration of **training data**"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-20T12:46:22.653062Z","start_time":"2019-10-20T12:46:20.930842Z"},"hidden":true},"cell_type":"markdown","source":"Findings from this section:\n\n- **20 Mio. samples, no missing values, no duplicates**\n- One year worth of data, **all timestamps from 2016**\n- Timewise the measurements are quite evenly distributed.\n- **Target variable highly skewed**\n- **Most readings measure electricity.**\n- **Generating steam consumes most energy.**\n- **Energy consumption peaks significantly in Spring, steam energy extremely so.**\n- Chilled water most consumed in late summer.\n- **Education buildings consume by far most of energy**, then come office buildings. \n- Buildings between 300-400k square feet by far with highest consumption.\n- **Buildings have up to four types of meters**, most have one (or three) types.\n- Distinct ranges of meter readings per building observable.\n- **3750 outliers for `meter_reading`**. They stem **from just 2 buildings (1099, 778)** and come **from steam and chilledwater**. \n- **Buildings 1099 and 778 are massive outliers. Examination yields very different results with and without these two buildings!**\n- **Building 1099 consumes lots of steam energy.**\n"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:17.883111Z","start_time":"2019-10-22T13:08:17.010019Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df.timestamp = pd.to_datetime(train_df.timestamp)\nprint(train_df.info(null_counts=True))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-09T20:04:33.834770Z","start_time":"2019-10-09T20:04:33.828486Z"},"hidden":true,"scrolled":false},"cell_type":"markdown","source":"Do we have duplicates? No, we don't either."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:35.326229Z","start_time":"2019-10-22T13:08:17.884285Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(train_df.shape)\nprint(train_df.drop_duplicates().shape)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:35.331580Z","start_time":"2019-10-22T13:08:35.328688Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"energy_types_dict = {0: \"electricity\", 1: \"chilledwater\", 2: \"steam\", 3: \"hotwater\"}\nenergy_types = ['electricity', 'chilledwater', 'steam', 'hotwater']","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-20T12:39:59.155214Z","start_time":"2019-10-20T12:39:57.490491Z"},"hidden":true},"cell_type":"markdown","source":"### Examine and plot"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:36.585585Z","start_time":"2019-10-22T13:08:35.333686Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\ntmp_df = train_df.meter.value_counts()\ntmp_df.index = energy_types\ntmp_df.sort_values().plot(kind=\"barh\")\nplt.title(f\"Most readings measure electricity\")\nplt.xlabel(\"Count of measurements\")\nplt.ylabel(f\"Meter type\")\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(16,5))\ntmp_df = train_df.groupby(\"meter\").meter_reading.sum()\ntmp_df.index = energy_types\ntmp_df.sort_values().plot(kind=\"barh\")\nplt.title(f\"Generating steam consumes most energy\")\nplt.xlabel(\"Sum of consumed energy\")\nplt.ylabel(f\"Type of energy\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:08:51.808739Z","start_time":"2019-10-22T13:08:36.586810Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\nsns.distplot(train_df.meter_reading, hist=False)\nplt.title(f\"Target variable meter_reading is highly skewed\")\nplt.ylabel(\"Count of readings\")\nplt.xlabel(f\"Measured consumption\")\nplt.xlim(0, train_df.meter_reading.max() + 100_000)\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(16,5))\nsns.distplot(np.log1p(train_df.meter_reading))\nplt.title(f\"After log transform, meter readings look more workable but still skewed\")\nplt.ylabel(\"Count of readings\")\nplt.xlabel(f\"Measured consumption\")\nplt.xlim(0, 12)\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(16,5))\nfor idx in range(0,4):\n sns.distplot(np.log1p(train_df[train_df.meter==idx].meter_reading), hist=False, label=energy_types[idx])\nplt.title(f\"After log transform, distributions of energy types look comparably skewed\")\nplt.ylabel(\"Count of readings\")\nplt.xlabel(f\"Measured consumption\")\nplt.legend()\nplt.xlim(0, 12)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:09:32.372683Z","start_time":"2019-10-22T13:08:51.809809Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,7))\n_ = stats.probplot(train_df['meter_reading'], plot=plt)\nplt.title(\"Probability plot for meter_reading shows extreme skewness\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:10:07.527569Z","start_time":"2019-10-22T13:09:32.374086Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,7))\n_ = stats.probplot(np.log(train_df['meter_reading']), plot=plt)\nplt.title(\"Even log transformed meter_reading is highly skewed\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-20T14:00:27.817534Z","start_time":"2019-10-20T14:00:27.573069Z"},"hidden":true},"cell_type":"markdown","source":"**Checking buildings with highest energy consumption.**"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:10:07.632046Z","start_time":"2019-10-22T13:10:07.528729Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df.groupby(\"building_id\").meter_reading.sum().sort_values(ascending=False)[:5]","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:10:11.337277Z","start_time":"2019-10-22T13:10:07.633207Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for bldg_id in [1099, 778, 1197, 1168, 1159]:\n plt.figure(figsize=(16,5))\n tmp_df = train_df[train_df.building_id == bldg_id].copy()\n tmp_df.set_index(\"timestamp\", inplace=True)\n tmp_df.resample(\"D\").meter_reading.sum().plot()\n plt.title(f\"Meter readings for building #{bldg_id} \")\n plt.xlabel(\"Sum of readings\")\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-20T14:24:23.788854Z","start_time":"2019-10-20T14:24:23.766837Z"},"hidden":true},"cell_type":"markdown","source":"Findings from the meter readings of the buildings with the highest energy consumption:\n\n- Building #1099 has a massive steam energy peak in Spring and November.\n- Building #778 has a massive consumption peak in Autumn.\n- #1168 and #1159 have very similar consumption patterns."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:10:17.874994Z","start_time":"2019-10-22T13:10:11.338278Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp_df = train_df.groupby(\"primary_use\").meter_reading.sum().sort_values()\n\noutliers_index = train_df[train_df.building_id.isin([1099, 778])].index\ntemp_df_inliers = train_df.drop(outliers_index).groupby(\"primary_use\").meter_reading.sum().sort_values()\n\nplt.figure(figsize=(16,9))\ntemp_df.plot(kind=\"barh\")\nplt.title(f\"Education buildings consume by far most of energy\")\nplt.xlabel(\"Sum of readings\")\nplt.ylabel(f\"Primary use\")\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(16,9))\ntemp_df_inliers.plot(kind=\"barh\")\nplt.title(f\"Less so without outliers 1099, 778\")\nplt.xlabel(\"Sum of readings\")\nplt.ylabel(f\"Primary use\")\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(16,9))\ntemp_df[:-1].plot(kind=\"barh\")\nplt.title(f\"Among other types, office buildings consume most energy\")\nplt.xlabel(\"Sum of readings\")\nplt.ylabel(f\"Primary use w/o «Education»\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:10:55.244418Z","start_time":"2019-10-22T13:10:52.588304Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sq_binned = pd.cut(train_df.square_feet, bins=np.arange(0, 1_000_000, 100_000))\nsq_binned = pd.DataFrame(sq_binned)\nsq_binned.columns = [\"sq_binned\"]\ntmp_df = pd.concat([train_df, sq_binned], axis=1).groupby(\"sq_binned\").meter_reading.mean().sort_index()\n\nplt.figure(figsize=(16,7))\ntmp_df.plot(kind=\"barh\")\nplt.title(f\"Buildings between 300-400k square feet consume by far most of energy\")\nplt.xlabel(\"Mean of consumption\")\nplt.ylabel(f\"Square feet of building (binned)\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:11:23.426195Z","start_time":"2019-10-22T13:11:17.531772Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sq_binned = pd.cut(train_df.drop(outliers_index).square_feet, bins=np.arange(0, 1_000_000, 100_000))\nsq_binned = pd.DataFrame(sq_binned)\nsq_binned.columns = [\"sq_binned\"]\ntmp_df = pd.concat([train_df.drop(outliers_index), sq_binned], axis=1).groupby(\"sq_binned\").meter_reading.mean().sort_index()\n\nplt.figure(figsize=(16,7))\ntmp_df.plot(kind=\"barh\")\nplt.title(f\"More balanced distribution if we remove outliers 1099, 778\")\nplt.xlabel(\"Mean of consumption\")\nplt.ylabel(f\"Square feet of building (binned)\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:11:26.738639Z","start_time":"2019-10-22T13:11:23.427374Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\ntrain_df.groupby(\"building_id\").meter.nunique().value_counts().sort_index().plot(kind=\"bar\")\nplt.title(f\"Buildings have up to four types of meters, most have one (or three) types\")\nplt.ylabel(\"Count of buildings\")\nplt.xlabel(f\"Number of different types of meters\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:11:30.948731Z","start_time":"2019-10-22T13:11:26.740686Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\ntrain_df.building_id.value_counts().plot(kind=\"bar\")\nplt.title(\"There are distinct ranges of meter readings per building\")\nplt.ylabel(\"Count of readings\")\nplt.xlabel(\"building_id\")\nplt.xticks([])\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(16,5))\ntmp_df = pd.cut(train_df.building_id.value_counts(), bins=np.arange(0, 45_000, 10_000))\ntmp_df = pd.DataFrame(tmp_df)\ntmp_df.building_id.value_counts().sort_index().plot(kind=\"bar\")\nplt.title(\"Distinct ranges of meter readings per building (binned)\")\nplt.ylabel(\"Count of readings\")\nplt.xlabel(\"building_id\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n\ndisplay(tmp_df.building_id.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-17T07:40:07.440065Z","start_time":"2019-10-17T07:40:07.140899Z"},"hidden":true},"cell_type":"markdown","source":"- 877 buildings have up to 10k readings.\n- 560 buildings have between 10k and 30k.\n- For just 12 buildings we have more than 30k."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:11:31.005768Z","start_time":"2019-10-22T13:11:30.950361Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"Timestamps in the training set range from {train_df.timestamp.min()} to {train_df.timestamp.max()}\")","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-17T07:50:43.282967Z","start_time":"2019-10-17T07:50:42.562348Z"},"hidden":true},"cell_type":"markdown","source":"We have exactly **one year (2016) worth of training data.**"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:11:38.386637Z","start_time":"2019-10-22T13:11:31.006931Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"timeframes = {\"month\" : train_df.timestamp.dt.month,\n \"week\" : train_df.timestamp.dt.week, \n \"weekday\" : train_df.timestamp.dt.weekday, \n \"hour\" : train_df.timestamp.dt.hour}\n\nfor timeframe_name, timeframe in timeframes.items():\n plt.figure(figsize=(16,5))\n train_df.groupby(timeframe).building_id.count().plot(kind=\"bar\")\n plt.title(f\"Quite even counts of meter readings per {timeframe_name}\")\n plt.ylabel(\"Count of readings\")\n plt.xlabel(f\"{timeframe_name}\")\n plt.xticks(rotation=45)\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-17T06:58:09.042161Z","start_time":"2019-10-17T06:58:09.037957Z"},"hidden":true},"cell_type":"markdown","source":"Timewise **the measurements are quite evenly distributed.** Only February and March stick out with a little less measurements."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:12:50.103963Z","start_time":"2019-10-22T13:12:22.319474Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for timeframe_name, timeframe in timeframes.items():\n plt.figure(figsize=(16,5))\n train_df.groupby(timeframe).meter_reading.median().plot(kind=\"bar\")\n plt.title(f\"Fairly stable median of energy consumption per {timeframe_name}\")\n plt.ylabel(\"Median of energy consumption\")\n plt.xlabel(f\"{timeframe_name}\")\n plt.xticks(rotation=45)\n \n if timeframe_name == \"weekday\":\n plt.title(\"Lower median consumption during weekend\")\n if timeframe_name == \"hour\":\n plt.title(\"Higher median consumption during daytime\")\n \n plt.tight_layout()\n plt.show()\n \n \n plt.figure(figsize=(16,5))\n train_df.groupby(timeframe).meter_reading.sum().plot(kind=\"bar\")\n plt.title(f\"Energy consumption peaks significantly in Spring (again due to outliers 1099, 778)\")\n plt.ylabel(\"Total energy consumption\")\n plt.xlabel(f\"{timeframe_name}\")\n plt.xticks(rotation=45)\n \n if timeframe_name == \"weekday\":\n plt.title(\"Lower total consumption during weekend\")\n if timeframe_name == \"hour\":\n plt.title(\"Higher total consumption during daytime and evening\")\n \n plt.tight_layout()\n plt.show()\n \n \n plt.figure(figsize=(16,7))\n sns.boxplot(x=timeframe, y=\"meter_reading\", data=train_df, showfliers=False)\n plt.title(f\"Noticable differences in distribution of meter readings per {timeframe_name}\")\n plt.ylabel(\"meter readings\")\n plt.xlabel(f\"{timeframe_name}\")\n plt.xticks(rotation=45)\n \n if timeframe_name == \"weekday\":\n plt.title(f\"Fairly stable distribution of meter readings per {timeframe_name}\")\n plt.xlabel(f\"{timeframe_name} (0 == Monday)\")\n if timeframe_name == \"hour\":\n plt.title(f\"Fairly stable distribution of meter readings per {timeframe_name}\")\n\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-17T09:44:03.723181Z","start_time":"2019-10-17T09:44:03.719399Z"},"hidden":true},"cell_type":"markdown","source":"**In general we observe more energy consumption during summer months (June to September), on weekdays and during the day.**"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:13:01.884721Z","start_time":"2019-10-22T13:12:50.105090Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for timeframe_name, timeframe in timeframes.items():\n plt.figure(figsize=(16,7))\n for idx in range(0,4):\n tmp_df = train_df[train_df.meter==idx].groupby(timeframe).meter_reading.sum()\n tmp_df.plot(kind=\"line\", label=energy_types[idx], use_index=True)\n plt.xticks(rotation=45)\n plt.ylabel(\"Median of consumption\")\n plt.xlabel(f\"{timeframe_name}\")\n plt.title(f\"Steam energy consumption fairly stable through out the day\")\n \n if timeframe_name in [\"month\", \"week\"]:\n plt.title(f\"Steam energy consumption peaks extremely in Spring due to outliers\")\n if timeframe_name in [\"weekday\"]:\n plt.title(f\"Steam energy consumption stronger on Wednesdays, less on weekends\")\n \n plt.legend(bbox_to_anchor=(1, 1), loc=2)\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:13:40.502517Z","start_time":"2019-10-22T13:13:01.886621Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for timeframe_name, timeframe in timeframes.items():\n plt.figure(figsize=(16,7))\n for idx in range(0,4):\n tmp_df = train_df.drop(outliers_index)\n tmp_df = tmp_df[tmp_df.meter==idx].groupby(timeframe).meter_reading.sum()\n tmp_df.plot(kind=\"line\", label=energy_types[idx], use_index=True)\n plt.xticks(rotation=45)\n plt.ylabel(\"Median of consumption\")\n plt.xlabel(f\"{timeframe_name}\")\n plt.title(f\"Energy consumption fairly stable through out the day\")\n \n if timeframe_name in [\"month\", \"week\"]:\n plt.title(f\"Steam energy consumption w/o outliers stronger in cold season\")\n if timeframe_name in [\"weekday\"]:\n plt.title(f\"Energy consumption a little less on weekends\")\n \n plt.legend(bbox_to_anchor=(1, 1), loc=2)\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:11.178365Z","start_time":"2019-10-22T13:13:40.503739Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for timeframe_name, timeframe in timeframes.items():\n plt.figure(figsize=(16,7))\n for idx in range(0,4):\n if idx == 2:\n continue\n tmp_df = train_df.drop(outliers_index)\n tmp_df = tmp_df[tmp_df.meter==idx].groupby(timeframe).meter_reading.sum()\n tmp_df.plot(kind=\"line\", label=energy_types[idx], use_index=True)\n plt.xticks(rotation=45)\n plt.ylabel(\"Median of consumption\")\n plt.xlabel(f\"{timeframe_name}\")\n \n if timeframe_name in [\"month\", \"week\"]:\n plt.title(f\"Chilled water most consumed in late summer / September\")\n if timeframe_name in [\"weekday\"]:\n plt.title(f\"Consumption fairly stable during week, less on weekend\")\n if timeframe_name in [\"hour\"]:\n plt.title(f\"Consumption stronger during daytime, less at night\\nMore hotwater consumption in mornings, chilledwater more on afternoons\")\n \n plt.legend(bbox_to_anchor=(1, 1), loc=2)\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T19:49:42.203528Z","start_time":"2019-09-16T19:49:42.200188Z"},"hidden":true},"cell_type":"markdown","source":"### Check for outliers\n\nWe quickly check for outliers that might negatively influence our regression algorithms. "},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T19:49:42.203528Z","start_time":"2019-09-16T19:49:42.200188Z"},"hidden":true},"cell_type":"markdown","source":"**But what exactly is an outlier?**\n\n[Taken from machinelearningmastery.com:](https://machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/)\n\n>**An outlier is an observation that is unlike the other observations.** It is rare, or distinct, or does not fit in some way. \n\n>_Outliers can have many causes, such as: Measurement or input error, data corruption or true outlier observation (e.g. Michael Jordan in basketball).\n**There is no precise way to define and identify outliers in general because of the specifics of each dataset.** Instead, you, or a domain expert, must interpret the raw observations and decide whether a value is an outlier or not.\nNevertheless, we can use statistical methods to identify observations that appear to be rare or unlikely given the available data._ \n\nOutliers will very likely decrease our models accuracy – since they make no sense and do not follow any regularities that can be learned by the algorithm. They are wrong and possibly have to be excluded from the training data. \n\nA sound practical approach for _normally distributed data_ is to filter values that lie beyond 3 standard deviations from the mean. "},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:11.486937Z","start_time":"2019-10-22T13:14:11.179434Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# calculate normal and extreme upper and lower cut off\n\ncut_off = train_df[\"meter_reading\"].std() * 3\nlower = train_df[\"meter_reading\"].mean() - cut_off \nupper = train_df[\"meter_reading\"].mean() + cut_off\ndf_lower = train_df[train_df[\"meter_reading\"] < lower]\ndf_upper = train_df[train_df[\"meter_reading\"] > upper]\n \nif df_lower.shape[0] != 0 or df_upper.shape[0] != 0:\n print(f\"{'meter_reading'}\")\n print(f\"lower bound: {lower:.2f}\\nupper bound: {upper:.2f}\")\nif df_lower.shape[0] != 0:\n display(train_df[train_df[\"meter_reading\"] < lower].sort_values(\"meter_reading\"))\nif df_upper.shape[0] != 0:\n display(train_df[train_df[\"meter_reading\"] > upper].sort_values(\"meter_reading\"))\n\ndisplay(df_upper.building_id.value_counts())\ndisplay(df_upper.meter.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-10T16:40:12.450883Z","start_time":"2019-10-10T16:40:12.447508Z"},"hidden":true},"cell_type":"markdown","source":"- We have **found 3750 outliers for `meter_reading`.** \n- They stem from just 2 buildings (1099, 778) and come from steam and chilledwater.\n- **These two buildings are massive outliers!**\n- Building 1099 consume lots of steam energy.\n\nWe might consider dropping these samples during training."},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# 2.b. Exploration of **building data**"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"Summary of findings from this section:\n\n- **1449 unique buildings** on **16 sites** built between **1900 and 2017** \n- **Wide variety of size and height**: 283 to 875k sqft, 1 to 26 floors\n- **16 types of usage**\n- 14 sites with mixed types of buildings, 2 with just one type\n- **`site_id` likely a city or area rather than a specific compound of buildings**\n- Distribution of square_feet highly skewed\n- Parking facilities much bigger on average than all other types\n- Food, parking, retail among the most recent built buildings\n- Technology and science related buildings on average the oldest\n- Lodging/residential on average the highest buildings\n- 29 outliers for `square_feet` and 5 for `floor_count`"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:11.494366Z","start_time":"2019-10-22T13:14:11.488315Z"},"hidden":true,"scrolled":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(bldg_df.info())","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-09T20:04:33.834770Z","start_time":"2019-10-09T20:04:33.828486Z"},"hidden":true,"scrolled":false},"cell_type":"markdown","source":"Do we have duplicates? No, we don't."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:11.502236Z","start_time":"2019-10-22T13:14:11.496050Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(bldg_df.shape)\nprint(bldg_df.drop_duplicates().shape)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-09T20:04:33.846199Z","start_time":"2019-10-09T20:04:33.836130Z"},"hidden":true,"scrolled":false},"cell_type":"markdown","source":"Check for missing values again."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:11.518563Z","start_time":"2019-10-22T13:14:11.504771Z"},"hidden":true,"scrolled":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"missing = [(c, bldg_df[c].isna().mean()*100) for c in bldg_df]\nmissing = pd.DataFrame(missing, columns=[\"feature\", \"percentage\"])\nmissing[\"count\"] = [bldg_df[c].isna().sum() for c in bldg_df]\nmissing = missing[missing.percentage > 0]\ndisplay(missing.sort_values(\"percentage\", ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T18:27:39.790598Z","start_time":"2019-09-16T18:27:39.711453Z"},"hidden":true},"cell_type":"markdown","source":"We have a **large number of missing floor counts and building years**. "},{"metadata":{"ExecuteTime":{"end_time":"2019-10-20T12:51:45.607358Z","start_time":"2019-10-20T12:51:44.997997Z"},"hidden":true},"cell_type":"markdown","source":"### Examine and plot"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:12.126217Z","start_time":"2019-10-22T13:14:11.520671Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"We have {bldg_df.building_id.nunique()} unique buildings and {bldg_df.site_id.nunique()} sites of buildings.\")\nsites_df = pd.DataFrame(bldg_df.site_id.value_counts())\nsites_df.sort_values(\"site_id\", inplace=True, ascending=False)\nsites_ordered_index = sites_df.index\nprint(f\"The number of buildings per site range from {sites_df.site_id.min()} to {sites_df.site_id.max()}.\")\n\nplt.figure(figsize=(16,5))\nsites_df.site_id.plot(kind=\"bar\")\nplt.title(\"Count of buildings per site_id vary between 274 and 5\")\nplt.ylabel(\"Count of buildings\")\nplt.xlabel(\"site id\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:12.145305Z","start_time":"2019-10-22T13:14:12.127311Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_stats(bldg_df[[\"square_feet\", \"year_built\", \"floor_count\"]])","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:12.187357Z","start_time":"2019-10-22T13:14:12.146579Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Smallest building\")\ndisplay(bldg_df[bldg_df.square_feet == bldg_df.square_feet.min()])\nprint(\"Largest building\")\ndisplay(bldg_df[bldg_df.square_feet == bldg_df.square_feet.max()])\nprint(\"Oldest buildings\")\ndisplay(bldg_df[bldg_df.year_built == bldg_df.year_built.min()])\nprint(\"Most recent building\")\ndisplay(bldg_df[bldg_df.year_built == bldg_df.year_built.max()])\nprint(\"Least tallest building\")\ndisplay(bldg_df[bldg_df.floor_count == bldg_df.floor_count.min()][:5])\nprint(\"Tallest building\")\ndisplay(bldg_df[bldg_df.floor_count == bldg_df.floor_count.max()])","execution_count":null,"outputs":[]},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- The buildings have a **wide variety of size and height:** from a tiny 283 square feet retail shop to a 875k square feet entertainment venue as well as from many one floor buildings to a 26 floor education building.\n- Houses in the dataset were **built between 1900 and 2017** with a mean / median around 1968/1970."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:12.193924Z","start_time":"2019-10-22T13:14:12.188932Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display(bldg_df.primary_use.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-16T12:31:50.773754Z","start_time":"2019-10-16T12:31:50.769727Z"},"hidden":true},"cell_type":"markdown","source":"According to the dataset description, these categories are based on [EnergyStar property type definitions that can be found here.](https://www.energystar.gov/buildings/tools-and-resources/list-portfolio-manager-property-types-definitions-and-use-details)\n\nInterestingly **many buildings seem to be public buildings especially in the education sector**, so probably schools and universities. "},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:13.602904Z","start_time":"2019-10-22T13:14:12.195362Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"graph_df = bldg_df.groupby(\"site_id\")[\"primary_use\"].value_counts().unstack()\ngraph_df = graph_df.reindex(sites_ordered_index)\ngraph_df.plot(kind=\"bar\", stacked=True, figsize=(16,7))\nplt.title(\"Primary use mapped to site_id\")\nplt.ylabel(\"Count of buildings\")\nplt.xlabel(\"site_id\")\nplt.legend(bbox_to_anchor=(1, 1), loc=2)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-14T15:23:12.463773Z","start_time":"2019-10-14T15:23:12.451811Z"},"hidden":true},"cell_type":"markdown","source":"- **Sites almost always are a mixed bag of several different usages.** \n- Only the sites with the least count of buildings (#7 and #11) are education only. \n- `site_id` **as a feature is therefore likely a city or an area** rather than a compound of buildings that was built for a certain type of usage."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:14.229378Z","start_time":"2019-10-22T13:14:13.603932Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\nsns.distplot(bldg_df[\"square_feet\"])\nplt.title(f\"Distribution of square feet of buildings\")\nplt.xlabel(f\"square feet\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-14T15:23:12.463773Z","start_time":"2019-10-14T15:23:12.451811Z"},"hidden":true},"cell_type":"markdown","source":"The distribution of the values is highly skewed. "},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:15.650661Z","start_time":"2019-10-22T13:14:14.230267Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"size_df = bldg_df.groupby(\"primary_use\")[\"square_feet\"].mean().sort_values(ascending=False)\norder = size_df.index\nplt.figure(figsize=(16,12))\nsns.boxplot(x=\"primary_use\", y=\"square_feet\", data=bldg_df, order=order)\nplt.title(\"Parking facilities by far the biggest on average\")\nplt.ylabel(\"square feet\")\nplt.xticks(rotation=80)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-14T15:23:12.463773Z","start_time":"2019-10-14T15:23:12.451811Z"},"hidden":true},"cell_type":"markdown","source":"- Parking facilities are on average much bigger\n- Service, healthcare, education and office buildings are usually big too. \n- Religious buildings in this data set are the smallest on average. "},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:16.358137Z","start_time":"2019-10-22T13:14:15.652569Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\nbldg_df[\"year_built\"].dropna().plot(kind=\"hist\", bins=117, rwidth=0.9)\nplt.title(f\"A significant peak of 55 buildings from 1976\")\nplt.xlabel(f\"Year built\")\nplt.tight_layout()\nplt.show()\n\ndisplay(bldg_df['year_built'].dropna().value_counts().iloc[:1])","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:17.756333Z","start_time":"2019-10-22T13:14:16.359219Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"size_df = bldg_df.groupby(\"primary_use\")[\"year_built\"].median().sort_values(ascending=False)\norder = size_df.index\nplt.figure(figsize=(16,12))\nsns.boxplot(x=\"primary_use\", y=\"year_built\", data=bldg_df, order=order)\nplt.title(\"Food, Parking, Healthcare and Retail buildings are the youngest\")\nplt.ylabel(\"year_built\")\nplt.xticks(rotation=80)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-14T15:23:12.463773Z","start_time":"2019-10-14T15:23:12.451811Z"},"hidden":true},"cell_type":"markdown","source":"Regarding the building years for the various usages we have to keep in mind that many values are missing here. \n\n- **«Technology/science» related buildings are on average quite old!**\n- Food, Parking, Healthcare and Retail facilities are on average of youngest age in the dataset."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:19.094058Z","start_time":"2019-10-22T13:14:17.757345Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"size_df = bldg_df.groupby(\"primary_use\")[\"floor_count\"].median().sort_values(ascending=False)\norder = size_df.index\n\nplt.figure(figsize=(16,12))\nsns.boxplot(x=\"primary_use\", y=\"floor_count\", data=bldg_df, order=order)\nplt.title(\"Lodging/residential on average the tallest buildings\")\nplt.ylabel(\"floor count\")\nplt.xticks(rotation=80)\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-14T15:23:12.463773Z","start_time":"2019-10-14T15:23:12.451811Z"},"hidden":true},"cell_type":"markdown","source":"Again we keep in mind that more than 75% of floor count values are missing. \n\nResidential buildings in the dataset are the tallest on average."},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T19:49:42.203528Z","start_time":"2019-09-16T19:49:42.200188Z"},"hidden":true},"cell_type":"markdown","source":"### Check for outliers in building data"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:19.125883Z","start_time":"2019-10-22T13:14:19.095263Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# calculate normal and extreme upper and lower cut off\nfor feature in bldg_df.select_dtypes(\"number\").columns:\n\n cut_off = bldg_df[feature].std() * 3\n lower = bldg_df[feature].mean() - cut_off \n upper = bldg_df[feature].mean() + cut_off\n df_lower = bldg_df[bldg_df[feature] < lower]\n df_upper = bldg_df[bldg_df[feature] > upper]\n \n if df_lower.shape[0] != 0 or df_upper.shape[0] != 0:\n print(f\"{feature}\")\n print(f\"lower bound: {lower:.2f}\\nupper bound: {upper:.2f}\")\n if df_lower.shape[0] != 0:\n display(bldg_df[bldg_df[feature] < lower].sort_values(feature))\n if df_upper.shape[0] != 0:\n display(bldg_df[bldg_df[feature] > upper].sort_values(feature))\n print()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-10T16:40:12.450883Z","start_time":"2019-10-10T16:40:12.447508Z"},"hidden":true},"cell_type":"markdown","source":"We have found \n\n- 29 outliers for `square_feet` \n- 5 outliers for `floor_count` \n\nWe might consider dropping these samples during training."},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# 2.c. Exploration of **weather data**"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"Summary of findings from this section:\n\n- 417016 samples, duplicates, missing values\n- All buildings from northern hemisphere, since seasonal temperature in general follows the same pattern\n- cloud_coverage for site_id 12 is much higher than on all other sites.\n- `sea_level_pressure` for site_id 5 is entirely missing (NaN)."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:19.151665Z","start_time":"2019-10-22T13:14:19.127173Z"},"hidden":true,"scrolled":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"weather_df.set_index(\"timestamp\", inplace=True)\nprint(weather_df.info())","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-09T20:04:33.834770Z","start_time":"2019-10-09T20:04:33.828486Z"},"hidden":true,"scrolled":false},"cell_type":"markdown","source":"Do we have **duplicates? Yes we have!**"},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:19.247166Z","start_time":"2019-10-22T13:14:19.153174Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(weather_df.shape)\nprint(weather_df.drop_duplicates().shape)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-09T20:04:33.846199Z","start_time":"2019-10-09T20:04:33.836130Z"},"hidden":true,"scrolled":false},"cell_type":"markdown","source":"Check for missing values again."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:19.267878Z","start_time":"2019-10-22T13:14:19.248463Z"},"hidden":true,"scrolled":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"missing = [(c, weather_df[c].isna().mean()*100) for c in weather_df]\nmissing = pd.DataFrame(missing, columns=[\"feature\", \"percentage\"])\nmissing[\"count\"] = [weather_df[c].isna().sum() for c in weather_df]\nmissing = missing[missing.percentage > 0]\ndisplay(missing.sort_values(\"percentage\", ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-09-16T18:27:39.790598Z","start_time":"2019-09-16T18:27:39.711453Z"},"hidden":true},"cell_type":"markdown","source":"We have a **larger number of missing values for `cloud_coverage` and `precip_depth_1_hr`**. "},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:19.961040Z","start_time":"2019-10-22T13:14:19.270654Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,5))\nweather_df.groupby(pd.Grouper(freq=\"W\"))['air_temperature'].sum().plot()\nplt.title('air_temperature')\nplt.ylabel('air_temperature')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:27.095276Z","start_time":"2019-10-22T13:14:19.962271Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"features = ['air_temperature', 'dew_temperature', \n 'sea_level_pressure', 'wind_direction', 'wind_speed', \n 'cloud_coverage', 'precip_depth_1_hr',]\nfor feature in features:\n plt.figure(figsize=(16,5))\n for sid in weather_df.site_id.unique():\n weather_df[weather_df.site_id==sid].groupby(pd.Grouper(freq=\"W\"))[feature].sum().plot()\n plt.title(f\"{feature}\")\n plt.ylabel(f\"{feature}\")\n plt.tight_layout()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:51.873586Z","start_time":"2019-10-22T13:14:27.096323Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# these plots are inspired by this kernel: \n# https://www.kaggle.com/blue07/eda-insights-on-weather-buildings\n\nfor feature in features:\n fig, axes = plt.subplots(nrows=4, ncols=4, sharex=True, sharey=True, figsize=(16, 12))\n for sid in weather_df.site_id.unique():\n row = int(sid / 4)\n col = sid%4\n tmp_df = weather_df[weather_df.site_id==sid]\n missing = 100 / len(tmp_df) * tmp_df[feature].isnull().sum()\n tmp_df.groupby(pd.Grouper(freq=\"M\"))[feature].sum().plot(ax=axes[row,col])\n if missing !=0:\n axes[row, col].set_title(f\"site {sid}, null:{missing :.2f}%\", fontsize=12, color=\"darkred\")\n else:\n axes[row, col].set_title(f\"site {sid}\", fontsize=12, color=\"darkgreen\")\n axes[row, col].set_xlabel(\"\")\n fig.suptitle(f\"{feature}\", fontsize=18)\n fig.subplots_adjust(top=0.92)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:54.830742Z","start_time":"2019-10-22T13:14:51.874721Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=4, ncols=4, sharex=True, sharey=True, figsize=(16, 12))\nfor sid in weather_df.site_id.unique():\n row = int(sid / 4)\n col = sid%4\n tmp_df = weather_df[weather_df.site_id==sid]\n missing = 100 / len(tmp_df) * tmp_df['wind_direction'].isnull().sum()\n tmp_df.groupby(tmp_df.index.hour)['wind_direction'].median().plot(ax=axes[row,col])\n if missing !=0:\n axes[row, col].set_title(f\"site {sid}, null:{missing :.2f}%\", fontsize=12, color=\"darkred\")\n else:\n axes[row, col].set_title(f\"site {sid}\", fontsize=12, color=\"darkgreen\")\n axes[row, col].set_xlabel(\"\")\nfig.suptitle(f\"wind_direction per hour of day\", fontsize=18)\nfig.subplots_adjust(top=0.92)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:14:54.838453Z","start_time":"2019-10-22T13:14:54.831631Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display(weather_df[weather_df.site_id ==1].cloud_coverage.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- **All buildings are on the northern hemisphere**, since seasonal temperature in general follows the same pattern\n- `cloud_coverage`: \n - much higher on site 12 than on all other sites\n - no values for site 7, 11\n- `sea_level_pressure`: \n - no values for site 5\n- `precip_depth`:\n - no values for site 1, 5, 12\n- [Change of wind direction on sites close to the ocean?](https://www.kaggle.com/blue07/eda-insights-on-weather-buildings)"},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# 2.d. Correlation of features"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"Summary of findings from this section:\n\n- Correlation of features differ substantially between sites\n- **Weak correlations to target variable**. Why is that?\n- Almost zero correlation from target to `precip_depth_1_hr` and `wind_direction`. Can probably be dropped."},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:15:00.149203Z","start_time":"2019-10-22T13:14:54.839678Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"lb = LabelEncoder()\ntrain_df.primary_use_lb = lb.fit_transform(train_df.primary_use)\ntrain_df.primary_use_lb = train_df.primary_use_lb.astype(\"int32\")\n# sample 100k to avoid crashing of kernel\ncorr_raw = train_df.sample(100_000).drop([\"timestamp\", \"primary_use\"], axis=1).astype(float)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:15:08.083142Z","start_time":"2019-10-22T13:15:00.151068Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# adding some temporary time related features\ncorr_raw[\"quarter\"] = train_df.timestamp.dt.quarter\ncorr_raw[\"quarter_start\"] = train_df.timestamp.dt.is_quarter_start\ncorr_raw[\"quarter_end\"] = train_df.timestamp.dt.is_quarter_end\ncorr_raw[\"month\"] = train_df.timestamp.dt.month\ncorr_raw[\"month_start\"] = train_df.timestamp.dt.is_month_start\ncorr_raw[\"month_end\"] = train_df.timestamp.dt.is_month_end\ncorr_raw[\"week\"] = train_df.timestamp.dt.week\ncorr_raw[\"dayofweek\"] = train_df.timestamp.dt.dayofweek\ncorr_raw[\"weekend\"] = corr_raw.dayofweek.apply(lambda x: True if x in [5, 6] else False)\ncorr_raw[\"day\"] = train_df.timestamp.dt.day\ncorr_raw[\"hour\"] = train_df.timestamp.dt.hour","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:16:07.738340Z","start_time":"2019-10-22T13:15:08.084829Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"corr = corr_raw.corr()\nplt.figure(figsize=(16,9));\ncorr[\"meter_reading\"].sort_values(ascending=True)[:-1].plot(kind=\"barh\")\nplt.title(\"Correlation of features to meter_reading\")\nplt.xlabel(\"Correlation to meter_reading\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:16:54.781494Z","start_time":"2019-10-22T13:16:07.739663Z"},"hidden":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# get correlation among all features with pandas .corr() function\ncorr = corr_raw.corr()\n# filter correlations less than 0.1\ncut_off = 0.1\ncorr = corr[(corr > cut_off) | (corr < -cut_off)]\n\nplt.subplots(figsize=(16,16));\nsns.heatmap(corr, cmap=\"RdBu\", square=True, annot=False, cbar_kws={\"shrink\": .6}, )\nplt.title(f\"Correlation of features greater than +/-{cut_off}\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-10-22T13:17:48.160550Z","start_time":"2019-10-22T13:16:54.782720Z"},"hidden":true,"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"## these heatmaps are inspired by this kernel: \n# https://www.kaggle.com/blue07/eda-insights-on-weather-buildings\n\nrows = 8\ncols = 2\n\nfig, axes = plt.subplots(nrows=rows, ncols=cols, sharex=True, sharey=True, figsize=(16, 64))\nfor sid in corr_raw.site_id.unique():\n row = int(sid/cols)\n col = int(sid%cols)\n tmp_df = corr_raw[corr_raw.site_id==sid]\n corr = tmp_df.corr()\n corr = corr[(corr > cut_off) | (corr < -cut_off)]\n sns.heatmap(corr, cmap=\"RdBu\", square=True, cbar=False, ax=axes[row,col])\n axes[row, col].set_xlabel(\"\")\n axes[row, col].set_title(f\"site {int(sid)}\", fontsize=12)\nfig.suptitle(f\"Correlation per site_id, greater than +/-{cut_off}\", fontsize=18)\nfig.subplots_adjust(top=0.965)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:04:29.865580Z","start_time":"2019-09-10T19:03:35.152398Z"},"heading_collapsed":true},"cell_type":"markdown","source":"# more to come..."},{"metadata":{"ExecuteTime":{"end_time":"2019-09-21T10:43:51.589442Z","start_time":"2019-09-21T10:43:51.582410Z"},"hidden":true},"cell_type":"markdown","source":"\n"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-21T14:37:51.975697Z","start_time":"2019-09-21T14:37:51.973773Z"},"heading_collapsed":true},"cell_type":"markdown","source":"# References"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:04:29.865580Z","start_time":"2019-09-10T19:03:35.152398Z"},"hidden":true},"cell_type":"markdown","source":"**Kaggle kernels**
\nhttps://www.kaggle.com/blue07/eda-insights-on-weather-buildings
\nhttps://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-ashrae
\nhttps://www.kaggle.com/jesucristo/starter-great-energy-predictor#ASHRAE---Great-Energy-Predictor-III
\nhttps://www.kaggle.com/drexpz/ashrae-lightgbm-prepro-and-data-visualization
\nhttps://www.kaggle.com/hmendonca/starter-eda-and-feature-selection-ashrae3
\n\n\n**Web**
\n[Jake VanderPlas's Data Science Handbook](https://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/00.00-Preface.ipynb)
\n\n\n**My other kernels**
\nhttps://www.kaggle.com/chmaxx/sklearn-pipeline-playground-for-12-classifiers
\nhttps://www.kaggle.com/chmaxx/extensive-data-exploration-modelling-python
\nhttps://www.kaggle.com/chmaxx/slim-data-cleaning-modelling-weighted-ensemble
\n\nhttps://www.kaggle.com/chmaxx/train-12-classifiers-with-one-line-of-code
\nhttps://www.kaggle.com/chmaxx/train-12-regressors-with-just-one-line-of-code
\n\n\n**Utility scripts**
\nhttps://www.kaggle.com/chmaxx/quick-regression
\nhttps://www.kaggle.com/chmaxx/quick-classification
"},{"metadata":{"ExecuteTime":{"end_time":"2019-09-10T19:14:27.552529Z","start_time":"2019-09-10T19:14:27.489365Z"},"hidden":true},"cell_type":"markdown","source":"---\n>*If you use parts of this notebook in your own scripts or kernels, please give credit (for example link back to this, upvote or send flowers). Thanks! \nAnd I very much appreciate your feedback or comments! Thanks for that too. 👍*\n\n---"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":false,"sideBar":true,"skip_h1_title":true,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"265px"},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":1}