{"cells":[{"metadata":{},"cell_type":"markdown","source":"## COMPREHENSIVE DATA EXPLORATION WITH PYTHON\n[Crislânio Macêdo](https://medium.com/sapere-aude-tech) - December, 31th, 2019\n\nModel in ⚡🔌ASHRAE : Lgbm Simple FE: [⚡🔌ASHRAE : Lgbm Simple FE](https://www.kaggle.com/caesarlupum/ashrae-ligthgbm-simple-fe)\n\n----------"},{"metadata":{},"cell_type":"markdown","source":"![image](https://www.shell.com/energy-and-innovation/the-energy-future/scenarios/shell-scenario-sky/_jcr_content/pageCarouselImage.img.960.jpeg/1521983923121/future-of-energy-homepage-r.jpeg?imwidth=960)\n\n\n> About the Host\n# 🔌ASHRAE -Start Here: A GENTLE Introduction\n\n- 1. Introduction: ASHRAE - Great Energy Predictor III\n - 1.1. Data Description\n - 1.2. Evaluation Metric\n- 2. Imports\n- 3. Read in Data\n- 4. Glimpse of Data\n- 5. Reducing Memory Size\n- 6. Exploratory Data Analysis\n - 6.1. Examine the Distribution of the Target Column\n - 6.2. Missing data and zeros visualized\n - 6.3. Examine Missing Values\n - 6.4. Column Types\n - 6.5. Correlations\n- 7. Ploting\n- 8. Simple Single Series Analysis\n- 9. Outlier Distribution\n - 9.1. Group data in a daily basis\n - 9.2. Aggregate the data for buildings\n - 9.3. Identifying outliers\n- 10. Simple Feature Engineering and Modeling\n - 10.1. Building DF merge through concat\n - 10.2. Weather DF merge over concat\n- 11. ASHRAE - Data minification\n- 12. Some Features\n- 13. Encoding Variables\n- 14. Handling missing values\n- 15. ASHRAE Ligthgbm Simple Fe \n- ASHRAE Energy prediction - small summary\n- 17. Final\n\n\n\n\n "},{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction: ASHRAE - Great Energy Predictor III"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"\n### How much energy will a building consume?\n\n* Q: How much does it cost to cool a skyscraper in the summer?\n* A: A lot! And not just in dollars, but in environmental impact.\n\nThankfully, significant investments are being made to improve building efficiencies to reduce costs and emissions. So, are the improvements working? That’s where you come in. Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.\n\nIn this competition, you’ll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.\n\nWith better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n\n\n Founded in 1894, [ASHRAE](https://www.kaggle.com/orgs-under-maintenance) serves to advance the arts and sciences of heating, ventilation, air conditioning refrigeration and their allied fields. ASHRAE members represent building system design and industrial process professionals around the world. With over 54,000 members serving in 132 countries, ASHRAE supports research, standards writing, publishing and continuing education - shaping tomorrow’s built environment today.\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n

ASHRAE 90.1-2016, Energy Standard for Buildings - Review of Changes

\n\n\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Suppress warnings \nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nfrom IPython.display import HTML\n\n\nHTML('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## 1.1 Data\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"

\n\nAssessing the value of energy efficiency improvements can be challenging as there's no way to truly know how much energy a building would have used without the improvements. The best we can do is to build counterfactual models. Once a building is overhauled the new (lower) energy consumption is compared against modeled values for the original building to calculate the savings from the retrofit. More accurate models could support better market incentives and enable lower cost financing.\n\nThis competition challenges you to build these counterfactual models across four energy types based on historic usage rates and observed weather. The dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.\n

\n\n### Files\n> #### train.csv\n- building_id - Foreign key for the building metadata.\n- meter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. Not every building has all meter types.\n- timestamp - When the measurement was taken\n- meter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n> #### building_meta.csv\n- site_id - Foreign key for the weather files.\n- building_id - Foreign key for training.csv\n- primary_use - Indicator of the primary category of activities for the building based on [EnergyStar](https://www.energystar.gov/buildings/facility-owners-and-managers/existing-buildings/use-portfolio-manager/identify-your-property-type) property type definitions\n- square_feet - Gross floor area of the building\n- year_built - Year building was opened\n- floor_count - Number of floors of the building\n> #### weather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.\n\n- site_id\n- air_temperature - Degrees Celsius\n- cloud_coverage - Portion of the sky covered in clouds, in [oktas](https://en.wikipedia.org/wiki/Okta)\n- dew_temperature - Degrees Celsius\n- precip_depth_1_hr - Millimeters\n- sea_level_pressure - Millibar/hectopascals\n- wind_direction - Compass direction (0-360)\n- wind_speed - Meters per second\n> #### test.csv\nThe submission files use row numbers for ID codes in order to save space on the file uploads. test.csv has no feature data; it exists so you can get your predictions into the correct order.\n\nrow_id - Row id for your submission file\n- building_id - Building id code\n- meter - The meter id code\n- timestamp - Timestamps for the test data period\n> #### sample_submission.csv\nA valid sample submission.\n\n

\n\nAll floats in the solution file were truncated to four decimal places; we recommend you do the same to save space on your file upload.\nThere are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored. \n

"},{"metadata":{},"cell_type":"markdown","source":"\n## 1.2 Evaluation Metric"},{"metadata":{},"cell_type":"markdown","source":"We will be evaluated by the metirc `Root Mean Squared Logarithmic Error`.\n\nThe RMSLE is calculated as:\nThe RMSLE is calculated as\n\n$ ϵ=1n∑i= \\sqrt{ 1/n (log(pi+1)−log(ai+1))^2 } $\nWhere:\n\n- ϵ is the RMSLE value (score)\n- n is the total number of observations in the (public/private) data set,\n- pi is your prediction of target, and\n- ai is the actual target for i.\n- log(x) is the natural logarithm of x\n\nUnderstanding and optimizing your predictions for this evaluation metric is paramount for this compeition."},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n

If you find this kernel useful or interesting, please don't forget to upvote the kernel =)\n\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"# 2. Imports "},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"> We are using a typical data science stack: `numpy`, `pandas`, `sklearn`, `matplotlib`. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport gc\n\n# matplotlib and seaborn for plotting\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\nimport matplotlib.patches as patches\n\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\npd.set_option('max_columns', 150)\n\npy.init_notebook_mode(connected=True)\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nimport os,random, math, psutil, pickle ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Read in Data "},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"

\n\nFirst, we can list all the available data files. There are a total of 6 files: 1 main file for training (with target) 1 main file for testing (without the target), 1 example submission file, and 4 other files containing additional information about energy types based on historic usage rates and observed weather.\n

"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(os.listdir(\"../input/ashrae-energy-prediction/\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\nroot = '../input/ashrae-energy-prediction/'\ntrain_df = pd.read_csv(root + 'train.csv')\ntrain_df[\"timestamp\"] = pd.to_datetime(train_df[\"timestamp\"], format='%Y-%m-%d %H:%M:%S')\n\nweather_train_df = pd.read_csv(root + 'weather_train.csv')\ntest_df = pd.read_csv(root + 'test.csv')\nweather_test_df = pd.read_csv(root + 'weather_test.csv')\nbuilding_meta_df = pd.read_csv(root + 'building_metadata.csv')\nsample_submission = pd.read_csv(root + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Glimpse of Data"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Size of train_df data', train_df.shape)\nprint('Size of weather_train_df data', weather_train_df.shape)\nprint('Size of weather_test_df data', weather_test_df.shape)\nprint('Size of building_meta_df data', building_meta_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Reducing Memory Size"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"\n

\nIt is necessary that after using this code, carefully check the output results for each column.\n

"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"## Function to reduce the DF size\ndef reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

\nReducing memory\n

"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df = reduce_mem_usage(train_df)\ntest_df = reduce_mem_usage(test_df)\n\nweather_train_df = reduce_mem_usage(weather_train_df)\nweather_test_df = reduce_mem_usage(weather_test_df)\nbuilding_meta_df = reduce_mem_usage(building_meta_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n\n\n

🔓MEMORY USAGE AFTER COMPLETION:

\n

Mem. usage decreased to : 289.19 Mb (53.1% reduction)

\n

Mem. usage decreased to : 6.08 Mb (68.1% reduction)

\n

Mem. usage decreased to : 0.03 Mb (60.3% reduction)

\n\n\n\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"> #### train_df data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"train_df.columns.values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.columns.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> #### weather_train_df data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"weather_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"weather_train_df.columns.values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"weather_train_df.columns.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> #### weather_test_df data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"weather_test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"weather_test_df.columns.values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"weather_test_df.columns.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> #### building_meta_df data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"building_meta_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"building_meta_df.columns.values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"building_meta_df.columns.values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# 6. Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"\n

\nExploratory Data Analysis (EDA) is an open-ended process where we calculate statistics and make figures to find trends, anomalies, patterns, or relationships within the data. \n

"},{"metadata":{},"cell_type":"markdown","source":"## 6.1 Examine the Distribution of the Target Column\n\n\n

\nThe target is meter_reading - Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.

\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for key, d in train_df.groupby('meter_reading'):\n break\n d.head()\nplt.figure(figsize = (20,5))\nd['meter'].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.scatter(range(train_df.shape[0]), np.sort(train_df['meter_reading'].values))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('meter_reading', fontsize=12)\nplt.title(\"Target Distribution\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (15,5))\ntrain_df['meter_reading'].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df['meter_reading'].plot(kind='hist',\n bins=25,\n figsize=(15, 5),\n title='Distribution of Target Variable (meter_reading)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2 Missing data and zeros visualized\n\n\nlook that by ganfear:[ Missing data and zeros visualized](https://www.kaggle.com/ganfear/missing-data-and-zeros-visualized)\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n

\nGoal: for each building and meter pair, visualize where target is missing and where target is zero VS time.\n\n\n

\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Load data\ntrain = train_df.set_index(['timestamp'])\n\n# Plot missing values per building/meter\nf,a=plt.subplots(1,4,figsize=(20,30))\nfor meter in np.arange(4):\n df = train[train.meter==meter].copy().reset_index()\n df['timestamp'] = pd.to_timedelta(df.timestamp).dt.total_seconds() / 3600\n df['timestamp'] = df.timestamp.astype(int)\n df.timestamp -= df.timestamp.min()\n missmap = np.empty((1449, df.timestamp.max()+1))\n missmap.fill(np.nan)\n for l in df.values:\n if l[2]!=meter:continue\n missmap[int(l[1]), int(l[0])] = 0 if l[3]==0 else 1\n a[meter].set_title(f'meter {meter:d}')\n sns.heatmap(missmap, cmap='Paired', ax=a[meter], cbar=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Vertical blue lines may be suspicious\n\nLegend:\n* X axis: hours elapsed since Jan 1st 2016, for each of the 4 meter types\n* Y axis: building_id\n* Brown: meter reading available with non-zero value\n* Light blue: meter reading available with zero value\n* White: missing meter reading"},{"metadata":{},"cell_type":"markdown","source":"## 6.3 Examine Missing Values\n\n\n

\nNext we can look at the number and percentage of missing values in each column. \n\n

\n\n"},{"metadata":{},"cell_type":"markdown","source":"### checking missing data for train_df"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"total = train_df.isnull().sum().sort_values(ascending = False)\npercent = (train_df.isnull().sum()/train_df.isnull().count()*100).sort_values(ascending = False)\nmissing__train_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing__train_data.head(4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### checking missing data for weather_train_df"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# checking missing data\ntotal = weather_train_df.isnull().sum().sort_values(ascending = False)\npercent = (weather_train_df.isnull().sum()/weather_train_df.isnull().count()*100).sort_values(ascending = False)\nmissing_weather_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_weather_data.head(9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### checking missing data for weather_test_df"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# checking missing data\ntotal = weather_test_df.isnull().sum().sort_values(ascending = False)\npercent = (weather_test_df.isnull().sum()/weather_test_df.isnull().count()*100).sort_values(ascending = False)\nmissing_weather_test_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_weather_test_data.head(9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### checking missing data for building_meta_df"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# checking missing data\ntotal = building_meta_df.isnull().sum().sort_values(ascending = False)\npercent = (building_meta_df.isnull().sum()/building_meta_df.isnull().count()*100).sort_values(ascending = False)\nmissing_building_meta_df = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_building_meta_df.head(6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.4 Column Types\n\n\nLet's look at the number of columns of each data type. `int64` and `float64` are numeric variables ([which can be either discrete or continuous](https://stats.stackexchange.com/questions/206/what-is-the-difference-between-discrete-data-and-continuous-data)). `object` columns contain strings and are [categorical features.](http://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/regression/supporting-topics/basics/what-are-categorical-discrete-and-continuous-variables/) . "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Number of each type of column\ntrain_df.dtypes.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Number of unique classes in each object column\ntrain_df.select_dtypes('object').apply(pd.Series.nunique, axis = 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.5 Correlations\n\nNow that we have dealt with the categorical variables and the outliers, let's continue with the EDA. One way to try and understand the data is by looking for correlations between the features and the target. We can calculate the Pearson correlation coefficient between every variable and the target using the `.corr` dataframe method.\n\nThe correlation coefficient is not the greatest method to represent \"relevance\" of a feature, but it does give us an idea of possible relationships within the data. Some [general interpretations of the absolute value of the correlation coefficent](http://www.statstutor.ac.uk/resources/uploaded/pearsons.pdf) are:\n\n\n* .00-.19 “**very weak**”\n* .20-.39 “**weak**”\n* .40-.59 “**moderate**”\n* .60-.79 “**strong**”\n* .80-1.0 “**very strong**”\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Find correlations with the target and sort\ncorrelations = train_df.corr()['meter_reading'].sort_values()\n\n# Display correlations\nprint('Most Positive Correlations:\\n', correlations.tail(15))\nprint('\\nMost Negative Correlations:\\n', correlations.head(15))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"corrs = train_df.corr()\ncorrs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (20, 8))\n\n# Heatmap of correlations\nsns.heatmap(corrs, cmap = plt.cm.RdYlBu_r, vmin = -0.25, annot = True, vmax = 0.6)\nplt.title('Correlation Heatmap');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Ploting"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.building_id.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Let's check the distribution of target value in train dataset"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df['meter_reading'].hist(figsize=(16, 8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_dist_col(column):\n '''plot dist curves for train and test weather data for the given column name'''\n fig, ax = plt.subplots(figsize=(10, 10))\n sns.distplot(weather_train_df[column].dropna(), color='green', ax=ax).set_title(column, fontsize=16)\n sns.distplot(weather_test_df[column].dropna(), color='purple', ax=ax).set_title(column, fontsize=16)\n plt.xlabel(column, fontsize=15)\n plt.legend(['train', 'test'])\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for air_temperature"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('air_temperature')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for cloud_coverage"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('cloud_coverage')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for dew_temperature"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('dew_temperature')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for precip_depth_1_hr"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('precip_depth_1_hr')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for sea_level_pressure"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('sea_level_pressure')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for wind_direction"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('wind_direction')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> plot dist. curves for train and test weather data for wind_speed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plot_dist_col('wind_speed')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8. Simple Single Series Analysis\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from statsmodels.tsa.seasonal import seasonal_decompose","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ts=train_df.groupby([\"timestamp\"])[\"meter_reading\"].sum()\nts.astype('float')\nplt.figure(figsize=(16,8))\nplt.title('meter_reading')\nplt.xlabel('timestamp')\nplt.ylabel('meter_reading')\nplt.plot(ts);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(16,6))\nplt.plot(ts.rolling(window=12,center=False).mean(),label='Rolling Mean');\nplt.plot(ts.rolling(window=12,center=False).std(),label='Rolling sd');\nplt.legend();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import statsmodels.api as sm\n# multiplicative\nres = sm.tsa.seasonal_decompose(ts.values,freq=12,model=\"multiplicative\")\nfig = res.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Additive model\nres = sm.tsa.seasonal_decompose(ts.values,freq=12,model=\"additive\")\nfig = res.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 9. Outlier Distribution\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"y_mean_time = train_df.groupby('timestamp').meter_reading.mean()\ny_mean_time.plot(figsize=(20, 8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"y_mean_time.rolling(window=10).std().plot(figsize=(20, 8))\nax = plt.axhline(y=0.009, color='red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"y_mean_time.rolling(window=10).std().plot(figsize=(20, 8))\nplt.axhline(y=0.009, color='red')\nplt.axvspan(0, 905, color='green', alpha=0.1)\nplt.axvspan(906, 1505, color='red', alpha=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.1. Group data in a daily basis\n"},{"metadata":{},"cell_type":"markdown","source":"Look this, by [juanmah](https://www.kaggle.com/juanmah) : [AHRAE Outliers](https://www.kaggle.com/juanmah/ashrae-outliers/notebook) (Upvote this!) "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df['meter'] = pd.Categorical(train_df['meter']).rename_categories({0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'})\ndaily_train = train_df.copy()\ndaily_train['date'] = daily_train['timestamp'].dt.date\ndaily_train = daily_train.groupby(['date', 'building_id', 'meter']).sum()\ndaily_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9.2. Aggregate the data for buildings\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_agg = daily_train.groupby(['date', 'meter']).agg(['sum', 'mean', 'idxmax', 'max'])\ndaily_train_agg = daily_train_agg.reset_index()\nlevel_0 = daily_train_agg.columns.droplevel(0)\nlevel_1 = daily_train_agg.columns.droplevel(1)\nlevel_0 = ['' if x == '' else '-' + x for x in level_0]\ndaily_train_agg.columns = level_1 + level_0\ndaily_train_agg.rename_axis(None, axis=1)\ndaily_train_agg.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Some plots"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig_total = px.line(daily_train_agg, x='date', y='meter_reading-sum', color='meter', render_mode='svg')\nfig_total.update_layout(title='Total kWh per energy aspect')\nfig_total.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sum, facetted for each energy aspect, shows some aberrant values."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig_maximum = px.line(daily_train_agg, x='date', y='meter_reading-max', color='meter', render_mode='svg')\nfig_maximum.update_layout(title='Maximum kWh value per energy aspect')\nfig_maximum.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the max value for each day, and for each energy aspect, shows that only a single building (for day and energy aspect) is causing the aberrant peaks"},{"metadata":{},"cell_type":"markdown","source":"## 9.3. Identifying outliers\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_agg['building_id_max'] = [x[1] for x in daily_train_agg['meter_reading-idxmax']]\ndaily_train_agg.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def show_building(building, energy_aspects=None):\n fig = px.line(daily_train.loc[(slice(None), building, slice(None)), :].reset_index(),\n x='date',\n y='meter_reading',\n color='meter',\n render_mode='svg')\n if energy_aspects:\n if 'electricity' not in energy_aspects:\n fig['data'][0].visible = 'legendonly'\n if 'chilledwater' not in energy_aspects:\n fig['data'][1].visible = 'legendonly'\n if 'steam' not in energy_aspects:\n fig['data'][2].visible = 'legendonly'\n if 'hotwater' not in energy_aspects:\n fig['data'][3].visible = 'legendonly'\n fig.update_layout(title='Building ID: {}'.format(building)) \n fig.show()\n display(building_metadata[building_metadata['building_id']==building])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Electricity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum electricity consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'electricity']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 6 buildings."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_electricity = daily_train_agg[daily_train_agg['meter']=='electricity'].copy()\ndaily_train_electricity['building_id_max'] = pd.Categorical(daily_train_electricity['building_id_max'])\nfig_daily_electricity = px.scatter(daily_train_electricity,\n x='date',\n y='meter_reading-max',\n color='building_id_max',\n render_mode='svg')\nfig_daily_electricity.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_electricity.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Chilledwater"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum chilledwater consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'chilledwater']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 10 buildings."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_chilledwater = daily_train_agg[daily_train_agg['meter']=='chilledwater'].copy()\ndaily_train_chilledwater['building_id_max'] = pd.Categorical(daily_train_chilledwater['building_id_max'])\nfig_daily_chilledwater = px.scatter(daily_train_chilledwater,\n x='date',\n y='meter_reading-max', \n color='building_id_max', \n render_mode='svg')\nfig_daily_chilledwater.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_chilledwater.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Steam"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum steam consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'steam']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 4 buildings."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_steam = daily_train_agg[daily_train_agg['meter']=='steam'].copy()\ndaily_train_steam['building_id_max'] = pd.Categorical(daily_train_steam['building_id_max'])\nfig_daily_steam = px.scatter(daily_train_steam,\n x='date',\n y='meter_reading-max',\n color='building_id_max',\n render_mode='svg')\nfig_daily_steam.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_steam.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Hotwate"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum hotwater consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'hotwater']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 7 buildings. Practically, two of them"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_hotwater = daily_train_agg[daily_train_agg['meter']=='hotwater'].copy()\ndaily_train_hotwater['building_id_max'] = pd.Categorical(daily_train_hotwater['building_id_max'])\nfig_daily_hotwater = px.scatter(daily_train_hotwater,\n x='date',\n y='meter_reading-max',\n color='building_id_max',\n render_mode='svg')\nfig_daily_hotwater.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_hotwater.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.3 Examine Missing Values\n\n\n

\n\n Taking only the buildings that consume more than the others, could be seen that there are a lot of measure scale errors.\n The error could be:\n\n The meter is not configured correctly. E.g., a bad voltage or current primary to secondary ratio.\n The software has not the units configured correctly. E.g., MJ/kg for steam.\n The software has not the decimal digits configured correctly.\n Using a power variable instead of an energy one.\n The measure could be done with an unique meter, or the sum of several of them.\n\n Some changes over time, values go to zero or the scale is changed, indicates that some buildings have more than one meter. One error in one meter and the overall measure is garbage.\n\n This notebook has only analised the outliers that influence the maximum consumption in a daily basis. This is only the tip of the iceberg. A sound analysis should be done to detect and correct these outliers.\n\n A solution to avoid scale errors is to normalize the values from 0 to 1, for each building and for each energy aspect.\n

\n\n"},{"metadata":{},"cell_type":"markdown","source":"# 10 Simple Feature Engineering and Modeling \n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 10.1 Building DF merge through concat \n\n"},{"metadata":{},"cell_type":"markdown","source":"\n> - Convert timestamp \n> - Convert Strings to category\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df['timestamp'] = pd.to_datetime(train_df['timestamp'])\ntest_df['timestamp'] = pd.to_datetime(test_df['timestamp'])\nweather_train_df['timestamp'] = pd.to_datetime(weather_train_df['timestamp'])\nweather_test_df['timestamp'] = pd.to_datetime(weather_test_df['timestamp'])\n \nbuilding_meta_df['primary_use'] = building_meta_df['primary_use'].astype('category')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"temp_df = train_df[['building_id']]\ntemp_df = temp_df.merge(building_meta_df, on=['building_id'], how='left')\ndel temp_df['building_id']\ntrain_df = pd.concat([train_df, temp_df], axis=1)\n\ntemp_df = test_df[['building_id']]\ntemp_df = temp_df.merge(building_meta_df, on=['building_id'], how='left')\n\ndel temp_df['building_id']\ntest_df = pd.concat([test_df, temp_df], axis=1)\ndel temp_df, building_meta_df\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## 10.2 Weather DF merge over concat \n\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"temp_df = train_df[['site_id','timestamp']]\ntemp_df = temp_df.merge(weather_train_df, on=['site_id','timestamp'], how='left')\n\ndel temp_df['site_id'], temp_df['timestamp']\ntrain_df = pd.concat([train_df, temp_df], axis=1)\n\ntemp_df = test_df[['site_id','timestamp']]\ntemp_df = temp_df.merge(weather_test_df, on=['site_id','timestamp'], how='left')\n\ndel temp_df['site_id'], temp_df['timestamp']\ntest_df = pd.concat([test_df, temp_df], axis=1)\n\ndel temp_df, weather_train_df, weather_test_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 11. ASHRAE - Data minification\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"> Use can use train_df.pkl, test_df.pkl for FE, FS for your baseline_predict"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.to_pickle('train_df.pkl')\ntest_df.to_pickle('test_df.pkl')\n \ndel train_df, test_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df = pd.read_pickle('train_df.pkl')\ntest_df = pd.read_pickle('test_df.pkl')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# 12. Some Features\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"Kaggle competitions are won by feature engineering\n\n> ### Stanford Professor Andrew Ng accurately said, “…applied machine learning is basically feature engineering.”\nSee this: https://blog.featurelabs.com/secret-to-data-science-success/"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df['age'] = train_df['year_built'].max() - train_df['year_built'] + 1\ntest_df['age'] = test_df['year_built'].max() - test_df['year_built'] + 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# 13. Encoding Variables\n\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"\nBefore we go any further, we need to deal with pesky categorical variables. A machine learning model unfortunately cannot deal with categorical variables (except for some models such as [LightGBM](Encoding Variables). Therefore, we have to find a way to encode (represent) these variables as numbers before handing them off to the model. There are two main ways to carry out this process:\n\nYou can see [this](https://www.kaggle.com/alexisbcook/categorical-variables):\n\n\n\n## Label Encoding:\nLabel encoding assigns each unique value to a different integer.\n\n![](https://raw.githubusercontent.com/WillKoehrsen/Machine-Learning-Projects/master/label_encoding.png)\n\nThis approach assumes an ordering of the categories: \"Never\" (0) < \"Rarely\" (1) < \"Most days\" (2) < \"Every day\" (3).\n\nThis assumption makes sense in this example, because there is an indisputable ranking to the categories. Not all categorical variables have a clear ordering in the values, but we refer to those that do as ordinal variables. For tree-based models (like decision trees and random forests), you can expect label encoding to work well with ordinal variables."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"le = LabelEncoder()\n# train_df['primary_use'] = train_df['primary_use'].astype(str)\ntrain_df['primary_use'] = le.fit_transform(train_df['primary_use']).astype(np.int8)\n\n# test_df['primary_use'] = test_df['primary_use'].astype(str)\ntest_df['primary_use'] = le.fit_transform(test_df['primary_use']).astype(np.int8)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# 14. Handling missing values\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"### Types of missing data\n> It is helpful to create and test hypotheses around why the data would be potentially missing, it is because the sensor recording the data disconnected from the server, the person feeding paper forms into the spreadsheet missed it or is the data missing for a particular category of rows.\n\nTo streamline this though process it is useful to know the 3 categories in which missing data can be classified into:\n\n* Missing Completely at Random (**MCAR**)\n* Missing at Random (**MAR**)\n* Missing Not at Random (**MNAR**)"},{"metadata":{},"cell_type":"markdown","source":"Time series imputation\n1. Non-time-series specific method\n1. mean imputation\n1. median imputation\n1. mode imputation\n1. calcucate the appropriate measure and replace NAs with the values.\n#### appropriate for stationary time series, for example, white noise data\n\n1. Random sample imputation\nreplace missing values with observations randomly selected from the remaining (either of it or just some section of it)\n#### It is not likely to work well unless the random select is carefully chosen.\n\n\nTime-Series specific method\n* Last observation carried forward **(LOCF)**\n* Next observation carried backward **(NOCB)**\n* Linear interpolation\n* Spline interpolation \n\n#### These methods rely on the assumption that adjacent observations are similar to one another. These methods do not work well when this assumption is not valid, especially when the presence of strong seasonality.\n\nThis kernel it's helpfull for this: https://www.kaggle.com/juejuewang/handle-missing-values-in-time-series-for-beginners\n"},{"metadata":{},"cell_type":"markdown","source":"Manually dealing with missing values will often improve model performance.\n\nOur approach we input **fill NaN** = -999 just for the 4 features with most missing values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\ntrain_df['floor_count'] = train_df['floor_count'].fillna(-999).astype(np.int16)\ntest_df['floor_count'] = test_df['floor_count'].fillna(-999).astype(np.int16)\n\ntrain_df['year_built'] = train_df['year_built'].fillna(-999).astype(np.int16)\ntest_df['year_built'] = test_df['year_built'].fillna(-999).astype(np.int16)\n\ntrain_df['age'] = train_df['age'].fillna(-999).astype(np.int16)\ntest_df['age'] = test_df['age'].fillna(-999).astype(np.int16)\n\ntrain_df['cloud_coverage'] = train_df['cloud_coverage'].fillna(-999).astype(np.int16)\ntest_df['cloud_coverage'] = test_df['cloud_coverage'].fillna(-999).astype(np.int16) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Some datetime features"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['month_datetime'] = train_df['timestamp'].dt.month.astype(np.int8)\ntrain_df['weekofyear_datetime'] = train_df['timestamp'].dt.weekofyear.astype(np.int8)\ntrain_df['dayofyear_datetime'] = train_df['timestamp'].dt.dayofyear.astype(np.int16)\n \ntrain_df['hour_datetime'] = train_df['timestamp'].dt.hour.astype(np.int8) \ntrain_df['day_week'] = train_df['timestamp'].dt.dayofweek.astype(np.int8)\ntrain_df['day_month_datetime'] = train_df['timestamp'].dt.day.astype(np.int8)\ntrain_df['week_month_datetime'] = train_df['timestamp'].dt.day/7\ntrain_df['week_month_datetime'] = train_df['week_month_datetime'].apply(lambda x: math.ceil(x)).astype(np.int8)\n \ntrain_df['year_built'] = train_df['year_built']-1900\ntrain_df['square_feet'] = np.log(train_df['square_feet'])\n \ntest_df['month_datetime'] = test_df['timestamp'].dt.month.astype(np.int8)\ntest_df['weekofyear_datetime'] = test_df['timestamp'].dt.weekofyear.astype(np.int8)\ntest_df['dayofyear_datetime'] = test_df['timestamp'].dt.dayofyear.astype(np.int16)\n \ntest_df['hour_datetime'] = test_df['timestamp'].dt.hour.astype(np.int8)\ntest_df['day_week'] = test_df['timestamp'].dt.dayofweek.astype(np.int8)\ntest_df['day_month_datetime'] = test_df['timestamp'].dt.day.astype(np.int8)\ntest_df['week_month_datetime'] = test_df['timestamp'].dt.day/7\ntest_df['week_month_datetime'] = test_df['week_month_datetime'].apply(lambda x: math.ceil(x)).astype(np.int8)\n \ntest_df['year_built'] = test_df['year_built']-1900\ntest_df['square_feet'] = np.log(test_df['square_feet'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"# Model in ⚡🔌ASHRAE : Lgbm Simple FE \n> ## See this: [⚡🔌ASHRAE : Lgbm Simple FE](https://www.kaggle.com/caesarlupum/ashrae-ligthgbm-simple-fe) "},{"metadata":{},"cell_type":"markdown","source":" > ⚡ Please, you can use parts of this notebook in your own scripts or kernels, no problem, but please give credit (for example link back to this, see this...)"},{"metadata":{},"cell_type":"markdown","source":"# ASHRAE Energy prediction - summary\n"},{"metadata":{},"cell_type":"markdown","source":"Top"},{"metadata":{},"cell_type":"markdown","source":"\n\n\n

ASHRAE Standard 90.1 2010, Part III -- HVAC Provisions

\n\n\n\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"HTML('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n\n

ASHRAE -- What It Is and Where It Is Going

\n\n\n\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"HTML('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ## General findings\n\nPublished Articles on Energy Consumtion Prediction\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113080#latest-665324\n\nGeo location:\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/115040#latest-667889\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/115698#latest-667385\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/115698#latest-667385\n\nOutliers:\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113254\n\nHolidays:\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113286\n\nMetric:\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113064#latest-663076\n\n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"\n\n\n\n

Don't hesitate to give your suggestions in the comment section

\n

Remember the upvote button is next to the fork button, and it's free too! ;)

\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"# 17. Final"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}