{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Introduction: ASHRAE - Great Energy Predictor III\n\n## Data    \n\nThe dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.\n\n### Files\n> #### train.csv\n- building_id - Foreign key for the building metadata.\n- meter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. Not every building has all meter types.\n- timestamp\n- meter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n\n\n> #### building_meta.csv\n- site_id - Foreign key for the weather files.\n- building_id - Foreign key for training.csv\n- primary_use - Indicator of the primary category of activities for the building based on [EnergyStar](https://www.energystar.gov/buildings/facility-owners-and-managers/existing-buildings/use-portfolio-manager/identify-your-property-type) property type definitions\n- square_feet - Gross floor area of the building\n- year_built - Year building was opened\n- floor_count - Number of floors of the building\n\n\n> #### weather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.\n\n- site_id\n- air_temperature - Degrees Celsius\n- cloud_coverage - Portion of the sky covered in clouds, in [oktas](https://en.wikipedia.org/wiki/Okta)\n- dew_temperature - Degrees Celsius\n- precip_depth_1_hr - Millimeters\n- sea_level_pressure - Millibar/hectopascals\n- wind_direction - Compass direction (0-360)\n- wind_speed - Meters per second\n\n\n> #### test.csv\nThe submission files use row numbers for ID codes in order to save space on the file uploads. test.csv has no feature data; it exists so you can get your predictions into the correct order.\n- row_id - Row id for your submission file\n- building_id - Building id code\n- meter - The meter id code\n- timestamp - Timestamps for the test data period\n\nAll floats in the solution file were truncated to four decimal places; we recommend you do the same to save space on your file upload. There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored. \n"},{"metadata":{},"cell_type":"markdown","source":"## Evaluation Metric\n\nWe will be evaluated by the metirc `Root Mean Squared Logarithmic Error`.\n\nThe RMSLE is calculated as:\n\n$$ \\epsilon = \\sqrt { \\frac{1}{n} \\sum_{i=1}^{n}{(log(p_i + 1) - log(a_i +1)) ^ 2} }$$ \n\nWhere:\n\n- $ \\epsilon $ is the RMSLE value (score)\n- $ n $ is the total number of observations in the (public/private) data set,\n- $ p_i $ is your prediction of target, and\n- $ a_i $ is the actual target for $i$.\n- $log(x)$ is the natural logarithm of $x$"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.offline as offline\noffline.init_notebook_mode() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input/ashrae-energy-prediction/\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nroot = '../input/ashrae-energy-prediction/'\ntrain_df = pd.read_csv(root + 'train.csv')\nweather_train_df = pd.read_csv(root + 'weather_train.csv')\ntest_df = pd.read_csv(root + 'test.csv')\nweather_test_df = pd.read_csv(root + 'weather_test.csv')\nbuilding_meta_df = pd.read_csv(root + 'building_metadata.csv')\nsample_submission = pd.read_csv(root + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Glimpse of Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Size of train_df data', train_df.shape)\nprint('Size of weather_train_df data', weather_train_df.shape)\nprint('Size of weather_test_df data', weather_test_df.shape)\nprint('Size of building_meta_df data', building_meta_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> #### weather_test_df data"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meta_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"### Examine the Distribution of the Target Column\n\nThe target is `meter_reading` - Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n\nMeter readings are for differnt meters, read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. Not every building has all meter types."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['meter'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Want to see meter reading by meter type\ntrain_df.groupby('meter')['meter_reading'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that meter readings by meter type are very different. Steam in particular, lives on a much higher range than all other meters."},{"metadata":{},"cell_type":"markdown","source":"## Examine Missing Values\n\nNext we can look at the number and percentage of missing values in each column. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = train_df.isnull().sum().sort_values(ascending = False)\npercent = (train_df.isnull().sum()/train_df.isnull().count()*100).sort_values(ascending = False)\nmissing__train_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing__train_data.head(4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = weather_train_df.isnull().sum().sort_values(ascending = False)\npercent = (weather_train_df.isnull().sum()/weather_train_df.isnull().count()*100).sort_values(ascending = False)\nmissing_weather_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_weather_data.head(9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = weather_test_df.isnull().sum().sort_values(ascending = False)\npercent = (weather_test_df.isnull().sum()/weather_test_df.isnull().count()*100).sort_values(ascending = False)\nmissing_weather_test_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_weather_test_data.head(9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = building_meta_df.isnull().sum().sort_values(ascending = False)\npercent = (building_meta_df.isnull().sum()/building_meta_df.isnull().count()*100).sort_values(ascending = False)\nmissing_building_meta_df  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_building_meta_df.head(6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have no missing data for meter readings, but a lot of missing data for:\n\n* building: floor_count (75%)\n\n* building: year_built (53%)\n\n* weather: cloud_coverage (51%)\n\n* weather: precip_depth_1_hr (35%)\n\n* weather: sea_level_pressure (8%)\n\n* weather: wind_direction (5%)"},{"metadata":{},"cell_type":"markdown","source":"## Column Types\n\nLet's look at the number of columns of each data type. `int64` and `float64` are numeric variables ([which can be either discrete or continuous](https://stats.stackexchange.com/questions/206/what-is-the-difference-between-discrete-data-and-continuous-data)). `object` columns contain strings and are  [categorical features.](http://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/regression/supporting-topics/basics/what-are-categorical-discrete-and-continuous-variables/) . "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of each type of column\ntrain_df.dtypes.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train_df.dtypes.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meta_df.dtypes.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of unique classes in each object column\ntrain_df.select_dtypes('object').apply(pd.Series.nunique, axis = 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Correlations\n\nNow that we have dealt with the categorical variables and the outliers, let's continue with the EDA. One way to try and understand the data is by looking for correlations between the features and the target. We can calculate the Pearson correlation coefficient between every variable and the target using the `.corr` dataframe method.\n\nThe correlation coefficient is not the greatest method to represent \"relevance\" of a feature, but it does give us an idea of possible relationships within the data. Some [general interpretations of the absolute value of the correlation coefficent](http://www.statstutor.ac.uk/resources/uploaded/pearsons.pdf) are:\n\n\n* .00-.19 “very weak”\n*  .20-.39 “weak”\n*  .40-.59 “moderate”\n*  .60-.79 “strong”\n* .80-1.0 “very strong”\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Find correlations with the target and sort\ncorrelations = train_df.corr()['meter_reading'].sort_values()\n\n# Display correlations\nprint('Most Positive Correlations:\\n', correlations.tail(15))\nprint('\\nMost Negative Correlations:\\n', correlations.head(15))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}