{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHRAE - Great Energy Predictor III (Beginner)"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n* Q: How much does it cost to cool a skyscraper in the summer?\n* A: A lot! And not just in dollars, but in environmental impact.\n\nThankfully, significant investments are being made to improve building efficiencies to reduce costs and emissions. The question is, are the improvements working? That’s where you come in. Under pay-for-performance financing, the building owner makes payments based on the difference between their real energy consumption and what they would have used without any retrofits. The latter values have to come from a model. Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nIn this competition, you’ll develop accurate models of metered building energy usage in the following areas: chilled water, electric, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe. With better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n\n>About the Host\n\n![image](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1095143%2Ff9ab8963dea5e7c1716f47310daa96ab%2FASHRAE_Logo_25.jpg?generation=1570808142334850&alt=media)\n\nFounded in 1894, ASHRAE serves to advance the arts and sciences of heating, ventilation, air conditioning refrigeration and their allied fields. ASHRAE members represent building system design and industrial process professionals around the world. With over 54,000 members serving in 132 countries, ASHRAE supports research, standards writing, publishing and continuing education - shaping tomorrow’s built environment today.\n\nBanner photo by Federico Beccari on Unsplash"},{"metadata":{},"cell_type":"markdown","source":"# Data\n\nAssessing the value of energy efficiency improvements can be challenging as there's no way to truly know how much energy a building would have used without the improvements. The best we can do is to build counterfactual models. Once a building is overhauled the new (lower) energy consumption is compared against modeled values for the original building to calculate the savings from the retrofit. More accurate models could support better market incentives and enable lower cost financing.\n\nThis competition challenges you to build these counterfactual models across four energy types based on historic usage rates and observed weather. The dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.\n\n## Files\n\n### train.csv\n\n* building_id - Foreign key for the building metadata.\n* meter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, 3: hotwater}. Not every building has all meter types.\n* timestamp - When the measurement was taken\n* meter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n\n### building_meta.csv\n\n* site_id - Foreign key for the weather files.\n* building_id - Foreign key for training.csv\n* primary_use - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n* square_feet - Gross floor area of the building\n* year_built - Year building was opened\n* floor_count - Number of floors of the building\n\n### weather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.\n\n* site_id\n* air_temperature - Degrees Celsius\n* cloud_coverage - Portion of the sky covered in clouds, in oktas\n* dew_temperature - Degrees Celsius\n* precip_depth_1_hr - Millimeters\n* sea_level_pressure - Millibar/hectopascals\n* wind_direction - Compass direction (0-360)\n* wind_speed - Meters per second\n\n### test.csv\nThe submission files use row numbers for ID codes in order to save space on the file uploads. test.csv has no feature data; it exists so you can get your predictions into the correct order.\n* row_id - Row id for your submission file\n* building_id - Building id code\n* meter - The meter id code\n* timestamp - Timestamps for the test data period\n\n### sample_submission.csv\nA valid sample submission.\n\n* All floats in the solution file were truncated to four decimal places; we recommend you do the same to save space on your file upload.\n* There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored."},{"metadata":{},"cell_type":"markdown","source":"## Input data files from Kaggle directory."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\n#import numpy as np # linear algebra\n#import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n#df_sample_submission = pd.read_csv(\"/kaggle/input/ashrae-energy-prediction/sample_submission.csv\")\n#df_building_metadata = pd.read_csv(\"/kaggle/input/ashrae-energy-prediction/building_metadata.csv\")\n#df_weather_train = pd.read_csv(\"/kaggle/input/ashrae-energy-prediction/weather_train.csv\")\n#df_weather_test = pd.read_csv(\"/kaggle/input/ashrae-energy-prediction/weather_test.csv\")\n#df_train = pd.read_csv(\"/kaggle/input/ashrae-energy-prediction/train.csv\")\n#df_test = pd.read_csv(\"/kaggle/input/ashrae-energy-prediction/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"markdown","source":"%%time\n\ndf_train['timestamp'] = pd.to_datetime(df_train['timestamp'])\ndf_test['timestamp'] = pd.to_datetime(df_test['timestamp'])\ndf_weather_train['timestamp'] = pd.to_datetime(df_weather_train['timestamp'])\ndf_weather_test['timestamp'] = pd.to_datetime(df_weather_test['timestamp'])"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"%%time\n\ndf_train.to_feather('train.feather')\ndf_test.to_feather('test.feather')\ndf_weather_train.to_feather('weather_train.feather')\ndf_weather_test.to_feather('weather_test.feather')\ndf_building_metadata.to_feather('building_metadata.feather')\ndf_sample_submission.to_feather('sample_submission.feather')"},{"metadata":{},"cell_type":"markdown","source":"Base on [ASHRAE: feather format for fast loading](https://www.kaggle.com/corochann/ashrae-simple-lgbm-submission) kernel. 'test.csv' is big data and takes time to load. I would like to use the code from the notebook to convert competition data to feather format for fast pandas.DataFrame loading!\n1. Using Add Data in the top of right hand corner.\n2. Search 'ashrae-feather-format-for-fast-loading' and add it in kernel\n3. Using .read_feather in pandas to read the feather"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\ndf_train = pd.read_feather('/kaggle/input/ashrae-feather-format-for-fast-loading/train.feather')\ndf_weather_train = pd.read_feather('/kaggle/input/ashrae-feather-format-for-fast-loading/weather_train.feather')\ndf_test = pd.read_feather('/kaggle/input/ashrae-feather-format-for-fast-loading/test.feather')\ndf_weather_test = pd.read_feather('/kaggle/input/ashrae-feather-format-for-fast-loading/weather_test.feather')\ndf_building_metadata = pd.read_feather('/kaggle/input/ashrae-feather-format-for-fast-loading/building_metadata.feather')\ndf_sample_submission = pd.read_feather('/kaggle/input/ashrae-feather-format-for-fast-loading/sample_submission.feather')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analytics\nI would like to show you what is the data talking about by graph step by step in point form.\n* I will describe my process and result before I run the code.\n* I will do the very beginner Data Analytics and model prediction. There are just three features in my model and I will not consider about the weather and buliding_meta data.\n\n    1. In df_train, there are 2,0216,100 rows of data 4 columns as below:\n        1.   building_id = 1448 unique. It means that there are 1448 buildings in this training data set.\n        2.   meter = 4 Read as {0: electricity, 1: chilledwater, 2: steam, 3: hotwater} *Some buliding had all 4 meters but some doesnt.\n        3.   timestamp = for each building the meter_readings recorded once per hour and the recording period is a whole year of 2016. It means that for 1 single buliding_id, it had around 8,000 meter_reading records.\n        1.   meter_reading\n    2. I plot a Time series line graph for building_id with max and min of mean meter records for you to have a easy outlook.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndf_train[\"timestamp\"] = pd.to_datetime(df_train[\"timestamp\"])\ndf_test[\"timestamp\"] = pd.to_datetime(df_test[\"timestamp\"])\n\ndf_train = df_train.assign(hour=df_train.timestamp.dt.hour,\n               day=df_train.timestamp.dt.day,\n               month=df_train.timestamp.dt.month,\n               year=df_train.timestamp.dt.year)\n\ndf_test = df_test.assign(hour=df_test.timestamp.dt.hour,\n               day=df_test.timestamp.dt.day,\n               month=df_test.timestamp.dt.month,\n               year=df_test.timestamp.dt.year)\n\ndf_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"########################### Helpers\n#################################################################################\n## -------------------\n## Memory Reducer\n# :df pandas dataframe to reduce size             # type: pd.DataFrame()\n# :verbose                                        # type: bool\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n## -------------------","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n########################### Base check\n#################################################################################\ndo_not_convert = ['category','datetime64[ns]','object']\nfor df in [df_train, df_test, df_building_metadata, df_weather_train, df_weather_test,df_sample_submission]:\n    original = df.copy()\n    df = reduce_mem_usage(df)\n\n    for col in list(df):\n        if df[col].dtype.name not in do_not_convert:\n            if (df[col]-original[col]).sum()!=0:\n                df[col] = original[col]\n                print('Bad transformation', col)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"%%time\nmeter_mean = df_train.groupby(['building_id', 'timestamp']).meter_reading.transform('mean')\nlen(meter_mean)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"df_train['meter_mean'] = meter_mean"},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\n\nmylist = []\n\nfor i in range(0,3):\n    x = random.randint(1,1448)\n    mylist.append(x)\n\nprint(mylist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_polt = df_train[df_train['building_id'].isin(mylist)]\ndf_train_polt ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_polt.sort_index(axis = 1) \ndf_train_polt= df_train_polt.reset_index(drop=True)\ndf_train_polt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_1Day =df_train_polt[((df_train_polt.month == 6) & (df_train_polt.day == 30))]\nimport seaborn as sns\nsns.set(rc={'figure.figsize':(11.2,8.27)})\nsns.set(style=\"darkgrid\")\nax = sns.lineplot(x=\"timestamp\", y=\"meter_reading\",hue= 'building_id' ,data= df_1Day)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_1month =df_train_polt[((df_train_polt.month == 6))]\nimport seaborn as sns\nsns.set(rc={'figure.figsize':(11.2,8.27)})\nsns.set(style=\"darkgrid\")\nax = sns.lineplot(x=\"timestamp\", y=\"meter_reading\",hue= 'building_id' ,data= df_1month)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_polt['meter_reading'][((df_train_polt.building_id == 329)&(df_train_polt.hour == 12))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_1year =df_train_polt[((df_train_polt.hour == 12))]\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nsns.set(style=\"dark\", context=\"talk\")\nsns.set(rc={'figure.figsize':(20,8.27)})\nf, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(20,8.27))\nsns.barplot(x=df_train_polt['timestamp'][((df_train_polt.building_id == 329)&(df_train_polt.hour == 12))], y=df_train_polt['meter_reading'][((df_train_polt.building_id == 329)&(df_train_polt.hour == 12))], ax=ax1, color='black')\nax1.axhline(0, color=\"c\", clip_on=False)\nax1.set_ylabel(\"329\")\n\nsns.barplot(x=df_train_polt['timestamp'][((df_train_polt.building_id == 358)&(df_train_polt.hour == 12))], y=df_train_polt['meter_reading'][((df_train_polt.building_id == 358)&(df_train_polt.hour == 12))], ax=ax2, color='black')\nax2.axhline(0, color=\"c\", clip_on=False)\nax2.set_ylabel(\"358\")\n\nsns.barplot(x=df_train_polt['timestamp'][((df_train_polt.building_id == 1019)&(df_train_polt.hour == 12))], y=df_train_polt['meter_reading'][((df_train_polt.building_id == 1019)&(df_train_polt.hour == 12))], ax=ax3, color='black')\nax3.axhline(0, color=\"c\", clip_on=False)\nax3.set_ylabel(\"1019\")\n\nsns.despine(bottom=True)\nplt.setp(f.axes, yticks=[])\nplt.tight_layout(h_pad=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}