{"cells":[{"metadata":{},"cell_type":"markdown","source":"Q: How much does it cost to cool a skyscraper in the summer?\nA: A lot! And not just in dollars, but in environmental impact.\n\nThankfully, significant investments are being made to improve building efficiencies to reduce costs and emissions. The question is, are the improvements working? That’s where you come in. Under pay-for-performance financing, the building owner makes payments based on the difference between their real energy consumption and what they would have used without any retrofits. The latter values have to come from a model. Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nIn this competition, you’ll develop accurate models of metered building energy usage in the following areas: chilled water, electric, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe. With better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n\nAbout the Host\n\n\nFounded in 1894, ASHRAE serves to advance the arts and sciences of heating, ventilation, air conditioning refrigeration and their allied fields. ASHRAE members represent building system design and industrial process professionals around the world. With over 54,000 members serving in 132 countries, ASHRAE supports research, standards writing, publishing and continuing education - shaping tomorrow’s built environment today.\n\nBanner photo by Federico Beccari on Unsplash"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Importing necessary files**"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"build_meta = pd.read_csv(\"../input/ashrae-energy-prediction/building_metadata.csv\")\ntest = pd.read_csv(\"../input/ashrae-energy-prediction/test.csv\")\ntrain_df = pd.read_csv(\"../input/ashrae-energy-prediction/train.csv\")\nweat_train = pd.read_csv(\"../input/ashrae-energy-prediction/weather_train.csv\")\nweat_test = pd.read_csv(\"../input/ashrae-energy-prediction/weather_test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Function to reduce the DF size\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Checking shapes of all the files**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Shape of building_metadata is: \", build_meta.shape)\nprint(\"Shape of train is: \", train_df.shape)\nprint(\"Shape of weat_train is: \", weat_train.shape)\nprint(\"Shape of test is: \", test.shape)\nprint(\"Shape of weat_test is: \", weat_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Submission**"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(\"../input/ashrae-energy-prediction/sample_submission.csv\")\nprint(\"Shape of submission is: \", sub.shape)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Taking a quick look at all the files**"},{"metadata":{"trusted":true},"cell_type":"code","source":"build_meta.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weat_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weat_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Merging Train**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Reduce Memory usage****"},{"metadata":{"trusted":true},"cell_type":"code","source":"build_meta = reduce_mem_usage(build_meta)\ntrain_df = reduce_mem_usage(train_df)\nweat_train = reduce_mem_usage(weat_train)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del test, sub, weat_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.merge(train_df, build_meta, on=\"building_id\", how=\"left\")\nprint(\"Shape after merging train and build train is:\", train.shape)\n\ndel train_df\ngc.collect()\n\ntrain = pd.merge(train, weat_train, on=['site_id','timestamp'], how=\"left\")\nprint(\"Shape after merging all train data is:\", train.shape)\n\ndel weat_train\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Before we move further let's visualize**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\ndef bar_plot(feature, df):\n    sns.set(style=\"darkgrid\")\n    ax = sns.countplot(x=feature , data=df)\n    \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualizing train**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> From above we can see that there is a lot of nan values present which we have to carefully handle"},{"metadata":{},"cell_type":"markdown","source":"**Label (meter_reading)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#bar_plot(\"meter_reading\", train)\nprint(\"total different value of meter_reading is:\", train.meter_reading.value_counts().shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**site_id**"},{"metadata":{"trusted":true},"cell_type":"code","source":"bar_plot(\"site_id\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. From the above plot we can conclude that there are total 16 different site_id\n2. id 2,3,9,13,14,15 has appeared most number of times\n3. id 7,11,12 has appeared least number of times"},{"metadata":{},"cell_type":"markdown","source":"**meter**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of meter is:\", train.meter.value_counts())\n\nbar_plot(\"meter\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. From the above plot we can conclude that there are total 4 different meter category\n2. 0 has appeared most number of times\n3. 3 has appeared least number of times\n"},{"metadata":{},"cell_type":"markdown","source":"**primary_use**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of primary use is:\", train.primary_use.value_counts())\nbar_plot(\"primary_use\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. this is a categorical variable\n2. From above we can see that office has maximum number of count and Religious worship has lowest"},{"metadata":{},"cell_type":"markdown","source":"**square feet**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of square feet is:\", train.square_feet.value_counts().shape)\nbar_plot(\"square_feet\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> there are total 1397 different types of square feet"},{"metadata":{},"cell_type":"markdown","source":"**year_build**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of year built is:\", train.year_built.value_counts())\nbar_plot(\"year_built\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> year 1994 has maximum built"},{"metadata":{},"cell_type":"markdown","source":"**floor count**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of floor count is:\", train.floor_count.value_counts())\nbar_plot(\"floor_count\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. From above we can conclude that\n2. floor 1 has maximum number of count and floor 14 and 16 has least"},{"metadata":{},"cell_type":"markdown","source":"**air_temperature**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of air temperature is:\", train.air_temperature.value_counts().shape)\nbar_plot(\"air_temperature\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> from above we can see that there are total 619 different temperature"},{"metadata":{},"cell_type":"markdown","source":"**cloud_coverage**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of cloud coverage is:\", train.cloud_coverage.value_counts())\nbar_plot(\"cloud_coverage\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> from above we can see that there are 10 different types of cloud coverage ranging from 0 to 9 and 0 has occured most of the time "},{"metadata":{},"cell_type":"markdown","source":"**dew temperature **"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of dew temperature is:\", train.dew_temperature.value_counts().shape)\nbar_plot(\"dew_temperature\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> there are 522 different dew temperture categories"},{"metadata":{},"cell_type":"markdown","source":"**precip_depth_1_hr**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of precip_depth_1_hr is:\", train.precip_depth_1_hr.value_counts().shape)\nbar_plot(\"precip_depth_1_hr\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> this too is a categorical variable with 128 different values\n> and we can see that 1 value has appeared a lot of times"},{"metadata":{},"cell_type":"markdown","source":"**sea_level_pressure**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of sea_level_pressure is:\", train.sea_level_pressure.value_counts().shape)\nbar_plot(\"sea_level_pressure\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**wind_direction**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of wind_direction is:\", train.wind_direction.value_counts().shape)\nprint(train.wind_direction.value_counts())\nbar_plot(\"wind_direction\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> here too 0 has appeared a lot of times"},{"metadata":{},"cell_type":"markdown","source":"**wind_speed **"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of wind_speed is:\", train.wind_speed.value_counts().shape)\nprint(train.wind_speed.value_counts())\nbar_plot(\"wind_speed\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> here too wind speed 0 has occured a lot of time"},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{},"cell_type":"markdown","source":"**Date**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import datetime\n\n#convert into datetime\ntrain[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\n\n#Extarct year, month, weeks etc from timestamp\ntrain[\"year\"] = pd.DatetimeIndex(train[\"timestamp\"]).year\ntrain[\"month\"] = pd.DatetimeIndex(train[\"timestamp\"]).month\ntrain[\"day\"] = pd.DatetimeIndex(train[\"timestamp\"]).day\ntrain[\"week\"] = pd.DatetimeIndex(train[\"timestamp\"]).week\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of year is:\", train.year.value_counts().shape)\nprint(train.year.value_counts())\nbar_plot(\"year\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> From above we can see that year is 2016 for all rows so we donot need to consider it"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(\"year\", axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of month is:\", train.month.value_counts().shape)\n#print(train.month.value_counts())\nbar_plot(\"month\", train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of day is:\", train.day.value_counts().shape)\n#print(train.day.value_counts())\nbar_plot(\"day\", train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"total different value of week is:\", train.week.value_counts().shape)\n#print(train.week.value_counts())\nbar_plot(\"week\", train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Sorting dataframe on basis of timestamp**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.sort_values(by='timestamp', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n\ntrain['timestamp'].plot()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Importing text data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(\"../input/ashrae-energy-prediction/test.csv\")\nweat_test = pd.read_csv(\"../input/ashrae-energy-prediction/weather_test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = reduce_mem_usage(test_df)\nweat_test = reduce_mem_usage(weat_test)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.merge(test_df, build_meta, on=\"building_id\", how=\"left\")\nprint(\"Shape after merging test and build test is:\", train.shape)\n\ndel test_df\ngc.collect()\n\ntest = pd.merge(test, weat_test, on=['site_id','timestamp'], how=\"left\")\nprint(\"Shape after merging all test data is:\", train.shape)\n\ndel weat_test, build_meta\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**DAtetime year for test**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#convert into datetime\ntest[\"timestamp\"] = pd.to_datetime(test[\"timestamp\"])\n\n#Extarct year, month, weeks etc from timestamp\n#train[\"year\"] = pd.DatetimeIndex(train[\"timestamp\"]).year\ntest[\"month\"] = pd.DatetimeIndex(test[\"timestamp\"]).month\ntest[\"day\"] = pd.DatetimeIndex(test[\"timestamp\"]).day\ntest[\"week\"] = pd.DatetimeIndex(test[\"timestamp\"]).week\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop([\"timestamp\"], axis=1, inplace=True)\ntest.drop([\"timestamp\"], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = train[\"meter_reading\"]\ndel train[\"meter_reading\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Target Encoding For categorical words**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from category_encoders import *\n\ndef target_encoder(feature):\n    cat_vectorizer = TargetEncoder().fit(train[feature].astype(str), target)\n\n    train[feature] = cat_vectorizer.transform(train[feature].astype(str))\n    test[feature] = cat_vectorizer.transform(test[feature].astype(str))\n    print(\"Done\")\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = [\"primary_use\"]\nfor i in cols:\n    target_encoder(feature = i)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" #cols = [\"floor_count\", \"year_built\", \"floor_count\", \"air_temperature \", \"cloud_coverage\",\\\n  #     \"dew_temperature\", \"precip_depth_1_hr\", \"sea_level_pressure\", \"wind_direction\", \"ind_speed\"]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.fillna(-99, inplace=True)\ntest.fillna(-99, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"feat_cols = [cols for cols in train.columns]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# work in progress. will update this kernel with few more beautiful insights"},{"metadata":{},"cell_type":"markdown","source":"**Please upvote if you like my work and give your suggestion on where I need to improve**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}