{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport gc\n\n# for plotting \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction to competition \nQ: How much does it cost to cool a skyscraper in the summer?\nA: A lot! And not just in dollars, but in environmental impact.\n\nThankfully, significant investments are being made to improve building efficiencies to reduce costs and emissions. The question is, are the improvements working? That’s where you come in. Under pay-for-performance financing, the building owner makes payments based on the difference between their real energy consumption and what they would have used without any retrofits. The latter values have to come from a model. Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nIn this competition, you’ll develop accurate models of metered building energy usage in the following areas: chilled water, electric, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe. With better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies."},{"metadata":{},"cell_type":"markdown","source":"# Input data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"building_metadata = pd.read_csv(\"../input/ashrae-energy-prediction/building_metadata.csv\")\nweather_train = pd.read_csv(\"../input/ashrae-energy-prediction/weather_train.csv\")\ntrain = pd.read_csv(\"../input/ashrae-energy-prediction/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Size of train data', train.shape)\nprint('Size of weather_train data', weather_train.shape)\nprint('Size of building_meta data', building_metadata.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the number of columns isn't big for each dataset , but the number of rows are huge. For train its 20 million+ and for test it is 41 million+. If we don't reduce the memory consumption, given kaggle's RAM allocation limit , the kernel will run out of memory limit and reboot frequently."},{"metadata":{},"cell_type":"markdown","source":"# Reducing memory size\nThes section is borrowed from the greate kernel https://www.kaggle.com/caesarlupum/ashrae-start-here-a-gentle-introduction. Reducing size is important as because of the very last train (20 million +) and test ( 40 million + ) the kernet RAM is being overflowed and restarting frequently. \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Function to reduce the DF size\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## REducing memory\nfor df in [train, weather_train, building_metadata]:\n    df = reduce_mem_usage(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data description\n## Files\n### train.csv\n* building_id - Foreign key for the building metadata.\n* meter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, 3: hotwater}. Not every building has all meter types.\n* timestamp - When the measurement was taken\n* meter_reading - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(3)\n# print(train.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['meter_reading'].hist(figsize=(6, 5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Very skewed and also have some outliers. It is less skewed after taking log. So it is better to train with log value. "},{"metadata":{"trusted":true},"cell_type":"code","source":"np.log1p(train['meter_reading']).hist(figsize=(6, 5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## weather_[train/test].csv\nWeather data from a meteorological station as close as possible to the site.\n* site_id\n* air_temperature - Degrees Celsius\n* cloud_coverage - Portion of the sky covered in clouds, in oktas\n* dew_temperature - Degrees Celsius\n* precip_depth_1_hr - Millimeters\n* sea_level_pressure - Millibar/hectopascals\n* wind_direction - Compass direction (0-360)\n* wind_speed - Meters per second"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.head(3)\n# print(weather_train.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train[\"air_temperature\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train[\"cloud_coverage\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train[\"dew_temperature\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train[\"sea_level_pressure\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train[\"wind_speed\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### building_metadata.csv\n* site_id - Foreign key for the weather files.\n* building_id - Foreign key for training.csv\n* primary_use - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n* square_feet - Gross floor area of the building\n* year_built - Year building was opened\n* floor_count - Number of floors of the building"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.head(3)\nprint(building_metadata.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata[\"square_feet\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata[\"square_feet\"] = building_metadata[\"square_feet\"].apply(np.log1p)\nbuilding_metadata[\"square_feet\"].hist(figsize=(6, 4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata[\"primary_use\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fixing missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_missing(df, ascending=False):\n    total = df.isnull().sum().sort_values(ascending = ascending)\n    percent = (df.isnull().sum()/df.isnull().count()*100).sort_values(ascending = ascending)\n    missing_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n    \n    # Only want to check columns with null values\n    missing_data = missing_data[missing_data['Total']!=0]\n    return missing_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"check_missing(train).head(len(train))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"check_missing(weather_train).head(len(weather_train))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So except cloud_coverage and precip_depth_1_hr others have less than 30% missing values. We'll fill them by their mean values. "},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.drop(['cloud_coverage', 'precip_depth_1_hr'], axis=1, inplace=True)\nweather_train.replace('NaN', np.nan, inplace=True)\nfor col in [\"air_temperature\", \"dew_temperature\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\"]:\n    weather_train[col].fillna(weather_train[col].mean(), inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"check_missing(building_metadata).head(len(building_metadata))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both of them have high missing values . So we'll drop them both for now."},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.drop(['floor_count', 'year_built'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Merging files"},{"metadata":{"trusted":true},"cell_type":"code","source":"def merging(df, weather):\n    df = df.merge(building_metadata, left_on = \"building_id\", right_on = \"building_id\", how = \"left\")\n    df = df.merge(weather, left_on = [\"site_id\", \"timestamp\"], right_on = [\"site_id\", \"timestamp\"], how = \"left\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = merging(train, weather_train)\ndel weather_train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"check_missing(train).head(len(train))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check that this train data which initially didn't have any null value, now has some . It is because, weather_train doesn't have value for all (site_id, timestamp) pair in train data."},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features = []\nprint(train.columns)\nfor col in train.columns:\n    if train[col].dtype == \"object\":\n        categorical_features.append(col)\nprint(categorical_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Though this only finds 2 categorical values , actually there are more. For example from data description, we know \n* The meter id code is int and has 4 unique values from 0 to 3. \n* year_built has 116 unique values except NAN. meter has only . \n*  site_id has values from 0 to 15. \n* building_id is an int value, but it has 1449 unique items.\nHowever to keep the data small, not trying one hot encoding right now."},{"metadata":{},"cell_type":"markdown","source":"## any column with unique value ?"},{"metadata":{"trusted":true},"cell_type":"code","source":"one_value_cols = [col for col in train.columns if train[col].nunique() <= 1]\nprint(f'columns with unique value in train{one_value_cols}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So no column with unique value"},{"metadata":{},"cell_type":"markdown","source":"# Feature engineering\nTimestamp related featuring. All timestamp are in year 2016."},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\ntrain[\"hour\"] = train[\"timestamp\"].dt.hour\ntrain[\"day\"] = train[\"timestamp\"].dt.day\ntrain[\"weekend\"] = train[\"timestamp\"].dt.weekday\ntrain[\"month\"] = train[\"timestamp\"].dt.month","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # train = pd.get_dummies(train, columns=categorical_features)\nfrom sklearn.preprocessing import LabelEncoder\nle = LabelEncoder()\nle.fit(train[\"primary_use\"])\ntrain[\"primary_use\"] = le.transform(train[\"primary_use\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"drop_columns = [\"timestamp\", \"meter_reading\"]\ntarget = np.log1p(train[\"meter_reading\"])\ntrain.drop(drop_columns, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train with LightGBM"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_test , y_train, y_test = train_test_split(train, target , test_size= 0.2, random_state=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n    'boosting_type': 'gbdt',\n    'objective': 'regression',\n    'metric': {'rmse'},\n    'learning_rate': 0.3,\n    'feature_fraction': 0.8,\n    'bagging_fraction': 0.8\n}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb_train = lgb.Dataset(x_train, y_train)\nlgb_test = lgb.Dataset(x_test, y_test)\ndel x_train, x_test , y_train, y_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gbm = lgb.train(params, lgb_train, num_boost_round=2000, valid_sets=[lgb_train, lgb_test], early_stopping_rounds=20, verbose_eval = 20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del lgb_train, lgb_test, train, target\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Input test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_test = pd.read_csv(\"../input/ashrae-energy-prediction/weather_test.csv\")\ntest = pd.read_csv(\"../input/ashrae-energy-prediction/test.csv\")\nprint('Size of weather_test_df data', weather_test.shape)\nprint('Size of test_df data', test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"row_id = test['row_id']\ntest.drop(['row_id'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test data conversion"},{"metadata":{"trusted":true},"cell_type":"code","source":"for df in [test, weather_test]:\n    df = reduce_mem_usage(df)\n\nweather_test.drop(['cloud_coverage', 'precip_depth_1_hr'], axis=1, inplace=True)\nweather_test.replace('NaN', np.nan, inplace=True)\nfor col in [\"air_temperature\", \"dew_temperature\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\"]:\n    weather_test[col].fillna(weather_test[col].mean(), inplace=True)\n\ntest = merging(test, weather_test)   \ndel weather_test, building_metadata\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test[\"timestamp\"] = pd.to_datetime(test[\"timestamp\"])\ntest[\"hour\"] = test[\"timestamp\"].dt.hour\ntest[\"day\"] = test[\"timestamp\"].dt.day\ntest[\"weekend\"] = test[\"timestamp\"].dt.weekday\ntest[\"month\"] = test[\"timestamp\"].dt.month\ntest.drop([\"timestamp\"], axis=1, inplace=True)\n\ntest[\"primary_use\"] = le.transform(test[\"primary_use\"])\n\ndel le\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Making prediction\nAs the RAM overflowing issue persists with predicting whole testset at once , predicting step by step resolved that problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = []\nstep = 50000\nfor i in range(0, len(test), step):\n    pred.extend(np.expm1(gbm.predict(test.iloc[i: min(i+step, len(test)), :], num_iteration=gbm.best_iteration)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({'row_id':row_id, 'meter_reading': pred})\nsubmission['meter_reading'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So there is no negative value in train and energy consumption can't be negative , but the model seems to predict them. Let's fix them and then write to csv. "},{"metadata":{"trusted":true},"cell_type":"code","source":"submission['meter_reading'] = submission['meter_reading'].apply(lambda x: 0 if x<0 else x)\nsubmission.to_csv(\"submission.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission['meter_reading'].describe()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}