{"cells":[{"metadata":{},"cell_type":"markdown","source":"**We are going to generate a baseline solution using plain statistics without any model. From this we can guess the pattern in test data**"},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport gc\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***In this Baseline Solution, we will be using the non null values of 2016 for both 2017 & 2018. We can see that for some buildings, the initial data points have target 0, and later the pattern changes. So, we treat the data points with six consecutive non nulls as our first data. We drop the rest of initial data.***"},{"metadata":{},"cell_type":"markdown","source":"### Read data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\ntrain[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv('../input/ashrae-energy-prediction/test.csv')\ntest[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Clean train data \nRemove the initial data points where the target is mostly zero. Here we impose condition that, there should be non-null target for six consecutive hours."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_len = len(train)\nprint (\"Initial Train Rows :\",train.shape[0])\ntrain['nonZero'] = (train.meter_reading>0).astype(int)\nlag = lambda x,l: x.shift(l).fillna(1)\nmask = train.groupby(['building_id','meter'])['nonZero'].transform(lambda x: (x*lag(x,1)*lag(x,2)*lag(x,3)*lag(x,4)*lag(x,5)).cumsum())\ntrain=train[mask>0]\ntrain= train.drop('nonZero',axis=1)\nprint (\"Clean Train Rows : \",train.shape[0])\nprint ('Number of Rows removed : ', train_len-len(train))\ngc.collect()\ntrain[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Merge datasets\nMerge the train and test set based on dates. (i.e. 2016-10-10 10:00:00,  2017-10-10 10:00:00 and  2017-10-10 10:00:00 will all have same meter reading)"},{"metadata":{"trusted":true},"cell_type":"code","source":"test.timestamp = test.timestamp.apply(lambda x: x.replace('2017','2016').replace('2018','2016'))\ntest = pd.merge(test, train, how='left')\nprint(pd.isna(test).sum())\ntest[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.groupby(['building_id','meter'])['meter_reading'].count().sort_values()[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Handle Null Targets"},{"metadata":{},"cell_type":"markdown","source":"We can see that there are over 2.6M missing meter readings in the merged set. We should impute them using interpolation.\nAlso there are few buildings which do not have any training data. We can impute them with zeros/mean."},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_target = np.expm1(np.log1p(train.meter_reading).mean())  ## Computing the geometric mean\ndel train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def impute(x):\n    if len(x.dropna())==0:\n        return x.fillna(0)\n        # return x.fillna(mean_target)    ## Uncomment if you want to use geometric mean\n    if pd.isna(x.iloc[0]):\n        last_val = x.dropna().iloc[-1]\n        x.iloc[0]=last_val\n    if pd.isna(x.iloc[-1]):\n        first_val = x.dropna().iloc[0]\n        x.iloc[-1]=first_val\n    return x.interpolate()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.meter_reading = test.groupby(['building_id','meter'])['meter_reading'].transform(impute)\ngc.collect()\ntest[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Write submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# test.loc[test.meter_reading==0, 'meter_reading'] = mean_target   ## Uncomment if you want to replace zeros with geometric mean of target\n\ntest[['row_id','meter_reading']].to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}