{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EDA for erratic sequences/trends","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode\ninit_notebook_mode(connected=True)\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-02-01T06:21:10.028281Z","iopub.execute_input":"2023-02-01T06:21:10.028747Z","iopub.status.idle":"2023-02-01T06:21:10.035605Z","shell.execute_reply.started":"2023-02-01T06:21:10.028702Z","shell.execute_reply":"2023-02-01T06:21:10.034741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on conclusions drawn after trying two powerful models like Random Forest and XGBoost. We needed to improve the data cleaning strategy. <br>Purpose of performing this EDA is to find erratic sequences related to very low values which could be there due to some technical faults while collecting training data. But as the **ASHRAE** organisation needs the  show up results to their customers that prove their technology can make a huge difference , they will be collecting test data with utmost care. ","metadata":{}},{"cell_type":"code","source":"# Reading the File\ntrain_df=pd.read_csv(f'/kaggle/input/ashrae-energy-prediction/train.csv')","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:12.273195Z","iopub.execute_input":"2023-02-01T06:11:12.273602Z","iopub.status.idle":"2023-02-01T06:11:20.354823Z","shell.execute_reply.started":"2023-02-01T06:11:12.273566Z","shell.execute_reply":"2023-02-01T06:11:20.353783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Original code from https://www.kaggle.com/gemartin/load-data-reduce-memory-usage by @gemartin\n\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype\n\ndef reduce_mem_usage(df, use_float16=False):\n    \"\"\"\n    Iterate through all the columns of a dataframe and modify the data type to reduce memory usage.        \n    \"\"\"\n    \n    start_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == \"int\":\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype(\"category\")\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df\n","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:20.356874Z","iopub.execute_input":"2023-02-01T06:11:20.357404Z","iopub.status.idle":"2023-02-01T06:11:20.371531Z","shell.execute_reply.started":"2023-02-01T06:11:20.357366Z","shell.execute_reply":"2023-02-01T06:11:20.370807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df=reduce_mem_usage(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:20.373059Z","iopub.execute_input":"2023-02-01T06:11:20.374042Z","iopub.status.idle":"2023-02-01T06:11:22.917083Z","shell.execute_reply.started":"2023-02-01T06:11:20.373998Z","shell.execute_reply":"2023-02-01T06:11:22.915982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's try visualizing 0 value trends by applying some grouping and aggregations to main dataframe to get how many 0 values do we have for different year days","metadata":{}},{"cell_type":"code","source":"# Defining a function over which the meter_reading column can be aggregated\ndef count_zeros(col):\n    count = 0\n    for _ in col:\n        if _ == 0:\n            count += 1\n    return count","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:22.919567Z","iopub.execute_input":"2023-02-01T06:11:22.920014Z","iopub.status.idle":"2023-02-01T06:11:22.925219Z","shell.execute_reply.started":"2023-02-01T06:11:22.919971Z","shell.execute_reply":"2023-02-01T06:11:22.924084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Density heatmap could be very nice visualization. Let's see the day wise trend","metadata":{}},{"cell_type":"code","source":"train_df.timestamp=pd.to_datetime(train_df.timestamp)\ntrain_df['dayofyear'] = train_df['timestamp'].dt.dayofyear\ntrainx_df=train_df.drop('timestamp', axis=1)\ntrainx_df = trainx_df.groupby(['dayofyear', 'building_id', 'meter']).agg([count_zeros]).dropna()\ntrainx_df.reset_index(inplace=True)\ntrainx_df.columns=trainx_df.columns.droplevel(1)\ntrainx_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:22.926436Z","iopub.execute_input":"2023-02-01T06:11:22.926815Z","iopub.status.idle":"2023-02-01T06:11:39.228402Z","shell.execute_reply.started":"2023-02-01T06:11:22.926778Z","shell.execute_reply":"2023-02-01T06:11:39.227359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems like some buildings have 0 meter readings for all 24 hours. Also let's rename the column to count of zeros","metadata":{}},{"cell_type":"code","source":"trainx_df.rename(columns={'meter_reading':'count 0'}, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:39.229758Z","iopub.execute_input":"2023-02-01T06:11:39.230104Z","iopub.status.idle":"2023-02-01T06:11:39.235344Z","shell.execute_reply.started":"2023-02-01T06:11:39.230075Z","shell.execute_reply":"2023-02-01T06:11:39.234288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Let's use this dataframe to plot a density heatmap. Also let's rename the columns meter_reading as it is no more showing us meter_reading.\n\nfig = px.density_heatmap(trainx_df,\n                         x='dayofyear',\n                         y='building_id',\n                         z='count 0',\n                         histfunc='sum',\n                         nbinsx=366,\n                         nbinsy=1449,\n                         height=1000,facet_col='meter',title='Desity Heatmap for Zero Readings')\nfig.update_layout(xaxis={'type': 'category'})\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:39.236492Z","iopub.execute_input":"2023-02-01T06:11:39.236828Z","iopub.status.idle":"2023-02-01T06:11:40.329153Z","shell.execute_reply.started":"2023-02-01T06:11:39.236782Z","shell.execute_reply":"2023-02-01T06:11:40.327942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We got huge chunk of 0 values for first 141 days of the year for building ids 0 to 104. So let's remove these values these are surely erratic. **So now considering month as category columns may not make much sense**. It may make sense if we are not considering to train different models based on meter and site id.\n\n\n<br><br>\nDefining a recursive function to have nice visualisation with plotly for all buildings in a form of grid.","metadata":{}},{"cell_type":"code","source":"del trainx_df","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:40.331860Z","iopub.execute_input":"2023-02-01T06:11:40.332209Z","iopub.status.idle":"2023-02-01T06:11:40.337019Z","shell.execute_reply.started":"2023-02-01T06:11:40.332172Z","shell.execute_reply":"2023-02-01T06:11:40.335937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"building_df=pd.read_csv('/kaggle/input/ashrae-energy-prediction/building_metadata.csv')\n# using same variable to keep memory in control. Will delete train_df and reload train.csv\ntrain_df=train_df.merge(building_df,how='left',on='building_id')\ntrain_df=train_df[['site_id','building_id','meter','timestamp','meter_reading']] #keeping only required columns","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:11:40.338367Z","iopub.execute_input":"2023-02-01T06:11:40.338672Z","iopub.status.idle":"2023-02-01T06:11:45.271226Z","shell.execute_reply.started":"2023-02-01T06:11:40.338644Z","shell.execute_reply":"2023-02-01T06:11:45.270221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_reading_trend(train_df,site=0,meter=0,x=32):\n    trainx_df = train_df.query(f'meter == {meter} & site_id=={site}')\n    buildings= trainx_df.building_id.unique()\n   # keeping data of 32 buildings only to have better visualisations\n    trainy_df=trainx_df.query(f'building_id < {x} & building_id>={x-32}')\n    # taking log of meter readings\n    trainy_df['meter_reading_log1p']=np.log1p(trainy_df['meter_reading'])\n    #grouping with sum aggregation\n    trainy_df = trainy_df.groupby(['building_id','timestamp'])['meter_reading_log1p'].sum()\n    #resetting the index to give parameters to plotly\n    trainy_df = trainy_df.to_frame().reset_index()\n    # ploting line graphs for each building id showing graphs of 8 buildings in 1 row\n    fig = px.line(trainy_df, x='timestamp', y='meter_reading_log1p',facet_col='building_id',facet_col_wrap=8,title='Building_id vs Sum of daywise log_meter_reading')\n    fig.update_yaxes(title_text=\"\"); # removing x,y titles\n    fig.update_xaxes(title_text=\"\");\n    fig.show()\n    # if there are any buildings left we again call this function\n    if x<trainx_df.building_id.nunique():\n        plot_reading_trend(train_df,site=0,meter=0,x=x+32)\n","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:21:25.065172Z","iopub.execute_input":"2023-02-01T06:21:25.065578Z","iopub.status.idle":"2023-02-01T06:21:25.076007Z","shell.execute_reply.started":"2023-02-01T06:21:25.065546Z","shell.execute_reply":"2023-02-01T06:21:25.074910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_reading_trend(train_df,site=0,meter=0,x=32)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:21:26.616764Z","iopub.execute_input":"2023-02-01T06:21:26.617438Z","iopub.status.idle":"2023-02-01T06:21:40.263101Z","shell.execute_reply.started":"2023-02-01T06:21:26.617392Z","shell.execute_reply":"2023-02-01T06:21:40.261962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we do have long erratic trends for all buildings in site 0 with meter 0 from start of year till 20th May. We should consider removing all these values","metadata":{}},{"cell_type":"code","source":"plot_reading_trend(train_df,site=0,meter=1,x=32)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T06:21:40.264930Z","iopub.execute_input":"2023-02-01T06:21:40.265910Z","iopub.status.idle":"2023-02-01T06:21:42.124804Z","shell.execute_reply.started":"2023-02-01T06:21:40.265867Z","shell.execute_reply":"2023-02-01T06:21:42.123584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There id no erratic trend with meter 1 except for building 7 for month of November. Such trends can be checked for other sites and meters as well. Site id 0 has meter 0 and 1 only.\nDue to time constraints all buildings and sites are not been checked here, site id 0 seemed to be most problematic in previous visualisation so that's why only removing these values for site 0. Removing other similarly erratic trends can help improve the scores significantly. ","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}