{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ASHRAE with fast.ai, Part 2: Training","metadata":{}},{"cell_type":"markdown","source":"This kernel leverages the convenient fast.ai API to prepare the dataset for training in just a few lines of code. It then trains a neural network adapted for tabular data.\n\nIn order to combine the large size of the ASHRAE dataset and the overhead of fast.ai's objects with the limited memory of Kaggle sessions, this kernel is part of a series which further includes:\n\n- https://www.kaggle.com/michelezoccali/ashrae-with-fast-ai-part-1 (preprocessing)\n- https://www.kaggle.com/michelezoccali/ashrae-with-fast-ai-part-3 (inference)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport sys\nimport psutil\n\nimport numpy as np\nimport pandas as pd\nimport datetime\nimport warnings\n\nfrom tqdm.notebook import tqdm\nfrom sklearn.metrics import mean_squared_error\nfrom fastai.tabular.all import *\n\n# plotting\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2021-07-04T07:49:32.592079Z","iopub.execute_input":"2021-07-04T07:49:32.592452Z","iopub.status.idle":"2021-07-04T07:49:33.461439Z","shell.execute_reply.started":"2021-07-04T07:49:32.592374Z","shell.execute_reply":"2021-07-04T07:49:33.460601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The pre-processed DataFrames can be accessed in the input folder, in the `ashrae-with-fast-ai-part-1` subdirectory.","metadata":{}},{"cell_type":"code","source":"path = '../input/ashrae-with-fast-ai-part-1/'\n\nfor dirname, _, filenames in os.walk(path):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2021-07-04T07:49:35.946708Z","iopub.execute_input":"2021-07-04T07:49:35.947041Z","iopub.status.idle":"2021-07-04T07:49:35.959808Z","shell.execute_reply.started":"2021-07-04T07:49:35.94701Z","shell.execute_reply":"2021-07-04T07:49:35.958755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data","metadata":{}},{"cell_type":"markdown","source":"Let's load the preprocessed training dataset without lag features.","metadata":{}},{"cell_type":"code","source":"#%%time\ndf_train = pd.read_hdf(path + 'preprocessing_no_lag.h5', 'train')\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T14:06:42.007751Z","iopub.execute_input":"2021-07-03T14:06:42.008156Z","iopub.status.idle":"2021-07-03T14:06:52.705032Z","shell.execute_reply.started":"2021-07-03T14:06:42.00812Z","shell.execute_reply":"2021-07-03T14:06:52.704174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.info(memory_usage='deep')","metadata":{"execution":{"iopub.status.busy":"2021-07-03T14:06:52.706425Z","iopub.execute_input":"2021-07-03T14:06:52.706812Z","iopub.status.idle":"2021-07-03T14:06:52.737336Z","shell.execute_reply.started":"2021-07-03T14:06:52.706773Z","shell.execute_reply":"2021-07-03T14:06:52.730574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T14:06:52.739233Z","iopub.execute_input":"2021-07-03T14:06:52.739722Z","iopub.status.idle":"2021-07-03T14:06:52.857577Z","shell.execute_reply.started":"2021-07-03T14:06:52.739649Z","shell.execute_reply":"2021-07-03T14:06:52.8565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# use for snapshot of RAM consumption by process\npsutil.test()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T14:06:52.859463Z","iopub.execute_input":"2021-07-03T14:06:52.860277Z","iopub.status.idle":"2021-07-03T14:06:52.878531Z","shell.execute_reply.started":"2021-07-03T14:06:52.860227Z","shell.execute_reply":"2021-07-03T14:06:52.877303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Modeling","metadata":{}},{"cell_type":"markdown","source":"We will be using fast.ai's TabularLearner NN class. One possible way to instantiate such a class is by first defining, among other things:\n\n- Categorical and continuous variables\n- A training/validation split\n- A set of transforms one wishes to apply to the data\n- A TabularPandas object\n- A DataLoaders object\n\nLet's do so in order below.","metadata":{}},{"cell_type":"code","source":"dep_var = 'meter_reading'\ncont, cat = cont_cat_split(df_train, max_card=25, dep_var=dep_var)\n#cont, cat","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[cat].nunique()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:25.146023Z","iopub.execute_input":"2021-07-03T13:53:25.146407Z","iopub.status.idle":"2021-07-03T13:53:25.986258Z","shell.execute_reply.started":"2021-07-03T13:53:25.146362Z","shell.execute_reply":"2021-07-03T13:53:25.98519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we will be doing a simple 11/1 train/validation split for starters. Let us extract the last month of 2016 as validation set.","metadata":{}},{"cell_type":"code","source":"cond = df_train.month<12\ntrain_idx = np.where( cond)[0]\nvalid_idx = np.where(~cond)[0]\n\nsplits = (list(train_idx), list(valid_idx))\n\ndf_train = df_train.drop(columns='month') # remove to avoid overfit\ncat.remove('month')","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:25.987965Z","iopub.execute_input":"2021-07-03T13:53:25.988337Z","iopub.status.idle":"2021-07-03T13:53:28.416212Z","shell.execute_reply.started":"2021-07-03T13:53:25.988291Z","shell.execute_reply":"2021-07-03T13:53:28.415254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's tell fast.ai which transforms we wish to apply. Note that here we do not include `FillMissing`, contrary to common practice, as we took care of missing values during preprocessing.","metadata":{}},{"cell_type":"code","source":"pd.options.mode.chained_assignment = None # to be included alongside reduce_memory=True\n\nprocs_nn = [Categorify, Normalize]\ndf_train = TabularPandas(df_train, procs_nn, cat, cont,\n                      splits=splits, y_names=dep_var,\n                      inplace=True, reduce_memory=True)\n\nlen(df_train.train), len(df_train.valid)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:28.417633Z","iopub.execute_input":"2021-07-03T13:53:28.418037Z","iopub.status.idle":"2021-07-03T13:53:52.906496Z","shell.execute_reply.started":"2021-07-03T13:53:28.41799Z","shell.execute_reply":"2021-07-03T13:53:52.905649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"TabularPandas instances have a handy method to directly create the `DataLoaders`:","metadata":{}},{"cell_type":"code","source":"dls = df_train.dataloaders(1024)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:52.907951Z","iopub.execute_input":"2021-07-03T13:53:52.908322Z","iopub.status.idle":"2021-07-03T13:53:58.64414Z","shell.execute_reply.started":"2021-07-03T13:53:52.908281Z","shell.execute_reply":"2021-07-03T13:53:58.643341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's establish the range of the target variable, so that we may rescale the outputs of the NN to fill this range by means of a sigmoid activation added as the last layer.","metadata":{}},{"cell_type":"code","source":"df_train.train.y.min(), df_train.train.y.max()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:58.645412Z","iopub.execute_input":"2021-07-03T13:53:58.645793Z","iopub.status.idle":"2021-07-03T13:53:59.742522Z","shell.execute_reply.started":"2021-07-03T13:53:58.645754Z","shell.execute_reply":"2021-07-03T13:53:59.741223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Instantiate the model and inspect its architecture.","metadata":{}},{"cell_type":"code","source":"learn = tabular_learner(dls, y_range=(0,17), layers=[500,250], n_out=1, loss_func=F.mse_loss)\nlearn.model","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:59.74428Z","iopub.execute_input":"2021-07-03T13:53:59.744706Z","iopub.status.idle":"2021-07-03T13:53:59.790576Z","shell.execute_reply.started":"2021-07-03T13:53:59.744649Z","shell.execute_reply":"2021-07-03T13:53:59.78961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find appropriate learning rate\nlearn.lr_find()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:53:59.792168Z","iopub.execute_input":"2021-07-03T13:53:59.792521Z","iopub.status.idle":"2021-07-03T13:54:28.742986Z","shell.execute_reply.started":"2021-07-03T13:53:59.792483Z","shell.execute_reply":"2021-07-03T13:54:28.742117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us now train the model with the 1-cycle policy.","metadata":{}},{"cell_type":"code","source":"learn.fit_one_cycle(10, 1e-2)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:54:28.744328Z","iopub.execute_input":"2021-07-03T13:54:28.744708Z","iopub.status.idle":"2021-07-03T13:58:53.893029Z","shell.execute_reply.started":"2021-07-03T13:54:28.744652Z","shell.execute_reply":"2021-07-03T13:58:53.891983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us check the results of the training by hand.","metadata":{}},{"cell_type":"code","source":"# this gets predictions on the validation set by default\npreds, targs = learn.get_preds()\n\nrmse_valid = np.sqrt(mean_squared_error(to_np(targs.squeeze()), to_np(preds.squeeze())))\nrmse_valid","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:59:04.036794Z","iopub.execute_input":"2021-07-03T13:59:04.037198Z","iopub.status.idle":"2021-07-03T13:59:04.053206Z","shell.execute_reply.started":"2021-07-03T13:59:04.037138Z","shell.execute_reply":"2021-07-03T13:59:04.052208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del cond, train_idx, valid_idx, splits\ndel df_train, dls, preds, targs, rmse_valid\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:59:04.054554Z","iopub.execute_input":"2021-07-03T13:59:04.05502Z","iopub.status.idle":"2021-07-03T13:59:07.713717Z","shell.execute_reply.started":"2021-07-03T13:59:04.054982Z","shell.execute_reply":"2021-07-03T13:59:07.712713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems that the model is performing well on the validation set (as well as LGBM \nin [this kernel](https://www.kaggle.com/michelezoccali/ashrae-energy-prediction-single-lgbm)). Let's save it.","metadata":{}},{"cell_type":"code","source":"with open('tabular_nn.pickle', mode='wb') as f:\n    pickle.dump(learn, f)","metadata":{"execution":{"iopub.status.busy":"2021-07-03T13:59:07.718015Z","iopub.execute_input":"2021-07-03T13:59:07.718351Z","iopub.status.idle":"2021-07-03T13:59:18.008889Z","shell.execute_reply.started":"2021-07-03T13:59:07.718322Z","shell.execute_reply":"2021-07-03T13:59:18.007738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's it. Now we only ought to do inference on the test set. However, trying to create a TabularPandas object for the entire test set in this kernel causes memory peaks which crash the session. Thus, let's continue in:\n\n- https://www.kaggle.com/michelezoccali/ashrae-with-fast-ai-part-3","metadata":{}}]}