{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle Transfer Learning Code for the City Square Feed Kaggle In-Class Competition\n\n**Author: Shawn S Lin and Qingyuan Gao(Group777)**\n\nThis workbook is a modified code for the [City Square Feed Kaggle In-Class Competition](https://www.kaggle.com/c/applications-of-deep-learning-wustlfall-2021)  This competition is one of the assignments for [T81-558: Applications of Deep Neural Netw1orks](https://sites.wustl.edu/jeffheaton/t81-558/) at [Washington University in St. Louis](https://www.wustl.edu).\n\nThis is a more advanced notebook that uses transfer learning; however, it is not particularly optimized.  It is also possable to run this project from Google CoLab.  ","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nPATH = \"/kaggle/input/applications-of-deep-learning-wustlfall-2021/city\"\nPATH_TRAIN = os.path.join(PATH, \"train.csv\")\nPATH_TEST = os.path.join(PATH, \"test.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:48.129708Z","iopub.execute_input":"2021-11-27T15:30:48.130272Z","iopub.status.idle":"2021-11-27T15:30:48.155333Z","shell.execute_reply.started":"2021-11-27T15:30:48.130149Z","shell.execute_reply":"2021-11-27T15:30:48.154665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we check versions and if the GPU is available. A GPU or TPU will be very helpful.","metadata":{}},{"cell_type":"code","source":"# What version of Python do you have?\nimport sys\n\nimport tensorflow.keras\nimport pandas as pd\nimport sklearn as sk\nimport tensorflow as tf\n\nprint(f\"Tensor Flow Version: {tf.__version__}\")\nprint(f\"Keras Version: {tensorflow.keras.__version__}\")\nprint()\nprint(f\"Python {sys.version}\")\nprint(f\"Pandas {pd.__version__}\")\nprint(f\"Scikit-Learn {sk.__version__}\")\nprint(\"GPU is\", \"available\" if tf.test.is_gpu_available() \\\n      else \"NOT AVAILABLE\")","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:48.157089Z","iopub.execute_input":"2021-11-27T15:30:48.157338Z","iopub.status.idle":"2021-11-27T15:30:55.90287Z","shell.execute_reply.started":"2021-11-27T15:30:48.157305Z","shell.execute_reply":"2021-11-27T15:30:55.902125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we prepare to read the training data (that we have labels for) and the test data that we must predict and send to Kaggle.","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(PATH_TRAIN)\ndf_test = pd.read_csv(PATH_TEST)\n\ndf_train['filename'] = df_train.id.astype(str) + \".jpg\"\ndf_test['filename'] = df_test.id.astype(str) + \".jpg\"","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:55.903868Z","iopub.execute_input":"2021-11-27T15:30:55.9043Z","iopub.status.idle":"2021-11-27T15:30:55.985662Z","shell.execute_reply.started":"2021-11-27T15:30:55.90426Z","shell.execute_reply":"2021-11-27T15:30:55.984929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Splitting training set and validation set","metadata":{}},{"cell_type":"markdown","source":"If we want to use early stopping, we need a validation set.  We will break the data into 90 percent test data and 10 validation.  Do not confuse this validation data with the test set provided by Kaggle.  This validation set is unique to your program and is just used for early stopping.","metadata":{}},{"cell_type":"code","source":"# splitting scheme from starter code\nTRAIN_PCT = 0.9\nTRAIN_CUT = int(len(df_train) * TRAIN_PCT)\n\ndf_train_cut = df_train[0:TRAIN_CUT]\ndf_validate_cut = df_train[TRAIN_CUT:]\n\nprint(f\"Training size: {len(df_train_cut)}\")\nprint(f\"Validate size: {len(df_validate_cut)}\")\nprint(len(df_train))","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:55.989584Z","iopub.execute_input":"2021-11-27T15:30:55.989786Z","iopub.status.idle":"2021-11-27T15:30:55.993101Z","shell.execute_reply.started":"2021-11-27T15:30:55.989762Z","shell.execute_reply":"2021-11-27T15:30:55.992404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we want to do kfold-validation, we could use `sklearn.model_selection.KFold`, however, since kaggle notebook has a 9-hour cap on single run, we can manually do the split to seperate each of the kfold, run seperately, and aggreagate the results later.","metadata":{}},{"cell_type":"code","source":"# # 10fold splitting scheme\n# k_fold_idx = [0,0.2,0.4,0.6,0.8,1]\n\n# # MANUALLY CHANGE PARAMETER TO RUN EACH FOLD!\n# fold = 5\n\n# VAL_START = int(len(df_train) * k_fold_idx[fold-1])\n# VAL_END = int(len(df_train) * k_fold_idx[fold])\n\n# df_train_cut = df_train[0:VAL_START].append(df_train[VAL_END:])\n# df_validate_cut = df_train[VAL_START:VAL_END]\n\n\n# print(f\"Training size: {len(df_train_cut)}\")\n# print(f\"Validate size: {len(df_validate_cut)}\")\n\n# df_train_cut","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:55.994439Z","iopub.execute_input":"2021-11-27T15:30:55.994915Z","iopub.status.idle":"2021-11-27T15:30:56.027642Z","shell.execute_reply.started":"2021-11-27T15:30:55.99488Z","shell.execute_reply":"2021-11-27T15:30:56.026963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_validate_cut","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:56.028725Z","iopub.execute_input":"2021-11-27T15:30:56.028956Z","iopub.status.idle":"2021-11-27T15:30:56.041069Z","shell.execute_reply.started":"2021-11-27T15:30:56.028925Z","shell.execute_reply":"2021-11-27T15:30:56.040397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we create the generators that will provide the images to the neural network as it is trained.  We normalize the images so that the RGB colors between 0-255 become ratios between 0 and 1.  We also use the **flow_from_dataframe** generator to connect the Pandas dataframe to the actual image files. We see here a straightforward implementation; you might also wish to use some of the image transformations provided by the data generator.\n\nFor the image transformations part, it should be noted that ImageDataGenerator would only randomly apply transformation to certain image in the data, it WOULD NOT increse the size of training dataset. 20k training data stays 20k. (Ref:https://www.pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/)\n\nThe **HEIGHT** and **WIDTH** constants specify the dimensions that the image will be scaled (or expanded) to. It is probably not a good idea to expand the images.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport keras_preprocessing\nfrom keras_preprocessing import image\nfrom keras_preprocessing.image import ImageDataGenerator\n\nWIDTH = 256\nHEIGHT = 256\n\n# don't tweak brightness it did not work\ntraining_datagen = ImageDataGenerator(\n  rescale = 1./255,\n  horizontal_flip=True,\n  #vertical_flip=True,\n  fill_mode='nearest')\n\ntrain_generator = training_datagen.flow_from_dataframe(\n        dataframe=df_train_cut,\n        directory=PATH,\n        x_col=\"filename\",\n        y_col=\"sqft\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=32, # Keeping the training batch size small USUALLY increases performance\n        class_mode='raw')\n\nvalidation_datagen = ImageDataGenerator(rescale = 1./255)\n\nval_generator = validation_datagen.flow_from_dataframe(\n        dataframe=df_validate_cut,\n        directory=PATH,\n        x_col=\"filename\",\n        y_col=\"sqft\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=256, # Make the validation batch size as large as you have memory for\n        class_mode='raw')","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:30:56.042424Z","iopub.execute_input":"2021-11-27T15:30:56.042941Z","iopub.status.idle":"2021-11-27T15:31:34.502876Z","shell.execute_reply.started":"2021-11-27T15:30:56.042907Z","shell.execute_reply":"2021-11-27T15:31:34.501725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Following utility snippet can be used to take a peek into the image data produced ImageDataGenerator","metadata":{}},{"cell_type":"code","source":"'''\nimport matplotlib.pyplot as plt\ndef plots(ims, figsize=(25,25), rows=3, interp=False, titles=None):\n    if type(ims[0]) is np.ndarray:\n        ims = np.array(ims)\n        if (ims.shape[-1] != 3):\n            ims = ims.transpose((0,2,3,1))\n    f = plt.figure(figsize=figsize)\n    cols = len(ims)//rows if len(ims) % 2 == 0 else len(ims)//rows + 1\n    for i in range(len(ims)):\n        sp = f.add_subplot(rows, cols, i+1)\n        sp.axis('Off')\n        if titles is not None:\n            sp.set_title(titles[i], fontsize=16)\n        plt.imshow(ims[i], interpolation=None if interp else 'none')\n\nimgs, labels = next(train_generator)\n\n# prints batch of imgs\n# plots(imgs, titles=labels)\n\nplt.imshow(imgs[0])\nplt.show()\n'''","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:31:34.504345Z","iopub.execute_input":"2021-11-27T15:31:34.504588Z","iopub.status.idle":"2021-11-27T15:31:34.511164Z","shell.execute_reply.started":"2021-11-27T15:31:34.504555Z","shell.execute_reply":"2021-11-27T15:31:34.510348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**(This block is copied from starter, might not be all true)**\nWe now create the neural network and fit it.  Some essential concepts are going on here.\n\n* **Batch Size** - The number of training samples that should be evaluated per training step.  Smaller batch sizes, or mini-batches, are generally preferred.\n* **Step** - A training step is one complete run over the batch.  At the end of a step, the weights are updated, and the neural network learns.\n* **Epoch** - An arbitrary point at which to measure results or checkpoint the model.  Generally, an epoch is one complete pass over the training set.  However, when generators are used, the training set size is theoretically infinite. Because of this, we set a **steps_per_epoch** parameter.\n* **validation steps** - The validation set may also be infinite; because of this, we must specify how many steps we wish to validate at the end of each Epoch.","metadata":{}},{"cell_type":"markdown","source":"After testing, we think there's a maximum`steps_per_epoch` that can be computed using `STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size`, as mentioned before, it would not increase the size of your training set.\nBy setting a`steps_per_epoch` larger than `STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size`, we would see the training stops at 1 epoch as there is no more training data that can be passed into the neural network.","metadata":{}},{"cell_type":"markdown","source":"# Transfer Learning\n\n(FROM STARTER CODE)We will now use a ResNet neural network as a basis for our neural network.  We will redefine both the input shape and output of the ResNet model, so we will not transfer the weights.  Since we redefine the input; the weights are of minimal value.  We begin by loading, from Keras, the ResNet50 network.","metadata":{}},{"cell_type":"markdown","source":"We took the time to try out different networks in `Keras.Applications`, for this competition, DenseNet121 is performing well.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications.densenet import DenseNet121\nfrom tensorflow.keras.layers import Input\n\ninput_tensor = Input(shape=(HEIGHT, WIDTH, 3))\n\nbase_model = DenseNet121(\n    include_top=False, weights=None, input_tensor=input_tensor,\n    input_shape=None)\n\n#base_model.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:31:34.515304Z","iopub.execute_input":"2021-11-27T15:31:34.516635Z","iopub.status.idle":"2021-11-27T15:31:36.839478Z","shell.execute_reply.started":"2021-11-27T15:31:34.516603Z","shell.execute_reply":"2021-11-27T15:31:36.83877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we must add a few layers to the end of the neural network so that it becomes a regression model.","metadata":{}},{"cell_type":"markdown","source":"We've tested out some other configurations but did not see much improvements over the 2\\*1024 dense layer one from starter.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout\nfrom tensorflow.keras.models import Model\n\nx=base_model.output\nx=GlobalAveragePooling2D()(x)\nx=Dense(1024,activation='relu')(x) \nx=Dense(1024,activation='relu')(x)\nmodel=Model(inputs=base_model.input,outputs=Dense(1)(x))\nmodel.summary()\n\n# design from keras doc, this one is very slow and did not see much improvement\n# x=base_model.output\n# x=GlobalAveragePooling2D()(x)\n# x =Dropout(0.2)(x)  # Regularize with dropout\n# model=Model(inputs=base_model.input,outputs=Dense(1)(x))\n# model.summary()","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-11-27T15:31:36.840696Z","iopub.execute_input":"2021-11-27T15:31:36.84097Z","iopub.status.idle":"2021-11-27T15:31:37.080733Z","shell.execute_reply.started":"2021-11-27T15:31:36.840937Z","shell.execute_reply":"2021-11-27T15:31:37.080078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we train just like before, the only difference is that we do not define the entire neural network here.","metadata":{}},{"cell_type":"markdown","source":"Some parameters you might want to work on:\n* **optimizer**: some articles states that a well tuned `sgd` (usually with momentum) performs better at the cost of longer training time, we did not pull off one that performs better than adam though.\n* **steps_per_epoch**: this works with `batch_size` defined in ImageDataGenerator, to use all data in training set, follow this formula`STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size`. \n* **callbacks**: a simple EarlyStopping performs well enough for this competition, we tried `ReduceLROnPlateau` along side with `sgd` but did not see much improvement.\n* **epochs**: too small could cause underfit or caught in the peak during loss fluctuation. If you choose to count on EarlyStopping you should be able to get away with setting this one as large as possible.\n* **patience**: this determines how much epochs the training would continue before no improvement greater than `min_delta` is perceived. A setting too small might get you trapped in a saddle point, too large would lead to longer training time. 25 worked fine for us.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping,ReduceLROnPlateau\nfrom tensorflow.keras.metrics import RootMeanSquaredError\n\n# Important, calculate a valid step size for the validation dataset\nSTEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=val_generator.n//val_generator.batch_size\n\nmodel.compile(loss = 'mean_squared_error', optimizer='adam', metrics=[RootMeanSquaredError(name=\"rmse\")])\n\n# callbacks: early stopping and learning rate decay\nearly_stop  = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=25, verbose=1, mode='auto',\n        restore_best_weights=True)\n# reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, min_lr=1e-6, verbose=1) # tune if training too slow; should not cause over fit as this only put lr down\n\nhistory = model.fit(train_generator, epochs=200, steps_per_epoch=STEP_SIZE_TRAIN, \n                    validation_data = val_generator, callbacks=[early_stop],\n                    verbose = 1, validation_steps=STEP_SIZE_VALID)","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:31:37.081758Z","iopub.execute_input":"2021-11-27T15:31:37.082021Z","iopub.status.idle":"2021-11-27T15:38:58.333141Z","shell.execute_reply.started":"2021-11-27T15:31:37.081984Z","shell.execute_reply":"2021-11-27T15:38:58.332351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Scrapping up after training","metadata":{}},{"cell_type":"markdown","source":"Since model training is very time consuming, and weight initilazation is stochastic, it is not that easy to build reproducible model. we would want to save up some information for later reference as","metadata":{}},{"cell_type":"code","source":"# plot training curve\n\nimport matplotlib.pyplot as plt\n\ndef display_training_curves(training, validation, title, subplot):\n  if subplot%10==1: # set up the subplots on the first call\n    plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n    plt.tight_layout()\n  ax = plt.subplot(subplot)\n  ax.set_facecolor('#F8F8F8')\n  ax.plot(training)\n  ax.plot(validation)\n  ax.set_title('model '+ title)\n  ax.set_ylabel(title)\n  ax.set_xlabel('epoch')\n  ax.legend(['train', 'valid.'])\n  plt.savefig('learning curve.png')\n\nprint(history.history.keys())\ndisplay_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 211) \ndisplay_training_curves(history.history['rmse'], history.history['val_rmse'], 'rmse', 212) ","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:38:58.349623Z","iopub.execute_input":"2021-11-27T15:38:58.351573Z","iopub.status.idle":"2021-11-27T15:38:59.231066Z","shell.execute_reply.started":"2021-11-27T15:38:58.351537Z","shell.execute_reply":"2021-11-27T15:38:59.229967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save the model for later transfer tuning\nmodel.save('./model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:38:59.232409Z","iopub.execute_input":"2021-11-27T15:38:59.232834Z","iopub.status.idle":"2021-11-27T15:39:00.634683Z","shell.execute_reply.started":"2021-11-27T15:38:59.232769Z","shell.execute_reply":"2021-11-27T15:39:00.633841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## loading from our trained model\n# from tensorflow import keras \n# model = keras.models.load_model('../input/model-v31/model_v31.h5')\n# model.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:00.636239Z","iopub.execute_input":"2021-11-27T15:39:00.636481Z","iopub.status.idle":"2021-11-27T15:39:00.641212Z","shell.execute_reply.started":"2021-11-27T15:39:00.636447Z","shell.execute_reply":"2021-11-27T15:39:00.639792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## access history\n# history.history['rmse']\n# history.history['val_rmse']","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:00.652521Z","iopub.execute_input":"2021-11-27T15:39:00.652784Z","iopub.status.idle":"2021-11-27T15:39:00.66064Z","shell.execute_reply.started":"2021-11-27T15:39:00.652749Z","shell.execute_reply":"2021-11-27T15:39:00.659961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this tells the final val_loss\ntextfile = open(\"final_val_rmse.txt\", \"w\")\ntextfile.write(str(history.history['val_rmse'][-1]))\ntextfile.close()","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:47:03.489477Z","iopub.execute_input":"2021-11-27T15:47:03.489756Z","iopub.status.idle":"2021-11-27T15:47:03.496574Z","shell.execute_reply.started":"2021-11-27T15:47:03.489726Z","shell.execute_reply":"2021-11-27T15:47:03.495543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Submission\n\nNow that the neural network is trained; we need to generate a submit CSV file to send to Kaggle.  We will use nearly the same technique to build the submit file.  However, these essential points that we must address:\n\n* We do not want the data generator to create an infinite date like we did when training.  We have a fixed number of cases to score for the Kaggle submit; we only want to process them.\n* We do not want the data generator to randomize the samples' order like it did when training. Therefore we set shuffle to false.\n* We want to always start at the beginning of the data, so we reset the generator.\n\nThese ensure that the predictions align with the id's.","metadata":{}},{"cell_type":"code","source":"submit_datagen = ImageDataGenerator(rescale = 1./255)\n\nsubmit_generator = submit_datagen.flow_from_dataframe(\n        dataframe=df_test,\n        directory=PATH,\n        x_col=\"filename\",\n        batch_size = 1,\n        shuffle = False,\n        target_size=(HEIGHT, WIDTH),\n        class_mode=None)\n\nsubmit_generator.reset()\npred = model.predict(submit_generator,steps=len(df_test))","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:00.662252Z","iopub.execute_input":"2021-11-27T15:39:00.662609Z","iopub.status.idle":"2021-11-27T15:39:25.657659Z","shell.execute_reply.started":"2021-11-27T15:39:00.662573Z","shell.execute_reply":"2021-11-27T15:39:25.656917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imgs = next(submit_generator)\n\n# prints batch of imgs\n# plots(imgs, titles=labels)\n\nplt.imshow(imgs[0])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:25.659115Z","iopub.execute_input":"2021-11-27T15:39:25.659386Z","iopub.status.idle":"2021-11-27T15:39:25.868712Z","shell.execute_reply.started":"2021-11-27T15:39:25.659348Z","shell.execute_reply":"2021-11-27T15:39:25.868021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submit = pd.DataFrame({\"id\":df_test['id'],'sqft':pred[:,0].flatten()})\ndf_submit.to_csv(\"/kaggle/working/submit.csv\",index = False)","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:25.870191Z","iopub.execute_input":"2021-11-27T15:39:25.870629Z","iopub.status.idle":"2021-11-27T15:39:25.882489Z","shell.execute_reply.started":"2021-11-27T15:39:25.870592Z","shell.execute_reply":"2021-11-27T15:39:25.881661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p2 = pred[:,0].flatten()","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:25.884099Z","iopub.execute_input":"2021-11-27T15:39:25.884554Z","iopub.status.idle":"2021-11-27T15:39:25.890407Z","shell.execute_reply.started":"2021-11-27T15:39:25.884517Z","shell.execute_reply":"2021-11-27T15:39:25.889578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submit[df_submit.id==24163]","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:25.891349Z","iopub.execute_input":"2021-11-27T15:39:25.891691Z","iopub.status.idle":"2021-11-27T15:39:25.91824Z","shell.execute_reply.started":"2021-11-27T15:39:25.89166Z","shell.execute_reply":"2021-11-27T15:39:25.917408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_generator.filenames","metadata":{"execution":{"iopub.status.busy":"2021-11-27T15:39:25.919969Z","iopub.execute_input":"2021-11-27T15:39:25.920304Z","iopub.status.idle":"2021-11-27T15:39:25.944955Z","shell.execute_reply.started":"2021-11-27T15:39:25.920267Z","shell.execute_reply":"2021-11-27T15:39:25.944272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensemble","metadata":{}},{"cell_type":"markdown","source":"This **might not** be the optimal/'textbook' way to conduct ensemble: What we did was taking an average of all the output .csv files that perfomed well on public test set. Following code were used to make things a bit easier when there's multiple outputs to ensemble. ","metadata":{}},{"cell_type":"markdown","source":"Note that this snippet was ran on a local pc, feel free to modify as needed.","metadata":{}},{"cell_type":"code","source":"'''\nimport pandas as pd\nimport numpy as np\n\nimport os\nPATH = 'workplace'\nfilenames = os.listdir(PATH)\nfilenames\n\ndf_list = []\nfor i in filenames:\n    df = pd.read_csv(PATH+'/'+i)\n    df_list.append(df)\ndf_submit = df_list[0][['id']].copy()\n\n# avg\nsqft_col = pd.DataFrame(0, index=np.arange(len(df_submit)), columns=['sqft'])\n\nfor i in df_list:\n    sqft_col += i[['sqft']]\n\ndf_submit['sqft'] = sqft_col/len(df_list)\ndf_submit\n\n# this is meant to keep output organized\nID = input('input submit number: ')\n\ndf_submit.to_csv(f\"submit_{ID}.csv\",index = False)\n'''","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`model.save()` in previous code should be helpful if we want to do proper bagging or anything like that.","metadata":{}}]}