{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Digit recognizer using Keras hyperparameter tuner \n### 10/11/2020"},{"metadata":{},"cell_type":"markdown","source":"### Content:\n1. **Introduction**\n    * 1.1. Prepare tensorboard\n    * 1.2. Tensorboard link\n2. **Preprocess the data**\n    * 2.1. Load data\n    * 2.2. Prepare data\n    * 2.3. X_train overview\n    * 2.4. y_train overview\n    * 2.5. Normalize, reshape and split data\n    * 2.6. Draw some digits\n3. **Data augmentation**\n4. **Hyperparameter tuning**\n    * 4.1. Build hypermodel\n    * 4.2. Hyperband tuner\n    * 4.3. Tuner search\n    * 4.4. Best models\n5. **Fit the model**\n    * 5.1. Summaries of best models\n    * 5.2. Select the model\n    * 5.3. Callbacks\n    * 5.4. Fit the model\n6. **Plot and evaluate the results**\n7. **Predict and submit results**\n"},{"metadata":{},"cell_type":"markdown","source":"## 1. Introduction \nThe idea in this notebook is to improve the \"Hello world\" of machine learning using [Keras tuner](https://github.com/keras-team/keras-tuner). To save precious time and to speed up and simplify the process of choosing the right hyperparameters of the model, it is possible to use a tuner. Kera's relatively new library is well suited for this. It allows to automatically play through different parameters and at the end - very conveniently - spits out the desired number of best models. These can then be processed in the further course. \n\n**Hyperparameter tuning is a time consuming process. This also applies to the automated version. So sit back and have a cup of coffee ☕️ and eat a cookie 🍪 - or simply adjust the tuner, epochs, etc. to your needs.**\n\nTo get an overview over the result of the fitting check out the [tensorboard](https://www.tensorflow.org/tensorboard/) (link provided in 1.2.)."},{"metadata":{},"cell_type":"markdown","source":"Some preparation for tensorboad running in Kaggle."},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf ./logs/ \n!mkdir ./logs/\n\n!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n!unzip ngrok-stable-linux-amd64.zip","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport tensorflow as tf\nfrom keras.utils.np_utils import to_categorical\nfrom tensorflow.keras import Model\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, Flatten, BatchNormalization, Dropout, AveragePooling2D\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.losses import CategoricalCrossentropy\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, TensorBoard, EarlyStopping\nfrom kerastuner.tuners import Hyperband\nimport datetime, multiprocessing, os, tensorboard, random, IPython\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.1. Prepare tensorboard\nTensorboard can be used to visualize metrics.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"pool = multiprocessing.Pool(processes = 10)\nresults_of_processes = [pool.apply_async(os.system, args=(cmd, ), callback=None)\n                        for cmd in [\n                        f\"tensorboard --logdir ./logs/ --host 0.0.0.0 --port 6006 &\",\n                        \"./ngrok http 6006 &\"\n                        ]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2. Tensorboard link\n**Use this link to open the tensorboard**. It may be a bit hard to see - it is the blue text on the black background. 👇"},{"metadata":{"trusted":true},"cell_type":"code","source":"!curl -s http://localhost:4040/api/tunnels | python3 -c \\\n    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Preprocess the data\n#### The data provided needs to be processed so it suits the model and can easier be proccessed. This contains e.g. reshaping it so it suits the `Conv2D` layer from Keras."},{"metadata":{},"cell_type":"markdown","source":"### 2.1. Load data\nLoad the `train` and the `test` data set using pandas `read_csv` method. "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2. Prepare data\nCreate `y_train` and drop the label layer from the data set."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = train['label']\nX_train = train.drop({'label'}, axis=1)\nprint(f'Shape of the raw X_train: {X_train.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.3. X_train overview\nGet an overview of the train data."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.4. y_train overview\nGet an overview of the tran labels to understand the distribution of th different digit images."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(y_train.head())\n\nplt.pie(y_train.value_counts(),\n        radius=2,\n        autopct='%1.0f%%'), \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The labels need to be processed from scalars to one-hot vectors. "},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train, num_classes=10)\nprint(f'Shape of the categorised procced y_train: {y_train.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.5. Normalize, reshape and split data\nNormalize data helps the CNN to converge faster."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train / 255.0\nX_test = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the Conv2D layer the data needs to be reshaped to a 28, 28, 1. The last dimension coresponds to the color channel. In this case it is a grayscaled image with only one color chanel. "},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\nprint(f'Shape of the reshaped X_train: {X_train.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using `train_test_split` for [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html?highlight=split#sklearn.model_selection.train_test_split) the X_train and y_train data will be split to training and validation data. The `test_size` is set to 10%."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.1, random_state=42)\nprint(f'Shape of the splited X_train: {X_train.shape}')\nprint(f'Shape of the splited X_valid: {X_valid.shape}')\nprint(f'Shape of the splited y_train: {y_train.shape}')\nprint(f'Shape of the splited y_valid: {y_valid.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.6. Draw some digits\nShow five random images from the train data and there according label (`y_train`)."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 5, figsize=(28, 28))\nfor idx, rand in enumerate(random.sample(range(1, len(X_train)), 5)):\n    ax[idx].imshow(X_train[rand][:,:,0], cmap='gray')\n    ax[idx].set_title(f\"Label: {np.argmax(y_train[rand])}\", fontdict={'size': 15})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add 10 random images to the tensorboard."},{"metadata":{"trusted":true},"cell_type":"code","source":"images = [X_train[rand][:,:,0] for rand in random.sample(range(1, len(X_train)), 10)]\nimages = np.expand_dims(images, axis=3)\nlog_dir_td = \"logs/train_data/\" + datetime.datetime.now().strftime(\"%d/%m/%y - %H:%M\")\nfile_writer = tf.summary.create_file_writer(log_dir_td)\nwith file_writer.as_default():\n    tf.summary.image(\"10 random training data examples\", images, max_outputs=10, step=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Data augmentation\nUse `ImageDataGenerator` for real-time data augmentation."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n    featurewise_center=False,\n    featurewise_std_normalization=False,\n    rotation_range=10,\n    zoom_range=0.1,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=False)\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Hyperparameter tuning\nThe `build_model` functions expects the `hp` parameter. \nAt the beginning we define various hyperparameter. Some provide a list of choices, others a range of values. \nAfterwards we use the functional API of Tensorflow to create a model using those hyperparameter."},{"metadata":{},"cell_type":"markdown","source":"### 4.1. Build hypermodel\n\nThe idea behind this implementation is to setup the hypermodel in such a way thay as many hyperparameters as possible can be tuned. The more parameters get tuned the longer takes the tuning process. \n\nThe model is using `Conv2D` layers with filter values from 8 to 256 and kernel sizes from 2, 3 or 5. The `Conv2D` layers are using either `tanh` or `relu` as activation functions. The `MaxPooling2D` and the `AveragePooling2D` layers have a fixed pool size of 2. After this layer there is a `BatchNormalisation` layer followed by a `Dropout` layer with a dropout rate from either 0.0, 0.25 or 0.5.\nAfter a `Flatten` there is at least 1 `Dense` layer, followed by a `Dropout` layer and the final `Dense` layer with the softmax activation and 10 units for categorical classification. The `epsilon` and the `learning_rate` of the Adam optemizer get also tuned. Let's see the impact of this. \n\nOne improvment I want to implement is making more layers optional. So, this part can definitively be more improved. Happy to receive some ideas!"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_hypermodel(hp):\n    \n    hp_dense_count = hp.Int('dense_count', min_value=1, max_value=7, step=1)\n    hp_conv_count_1 = hp.Int('conv_count_1', min_value=1, max_value=4, step=1)\n    hp_conv_count_2 = hp.Int('conv_count_2', min_value=1, max_value=4, step=1)\n    hp_dropout_final = hp.Choice('dropout_final', values=[0.0, 0.25, 0.5])\n    hp_learning_rate = hp.Choice('adam_learning_rate', values =[0.001, 0.0001])\n    hp_adam_epsilon = hp.Choice('adam_epsilon', values=[1e-07, 1e-08])\n    hp_conv_dropout = hp.Choice('conv_dropout', values=[0.0, 0.25, 0.5])\n    hp_filter_1 = hp.Int('conv_filter_1', min_value=16, max_value=64, step=16)\n    hp_filter_2 = hp.Int('conv_filter_2', min_value=64, max_value=256, step=32)\n    hp_kernel_size_1 = hp.Choice('conv_kernel_size_1', values=[2, 3, 5])\n    hp_kernel_size_2 = hp.Choice('conv_kernel_size_2', values=[2, 3, 5])\n    hp_conv_activation = hp.Choice('conv_activation', values=['tanh', 'relu'])\n    hp_pooling_type = hp.Choice('pooling_type', values=['avg', 'max'])\n   \n    inputs = Input((28,28,1))\n    x = inputs\n\n    for i in range(hp_conv_count_1): \n      x = Conv2D(hp_filter_1, hp_kernel_size_1, padding='same', activation=hp_conv_activation)(x)\n      x = BatchNormalization()(x)\n\n    if hp_pooling_type == 'max': \n      x = MaxPooling2D()(x)\n    else:\n      x = AveragePooling2D()(x)\n    x = Dropout(hp_conv_dropout)(x)\n\n    for i in range(hp_conv_count_2): \n      x = Conv2D(hp_filter_2, hp_kernel_size_2, padding='same', activation=hp_conv_activation)(x)\n      x = BatchNormalization()(x)\n \n    if hp_pooling_type == 'max': \n      x = MaxPooling2D()(x)\n    else:\n      x = AveragePooling2D()(x)\n    x = Dropout(hp_conv_dropout)(x)        \n\n    x = Flatten()(x)\n    \n    for i in range(hp_dense_count):    \n        hp_dense = hp.Int(f'dense_units_{i}', min_value=64, max_value=512, step=32)\n        hp_dense_activation = hp.Choice(f'dense_activation_{i}', values=['tanh', 'relu'])\n        x = Dense(hp_dense, activation=hp_dense_activation)(x)\n        x = BatchNormalization()(x)\n    \n    x = Dropout(hp_dropout_final)(x)\n    outputs = Dense(10, activation='softmax')(x)\n\n    model = Model(inputs, outputs)\n    model.compile(optimizer=Adam(learning_rate=hp_learning_rate, epsilon=hp_adam_epsilon), loss=CategoricalCrossentropy(), metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Hyperband tuner\nUsing hyperband for seaching for the best model. This tuner uses the feature of early stopping to speed up the tuning process. "},{"metadata":{"trusted":true},"cell_type":"code","source":"tuner = Hyperband(build_hypermodel, \n                  objective='val_accuracy',\n                  executions_per_trial=1,\n                  hyperband_iterations=1,\n                  max_epochs=3, \n                  project_name='cnn_keras_tuner')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3. Tuner search\nSearching for the best hyperparameters. This may take quite a while 😉. For the purpose of the speed I set `executions_per_trial`, `hyperband_iterations`, `max_epochs` and `epochs` to low numbers. "},{"metadata":{"trusted":true},"cell_type":"code","source":"class ClearTrainingOutput(tf.keras.callbacks.Callback):\n  def on_train_end(*args, **kwargs):\n    IPython.display.clear_output(wait=True)\n\ntuner.search(datagen.flow(X_train, y_train, batch_size=64), \n             epochs=1,\n             validation_data=(X_valid, y_valid),\n             callbacks=[ClearTrainingOutput(), EarlyStopping('val_accuracy', patience=1)],\n             verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.4. Best models\nCollect the best 3 models found by the hyperband and show a summary of the results."},{"metadata":{"trusted":true},"cell_type":"code","source":"best_models = tuner.get_best_models(num_models=3)\nprint(tuner.results_summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Fit the model \n### 5.1. Summaries of best models\nShows the summary of each of the `best_models` selected by the hyperparameter tuning."},{"metadata":{"trusted":true},"cell_type":"code","source":"for model in best_models:\n    model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2. Select the model\nSelect the most promising model (0, 1 or 2)."},{"metadata":{"trusted":true},"cell_type":"code","source":"chosen_model = best_models[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.3. Callbacks\nThe `tensorbord` callback is needed to store the log after each epoch to show it in the board (open it by clicking on the link in 1.2. `reduce_on_plateau` reduces the learning rate to the selected minimum `min_lr`.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"log_dir = \"logs/fit/\" + datetime.datetime.now().strftime(\"%d/%m/%y - %H:%M\")\ntensorboard = TensorBoard(log_dir=log_dir, histogram_freq=1)\nreduce_on_plateau = ReduceLROnPlateau(monitor='val_accuracy', patience=3, factor=0.5, min_lr=0.0001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.4. Fit the model\nFit the chosen model using the augmented data. 30 epochs seems to be a valid number. But less (~15) should also work fine. Just check the tensorboard or the plots for more information. The `reduce_on_plateau` and the `tensorboard` callbacks are set and get called after each epoch."},{"metadata":{"trusted":true},"cell_type":"code","source":"chosen_model.summary()\nhistory = chosen_model.fit(datagen.flow(X_train, y_train, batch_size=64), \n                    epochs=30, \n                    validation_data=(X_valid, y_valid), \n                    callbacks=[reduce_on_plateau, tensorboard],\n                    verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Plot and evaluate the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(2,1, figsize=(30, 20))\n\naxs[0].plot(history.history['accuracy'], color='red')\naxs[0].plot(history.history['val_accuracy'], color='green')\naxs[0].legend(labels=['Training accuracy','Validation accuracy'])\naxs[0].set_xlim(left=1, right=20)\naxs[0].set_xticks(range(1,20))\naxs[0].set_xlabel('Epochs')\naxs[0].set_ylabel('Accuracy')\naxs[0].grid()\n\naxs[1].plot(history.history['loss'], color='red')\naxs[1].plot(history.history['val_loss'], color='green')\naxs[1].legend(labels=['Training loss','Validation loss'])\naxs[1].set_xlim(left=1, right=20)\naxs[1].set_xticks(range(1,20))\naxs[1].set_xlabel('Epochs')\naxs[1].set_ylabel('Loss')\naxs[1].grid()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Predict and submit results"},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction = chosen_model.predict(X_test)\nprint(f'Prediction shape: {prediction.shape}')\nprediction = np.argmax(prediction, axis=1)\n\nresults = pd.Series(prediction, name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001), name='ImageId'), results], axis=1)\nsubmission.to_csv('mnist.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}