{"cells":[{"metadata":{},"cell_type":"markdown","source":"
Keras: Introduction to CNN with TPUs
\n
on MNIST digit dataset
\n
\n
"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"The objective of this notebook is to create a model running on TPUs that allows to correctly classify a handwritten digit. The TPUs will allow to distribute the calculations during model training."},{"metadata":{},"cell_type":"markdown","source":"#
Summary
\n\n**1. Load libraries and check TPU settings**\n**
2. EDA and preprocessing**\n**
3. CNN**\n**
4. Evaluation**"},{"metadata":{},"cell_type":"markdown","source":"#
1. Load libraries and check TPU settings
"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Remove warning messages\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport random\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport plotly\nimport plotly.graph_objects as go\n%matplotlib inline\n\nimport os\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nimport tensorflow as tf\nfrom keras.utils.np_utils import to_categorical\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom tensorflow.keras.optimizers import RMSprop\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.metrics import top_k_categorical_accuracy, categorical_accuracy\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set seed\nnp.random.seed(42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"print(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_TO_DATA = '../input/digit-recognizer/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load train and test\ntrain = pd.read_csv(PATH_TO_DATA + 'train.csv')\ntest = pd.read_csv(PATH_TO_DATA + 'test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# First rows of train\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"--------\n\n**Back to summary**"},{"metadata":{},"cell_type":"markdown","source":"#
2. EDA and preprocessing
"},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Class distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_distribution_classes(x_values, y_values):\n\n fig = go.Figure(data=[go.Bar(\n x=x_values, \n y=y_values,\n text=y_values\n )])\n\n fig.update_layout(height=600, width=1200, title_text=\"Distribution of classes\")\n fig.update_xaxes(type=\"category\")\n\n fig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = np.sort(train.label.unique())\ny = train.label.value_counts().sort_index()\n\nplot_distribution_classes(x, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocessing(train, test, split_train_size = 0.1):\n\n X_train = train.drop([\"label\"],\n axis = 1)\n y_train = train[\"label\"]\n\n # Normalize the data\n X_train = X_train / 255.0\n test = test / 255.0\n\n # Reshape into right format vectors. One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions.\n X_train = X_train.values.reshape(-1,28,28,1)\n X_test = test.values.reshape(-1,28,28,1)\n\n # Apply ohe on labels\n y_train = to_categorical(y_train, num_classes = 10)\n \n # Split the train and the validation set for the fitting\n X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = split_train_size, random_state=42)\n \n return X_train, y_train, X_val, y_val, X_test\n\nX_train, y_train, X_val, y_val, X_test = preprocessing(train, test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Shape of training data: {X_train.shape}')\nprint(f'Shape training labels: {y_train.shape}')\nprint(f'Shape of validation data: {X_val.shape}')\nprint(f'Shape of valiation labels: {y_val.shape}')\nprint(f'Shape of testing data: {X_test.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Display some examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_images(graph_indexes = np.arange(9)):\n \n # plot first few images\n plt.figure(figsize=(12,12))\n \n for graph_index in graph_indexes:\n \n index = random.randint(1, X_train.shape[0])\n \n # Get corresponding label\n label = list(y_train[index]).index(1)\n \n # define subplot\n plt.subplot(330 + 1 + graph_index)\n plt.title('Label: %s \\n'%label,\n fontsize=18)\n # plot raw pixel data\n plt.imshow(X_train[index][:,:,0], cmap=plt.get_cmap('gray'))\n \n plt.subplots_adjust(bottom = 0.001) # the bottom of the subplots of the figure\n plt.subplots_adjust(top = 0.99)\n # show the figure\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_images()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.4 Convert data to a tensorflow dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 32 * strategy.num_replicas_in_sync # this is 8 on TPU v3-8, it is 1 on CPU and GPU","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Put data in a tensor format for parallelization\n\ntrain_dataset = (\n tf.data.Dataset\n # The given tensors are sliced along their first dimension. This operation preserves the structure of the input tensors, \n # removing the first dimension of each tensor and using it as the dataset dimension.\n .from_tensor_slices((X_train.astype(np.float32), \n y_train.astype(np.float32)))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nval_dataset = (\n tf.data.Dataset\n .from_tensor_slices((X_val.astype(np.float32), \n y_val.astype(np.float32)))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\ntest_dataset = (\n tf.data.Dataset\n .from_tensor_slices(X_test.astype(np.float32))\n .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Back to summary**\n\n-------"},{"metadata":{},"cell_type":"markdown","source":"#
3. CNN
"},{"metadata":{},"cell_type":"markdown","source":"## 3.1 What is a CNN ?\n\nA CNN is quite similar to Classic Neural Networks (RegularNets) where there are neurons with weights and biases. Just like in RegularNets, we use a loss function and an optimizer in CNNs. Additionally though, in CNNs, there are Convolutional Layers, Pooling Layers, and Flatten Layers. CNNs are mainly used for image classification.\n\n### CNN layers\n* **Convolutional layer** \n\nThe very first layer where we extract features from the images in our datasets. Due to the fact that pixels are only related with the adjacent and close pixels, convolution allows us to preserve the relationship between different parts of an image. Convolution is basically filtering the image with a smaller pixel filter to decrease the size of the image without loosing the relationship between pixels. When we apply convolution to 5x5 image by using a 3x3 filter with 1x1 stride (1 pixel shift at each step). We will end up having a 3x3 output (64% decrease in complexity).\n\n\n* **Pooling layer**\n\nWhen constructing CNNs, it is common to insert pooling layers after each convolution layer to reduce the spatial size of the representation to reduce the parameter counts which reduces the computational complexity. In addition, pooling layers also **helps with the overfitting problem**. Basically we select a pooling size to reduce the amount of the parameters by selecting the maximum, average, or sum values inside these pixels.\n\n\n* **Flatten layer**\n\nFlattens the input. Does not affect the batch size."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parameters\nN_ITER = 250\nEPOCHS = 30","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def top_5_categorical_accuracy(y_true, y_pred):\n return top_k_categorical_accuracy(y_true, y_pred, k=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def CNN_model():\n \n model = tf.keras.Sequential()\n\n model.add(tf.keras.layers.Conv2D(filters = 16, kernel_size = (5,5), padding = 'Same', \n activation ='relu', input_shape = (28,28,1)))\n model.add(tf.keras.layers.MaxPool2D(pool_size=(2,2)))\n model.add(tf.keras.layers.Dropout(0.25))\n\n model.add(tf.keras.layers.Conv2D(filters = 32, kernel_size = (3,3), padding = 'Same', \n activation ='relu'))\n model.add(tf.keras.layers.MaxPool2D(pool_size=(2,2), strides=(2,2)))\n model.add(tf.keras.layers.Dropout(0.25))\n\n model.add(tf.keras.layers.Flatten())\n model.add(tf.keras.layers.Dense(256, activation = \"relu\"))\n model.add(tf.keras.layers.Dropout(0.25))\n model.add(tf.keras.layers.Dense(10, activation = \"softmax\"))\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Create model with TPU"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TPU\nwith strategy.scope():\n model = CNN_model()\n \nmodel.summary()\n\n# Compile the model\nmodel.compile(optimizer = 'Adam', \n loss = \"categorical_crossentropy\", \n metrics=[\"accuracy\", top_5_categorical_accuracy])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 Good reflexes to have\n\n* **Add dropout**\n\nDropout refers to ignoring neurons during the training phase of certain set of neurons which is chosen at random.\n\n* **LeakyRelu**\n\nThe advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of the output for negative inputs is a learnable parameter while in the Leaky ReLU it's a hyperparameter.\n\n* **Add callbacks**\n\nA callback is a function that is to be executed after another function has finished executing hence the name 'call back'. With callbacks, you can define earlystopping criterias for your model if it doesn't learn anymore through epochs. Callback allows you to store some information at the end of each epoch so you can check your model's performance."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define callbacks\n\n# Save weights only for best model\ncheckpointer = ModelCheckpoint(filepath = 'weights_best_MNIST.hdf5', \n verbose = 2, \n save_best_only = True)\n\n# LR strategy\nlearning_rate = ReduceLROnPlateau(monitor='accuracy', \n patience=5, \n verbose=2, \n factor=0.5)\n\n# If score doesn't improve during patience epochs, stop learning\nestopping = EarlyStopping(monitor='val_loss', \n patience=10, \n verbose=2)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"history = model.fit(train_dataset, \n steps_per_epoch = N_ITER, \n epochs = EPOCHS, \n validation_data=(val_dataset),\n callbacks = [checkpointer, learning_rate, estopping])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 History of CNN"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_history(model_history):\n\n plt.figure(figsize = (20,15))\n \n plt.subplot(221)\n # summarize history for accuracy\n plt.plot(model_history.history['top_5_categorical_accuracy'])\n plt.plot(model_history.history['val_top_5_categorical_accuracy'])\n plt.title('top_5_categorical_accuracy')\n plt.ylabel('accuracy')\n plt.xlabel('epoch')\n plt.legend(['train', 'test'], loc='upper left')\n plt.grid()\n \n plt.subplot(222)\n # summarize history for accuracy\n plt.plot(model_history.history['accuracy'])\n plt.plot(model_history.history['val_accuracy'])\n plt.title('model accuracy')\n plt.ylabel('accuracy')\n plt.xlabel('epoch')\n plt.legend(['train', 'test'], loc='upper left')\n plt.grid()\n \n plt.subplot(223)\n # summarize history for loss\n plt.plot(model_history.history['loss'])\n plt.plot(model_history.history['val_loss'])\n plt.title('model loss')\n plt.ylabel('loss')\n plt.xlabel('epoch')\n plt.legend(['train', 'test'], loc='upper left')\n plt.grid()\n \n plt.subplot(224)\n # summarize history for lr\n plt.plot(model_history.history['lr'])\n plt.title('learning rate')\n plt.ylabel('lr')\n plt.xlabel('epoch')\n plt.grid()\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_history(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Back to summary**\n\n--------"},{"metadata":{},"cell_type":"markdown","source":"#
4. Evaluation
"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TPU\nwith strategy.scope():\n # loading the model with the best validation accuracy\n model.load_weights('weights_best_MNIST.hdf5')\n \nmodel.evaluate(val_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Confusion Matrix"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_confusion_matrix(confusion_matrix, \n cmap=plt.cm.Reds):\n \n classes = range(10)\n \n plt.figure(figsize=(8,8))\n plt.imshow(confusion_matrix, \n interpolation='nearest', \n cmap=cmap)\n plt.title('Confusion matrix')\n plt.colorbar()\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n thresh = confusion_matrix.max() / 2.\n for i, j in itertools.product(range(confusion_matrix.shape[0]), range(confusion_matrix.shape[1])):\n plt.text(j, i, confusion_matrix[i, j],\n horizontalalignment=\"center\",\n color=\"white\" if confusion_matrix[i, j] > thresh else \"black\")\n\n plt.tight_layout()\n plt.ylabel('True label')\n plt.xlabel('Predicted label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predict the values from the validation dataset\ny_pred = model.predict(X_val.astype(np.float32))\n\n# Convert predictions classes to one hot vectors \ny_pred_classes = np.argmax(y_pred, axis = 1) \n\n# Convert validation observations to one hot vectors\ny_true = np.argmax(y_val, axis = 1) \n\n# compute the confusion matrix\ncm = confusion_matrix(y_true, y_pred_classes) \n\n# plot the confusion matrix\nplot_confusion_matrix(cm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Some examples of predicted images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def display_predicted_images(graph_indexes = np.arange(9)):\n \n # plot first few images\n plt.figure(figsize=(12,12))\n \n for graph_index in graph_indexes:\n \n index = random.randint(1, X_val.shape[0])\n \n # Get corresponding label\n predicted_label = y_pred_classes[index]\n true_label = y_true[index]\n \n \n # define subplot\n plt.subplot(330 + 1 + graph_index)\n plt.title('Predicted label: %s \\n'%predicted_label+\\\n 'True label %s \\n'%true_label,\n fontsize=18)\n # plot raw pixel data\n plt.imshow(X_val[index][:,:,0], cmap=plt.get_cmap('gray'))\n \n plt.subplots_adjust(bottom = 0.001) # the bottom of the subplots of the figure\n plt.subplots_adjust(top = 0.99)\n # show the figure\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_predicted_images()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is most likely to see a perfect prediction on this short sample of predictions, let's print predictions that have been the least accurate for our model"},{"metadata":{},"cell_type":"markdown","source":"## 4.3 \"Best\" errors"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display errors \nerrors = (y_pred_classes - y_true != 0)\n\ny_pred_classes_errors = y_pred_classes[errors]\ny_pred_errors = y_pred[errors]\ny_true_errors = y_true[errors]\nX_val_errors = X_val[errors]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def display_top9_wrongly_predicted_images(list_of_indexes, graph_indexes = np.arange(9)):\n \n # plot first few images\n plt.figure(figsize=(12,12))\n \n for graph_index in graph_indexes:\n \n index = list_of_indexes[graph_index]\n \n # Get corresponding label\n predicted_label = y_pred_classes_errors[index]\n true_label = y_true_errors[index]\n \n \n # define subplot\n plt.subplot(330 + 1 + graph_index)\n plt.title('Predicted label: %s \\n'%predicted_label+\\\n 'True label %s \\n'%true_label,\n fontsize=18)\n # plot raw pixel data\n plt.imshow(X_val_errors[index][:,:,0], cmap=plt.get_cmap('gray'))\n \n plt.subplots_adjust(bottom = 0.001) # the bottom of the subplots of the figure\n plt.subplots_adjust(top = 0.99)\n # show the figure\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Probabilities of the wrong predicted numbers\ny_pred_errors_prob = np.max(y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(y_pred_errors, y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 9 errors\nmost_important_errors = sorted_dela_errors[-9:]\n\n# Show the top 9 errors\ndisplay_top9_wrongly_predicted_images(list_of_indexes = most_important_errors)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Back to summary**\n\n--------"},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict results\ny_test_pred = model.predict(test_dataset)\n\n# Associate max probability obs with label class\ny_test_pred = np.argmax(y_test_pred, axis = 1)\ny_test_pred = pd.Series(y_test_pred, name=\"Label\")\n\nsubmission = pd.concat([pd.Series(range(1,28001), name = \"ImageId\"), y_test_pred], axis = 1)\n\nsubmission.to_csv(\"CNN_model_TPU_submission.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n\n* Thanks to yassineghouzam for his inspiring notebook\n\n* TPU usage on flowers classification\n\n* TPU documentation\n\n* My previous notebook: CNN for Handwritten Letters Classification"},{"metadata":{},"cell_type":"markdown","source":"
\n
There is also CNN implementation using Pytorch.
\n
Pytorch CNN for MNIST digit recognition with GPUs
\n
\n
Thank you for taking the time to read this notebook. I hope that I was able to answer your questions or your curiosity and that it was quite understandable. any constructive comments are welcome. They help me progress and motivate me to share better quality content. I am above all a passionate person who tries to advance my knowledge but also that of others. If you liked it, feel free to upvote and share my work.
\n
\n
Thank you and may passion guide you.
"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}