{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CAIS 2022 Workshop 6: Introduction to CNN\n\nConvolutional neural network (CNN) is very popular type of neural network used in image recognition and processing that is specifically designed to process pixel data. CNN are typical used in computer vision like image and video recognition software.\n\n**In this workshop we will cover:**\n1. Loading Data\n2. Defining a CNN Model\n3. Compiling a CNN Model\n4. Fitting a CNN Model\n5. Evaluating CNN Model","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-03-01T01:05:00.772474Z","iopub.execute_input":"2022-03-01T01:05:00.772765Z","iopub.status.idle":"2022-03-01T01:05:00.779673Z","shell.execute_reply.started":"2022-03-01T01:05:00.772737Z","shell.execute_reply":"2022-03-01T01:05:00.77865Z"}}},{"cell_type":"markdown","source":"## Import you packages/modules","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\n# import warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:39.642524Z","iopub.execute_input":"2022-03-09T23:25:39.644559Z","iopub.status.idle":"2022-03-09T23:25:45.844226Z","shell.execute_reply.started":"2022-03-09T23:25:39.644524Z","shell.execute_reply":"2022-03-09T23:25:45.843302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Loading the Data Set\n* In this part we load and visualize the data.","metadata":{}},{"cell_type":"code","source":"# read train \ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")\nprint(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:45.845911Z","iopub.execute_input":"2022-03-09T23:25:45.846152Z","iopub.status.idle":"2022-03-09T23:25:49.44034Z","shell.execute_reply.started":"2022-03-09T23:25:45.846124Z","shell.execute_reply":"2022-03-09T23:25:49.439583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read test \ntest= pd.read_csv(\"../input/digit-recognizer/test.csv\")\nprint(test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:49.441502Z","iopub.execute_input":"2022-03-09T23:25:49.441883Z","iopub.status.idle":"2022-03-09T23:25:51.46957Z","shell.execute_reply.started":"2022-03-09T23:25:49.441843Z","shell.execute_reply":"2022-03-09T23:25:51.468665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# put labels into y_train variable\nY_train = train[\"label\"]\n# Drop 'label' column\nX_train = train.drop(labels = [\"label\"],axis = 1) \n\n# visualize number of digits classes\nplt.figure(figsize=(10,7))\ng = sns.countplot(Y_train, palette=\"icefire\")\nplt.title(\"Number of digit classes\")\nY_train.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:51.472115Z","iopub.execute_input":"2022-03-09T23:25:51.472384Z","iopub.status.idle":"2022-03-09T23:25:51.854835Z","shell.execute_reply.started":"2022-03-09T23:25:51.472349Z","shell.execute_reply":"2022-03-09T23:25:51.854217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot some samples\nimg = X_train.iloc[0].to_numpy()\nimg = img.reshape((28,28))\nplt.imshow(img,cmap='gray')\nplt.title(train.iloc[0,0])\nplt.axis(\"off\")\nplt.show()\n\nimg = X_train.iloc[3].to_numpy()\nimg = img.reshape((28,28))\nplt.imshow(img,cmap='gray')\nplt.title(train.iloc[3,0])\nplt.axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:51.856105Z","iopub.execute_input":"2022-03-09T23:25:51.856813Z","iopub.status.idle":"2022-03-09T23:25:52.040085Z","shell.execute_reply.started":"2022-03-09T23:25:51.856773Z","shell.execute_reply":"2022-03-09T23:25:52.039069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Normalization, Reshape and Label Encoding \n* Normalization\n * We perform a grayscale normalization to reduce the effect of illumination's differences.\n * A 8-bit grayscale was used (2^8 = 256)\n * If we perform normalization, CNN works faster.\n* Reshape\n * Train and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values.\n * We reshape all data to 28x28x1 3D matrices.\n * Keras needs an extra dimension in the end which correspond to channels. Our images are gray scaled so it use only one channel. \n* Label Encoding \n * Encode labels to one hot vectors \n * 2 => [0,0,1,0,0,0,0,0,0,0]\n * 4 => [0,0,0,0,1,0,0,0,0,0]","metadata":{}},{"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255.0\ntest = test / 255.0\nprint(\"x_train shape: \",X_train.shape)\nprint(\"test shape: \",test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.041633Z","iopub.execute_input":"2022-03-09T23:25:52.041988Z","iopub.status.idle":"2022-03-09T23:25:52.177755Z","shell.execute_reply.started":"2022-03-09T23:25:52.041945Z","shell.execute_reply":"2022-03-09T23:25:52.17711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reshape\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)\nprint(\"x_train shape: \",X_train.shape)\nprint(\"test shape: \",test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.178852Z","iopub.execute_input":"2022-03-09T23:25:52.179035Z","iopub.status.idle":"2022-03-09T23:25:52.184658Z","shell.execute_reply.started":"2022-03-09T23:25:52.179011Z","shell.execute_reply":"2022-03-09T23:25:52.183833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Label Encoding \nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nY_train = to_categorical(Y_train, num_classes = 10)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.186014Z","iopub.execute_input":"2022-03-09T23:25:52.186937Z","iopub.status.idle":"2022-03-09T23:25:52.195418Z","shell.execute_reply.started":"2022-03-09T23:25:52.186896Z","shell.execute_reply":"2022-03-09T23:25:52.194768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train Test Split\n* The data is split into train and test sets.\n* Test size is 10%.\n* Train size is 90%.","metadata":{}},{"cell_type":"code","source":"# Split the train and the validation set for the fitting\nfrom sklearn.model_selection import train_test_split\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=2)\nprint(\"x_train shape\",X_train.shape)\nprint(\"x_test shape\",X_val.shape)\nprint(\"y_train shape\",Y_train.shape)\nprint(\"y_test shape\",Y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.196612Z","iopub.execute_input":"2022-03-09T23:25:52.19706Z","iopub.status.idle":"2022-03-09T23:25:52.626323Z","shell.execute_reply.started":"2022-03-09T23:25:52.197031Z","shell.execute_reply":"2022-03-09T23:25:52.62551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Some examples\nplt.imshow(X_train[2][:,:,0],cmap='gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.628751Z","iopub.execute_input":"2022-03-09T23:25:52.628978Z","iopub.status.idle":"2022-03-09T23:25:52.795304Z","shell.execute_reply.started":"2022-03-09T23:25:52.628951Z","shell.execute_reply":"2022-03-09T23:25:52.79459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Defining a CNN Model\n* This is a 5 layers Sequential Convolutional Neural Network for digits recognition trained on MNIST dataset\n* \"gec2\"","metadata":{}},{"cell_type":"markdown","source":"* The convolution layer uses a filter also referred to as Feature detector or kernel to detect features like edges or convex shapes. Example, if out input is dog, feature detector can detect features like ear or tail of the dog.\n\n* feature map = conv(input image, feature detector). Element wise multiplication of matrices.\n\n* The size of the imaged is reduce by convolution operation therefore output image will have less information. However, training the CNN will be faster and we captured the most important information use the convolution operation.\n\n* \"gec\"\n\n* After having convolution layer we use ReLU to remove the negative values and to increase nonlinearity of the image\n* \"RELU\"","metadata":{}},{"cell_type":"markdown","source":"### Same Padding\n* As mention before conv layers will shrink the image every iteration so surround the output image with a layer of zero (padding) to ensure that output image is the same size of the input image \n\n* In the early layers of our network, we want to preserve as much information about the original input volume so that we can extract those low level features.\n\n* \"padding\"","metadata":{}},{"cell_type":"markdown","source":"### Max Pooling\n* Max pooling allows us to downsampling or sub-sampling number of parameter\n* It makes the detection of features invariant to scale or orientation changes.\n* The lower number of of parameters and computation in the network control the effects of overfitting.\n* \"maxpool\"","metadata":{}},{"cell_type":"markdown","source":"\n### Flattening\n\n* Flatten reshape the matrix to have one column which Nx1 matrix. \n* \"flattenigng\"","metadata":{}},{"cell_type":"markdown","source":"### Full Connection\n* Neurons in a fully connected layer have connections to all activations in the previous layer\n* Artificial Neural Network\n\n* \"fullyc\"","metadata":{}},{"cell_type":"markdown","source":"## Implementing with Keras\n\n### Create Model\n* conv => max pool => dropout => conv => max pool => dropout => fully connected (2 layer)\n* Dropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting.\n* \"dropout\"","metadata":{}},{"cell_type":"code","source":"model = Sequential()\n# When padding=\"same\" and strides=1, the output has the same size as the input.\nmodel.add(Conv2D(filters = 8, kernel_size = (5,5),padding = 'Same', \n activation ='relu', input_shape = (28,28,1)))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n#\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', \n activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n# fully connected\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation = \"softmax\"))\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.796772Z","iopub.execute_input":"2022-03-09T23:25:52.797154Z","iopub.status.idle":"2022-03-09T23:25:52.944458Z","shell.execute_reply.started":"2022-03-09T23:25:52.797114Z","shell.execute_reply":"2022-03-09T23:25:52.942136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Optimizer \nAdam combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimization algorithm that can handle sparse gradients on noisy problems.\n\nWe could have used Stochastic Gradient Descent ('sgd') or RMSProp but these optimizers are slower then Adam\n\nThe metric function \"accuracy\" is used is to evaluate the performance our model.\nThis metric function is similar to the loss function, except that the results from the metric evaluation are not used when training the model (only for evaluation).","metadata":{}},{"cell_type":"code","source":"# Define the optimizer\nfrom tensorflow.keras.optimizers import Adam\noptimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:52.946047Z","iopub.execute_input":"2022-03-09T23:25:52.946285Z","iopub.status.idle":"2022-03-09T23:25:53.279933Z","shell.execute_reply.started":"2022-03-09T23:25:52.946259Z","shell.execute_reply":"2022-03-09T23:25:53.279068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Compiling a CNN Model\n* categorical crossentropy\n* The error rate between the label image and predicated one can be used using categorical_crossentropy. It measure how poorly our model performs on images with known labels.\n* \"cce\"","metadata":{}},{"cell_type":"code","source":"# Compile the model\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:53.283333Z","iopub.execute_input":"2022-03-09T23:25:53.283569Z","iopub.status.idle":"2022-03-09T23:25:53.297308Z","shell.execute_reply.started":"2022-03-09T23:25:53.28354Z","shell.execute_reply":"2022-03-09T23:25:53.295822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Epochs and Batch Size\n* Say you have a dataset of 10 examples (or samples). You have a **batch size** of 2, and you've specified you want the algorithm to run for 3 **epochs**. Therefore, in each epoch, you have 5 **batches** (10/2 = 5). Each batch gets passed through the algorithm, therefore you have 5 iterations **per epoch**.\n* reference: https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks","metadata":{}},{"cell_type":"code","source":"epochs = 10 # for better result increase the epochs\nbatch_size = 250","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:53.298504Z","iopub.execute_input":"2022-03-09T23:25:53.299253Z","iopub.status.idle":"2022-03-09T23:25:53.311858Z","shell.execute_reply.started":"2022-03-09T23:25:53.299208Z","shell.execute_reply":"2022-03-09T23:25:53.311174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Fitting a CNN Model","metadata":{}},{"cell_type":"code","source":"# Fit the model\nhistory = model.fit(X_train, Y_train, batch_size = batch_size, epochs = epochs, \n validation_data = (X_val, Y_val), verbose = 2)","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:25:53.312836Z","iopub.execute_input":"2022-03-09T23:25:53.31316Z","iopub.status.idle":"2022-03-09T23:26:27.287774Z","shell.execute_reply.started":"2022-03-09T23:25:53.313121Z","shell.execute_reply":"2022-03-09T23:26:27.287102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#datagen = ImageDataGenerator(\n #featurewise_center=False, # set input mean to 0 over the dataset\n #samplewise_center=False, # set each sample mean to 0\n #featurewise_std_normalization=False, # divide inputs by std of the dataset\n #samplewise_std_normalization=False, # divide each input by its std\n #zca_whitening=False, # apply ZCA whitening\n #rotation_range=10, # randomly rotate images in the range (degrees, 0 to 180)\n #zoom_range = 0.1, # Randomly zoom image \n #width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)\n #height_shift_range=0.1, # randomly shift images vertically (fraction of total height)\n #horizontal_flip=False, # randomly flip images\n #vertical_flip=False) # randomly flip images\n\n\n#datagen.fit(X_train)\n#history = model.fit_generator(datagen.flow(X_train,Y_train, batch_size=batch_size),\n #epochs = epochs, validation_data = (X_val,Y_val),\n #verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size\n #, callbacks=[learning_rate_reduction])","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:26:27.290015Z","iopub.execute_input":"2022-03-09T23:26:27.290295Z","iopub.status.idle":"2022-03-09T23:26:27.295407Z","shell.execute_reply.started":"2022-03-09T23:26:27.290258Z","shell.execute_reply":"2022-03-09T23:26:27.294029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. Evaluating CNN Model\n* Test Loss visualization\n* Confusion matrix","metadata":{}},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nplt.plot(history.history['val_loss'], color='b', label=\"validation loss\")\nplt.title(\"Test Loss\")\nplt.xlabel(\"Number of Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:26:27.296759Z","iopub.execute_input":"2022-03-09T23:26:27.297171Z","iopub.status.idle":"2022-03-09T23:26:27.488263Z","shell.execute_reply.started":"2022-03-09T23:26:27.297135Z","shell.execute_reply":"2022-03-09T23:26:27.487324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# confusion matrix\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n\n# plot the confusion matrix\nf,ax = plt.subplots(figsize=(8, 8))\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.01,cmap=\"Blues\",linecolor=\"gray\", fmt= '.1f',ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-09T23:26:27.489404Z","iopub.execute_input":"2022-03-09T23:26:27.489608Z","iopub.status.idle":"2022-03-09T23:26:28.644619Z","shell.execute_reply.started":"2022-03-09T23:26:27.489583Z","shell.execute_reply":"2022-03-09T23:26:28.643819Z"},"trusted":true},"execution_count":null,"outputs":[]}]}