{"cells":[{"metadata":{"_uuid":"b86a595d274700f3d3881597d9de1463255583a0"},"cell_type":"markdown","source":"# Introduction to CNN with keras\n\n* ** 1. introduction ** \n* ** 2. Data preparation **\n * 2.1 loading data \n\n * 2.2 Normalization\n * 2.3 Reshape\n * 2.4 Label Encoding\n * 2.5 splitting into training and validation data\n \n \n ** 3. CNN Architecure **\n* 3.1. Define Model\n* 3.2 Data Augmentation\n* 3.3 optimizer and Learning Rate scheduler\n\n** 4. Model Evaluation **\n* 4.1 plotting training and validation loss\n* 4.2 plotting training and validation loss\n* 4.3 confusion matrix\n\n\n"},{"metadata":{"_uuid":"65ae35121f95010bb149515f39d4f61f71cb1c8f"},"cell_type":"markdown","source":"

1.Introduction to Convolution Neural network

\n\n![](https://adeshpande3.github.io/assets/Cover.png)\n
\n
\n

A CNN is a neural network that typically contains several types of layers, one of which is a convolutional layer, as well as pooling, and activation layers.

\n\n

convolutional layer

\n
\n

The Conv layer is the core building block of a Convolutional Network that does most of the computational heavy lifting.

\n\n\n\n\n

Imagine you have an image represented as a 5x5 matrix of values, and you take a 3x3 matrix and slide that 3x3 window around the image. At each position the 3x3 visits, you matrix multiply element wise the values of your 3x3 window by the values in the image that are currently being covered by the window and it also passes through RELU Activation.. This results in a single number the represents all the values in that window of the image.

\n\n\n\n\n\n\n

The “window” that moves over the image is called a kernel. . The weigts of Kernel are randomly initialize and later it learn them.
The distance the window moves each time is called the stride. \n\n

pooling layer

\n\n

Convolutional networks may include local or global pooling layers, which combine the outputs of neuron clusters at one layer into a single neuron in the next layer. For example, max pooling uses the maximum value from each of a cluster of neurons at the prior layer.Another is average pooling, which uses the average value from each of a cluster of neurons at the prior layer.

\n\n![](https://www.embedded-vision.com/sites/default/files/technical-articles/CadenceCNN/Figure7.jpg)\n\nFor more detail [Convolutional Neural Networks](http://cs231n.github.io/convolutional-networks/)
\n\n
\n\n

Activation layer

\n\n

Activation functions are important for a Artificial Neural Network to learn and understand the complex patterns. The main function of it is to introduce non-linear properties into the network. What it does is, it calculates the ‘weighted sum’ and adds direction and decides whether to ‘fire’ a particular neuron or not. There are several kinds of non-linear activation functions, like Sigmoid, Tanh, ReLU and leaky ReLU. The non linear activation function will help the model to understand the complexity and give accurate results.

\n![](https://i.stack.imgur.com/iIcbq.gif)\n\nFor more detail. [Types Of Activation Functions In Neural Networks And Rationale Behind It](https://i.stack.imgur.com/iIcbq.gif)\n\n\n"},{"metadata":{"_uuid":"ca41245ecacda3b99ce9add76da61925f2bfc5d7"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a82c569fc9e72bab02e56189d7c29b54fbf37d42"},"cell_type":"markdown","source":"## 2.Data preprocess\n"},{"metadata":{"_uuid":"c87092c95ab5474e788092169b514d4a2f93ec1c"},"cell_type":"markdown","source":"### 2.1 loading data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"collapsed":true},"cell_type":"code","source":"\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9380455fb543b7d4cb8d3135f2bbd2f79967f5c0","trusted":true},"cell_type":"code","source":"print(train.shape)\nprint(test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04f83390ba69e93f499bf46dd6eef4882be366aa","trusted":true},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a3617924581908916d04e36aa887765f741215c0","trusted":true},"cell_type":"code","source":"test.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ded7b150dda947ae5fa57aa9da536880059b3432","trusted":true},"cell_type":"code","source":"X_train=train.drop(labels = [\"label\"],axis = 1) \nY_train=train['label']\nprint(X_train.shape)\nprint(Y_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40d48b1b46c7efa6136af815773f12acd05e0b71","trusted":true},"cell_type":"code","source":"Y_train.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e6d7bdccb69186196cb93688adef6acc251f7ae"},"cell_type":"markdown","source":"### visualizing the number of different labels in traing data"},{"metadata":{"_uuid":"424409a5832eb2371ce685ca7f4919cfbe669c80","trusted":true},"cell_type":"code","source":"import seaborn as sns\nplt.figure(figsize=(8,4))\nsns.countplot(x='label', data=train);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e455f1002b29723061c320187a92eeeeb8b3f60e"},"cell_type":"markdown","source":"### 2.2 Normalizing data"},{"metadata":{"_uuid":"1a77be37f59ef86c3f93094a683a6234f25019be","trusted":true,"collapsed":true},"cell_type":"code","source":"X_train=X_train.astype('float32')/255\ntest=test.astype('float32')/255","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"beeaafdbc47e99ab11341a29dffbd7d2eab2e29e"},"cell_type":"markdown","source":"### 2.3 Reshape \n\nReshaping image into 3D matrix"},{"metadata":{"_uuid":"01917deae085f57365e28ac3c0b1c20c291a6b84","trusted":true,"collapsed":true},"cell_type":"code","source":"X_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a034aaf809a37374d421523a78f99b7c33c6471","trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e48bb64e66b829bf3862e99282a6f6c6a9c0b8f","trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee637a33e11a7c7d0e8b3fd91a78e35abf0e6d1c"},"cell_type":"markdown","source":"### 2.4 Label Encoding"},{"metadata":{"_uuid":"85b81e0fa8eb3ab4aa0b09ae262f7af535f6f40d","trusted":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical\nY_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83e4d885daccfa9aaaa0e71305f676fdcdd6d813","trusted":true},"cell_type":"code","source":"Y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07e842439aac978a9b81e4ec0124f740e088e2e4","trusted":true},"cell_type":"code","source":"print(Y_train[:5])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"76aa18b22631db9e07f1909f2dd6594ffe35249a"},"cell_type":"markdown","source":"** 2.5 Now we will split training data into training data and validation data **"},{"metadata":{"_uuid":"8066b776e4e1b297e89680dfbf513521982b494f","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_valid, Y_train, Y_valid = train_test_split(X_train, Y_train, test_size = 0.1, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0fed0df1e29dcbb025b680318d7ca5d22f11768b","trusted":true,"scrolled":true},"cell_type":"code","source":"plt.figure(figsize=(6,6))\nplt.imshow(X_train[1][:,:,0])\nplt.title(Y_train[1].argmax());","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f9551b231411c0c05f05c817ab5371c997cb9a6"},"cell_type":"markdown","source":"\n\n"},{"metadata":{"_uuid":"507e67996b674cb1a30590fc220462cdf63e5628"},"cell_type":"markdown","source":"## 3. Building CNN architecture using keras"},{"metadata":{"_uuid":"a490f5c79c87e4b6c0375eeff3b94b5c3391124d"},"cell_type":"markdown","source":"### 3.1 Defining cnn model"},{"metadata":{"_uuid":"79fb7ec14b8d0d702642b51f59201b2c09277241","trusted":true,"collapsed":true},"cell_type":"code","source":"from keras.layers import Input,InputLayer, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D\nfrom keras.layers import AveragePooling2D, MaxPooling2D, Dropout\nfrom keras.models import Sequential,Model\nfrom keras.optimizers import SGD\nfrom keras.callbacks import ModelCheckpoint,LearningRateScheduler\nimport keras\nfrom keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b8015adbd0738bee5ec2bd8cd137e4cf362d89e","trusted":true,"collapsed":true},"cell_type":"code","source":"inputShape=(28,28,1)\ninput = Input(inputShape)\n\nx = Conv2D(64,(3,3),strides = (1,1),name='layer_conv1',padding='same')(input)\nx = BatchNormalization()(x)\nx = Activation('relu')(x)\nx = MaxPooling2D((2,2),name='maxPool1')(x)\n\n\n\nx = Conv2D(64,(3,3),strides = (1,1),name='layer_conv2',padding='same')(x)\nx = BatchNormalization()(x)\nx = Activation('relu')(x)\nx = MaxPooling2D((2,2),name='maxPool2')(x)\n\nx = Conv2D(32,(3,3),strides = (1,1),name='conv3',padding='same')(x)\nx = BatchNormalization()(x)\nx = Activation('relu')(x)\nx = MaxPooling2D((2,2),name='maxPool3')(x)\n\n\nx = Flatten()(x)\nx = Dense(64,activation = 'relu',name='fc0')(x)\nx = Dropout(0.25)(x)\nx = Dense(32,activation = 'relu',name='fc1')(x)\nx = Dropout(0.25)(x)\nx = Dense(10,activation = 'softmax',name='fc2')(x)\n\nmodel = Model(inputs = input,outputs = x,name='Predict')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f10e2c28421cb95e0710d10f633e993741f47045","trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"adbab78f5d37efdd64874e503555d0c0c520604e"},"cell_type":"markdown","source":"## 3.2 Data Augmentation"},{"metadata":{"_uuid":"a83cf1b5c3521c0dbe857f9965fd1e79a254449c"},"cell_type":"markdown","source":"```\ndatagen_train = ImageDataGenerator(\n width_shift_range=0.2, # randomly shift images horizontally \n height_shift_range=0.2,# randomly shift images vertically \n \n horizontal_flip=True) # randomly flip images horizontally\n\n# fit augmented image generator on data\ndatagen_train.fit(X_train)\n```"},{"metadata":{"_uuid":"9c28882af032ba0e812f1eac6461034883bc300c"},"cell_type":"markdown","source":"## 3.3 optimizer "},{"metadata":{"_uuid":"5c9cbb460562800bd46771f0ff13fcad864bb092","trusted":true,"collapsed":true},"cell_type":"code","source":"# define SGD optimizer\nmomentum = 0.5\nsgd = SGD(lr=0.01, momentum=momentum, decay=0.0, nesterov=False) \n\n# compile the model\nmodel.compile(loss='categorical_crossentropy',optimizer=sgd, metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6faf650d641c9e57f8fe6c7ef62ddb05bdda313f"},"cell_type":"markdown","source":"## Learning Rate Schedules\n\nLearning rate schedules seek to adjust the learning rate during training by reducing the learning rate according to a pre-defined schedule. Common learning rate schedules include time-based decay, step decay and exponential decay\n\n** Here we will implement Step Decay **\n\nStep decay schedule drops the learning rate by a factor every few epochs. The mathematical form of step decay is :\n```\nlr = lr0 * drop^floor(epoch / epochs_drop)\n```\n** we will drop learning rate after every 3 epochs **"},{"metadata":{"_uuid":"9ec461e5a118bf591494c0f03f37636ae777c27e","trusted":true,"collapsed":true},"cell_type":"code","source":"import math\ndef step_decay(epoch):\n \n \n initial_lrate=0.1\n drop=0.6\n epochs_drop = 3.0\n lrate= initial_lrate * math.pow(drop, \n math.floor((1+epoch)/epochs_drop))\n return lrate\n \n\nlrate = LearningRateScheduler(step_decay)\ncallbacks_list = [ lrate]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa01817d6a9e90ce9b8f71b2051140033d5c87d9","trusted":true},"cell_type":"code","source":"history=model.fit(X_train, Y_train, validation_data=(X_valid, Y_valid),\n epochs=35,callbacks=callbacks_list,verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b4222a2b671f67957c8be08cd2532b51893fbee"},"cell_type":"markdown","source":"### above we have not run our model with augmented data\n** we can run our model model with augmentated data like below **\n```\nmodel.fit_generator(datagen_train.flow(X_train, Y_train, batch_size=16), validation_data=(X_valid, Y_valid),\n epochs=10,steps_per_epoch=X_train.shape[0],callbacks=[checkpointer,lrate], verbose=1)\n \n ``` "},{"metadata":{"_uuid":"b3d8b58b5cd82efb398ff7bb70a92238761a2407"},"cell_type":"markdown","source":"## model Evaluation"},{"metadata":{"_uuid":"fed251f0a09299a051a602e2e33d0fe0d235f59c"},"cell_type":"markdown","source":"## 4.1 plotting training and validation loss"},{"metadata":{"_uuid":"6963e128c3ab1153857acbaf98ea45124c1ef0a8","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nloss = history.history['loss']\nval_loss = history.history['val_loss']\nepochs = range(1, len(loss) + 1)\nplt.plot(epochs, loss, color='red', label='Training loss')\nplt.plot(epochs, val_loss, color='green', label='Validation loss')\nplt.title('Training and validation loss')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0770e52643a8025f11f5fd089f952246aa003a0a"},"cell_type":"markdown","source":"## 4.2 plotting training and validation accuracy"},{"metadata":{"_uuid":"ac9371e056a345a67c02a73120777919f5408839","trusted":true},"cell_type":"code","source":"acc = history.history['acc']\nval_acc = history.history['val_acc']\nplt.plot(epochs, acc, color='red', label='Training acc')\nplt.plot(epochs, val_acc, color='green', label='Validation acc')\nplt.title('Training and validation accuracy')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92b4c43e39e88cc03336714092410f5ad0a463be","trusted":true},"cell_type":"code","source":"print(\"on valid data\")\npred1=model.evaluate(X_valid,Y_valid)\nprint(\"accuaracy\", str(pred1[1]*100))\nprint(\"Total loss\",str(pred1[0]*100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d7d1158f4191848b65b4a8149e195762e1a29b8"},"cell_type":"markdown","source":"## Visualize CNN Layers"},{"metadata":{"trusted":true,"_uuid":"1af8d81b0b5ed3078581bfa909d04dfd136e84f2","collapsed":true},"cell_type":"code","source":"from keras.models import Model\nlayer_outputs = [layer.output for layer in model.layers]\nactivation_model = Model(inputs=model.input, outputs=layer_outputs)\nactivations = activation_model.predict(X_train[10].reshape(1,28,28,1))\n \ndef display_activation(activations, col_size, row_size, act_index): \n activation = activations[act_index]\n activation_index=0\n fig, ax = plt.subplots(row_size, col_size, figsize=(row_size*2.5,col_size*1.5))\n for row in range(0,row_size):\n for col in range(0,col_size):\n ax[row][col].imshow(activation[0, :, :, activation_index], cmap='gray')\n activation_index += 1\n \n \n ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3250896f767195f3141ecc5a108650d993aae14b"},"cell_type":"markdown","source":"### Displaying original Image"},{"metadata":{"trusted":true,"_uuid":"72660c765e4b95ac978f6c990219e771a98943c7"},"cell_type":"code","source":"plt.imshow(X_train[10][:,:,0]);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc3bdfa0bed1d0c43a402f7469b07a403c30092d"},"cell_type":"markdown","source":"### Desplaying above image after layer 2 .\n** layer 1 is input layer **."},{"metadata":{"trusted":true,"_uuid":"3210f437a8e8050c0fd465ba8fe8d629abfee01f"},"cell_type":"code","source":"display_activation(activations, 8, 8, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0ca318b7147bf6838300492d4cf99c099129d3a"},"cell_type":"markdown","source":"### Displaying output of layer 4"},{"metadata":{"trusted":true,"_uuid":"1d0110e63cf04085b685f061140b3ccd0198e1b5"},"cell_type":"code","source":"display_activation(activations, 8, 8, 3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2ac77a1befff899048706bc61128dbd00a1d2d3"},"cell_type":"markdown","source":"## Displaying output of layer 8"},{"metadata":{"trusted":true,"_uuid":"959873fcb8522397a9f47d976c2b26f27837a2ea"},"cell_type":"code","source":"display_activation(activations, 8, 8, 7)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f482a3649b879220b53b6c51c224d292fb03c284","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nY_prediction = model.predict(X_valid)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_prediction,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_valid,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02603218e4169d1c7d9870eeef9ef332d5f39876"},"cell_type":"markdown","source":"### 4.3 confusion matrix "},{"metadata":{"_uuid":"74a3439aed53d2c3243f529d2e8b10c844cd1e0d","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,8))\nsns.heatmap(confusion_mtx, annot=True, fmt=\"d\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14e0836d226825e9841ae46ee0de49fd97dc83cc","trusted":true,"collapsed":true},"cell_type":"code","source":"# predict results\nresults = model.predict(test)\n\n# select the indix with the maximum probability\nresults = np.argmax(results,axis = 1)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d6f82264d238ecfe3db811f267b9a375b7d77e6","trusted":true,"collapsed":true},"cell_type":"code","source":"submissions=pd.DataFrame({\"ImageId\": list(range(1,len(results)+1)),\n \"Label\": results})\nsubmissions.to_csv(\"re2-submission.csv\", index=False, header=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"683dc41e6ede282dcf88e4a8cdf6b433e1dd4e5e"},"cell_type":"markdown","source":"Refrences \n1.[visualize-convolutional-neural-network ](http://www.codeastar.com/visualize-convolutional-neural-network/)\n
\n2.[learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning ](https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1)\n "},{"metadata":{"_uuid":"4bd0253845fcc33bcd24bbd00c20be92ee88280a","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}