{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":9432,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n## Convolutional Neural Network \n* CNN is used for image classification, object detection \n* \"gec2\"","metadata":{"_uuid":"caf4b9f7fcae153ac1d4c0246b1defc9d9d30776"}},{"cell_type":"markdown","source":"\n### What is Convolution Operation?\n* We have some image and feature detector(3*3)\n* Feature detector does not need to be 3 by 3 matrix. It can be 5 by 5 or 7 by 7.\n* Feature detector = kernel = filter\n* Feauture detector detects features like edges or convex shapes. Example, if out input is dog, feature detector can detect features like ear or tail of the dog.\n* feature map = conv(input image, feature detector). Element wise multiplication of matrices.\n* feature map = convolved feature\n* Stride = navigating in input image.\n* We reduce the size of image. This is important bc code runs faster. However, we lost information. \n* We create multiple feature maps bc we use multiple feature detectors(filters).\n* Lets look at gimp. Edge detect: [0,10,0],[10,-4,10],[0,10,0]\n* \"gec\"\n* After having convolution layer we use ReLU to break up linearity. Increase nonlinearity. Because images are non linear.\n* \"RELU\"","metadata":{"_uuid":"682cb6f5144794038c0bef1dc0814dde0f7fe6c0"}},{"cell_type":"markdown","source":"\n### Same Padding\n* As we keep applying conv layers, the size of the volume will decrease faster than we would like. In the early layers of our network, we want to preserve as much information about the original input volume so that we can extract those low level features.\n* input size and output size are same.\n* \"padding\"","metadata":{"_uuid":"3af8fb49243719ad70016b64cd4dff6e62c413d9"}},{"cell_type":"markdown","source":"\n### Max Pooling\n* It makes down-sampling or sub-sampling (Reduces the number of parameters)\n* It makes the detection of features invariant to scale or orientation changes.\n* It reduce the amount of parameters and computation in the network, and hence to also control overfitting. \n* \"maxpool\"","metadata":{"_uuid":"6694b8ea366f974d6bd055a2915a5dc06f7a96bb"}},{"cell_type":"markdown","source":"\n### Flattening\n* \"flattenigng\"","metadata":{"_uuid":"c2fb971fc473c9104a27975242a36f8e9183742c"}},{"cell_type":"markdown","source":"\n### Full Connection\n* Neurons in a fully connected layer have connections to all activations in the previous layer\n* Artificial Neural Network\n* \"fullyc\"","metadata":{"trusted":true,"_uuid":"f437407cf18fdfacf8626baa5e76ed5d64be1a11"}},{"cell_type":"markdown","source":"\n## Implementing with Keras","metadata":{"trusted":true,"_uuid":"7b290eb3f3b111d6841f8e4093277be1bde05078"}},{"cell_type":"markdown","source":"\n### Create Model\n* conv => max pool => dropout => conv => max pool => dropout => fully connected (2 layer)\n* Dropout: Dropout is a technique where randomly selected neurons are ignored during training\n* \"dropout\"","metadata":{"trusted":true,"_uuid":"fd9feb4c4dadbb8a5cc8a32b5ef582d41b5c0698"}},{"cell_type":"code","source":"# \nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.optimizers import RMSprop,Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\n\nmodel = Sequential()\n#\nmodel.add(Conv2D(filters = 8, kernel_size = (5,5),padding = 'Same', \n activation ='relu', input_shape = (28,28,1)))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n#\nmodel.add(Conv2D(filters = 16, kernel_size = (3,3),padding = 'Same', \n activation ='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n# fully connected\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation = \"softmax\"))","metadata":{"trusted":true,"_uuid":"c441d7b3852cee5d3636272d4da2f96b169f81ac"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Define Optimizer \n* Adam optimizer: Change the learning rate\n","metadata":{"_uuid":"9a432df491777ca0019db6f6b972581f2c5bd9f9"}},{"cell_type":"code","source":"# Define the optimizer\noptimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999)","metadata":{"trusted":true,"_uuid":"607a02b42636e3115a1ac7a8edcadf61cf5ea1b0"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Compile Model\n* categorical crossentropy\n* We make binary cross entropy at previous parts and in machine learning tutorial\n* At this time we use categorical crossentropy. That means that we have multi class.\n* \"cce\"\n","metadata":{"_uuid":"91bf542a979c5b191c534876186bf31e70ec7f06"}},{"cell_type":"code","source":"# Compile the model\nmodel.compile(optimizer = optimizer , loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])","metadata":{"trusted":true,"_uuid":"0d1eefc68470b4cdcec04c2570651da3d97676d0"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Epochs and Batch Size\n* Say you have a dataset of 10 examples (or samples). You have a **batch size** of 2, and you've specified you want the algorithm to run for 3 **epochs**. Therefore, in each epoch, you have 5 **batches** (10/2 = 5). Each batch gets passed through the algorithm, therefore you have 5 iterations **per epoch**.\n* reference: https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks","metadata":{"_uuid":"d5d8fea4cbb9b53cf1bb5089357a33dc179ee981"}},{"cell_type":"code","source":"epochs = 10 # for better result increase the epochs\nbatch_size = 250","metadata":{"trusted":true,"_uuid":"a237feb5e53ecbc8799101cb6e699877faafde77"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Data Augmentation\n* To avoid overfitting problem, we need to expand artificially our handwritten digit dataset\n* Alter the training data with small transformations to reproduce the variations of digit.\n* For example, the number is not centered The scale is not the same (some who write with big/small numbers) The image is rotated.\n* \"augment\"\n \n","metadata":{"_uuid":"e87c60e45759e0c52c2b72ee562fdb964c8008d5"}},{"cell_type":"code","source":"# data augmentation\ndatagen = ImageDataGenerator(\n featurewise_center=False, # set input mean to 0 over the dataset\n samplewise_center=False, # set each sample mean to 0\n featurewise_std_normalization=False, # divide inputs by std of the dataset\n samplewise_std_normalization=False, # divide each input by its std\n zca_whitening=False, # dimesion reduction\n rotation_range=5, # randomly rotate images in the range 5 degrees\n zoom_range = 0.1, # Randomly zoom image 10%\n width_shift_range=0.1, # randomly shift images horizontally 10%\n height_shift_range=0.1, # randomly shift images vertically 10%\n horizontal_flip=False, # randomly flip images\n vertical_flip=False) # randomly flip images\n\ndatagen.fit(X_train)","metadata":{"trusted":true,"_uuid":"c339fa7d06e9b73a519ca661bed19de482707d1f"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Fit the model","metadata":{"_uuid":"e42e43f1b3b9e6da0d99dafff8f4a4b514a07f4c"}},{"cell_type":"code","source":"# Fit the model\nhistory = model.fit_generator(datagen.flow(X_train,Y_train, batch_size=batch_size),\n epochs = epochs, validation_data = (X_val,Y_val), steps_per_epoch=X_train.shape[0] // batch_size)","metadata":{"trusted":true,"_uuid":"4b2957bb8976a25cdbbbdc3110d68c5035a9773c"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### Evaluate the model\n* Test Loss visualization\n* Confusion matrix\n","metadata":{"_uuid":"debd5424728e11aa30a8513ac7b4f7377193a2da"}},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nplt.plot(history.history['val_loss'], color='b', label=\"validation loss\")\nplt.title(\"Test Loss\")\nplt.xlabel(\"Number of Epochs\")\nplt.ylabel(\"Loss\")\nplt.legend()\nplt.show()","metadata":{"trusted":true,"_uuid":"180a06f7ae01e69117c6c8258411cfe1b9b7b991"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# confusion matrix\nimport seaborn as sns\n# Predict the values from the validation dataset\nY_pred = model.predict(X_val)\n# Convert predictions classes to one hot vectors \nY_pred_classes = np.argmax(Y_pred,axis = 1) \n# Convert validation observations to one hot vectors\nY_true = np.argmax(Y_val,axis = 1) \n# compute the confusion matrix\nconfusion_mtx = confusion_matrix(Y_true, Y_pred_classes) \n# plot the confusion matrix\nf,ax = plt.subplots(figsize=(8, 8))\nsns.heatmap(confusion_mtx, annot=True, linewidths=0.01,cmap=\"Greens\",linecolor=\"gray\", fmt= '.1f',ax=ax)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix\")\nplt.show()","metadata":{"trusted":true,"_uuid":"6586e37bd470db822086e191a90388e7175d504f"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## Conclusion\n* http://scs.ryerson.ca/~aharley/vis/conv/flat.html\n* HW\n* If you have any question I will be very happy to hear it.","metadata":{"_uuid":"9409bc6a7e905d5887c8eec8d056c9f015bf4878"}}]}