{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Digit-Recognition by Convolutional Neural Networks (CNN)"},{"metadata":{},"cell_type":"markdown","source":"### 1. Background"},{"metadata":{},"cell_type":"markdown","source":"#### 1.1 Artificial neural networks"},{"metadata":{},"cell_type":"markdown","source":"[Artificial neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network) are computing systems that are inspired by, but not identical to, biological neural networks that constitute animal brains. Such systems \"learn\" to perform tasks by considering examples, generally without being programmed with task-specific rules."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"#### 1.2 Convolutional neural network CNN)"},{"metadata":{},"cell_type":"markdown","source":"In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. For example, in image recognition, they might learn to identify images that contain cats by analyzing example images that have been manually labeled as \"cat\" or \"no cat\" and using the results to identify cats in other images. They do this without any prior knowledge of cats, for example, that they have fur, tails, whiskers and cat-like faces. Instead, they automatically generate identifying characteristics from the examples that they process."},{"metadata":{},"cell_type":"markdown","source":"here is a simple example of a standard 2D CNN"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"input dimension is (w,n,3) - (wxn) image with 3 channels (R,G,B)"},{"metadata":{},"cell_type":"markdown","source":"#### 1.2.1 CNN Layers"},{"metadata":{},"cell_type":"markdown","source":"1.2.1.1 Convolutional Layers\n\nthe major block in CNN is Convolutional layers, which apply filtering to the input that results an activation, repeating applying the filter (kernel) on the input will create a feature map that summarizes the presence of detected features in the input."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"1.2.1.2 Pooling Layer\n\nPooling is required to down sample the detection of features in feature maps by summarizing the presence of features in patches of the feature map. there is many pooling methods one of the common pooling methods is 'max pooling', which summarize the max activated presence of a feature."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"1.2.1.3 Fully Connected Layer\n\nAfter the two previous steps in CNN process ends, breaking down the image into features, and analyzing them independently. The result of this process will be flatted and feeds into a fully connected neural network structure that drives the final classification decision."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"#### 1.3 MNIST"},{"metadata":{},"cell_type":"markdown","source":"[MNIST](https://www.kaggle.com/c/digit-recognizer) (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"#### 1.4 Images\nRGB images is stored as 3D numpy array (rows, columns, channels), rows is images height of the image and columns is the width of the image.\nChannels consists of Red, Green and Blue components of each individual pixel. for example a (0,0,0) pixel is displayed as black, and a pixel whose color components are (255,255,255) is displayed as white."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"for our case, we will feed the CNN model with graysacle images, graysacle image is one in which the value of each pixel is a single sample representing only an amount of light.\nwhich will be stored as (rows, columns, 1) in our case it will be (28, 28, 1)"},{"metadata":{},"cell_type":"markdown","source":"### 2. Problem"},{"metadata":{},"cell_type":"markdown","source":"The goal is to correctly identify digits from a dataset of tens of thousands of handwritten images."},{"metadata":{},"cell_type":"markdown","source":"### 3. Import Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport keras\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential, load_model\nfrom keras.layers import Conv2D, MaxPooling2D, Dense, Dropout, Flatten, Activation, MaxPool2D\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom sklearn.model_selection import train_test_split\nfrom keras.optimizers import Adam,RMSprop\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nfrom sklearn.metrics import confusion_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4. Gathering data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5. EDA - Exploratory data analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"42,000 images to train and learn from."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"as you can see we have label wich define the actual digit and 784pixels - pixels[0-783].\ni will seperate train data to (X, y) - y will be label only, and X will be the whole data without the label."},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train.drop('label', axis=1)\ny = train['label']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"28,000 images to test"},{"metadata":{},"cell_type":"markdown","source":"Let us count the labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train['label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6. Preprocessing that data"},{"metadata":{"trusted":true},"cell_type":"code","source":"X.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us take a look to the first row"},{"metadata":{"trusted":true},"cell_type":"code","source":"first_row = X.iloc[0].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"reshape it to 28x28"},{"metadata":{"trusted":true},"cell_type":"code","source":"first_mat = first_row.values.reshape(28,28)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"now we can plot that image"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(first_mat)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"it looks like the digit ```1```"},{"metadata":{},"cell_type":"markdown","source":"let us plot the first 10 images"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nfor i in range(10):\n plt.subplot(2,5,i+1)\n plt.xticks([])\n plt.yticks([])\n plt.grid(False)\n plt.imshow(X.iloc[i].values.reshape(28,28))\n plt.xlabel(y[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 7. Neural network model"},{"metadata":{},"cell_type":"markdown","source":"#### 7.1 Define Variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"input_shape = (28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let see how many unique labels we have"},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_labels = y.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_labels = len(unique_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 7.2 Train/Test Split"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 7.4 Reshaping and Scaling data"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.values.reshape(-1, 28, 28, 1)\nX_test = X_test.values.reshape(-1, 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train / 255.\nX_test = X_test / 255.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 7.5 Build Neural Network Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Model Definition\nmodel = Sequential()\n\n# add 32 convolution filters used each of size 5x5 with relu activation\nmodel.add(Conv2D(filters=32, kernel_size=(5, 5), padding='Valid', activation='relu', input_shape=(28, 28, 1)))\n\n\n# add another 32 convolution filters used each of size 3x3 with relu activation\nmodel.add(Conv2D(filters=32, kernel_size=(3, 3), padding='Same', activation='relu'))\n\n# adding pooling layer with a MaxPool2D filter of size 2x2 summarize the presence of features\n# in patches of the feature map.\nmodel.add(MaxPool2D(pool_size=(2, 2)))\n\n\n# turn on and off neurons randomly for reducing interdependent learning amongst the neurons.\nmodel.add(Dropout(0.2))\n\n# add 64 convolution filters used each of size 5x5 with relu activation\nmodel.add(Conv2D(filters=64, kernel_size=(5, 5), padding='Valid', activation='relu'))\n\n# add 64 convolution filters used each of size 3x3 with relu activation\nmodel.add(Conv2D(filters=64, kernel_size=(3, 3), padding='Same', activation='relu'))\n\n# adding pooling layer with a MaxPool2D filter of size 2x2 summarize the presence of features\n# in patches of the feature map.\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=(2, 2)))\n\n# turn on and off neurons randomly for reducing interdependent learning amongst the neurons.\nmodel.add(Dropout(0.2))\n\n# # Flattens the data.\nmodel.add(Flatten())\n\n# add densely-connected NN layer, to fully connected to drives the final classification decision.\nmodel.add(Dense(519, activation=\"relu\"))\n\n# turn on and off neurons randomly for reducing interdependent learning amongst the neurons.\nmodel.add(Dropout(0.5))\n\n# output a softmax to let the output to be interpreted as probabilities\nmodel.add(Dense(10, activation=\"softmax\"))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's display the architecture of our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 7.6 Model Compiling and Training"},{"metadata":{},"cell_type":"markdown","source":"Before training the model, we need to compile :\n* Loss function — This measures how accurate the model is during training. You want to minimize this function to \"steer\" the model in the right direction.\n* Optimizer —This is how the model is updated based on the data it sees and its loss function.\n* Metrics —Used to monitor the training and testing steps. The following example uses accuracy, the fraction of the images that are correctly classified"},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=1e-3), metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss='categorical_crossentropy',\n optimizer=keras.optimizers.RMSprop(),\n metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"because we are using the ```categorical_crossentropy``` loss method, we need to convert ```y_train```, ```y_test``` using one hot encoder."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train, num_classes = num_labels)\ny_test = to_categorical(y_test, num_classes = num_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 7.7 Reduce Learning Rate (LR)"},{"metadata":{},"cell_type":"markdown","source":"Reduce learning rate when a metric has stopped improving.\nModels often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=2, factor=0.5, min_lr=0.0000001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 7.8 Image proccessing"},{"metadata":{},"cell_type":"markdown","source":"because we want to reduce over-fitting, i will use Data Augmentation technique \nData augmentation Applies a transformation to an image according to given parameters for example rotates, shears, zooms and other transformations to the image and make the model learns to generalize and not remember specific data. If the model overfits, it will perform very well on the images that it already knows but will fail if new images are given to it."},{"metadata":{"trusted":true},"cell_type":"code","source":"img_data_gen = ImageDataGenerator(\n featurewise_center=False,\n samplewise_center=False,\n featurewise_std_normalization=False,\n samplewise_std_normalization=False,\n zca_whitening=False,\n rotation_range=10,\n zoom_range=0.1,\n width_shift_range=0.1,\n height_shift_range=0.1,\n horizontal_flip=False,\n vertical_flip=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# epochs - One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE\nnum_epochs = 1 # replace it to 30\n# batch size -Total number of training examples present in a single batch.\nbatch_size = 64\n\ntrain_generator = img_data_gen.flow(X_train, y_train, batch_size=batch_size)\ntest_generator = img_data_gen.flow(X_test, y_test, batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save the model to disk\nmodel.save('MNIST-1.h5')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"start train"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\nhistory = model.fit_generator(train_generator,\n epochs=num_epochs,\n validation_data=test_generator,\n callbacks=[reduce_lr])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 8. Model Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"score = model.evaluate(X_test, y_test, verbose=0)\nprint('Test loss:', score[0])\nprint('Test accuracy:', score[1]) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Confusion matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_true = [np.argmax(i) for i in y_test]\npredictions = model.predict(X_test)\ny_pred = [np.argmax(i) for i in predictions]\nplt.figure(figsize=(15,8))\nsns.heatmap(confusion_matrix(y_true, y_pred), cmap=\"coolwarm\", annot=True , fmt=\"d\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"accuracy vs. validation accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'], label='accuracy')\nplt.plot(history.history['val_accuracy'], label = 'val_accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.ylim([0.5, 1])\nplt.legend(loc='lower right')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us take a loot to the first prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the prediction is an array of 10 elements, each element representing the model \"confidence\" value for the corresponding class.\nand to get the correct prediction we need to get the Max \"confidence\" value for each input"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.argmax(predictions[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model says that the first digit is ```3```, let see the acutal class"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.argmax(y_test[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Cool the model is right !"},{"metadata":{},"cell_type":"markdown","source":"Let us see the first 10 prediction images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_image(i, predictions_array, true_label, img):\n predictions_array, true_label, img = predictions_array, true_label[i], img[i]\n plt.grid(False)\n plt.xticks([])\n plt.yticks([])\n\n plt.imshow(img, cmap=plt.cm.binary)\n\n predicted_label = np.argmax(predictions_array)\n if predicted_label == true_label:\n color = 'blue'\n else:\n color = 'red'\n\n plt.xlabel(\"{} {:2.0f}% ({})\".format(predicted_label,\n 100*np.max(predictions_array),\n true_label),\n color=color)\n\ndef plot_value_array(i, predictions_array, true_label):\n predictions_array, true_label = predictions_array, true_label[i]\n plt.grid(False)\n plt.xticks(range(10))\n plt.yticks([])\n thisplot = plt.bar(range(10), predictions_array, color=\"#777777\")\n plt.ylim([0, 1])\n predicted_label = np.argmax(predictions_array)\n\n thisplot[predicted_label].set_color('red')\n thisplot[true_label].set_color('blue')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_rows = 5\nnum_cols = 3\nnum_images = num_rows*num_cols\nplt.figure(figsize=(2*2*num_cols, 2*num_rows))\nfor i in range(num_images):\n plt.subplot(num_rows, 2*num_cols, 2*i+1)\n plot_image(i, predictions[i], np.argmax(np.array(y_test), axis=1), X_test.reshape(-1,28,28))\n plt.subplot(num_rows, 2*num_cols, 2*i+2)\n plot_value_array(i, predictions[i], np.argmax(np.array(y_test), axis=1))\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let display the model errors"},{"metadata":{"trusted":true},"cell_type":"code","source":"errors = pd.DataFrame(np.argmax(y_test, axis=1), columns=['label'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"errors.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"errors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"errors['predictions'] = y_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"errors.loc[errors['label'] - errors['predictions'] != 0, 'error'] = 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"errors[errors['error']==1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_errors = len(errors[errors['error']==1].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"number of errors is: {}\".format(num_errors))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"err_index = errors[errors['error']==1].index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nfor i in range(10):\n err_index = errors[errors['error']==1].index[i]\n plt.subplot(2,5,i+1)\n plt.xticks([])\n plt.yticks([])\n plt.grid(False)\n plt.imshow(X_test[err_index].reshape(28,28))\n plt.xlabel(\"ture is {}, predicted as {}\".format(np.argmax(y_test[err_index]), y_pred[err_index]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest = test / 255\ntest = test.values.reshape(-1, 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_predictions = model.predict(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_predictions = list(map(lambda x : np.argmax(np.round(x)), final_predictions))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_predictions[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_labels = pd.Series(final_predictions, name=\"Label\")\nimage_id = pd.Series(range(1, len(predicted_labels)+1),name=\"ImageId\")\n\nresults = pd.concat([image_id,predicted_labels],axis=1)\n\nresults.to_csv(\"MNIST.csv\",index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}