{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Introduction\nI have done a notebook while following a [Neural Network Web book by Micheal Nelson](http://neuralnetworksanddeeplearning.com). The notebook was [this one](https://www.kaggle.com/ankitdatascience/neural-network-for-digit-recogniser-from-scratch). \n\nNow I am trying the same implementation using keras (the first notebook has the neural network code from scratch).\n\nFew points on the notebook :-\n\n1. Here we have 42000 examples.\n2. We will create a validation dataset from 42000 training dataset provided.\n3. We will creating a multiplayer perceptron and then will use Convolutional Neural Network and compare the results."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n#Import from keras\nfrom keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.utils import np_utils\nfrom keras.layers.convolutional import Convolution2D, MaxPooling2D, Conv2D\n\n# Import from matplotlib\nimport matplotlib.pyplot as plt\n\n# Import from sklearn\nfrom sklearn.preprocessing import scale, StandardScaler\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Reading the kaggle dataset in our environment.\ndata = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndata.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First column in the dataset is the label. Rest of the columns are the pixel value."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Converting the dataframe to a numpy array.\ndata_matrix = np.array(data.drop(['label'], axis = 1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot a few examples from the dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting few examples\nplt.subplot(221)\n# Reshaping the pixel vector to a 28 X 28 image pixel and plotting\nplt.imshow(data_matrix[1].reshape(28,28), cmap = plt.get_cmap('gray')) \nplt.subplot(222)\nplt.imshow(data_matrix[2].reshape(28,28), cmap = plt.get_cmap('gray'))\nplt.subplot(223)\nplt.imshow(data_matrix[3].reshape(28,28), cmap = plt.get_cmap('gray'))\nplt.subplot(224)\nplt.imshow(data_matrix[6].reshape(28,28), cmap = plt.get_cmap('gray'))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Scaling the data.\n\nOne thing, you will notice here is that the data is not scaled. So before doing anything let's scale our predictor space."},{"metadata":{"trusted":true},"cell_type":"code","source":"X = scale(data.drop('label', axis = 1).values)\ny = np.array(data['label'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train and test split of the data\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 2000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting the random seed\nseed = 7\nnp.random.seed(seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we will create dummy variables for each of the response class (one hot encode)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# one hot encode outputs\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Multilayer Perceptron"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the baseline model\ndef baseline_model():\n    #Create Model\n    model = Sequential()\n    model.add(Dense(784, input_dim = 784, kernel_initializer = 'normal', activation = 'relu')) # first hidden layer\n    model.add(Dense(num_classes, kernel_initializer = 'normal', activation = 'softmax')) # output layer\n    # Compile Model\n    model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build the model\nmodel = baseline_model()\n# Fit the model\nmodel.fit(X_train, y_train, validation_data = (X_test, y_test), \n          epochs = 20, batch_size = 200, verbose = 2)\n\n# Evaluation of the model\nscores = model.evaluate(X_test, y_test, verbose = 0)\nprint(\"Baseline Error: %.2f%%\" %(100- scores[1] * 100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have an error of 2.45% only using a simple multilayer perceptron model. Now let's try a simple CNN model.\n\n## Simple Convolutional Neural Network Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.reshape(X_train.shape[0], 28, 28).astype(\"float32\")\nX_test = X_test.reshape(X_test.shape[0], 28, 28).astype(\"float32\")\n\n# reshape to be [samples][channels][width][height]\nX_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype(\"float32\")\nX_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype(\"float32\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# one hot encode outputs\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Our CNN structure\n\n1. The first hidden layer is a convolution layer. The layer has 32 feature map, which with the size of 5 X 5 and a rectifier activation function.\n\n2. A pooling layer with a pool size of 2 X 2.\n\n3. The next layer is a regularization layer using dropout.\n\n4. Next is a layer that converts the 2D matrix data to a vector called Flatten.\n\n5. Next a fully connected layer with 128 neurons and rectifier activation function is used.\n\n6. The output layer has 10 neurons for the 10 classes and a softmax activation function."},{"metadata":{"trusted":true},"cell_type":"code","source":"def baseline_model():\n    model = Sequential()\n    model.add(Conv2D(32, (5,5), padding='valid', input_shape = (28,28,1),\n                     activation = 'relu'))\n    model.add(MaxPooling2D(pool_size = (2,2)))\n    model.add(Dropout(0.2))\n    model.add(Flatten())\n    model.add(Dense(128, activation = 'relu'))\n    model.add(Dense(num_classes, activation = 'softmax'))\n    \n    model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build the model\nmodel = baseline_model()\n# Fit the model\nmodel.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 10, batch_size = 200, verbose = 2)\n\n# Evaluation of the model\nscores = model.evaluate(X_test, y_test, verbose = 0)\nprint(\"Baseline Error: %.2f%%\" %(100- scores[1] * 100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The error is smaller than the Multilayer perceptron. Now we have only 1.65% error. Let's try a bigger convolutional NN.\n\n## Large Convolutional Neural Network\n\n#### Our CNN structure\n\n1. The first hidden layer is a convolution layer. The layer has 30 feature map, which with the size of 5 X 5 and a rectifier activation function.\n\n2. A pooling layer with a pool size of 2 X 2.\n\n3. A convolutional layer with 15 feature maps of size 3 X 3.\n\n4. A pooling layer with a pool size of 2 X 2.\n\n5. The next layer is a regularization layer using dropout.\n\n6. Next is a layer that converts the 2D matrix data to a vector called Flatten.\n\n7. Next a fully connected layer with 128 neurons and rectifier activation function is used.\n\n8. Fully connected layer with 50 neurons and rectifier activation.\n\n9. The output layer has 10 neurons for the 10 classes and a softmax activation function."},{"metadata":{"trusted":true},"cell_type":"code","source":"def larger_model():\n    model = Sequential()\n    model.add(Conv2D(30, (5,5), padding='valid', input_shape = (28,28,1),\n                     activation = 'relu'))\n    model.add(MaxPooling2D(pool_size = (2,2)))\n    model.add(Conv2D(15, (3,3), activation = 'relu'))\n    model.add(MaxPooling2D(pool_size = (2,2)))\n    model.add(Dropout(0.2))\n    model.add(Flatten())\n    model.add(Dense(128, activation = 'relu'))\n    model.add(Dense(50, activation = 'relu'))\n    model.add(Dense(num_classes, activation = 'softmax'))\n    \n    model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build the model\nmodel = larger_model()\n# Fit the model\nmodel.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 10, batch_size = 200, verbose = 2)\n\n# Evaluation of the model\nscores = model.evaluate(X_test, y_test, verbose = 0)\nprint(\"Baseline Error: %.2f%%\" %(100- scores[1] * 100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have a slight improvement over the simple CNN."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}