{"cells":[{"metadata":{},"cell_type":"markdown","source":"# AI leads the way to computer vision\n\nMost traditional methods of analysing and processing of images and video have been quickly surpassed by the power of artificial intelligence and deep learning. Convolutional Neural Networks, that work on the same basic principal as the neurons in the visual cortex, can analyse images and identify objects in them with staggering accuracy.\n\nThis is why for the solution to the problem of recognizing handwritten digits, CNNs are the best approach to use."},{"metadata":{},"cell_type":"markdown","source":"## Importing the libraries\n\nFirst we need to import the basic libraries that we will use for preprocessing and visualization of the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the data\n\nNext, we load the training and test datasets into separate variables using *Pandas*."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"train_data = pd.read_csv('../input/digit-recognizer/train.csv')\ntest_data = pd.read_csv('../input/digit-recognizer/test.csv')\n\ntrain_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the training dataset consists of one column for the label (which represents what digit was written), and 784 columns for each pixel of the image that contains the handwritten digit. To be able to use a 2D Convolutional Neural Network, we need to reshape the data so we can get the actual image of the digit."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reshaping the data\n\nThe current shape of the training dataset given above tells us that there are 42000 images which are represented by 784 pixels (plus one column for the label). That means that in order to get a perfect square image, the dimensions of the pixel need to be ($\\sqrt{784}, \\sqrt{784}$), or (28, 28) pixels. "},{"metadata":{},"cell_type":"markdown","source":"## Separating the label column\n\nIn order to reshape the data, first we must separate the labels from the feature map."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels = train_data['label']\ntrain_data = train_data.drop('label', axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reshaping the datasets\n\nAs mentioned previously, the required shape of each image is 28x28 pixels, and we can easily achieve that using the *NumPy* function *reshape*."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_data = train_data.values[:, :, np.newaxis].reshape(-1, 28, 28)\ntest_data = test_data.values[:, :, np.newaxis].reshape(-1, 28, 28)\n\ntrain_data.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can observe that the training data now has a shape of (42000, 28, 28), which represents the 42000 images of handwritten digits, with dimensions 28x28."},{"metadata":{},"cell_type":"markdown","source":"# Visualization of the data\n\nBefore moving on, we can visualize the images of the digits to get a better understanding of how they look. This can be easily done using the *matplotlib* library.\n\nWe will plot 9 random images from the dataset that will be picked at random."},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(157)\n\nidxs = np.random.randint(0, train_data.shape[0], 9)\n\nfig, ax = plt.subplots(3, 3, figsize=(10, 10))\n\nax[0, 0].imshow(train_data[idxs[0], :, :], cmap='Greys')\nax[0, 0].set_title('Handwritten number ' + str(train_labels[idxs[0]]))\n\nax[0, 1].imshow(train_data[idxs[1], :, :], cmap='Greys')\nax[0, 1].set_title('Handwritten number ' + str(train_labels[idxs[1]]))\n\nax[0, 2].imshow(train_data[idxs[2], :, :], cmap='Greys')\nax[0, 2].set_title('Handwritten number ' + str(train_labels[idxs[2]]))\n\nax[1, 0].imshow(train_data[idxs[3], :, :], cmap='Greys')\nax[1, 0].set_title('Handwritten number ' + str(train_labels[idxs[3]]))\n\nax[1, 1].imshow(train_data[idxs[4], :, :], cmap='Greys')\nax[1, 1].set_title('Handwritten number ' + str(train_labels[idxs[4]]))\n\nax[1, 2].imshow(train_data[idxs[5], :, :], cmap='Greys')\nax[1, 2].set_title('Handwritten number ' + str(train_labels[idxs[5]]))\n\nax[2, 0].imshow(train_data[idxs[6], :, :], cmap='Greys')\nax[2, 0].set_title('Handwritten number ' + str(train_labels[idxs[6]]))\n\nax[2, 1].imshow(train_data[idxs[7], :, :], cmap='Greys')\nax[2, 1].set_title('Handwritten number ' + str(train_labels[idxs[7]]))\n\nax[2, 2].imshow(train_data[idxs[8], :, :], cmap='Greys')\nax[2, 2].set_title('Handwritten number ' + str(train_labels[idxs[8]]))\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, some of these numbers are written with very poor handwritting that can even confuse a human, meaning the CNN will have quite a difficult task to acomplish."},{"metadata":{},"cell_type":"markdown","source":"# Classification"},{"metadata":{},"cell_type":"markdown","source":"## Preparing the data for classification\n\nThere are a couple of preprocessing steps we need to complete before the training of the model:\n\n* First, the training labels need to be one-hot encoded so we can use a softmax output layer at the end of the network.\n* Next, we need to normalize the values of the data between 0 and 1, which leads to an improvement of the results of the classification.\n* We also need to reshape the data again by adding a new fourth axis which represents the color channel of the image.\n* Finally, we need to create a new dataset which will be used for validation during the training phase of the network."},{"metadata":{},"cell_type":"markdown","source":"### One-hot encoding the labels\n\nOne-hot encodin the labels so we can use the *softmax* output layer with the *categorical crossentropy* loss function."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelBinarizer\n\nencoder = LabelBinarizer()\ntrain_labels = encoder.fit_transform(train_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Normalization\n\nScaling the values of the pixels of the images in range between 0 and 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data.astype('float32') / 255\ntest_data = test_data.astype('float32') / 255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reshaping the data\n\n2D convolutional neural networks require the images to be 3D, where one of the dimensions represents the colour channel. Seeing as our images are black and white, we simply add an empty new axis at the end of the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data[:, :, :, np.newaxis]\ntest_data = test_data[:, :, :, np.newaxis]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating the validation dataset\n\nDuring the training, the network requires a validation dataset to be able to calculate its performace. We can create this dataset by setting aside 10% of the training data to be used as validation, using the *train_test_split* function from the *sklearn* library."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_val, y_train, y_val = train_test_split(train_data, train_labels, test_size=0.1, random_state=157, stratify=train_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation\n\nBefore training, a good practice to use (especially when working with image data) is to perform data augmentation. This can be easily done using the *ImageDataGenerator* function from the *Keras* library.\n\nWe augment the images in the training set by performing certain transformations upon them, such as:\n\n* Rotating by 10 degrees\n* Zooming by 10%\n* Horizontal and vertical shift of the image by 10%\n\nFlipping and whitening of the images is not performed due to the possibilty of hurting the final results rather than aiding them."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n\ndata_generator = ImageDataGenerator(\n        featurewise_center=False,\n        samplewise_center=False,\n        featurewise_std_normalization=False,\n        samplewise_std_normalization=False,\n        zca_whitening=False,\n        horizontal_flip=False,\n        vertical_flip=False,\n        rotation_range=10,\n        zoom_range = 0.1,\n        width_shift_range=0.1,\n        height_shift_range=0.1\n        )\n\n\ndata_generator.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building the Keras CNN model"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import BatchNormalization, Activation, Dropout, Dense, MaxPooling2D, Conv2D, Flatten\nfrom keras.initializers import he_uniform, glorot_uniform\nfrom keras.activations import relu\nfrom keras.constraints import maxnorm\n\ninit_relu = he_uniform(seed=157)        # used for ReLU\ninit_tanh = glorot_uniform(seed=157)    # used for tanh and softmax\n\nmodel = Sequential()\n\n# input and first Convolutional layer\nmodel.add(Conv2D(name='Conv_1', input_shape=(28, 28, 1), filters=64, kernel_size=(3, 3), padding='same', kernel_initializer=init_relu, kernel_constraint=maxnorm(3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(relu))\n\n\n# second Convolutional layer and first MaxPooling layer\nmodel.add(Conv2D(name='Conv_2', filters=64, kernel_size=(3, 3), padding='same', kernel_initializer=init_relu, kernel_constraint=maxnorm(3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(relu))\nmodel.add(MaxPooling2D(name='Pooling_1', pool_size=(2, 2), strides=2))\nmodel.add(Dropout(0.20))\n\n\n# fourth Convolutional layer\nmodel.add(Conv2D(name='Conv_4', filters=32, kernel_size=(3, 3), padding='same', kernel_initializer=init_relu, kernel_constraint=maxnorm(3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(relu))\n\n\n# fifth Convolutional layer and second MaxPooling layer\nmodel.add(Conv2D(name='Conv_5', filters=32, kernel_size=(3, 3), padding='same', kernel_initializer=init_relu, kernel_constraint=maxnorm(3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(relu))\nmodel.add(MaxPooling2D(name='Pooling_2', pool_size=(2, 2), strides=2))\nmodel.add(Dropout(0.20))\n\n\n# first Dense layer\nmodel.add(Flatten())\nmodel.add(Dense(name='FC_1', units=512, kernel_initializer=init_relu, kernel_constraint=maxnorm(3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(relu))\nmodel.add(Dropout(0.20))\n\n# second Dense layer\nmodel.add(Dense(name='FC_2', units=128, kernel_initializer=init_relu, kernel_constraint=maxnorm(3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(relu))\n\n# softmax output Dense layer\nmodel.add(Dense(name='output', units=10, activation='softmax', kernel_initializer=init_tanh, kernel_constraint=maxnorm(3)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setting all random variables to a single seed for reproducibility"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport random\nimport tensorflow as tf\n\n# 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n\nos.environ[\"CUDA_DEVICE_ORDER\"] = \"PCI_BUS_ID\"\nos.environ[\"CUDA_VISIBLE_DEVICES\"] = \"\"\nos.environ['PYTHONHASHSEED'] = str(157)\n\n# 2. Set `python` built-in pseudo-random generator at a fixed value\nrandom.seed(157)\n\n# 3. Set `numpy` pseudo-random generator at a fixed value\nnp.random.seed(157)\n\n# 4. Set `tensorflow` pseudo-random generator at a fixed value\ntf.compat.v1.set_random_seed(157)\n\n# 5. Configure a new global `tensorflow` session\nfrom tensorflow.python.keras import backend as k\n\nsession_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,\n                                        inter_op_parallelism_threads=1)\n\nsess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\nk.set_session(sess)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training the network"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard, LearningRateScheduler\nfrom keras.optimizers import Adam\nfrom keras.models import save_model\n\n\nlr = 0.001\n\nopt = Adam(learning_rate=lr, amsgrad=True)\n\nmodel.compile(optimizer=opt,\n            loss='categorical_crossentropy',\n            metrics=['accuracy'])\n\ncheckpoint = ModelCheckpoint('neural_network_checkpoint_training.h5', monitor='val_loss', verbose=1, save_best_only=True, mode='min')\nlr_decay = LearningRateScheduler(lambda x: lr * 0.95 ** x)\n\ntensorboard = TensorBoard(log_dir='./logs',\n                          histogram_freq=0,\n                          write_graph=True,\n                          write_images=True)\n\nmodel.fit_generator(data_generator.flow(X_train, y_train, batch_size=64),\n        epochs=50,\n        validation_data=(X_val, y_val),\n        shuffle=True,\n        callbacks=[tensorboard, checkpoint, lr_decay],\n        verbose=1)\n\nsave_model(checkpoint.model, 'neural_network_latest_saved.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The final prediction and submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import load_model\n\nclf = load_model('neural_network_checkpoint_training.h5')\n\nprediction = clf.predict(test_data)\n\nprediction = pd.DataFrame(np.argmax(prediction, axis=-1), columns=['Label'])\nimg_idx = pd.DataFrame(np.arange(1, len(prediction) + 1), columns=['ImageId'])\n\nprediction = pd.concat([img_idx, prediction], axis=1)\n\nprediction.to_csv('submission_cnn.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nWith sufficient data augmentation and tuning of the architecture and hyperparameters of the Convolutional Neural Network, we achieve a significant accuracy of ~99.6%, which more than acceptable for the task at hand. This further proves the point that artificial intelligence is the best new approach to the problem of computer vision, that will only continue to get better and better."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}