{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":29867,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"I've been learning about Convolutional Neural Networks (CNNs) recently and wanted to give them a go on a challenge I have attempted with a Deep Neural Network in the past. I got 0.94285 accuracy with a Deep Neural Network so I'm keen to do better with a CNN.\n\nLet's begin by loading the libraries we will be using.","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport tensorflow as tf","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-18T00:00:02.512000Z","iopub.execute_input":"2024-01-18T00:00:02.512349Z","iopub.status.idle":"2024-01-18T00:00:05.456104Z","shell.execute_reply.started":"2024-01-18T00:00:02.512313Z","shell.execute_reply":"2024-01-18T00:00:05.454798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load and data analysis\n\nFirst things first, load the training dataset. Although this is an image recognition task, the data comes in a csv format rather than a directory of PNGs/JPGs. The easiest way to load this data is to use pandas. We'll do this and have a look at the first row.","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\nprint(data.info())","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:05.458293Z","iopub.execute_input":"2024-01-18T00:00:05.458607Z","iopub.status.idle":"2024-01-18T00:00:09.401796Z","shell.execute_reply.started":"2024-01-18T00:00:05.458575Z","shell.execute_reply":"2024-01-18T00:00:09.400573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we can see that each row of data represents an image. Each image is made up 784 pixels (which is a 28x28 image). Each pixel is represented in the columns with a prefix of \"pixel\". These columns contain a number between 0 and 255 where 0 represents a white pixel, 255 a black pixel and the other numbers represent shades of grey.\n\nWe also have a column called \"label\" that represents the number in the image (the images class). This number will be anything between 0 and 9.\n\nTo test the model as it trains I will split the dataset into a training and validation dataset. Then I'll remove the labels from the datasets and put them into their own dataframes. I want to make a lot of the images available for training so taking some influence from other notebooks in this competition I'm going to go for a 90%-10% split.","metadata":{}},{"cell_type":"code","source":"train_data = data.head(37800)\nval_data = data.tail(4200)\n\ntrain_labels = train_data.pop('label')\nval_labels = val_data.pop('label')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:09.403337Z","iopub.execute_input":"2024-01-18T00:00:09.403654Z","iopub.status.idle":"2024-01-18T00:00:09.489896Z","shell.execute_reply.started":"2024-01-18T00:00:09.403620Z","shell.execute_reply":"2024-01-18T00:00:09.488951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'll be using Tensorflow to train an image classifier model. Tensorflow has it's own api for creating a data pipeline which makes it easier to feed the data into a Tensorflow model. To use this api the data needs to be loaded into a Tensorflow data object. ","metadata":{}},{"cell_type":"code","source":"tf_train_data = tf.data.Dataset.from_tensor_slices((train_data.values, train_labels.values))\ntf_val_data = tf.data.Dataset.from_tensor_slices((val_data.values, val_labels.values))\n\nprint(tf_train_data)\nprint(tf_val_data)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:09.492035Z","iopub.execute_input":"2024-01-18T00:00:09.492619Z","iopub.status.idle":"2024-01-18T00:00:09.870480Z","shell.execute_reply.started":"2024-01-18T00:00:09.492582Z","shell.execute_reply":"2024-01-18T00:00:09.869400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before going on to form the data pipeline, I'll have a look at some of the images in the dataset and visualise them with their labels. As pixels for the images have been flattened into a 2D array, I have used a reshape function to put the images into a 28x28 shape that is more human friendly.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\ni = 0\n\nfor image, label in tf_train_data.take(5):\n    plt.subplot(1,5,i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.grid(False)\n\n    plt.imshow(image.numpy().reshape((28, 28)), cmap='gray')\n    plt.xlabel(label.numpy())\n\n    i = i + 1","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:09.874513Z","iopub.execute_input":"2024-01-18T00:00:09.874907Z","iopub.status.idle":"2024-01-18T00:00:10.381615Z","shell.execute_reply.started":"2024-01-18T00:00:09.874862Z","shell.execute_reply":"2024-01-18T00:00:10.380463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data pipeline\n\nNow that the data is loaded and a quick piece of analysis has been done I am ready to form the pipeline that will feed the images into the model. The first step is to perform some small transformations to the images. I'll do the following to each image:\n\n1. **Reshape:** The images are currently formatted as a flat list of pixels. CNNs need the data to be reshaped into a 3D matrix which in this case is a 28x28 matrix. I also need to specify the channels. As these images are greyscale I only need the one. The shape of the data then is (28, 28, 1).\n2. **Scale:** Models seem to perform better when numeric features are put onto a 0-1 scale. As all the features are numeric and on a 0-255 scale it is easy to put the data onto this scale by simply dividing the features by 255.","metadata":{}},{"cell_type":"code","source":"def preprocess_image(image, label):\n    image = tf.reshape(image, [28, 28, 1])\n    image = tf.cast(image, tf.float32) / 255.\n    \n    return image, label\n\ntf_train_data = tf_train_data.map(\n    preprocess_image, \n    num_parallel_calls=tf.data.experimental.AUTOTUNE\n)\n\ntf_val_data = tf_val_data.map(\n    preprocess_image, \n    num_parallel_calls=tf.data.experimental.AUTOTUNE\n)\n\nprint(tf_train_data)\nprint(tf_val_data)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:10.383715Z","iopub.execute_input":"2024-01-18T00:00:10.384017Z","iopub.status.idle":"2024-01-18T00:00:10.604937Z","shell.execute_reply.started":"2024-01-18T00:00:10.383984Z","shell.execute_reply":"2024-01-18T00:00:10.604200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now the data is preprocessed there are few configurations that can be made to the pipeline. Here's a list of those configurations:\n\n1. **Shuffle:** To make sure the model doesn't pick up anything from the order of the rows in the dataset the top 100 rows of data will be shuffled per training step. \n2. **Batch:** This is quite a large dataset in terms of both rows and columns. To ensure the processor doesn't get overloaded the data will be fed in 32 images at a time.\n3. **Prefetch:** The speed up the training the pipeline can be set to fetch the next batch of data while the current batch of data is in training. When the current step ends there is no time wasted loading the next batch as it is ready to go.","metadata":{}},{"cell_type":"code","source":"def pipeline(tf_data):\n    tf_data = tf_data.shuffle(100)\n    tf_data = tf_data.batch(32)\n    tf_data = tf_data.prefetch(tf.data.experimental.AUTOTUNE)\n    \n    return tf_data\n\ntf_train_data = pipeline(tf_train_data)\ntf_val_data = pipeline(tf_val_data)\n\nprint(tf_train_data)\nprint(tf_val_data)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:10.606247Z","iopub.execute_input":"2024-01-18T00:00:10.606677Z","iopub.status.idle":"2024-01-18T00:00:10.616561Z","shell.execute_reply.started":"2024-01-18T00:00:10.606646Z","shell.execute_reply":"2024-01-18T00:00:10.615584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train model\n\nThe data is now ready to be fed into a model. I've chosen to use a CNN here and will be using an architecture based on the LeNet-5 architecture. I won't go into how CNNs work or why they are good for image classification tasks in this notebook. If you would like to learn more about them though I learnt about them on this [course](https://www.coursera.org/learn/convolutional-neural-networks?specialization=deep-learning) by Andrew Ng and I can't recommend it highly enough.\n\nAlthough I learnt about the LeNet-5 CNN architecture on the course and not by reading the paper it only seems right to quote the paper that LeNet-5 comes from:\n*\"LeCun et al., 1998, Gradient-based learning applied to document recognition\"*\n\nSo without further ado I'll use Tensorflow to define the model. As you will see the model is much the same as the LeNet-5 model except for a few changes:\n\n1. All average pooling layers have been replaced with max pooling layers\n2. The input shape is slightly smaller than the 32x32 shape in the paper. The first convolutional layer uses \"same\" padding to ensure the data is in the same shape as it is in the paper by the time it hits the first pooling layer.","metadata":{}},{"cell_type":"code","source":"model = tf.keras.Sequential([\n    tf.keras.layers.Conv2D(6, (5, 5), activation='relu', padding='same', input_shape=(28, 28, 1)),\n    tf.keras.layers.MaxPooling2D((2, 2)),\n    \n    tf.keras.layers.Conv2D(16, (5, 5), activation='relu', padding='valid'),\n    tf.keras.layers.MaxPooling2D((2, 2)),\n    \n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(120, activation='relu'),\n    tf.keras.layers.Dense(84, activation='relu'),\n    \n    tf.keras.layers.Dense(10, activation='softmax'),\n])","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:10.618056Z","iopub.execute_input":"2024-01-18T00:00:10.618338Z","iopub.status.idle":"2024-01-18T00:00:10.733318Z","shell.execute_reply.started":"2024-01-18T00:00:10.618307Z","shell.execute_reply":"2024-01-18T00:00:10.732383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I'll compile the model and get a print out describing it.","metadata":{}},{"cell_type":"code","source":"optimiser = tf.keras.optimizers.Adam(learning_rate=0.001)\n\nmodel.compile(\n    optimizer=optimiser, \n    loss='sparse_categorical_crossentropy', \n    metrics=['accuracy'])\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:10.735696Z","iopub.execute_input":"2024-01-18T00:00:10.736082Z","iopub.status.idle":"2024-01-18T00:00:10.795817Z","shell.execute_reply.started":"2024-01-18T00:00:10.736036Z","shell.execute_reply":"2024-01-18T00:00:10.794891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Tensorflow has a piece of functionality called callbacks that enable us to monitor the models metrcis during training and take some action as a result. Using this functionality I will implement the following:\n\n1. **Learning Rate Decay:** If the models loss does not reduce for 2 consecutive epochs, reduce the learning rate. This should stop the model stepping over the optimum too many times.\n2. **Early Stopping:** If the models loss does not reduce for 5 consecutive epochs then it's clear the model is not learning anything anymore. Save some computation and stop training.\n\nNote: the verbose parameter simply states whether a message should be printed in the console (1) or not (0) when a callback is made.","metadata":{}},{"cell_type":"code","source":"callbacks = [\n    tf.keras.callbacks.ReduceLROnPlateau(monitor='loss', patience=2, verbose=1),\n    tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5, verbose=1),\n]","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:10.797355Z","iopub.execute_input":"2024-01-18T00:00:10.797656Z","iopub.status.idle":"2024-01-18T00:00:10.804009Z","shell.execute_reply.started":"2024-01-18T00:00:10.797624Z","shell.execute_reply":"2024-01-18T00:00:10.802875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the model compiled and the data sitting ready in the pipeline it is time to train the model. The validation dataset has been included here so that the model validates its accuracy after each step by making predictions against the validation dataset.\n\nNote: Each epoch takes around 22 seconds to complete so this may take a few minutes to run depending on how many epochs this is run for.","metadata":{}},{"cell_type":"code","source":"train_log = model.fit(\n    tf_train_data,\n    validation_data=tf_val_data,\n    epochs=10,\n    callbacks=callbacks\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:00:10.805741Z","iopub.execute_input":"2024-01-18T00:00:10.806146Z","iopub.status.idle":"2024-01-18T00:03:26.562516Z","shell.execute_reply.started":"2024-01-18T00:00:10.806100Z","shell.execute_reply":"2024-01-18T00:03:26.561275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the model trained I'll plot the accuracy achieved per epoch per dataset on a chart so that training progress is clearer.","metadata":{}},{"cell_type":"code","source":"plt.plot(train_log.history['accuracy'], label='accuracy')\nplt.plot(train_log.history['val_accuracy'], label = 'val_accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend()\n\nprint('Training accuracy: %f' % train_log.history['accuracy'][-1])\nprint('Validation accuracy: %f' % train_log.history['val_accuracy'][-1])","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:03:26.564159Z","iopub.execute_input":"2024-01-18T00:03:26.564481Z","iopub.status.idle":"2024-01-18T00:03:26.776322Z","shell.execute_reply.started":"2024-01-18T00:03:26.564412Z","shell.execute_reply":"2024-01-18T00:03:26.775414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference\n\nThe final step is to use this trained model to classify the images in the test dataset. The first step is to load the training dataset and put it into the Tensorflow data format.","metadata":{"trusted":true}},{"cell_type":"code","source":"test_data = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\ntf_test_data = tf.data.Dataset.from_tensor_slices(([test_data.to_numpy().reshape(len(test_data), 28, 28, 1)]))","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:03:26.777660Z","iopub.execute_input":"2024-01-18T00:03:26.777926Z","iopub.status.idle":"2024-01-18T00:04:10.680164Z","shell.execute_reply.started":"2024-01-18T00:03:26.777896Z","shell.execute_reply":"2024-01-18T00:04:10.679113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then use the model to classify each image. The model will not just output one class but will instead output a probability for each possible class. Numpys argmax is thus used to find the class with the highest probability and this class is used for the image.","metadata":{}},{"cell_type":"code","source":"predictions = model.predict(tf_test_data)\npredictions = np.argmax(predictions, axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:04:10.681607Z","iopub.execute_input":"2024-01-18T00:04:10.681871Z","iopub.status.idle":"2024-01-18T00:04:13.562347Z","shell.execute_reply.started":"2024-01-18T00:04:10.681841Z","shell.execute_reply":"2024-01-18T00:04:13.561544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To see if the model is performing as expected I will check visualisations of the first fifteen images against the classifications produced by the model.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\n\nfor i, row in test_data.head(15).iterrows():\n    plt.subplot(3,5,i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.grid(False)\n\n    plt.imshow(row.values.reshape((28, 28)), cmap='gray')\n    plt.xlabel(predictions[i])","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:04:13.563749Z","iopub.execute_input":"2024-01-18T00:04:13.564257Z","iopub.status.idle":"2024-01-18T00:04:14.256150Z","shell.execute_reply.started":"2024-01-18T00:04:13.564221Z","shell.execute_reply":"2024-01-18T00:04:14.255259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While it is unfeasible to eyeball all 28000 test images and predicted classes, checking the first fifteen and seeing 98% accuracy on the validation dataset gives me some confidence that the model is performing well. The last thing to do then is to put the classifications into a submission dataframe and write it to a csv file.","metadata":{}},{"cell_type":"code","source":"predictions_df = pd.DataFrame(data={'Label': predictions}, index=pd.RangeIndex(start=1, stop=28001))\npredictions_df.index = predictions_df.index.rename('ImageId')\n\npredictions_df.to_csv('submission_file.csv')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T00:04:14.257702Z","iopub.execute_input":"2024-01-18T00:04:14.258241Z","iopub.status.idle":"2024-01-18T00:04:14.560679Z","shell.execute_reply.started":"2024-01-18T00:04:14.258202Z","shell.execute_reply":"2024-01-18T00:04:14.559423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Final note\n\nI'm still relatively new to data science and I'm keen to learn. Any comments, ideas and points to improve on would be much appreciated.","metadata":{"trusted":true}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}