{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Project Goal\nThe primary objective of this project is to develop and train a machine learning model capable of accurately recognizing handwritten digits. This challenge is a part of the Kaggle competition named \"Digit Recognizer\"\n\nHandwritten digit recognition has vast applications in areas where digitizing handwritten documents is necessary, such as processing bank cheques, sorting postal mail, or digitizing old manuscripts. By participating in this competition, we aim to explore various deep learning techniques, understand the intricacies of CNN architectures, and apply best practices in model training and evaluation to achieve the highest possible accuracy on the given dataset.\n\nThe dataset for this competition is derived from the MNIST dataset, a large collection of handwritten digits that has been a standard benchmark for classification algorithms. Our challenge is to correctly identify digits from a dataset of tens of thousands of handwritten images. Success in this project will demonstrate a foundational understanding of image classification tasks using deep learning and provide a stepping stone towards more complex computer vision challenges.","metadata":{}},{"cell_type":"markdown","source":"# Loading the dataset","metadata":{}},{"cell_type":"code","source":"# Load train dataset\ndata = pd.read_csv('train.csv')\n\n# Check the first 5 rows of the dataset\nprint(data.head())","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"|    | label | pixel0 | pixel1 | pixel2 | ... | pixel780 | pixel781 | pixel782 | pixel783 |\n|----|-------|--------|--------|--------|-----|----------|----------|----------|----------|\n| 0  | 1     | 0      | 0      | 0      | ... | 0        | 0        | 0        | 0        |\n| 1  | 0     | 0      | 0      | 0      | ... | 0        | 0        | 0        | 0        |\n| 2  | 1     | 0      | 0      | 0      | ... | 0        | 0        | 0        | 0        |\n| 3  | 4     | 0      | 0      | 0      | ... | 0        | 0        | 0        | 0        |\n| 4  | 0     | 0      | 0      | 0      | ... | 0        | 0        | 0        | 0        |\n","metadata":{}},{"cell_type":"markdown","source":"A first view at the data shows the target digit in the first column, the other columns are either 0 or 1, representing a black or white pixel in 28x28 pixel wide images.\nSo we have a total of 784 pixels.","metadata":{}},{"cell_type":"code","source":"# Split the dataset into Features and Target\nx = data.iloc[:, 1:].values\ny = data.iloc[:, 0].values\n\n# Reshape the features to 28x28 pixels\nx = x.reshape(-1, 28, 28, 1)\n\n# Show first 8 images\nfig, axes = plt.subplots(nrows=2, ncols=4, figsize=(10, 5))\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(x[i].reshape(28, 28), cmap='gray')\n    ax.set_title(f\"Label: {y[i]}\")\n    ax.axis('off')\n\nfig.suptitle('First 8 images of the dataset')\nplt.tight_layout()\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![First 8 images of the dataset](https://i.imgur.com/ni5ZSS9.png)\n\nThe first 8 entries in a visualized state give us a better understanding of what we are working with.","metadata":{}},{"cell_type":"markdown","source":"# Model Creation\n\nIn this section, we construct our Convolutional Neural Network (CNN) model, designed to recognize and classify the handwritten digits from the dataset. Our model's architecture is sequential, meaning that each layer's output feeds directly into the next layer as input. Here's a breakdown of the layers used:\n\n- **First Convolutional Layer**: This layer uses 32 filters, each of size \\(1 *1\\), with ReLU (Rectified Linear Unit) activation. We also apply L2 regularization to help prevent overfitting by penalizing large weights. The input shape is set to \\(28 * 28 * 1\\), matching our pre-processed image dimensions.\n\n- **First Max Pooling Layer**: Following the initial convolution, we use a Max Pooling layer with a pool size of \\(2 * 2\\), reducing the spatial dimensions (height and width) of the output from the previous layer, which helps in reducing computation and controlling overfitting.\n\n- **Second Convolutional Layer**: This layer increases the filter size to \\(3 * 3\\), sticking with 32 filters and ReLU activation. L2 regularization is again applied.\n\n- **Second Max Pooling Layer**: We apply another Max Pooling layer with the same pool size of \\(2 * 2\\), further downsampling the feature maps.\n\n- **Third Convolutional Layer**: Increasing complexity, this layer has 64 filters of size \\(3 * 3\\), with L2 regularization but initially no activation function. \n\n- **Batch Normalization**: Before applying the activation function, we use Batch Normalization to normalize the inputs of the Activation layer, helping to speed up training and improve the overall performance by reducing internal covariate shift.\n\n- **Activation Function**: ReLU is applied after batch normalization to introduce non-linearity, allowing the model to learn more complex patterns.\n\n- **Third Max Pooling Layer**: The final Max Pooling layer again uses a \\(2 * 2\\) pool size, preparing the data for the dense layers by reducing its dimensions.\n\n- **Flatten Layer**: The Flatten layer converts the 2D feature maps into a 1D feature vector, making it possible to feed the convolutional network's output into the dense layers.\n\n- **First Dense Layer**: A fully connected layer with 128 neurons and ReLU activation.\n\n- **Output Layer**: The final layer is a Dense layer with 10 neurons, one for each digit (0-9), using the softmax activation function to output probabilities for each class.\n\nThis model structure is chosen to progressively learn more complex features from the images, with pooling layers reducing the spatial size of the representations to make the computations more manageable, and regularization techniques like L2 and batch normalization ensuring the model generalizes well to unseen data.\n","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"| Layer (type)                | Output Shape          | Param #  |\n|-----------------------------|-----------------------|----------|\n| conv2d (Conv2D)             | (None, 28, 28, 32)    | 64       |\n| max_pooling2d (MaxPooling2D)| (None, 14, 14, 32)    | 0        |\n| conv2d_1 (Conv2D)           | (None, 12, 12, 32)    | 9,248    |\n| max_pooling2d_1 (MaxPooling2D)| (None, 6, 6, 32)     | 0        |\n| conv2d_2 (Conv2D)           | (None, 4, 4, 64)      | 18,496   |\n| batch_normalization (BatchNormalization) | (None, 4, 4, 64) | 256      |\n| activation (Activation)     | (None, 4, 4, 64)      | 0        |\n| max_pooling2d_2 (MaxPooling2D)| (None, 2, 2, 64)     | 0        |\n| flatten (Flatten)           | (None, 256)           | 0        |\n| dense (Dense)               | (None, 128)           | 32,896   |\n| dense_1 (Dense)             | (None, 10)            | 1,290    |\n|-----------------------------|-----------------------|----------|\n| **Total params**: 62,250    |                       |          |\n| **Trainable params**: 62,122|                       |          |\n| **Non-trainable params**: 128 |                     |          |","metadata":{}},{"cell_type":"markdown","source":"# Early Stopping\n\nTo optimize our neural network, we employ a technique known as **Early Stopping**. This method monitors the model's performance on a validation set and halts the training process when the performance no longer improves, effectively preventing overfitting. Here's how we configure Early Stopping for our model:\n\n- **Monitor**: We choose `val_loss` as the metric to monitor, meaning the training will focus on minimizing loss on the validation set.\n- **Patience**: Set to 15, this parameter specifies the number of epochs to continue training after the monitored metric has stopped improving. This buffer allows the model to navigate past small plateaus in search of better weights.\n- **Restore Best Weights**: By setting this to `True`, our model will revert to the weights from the epoch with the best value of the monitored metric (`val_loss` in our case) once the training is halted. This ensures that our model retains the most effective parameters even if the performance dips in subsequent epochs.","metadata":{}},{"cell_type":"code","source":"early_stopping = EarlyStopping(\n    monitor='val_loss',  \n    patience=15,  \n    restore_best_weights=True\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training\nFollowing the configuration of Early Stopping, we proceed to train our model using the `fit` method with the following parameters:\n\n- **Input Data (`x`)**: Our training features, reshaped to 28x28 pixels and with a single color channel.\n- **Target Data (`y`)**: The labels indicating the correct digit each image represents.\n- **Epochs**: We set an initial cap at 50 epochs, allowing ample opportunity for the model to learn from the data. However, due to Early Stopping, training may halt before reaching this limit if no improvement is seen.\n- **Batch Size**: Set to 32, this determines the number of samples that will be propagated through the network at a time. Smaller batch sizes require less memory and can offer a regularizing effect, potentially improving model generalization.","metadata":{}},{"cell_type":"code","source":"history = model.fit(x, y, epochs=50, \n                    batch_size=32, \n                    validation_split=0.2, \n                    callbacks=[early_stopping])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After completing the training process, our model achieved an impressive accuracy score of 0.984 on the validation data, indicating a high level of accuracy ","metadata":{}},{"cell_type":"markdown","source":"# Load Test Dataset\n\nAfter training our model, the next step involves evaluating its performance on unseen data. To do this, we load our test dataset using Pandas:\n","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('test.csv')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing Test Data\nJust like our training data, we need to reshape the test images to fit our model's input expectations. Each image is reshaped to 28x28 pixels with a single color channel:","metadata":{}},{"cell_type":"code","source":"x_test = test.values.reshape(-1, 28, 28, 1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Making Predictions\nWith our test data prepared, we can now use our trained model to make predictions:","metadata":{}},{"cell_type":"code","source":"predictions = model.predict(x_test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We're interested in the digit with the highest prediction probability for each image, so we extract the index of the highest probability, which corresponds to our model's predicted digit:","metadata":{}},{"cell_type":"code","source":"predictions = [p.argmax() for p in predictions]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualizing Predictions\nFinally, to visually confirm our model's performance, we display the first 8 predictions. This step is not only helpful for a quick accuracy check but also insightful to understand how our model interprets different handwritten digits:","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=4, figsize=(10, 4))\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(x_test[i].reshape(28, 28), cmap='gray')\n    ax.set_title(f\"Pred: {submission.iloc[i]['Label']}\")\n    ax.axis('off')\nfig.suptitle('First 8 predictions of the train dataset')\nplt.tight_layout()\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![First 8 predictions](https://i.imgur.com/QH3s6Bv.png)","metadata":{}},{"cell_type":"markdown","source":"This visualization displays the first eight predictions made by our model on the test dataset, proving its ability to predict the# Bonus Part: Predicting Own Handwritten Digits\n\n","metadata":{}},{"cell_type":"markdown","source":"# Bonus Part: Predicting Own Handwritten Digits\n\nAfter evaluating our model on the provided test dataset, we take a step further to test its capability with our own handwritten digits. This not only demonstrates the model's practical application but also its adaptability to real-world variability in handwriting styles.\n\n## Preparing and Predicting Own Images\n\nWe prepared four handwritten digit images (`7`, `8`, `0`, and `3`) and saved them as PNG files. These images were then loaded, reshaped to match our model's input expectations (28x28 pixels, grayscale), and fed into the model for prediction:\n\n- `own_predict_7.png`\n- `own_predict_8.png`\n- `own_predict_0.png`\n- `own_predict_3.png`\n\nThe predictions were obtained by identifying the class (digit) with the highest probability from the model's output.\n\n## Results\n\nThe model predicted the following digits for our images:\n\n- Prediction for `7.png`: 7\n- Prediction for `8.png`: 8\n- Prediction for `0.png`: 0\n- Prediction for `3.png`: 3\n\nThese results were visualized to provide a clear comparison between the actual images and the model's predictions:","metadata":{}},{"cell_type":"markdown","source":"![own handwritten predictions](https://i.imgur.com/PTA1jWy.png)","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nAs we reach the end of this project, it's valuable to reflect on the journey and the key takeaways.\nThis project's primary goal was to develop a machine learning model capable of accurately recognizing handwritten digits. By achieving a validation score of approximately 0.984, we demonstrated that our model could effectively interpret and classify a wide range of handwritten styles with high accuracy.","metadata":{}}]}