","metadata":{}},{"cell_type":"markdown","source":"In the rapidly evolving field of artificial intelligence (AI), neural networks have become a powerful tool for solving complex problems, such as digit recognition. This project begins by establishing a baseline using dense neural networks, before transitioning to more advanced convolutional neural networks (CNNs). The aim is to showcase how different types of neural networks can be applied to the MNIST dataset, which contains 28,000 images of handwritten digits, and progressively improve accuracy in digit classification.\n\n## Understanding Neural Networks\nNeural networks are inspired by the structure and function of the human brain. Just as neurons in the brain process information, artificial neurons in a neural network take inputs, process them, and pass on the result. These neurons are organized into layers, where each layer learns progressively more complex patterns in the data.\n\nWe begin this project with dense (fully connected) neural networks, where each neuron in one layer is connected to every neuron in the next. These layers extract features from the input data (in our case, the pixel values of handwritten digits) and perform non-linear transformations to predict the digit. Dense networks are effective for simple datasets, but they often struggle with image data, where spatial relationships between pixels are key.\n\n## From Dense Networks to CNNs\nWhile dense neural networks can serve as a strong baseline, convolutional neural networks (CNNs) are far better suited for image classification tasks. CNNs take advantage of the spatial structure of images by applying filters (convolutions) to detect patterns such as edges, curves, and textures, allowing the network to \"see\" the image in a more structured way. These filters automatically learn to detect features at different levels of abstraction, making CNNs more effective in recognizing and classifying complex patterns in images, such as handwritten digits.","metadata":{}},{"cell_type":"markdown","source":"#
","metadata":{}},{"cell_type":"markdown","source":"## Loading data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2024-09-23T20:18:55.059193Z","iopub.execute_input":"2024-09-23T20:18:55.059570Z","iopub.status.idle":"2024-09-23T20:18:59.248211Z","shell.execute_reply.started":"2024-09-23T20:18:55.059534Z","shell.execute_reply":"2024-09-23T20:18:59.246987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Separating features and labels","metadata":{}},{"cell_type":"code","source":"X = train.drop(\"label\", axis=1) \ny = train.label","metadata":{"execution":{"iopub.status.busy":"2024-09-23T20:18:59.249851Z","iopub.execute_input":"2024-09-23T20:18:59.250213Z","iopub.status.idle":"2024-09-23T20:18:59.337787Z","shell.execute_reply.started":"2024-09-23T20:18:59.250175Z","shell.execute_reply":"2024-09-23T20:18:59.336665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data normalization","metadata":{}},{"cell_type":"markdown","source":"This step ensures that our input data is in a suitable range for training a neural network, leading to better performance and faster learning.","metadata":{}},{"cell_type":"code","source":"# Normalize the data (scale pixel values between 0 and 1)\nX = X / 255.0\nX_test = test / 255.0","metadata":{"execution":{"iopub.status.busy":"2024-09-23T20:18:59.339192Z","iopub.execute_input":"2024-09-23T20:18:59.339592Z","iopub.status.idle":"2024-09-23T20:18:59.447744Z","shell.execute_reply.started":"2024-09-23T20:18:59.339542Z","shell.execute_reply":"2024-09-23T20:18:59.446801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Spliting data into training and testing sets","metadata":{}},{"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.1, random_state=1)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T20:18:59.449769Z","iopub.execute_input":"2024-09-23T20:18:59.450421Z","iopub.status.idle":"2024-09-23T20:18:59.634527Z","shell.execute_reply.started":"2024-09-23T20:18:59.450375Z","shell.execute_reply":"2024-09-23T20:18:59.633659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preview Images","metadata":{}},{"cell_type":"code","source":"X_reshaped = X_train.to_numpy().reshape(-1, 28, 28, 1)\n\nplt.figure(figsize=(15, 5))\nfor i in range(30): \n plt.subplot(3, 10, i+1)\n plt.imshow(X_reshaped[i, :, :, 0], cmap=plt.cm.binary) # Access the 28x28 image directly\n plt.axis('off')\nplt.subplots_adjust(wspace=-0.1, hspace=-0.1)\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-23T20:23:36.435188Z","iopub.execute_input":"2024-09-23T20:23:36.435574Z","iopub.status.idle":"2024-09-23T20:23:37.751480Z","shell.execute_reply.started":"2024-09-23T20:23:36.435537Z","shell.execute_reply":"2024-09-23T20:23:37.750516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
4. Dense Neural Network
","metadata":{}},{"cell_type":"markdown","source":"A dense neural network, also known as a fully connected neural network (FCNN) or multi-layer perceptron (MLP), is a machine learning model that connects each layer to the previous layer.","metadata":{}},{"cell_type":"markdown","source":"We will train the model through iterative steps, starting with a simple version. If the model shows signs of high bias or underfitting, we'll increase its complexity. If the model shows high variance or overfitting, we'll boost the regularization. This approach should help us achieve better results.","metadata":{}},{"cell_type":"code","source":"# Define a simple model\nmodel = Sequential([\n Dense(units=25, activation='relu'),\n Dense(units=15, activation='relu'),\n Dense(units=10, activation='softmax')\n])","metadata":{"execution":{"iopub.status.busy":"2024-09-23T13:23:02.837714Z","iopub.execute_input":"2024-09-23T13:23:02.838588Z","iopub.status.idle":"2024-09-23T13:23:02.847213Z","shell.execute_reply.started":"2024-09-23T13:23:02.838535Z","shell.execute_reply":"2024-09-23T13:23:02.846191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compile the model with an Adam optimizer and metrics\nmodel.compile(\n optimizer = Adam(learning_rate=1e-3),\n loss=SparseCategoricalCrossentropy(),\n metrics=['accuracy']\n)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T13:23:03.051429Z","iopub.execute_input":"2024-09-23T13:23:03.051716Z","iopub.status.idle":"2024-09-23T13:23:03.060489Z","shell.execute_reply.started":"2024-09-23T13:23:03.051685Z","shell.execute_reply":"2024-09-23T13:23:03.059637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train the model\nmodel.fit(X_train, y_train, epochs=30, validation_data=(X_val, y_val))","metadata":{"execution":{"iopub.status.busy":"2024-09-23T13:11:50.715711Z","iopub.execute_input":"2024-09-23T13:11:50.716094Z","iopub.status.idle":"2024-09-23T13:12:51.699427Z","shell.execute_reply.started":"2024-09-23T13:11:50.716057Z","shell.execute_reply":"2024-09-23T13:12:51.698597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model is performing much better on the training data than on the validation data, which means it's overfitting. High variance or overfitting happens when the model learns too much from the training data, including noise, and struggles to generalize to new data. To fix this, we will add a regularization term, which acts like a penalty and helps the model focus on the important patterns rather than memorizing the training data.","metadata":{}},{"cell_type":"code","source":"# Define a model with regularization\nmodel = Sequential([\n Dense(units=25, activation='relu', kernel_regularizer=L2(0.001)),\n Dense(units=15, activation='relu', kernel_regularizer=L2(0.001)),\n Dense(units=10, activation='softmax', kernel_regularizer=L2(0.001))\n])\n\n# Compile the model with an Adam optimizer and metrics\nmodel.compile(\n optimizer = Adam(learning_rate=1e-3),\n loss=SparseCategoricalCrossentropy(),\n metrics=['accuracy']\n)\n\n# Train the model\nmodel.fit(X_train, y_train, epochs=30, validation_data=(X_val, y_val))","metadata":{"execution":{"iopub.status.busy":"2024-09-23T13:13:56.135606Z","iopub.execute_input":"2024-09-23T13:13:56.136012Z","iopub.status.idle":"2024-09-23T13:14:55.112844Z","shell.execute_reply.started":"2024-09-23T13:13:56.135974Z","shell.execute_reply":"2024-09-23T13:14:55.111823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The difference between training and validation performance has reduced, but there's still room for improvement in accuracy. To enhance the model, we'll increase its complexity while maintaining regularization and adding Dropout. This will help the model generalize better and improve overall performance.","metadata":{}},{"cell_type":"code","source":"# Define a deeper and more complex model\nmodel = Sequential([\n Dense(units=512, activation='relu', kernel_regularizer=L2(0.001)), # Increased units\n Dropout(0.4), # Add dropout to prevent overfitting\n Dense(units=256, activation='relu', kernel_regularizer=L2(0.001)),\n Dropout(0.4),\n Dense(units=128, activation='relu', kernel_regularizer=L2(0.001)),\n Dropout(0.3),\n Dense(units=64, activation='relu', kernel_regularizer=L2(0.001)),\n Dropout(0.3),\n Dense(units=32, activation='relu', kernel_regularizer=L2(0.001)),\n Dense(units=10, activation='softmax', kernel_regularizer=L2(0.001)) \n])\n\n# Compile the model with an Adam optimizer and metrics\nmodel.compile(\n optimizer=Adam(learning_rate=5e-5), # Reduced learning rate for better convergence\n loss=SparseCategoricalCrossentropy(),\n metrics=['accuracy']\n)\n\n# Train the model\nmodel.fit(X_train, y_train, epochs=50, validation_data=(X_val, y_val)) # Increased epochs","metadata":{"execution":{"iopub.status.busy":"2024-09-23T13:16:46.759796Z","iopub.execute_input":"2024-09-23T13:16:46.760469Z","iopub.status.idle":"2024-09-23T13:19:11.656980Z","shell.execute_reply.started":"2024-09-23T13:16:46.760427Z","shell.execute_reply":"2024-09-23T13:19:11.656123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results are promising, but now let's take it a step further by trying a convolutional neural network (CNN).","metadata":{}},{"cell_type":"markdown","source":"#
5. Convolutional Neural Network
","metadata":{}},{"cell_type":"markdown","source":"Convolutional Neural Networks (CNNs) are powerful for image recognition tasks because they can capture spatial features, like edges, patterns, and shapes, making them much more effective than dense neural networks for working with images.\n\nHowever, to take advantage of CNNs, we need to reshape our data. The images in the dataset are originally stored as flat 1D vectors (with 784 elements for 28x28 pixel images). CNNs, though, require the data to be in 2D (28x28) with a depth of 1 for grayscale images. This allows the CNN to process the pixel structure properly and detect patterns.\n\nThatโs why we convert our data from DataFrames to NumPy arrays and reshape them to (28, 28, 1). This gives CNNs the right format to work with, preserving the spatial relationships between pixels.","metadata":{}},{"cell_type":"code","source":"# Convert DataFrames to NumPy arrays\nX_train = X_train.to_numpy().reshape(-1, 28, 28, 1)\nX_val = X_val.to_numpy().reshape(-1, 28, 28, 1)\nX_test = X_test.to_numpy().reshape(-1, 28, 28, 1)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:49:28.828469Z","iopub.execute_input":"2024-09-23T18:49:28.828800Z","iopub.status.idle":"2024-09-23T18:49:28.834174Z","shell.execute_reply.started":"2024-09-23T18:49:28.828766Z","shell.execute_reply":"2024-09-23T18:49:28.833203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define a model with Conv2D layers\nmodel = Sequential([\n Conv2D(32, kernel_size=(3, 3), activation='relu', padding = 'Same', kernel_regularizer=L2(0.0001), input_shape=(28, 28, 1)), # First conv layer\n MaxPooling2D(pool_size=(2, 2)), # Pooling layer\n Dropout(0.2), # Dropout to reduce overfitting\n \n Conv2D(64, kernel_size=(3, 3), activation='relu', padding = 'Same', kernel_regularizer=L2(0.0001)), # Second conv layer\n MaxPooling2D(pool_size=(2, 2)),\n Dropout(0.3),\n \n Conv2D(128, kernel_size=(3, 3), activation='relu', padding = 'Same', kernel_regularizer=L2(0.0001)), # Third conv layer\n MaxPooling2D(pool_size=(2, 2)),\n Dropout(0.3),\n \n Flatten(), # Flatten the output for dense layers\n Dense(64, activation='relu', kernel_regularizer=L2(0.0001)), # Fully connected layer\n Dropout(0.2),\n Dense(10, activation='softmax') # Output layer\n])\n\n# Compile the model\nmodel.compile(\n optimizer=Adam(learning_rate=1e-4), \n loss=SparseCategoricalCrossentropy(),\n metrics=['accuracy']\n)\n\n# Train the model\nmodel.fit(X_train, y_train, epochs=50, validation_data=(X_val, y_val)) ","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:50:16.215866Z","iopub.execute_input":"2024-09-23T18:50:16.216239Z","iopub.status.idle":"2024-09-23T18:53:04.719869Z","shell.execute_reply.started":"2024-09-23T18:50:16.216202Z","shell.execute_reply":"2024-09-23T18:53:04.719094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Augmentation","metadata":{}},{"cell_type":"markdown","source":"Data augmentation is a powerful technique used to improve the performance and generalization of machine learning models, especially in image recognition tasks like digit recognition. The idea behind data augmentation is to artificially increase the size and diversity of the training dataset by applying random transformations to the original images, such as rotations, zooms, shifts, or flips. This helps the model learn more robust features and reduces the risk of overfitting.\n\nSince our training dataset may not be large or varied enough, data augmentation allows us to create new variations of the existing images, simulating real-world scenarios where digits might be slightly tilted, resized, or shifted. This technique makes the model more adaptable and better at recognizing digits in various conditions.\n\nFor this project, we use data augmentation to randomly rotate, shift, or zoom in on images, enhancing the model's ability to generalize well to unseen data. This ensures that the model doesn't simply memorize the training images but learns patterns that can be applied to new data.","metadata":{}},{"cell_type":"markdown","source":"#### Key Aspects:\n- **Rotation Range (5 degrees)**: Digit recognition can benefit from slight rotations as handwritten digits may not be perfectly aligned. A small range like 5 degrees keeps the transformation realistic without distorting the digits.\n\n- **Zoom Range (0.1)**: A 10% zoom allows the model to learn from digits that may be written smaller or larger, which adds robustness.\n\n- **Width/Height Shift (0.1)**: Small translations mimic slight variations in where digits might appear in images. This helps the model handle digits that are not perfectly centered.","metadata":{}},{"cell_type":"code","source":"datagen = ImageDataGenerator(\n rotation_range=5, # Randomly rotate images by up to 5 degrees\n zoom_range=0.1, # Randomly zoom images by up to 10%\n width_shift_range=0.1, # Randomly shift images horizontally by up to 10% of width\n height_shift_range=0.1, # Randomly shift images vertically by up to 10% of height\n)\n\ndatagen.fit(X_train)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:53:38.383000Z","iopub.execute_input":"2024-09-23T18:53:38.383784Z","iopub.status.idle":"2024-09-23T18:53:38.478856Z","shell.execute_reply.started":"2024-09-23T18:53:38.383741Z","shell.execute_reply":"2024-09-23T18:53:38.477965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define a more complex model with adjusted architecture\nmodel = Sequential([\n # First convolutional block\n Conv2D(64, kernel_size=(3, 3), activation='relu', padding = 'Same', kernel_regularizer=L2(0.0001), input_shape=(28, 28, 1)),\n BatchNormalization(),\n MaxPooling2D(pool_size=(2, 2)),\n Dropout(0.3),\n \n # Second convolutional block\n Conv2D(128, kernel_size=(3, 3), activation='relu', padding = 'Same', kernel_regularizer=L2(0.0001)),\n BatchNormalization(),\n MaxPooling2D(pool_size=(2, 2)),\n Dropout(0.3),\n \n # Third convolutional block\n Conv2D(256, kernel_size=(3, 3), activation='relu', padding = 'Same', kernel_regularizer=L2(0.0001)),\n BatchNormalization(),\n MaxPooling2D(pool_size=(2, 2)),\n Dropout(0.4),\n \n # Flatten and fully connected layers\n Flatten(),\n Dense(128, activation='relu', kernel_regularizer=L2(0.0001)),\n Dropout(0.4),\n Dense(64, activation='relu', kernel_regularizer=L2(0.0001)),\n Dropout(0.3),\n \n # Output layer for classification\n Dense(10, activation='softmax')\n])\n\n# Compile the model\nmodel.compile(\n optimizer=Adam(learning_rate=1e-4),\n loss=SparseCategoricalCrossentropy(),\n metrics=['accuracy']\n)\n\n# Train the model\nhistory = model.fit(datagen.flow(X_train, y_train, batch_size=32),\n epochs=50, \n validation_data=(X_val, y_val))","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:53:41.009163Z","iopub.execute_input":"2024-09-23T18:53:41.009595Z","iopub.status.idle":"2024-09-23T19:10:56.265231Z","shell.execute_reply.started":"2024-09-23T18:53:41.009560Z","shell.execute_reply":"2024-09-23T19:10:56.264437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Evaluation","metadata":{}},{"cell_type":"markdown","source":"### Training and Validation Curves","metadata":{}},{"cell_type":"code","source":"# Plot accuracy\nplt.plot(history.history['accuracy'], label='train accuracy')\nplt.plot(history.history['val_accuracy'], label='validation accuracy')\nplt.legend()\nplt.show()\n\n# Plot loss\nplt.plot(history.history['loss'], label='train loss')\nplt.plot(history.history['val_loss'], label='validation loss')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T19:22:56.421409Z","iopub.execute_input":"2024-09-23T19:22:56.422314Z","iopub.status.idle":"2024-09-23T19:22:57.260471Z","shell.execute_reply.started":"2024-09-23T19:22:56.422272Z","shell.execute_reply":"2024-09-23T19:22:57.259455Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Confusion Matrix","metadata":{}},{"cell_type":"code","source":"# 1. Make predictions on the validation or test set\ny_pred_probs = model.predict(X_val) # Predict probabilities\ny_pred = np.argmax(y_pred_probs, axis=1) # Convert probabilities to class labels\n\n# 2. Generate the confusion matrix\nconf_matrix = confusion_matrix(y_val, y_pred)\n\n# 3. Define class names (digits 0 to 9)\nclass_names = [i for i in range(10)]\n\n# 4. Plot the confusion matrix\nplt.figure(figsize=(8, 6))\nsns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=class_names, yticklabels=class_names)\nplt.xlabel('Predicted Label')\nplt.ylabel('True Label')\nplt.title('Confusion Matrix')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T19:28:55.745052Z","iopub.execute_input":"2024-09-23T19:28:55.745784Z","iopub.status.idle":"2024-09-23T19:28:56.579939Z","shell.execute_reply.started":"2024-09-23T19:28:55.745742Z","shell.execute_reply":"2024-09-23T19:28:56.579000Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predictions","metadata":{}},{"cell_type":"code","source":"predictions = model.predict(X_test, batch_size=128)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T19:25:59.498886Z","iopub.execute_input":"2024-09-23T19:25:59.499764Z","iopub.status.idle":"2024-09-23T19:26:01.547268Z","shell.execute_reply.started":"2024-09-23T19:25:59.499722Z","shell.execute_reply":"2024-09-23T19:26:01.546419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_classes = np.argmax(predictions, axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-09-18T17:09:12.494937Z","iopub.execute_input":"2024-09-18T17:09:12.495327Z","iopub.status.idle":"2024-09-18T17:09:12.500153Z","shell.execute_reply.started":"2024-09-18T17:09:12.495281Z","shell.execute_reply":"2024-09-18T17:09:12.499244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.DataFrame({\n 'ImageId': range(1, len(predicted_classes) + 1),\n 'Label': predicted_classes\n})\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-09-18T17:09:18.681524Z","iopub.execute_input":"2024-09-18T17:09:18.681901Z","iopub.status.idle":"2024-09-18T17:09:18.720029Z","shell.execute_reply.started":"2024-09-18T17:09:18.681863Z","shell.execute_reply":"2024-09-18T17:09:18.719332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
๐ Thank You for Reading the Notebook!
\n
Your support is truly appreciated. If you found this helpful, feel free to upvote and leave feedback.