{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30664,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"raw","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:22.790793Z","iopub.execute_input":"2024-03-06T05:41:22.791205Z","iopub.status.idle":"2024-03-06T05:41:22.799431Z","shell.execute_reply.started":"2024-03-06T05:41:22.791174Z","shell.execute_reply":"2024-03-06T05:41:22.798499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Notebook Outline\n**Introduction**\n\n* Brief overview of the project objective and the dataset.\n**Setup**\n\n* Import necessary libraries (e.g., NumPy, pandas, TensorFlow/Keras).\n**Data Loading**\n\n* Code to load train.csv and test.csv datasets from the zipped files.\n* Display the first few rows of the dataset to ensure it's loaded correctly.\n**Data Preprocessing**\n\n* Normalize pixel values to be between 0 and 1.\n* Reshape images from flat vectors to 28x28 pixel arrays.\n* One-hot encode the labels for the training data.\n**Exploratory Data Analysis (EDA)**\n\n* Visualize some images from the dataset.\n* Analyze the distribution of digits in the training data.\n**Model Building**\n\n* Define a neural network model for digit classification. Considering the data, a simple CNN or MLP model will be used.\n* Compile the model with an appropriate optimizer, loss function, and metrics.\n**Model Training**\n\n* Train the model with the training dataset, using a validation split to monitor performance.\n* Plot training history (accuracy and loss over epochs).\n**Model Evaluation**\n\n* Evaluate the model on a test set or a validation set to get the final accuracy.\n* Display a confusion matrix and classification report.\n**Predictions for Submission**\n\n* Predict labels for the test dataset.\n* Prepare a submission file format as specified (ImageId,Label).\n**Conclusion**\n\n* Summarize the findings, model performance, and any insights gained during the project.\n# Starting with Data Loading\n","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nimport numpy as np\nimport pandas as pd\nfrom tensorflow.keras.utils import to_categorical\nfrom sklearn.model_selection import train_test_split\n# Load the dataset\ntrain_df = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_df = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\n# Display the first few rows of the train dataset\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:22.804168Z","iopub.execute_input":"2024-03-06T05:41:22.804435Z","iopub.status.idle":"2024-03-06T05:41:26.822719Z","shell.execute_reply.started":"2024-03-06T05:41:22.804413Z","shell.execute_reply":"2024-03-06T05:41:26.821699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing\n1. **Normalize the Pixel Values:** Neural networks work best with data normalized to have a mean close to 0 and a standard deviation close to 1. Since our pixel values range from 0 to 255, we can simply divide by 255 to normalize them to the range [0, 1].\n \n1. **Reshape the Images:** The images in the MNIST dataset are 28x28 pixels, but they are flattened into 1D arrays of 784 values in the CSV file. We need to reshape them back into 2D format to feed them into our model correctly.\n \n1. **One-hot Encode the Labels:** The labels are currently single integers representing the digit in the image. We'll convert these to one-hot encoded vectors, which is a standard format for categorical targets in machine learning models.","metadata":{}},{"cell_type":"code","source":"# Separate features and target variable\nX_train = train_df.drop(\"label\", axis=1).values.astype('float32')\ny_train = train_df[\"label\"].values\nX_test = test_df.values.astype('float32')\n\n# Normalize the pixel values to be between 0 and 1\nX_train /= 255\nX_test /= 255\n\n# Reshape the images to 28x28 pixels\nX_train = X_train.reshape(-1, 28, 28, 1)  # Adding dimension for channel (grayscale)\nX_test = X_test.reshape(-1, 28, 28, 1)\n\n# One-hot encode the labels\ny_train_encoded = to_categorical(y_train, num_classes=10)\n\n# Display the new shape of the datasets\nprint(\"X_train shape:\", X_train.shape)\nprint(\"X_test shape:\", X_test.shape)\nprint(\"y_train_encoded shape:\", y_train_encoded.shape)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:26.824938Z","iopub.execute_input":"2024-03-06T05:41:26.825687Z","iopub.status.idle":"2024-03-06T05:41:27.026428Z","shell.execute_reply.started":"2024-03-06T05:41:26.825649Z","shell.execute_reply":"2024-03-06T05:41:27.025420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA)\n1. **Visualize Images from the Dataset:** It's helpful to see what the images look like and understand the variability in handwriting styles across the dataset.\n\n1. **Analyze the Distribution of Digits:** Checking the distribution of digits (0 through 9) can help us identify if the dataset is balanced or if there are any biases we need to be aware of.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Function to display images in a grid\ndef display_images(images, labels, nrows=2, ncols=5):\n    plt.figure(figsize=(10, 5))\n    for i in range(nrows * ncols):\n        plt.subplot(nrows, ncols, i+1)\n        plt.imshow(images[i].reshape(28,28), cmap='gray')\n        plt.title('Digit: {}'.format(labels[i]))\n        plt.axis('off')\n    plt.tight_layout()\n    plt.show()\n\n# Display images from the training dataset\ndisplay_images(X_train, y_train, nrows=2, ncols=5)\n\n# Analyze the distribution of digits in the training dataset\nplt.figure(figsize=(10, 5))\nsns.countplot(y_train)\nplt.title('Distribution of Digits in Training Data')\nplt.xlabel('Digit')\nplt.ylabel('Count')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:27.027508Z","iopub.execute_input":"2024-03-06T05:41:27.027783Z","iopub.status.idle":"2024-03-06T05:41:28.008956Z","shell.execute_reply.started":"2024-03-06T05:41:27.027759Z","shell.execute_reply":"2024-03-06T05:41:28.007973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Building\n1. **Import Necessary Libraries:** First, we import the necessary components from Keras to build our neural network.\n\n1. **Define the Model Architecture:** We will design a simple yet effective CNN architecture for our task.\n \n1. **Compile the Model:** After defining the model, we need to compile it with an appropriate optimizer, loss function, and metric(s) for evaluation.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam\n\n# Define the CNN model\ndef create_model():\n    model = Sequential([\n        Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)),\n        MaxPooling2D(pool_size=(2, 2)),\n        Conv2D(64, kernel_size=(3, 3), activation='relu'),\n        MaxPooling2D(pool_size=(2, 2)),\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),\n        Dense(10, activation='softmax')\n    ])\n    \n    return model\n\n# Create the model\nmodel = create_model()\n\n# Compile the model\nmodel.compile(optimizer=Adam(), \n              loss='categorical_crossentropy', \n              metrics=['accuracy'])\n\n# Model summary\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:28.011446Z","iopub.execute_input":"2024-03-06T05:41:28.011729Z","iopub.status.idle":"2024-03-06T05:41:28.074955Z","shell.execute_reply.started":"2024-03-06T05:41:28.011705Z","shell.execute_reply":"2024-03-06T05:41:28.074045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This model starts with two sets of convolutional and max pooling layers for feature extraction from the images, followed by a flattening layer to transform the 2D feature maps into a 1D vector. After that, it includes a dense layer for classification, a dropout layer to prevent overfitting, and finally, a dense output layer with softmax activation to output the probabilities of the 10 digits.\n\nThe model is compiled with the Adam optimizer, a common choice for deep learning tasks due to its adaptive learning rate properties, and categorical crossentropy as the loss function since we're dealing with a multi-class classification problem. The accuracy metric will help us evaluate the model's performance.","metadata":{}},{"cell_type":"markdown","source":"# Model Training and Evaluation\n1. **Split the Training Data:** Although we have a separate test dataset, it's good practice to also have a validation set during training to monitor the model's performance on unseen data. This helps in detecting overfitting early.\n \n1. **Train the Model:** We will fit the model to the training data using the model.fit method, also passing in the validation data.\n\n1. **Evaluate the Model:** After training, we'll evaluate the model's performance on the validation set to see how well it does.","metadata":{}},{"cell_type":"code","source":"# Splitting the training data into training and validation sets\nX_train_split, X_val_split, y_train_split, y_val_split = train_test_split(X_train, y_train_encoded, test_size=0.2, random_state=42)\n\n# Train the model\nhistory = model.fit(X_train_split, y_train_split, \n                    epochs=10, \n                    batch_size=128, \n                    validation_data=(X_val_split, y_val_split),\n                    verbose=2)\n\n# Plotting training history\nplt.figure(figsize=(12, 5))\n\nplt.subplot(1, 2, 1)\nplt.plot(history.history['accuracy'], label='Training Accuracy')\nplt.plot(history.history['val_accuracy'], label='Validation Accuracy')\nplt.title('Accuracy over epochs')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend()\n\nplt.subplot(1, 2, 2)\nplt.plot(history.history['loss'], label='Training Loss')\nplt.plot(history.history['val_loss'], label='Validation Loss')\nplt.title('Loss over epochs')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend()\n\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:28.075988Z","iopub.execute_input":"2024-03-06T05:41:28.076265Z","iopub.status.idle":"2024-03-06T05:41:44.406567Z","shell.execute_reply.started":"2024-03-06T05:41:28.076242Z","shell.execute_reply":"2024-03-06T05:41:44.405524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make predictions on the test dataset\npredictions = model.predict(X_test)\npredicted_classes = np.argmax(predictions, axis=1)\n\n# Prepare submission DataFrame\nsubmission = pd.DataFrame({\n    \"ImageId\": list(range(1, len(predicted_classes) + 1)),\n    \"Label\": predicted_classes\n})\n\n# Generate submission file\nsubmission_filepath = '/kaggle/working/submission.csv'\nsubmission.to_csv(submission_filepath, index=False)\n\nprint(f\"Submission file ready: {submission_filepath}\")\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:44.408144Z","iopub.execute_input":"2024-03-06T05:41:44.408600Z","iopub.status.idle":"2024-03-06T05:41:46.421580Z","shell.execute_reply.started":"2024-03-06T05:41:44.408562Z","shell.execute_reply":"2024-03-06T05:41:46.420540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The Final Score\n\nScore: 0.98882","metadata":{}},{"cell_type":"markdown","source":"# A BETTER VERSION","metadata":{}},{"cell_type":"markdown","source":"**Step 1: Data Augmentation**\nData augmentation increases the diversity of your training data, helping to prevent overfitting and improving generalization. We'll use ImageDataGenerator from Keras for this purpose.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n\n# Define data augmentation generator\ndatagen = ImageDataGenerator(\n    rotation_range=10,\n    zoom_range=0.1,\n    width_shift_range=0.1,\n    height_shift_range=0.1\n)\n\n# Fit the generator to your data\ndatagen.fit(X_train)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:46.422672Z","iopub.execute_input":"2024-03-06T05:41:46.423103Z","iopub.status.idle":"2024-03-06T05:41:46.476807Z","shell.execute_reply.started":"2024-03-06T05:41:46.423076Z","shell.execute_reply":"2024-03-06T05:41:46.476051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Step 2: Advanced Model Architecture**\nWe'll use a simplified ResNet-like architecture, which is known for its performance in image classification tasks. This architecture uses residual connections to enable the training of very deep networks.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, BatchNormalization, Activation, Add\n\ndef residual_block(x, filters, kernel_size=3, stride=1):\n    shortcut = x\n    x = Conv2D(filters, kernel_size=kernel_size, strides=stride, padding='same')(x)\n    x = BatchNormalization()(x)\n    x = Activation('relu')(x)\n    x = Conv2D(filters, kernel_size=kernel_size, strides=stride, padding='same')(x)\n    x = BatchNormalization()(x)\n    x = Add()([x, shortcut])\n    x = Activation('relu')(x)\n    return x\n\ndef build_resnet(input_shape=(28, 28, 1), num_classes=10):\n    inputs = Input(shape=input_shape)\n    x = Conv2D(32, kernel_size=(3, 3), padding='same')(inputs)\n    x = BatchNormalization()(x)\n    x = Activation('relu')(x)\n    \n    # Add residual blocks\n    for _ in range(3):\n        x = residual_block(x, 32)\n    \n    x = MaxPooling2D(pool_size=(2, 2))(x)\n    x = Flatten()(x)\n    x = Dense(128, activation='relu')(x)\n    outputs = Dense(num_classes, activation='softmax')(x)\n    \n    model = Model(inputs=inputs, outputs=outputs)\n    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n    return model\n\n# Build and compile the ResNet model\nmodel = build_resnet()\n\n# Model summary\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:46.477914Z","iopub.execute_input":"2024-03-06T05:41:46.478233Z","iopub.status.idle":"2024-03-06T05:41:46.672093Z","shell.execute_reply.started":"2024-03-06T05:41:46.478208Z","shell.execute_reply":"2024-03-06T05:41:46.671203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Step 3: Model Training with Data Augmentation**\nTrain the model using the augmented data. This approach helps the model learn from a wider variety of handwriting styles.","metadata":{}},{"cell_type":"code","source":"history = model.fit(datagen.flow(X_train_split, y_train_split, batch_size=64),\n                    epochs=30,\n                    validation_data=(X_val_split, y_val_split),\n                    verbose=2)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:41:46.673262Z","iopub.execute_input":"2024-03-06T05:41:46.673530Z","iopub.status.idle":"2024-03-06T05:48:31.128515Z","shell.execute_reply.started":"2024-03-06T05:41:46.673507Z","shell.execute_reply":"2024-03-06T05:48:31.127548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make predictions on the test dataset\npredictions = model.predict(X_test)\npredicted_classes = np.argmax(predictions, axis=1)\n\n# Prepare submission DataFrame\nsubmission = pd.DataFrame({\n    \"ImageId\": list(range(1, len(predicted_classes) + 1)),\n    \"Label\": predicted_classes\n})\n\n# Generate submission file\nsubmission_filepath = '/kaggle/working/submission.csv'\nsubmission.to_csv(submission_filepath, index=False)\n\nprint(f\"Submission file ready: {submission_filepath}\")","metadata":{"execution":{"iopub.status.busy":"2024-03-06T05:48:31.132443Z","iopub.execute_input":"2024-03-06T05:48:31.132758Z","iopub.status.idle":"2024-03-06T05:48:33.721139Z","shell.execute_reply.started":"2024-03-06T05:48:31.132730Z","shell.execute_reply":"2024-03-06T05:48:33.720158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Score: 0.99242","metadata":{}}]}