{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Introduction**\n\nThe objective of this project is to build a computer program that can recognize handwritten digits from images. We will use a dataset of images of handwritten digits and we will teach our program to learn from these images and predict what digit is shown in each image.\n\nTo achieve this, we will use a type of neural network called a Convolutional Neural Network (CNN). A CNN is a special type of neural network that can \"see\" patterns in an image by looking at small parts of the image at a time.\n\nWe will use the programming language Python to build our CNN. We will first load the dataset into our program and split it into a training dataset and a validation dataset. We will then create the CNN model and train it using the training dataset.\n\nOnce the model is trained, we will evaluate its performance using the validation dataset. We will measure how accurately the model can predict the digits in the validation dataset.\n\nFinally, we will make predictions on a separate test dataset using our trained model and save the predictions to a CSV file. We can submit this file to Kaggle, an online platform for data science competitions, to see how well our model performs compared to other models.\n\nThis Notebook consists of 3 main section:\n* Section 1: Data Preprocessing\n* Section 2: Building the Neural Network Model\n* Section 3: Training and Evaluating the Model\n","metadata":{}},{"cell_type":"markdown","source":"# Section 1: Data Preprocessing","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nfrom keras.utils.np_utils import to_categorical\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.callbacks import ReduceLROnPlateau\nfrom sklearn.metrics import confusion_matrix\n\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:36.838367Z","iopub.execute_input":"2023-04-26T23:17:36.839365Z","iopub.status.idle":"2023-04-26T23:17:36.854117Z","shell.execute_reply.started":"2023-04-26T23:17:36.839290Z","shell.execute_reply":"2023-04-26T23:17:36.852916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the training and test data\ntrain_df = pd.read_csv('../input/digit-recognizer/train.csv')\ntest_df = pd.read_csv('../input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:36.856927Z","iopub.execute_input":"2023-04-26T23:17:36.857379Z","iopub.status.idle":"2023-04-26T23:17:39.601498Z","shell.execute_reply.started":"2023-04-26T23:17:36.857325Z","shell.execute_reply":"2023-04-26T23:17:39.600243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# EDA\nprint(train_df.shape)\nprint(test_df.shape)","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.604103Z","iopub.execute_input":"2023-04-26T23:17:39.604567Z","iopub.status.idle":"2023-04-26T23:17:39.615196Z","shell.execute_reply.started":"2023-04-26T23:17:39.604522Z","shell.execute_reply":"2023-04-26T23:17:39.613980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# EDA\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.616943Z","iopub.execute_input":"2023-04-26T23:17:39.617264Z","iopub.status.idle":"2023-04-26T23:17:39.636954Z","shell.execute_reply.started":"2023-04-26T23:17:39.617217Z","shell.execute_reply":"2023-04-26T23:17:39.635753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Section 2: Building the Neural Network Model","metadata":{}},{"cell_type":"code","source":"# Split the training data into features (X_train) and labels (y_train)\nX_train = train_df.drop('label', axis=1)\ny_train = train_df['label']\nX_test = test_df\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.639839Z","iopub.execute_input":"2023-04-26T23:17:39.640582Z","iopub.status.idle":"2023-04-26T23:17:39.729508Z","shell.execute_reply.started":"2023-04-26T23:17:39.640535Z","shell.execute_reply":"2023-04-26T23:17:39.728260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255.0\nX_test = X_test / 255.0\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.732029Z","iopub.execute_input":"2023-04-26T23:17:39.732896Z","iopub.status.idle":"2023-04-26T23:17:39.912272Z","shell.execute_reply.started":"2023-04-26T23:17:39.732850Z","shell.execute_reply":"2023-04-26T23:17:39.911094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The pixel values in the images range from 0 to 255, where 0 represents black and 255 represents white. By dividing all pixel values by 255, we scale the pixel values to be in the range of 0 to 1. Normalization is important for training neural networks because it helps to improve convergence and speed up training.","metadata":{}},{"cell_type":"code","source":"# Reshape the data\nX_train = X_train.values.reshape(-1,28,28,1)\nX_test = X_test.values.reshape(-1,28,28,1)\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.914068Z","iopub.execute_input":"2023-04-26T23:17:39.914505Z","iopub.status.idle":"2023-04-26T23:17:39.921527Z","shell.execute_reply.started":"2023-04-26T23:17:39.914443Z","shell.execute_reply":"2023-04-26T23:17:39.920079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The input data originally has a 2-dimensional shape of (n_samples, n_features) where n_samples is the number of images in the dataset and n_features is the number of pixels in each image. By reshaping it to a 4-dimensional shape, we are indicating to the neural network that the input is a set of images with a height and width of 28 pixels and a depth of 1 (because the images are grayscale).\n> \n> The \"-1\" argument is used to automatically determine the number of samples (i.e., the number of images) in the dataset. The \"values\" attribute is used to convert the Pandas data frame to a numpy array which is required by Keras.","metadata":{}},{"cell_type":"code","source":"# One-hot encode the target variable\ny_train = to_categorical(y_train, num_classes=10)","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.923368Z","iopub.execute_input":"2023-04-26T23:17:39.924002Z","iopub.status.idle":"2023-04-26T23:17:39.932160Z","shell.execute_reply.started":"2023-04-26T23:17:39.923961Z","shell.execute_reply":"2023-04-26T23:17:39.931008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the train dataset into train and validation datasets\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.1, random_state=2)","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:39.934155Z","iopub.execute_input":"2023-04-26T23:17:39.934676Z","iopub.status.idle":"2023-04-26T23:17:40.318275Z","shell.execute_reply.started":"2023-04-26T23:17:39.934608Z","shell.execute_reply":"2023-04-26T23:17:40.317169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Section 3: Training and Evaluating the Model","metadata":{}},{"cell_type":"code","source":"# Build a convolutional neural network using Keras.\nmodel = Sequential()\n\nmodel.add(Conv2D(filters=32, kernel_size=(5,5), padding='Same', activation='relu', input_shape=(28,28,1)))\nmodel.add(Conv2D(filters=32, kernel_size=(5,5), padding='Same', activation='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters=64, kernel_size=(3,3), padding='Same', activation='relu'))\nmodel.add(Conv2D(filters=64, kernel_size=(3,3), padding='Same', activation='relu'))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:40.320052Z","iopub.execute_input":"2023-04-26T23:17:40.320505Z","iopub.status.idle":"2023-04-26T23:17:40.446123Z","shell.execute_reply.started":"2023-04-26T23:17:40.320463Z","shell.execute_reply":"2023-04-26T23:17:40.445292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> This block of code is responsible for defining the architecture of the convolutional neural network (CNN) using Keras. The Sequential function is used to create a sequential stack of layers, where each layer is added one after the other.\n> \n> The CNN architecture consists of several layers that perform different operations such as convolution, pooling, and dropout. These layers help in extracting relevant features from the input image and reduce the dimensionality of the data, making it easier for the network to learn and generalize.\n> \n> The Conv2D layers are convolutional layers that perform the operation of convolution, which applies a filter to the input image and produces a feature map. The filters are learned during training and help in identifying important features in the input image.\n> \n> The MaxPool2D layers are pooling layers that downsample the feature maps by taking the maximum value in a local region. This helps in reducing the spatial size of the feature maps and makes the network more robust to translation invariance.\n> \n> The Dropout layers randomly drop out some of the neurons during training, which helps in preventing overfitting and improves the generalization of the network.\n> \n> The Flatten layer is used to convert the 2D feature maps into a 1D feature vector, which is then fed into a fully connected layer. The Dense layers are fully connected layers that perform the operation of matrix multiplication and apply an activation function to produce the output.\n>\n> The model.summary() function is used to print a summary of the model architecture, which includes the number of parameters in each layer and the total number of parameters in the model. This helps in debugging and optimizing the model architecture","metadata":{}},{"cell_type":"code","source":"# Compile the model with an optimizer, loss function, and metrics.\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:40.449646Z","iopub.execute_input":"2023-04-26T23:17:40.450112Z","iopub.status.idle":"2023-04-26T23:17:40.471789Z","shell.execute_reply.started":"2023-04-26T23:17:40.450072Z","shell.execute_reply":"2023-04-26T23:17:40.470748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train the model using the training and validation sets\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy', patience=3, verbose=1, factor=0.5, min_lr=0.00001)\n\nhistory = model.fit(X_train, y_train, batch_size=86, epochs=30, validation_data=(X_val, y_val), verbose=1, callbacks=[learning_rate_reduction])\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:17:40.473489Z","iopub.execute_input":"2023-04-26T23:17:40.473862Z","iopub.status.idle":"2023-04-26T23:20:03.711081Z","shell.execute_reply.started":"2023-04-26T23:17:40.473821Z","shell.execute_reply":"2023-04-26T23:20:03.709963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> This output shows the training progress of a neural network model over several epochs. Each epoch is a complete pass through the entire training dataset.\n> \n> The output shows the current epoch number and the number of batches processed per epoch. The batch size is determined by the user and can affect the speed and quality of training.\n> \n> For each epoch, the output shows the loss and accuracy of the model on the training dataset, as well as the loss and accuracy on a separate validation dataset. The loss is a measure of how well the model is predicting the correct output, while the accuracy is the percentage of correct predictions made by the model.\n> \n> The output also shows the learning rate, which is a hyperparameter that determines how quickly the model adapts to new information during training. The learning rate can be adjusted during training to improve the model's performance.\n> \n> This model's loss and accuracy on both the training and validation sets are improving over time, which indicates that the model is learning and becoming more accurate at predicting the correct output. The learning rate is also being reduced automatically using a callback function, which can help improve the model's performance.","metadata":{}},{"cell_type":"code","source":"# Evaluate the model\nval_loss, val_acc = model.evaluate(X_val, y_val)\nprint('Validation Loss:', val_loss)\nprint('Validation Accuracy:', val_acc)\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:20:03.713267Z","iopub.execute_input":"2023-04-26T23:20:03.713972Z","iopub.status.idle":"2023-04-26T23:20:04.413552Z","shell.execute_reply.started":"2023-04-26T23:20:03.713929Z","shell.execute_reply":"2023-04-26T23:20:04.412381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Plot the training and validation accuracy curves\nplt.plot(history.history['accuracy'], label='Training Accuracy')\nplt.plot(history.history['val_accuracy'], label='Validation Accuracy')\nplt.title('Training and Validation Accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend(loc='lower right')\nplt.show()\n\n# Plot the training and validation loss curves\nplt.plot(history.history['loss'], label='Training Loss')\nplt.plot(history.history['val_loss'], label='Validation Loss')\nplt.title('Training and Validation Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend(loc='upper right')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:20:04.415243Z","iopub.execute_input":"2023-04-26T23:20:04.415939Z","iopub.status.idle":"2023-04-26T23:20:04.842289Z","shell.execute_reply.started":"2023-04-26T23:20:04.415894Z","shell.execute_reply":"2023-04-26T23:20:04.841286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> The plots shows the training and validation accuracy over epochs, provides information about how well the model is learning from the training data and how well it generalizes to new, unseen data (validation data). A model with high training accuracy but low validation accuracy may be overfitting, meaning it has learned to perform well on the training data but doesn't generalize well to new data. On the other hand, a model with low training accuracy may not be learning enough from the data.","metadata":{}},{"cell_type":"code","source":"# Get the predictions on the test set\ny_pred = model.predict(X_test)\n\n# Convert the predictions to integer labels\ny_pred = np.argmax(y_pred, axis=1)\n\n# Plot some examples of the test set with their predicted labels\nnum_examples = 9\nplt.figure(figsize=(10,10))\nfor i in range(num_examples):\n    plt.subplot(3, 3, i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.grid(False)\n    plt.imshow(X_test[i], cmap=plt.cm.binary)\n    plt.xlabel('Predicted Label: {}'.format(y_pred[i]))\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:20:04.843854Z","iopub.execute_input":"2023-04-26T23:20:04.844188Z","iopub.status.idle":"2023-04-26T23:20:07.524321Z","shell.execute_reply.started":"2023-04-26T23:20:04.844151Z","shell.execute_reply":"2023-04-26T23:20:07.523276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above plots shows the training and validation loss over epochs, provides information about how well the model is minimizing the error between its predictions and the true labels. A model with low training and validation loss indicates that it is making accurate predictions on both the training and validation data. If the validation loss is higher than the training loss, it may suggest that the model is overfitting","metadata":{}},{"cell_type":"code","source":"# Make predictions\npredictions = model.predict(X_test)\npredictions = np.argmax(predictions,axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:20:07.526050Z","iopub.execute_input":"2023-04-26T23:20:07.526449Z","iopub.status.idle":"2023-04-26T23:20:09.732653Z","shell.execute_reply.started":"2023-04-26T23:20:07.526411Z","shell.execute_reply":"2023-04-26T23:20:09.731630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save to csv file\nsubmission = pd.DataFrame({'ImageId':list(range(1,len(predictions)+1)), 'Label':predictions})\nsubmission.to_csv('submission.csv', index=False)\n","metadata":{"execution":{"iopub.status.busy":"2023-04-26T23:20:09.735434Z","iopub.execute_input":"2023-04-26T23:20:09.736203Z","iopub.status.idle":"2023-04-26T23:20:09.774115Z","shell.execute_reply.started":"2023-04-26T23:20:09.736163Z","shell.execute_reply":"2023-04-26T23:20:09.773184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Conclusion:\n\nBased on the provided output, it seems that the model achieved high accuracy on both the training and validation sets. The training accuracy was not provided in the output, but the validation accuracy was reported as 0.994, which means that the model correctly classified 99.4% of the images in the validation set. The validation loss was also very low at 0.035, which indicates that the model was able to minimize the error between its predictions and the true labels.","metadata":{}}]}