{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30615,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognition with Deep Learning\n\n## Introduction\n\nThis is a contribution to the Kaggle Digit Recognizer competition: AstroDave, Will Cukierski. (2012). Digit Recognizer. Kaggle. https://kaggle.com/competitions/digit-recognizer.\n\nIn the realm of computer vision and machine learning, digit recognition serves as a fundamental problem with diverse applications ranging from automated postal code sorting to OCR (Optical Character Recognition). This notebook explores the creation of a digit recognition model using deep learning techniques.\n\nTraditionally, convolutional neural networks (CNNs) have been the go-to architecture for image-related tasks, including digit recognition. However, this project takes a different approach by utilizing a deep learning model without CNNs. Instead, we use the provided input vector of black and white pixels, exploring the capacity of the model to discern patterns and features inherent in the pixel representations.\n\nThe objective is to develop an efficient and accurate digit recognition system that can generalize well to unseen data. Throughout the notebook, we will walk through data preprocessing, model architecture design, training, and evaluation processes. Additionally, we will incorporate best practices in hyperparameter tuning and regularization to enhance the model's robustness.\n\nBy the end of this exploration, we aim to have a well-optimized and interpretable digit recognition model, shedding light on the inner workings of deep learning when applied to image classification tasks without the conventional use of CNNs.\n\nLet's embark on this journey of digit recognition through deep learning, delving into the intricacies of pixel-based input representations and uncovering insights into the model's decision-making processes.\n\n\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-12-12T19:04:37.285837Z","iopub.execute_input":"2023-12-12T19:04:37.286220Z","iopub.status.idle":"2023-12-12T19:04:37.670678Z","shell.execute_reply.started":"2023-12-12T19:04:37.286190Z","shell.execute_reply":"2023-12-12T19:04:37.669411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training and Testing dataframes are created\ndf_train_digits = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndf_test_digits = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:04:40.069330Z","iopub.execute_input":"2023-12-12T19:04:40.069830Z","iopub.status.idle":"2023-12-12T19:04:45.520250Z","shell.execute_reply.started":"2023-12-12T19:04:40.069794Z","shell.execute_reply":"2023-12-12T19:04:45.518986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_digits.head()","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:04:45.521790Z","iopub.execute_input":"2023-12-12T19:04:45.522100Z","iopub.status.idle":"2023-12-12T19:04:45.552046Z","shell.execute_reply.started":"2023-12-12T19:04:45.522071Z","shell.execute_reply":"2023-12-12T19:04:45.551115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Inspection and Exploration\n\n### Checking for Missing Values\n\nBefore starting the model development, it is essential to ensure the integrity of our dataset. Here, we check for any missing values and find that there are none.","metadata":{}},{"cell_type":"code","source":"print('Number of missing values in the data set: ',df_train_digits.isna().sum().sum())","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:04:45.553851Z","iopub.execute_input":"2023-12-12T19:04:45.554251Z","iopub.status.idle":"2023-12-12T19:04:45.598237Z","shell.execute_reply.started":"2023-12-12T19:04:45.554213Z","shell.execute_reply":"2023-12-12T19:04:45.596899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Illustrating the Distribution of Digits\n\nUnderstanding the distribution of digits in our dataset provides valuable insights into potential class imbalances. We visualize the label distribution as a bar plot and find that the data is mostly balanced between the different classes and we have roughly 4000 images for each digit.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Assuming the digit labels are in a column named 'label'\ndigit_distribution = df_train_digits['label'].value_counts().sort_index()\n\n# Plotting the distribution\nplt.figure(figsize=(10, 6))\nsns.barplot(x=digit_distribution.index, y=digit_distribution.values, palette=\"viridis\")\nplt.title(\"Distribution of Digits in the Dataset\")\nplt.xlabel(\"Digit\")\nplt.ylabel(\"Count\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:04:47.943997Z","iopub.execute_input":"2023-12-12T19:04:47.944399Z","iopub.status.idle":"2023-12-12T19:04:48.935519Z","shell.execute_reply.started":"2023-12-12T19:04:47.944364Z","shell.execute_reply":"2023-12-12T19:04:48.934166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Digit Illustration: Visualizing Pixel Representations\n\nTo gain deeper insights into our digit recognition dataset, we present visual representations of digits. \nThe visualizations below display pixel representations of the selected digits, allowing us to observe the inherent variations and similarities. This step is crucial for understanding the challenges our model may face, especially when distinguishing between digits that share visual similarities.\nBy presenting digit triplets, we aim to highlight nuances in pixel patterns that contribute to the complexity of digit recognition.\n\n","metadata":{}},{"cell_type":"code","source":"# Function to show three digits in a row\ndef show_three_digits_in_row(pixel_values1, label1, pixel_values2, label2, pixel_values3, label3):\n    fig, axes = plt.subplots(1, 3, figsize=(12, 4))\n\n    # Display the first digit\n    axes[0].imshow(np.array(pixel_values1).reshape(28, 28), cmap='gray')\n    axes[0].set_title(f\"Digit: {label1}\")\n    axes[0].axis('off')\n\n    # Display the second digit\n    axes[1].imshow(np.array(pixel_values2).reshape(28, 28), cmap='gray')\n    axes[1].set_title(f\"Digit: {label2}\")\n    axes[1].axis('off')\n\n    # Display the third digit\n    axes[2].imshow(np.array(pixel_values3).reshape(28, 28), cmap='gray')\n    axes[2].set_title(f\"Digit: {label3}\")\n    axes[2].axis('off')\n\n    plt.show()\n\n# Display three digits in a row for each triplet (0, 1, and 2; 2, 3, and 4; etc.), excluding 9\nfor digit in range(0, 9, 3):  # Start from 0, increment by 3, exclude 9\n    digit_data_1 = df_train_digits[df_train_digits['label'] == digit].iloc[0]\n    digit_data_2 = df_train_digits[df_train_digits['label'] == digit + 1].iloc[0]\n    digit_data_3 = df_train_digits[df_train_digits['label'] == digit + 2].iloc[0]\n\n    pixel_values1, label1 = digit_data_1.drop('label').values, digit_data_1['label']\n    pixel_values2, label2 = digit_data_2.drop('label').values, digit_data_2['label']\n    pixel_values3, label3 = digit_data_3.drop('label').values, digit_data_3['label']\n\n    show_three_digits_in_row(pixel_values1, label1, pixel_values2, label2, pixel_values3, label3)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:04:50.852797Z","iopub.execute_input":"2023-12-12T19:04:50.853159Z","iopub.status.idle":"2023-12-12T19:04:51.992442Z","shell.execute_reply.started":"2023-12-12T19:04:50.853129Z","shell.execute_reply":"2023-12-12T19:04:51.991254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Principal Component Analysis (PCA)\n\nThe next step is exploring the data set through Principal Component Analysis (PCA). \n\n#### **Observations:**\n\nIn the scatter plot of principal components, each digit forms its own cloud of values, contributing to the overall complexity of digit representation. Notably, digits 0 and 1 exhibit distinct and well-separated clouds, indicating clear and easily distinguishable features. This aligns with our expectations, given the simple forms of these digits as a line and circle that are the most easily distinguishable among the digits.\n\n#### **Implications:**\n\n1. **Distinguishable Features of 0 and 1:**\n   - The unique and separate clouds formed by digits 0 and 1 highlight their simplicity and straightforward spatial representations. This distinctiveness bodes well for the effectiveness of our model in discerning between these digits.\n   \n2. **Overlapping Digit Clouds:**\n   - Digits 2 through 9 showcase varying degrees of overlap, suggesting shared spatial characteristics and challenges in their discrimination. The clustering of digit clouds reflects the intricacies of distinguishing between these numerals based on spatial arrangements of binary pixels. For instance, the digits 4, 7 and 9 (or 6 and 8) are harder to distinguish and their clouds mostly overlap. \n\n#### **Interpretation:**\n\n- **Spatial Complexity Across Digits:**\n  - The spatial complexity observed across digit clouds underscores the need for a model capable of capturing intricate spatial relationships for accurate digit recognition.\n\n- **Model Considerations:**\n  - While digits 0 and 1 present clear features, the challenge lies in developing a model that can navigate the overlapping digit clouds of 2 through 9. Model adaptations and feature engineering strategies should address this complexity for robust performance.\n\nThis PCA analysis serves as a valuable guide, emphasizing the varying degrees of overlap between digit clouds and the distinctive features exhibited by certain digits.\n\n","metadata":{}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\n\n# Extract pixel values\npixel_columns = df_train_digits.columns[df_train_digits.columns.str.startswith('pixel')]\npixel_data = df_train_digits[pixel_columns]\n\n# Apply PCA\npca = PCA(n_components=2)\nprincipal_components = pca.fit_transform(pixel_data)\n\n# Create a DataFrame with the principal components\nprincipal_df = pd.DataFrame(data=principal_components, columns=['Principal Component 1', 'Principal Component 2'])\nfinal_df = pd.concat([principal_df, df_train_digits['label']], axis=1)\n\n# Set the hue_order to ensure all digits are represented in the legend\nhue_order = sorted(df_train_digits['label'].unique())\n\n# Plot the PCA results with manual hue_order\nplt.figure(figsize=(12, 8))\nscatter = sns.scatterplot(x='Principal Component 1', y='Principal Component 2', hue='label', hue_order=hue_order, data=final_df, palette='viridis', alpha=0.8)\nplt.title('PCA: Visualization of Digit Data')\n\n# Manually adjust the legend outside Seaborn\nscatter.legend(title='Digit', bbox_to_anchor=(1.05, 1), loc='upper left')\nplt.show()\n\n\n","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:04:58.133686Z","iopub.execute_input":"2023-12-12T19:04:58.134167Z","iopub.status.idle":"2023-12-12T19:05:02.985141Z","shell.execute_reply.started":"2023-12-12T19:04:58.134124Z","shell.execute_reply":"2023-12-12T19:05:02.984187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Splitting: Training and Testing Sets\n\nTo assess our model's performance and ensure its generalization ability, we split our dataset into two subsets:\n\n1. **Training Set:**\n   - Used to train the neural network, the majority of the data teaches the model patterns and features.\n\n2. **Validation Set:**\n   - Kept separate for evaluation, this unseen dataset assesses the model's ability to generalize to new, unseen data.\n\n### Benefits:\n\n- **Performance Evaluation:**\n  - Assesses the model's accuracy on new, real-world data.\n\n- **Preventing Overfitting:**\n  - Guards against overfitting by ensuring consistent performance across training and validation sets.\n\n### Implementation:\n\nWe use `train_test_split` from scikit-learn to randomly divide the dataset.\n","metadata":{}},{"cell_type":"code","source":"# Split data into training and testing data\nfrom sklearn.model_selection import train_test_split\n\nX_train, X_val, y_train, y_val = train_test_split(df_train_digits.drop('label',axis=1), df_train_digits['label'], test_size=0.33, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:05:03.991969Z","iopub.execute_input":"2023-12-12T19:05:03.993020Z","iopub.status.idle":"2023-12-12T19:05:04.378060Z","shell.execute_reply.started":"2023-12-12T19:05:03.992978Z","shell.execute_reply":"2023-12-12T19:05:04.376637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the input size \ninput_shape = X_train.iloc[0].shape","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:05:07.724158Z","iopub.execute_input":"2023-12-12T19:05:07.724595Z","iopub.status.idle":"2023-12-12T19:05:07.730095Z","shell.execute_reply.started":"2023-12-12T19:05:07.724559Z","shell.execute_reply":"2023-12-12T19:05:07.728467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Description\n\n## Model Architecture:\n\nOur deep learning model is designed for digit recognition using TensorFlow and Keras. Here's an overview of the architecture:\n\n1. **Input Layer:**\nThe input layer assumes each image is represented as a flattened 1D array of pixel values. The shape is determined by the input_shape parameter.\n2. **Hidden Layers:**\nThe model includes two hidden layers with 128 and 64 units, respectively, using the ReLU activation function. These layers capture hierarchical features in the data.\nTo enhance generalization and prevent overfitting, dropout layers with a rate of 0.3 are inserted after each hidden layer. Additionally, batch normalization is applied to standardize and stabilize the input to each layer.\n3. **Output Layer:**\nThe output layer consists of 10 units, corresponding to the possible digits (assuming 10 classes for digits 0-9). It uses the softmax activation function for multiclass classification.\n4. **Compilation:**\nThe model is compiled with the Adam optimizer, sparse categorical crossentropy loss (since labels are integers), and accuracy as the evaluation metric.\n5. **Early Stopping:**\nTo prevent overfitting and speed up training, an early stopping callback is implemented. Training stops if the validation loss does not improve for a specified number of epochs (patience).","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers, models, callbacks\n\n# Assume your data is stored in X_train and y_train (features and labels)\n# X_train should be a 2D array where each row represents a flattened image of pixels\n\n# Define the model\nmodel = models.Sequential([\n    # Input layer (assuming each image is a 1D array of pixel values)\n    layers.InputLayer(input_shape=input_shape),\n\n    # Hidden layer with 128 units and ReLU activation\n    layers.Dense(128, activation='relu'),\n    layers.Dropout(0.3),\n    layers.BatchNormalization(),\n\n    # Add another hidden layer if needed\n    layers.Dense(64, activation='relu'),\n    layers.Dropout(0.3),\n    layers.BatchNormalization(),\n\n    # Output layer with 10 units (assuming 10 digits) and softmax activation for multiclass classification\n    layers.Dense(10, activation='softmax')\n])\n\n# Compile the model\nmodel.compile(optimizer='adam',\n              loss='sparse_categorical_crossentropy',  # Use 'categorical_crossentropy' if labels are one-hot encoded\n              metrics=['accuracy'])\n\n# Display the model summary\nmodel.summary()\n\n# Early stopping callback\nearly_stopping = callbacks.EarlyStopping(\n    monitor='val_loss',  # Stop training when the validation loss doesn't improve\n    patience=3,           # Number of epochs with no improvement after which training will be stopped\n    restore_best_weights=True  # Restore model weights from the epoch with the best value of the monitored quantity\n)\n\n# Fit the model to the training data\nmodel.fit(X_train, y_train, validation_data = [X_val,y_val],epochs=100, batch_size=32,callbacks=[early_stopping])","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:09:50.085934Z","iopub.execute_input":"2023-12-12T19:09:50.086355Z","iopub.status.idle":"2023-12-12T19:11:02.891841Z","shell.execute_reply.started":"2023-12-12T19:09:50.086321Z","shell.execute_reply":"2023-12-12T19:11:02.890726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model illustration\n\nHere, we see some further illustration of the model","metadata":{}},{"cell_type":"code","source":"!pip install pydot\n!pip install pydot_ng\n!pip install graphviz\n!pip install plot_model","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:19:39.430063Z","iopub.execute_input":"2023-12-12T19:19:39.430477Z","iopub.status.idle":"2023-12-12T19:20:26.970801Z","shell.execute_reply.started":"2023-12-12T19:19:39.430446Z","shell.execute_reply":"2023-12-12T19:20:26.968725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.utils import plot_model\n\n# Plot the model architecture with custom styling\nplot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=False, rankdir='TB', expand_nested=True, dpi=96)#, layer_colors=layer_colors)","metadata":{"execution":{"iopub.status.busy":"2023-12-12T19:21:08.527783Z","iopub.execute_input":"2023-12-12T19:21:08.528192Z","iopub.status.idle":"2023-12-12T19:21:08.617329Z","shell.execute_reply.started":"2023-12-12T19:21:08.528143Z","shell.execute_reply":"2023-12-12T19:21:08.616187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Misclassification Analysis and Illustration\n\nUnderstanding the misclassification patterns of our model provides valuable insights into its strengths and weaknesses. Misclassification analysis involves examining instances where the model predicts the wrong class and identifying potential patterns or challenges.\n\n1. **Methodology:**\nIdentification of Misclassified Samples:\nAfter training the model, predictions are made on the validation set, and misclassified samples are identified by comparing predicted labels with true labels.\n2. **Visualization of Misclassified Images:**\nA selection of misclassified images is visualized to gain a qualitative understanding of the types of errors the model makes. This visualization helps identify common patterns or challenges in misclassification.\n3. **Distribution of Misclassified Numbers:**\nThe distribution of misclassified numbers is plotted to highlight which digits are more prone to misclassification. This analysis can guide further investigation into specific digits that might pose challenges for the model.\n4. **Heatmap for Digit Misclassification Patterns:**\nTo delve deeper into the misclassification patterns, creating a heatmap is an insightful approach. The heatmap visually represents the confusion matrix, showcasing how often each true digit is misclassified as another digit. Rows correspond to true labels, while columns represent predicted labels. Lighter regions on the heatmap indicate higher frequencies of misclassification between specific digit pairs. Analyzing this heatmap allows us to pinpoint which digits tend to be confused with each other, providing valuable information for model refinement.\n\nAs suggested by the PCA, distinguishing between 4 and 9 respectively 7 and 9 is hard for the model. Further pairs that are hard to distinguish in this data set are 3 and 5 as well as 3 and 8. Looking at the form of the digits, we see that each of them share similar characteristics such as the form.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\n\n# Assuming you have a trained model named 'model' and validation data named 'X_val' and 'y_val'\n\n# Step 1: Make predictions on the validation data\npredictions = model.predict(X_val)\n\n# Step 2: Identify misclassified samples\nmisclassified_indices = np.argmax(predictions, axis=1) != y_val\nmisclassified_images = X_val[misclassified_indices]\ntrue_labels = y_val[misclassified_indices]\npredicted_labels = np.argmax(predictions[misclassified_indices], axis=1)\n# Step 3: Visualize misclassified images with increased space between subplots\nnum_samples_to_visualize = 5\n\n# Adjusting spacing between subplots\nfig, axes = plt.subplots(1, num_samples_to_visualize, figsize=(15, 5), gridspec_kw={'hspace': 0.5})\n\n# Flatten the axes array for easier indexing\naxes = axes.flatten()\n\nfor i in range(num_samples_to_visualize):\n    axes[i].imshow(misclassified_images.iloc[i].to_numpy().reshape(28, 28), cmap='gray')\n    axes[i].set_title(f'True: {true_labels.iloc[i]}, Predicted: {predicted_labels[i]}')\n    axes[i].axis('off')\n\nplt.suptitle('Misclassified Images on Validation Data', y=1.05,fontsize=20)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-12-12T15:24:57.864755Z","iopub.execute_input":"2023-12-12T15:24:57.865222Z","iopub.status.idle":"2023-12-12T15:24:58.396374Z","shell.execute_reply.started":"2023-12-12T15:24:57.865184Z","shell.execute_reply":"2023-12-12T15:24:58.394881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Step 4: Analyze misclassification patterns\nmisclassification_analysis = {}\nfor true_label, predicted_label in zip(true_labels, predicted_labels):\n    if true_label not in misclassification_analysis:\n        misclassification_analysis[true_label] = {predicted_label: 1}\n    else:\n        if predicted_label not in misclassification_analysis[true_label]:\n            misclassification_analysis[true_label][predicted_label] = 1\n        else:\n            misclassification_analysis[true_label][predicted_label] += 1\n            \n# Print misclassification analysis\nprint(\"Misclassification Analysis:\")\nfor true_label, predictions in misclassification_analysis.items():\n    print(f\"True Label {true_label}: {predictions}\")\n\n# Visualize misclassification patterns\nfig, ax = plt.subplots(figsize=(10, 6))\nheatmap_data = np.zeros((10, 10))\nfor true_label, predictions in misclassification_analysis.items():\n    for predicted_label, count in predictions.items():\n        heatmap_data[true_label, predicted_label] = count\n\nsns.heatmap(heatmap_data, annot=True, fmt='g', cmap='viridis', cbar=True, ax=ax)\nax.set_xlabel('Predicted Label')\nax.set_ylabel('True Label')\nax.set_title('Misclassification Patterns')\nplt.show()\n\n\n# Distribution of digits that are wrongly labeled\nplt.figure(figsize=(8, 5))\nsns.countplot(x=true_labels, palette='viridis')\nplt.title('Distribution of Misclassified (True) Digits')\nplt.xlabel('Predicted Number')\nplt.ylabel('Count')\nplt.show()\n\n# Distribution of digits that are wrongly labeled\nplt.figure(figsize=(8, 5))\nsns.countplot(x=predicted_labels, palette='viridis')\nplt.title('Distribution of Misclassified (False) Digits')\nplt.xlabel('Predicted Number')\nplt.ylabel('Count')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-12T15:30:07.714594Z","iopub.execute_input":"2023-12-12T15:30:07.715097Z","iopub.status.idle":"2023-12-12T15:30:09.038722Z","shell.execute_reply.started":"2023-12-12T15:30:07.715053Z","shell.execute_reply":"2023-12-12T15:30:09.037446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test Predictions\n\nAfter training the model and fine-tuning its performance, it's time to make predictions on the test dataset and prepare the results for submission. The process involves utilizing the trained model to predict the labels of the test data. The predictions are then organized into a DataFrame, which is a structured tabular data format. Each row in the DataFrame corresponds to an image in the test set, and the columns represent the predicted probability scores for each digit class (0-9). This DataFrame is crucial for generating a submission file that can be uploaded to the competition platform. ","metadata":{}},{"cell_type":"code","source":"# Make predictions on the test data\npredictions = model.predict(df_test_digits)\n# If you are doing classification and want the predicted class labels\npredicted_labels = predictions.argmax(axis=1)\n\n# Create a submission dataframe\nsubmission_df = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\nsubmission_df['Label'] = predicted_labels.squeeze()\nsubmission_df.to_csv('submission.csv', index=False)\n\nprint(f\"Predictions saved to DataFrame\")","metadata":{"execution":{"iopub.status.busy":"2023-12-11T19:09:53.595653Z","iopub.execute_input":"2023-12-11T19:09:53.596631Z","iopub.status.idle":"2023-12-11T19:09:56.100603Z","shell.execute_reply.started":"2023-12-11T19:09:53.596588Z","shell.execute_reply":"2023-12-11T19:09:56.099110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Future Work - Other Deep Learning Models:\nWhile this model gives us a decent accuracy, there is still a lot of room for improvement. Future work can use some of the following ideas:\n\n1. **Convolutional Neural Network (CNN):**\nWell-suited for image data, CNNs can capture spatial hierarchies and are robust to variations in position and scale.\n2. **Recurrent Neural Network (RNN):**\nEffective for sequential data, RNNs can be useful if there is temporal information or dependencies between pixels.\n3. **Transfer Learning (e.g., using pre-trained models like ResNet or VGG):**\nLeveraging pre-trained models on large image datasets can boost performance, especially with limited labeled data.\n4. **Ensemble Models:**\nCombining multiple models (e.g., stacking or bagging) can enhance overall predictive performance.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}