{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30822,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognition Using MNIST - CNN","metadata":{"execution":{"iopub.status.busy":"2025-01-08T13:32:24.294287Z","iopub.execute_input":"2025-01-08T13:32:24.294536Z","iopub.status.idle":"2025-01-08T13:32:24.298735Z","shell.execute_reply.started":"2025-01-08T13:32:24.294497Z","shell.execute_reply":"2025-01-08T13:32:24.297729Z"}}},{"cell_type":"markdown","source":"## Intro","metadata":{}},{"cell_type":"markdown","source":"Hello! \n\nWelcome to my first Neural Networks project. For this project I'll be attempting the famous MNIST handwriiten number classification problem. I'll use Convolutional Neural Networks (CNN) to correctly identify and classify the numbers.\n\n### CNN\n\nCNN is a specialized type of neural network designed for processing grid like data searches such as images. CNN is a great use-case for this project because it uses ***filters(kernels)*** to scan small parts of the image at a time, capturing special patterns like edges, textures, or shapes.\n","metadata":{"jupyter":{"source_hidden":true}}},{"cell_type":"markdown","source":"The **MNIST database** (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems.The database is also widely used for training and testing in the field of machine learning. It was created by \"re-mixing\" the samples from NIST's original datasets.The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments. Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels.\n\nThe MNIST database contains **60,000** training images and **10,000** testing images.Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.The original creators of the database keep a list of some of the methods tested on it. In their original paper, they use a support-vector machine to get an error rate of 0.8%.","metadata":{}},{"cell_type":"markdown","source":"## Overview of MNIST","metadata":{}},{"cell_type":"markdown","source":"## Import Libraries","metadata":{}},{"cell_type":"code","source":"#to suppress warnings:\ndef warn(*args, **kwargs):\n    pass\n\nimport warnings\nwarnings.warn = warn\nwarnings.filterwarnings('ignore')\n\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing\nimport matplotlib.pyplot as plt #visualizations\n%matplotlib inline\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix #to evaluate accuracy of a classification\n\n#from keras.utils.np_utils import to_categorical #for one hot encoding/ vecotrization of varibles\nfrom keras.utils import to_categorical\nfrom keras.models import Sequential #Sequential Model/API for our model\n\n#Dense allows us to connect each neuron/output \n#Dropout layer randomly ses input units to zero, batchnorm - stabilizes the learning\n#Conv2D a 2x2 Conv Kernel # Flatten - flattens the image, MaxPool2D takes the max when filtering\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\n\nfrom keras.optimizers import Adam #for optimization\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator #random data from our train set\nfrom keras.callbacks import LearningRateScheduler # adjust LR each epoch","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:39:51.268398Z","iopub.execute_input":"2025-02-07T19:39:51.268702Z","iopub.status.idle":"2025-02-07T19:40:02.664431Z","shell.execute_reply.started":"2025-02-07T19:39:51.268664Z","shell.execute_reply":"2025-02-07T19:40:02.663513Z"},"_kg_hide-output":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Import Dataset","metadata":{}},{"cell_type":"code","source":"# get the dataset path.\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:02.665426Z","iopub.execute_input":"2025-02-07T19:40:02.666143Z","iopub.status.idle":"2025-02-07T19:40:02.673833Z","shell.execute_reply.started":"2025-02-07T19:40:02.666110Z","shell.execute_reply":"2025-02-07T19:40:02.672951Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Exploration","metadata":{}},{"cell_type":"markdown","source":"### Objective\n\n\nThe main objective of this analysis is to build an effective and accurate model for recognizing handwritten digits using the MNIST dataset. This analysis aims to explore the dataset comprehensively, understand the underlying patterns, and implement machine learning models for digit classification. The approach includes preprocessing the data, visualizing pixel intensity distributions, and testing various algorithms to identify the best-performing one. Potential challenges, such as variations in digit styles or overlapping pixel patterns, are anticipated and factored into the analysis to ensure robustness. By addressing these aspects, the project ensures a structured and insightful exploration of the MNIST data.","metadata":{}},{"cell_type":"markdown","source":"## Overview of Data","metadata":{}},{"cell_type":"markdown","source":"The MNIST (Modified National Institute of Standards and Technology) dataset consists of 70,000 handwritten digit images, specifically:\n\n* 60,000 training images\n* 10,000 test images\n\n**Features:**\n\n* Image size: 28x28 pixels (784 features)\n* Grayscale values: 0-255 \n* Labels: Digits 0-9 (10 classes)\n\n**Data Characteristics:**\n\n* Each pixel represents a feature\n* Input shape: (n_samples, 784)\n* Target: Single integer 0-9\n* Balanced dataset with ~3,000+ examples per digit\n\n**Data Processing:**\n\n* Flattened 2D images to 1D arrays\n* Normalized pixel values for consistent scaling\n* One-hot encoded target labels for classification\n","metadata":{}},{"cell_type":"markdown","source":"For the purpose of analysis and understanding of our data we'll load them as CSV/ Dataframes. We'll perform various data cleaning and visualizations to get a good grasp of our data.","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_data = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\noutput_data = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:02.674809Z","iopub.execute_input":"2025-02-07T19:40:02.675273Z","iopub.status.idle":"2025-02-07T19:40:07.593114Z","shell.execute_reply.started":"2025-02-07T19:40:02.675236Z","shell.execute_reply":"2025-02-07T19:40:07.592110Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(train_data.shape, test_data.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.594164Z","iopub.execute_input":"2025-02-07T19:40:07.594539Z","iopub.status.idle":"2025-02-07T19:40:07.600347Z","shell.execute_reply.started":"2025-02-07T19:40:07.594503Z","shell.execute_reply":"2025-02-07T19:40:07.599205Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Looking at a few rows from the data isn't a bad idea.\ntrain_data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.601455Z","iopub.execute_input":"2025-02-07T19:40:07.601834Z","iopub.status.idle":"2025-02-07T19:40:07.641935Z","shell.execute_reply.started":"2025-02-07T19:40:07.601798Z","shell.execute_reply":"2025-02-07T19:40:07.640608Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_data.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.644830Z","iopub.execute_input":"2025-02-07T19:40:07.645168Z","iopub.status.idle":"2025-02-07T19:40:07.652621Z","shell.execute_reply.started":"2025-02-07T19:40:07.645139Z","shell.execute_reply":"2025-02-07T19:40:07.651528Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_data.isna().sum().any()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.654912Z","iopub.execute_input":"2025-02-07T19:40:07.655282Z","iopub.status.idle":"2025-02-07T19:40:07.710099Z","shell.execute_reply.started":"2025-02-07T19:40:07.655253Z","shell.execute_reply":"2025-02-07T19:40:07.708866Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_data.isna().sum().any()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.711117Z","iopub.execute_input":"2025-02-07T19:40:07.711504Z","iopub.status.idle":"2025-02-07T19:40:07.736541Z","shell.execute_reply.started":"2025-02-07T19:40:07.711472Z","shell.execute_reply":"2025-02-07T19:40:07.735549Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mnist_train_data = train_data.iloc[:, 1:]\nmnist_train_label = train_data.loc[:, 'label']\n\nmnist_train_data = mnist_train_data/255\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.737473Z","iopub.execute_input":"2025-02-07T19:40:07.737839Z","iopub.status.idle":"2025-02-07T19:40:07.828523Z","shell.execute_reply.started":"2025-02-07T19:40:07.737804Z","shell.execute_reply":"2025-02-07T19:40:07.827457Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import seaborn as sns\nplot_df = mnist_train_label.value_counts().to_frame(name='count')\nplot_df.reset_index(inplace=True)\nplot_df.rename(columns={'index': 'handwritten_no'},inplace=True)\n\nsns.barplot(data=plot_df,x='label',y='count',palette='dark:b_r')\nplt.xlabel(\"Handwritten Value\")\nplt.ylabel(\"Count of Value\")\nplt.title(\"Handwritten Digits in our DataFrame\")\nsns.despine()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:07.829807Z","iopub.execute_input":"2025-02-07T19:40:07.830495Z","iopub.status.idle":"2025-02-07T19:40:08.171535Z","shell.execute_reply.started":"2025-02-07T19:40:07.830455Z","shell.execute_reply":"2025-02-07T19:40:08.170406Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> All values from 0 to 1 are in our dataframe so we can proceed.","metadata":{}},{"cell_type":"code","source":"digit = train_data.loc[3, 'pixel0':]\narr = np.array(digit)\n\nimage_arr = np.reshape(arr, (28,28))\ndig_img = plt.imshow(image_arr, cmap=plt.cm.binary)\nplt.colorbar(dig_img)\nplt.title(\"Image Label: {}\".format(train_data.loc[3, 'label']))\nsns.despine()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:08.172720Z","iopub.execute_input":"2025-02-07T19:40:08.173147Z","iopub.status.idle":"2025-02-07T19:40:08.526651Z","shell.execute_reply.started":"2025-02-07T19:40:08.173104Z","shell.execute_reply":"2025-02-07T19:40:08.525644Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_data","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:08.527721Z","iopub.execute_input":"2025-02-07T19:40:08.528142Z","iopub.status.idle":"2025-02-07T19:40:08.550357Z","shell.execute_reply.started":"2025-02-07T19:40:08.528103Z","shell.execute_reply":"2025-02-07T19:40:08.549068Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create a DataFrame for the labels\nlabels_df = pd.DataFrame(train_data, columns=['label'])\n\n# Plot the distribution of digits\nsns.countplot(data=labels_df, x='label', palette='viridis')\nplt.xlabel(\"Handwritten Value\")\nplt.ylabel(\"Count of Value\")\nplt.title(\"Handwritten Digits in our DataFrame\")\nsns.despine()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:08.551405Z","iopub.execute_input":"2025-02-07T19:40:08.551784Z","iopub.status.idle":"2025-02-07T19:40:08.797669Z","shell.execute_reply.started":"2025-02-07T19:40:08.551748Z","shell.execute_reply":"2025-02-07T19:40:08.796440Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Flatten the dataset to combine pixel values\npixel_values = train_data.drop(columns=['label']).to_numpy().flatten()\n#pixel_values = train_data.flatten()\n\n# Plot the histogram of pixel intensities\nplt.hist(pixel_values, bins=50, color='skyblue', edgecolor='black')\nplt.title(\"Distribution of Pixel Intensity Values\")\nplt.xlabel(\"Pixel Intensity\")\nplt.ylabel(\"Frequency\")\nsns.despine()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:08.798796Z","iopub.execute_input":"2025-02-07T19:40:08.799218Z","iopub.status.idle":"2025-02-07T19:40:09.916417Z","shell.execute_reply.started":"2025-02-07T19:40:08.799178Z","shell.execute_reply":"2025-02-07T19:40:09.915387Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## PCA","metadata":{}},{"cell_type":"markdown","source":"### Principle Component Analysis\n\nPCA is a linear dimensionality reduction technique. PCA allows us to reduce the data in this case the images in to 2 or 3 dimensions.\n\nThis will allow us to see how differenct digit cases form a cluster. Also, we might be able to see classes overlap, digits that might have similar features such as (\"3\" or \"8\") will likely overlap.\n\nPCA will also help us capture significant information about our data. It might also help us point out points that might be outliers meaning that they were poorly wriiten.\n\n> All of these are assumptions that the PCA will help us distinguish and find answers to.","metadata":{}},{"cell_type":"code","source":"#Before feeding our data to PCA we'll have to standardize our data\n# StandardScaler will convert our dataframe to a mean of 0 and standard deviation of 1\nfrom sklearn.preprocessing import StandardScaler\nstandard_scaler = StandardScaler()\nstandardized_df = standard_scaler.fit_transform(mnist_train_data)\n\nfrom sklearn import decomposition\npca = decomposition.PCA()\npca.n_components = 2\npca_data = pca.fit_transform(standardized_df)\npca_data.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:09.917422Z","iopub.execute_input":"2025-02-07T19:40:09.917757Z","iopub.status.idle":"2025-02-07T19:40:11.779275Z","shell.execute_reply.started":"2025-02-07T19:40:09.917730Z","shell.execute_reply":"2025-02-07T19:40:11.778192Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"standardized_df.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:11.780194Z","iopub.execute_input":"2025-02-07T19:40:11.780579Z","iopub.status.idle":"2025-02-07T19:40:11.786528Z","shell.execute_reply.started":"2025-02-07T19:40:11.780542Z","shell.execute_reply":"2025-02-07T19:40:11.785534Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.DataFrame(pca_data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:11.787544Z","iopub.execute_input":"2025-02-07T19:40:11.787880Z","iopub.status.idle":"2025-02-07T19:40:11.810273Z","shell.execute_reply.started":"2025-02-07T19:40:11.787854Z","shell.execute_reply":"2025-02-07T19:40:11.809304Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"standardized_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:11.811158Z","iopub.execute_input":"2025-02-07T19:40:11.811469Z","iopub.status.idle":"2025-02-07T19:40:11.831307Z","shell.execute_reply.started":"2025-02-07T19:40:11.811444Z","shell.execute_reply":"2025-02-07T19:40:11.830132Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pca_data = pd.DataFrame(data=pca_data,columns=['PC1', 'PC2'])\nlabel = pd.DataFrame(mnist_train_label)\npca_data['label'] = mnist_train_label\npca_data","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:11.832310Z","iopub.execute_input":"2025-02-07T19:40:11.832611Z","iopub.status.idle":"2025-02-07T19:40:11.859134Z","shell.execute_reply.started":"2025-02-07T19:40:11.832586Z","shell.execute_reply":"2025-02-07T19:40:11.857650Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nlabel = pd.DataFrame(mnist_train_label)\npca_data = pd.DataFrame(data=pca_data,columns=['PC1', 'PC2'])\npca_data['label'] = mnist_train_label","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:11.864176Z","iopub.execute_input":"2025-02-07T19:40:11.864520Z","iopub.status.idle":"2025-02-07T19:40:11.881159Z","shell.execute_reply.started":"2025-02-07T19:40:11.864490Z","shell.execute_reply":"2025-02-07T19:40:11.879964Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nsns.FacetGrid(pca_data,hue='label',height=6, aspect=8/6).map(plt.scatter, 'PC1','PC2').add_legend()\nsns.despine()\nplt.xlabel(\"Principal Component 2\")\nplt.ylabel(\"Principal Component 1\")\nplt.title(\"Handwritten Digits With 2 PCA\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:11.883594Z","iopub.execute_input":"2025-02-07T19:40:11.884016Z","iopub.status.idle":"2025-02-07T19:40:12.974288Z","shell.execute_reply.started":"2025-02-07T19:40:11.883964Z","shell.execute_reply":"2025-02-07T19:40:12.972972Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> The graph clearly demonstrates that clusters that the numbers form interms of their similarity. Also, rememember our assumption that a few features in the digits the likes of 3 or 8 would have commonality so they would overlap. It's clear to see that they do.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.scatterplot(data=pca_data, x='PC1', y='PC2', hue='label')\nsns.despine()\nplt.xlabel(\"Principal Component 2\")\nplt.ylabel(\"Principal Component 1\")\nplt.title(\"Handwritten Digits With 2 PCA\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:12.975640Z","iopub.execute_input":"2025-02-07T19:40:12.975950Z","iopub.status.idle":"2025-02-07T19:40:15.226381Z","shell.execute_reply.started":"2025-02-07T19:40:12.975921Z","shell.execute_reply":"2025-02-07T19:40:15.225035Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create Model","metadata":{"execution":{"iopub.status.busy":"2025-01-08T17:41:26.190007Z","iopub.execute_input":"2025-01-08T17:41:26.190378Z","iopub.status.idle":"2025-01-08T17:41:26.202553Z","shell.execute_reply.started":"2025-01-08T17:41:26.190308Z","shell.execute_reply":"2025-01-08T17:41:26.201347Z"}}},{"cell_type":"code","source":"train_ = '/kaggle/input/digit-recognizer/train.csv'\ntest_ = '/kaggle/input/digit-recognizer/test.csv'\noutput_ = '/kaggle/input/digit-recognizer/sample_submission.csv'\n\nraw_data = np.loadtxt(train_, skiprows=1, dtype='int', delimiter=',')\nx_train, x_val, y_train, y_val = train_test_split(raw_data[:,1:], raw_data[:,0], test_size=0.1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:15.227342Z","iopub.execute_input":"2025-02-07T19:40:15.227619Z","iopub.status.idle":"2025-02-07T19:40:16.743033Z","shell.execute_reply.started":"2025-02-07T19:40:15.227594Z","shell.execute_reply":"2025-02-07T19:40:16.742076Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.DataFrame(raw_data).head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:16.744324Z","iopub.execute_input":"2025-02-07T19:40:16.744717Z","iopub.status.idle":"2025-02-07T19:40:16.759496Z","shell.execute_reply.started":"2025-02-07T19:40:16.744675Z","shell.execute_reply":"2025-02-07T19:40:16.758443Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, ax = plt.subplots(2, 1, figsize = (10, 8))\nax[0].plot(x_train[0])\nax[0].set_title('784 x 1 Data')\nax[1].imshow(x_train[0].reshape(28,28),cmap='gray')\nax[1].set_title('28x28 Data',color='navy')\nsns.despine()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:16.760477Z","iopub.execute_input":"2025-02-07T19:40:16.760748Z","iopub.status.idle":"2025-02-07T19:40:17.229703Z","shell.execute_reply.started":"2025-02-07T19:40:16.760722Z","shell.execute_reply":"2025-02-07T19:40:17.228638Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> **We now reshape all data this way. Keras wants an extra dimension in the end, for channels. If this had been RGB images, there would have been 3 channels, but as MNIST is gray scale it only uses one.**","metadata":{}},{"cell_type":"markdown","source":"### Reshape Data","metadata":{}},{"cell_type":"markdown","source":"We have to now store our data/ images in 1D arrays, will all the pixels lined up in a row.\n\nIn this case:\n\n* 37800: Number of images.\n* 28, 28: Height and width of each image.\n* 1: Number of channels (grayscale).","metadata":{"execution":{"iopub.status.busy":"2025-01-08T18:43:07.878466Z","iopub.execute_input":"2025-01-08T18:43:07.878867Z","iopub.status.idle":"2025-01-08T18:43:07.887692Z","shell.execute_reply.started":"2025-01-08T18:43:07.878834Z","shell.execute_reply":"2025-01-08T18:43:07.886067Z"}}},{"cell_type":"code","source":"print(x_train.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.230811Z","iopub.execute_input":"2025-02-07T19:40:17.231196Z","iopub.status.idle":"2025-02-07T19:40:17.236339Z","shell.execute_reply.started":"2025-02-07T19:40:17.231156Z","shell.execute_reply":"2025-02-07T19:40:17.235231Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"x_train","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.237499Z","iopub.execute_input":"2025-02-07T19:40:17.237892Z","iopub.status.idle":"2025-02-07T19:40:17.255606Z","shell.execute_reply.started":"2025-02-07T19:40:17.237841Z","shell.execute_reply":"2025-02-07T19:40:17.254538Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"x_train = x_train.reshape(-1,28,28,1)\nx_val = x_val.reshape(-1,28,28,1)\nprint(x_train.shape)\nprint(x_val.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.256522Z","iopub.execute_input":"2025-02-07T19:40:17.256860Z","iopub.status.idle":"2025-02-07T19:40:17.272403Z","shell.execute_reply.started":"2025-02-07T19:40:17.256822Z","shell.execute_reply":"2025-02-07T19:40:17.271326Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Standardize our Data","metadata":{}},{"cell_type":"code","source":"x_train = x_train.astype('float32')/255\nx_val = x_val.astype('float32')/255","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.273513Z","iopub.execute_input":"2025-02-07T19:40:17.273862Z","iopub.status.idle":"2025-02-07T19:40:17.365590Z","shell.execute_reply.started":"2025-02-07T19:40:17.273827Z","shell.execute_reply":"2025-02-07T19:40:17.364487Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### One Hot Encode","metadata":{}},{"cell_type":"markdown","source":"The labels were given as integers between 0 and 9. We need to convert these to one-hot encoding, i.e. a 10x1 array with one 1 and nine 0:s, with the position of the 1 showing us the value. See the example, with the position of the 1 showing the correct value for the digit in the graph above.","metadata":{}},{"cell_type":"code","source":"y_train = to_categorical(y_train)\ny_val = to_categorical(y_val)\n#example:\nprint(y_train[0])\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.366534Z","iopub.execute_input":"2025-02-07T19:40:17.366808Z","iopub.status.idle":"2025-02-07T19:40:17.374804Z","shell.execute_reply.started":"2025-02-07T19:40:17.366785Z","shell.execute_reply":"2025-02-07T19:40:17.373682Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Train Model","metadata":{}},{"cell_type":"markdown","source":"For this project we will be using the **Sequential API**, where we've to add on one layer at a time, starting from the input\n\nThe most important part are teh **Convolutional Layers (Conv2D)**. Here they have 16-32 filters that use nine weights each to transform a pixel to a weighted average of itself and its eight neighbors. As the same nine weights are used over the whole image, the net will pick up features that are useful everywhere. As it is only nine weights, we can stack many convolutional layers on top of each other without running out of memory/time.","metadata":{}},{"cell_type":"code","source":"model = Sequential()\ninput_shape = (28, 28, 1)\n\n# Layer One\nmodel.add(Conv2D(filters=16, kernel_size=(3, 3), activation='relu',input_shape=input_shape))\nmodel.add(BatchNormalization())\n\n#Layers\nmodel.add(Conv2D(filters=16, kernel_size=(3, 3), activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(strides=(2,2)))\nmodel.add(Dropout(0.25))\n\n# Layers\nmodel.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\nmodel.add(BatchNormalization())\n\n# Layers\nmodel.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(MaxPool2D(strides=(2,2)))\nmodel.add(Dropout(0.25))\n\n# Utility Layer\nmodel.add(Flatten())\n\n# Dense Layer\nmodel.add(Dense(512,activation='relu'))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(1024,activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10, activation='softmax'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.375962Z","iopub.execute_input":"2025-02-07T19:40:17.376393Z","iopub.status.idle":"2025-02-07T19:40:17.683099Z","shell.execute_reply.started":"2025-02-07T19:40:17.376351Z","shell.execute_reply":"2025-02-07T19:40:17.682053Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.684043Z","iopub.execute_input":"2025-02-07T19:40:17.684310Z","iopub.status.idle":"2025-02-07T19:40:17.716776Z","shell.execute_reply.started":"2025-02-07T19:40:17.684288Z","shell.execute_reply":"2025-02-07T19:40:17.715764Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Important method to improve generalization is augmentation. Augmentaion means generating more training data by randomly perturbing the images. If done in the right way, it can force the net to only learn translation-invariant features. If you train this model over hundreds of epochs, augmentation will definitely improve your performance. Here in the Kernel, we will only look at each image 4-5 times, so the difference is smaller. We use a Keras function for augmentation.","metadata":{}},{"cell_type":"code","source":"# Data augmentaion class\n# for our training, applies real time image augmentaion for the model to create variations in the data for maximum learning\ndatagen = ImageDataGenerator(zoom_range = 0.1,height_shift_range = 0.1,\n                             width_shift_range = 0.1,rotation_range = 10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.717751Z","iopub.execute_input":"2025-02-07T19:40:17.718079Z","iopub.status.idle":"2025-02-07T19:40:17.722295Z","shell.execute_reply.started":"2025-02-07T19:40:17.718052Z","shell.execute_reply":"2025-02-07T19:40:17.721348Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nmodel.compile(loss='categorical_crossentropy', optimizer = Adam(learning_rate=1e-4), metrics=[\"accuracy\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.723214Z","iopub.execute_input":"2025-02-07T19:40:17.723498Z","iopub.status.idle":"2025-02-07T19:40:17.752614Z","shell.execute_reply.started":"2025-02-07T19:40:17.723468Z","shell.execute_reply":"2025-02-07T19:40:17.751663Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#learning rate adjusting object for our training\nannealer = LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.753707Z","iopub.execute_input":"2025-02-07T19:40:17.754088Z","iopub.status.idle":"2025-02-07T19:40:17.758573Z","shell.execute_reply.started":"2025-02-07T19:40:17.754044Z","shell.execute_reply":"2025-02-07T19:40:17.757334Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# create a history object storing the training progress, including the loss and accuracy\nhistory = model.fit(datagen.flow(x_train, y_train, batch_size=10), #data augmentation\n                             steps_per_epoch=500, #training samples per epoch 500 * 15 = 7500 training samples\n                             epochs=40, #No of iterations of training\n                             verbose=2, #output of the training\n                             validation_data=(x_val,y_val), #model evaluation at each epoch\n                             callbacks=[annealer]) #learning rate adjuster","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:40:17.759543Z","iopub.execute_input":"2025-02-07T19:40:17.759840Z","iopub.status.idle":"2025-02-07T19:46:25.534075Z","shell.execute_reply.started":"2025-02-07T19:40:17.759815Z","shell.execute_reply":"2025-02-07T19:46:25.533049Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"> A few parameters that could be tweaked (number of layers, number of filters, Dropout parameters, learning rate, augmentation settings). This is often done with trial and error, and there is no easy shortcut.\n\n> Achieving convergence shouldn't be an issue unless you use an excessively large learning rate. However, it’s common to create a model that overfits, performing exceptionally well on the training set but poorly on the validation data. If you encounter this, consider increasing the Dropout parameters, enhancing data augmentation, or stopping training sooner. If your goal is to boost accuracy, you might want to add two additional layers or increase the number of filters.","metadata":{}},{"cell_type":"markdown","source":"### Model 2","metadata":{}},{"cell_type":"code","source":"model_2 = Sequential()\ninput_shape = (28, 28, 1)\n\n# Layer One\nmodel_2.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu',input_shape=input_shape))\nmodel_2.add(BatchNormalization())\n\n#Layers\nmodel_2.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))\nmodel_2.add(BatchNormalization())\nmodel_2.add(MaxPool2D(strides=(2,2)))\nmodel_2.add(Dropout(0.25))\n\n# Layers\nmodel_2.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))\nmodel_2.add(BatchNormalization())\n\n# Layers\nmodel_2.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))\nmodel_2.add(BatchNormalization())\nmodel_2.add(MaxPool2D(strides=(2,2)))\nmodel_2.add(Dropout(0.25))\n\n# Utility Layer\nmodel_2.add(Flatten())\n\n# Dense Layer\nmodel_2.add(Dense(512,activation='relu'))\nmodel_2.add(Dropout(0.25))\nmodel_2.add(Dense(1024,activation='relu'))\nmodel_2.add(Dropout(0.5))\nmodel_2.add(Dense(10, activation='softmax'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:46:25.535188Z","iopub.execute_input":"2025-02-07T19:46:25.535458Z","iopub.status.idle":"2025-02-07T19:46:25.777428Z","shell.execute_reply.started":"2025-02-07T19:46:25.535436Z","shell.execute_reply":"2025-02-07T19:46:25.776405Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model_2.compile(loss='categorical_crossentropy', optimizer = Adam(learning_rate=1e-4), metrics=[\"accuracy\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:46:25.778393Z","iopub.execute_input":"2025-02-07T19:46:25.778664Z","iopub.status.idle":"2025-02-07T19:46:25.787847Z","shell.execute_reply.started":"2025-02-07T19:46:25.778642Z","shell.execute_reply":"2025-02-07T19:46:25.787007Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# create a history object storing the training progress, including the loss and accuracy\nhistory_2 = model_2.fit(datagen.flow(x_train, y_train, batch_size=10), #data augmentation\n                             steps_per_epoch=500, #training samples per epoch 500 * 15 = 7500 training samples\n                             epochs=40, #No iterations of training\n                             verbose=2, #output of the training\n                             validation_data=(x_val,y_val), #model evaluation at each epoch\n                             callbacks=[annealer]) #learning rate adjuster","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:46:25.788837Z","iopub.execute_input":"2025-02-07T19:46:25.789139Z","iopub.status.idle":"2025-02-07T19:56:00.209430Z","shell.execute_reply.started":"2025-02-07T19:46:25.789114Z","shell.execute_reply":"2025-02-07T19:56:00.208287Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Evaluate","metadata":{}},{"cell_type":"code","source":"# Model 1\nfinal_loss, final_acc = model.evaluate(x_val, y_val, verbose=0)\nprint(\"Model 1 - Final Loss: {}, and Final Accuracy: {}\".format(final_loss, final_acc))\n\n# Model 2\nfinal_loss, final_acc = model_2.evaluate(x_val, y_val, verbose=0)\nprint(\"Model 2 - Final Loss: {}, and Final Accuracy: {}\".format(final_loss, final_acc))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:00.210738Z","iopub.execute_input":"2025-02-07T19:56:00.211131Z","iopub.status.idle":"2025-02-07T19:56:03.014716Z","shell.execute_reply.started":"2025-02-07T19:56:00.211101Z","shell.execute_reply":"2025-02-07T19:56:03.013743Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insights:**\n\n> Model 2 outperformed Model 1, achieving 99.50% accuracy vs. 99.31% with a lower validation loss (0.0172 vs. 0.0215).\nData augmentation improved generalization, while learning rate annealing refined performance, stabilizing validation accuracy above 99.43%.\nPerformance gains were minimal after epoch 30, indicating early stopping could save resources.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplt.subplot(1,2,1)\nplt.plot(history.history['loss'], color='c',label='Training Loss')\nplt.plot(history.history['val_loss'], color='r', label='Validation_loss')\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"\")\nplt.title(\"Loss and Validtion Loss for Model 1\")\nplt.legend(loc='upper right')\nsns.despine()\n\n\nplt.subplot(1,2,2)\nplt.plot(history.history['accuracy'], color='c',label='Training Loss')\nplt.plot(history.history['val_accuracy'], color='r', label='Validation_loss')\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"\")\nplt.title(\"Accuracy and Validtion Accuracy for Model 1\")\nsns.despine()\nplt.legend(loc='upper right')\n\nplt.show()\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:03.015647Z","iopub.execute_input":"2025-02-07T19:56:03.015969Z","iopub.status.idle":"2025-02-07T19:56:03.499120Z","shell.execute_reply.started":"2025-02-07T19:56:03.015942Z","shell.execute_reply":"2025-02-07T19:56:03.498092Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplt.subplot(1,2,1)\nplt.plot(history_2.history['loss'], color='y',label='Training Loss')\nplt.plot(history_2.history['val_loss'], color='r',label='Validation Loss')\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"\")\nplt.title(\"Loss and Validtion Loss for Model 2\")\nsns.despine()\n\n\nplt.subplot(1,2,2)\nplt.plot(history_2.history['accuracy'], color='y',label='Training Loss')\nplt.plot(history_2.history['val_accuracy'], color='r',label='Validation Loss')\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"\")\nplt.title(\"Accuracy and Validtion Accuracy for Model 2\")\nsns.despine()\n\nplt.show()\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:03.500208Z","iopub.execute_input":"2025-02-07T19:56:03.500601Z","iopub.status.idle":"2025-02-07T19:56:03.948417Z","shell.execute_reply.started":"2025-02-07T19:56:03.500564Z","shell.execute_reply":"2025-02-07T19:56:03.947370Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\nplt.figure(figsize=(20,10))\ny_hat = model.predict(x_val)\ny_pred = np.argmax(y_hat, axis=1)\ny_true = np.argmax(y_val, axis=1)\ncm = confusion_matrix(y_true, y_pred)\n# mask=np.triu(np.ones(cm.shape),k=1)\n# np.fill_diagonal(cm,0)\n# sns.heatmap(data=cm, annot=True, cmap='viridis',mask=mask)\ndisp = ConfusionMatrixDisplay(confusion_matrix=cm)\ndisp.plot()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:03.949371Z","iopub.execute_input":"2025-02-07T19:56:03.949709Z","iopub.status.idle":"2025-02-07T19:56:05.978506Z","shell.execute_reply.started":"2025-02-07T19:56:03.949682Z","shell.execute_reply":"2025-02-07T19:56:05.977396Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Possible Flaws of Model\n\n> The model relies heavily on data augmentation, which might not fully simulate real-world variations.\nHigh accuracy may indicate potential overfitting, especially if training and validation data are not diverse enough.\nLimited exploration of alternative architectures or regularization techniques like dropout.\n","metadata":{}},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"mnist_testset = np.loadtxt(test_, skiprows=1, dtype='int', delimiter=',')\nx_test = mnist_testset.astype(\"float32\")\nx_test = x_test.reshape(-1, 28, 28, 1)/255.","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:05.979437Z","iopub.execute_input":"2025-02-07T19:56:05.979700Z","iopub.status.idle":"2025-02-07T19:56:06.994178Z","shell.execute_reply.started":"2025-02-07T19:56:05.979678Z","shell.execute_reply":"2025-02-07T19:56:06.993219Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Model 1 Predictions\ny_hat_1 = model.predict(x_test, batch_size=64)\ny_pred_1 = np.argmax(y_hat_1, axis=1)\n\n\n# Model 2 Predictions\ny_hat_2 = model_2.predict(x_test, batch_size=64)\ny_pred_2 = np.argmax(y_hat_2, axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:06.995014Z","iopub.execute_input":"2025-02-07T19:56:06.995313Z","iopub.status.idle":"2025-02-07T19:56:22.990942Z","shell.execute_reply.started":"2025-02-07T19:56:06.995287Z","shell.execute_reply":"2025-02-07T19:56:22.989827Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mnist_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:22.992203Z","iopub.execute_input":"2025-02-07T19:56:22.992583Z","iopub.status.idle":"2025-02-07T19:56:24.354459Z","shell.execute_reply.started":"2025-02-07T19:56:22.992543Z","shell.execute_reply":"2025-02-07T19:56:24.353329Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission_1 =  pd.DataFrame({\n        \"ImageId\": mnist_test.index+1,\n        \"Label\": y_pred_1\n    })\n\nsubmission_1.to_csv('my_submission_1.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:24.355419Z","iopub.execute_input":"2025-02-07T19:56:24.355781Z","iopub.status.idle":"2025-02-07T19:56:24.385754Z","shell.execute_reply.started":"2025-02-07T19:56:24.355744Z","shell.execute_reply":"2025-02-07T19:56:24.384593Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission_2 =  pd.DataFrame({\n        \"ImageId\": mnist_test.index+1,\n        \"Label\": y_pred_2\n    })\n\nsubmission_2.to_csv('my_submission_2.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T19:56:24.387092Z","iopub.execute_input":"2025-02-07T19:56:24.387469Z","iopub.status.idle":"2025-02-07T19:56:24.414726Z","shell.execute_reply.started":"2025-02-07T19:56:24.387441Z","shell.execute_reply":"2025-02-07T19:56:24.413775Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Save the entire model to a HDF5 file\nmodel_2.save('best_model.h5')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T20:17:43.215324Z","iopub.execute_input":"2025-02-07T20:17:43.215833Z","iopub.status.idle":"2025-02-07T20:17:43.337354Z","shell.execute_reply.started":"2025-02-07T20:17:43.215799Z","shell.execute_reply":"2025-02-07T20:17:43.336320Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Save only the model weights to a HDF5 file\nmodel_2.save_weights('best_model_weihts.weights.h5')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T20:21:29.704160Z","iopub.execute_input":"2025-02-07T20:21:29.704561Z","iopub.status.idle":"2025-02-07T20:21:29.785630Z","shell.execute_reply.started":"2025-02-07T20:21:29.704530Z","shell.execute_reply":"2025-02-07T20:21:29.784571Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pickle\n\n# Save the model using pickle\nwith open('my_cnn_model.pkl', 'wb') as f:\n    pickle.dump(model_2, f)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport subprocess\nfrom IPython.display import FileLink, display\n\ndef download_file(path, download_file_name):\n    os.chdir('/kaggle/working/')\n    zip_name = f\"/kaggle/working/{download_file_name}.zip\"\n    command = f\"zip {zip_name} {path} -r\"\n    result = subprocess.run(command, shell=True, capture_output=True, text=True)\n    if result.returncode != 0:\n        print(\"Unable to run zip command!\")\n        print(result.stderr)\n        return\n    display(FileLink(f'{download_file_name}.zip'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T20:22:50.529862Z","iopub.execute_input":"2025-02-07T20:22:50.530261Z","iopub.status.idle":"2025-02-07T20:22:50.536197Z","shell.execute_reply.started":"2025-02-07T20:22:50.530228Z","shell.execute_reply":"2025-02-07T20:22:50.534969Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"download_file('/kaggle/working/best_model_weihts.weights.h5', 'best_model_weights')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T20:23:37.485844Z","iopub.execute_input":"2025-02-07T20:23:37.486294Z","iopub.status.idle":"2025-02-07T20:23:38.227162Z","shell.execute_reply.started":"2025-02-07T20:23:37.486252Z","shell.execute_reply":"2025-02-07T20:23:38.226104Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"download_file('/kaggle/working/best_model.h5', 'best_model')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-07T20:24:17.152718Z","iopub.execute_input":"2025-02-07T20:24:17.153083Z","iopub.status.idle":"2025-02-07T20:24:17.896408Z","shell.execute_reply.started":"2025-02-07T20:24:17.153054Z","shell.execute_reply":"2025-02-07T20:24:17.895348Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}