{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Step 1: Importing the necessary libraries","metadata":{"noteable":{"cell_type":"markdown"},"id":"afd5177d-a137-4399-913d-420e25fdf74e"}},{"cell_type":"code","source":"import tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras import backend as K","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:26:17.138912+00:00","start_time":"2023-06-18T15:26:14.487770+00:00"},"id":"2154d8a7-8636-4257-bedf-696572f45724","execution":{"iopub.status.busy":"2023-06-24T19:01:12.374314Z","iopub.execute_input":"2023-06-24T19:01:12.374735Z","iopub.status.idle":"2023-06-24T19:01:20.522755Z","shell.execute_reply.started":"2023-06-24T19:01:12.374704Z","shell.execute_reply":"2023-06-24T19:01:20.521804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Loading the Data\n\nThe first step in our data preprocessing is to load the data. We have two datasets: 'train.csv' and 'test.csv'. The 'train.csv' file contains the training data that we will use to train our model, and the 'test.csv' file contains the test data that we will use to evaluate our model.\n\nEach row in these files corresponds to an image: the first column in 'train.csv' is the label (the digit) and the remaining columns are the pixel values. In 'test.csv', each row corresponds to an image, and the columns are the pixel values.","metadata":{"noteable":{"cell_type":"markdown"},"id":"cf8e8c3c-0254-4bb1-8605-47d07901d6a0"}},{"cell_type":"code","source":"# Load the data\ntrain = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\n# Display the first few rows of the train dataset\ntrain.head()","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:27:05.337681+00:00","start_time":"2023-06-18T15:27:00.112491+00:00"},"datalink":{"367c3467-3020-472f-b937-b5249d615588":{"applied_filters":[],"dataframe_info":{"default_index_used":true,"orig_num_cols":785,"orig_num_rows":5,"orig_size_bytes":31440,"truncated_num_cols":100,"truncated_num_rows":5,"truncated_size_bytes":4040,"truncated_string_columns":[]},"display_id":"367c3467-3020-472f-b937-b5249d615588","dx_settings":{"ALLOW_NOTEABLE_ATTRS":true,"COLUMN_SAMPLING_METHOD":"outer","DB_LOCATION":":memory:","DEV_MODE":false,"DISPLAY_MAX_COLUMNS":100,"DISPLAY_MAX_ROWS":50000,"DISPLAY_MODE":"simple","ENABLE_ASSIGNMENT":true,"ENABLE_DATALINK":true,"FLATTEN_COLUMN_VALUES":true,"FLATTEN_INDEX_VALUES":false,"GENERATE_DEX_METADATA":false,"HTML_TABLE_SCHEMA":false,"LOG_LEVEL":30,"MAX_RENDER_SIZE_BYTES":104857600,"MAX_STRING_LENGTH":250,"NUM_PAST_SAMPLES_TRACKED":3,"RANDOM_STATE":12648430,"RESET_INDEX_VALUES":false,"ROW_SAMPLING_METHOD":"random","SAMPLING_FACTOR":0.1,"SAMPLING_METHOD":"random","STRINGIFY_COLUMN_VALUES":true,"STRINGIFY_INDEX_VALUES":false},"sample_history":[],"sampling_time":"2023-06-18T15:27:05.061148","user_variable_name":null,"variable_name":"unk_dataframe_1e925b9ba2d84aa0bc12f7e6022c4ee8"}},"id":"231a771f-3e3a-4aec-83aa-a5e8805700f3","outputId":"0bc5da00-8a59-42a7-cdf9-116022b7152f","execution":{"iopub.status.busy":"2023-06-24T19:01:20.524686Z","iopub.execute_input":"2023-06-24T19:01:20.525494Z","iopub.status.idle":"2023-06-24T19:01:26.567814Z","shell.execute_reply.started":"2023-06-24T19:01:20.525459Z","shell.execute_reply":"2023-06-24T19:01:26.566933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The data has been successfully loaded. As we can see, the 'label' column contains the target labels, and the 'pixel' columns contain the pixel values of the images. Each image is 28x28 pixels, so we have 784 pixel columns.\n\nNext, we will separate the features (pixel values) from the labels. We will also normalize the pixel values to be in the range [0, 1] by dividing by 255 (since the original pixel values are in the range [0, 255]). Finally, we will reshape the data to be in the format that our neural network expects.","metadata":{"noteable":{"cell_type":"markdown"},"id":"450f7144-25e2-4068-b81f-c997ab3b0350"}},{"cell_type":"code","source":"# Separate the features and the labels\nX_train = train.drop('label', axis=1)\ny_train = train['label']\n\n# Normalize the data\nX_train = X_train / 255.0\ntest = test / 255.0\n\n# Reshape the data\nX_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:27:44.300213+00:00","start_time":"2023-06-18T15:27:43.934295+00:00"},"id":"7370181c-edb1-4b1f-bed5-a52000a78830","execution":{"iopub.status.busy":"2023-06-24T19:01:26.569248Z","iopub.execute_input":"2023-06-24T19:01:26.569681Z","iopub.status.idle":"2023-06-24T19:01:26.819283Z","shell.execute_reply.started":"2023-06-24T19:01:26.569646Z","shell.execute_reply":"2023-06-24T19:01:26.818239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the first 5 images in the training set\nfor i in range(5):\n    plt.figure(figsize=(2, 2))\n    plt.imshow(X_train[i].reshape(28, 28), cmap='gray')\n    plt.title(f'Label: {y_train[i]}')\n    plt.show()","metadata":{"id":"6cuZP0Mxz_-M","outputId":"fa80fa6d-fabd-4058-984a-6093ccaa9ccb","execution":{"iopub.status.busy":"2023-06-24T19:01:26.823502Z","iopub.execute_input":"2023-06-24T19:01:26.825691Z","iopub.status.idle":"2023-06-24T19:01:27.837499Z","shell.execute_reply.started":"2023-06-24T19:01:26.825663Z","shell.execute_reply":"2023-06-24T19:01:27.836464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The features and labels have been separated, the data has been normalized, and the data has been reshaped to the format that our neural network expects.\n\nThe next step is to split our training data into a training set and a validation set. We will use the training set to train our model, and the validation set to tune the hyperparameters and evaluate the model's performance during training.","metadata":{"noteable":{"cell_type":"markdown"},"id":"160e32ee-6b16-4561-ac2b-18086db5a88e"}},{"cell_type":"code","source":"# Split the data into a training set and a validation set\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:28:19.384515+00:00","start_time":"2023-06-18T15:28:18.738913+00:00"},"id":"7507e65d-317a-420d-9ac4-14e6b1817547","execution":{"iopub.status.busy":"2023-06-24T19:01:27.838734Z","iopub.execute_input":"2023-06-24T19:01:27.839079Z","iopub.status.idle":"2023-06-24T19:01:28.215587Z","shell.execute_reply.started":"2023-06-24T19:01:27.839046Z","shell.execute_reply":"2023-06-24T19:01:28.214605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have successfully split our training data into a training set and a validation set. We used 80% of the data for training and 20% for validation.\n\nNow, we are ready to train our model. We will use the fit method of our model to train it. We will also specify the validation data to use, so that we can monitor the validation loss and accuracy during training.","metadata":{"noteable":{"cell_type":"markdown"},"id":"92e01211-c86e-40bf-83f5-96637d2c4c4f"}},{"cell_type":"markdown","source":"# Step 3: Building the Model\n\nWe will build a Convolutional Neural Network (CNN) for this task. CNNs are a type of neural network that are especially good at dealing with image data. Our network will have the following layers:\n\n1. Convolutional layer with 32 filters, a kernel size of 3x3, 'relu' activation, and 'same' padding.\n2. Convolutional layer with 32 filters, a kernel size of 3x3, 'relu' activation, and 'same' padding.\n3. Max pooling layer with pool size of 2x2.\n4. Dropout layer with a rate of 0.25.\n5. Flatten layer to flatten the output of the previous layers.\n6. Dense (fully connected) layer with 128 units and 'relu' activation.\n7. Dropout layer with a rate of 0.5.\n8. Output layer with 10 units (one for each digit) and 'softmax' activation.\n\nWe will use the 'adam' optimizer, the 'categorical_crossentropy' loss function (since this is a multi-class classification problem), and we will monitor the 'accuracy' metric.","metadata":{"noteable":{"cell_type":"markdown"},"id":"8bd92bcb-6bd2-4a16-bc65-52cd4c6cd1cf"}},{"cell_type":"markdown","source":"The model has been successfully created. As we can see from the summary, our model has a total of 813,802 parameters that it can learn from.\n\nNow, we are ready to train our model. We will use the fit method of our model to train it. We will also specify the validation data to use, so that we can monitor the validation loss and accuracy during training.","metadata":{"noteable":{"cell_type":"markdown"},"id":"ac50b1d5-b832-43ce-ba9a-824a2672975d"}},{"cell_type":"code","source":"# Define the model again\nmodel = Sequential()\nmodel.add(Conv2D(filters=64, kernel_size=3, padding='same', activation='relu', input_shape=(28,28,1)))\nmodel.add(Conv2D(filters=64, kernel_size=3, padding='same', activation='relu'))\nmodel.add(Conv2D(filters=128, kernel_size=3, padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=2))\nmodel.add(Conv2D(filters=128, kernel_size=3, padding='same', activation='relu'))\nmodel.add(Conv2D(filters=192, kernel_size=3, padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=2))\nmodel.add(Conv2D(filters=192, kernel_size=5, padding='same', activation='relu'))\nmodel.add(MaxPooling2D(pool_size=2, padding='same'))\nmodel.add(Flatten())\nmodel.add(Dense(256, activation='relu'))\nmodel.add(Dense(10, activation='softmax'))\n# Compile the model\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n\n# Display the model's architecture\nmodel.summary()","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:32:38.048312+00:00","start_time":"2023-06-18T15:32:37.734724+00:00"},"id":"afa0c99f-4af9-4d69-b715-c3f2cd7b04c5","outputId":"e0c39af2-6fd4-441d-c3b5-8b9aa0a9f33c","execution":{"iopub.status.busy":"2023-06-24T19:01:28.217177Z","iopub.execute_input":"2023-06-24T19:01:28.217526Z","iopub.status.idle":"2023-06-24T19:01:31.053223Z","shell.execute_reply.started":"2023-06-24T19:01:28.217493Z","shell.execute_reply":"2023-06-24T19:01:31.052470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert labels to one-hot encoded format\ny_train = to_categorical(y_train, num_classes=10)\ny_val = to_categorical(y_val, num_classes=10)","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:31:34.803114+00:00","start_time":"2023-06-18T15:31:34.644200+00:00"},"id":"14def6c4-7a4a-4e54-af13-a4146fd894d7","execution":{"iopub.status.busy":"2023-06-24T19:01:31.054272Z","iopub.execute_input":"2023-06-24T19:01:31.054641Z","iopub.status.idle":"2023-06-24T19:01:31.068374Z","shell.execute_reply.started":"2023-06-24T19:01:31.054609Z","shell.execute_reply":"2023-06-24T19:01:31.067613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Clear the TensorFlow session\nK.clear_session()","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T15:32:06.496004+00:00","start_time":"2023-06-18T15:32:06.334577+00:00"},"id":"7f8f3b68-bde2-4b81-b423-ed76dc91b7ad","execution":{"iopub.status.busy":"2023-06-24T19:01:31.069404Z","iopub.execute_input":"2023-06-24T19:01:31.069740Z","iopub.status.idle":"2023-06-24T19:01:31.080220Z","shell.execute_reply.started":"2023-06-24T19:01:31.069711Z","shell.execute_reply":"2023-06-24T19:01:31.079345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data augmentation\ndatagen = tf.keras.preprocessing.image.ImageDataGenerator(\n    featurewise_std_normalization=False,\n    samplewise_std_normalization=False,\n    zca_whitening=False,\n    rotation_range=5,\n    zoom_range=0.1,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=False,\n    vertical_flip=False\n)\n\ndatagen.fit(X_train)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T19:01:31.082406Z","iopub.execute_input":"2023-06-24T19:01:31.083260Z","iopub.status.idle":"2023-06-24T19:01:31.172250Z","shell.execute_reply.started":"2023-06-24T19:01:31.083228Z","shell.execute_reply":"2023-06-24T19:01:31.171281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.callbacks import ReduceLROnPlateau\nreduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.3, verbose=1,\n                              patience=2, min_lr=0.00000001)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T19:01:31.176123Z","iopub.execute_input":"2023-06-24T19:01:31.176554Z","iopub.status.idle":"2023-06-24T19:01:31.181347Z","shell.execute_reply.started":"2023-06-24T19:01:31.176506Z","shell.execute_reply":"2023-06-24T19:01:31.180234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train the model\nhistory = model.fit(X_train, y_train, batch_size=100, epochs=25,\n          validation_data=(X_val, y_val), callbacks=[reduce_lr],\n          verbose=1, shuffle=True)","metadata":{"noteable":{"cell_type":"code"},"ExecuteTime":{"end_time":"2023-06-18T16:02:56.099168+00:00","start_time":"2023-06-18T15:32:52.397062+00:00"},"id":"3970a8df-b3fa-400b-ae4c-0818b995999d","outputId":"6095f90f-209b-4eaa-ed5e-4108a7d4bc28","execution":{"iopub.status.busy":"2023-06-24T19:01:31.182909Z","iopub.execute_input":"2023-06-24T19:01:31.183241Z","iopub.status.idle":"2023-06-24T19:04:54.752831Z","shell.execute_reply.started":"2023-06-24T19:01:31.183211Z","shell.execute_reply":"2023-06-24T19:04:54.751823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# summarize history for accuracy\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T19:14:01.534570Z","iopub.execute_input":"2023-06-24T19:14:01.535250Z","iopub.status.idle":"2023-06-24T19:14:01.845299Z","shell.execute_reply.started":"2023-06-24T19:14:01.535217Z","shell.execute_reply":"2023-06-24T19:14:01.844428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# summarize history for loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T19:14:35.822131Z","iopub.execute_input":"2023-06-24T19:14:35.822501Z","iopub.status.idle":"2023-06-24T19:14:36.092260Z","shell.execute_reply.started":"2023-06-24T19:14:35.822470Z","shell.execute_reply":"2023-06-24T19:14:36.091233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4 : Predict the labels of the test set\n\n","metadata":{}},{"cell_type":"code","source":"# Predict the labels of the test set\ny_pred = model.predict(test)\n\n# Convert the predictions from categorical to label encoded\ny_pred = np.argmax(y_pred, axis=1)\n\n# Create a DataFrame with the predictions\npredictions = pd.DataFrame({'ImageId': range(1, len(y_pred) + 1), 'Label': y_pred})\n\n# Display the first few rows of the predictions DataFrame\npredictions.head()","metadata":{"noteable":{"cell_type":"code"},"id":"9b073b04-75ee-4cb0-af15-d5db4cdb020a","outputId":"bd7db83e-3291-4461-f8be-c19a9b7b41fd","execution":{"iopub.status.busy":"2023-06-24T19:04:54.754729Z","iopub.execute_input":"2023-06-24T19:04:54.755109Z","iopub.status.idle":"2023-06-24T19:05:00.257694Z","shell.execute_reply.started":"2023-06-24T19:04:54.755077Z","shell.execute_reply":"2023-06-24T19:05:00.256708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"# Save the predictions to a CSV file\npredictions.to_csv('submission.csv', index=False)","metadata":{"id":"5ixvKArTzpld","execution":{"iopub.status.busy":"2023-06-24T19:05:00.262030Z","iopub.execute_input":"2023-06-24T19:05:00.264278Z","iopub.status.idle":"2023-06-24T19:05:00.356408Z","shell.execute_reply.started":"2023-06-24T19:05:00.264244Z","shell.execute_reply":"2023-06-24T19:05:00.355405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display the Test Data with Label","metadata":{}},{"cell_type":"code","source":" #Display the first 50 images in the test set\nfor i in range(50):\n    plt.figure(figsize=(2, 2))\n    plt.imshow(test[i].reshape(28, 28), cmap='gray')\n    plt.title(f'Label: {y_pred[i]}')\n    plt.show()","metadata":{"id":"qa8Y01tB0qY7","outputId":"ad5e188a-f6c3-44d9-9ee6-97e6dd112945","execution":{"iopub.status.busy":"2023-06-24T19:05:00.361172Z","iopub.execute_input":"2023-06-24T19:05:00.363430Z","iopub.status.idle":"2023-06-24T19:05:11.489526Z","shell.execute_reply.started":"2023-06-24T19:05:00.363396Z","shell.execute_reply":"2023-06-24T19:05:11.488644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The predictions have been successfully made on the test set. The 'ImageId' column contains the ID of the image, and the 'Label' column contains the predicted label for that image.\n\nThese predictions can now be submitted to the competition for evaluation. The competition will calculate the accuracy of the model based on these predictions.","metadata":{"noteable":{"cell_type":"markdown"},"id":"ed58f904-76af-482a-9519-4088743c8df7"}}]}