https://www.youtube.com/watch?v=yEXkEUqK52Q","metadata":{}},{"cell_type":"markdown","source":"\n
\n 2. Import the required packages and load the data\n
\n","metadata":{}},{"cell_type":"markdown","source":"In this section, I will perform the following steps to prepare the data for the digit recognition competition:\n1. Import the necessary packages for data manipulation,visualization,etc.\n2. Load the train.csv file and split it into train and validation sets randomly.\n3. Process the train data by doing the following:\n * Separate the labels (Y) from the features (X) and correct some of the mislabeled instances.\n * Convert the labels to one-hot encoding using pandas.get_dummies()\n * Normalize the features between 0 and 1 by dividing them by 255\n * Reshape the features to (m, 28, 28) to represent them as images\n4. Process the validation data in the same way as the train data\n5. Load the test.csv file as test data and process it in the same way as the train and validation data\n6. Print the shapes of the variables to check their dimensions\n\nThe following code shows how to do these steps:","metadata":{}},{"cell_type":"code","source":"# ================ Import the required packages\nimport tensorflow as tf\nimport numpy as np\nimport cv2\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n# ================ Load the train data and randomly split it to train and validation sets\ndata = pd.read_csv(\"train.csv\")\ntrain_data =data.sample(frac=0.9,random_state=200)\nvalidation_data =data.drop(train_data.index)\n\n# ================ prepare the train data\nY_train_digits = train_data[\"label\"]\n#edit the mislabeld instances:\nY_train_digits.iloc[208] = 4\nY_train_digits.iloc[1941] = 7\nY_train_digits.iloc[16942] = 7\nY_train_digits.iloc[17868] = 1\nY_train_digits.iloc[23234] = 9\nY_train_digits.iloc[34095] = 5\nY_train_digits.iloc[15808] = 4\n\n\nY_train_encoded = pd.get_dummies(Y_train_digits)\n\nX_train_pixels = train_data.drop(['label'], axis=1)\n# Normalize the X data to be in range [0 1]\nX_train_pixels = X_train_pixels / 255\n\nX_train_images = X_train_pixels.to_numpy()\nX_train_images = X_train_images.reshape((X_train_images.shape[0],(int)(np.sqrt(X_train_images.shape[1])),(int)(np.sqrt(X_train_images.shape[1]))))\n\n# ================ prepare the validation data\nY_valid_digits = validation_data[\"label\"]\n#edit the mislabeld instances:\nY_valid_digits.iloc[232] = 7\nY_valid_digits.iloc[1542] = 4\nY_valid_digits.iloc[2505] = 6\nY_valid_digits.iloc[4164] = 1\nY_valid_digits.iloc[425] = 6\n\n\n\nY_valid_encoded = pd.get_dummies(Y_valid_digits)\n\nX_valid_pixels = validation_data.drop(['label'], axis=1)\n# Normalize the X data to be in range [0 1]\nX_valid_pixels = X_valid_pixels / 255\n\nX_valid_images = X_valid_pixels.to_numpy()\nX_valid_images = X_valid_images.reshape((X_valid_images.shape[0],(int)(np.sqrt(X_valid_images.shape[1])),(int)(np.sqrt(X_valid_images.shape[1]))))\n\n# ================ Load the test data and prepare it\ntest_data = pd.read_csv(\"test.csv\")\nX_test_pixels = test_data\n# Normalize the X data to be in range [0 1]\nX_test_pixels = X_test_pixels / 255\n\nX_test_images = X_test_pixels.to_numpy()\nX_test_images = X_test_images.reshape((X_test_images.shape[0],(int)(np.sqrt(X_test_images.shape[1])),(int)(np.sqrt(X_test_images.shape[1]))))\n\n# ================ Print the shape of the defined variables\n\nprint(\"Train data: \",\"X_train_pixels:\",X_train_pixels.shape, \" X_train_images:\", X_train_images.shape, \\\n \" Y_train_digits:\", Y_train_digits.shape, \" Y_train_encoded:\", Y_train_encoded.shape)\n\nprint(\"Validation data: \",\"X_valid_pixels:\",X_valid_pixels.shape, \" X_valid_images:\", X_valid_images.shape, \\\n \" Y_valid_digits:\", Y_valid_digits.shape, \" Y_valid_encoded:\", Y_valid_encoded.shape)\n\nprint(\"Test data: \",\"X_test_pixels: \",X_test_pixels.shape, \" X_test_images: \", X_test_images.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 3. Learning Rate Tracker and Scheduler\n
\n\n\nOne of the important factors that influenced my results in this competition was the learning rate. The **learning rate is a hyperparameter** that controls how much the model updates its weights in each iteration of the training process. A too high learning rate may cause the model to overshoot the optimal solution, while a too low learning rate may cause the model to converge too slowly or get stuck in a local minimum. Therefore, choosing a suitable learning rate is crucial for the performance of the model.\n\nTo find the optimal learning rate, I used a **learning rate schedule**, which is a function that changes the learning rate according to some criteria, such as the number of epochs or the loss value. A learning rate schedule can help the model to start with a high learning rate to explore the search space quickly, and then gradually reduce the learning rate to fine-tune the solution. **I also used a LearningRateTracker**, which is a custom callback that records and plots the learning rate during the training process. This helped me to monitor how the learning rate changes and how it affects the model's accuracy and loss.\n\nBy using these two techniques, I was able to experiment **with different architectures** and hyperparameters, such as epochs or final_learning_rate, and find the best combination for this competition. I will show you how to implement and use them in the following sections","metadata":{}},{"cell_type":"code","source":"# Define a callback to track the learning rate at each epoch:\n\nclass LearningRateTracker(tf.keras.callbacks.Callback):\n def on_epoch_end(self, epoch, logs=None):\n optimizer = self.model.optimizer\n _lr = tf.keras.backend.get_value(optimizer.lr)\n print(f'\\nEpoch {epoch+1}: Learning rate is {_lr:.6f}.')\nlr_tracker = LearningRateTracker()\n\n# Define the learning rate schedule to decrease it by increaseing the num. of learned epochs:\nepochs = 50\nbatch_size = 100\ntrain_size = 37800\ninitial_learning_rate = 0.0004\nfinal_learning_rate = 0.00002\nlearning_rate_decay_factor = (final_learning_rate / initial_learning_rate)**(1/epochs)\nsteps_per_epoch = int(train_size/batch_size)\n\nlr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(\n initial_learning_rate=initial_learning_rate,\n decay_steps=steps_per_epoch,\n decay_rate=learning_rate_decay_factor,\n staircase=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 4. Saving The Best Model Weights\n
\n\n\nAnother technique that I found useful was **saving the best model weights** during the training process. This is because sometimes the model may achieve the highest validation accuracy at an earlier epoch than the final one, and then start to overfit or fluctuate. If we only use the final model weights, we may lose the optimal solution. Therefore, it is better to save the model weights whenever the validation accuracy improves.\n\nTo do this, I used a callback **function called ModelCheckpoint**, which is provided by Keras. This function allows you to specify a file path and a monitor metric, such as validation accuracy, and automatically save the model weights to that file whenever the metric improves. You can also set the save_best_only argument to True, so that only the best model weights are saved.\n\nIn my code, I **saved the model weights to my Google Drive folder**. This way, I can access and load the model weights from any device and avoid losing them if Google Colab disconnects or restarts. Latter I will demonstrate how this helped me to use **more free time of Colab GPUs**.\n","metadata":{}},{"cell_type":"code","source":"checkpoint_filepath = '/content/drive/MyDrive/model_weight'\nmodel_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n filepath=checkpoint_filepath,\n save_weights_only=True,\n monitor='val_accuracy',\n mode='max',\n save_best_only=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 5. Data Augmentation\n
\n\n\nOne of the challenges I faced in this competition was **overfitting**, which means that the model had **high variance** and did **not generalize well to new data**. To overcome this problem, I applied **data augmentation techniques** using the following layers:\n\n* RandomRotation\n* RandomTranslation\n* RandomZoom\n* RandomBrightness\n\nData augmentation is a way of creating new training examples **by applying random transformations to the original images**, such as rotating, translating, zooming, or changing the brightness. This helps the model learn more robust features and reduce the gap between the training and validation performance.\n\nBy **analyzing the errors in the training set**, I found that **the most effective data augmentation technique for this problem was RandomBrightness**, which randomly adjusts the brightness of the images. You can see in the code below that I used this layer with higher factor than the others.\n\nPlease do not run the cell below, as it is only a part of my model that I will show in the next section.\n","metadata":{}},{"cell_type":"code","source":"x = tf.keras.layers.RandomRotation(factor = 0.1,fill_mode='constant')(inputs)\nx = tf.keras.layers.RandomTranslation(height_factor=0.1,width_factor= 0.1,fill_mode='constant')(x)\nx = tf.keras.layers.RandomZoom(height_factor= 0.1,width_factor= 0.1,fill_mode='constant')(x)\nx = tf.keras.layers.RandomBrightness(factor=0.4, value_range=(0, 1))(inputs)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 6. The CNN model\n
\n\n\nIn the cell below, I use keras to build a CNN model for image classification. The model consists of the following components:\n\n* **Data augmentation** layers, as explained in the previous section.\n* **Conv2D layers** that perform convolution operations on the images and apply ReLU activations to introduce non-linearity.\n* **BatchNormalization layers** that normalize the inputs of each layer, which helps prevent overfitting and speed up the training process.\n* **Dropout layers** that randomly drop out some of the units in each layer and reduce the co-adaptation of features, which also helps prevent overfitting and improve generalization.\n* A **MaxPool2D layer** that performs max pooling on the feature maps and reduces their spatial dimensions.\n* A **Flatten layer** that flattens the output of the previous layer into a one-dimensional vector, which is then fed into the final layer.\n* A **Dense layer** that performs a linear transformation on the input vector and applies a softmax activation to produce a probability distribution over the classes.","metadata":{}},{"cell_type":"code","source":"inputs = tf.keras.Input(shape=(28,28,1))\ninitializer = tf.keras.initializers.HeNormal(seed = 1)\nx = tf.keras.layers.RandomRotation(factor = 0.1,fill_mode='constant')(inputs)\nx = tf.keras.layers.RandomTranslation(height_factor=0.1,width_factor= 0.1,fill_mode='constant')(x)\nx = tf.keras.layers.RandomZoom(height_factor= 0.1,width_factor= 0.1,fill_mode='constant')(x)\nx = tf.keras.layers.RandomBrightness(factor=0.4, value_range=(0, 1))(inputs)\nx = tf.keras.layers.Conv2D(filters = 28, kernel_size = 3,padding = \"same\", activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.35)(x)\nx = tf.keras.layers.Conv2D(filters = 56, kernel_size = 3,padding = \"same\",activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.4)(x)\nx = tf.keras.layers.Conv2D(filters = 112, kernel_size = 3,padding = \"same\",activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.45)(x)\nx = tf.keras.layers.Conv2D(filters = 224, kernel_size = 3,padding = \"same\" ,activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.5)(x)\nx = tf.keras.layers.Conv2D(filters = 448, kernel_size = 3,padding = \"same\" ,activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.5)(x)\nx = tf.keras.layers.Conv2D(filters = 224, kernel_size = 3,padding = \"same\" ,activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.5)(x)\nx = tf.keras.layers.Conv2D(filters = 112, kernel_size = 3,strides = 2,activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.4)(x)\nx = tf.keras.layers.Conv2D(filters = 56, kernel_size = 3,strides = 2,activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.35)(x)\nx = tf.keras.layers.Conv2D(filters = 28, kernel_size = 3,strides = 2, activation='relu',kernel_initializer=initializer)(x)\nx = tf.keras.layers.BatchNormalization()(x)\nx = tf.keras.layers.Dropout(0.35)(x)\nx = tf.keras.layers.MaxPooling2D(pool_size=(2, 2),strides=(2, 2), padding='valid')(x)\nx = tf.keras.layers.Flatten()(x)\noutputs = tf.keras.layers.Dense(10,activation=tf.keras.activations.softmax,kernel_initializer=initializer)(x)\n\nmodel = tf.keras.Model(inputs=inputs, outputs=outputs)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 7. Compile and Train the Model\n
\n\n\nThe cell below shows how to compile and train the model. The following steps are performed:\n\n* The model is compiled with the **Adam optimizer**, which is an adaptive gradient descent algorithm that adjusts the learning rate dynamically. The **lr_schedule** schedule is used to update the learning rate according to the predefined rule. A **weight decay** term is added to the loss function to penalize large weights and prevent overfitting.\n* The model is trained with the .fit() function, which takes the training data and labels as inputs and performs backpropagation to update the model parameters. Two callbacks are passed to the function: **lr_tracker** and **model_checkpoint_callback**. These are functions that are executed at certain points during the training process, such as at the end of each epoch. We have defined them at the previous sections.\n* The **accuracy and loss curves** for the training and validation sets are plotted to evaluate the model performance.","metadata":{}},{"cell_type":"code","source":"# Define the optimizer, the loss function and compile the model\nopt = tf.keras.optimizers.Adam(learning_rate= lr_schedule , beta_1 = 0.9, beta_2 = 0.999, weight_decay=1)\nlos = tf.keras.losses.CategoricalCrossentropy()\nmodel.compile(optimizer = opt,loss = los, metrics = ['accuracy'] )\n\nmodel.summary()\n\n# Train the model based on the training data, it will also reports the validation loss and accuracy\nhistory = model.fit(X_train_images, Y_train_encoded, validation_data=(X_valid_images,Y_valid_encoded),\\\n batch_size = batch_size, epochs = epochs,callbacks=[lr_tracker,model_checkpoint_callback], verbose = True)\nprint(\"last accuracy:\",history.history['accuracy'][-1])\n\nfig, axs = plt.subplots(1,2)\naxs[0].plot(history.history['accuracy'])\naxs[0].plot(history.history['val_accuracy'])\naxs[0].set_ylim([0.9, 1])\n\n\naxs[1].plot(history.history['loss'])\naxs[1].plot(history.history['val_loss'])\naxs[1].set_ylim([0, 0.1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 8. Analyze the results for the train and validation sets\n
\n\n\nThe following code displays the images from the validation set that are **incorrectly classified** by the model. I use the **number of misclassified images** in the validation set as a **metric to evaluate the performance** of the model, as it is **easy to compare**. For instance, if I see 25 misclassified images in my previous attempt and 20 misclassified images in my current attempt, it indicates that my recent modifications have improved the model.","metadata":{}},{"cell_type":"code","source":"# Predict the result for the validation data\nY_valid_pred_encoded = model.predict(X_valid_images)\nY_valid_pred_digits = np.argmax(Y_valid_pred_encoded,axis = 1)\n\n# Find the failed ones.\nvalid_error_idxs = np.where(Y_valid_pred_digits!=Y_valid_digits)[0]\nprint(\"Num. of mislabeled images in validation data: \",valid_error_idxs.shape[0])\nprint(valid_error_idxs)\nfig, axs = plt.subplots(1,np.min((10,valid_error_idxs.shape[0])))\nfor i in range(np.min((10,valid_error_idxs.shape[0]))):\n axs[i].axis('off')\n axs[i].imshow(X_valid_images[valid_error_idxs[i],:,:])\n axs[i].title.set_text(\"\"+ str(Y_valid_digits.iloc[valid_error_idxs[i]])+ \"\\n\"+str(Y_valid_pred_digits[valid_error_idxs[i]]) )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I also noticed an interesting thing when I checked the predictions on the training data. **The accuracy of the model** on the training set during the \"fit\" function is different from its accuracy when using the \"predict\" function and comparing the predicted labels with the original ones. This is because during training, **we only see the accuracy of the last epoch, which uses different weights for each step.** But when using the \"predict\" function, we **use the same final weights** for all the data.\n\nAnother point is that the training set has **some instances with incorrect labels**. I have fixed some of the labels when loading the data.","metadata":{}},{"cell_type":"code","source":"# Predict the result for the train data\nY_train_pred_encoded = model.predict(X_train_images)\nY_train_pred_digits = np.argmax(Y_train_pred_encoded,axis = 1)\n\n# Find the failed ones.\ntrain_error_idxs = np.where(Y_train_pred_digits!=Y_train_digits)[0]\nprint(\"Num. of mislabeled images in train data: \",train_error_idxs.shape[0])\nprint(train_error_idxs)\nfig, axs = plt.subplots(1,np.min((10,train_error_idxs.shape[0])))\nfor i in range(np.min((10,train_error_idxs.shape[0]))):\n axs[i].axis('off')\n axs[i].imshow(X_train_images[train_error_idxs[i],:,:])\n axs[i].title.set_text(\"\"+ str(Y_train_digits.iloc[train_error_idxs[i]])+ \"\\n\"+str(Y_train_pred_digits[train_error_idxs[i]]) )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 9. Loading the model weights and train more\n
\n\n\nSometimes, the best result is achieved in an epoch other than the last one. You can **load the weights** from that epoch by using the code below. The weights are saved in your specified directory.","metadata":{}},{"cell_type":"code","source":"model.load_weights(checkpoint_filepath)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can also resume your model training if you think it is not fully converged yet. The code below will continue the training from the last weights.","metadata":{}},{"cell_type":"code","source":"history1 = model.fit(X_train_images, Y_train_encoded, validation_data=(X_valid_images,Y_valid_encoded),\\\n batch_size = batch_size, epochs = 70,initial_epoch = 50,callbacks=[lr_tracker,model_checkpoint_callback], verbose = True)\nprint(\"last accuracy:\",history1.history['accuracy'][-1])\n\n\nfig, axs = plt.subplots(1,2)\naxs[0].plot(history.history['accuracy'] + history1.history['accuracy'])\naxs[0].plot(history.history['val_accuracy'] + history1.history['val_accuracy'])\naxs[0].set_ylim([0.9, 1])\n\naxs[1].plot(history.history['loss']+ history1.history['loss'])\naxs[1].plot(history.history['val_loss']+ history1.history['val_loss'])\naxs[1].set_ylim([0, 0.1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 10. Test set results and upload on Kaggle\n
\n","metadata":{}},{"cell_type":"markdown","source":"The cell below will generate the predictions on the test dataset and save them in a CSV file. Then it uploads this file to Kaggle and you can see your score on the leaderboard.","metadata":{}},{"cell_type":"code","source":"Y_test_encoded = model.predict(X_test_images)\nY_test_digits = np.argmax(Y_test_encoded,axis = 1)\n\noutput = pd.DataFrame({'ImageId': range(1,Y_test_digits.shape[0]+1,1), 'Label': Y_test_digits})\noutput.to_csv('submission1.csv', index=False)\n\n! kaggle competitions submit -c digit-recognizer -f submission1.csv -m \"Message\"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n 11. How to use Google Colab free GPUs for longer time\n
\n\n\nOne of the challenges that I faced during this competition was the limited time of using the free GPUs on Google Colab. Usually, Colab did not let me use its free GPUs for more than 5 hours per day.\n\nI am sharing this for **beginners who cannot afford to buy a Colab premium account**. If you can, I strongly recommend you to buy it and use Colab more efficiently.\n\nUnfortunately, I could not buy a Colab premium account because of **the restrictions in Iran**. To overcome this issue and use more free GPU time, **I followed these steps**:\n\n1. I saved my Colab notebook in my Google Drive\n2. I also saved my best model weights in my Google Drive\n3. I shared them with another Google account\n4. I opened my Colab notebook with my second Google account\n5. I loaded the saved weights\n6. I continued the training with the second account\n\nBy doing this, I was able to use Google Colab **free GPUs for almost 10 hours per day**.","metadata":{}},{"cell_type":"markdown","source":"\n
\n 12. Conclusion and feedback\n
\n\n\nI hope you enjoyed reading this notebook and learned something new from it. If you did, please upvote and leave a comment. I appreciate your feedback and suggestions. Thank you for your time and attention. Happy Kaggling! ","metadata":{}}]}