{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\ntrain = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing"},{"metadata":{},"cell_type":"markdown","source":"First we'll split the training data into `training` and `validation` sets, while stratifying on the label."},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_validation, y_train, y_validation = train_test_split(train, \n    train['label'], \n    test_size=0.2, \n    random_state=42,\n    stratify=train['label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Convert image data to 28x28 arrays\n\nFirst we'll convert the dataframe to a format that can be used in keras/tensorflow. Currently the data is in a pandas dataframe, which is common for Kaggle datasets. We can see, for example, that the training dataset contains 42,000 rows of 785 columns. The first column contains the label, and the 784 columns after that contain the pixel values.\n\nWe want to put the label in a one array, and then take the pixel data and put it in a separate array of 28x28 arrays, and that image data should have this shape in the end: `(42000, 28, 28)`. We'll apply the same image data transformation to the test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We'll create a \"split\" of training, for cross-validation\ntrain_images_split = []\ntrain_labels_split = []\nvalidation_images = []\nvalidation_labels = []\n\n# We'll create a full training set\ntrain_images_full = []\ntrain_labels_full = []\n\n# And the test set\ntest_images = []\n\nfor index, row in x_train.iterrows():\n    train_images_split.append(row.values[1 : ].reshape((28, 28)))\n    train_labels_split.append(row['label'])\n    train_images_full.append(row.values[1 : ].reshape((28, 28)))\n    train_labels_full.append(row['label'])\n    \nfor index, row in x_validation.iterrows():\n    validation_images.append(row.values[1 : ].reshape((28, 28)))\n    validation_labels.append(row['label'])\n    train_images_full.append(row.values[1 : ].reshape((28, 28)))\n    train_labels_full.append(row['label'])\n\nfor index, row in test.iterrows():\n    test_images.append(row.values.reshape((28, 28)))\n    \n# Convert numpy array, while normalizing the image data\ntrain_labels_split = np.array(train_labels_split)\ntrain_images_split = np.array(train_images_split) / 255.\nvalidation_labels = np.array(validation_labels)\nvalidation_images = np.array(validation_images) / 255.\ntrain_labels_full = np.array(train_labels_full)\ntrain_images_full = np.array(train_images_full) / 255.\ntest_images = np.array(test_images) / 255.\n\nprint(train_labels_full.shape)\nprint(train_images_full.shape)\nprint(train_labels_split.shape)\nprint(train_images_split.shape)\nprint(validation_labels.shape)\nprint(validation_images.shape)\nprint(test_images.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot some digits to make sure we did that correctly."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(2, 2)\naxs[0, 0].imshow(train_images_full[42], cmap='gray')\naxs[0, 1].imshow(train_images_full[7], cmap='gray')\naxs[1, 0].imshow(train_images_full[2020], cmap='gray')\naxs[1, 1].imshow(train_images_full[0], cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Convolutional Neural Network (CNN) with keras/tf\n\nFinally, we'll train a simple Convoluted Neural Network (CNN) using keras/tensorflow. We'll set it up to do early stopping once the training accuracy reaches 99.5%, then we'll look at validation accuracy at that point."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Callback function so we can stop training once we've reached a desired level of accuracy.\nclass myCallback(tf.keras.callbacks.Callback):\n  def on_epoch_end(self, epoch, logs={}):\n    if(logs.get('accuracy')>=0.995):\n      print(\"\\n99.5% accuracy reached, stopping!\")\n      self.model.stop_training = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add an extra dimension\ntrain_images = np.expand_dims(train_images_split, axis=3)\nvalidation_images = np.expand_dims(validation_images, axis=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Model definition\nmodel = tf.keras.models.Sequential([\n    # Convolutional layer 1\n    tf.keras.layers.Conv2D(64, (3, 3), activation='relu', input_shape=(28, 28, 1)),\n    tf.keras.layers.MaxPooling2D(2, 2),\n    # Convolutional layer 2\n    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),\n    tf.keras.layers.MaxPooling2D(2, 2),\n    # Final layers\n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(512, activation=tf.nn.relu),\n    tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n])\n\n# Model compiler\nmodel.compile(optimizer='adam',\n              loss='sparse_categorical_crossentropy',\n              metrics=['accuracy'])\n\n# Generators\ntrain_gen = ImageDataGenerator().flow(train_images, train_labels_split, batch_size=32)\nvalidation_gen = ImageDataGenerator().flow(validation_images, validation_labels, batch_size=32)\n\n# Model fit\nhistory = model.fit(train_gen,\n                    validation_data = validation_gen,\n                    steps_per_epoch = len(train_images) / 32,\n                    validation_steps = len(validation_images) / 32,    \n                    epochs=10, \n                    callbacks=[myCallback()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We reached 99.5% training accuracy and the validation accuracy was very close!\n\nWe can visualize the performance of the model (training vs. validation) over the epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"acc = history.history['accuracy']\nval_acc = history.history['val_accuracy']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs = range(len(acc))\n\nplt.plot(epochs, acc, 'r', label='Training accuracy')\nplt.plot(epochs, val_acc, 'b', label='Validation accuracy')\nplt.title('Training and validation accuracy')\nplt.legend()\nplt.figure()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we see, the validation accuracy sort of peaked after the second epoch, after which there was marginal improvement, but by epoch 3 we begin to see some overfitting (i.e., improvement in model fit to the training data without much improvement on the validation set) before our callback method eventually stopped the training."},{"metadata":{},"cell_type":"markdown","source":"# Re-train with all data\n\nNow that we're sure our approach works, we'll re-train the model using all of the training data available."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Redefine training images and labels, using full sets\ntrain_images = np.expand_dims(train_images_full, axis=3)\ntrain_labels = train_labels_full\n\n# Re-define generator\ntrain_gen = ImageDataGenerator().flow(train_images, train_labels, batch_size=32)\n\n# Model fit\nhistory = model.fit(train_gen,\n                    steps_per_epoch = len(train_images) / 32,  \n                    epochs=10, \n                    callbacks=[myCallback()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission\n\nNow that we have our model finished, we can generate predictions on the test set and prepare our submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_images = np.expand_dims(test_images, axis=3)\npreds = model.predict_classes(test_images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_submission = pd.concat([pd.Series(range(1,28001), name = \"ImageId\")],axis = 1)\nmy_submission['Label'] = preds\nmy_submission.to_csv(\"my_submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}