{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction\n\nIn the [last notebook (Beginner Guide to Digit Recognition)](https://www.kaggle.com/craigmthomas/beginner-guide-to-digit-recognition), we examined how to quickly read input data from the training dataset, display it to see what it looked like, create a simple classifier, and then submit it to competition. In this notebook, we're going to take the next logical step to improve on our past kernel. In particular we will:\n\n1. Take a closer look at what the classifier got wrong.\n2. Augment our data so that we have more training examples to work with.\n3. See how much better (or worse) the classifier performs.\n4. Submit new results to the competition."},{"metadata":{},"cell_type":"markdown","source":"# 2. Load the Data\n\nIn this section, we'll go ahead and load the data into a Pandas DataFrame class. DataFrames are nice to work with because they provide all sorts of useful features to examine, analyze, and summarize data that is loaded into them. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntrain","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Reconstruct the Model\n\nLet's go ahead and rebuild our classifier from the first kernel. This time however, we're going to take our training data and split it into a training and validation set. We did something similar with cross-validation last time, but this time we're only going to create a single split. We want to see what our classifier is doing this time. To explain a little about what `train_test_split` does:\n\n* `x_train` is the training data without the `label` column.\n* `y_train` is the training data with _only_ the `label` column (ground truth).\n* `x_valid` is the training data without the `label` column that we're using for validation.\n* `y_valid` is the training data with _only_ the `label` column (ground truth) that we're going to use for validation."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\nfrom sklearn.svm import SVC\n\nfeatures = [\"pixel{}\".format(pixel_num) for pixel_num in range(0, 784)]\n\nx_train, x_valid, y_train, y_valid = train_test_split(\n    train[features], \n    train[\"label\"], \n    random_state=2020\n)\n    \nclassifier = SVC(kernel=\"rbf\", random_state=2020)\nclassifier.fit(x_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Examine Misclassified Digits\n\nNow that we have the classifier built, let's see what it's doing wrong. First, let's run it through the validation set and make predictions. We'll see if the precision, recall, and f1 scores can tell us anything interesting about how the classifier is mis-behaving."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import f1_score\n\npredictions = classifier.predict(x_valid)\nprint(classification_report(y_valid, predictions))\nprint(\": F1 micro score {:0.5}\".format(f1_score(y_valid, predictions, average=\"micro\")))\nprint(\": F1 macro score {:0.5}\".format(f1_score(y_valid, predictions, average=\"macro\")))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As expected, our precition, recall, and f1 scores are all very high. We knew this previously due to the high accuracy we saw in the original kernel. However, there is a noticeable issue: the numbers `9`, `8` and `3` have lower recalls values. This means we are not effective at identifying them. We can also see that our initial concern for the digit `5` may not be as bad as we thought - our precision and recall are both relatively high. What the classification report is telling us is that across the board we are doing very well on each class (with `9`, `8` and `3` being examples where we may need to investigate further). Let's take a look next as to how we're mis-classifying each digit. We can do this by looking at the confusion matrix."},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt     \nfrom sklearn.metrics import confusion_matrix\n\nconf_matrix = confusion_matrix(y_valid, predictions)\n_ = plt.figure(figsize=(10, 7))\nax = plt.subplot()\nhm = sns.heatmap(conf_matrix, annot=True, ax=ax)\n\n_ = ax.set_xlabel(\"Predicted labels\")\n_ = ax.set_ylabel(\"True labels\")\n_ = ax.set_title(\"Confusion Matrix\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, so now we can see a little more about what is going wrong. Let's look at some of the bigger problems:\n\n* The digit `3` is being recognized as a `5` twelve times. \n* The digit `4` is being recognized as a `9` nineteen times.\n* The digit `8` is being recognized as all other numbers several times.\n* The digit `9` is being recognized as a `4` twelve times, and a `7` sixteen times.\n\nThis gives us some areas to concentrate on. Let's see if we can figure out why we're seeing mis-classifications. To do this, let's pull up some examples of misclassified digits."},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Digit 3 Misclassified Examples\n\nLet's take a look at examples where the digit `3` was misclassified as another number"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\nfrom matplotlib.pyplot import imshow\nfrom math import ceil\n\ncombined_results = pd.DataFrame(x_valid)\ncombined_results[\"label\"] = y_valid\ncombined_results[\"predicted\"] = predictions\n\ndef get_df_with_mislabelled_digit(digit, features, df):\n    return df[features].loc[(df[\"label\"] == digit) & (df[\"predicted\"] != digit)]\n\ndef display_images(df):\n    images = df.values.reshape(df.shape[0], 28, 28)\n    height = int(ceil(len(images) / 10.0))\n    _, axs = plt.subplots(\n        height, \n        10, \n        figsize=(10, height), \n        gridspec_kw={'hspace': 0, 'wspace': 0}, \n        sharex='col', \n        sharey='row'\n    )\n    for index, ax in enumerate(axs.flatten()):\n        ax.axis(\"off\")\n        if index < len(images):\n            _ = ax.imshow(images[index], cmap=\"gray\")\n\ndisplay_images(get_df_with_mislabelled_digit(3, features, combined_results))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Digit 4 Misclassified Examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_images(get_df_with_mislabelled_digit(4, features, combined_results))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Digit 8 Misclassified Examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_images(get_df_with_mislabelled_digit(8, features, combined_results))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4 Digit 9 Misclassified Examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_images(get_df_with_mislabelled_digit(9, features, combined_results))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Data Augmentation\n\nFrom our examples above, we're seeing instances where our numbers are very distorted, or are very similar looking to other numbers. One thing we can do to try and offset the problem that we're having, is to take all of our training examples, and slightly modify them to give us more examples to learn from. The idea behind this suggestion is that by providing more examples to the classifier, we'll provide more examples of these edge cases of distorted numbers to the classifier during training, and be better able to separate them when we come upon them in unseen data.\n\nFor this particular task, we'll keep our training and testing set that we used for our classifier above, so we can compare how much lift we're actually getting. We'll take each existing example and perform a random transform on it (a random rotation of 20 degrees, a translation of 2 pixels in the x or y direction, and a random shear of 5 degrees). By applying this transform to all of our training examples 5 times, we will increase the amount of training data that we have available for the classifier by 500%.\n\nFirst, let's import the `torch` library, and set the random seed to a specific value so that we can replicate our results."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\n\nfrom torchvision import transforms\n\n_ = torch.manual_seed(2020)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we'll take take our classifier training data and re-combine it with our labels so that we can build new training examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train_with_label = pd.DataFrame(x_train)\nx_train_with_label[\"label\"] = y_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our transformation pipeline will convert our row data to an image, perform the affine transformation, and then convert it into a tensor (more on what a tensor is in a minute)."},{"metadata":{"trusted":true},"cell_type":"code","source":"random_transform = transforms.Compose(\n[\n    transforms.ToPILImage(),\n    transforms.RandomAffine(40, translate=(0.2, 0.2), shear=30),\n    transforms.ToTensor(),\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, now that we have our transform, we need to define a function that will take our row data and convert it into a tensor. We need to create a matching function that will take a tensor and convert it into an image. A tensor is a multi-dimensional matrix. It's similar to a `numpy` array, but is in the format that the `torch` library likes. Converting to a tensor is relatively straightforward."},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_to_tensor(df, features):\n    images = df[features].values.reshape(df[features].shape[0], 28, 28)\n    tensors = [torch.as_tensor(image).type(torch.ByteTensor) for image in images]\n    df[\"tensor\"] = tensors\n\nconvert_to_tensor(x_train_with_label, features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we've got tensor data for every image, we can go ahead and run our conversion."},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform_dataset(df):\n    new_df = df.copy()\n    new_df[\"tensor\"] = new_df[\"tensor\"].apply(lambda x: random_transform(x))\n    return new_df\n\nextra_dataset_1 = transform_dataset(x_train_with_label)\nextra_dataset_2 = transform_dataset(x_train_with_label)\nextra_dataset_3 = transform_dataset(x_train_with_label)\nextra_dataset_4 = transform_dataset(x_train_with_label)\nextra_dataset_5 = transform_dataset(x_train_with_label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that our random transforms have been applied, we need to unpack the tensor, and put the image data back into the proper column of our Pandas DataFrame."},{"metadata":{"trusted":true},"cell_type":"code","source":"def repopulate_rows(df):\n    for index, row in df.iterrows():\n        image_data = row[\"tensor\"].numpy().flatten() * 255\n        image_data = image_data.astype(\"uint8\")\n        for index, pixel in enumerate(image_data):\n            row[\"pixel{}\".format(index)] = pixel\n            \nrepopulate_rows(extra_dataset_1)\nrepopulate_rows(extra_dataset_2)\nrepopulate_rows(extra_dataset_3)\nrepopulate_rows(extra_dataset_4)\nrepopulate_rows(extra_dataset_5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's go ahead and append the new training datasets to our existing dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train_with_label = x_train_with_label.append(extra_dataset_1)\nx_train_with_label = x_train_with_label.append(extra_dataset_2)\nx_train_with_label = x_train_with_label.append(extra_dataset_3)\nx_train_with_label = x_train_with_label.append(extra_dataset_4)\nx_train_with_label = x_train_with_label.append(extra_dataset_5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's rebuild our classifier with the new examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"new_x_train = x_train_with_label[features]\nnew_y_train = x_train_with_label[\"label\"]\nclassifier = SVC(kernel=\"rbf\", random_state=2020)\nclassifier.fit(new_x_train, new_y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we can re-run our predictions and see if we got any lift."},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = classifier.predict(x_valid[features])\nprint(classification_report(y_valid, predictions))\nprint(\": F1 micro score {:0.5}\".format(f1_score(y_valid, predictions, average=\"micro\")))\nprint(\": F1 macro score {:0.5}\".format(f1_score(y_valid, predictions, average=\"macro\")))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we compare our results to the previous run at the start, we're seeing single point increases in either the precision or recall of nearly every digit. This is good news! Overall, we're seeing a lift of roughly 0.004 (0.4%) to our overall results. This is quite encouraging given how well the classifier was already working. "},{"metadata":{},"cell_type":"markdown","source":"# 6. Rebuilding and Submitting New Results\n\nNow that we know we get some lift, it's time to rebuild the classifier using all of our data, and then make a new set of submissions. Part of our dataset has already been augmented. We should augment the validation portion of our dataset, and then add that to our existing training data. This will give us a few thousand more examples to learn from. The code below takes our validation set, generates tensors, augments each example randomly 5 times, and then adds it back to our large dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"x_valid_with_label = pd.DataFrame(x_valid)\nx_valid_with_label[\"label\"] = y_valid\n\nconvert_to_tensor(x_valid_with_label, features)\n\nextra_dataset_1 = transform_dataset(x_valid_with_label)\nextra_dataset_2 = transform_dataset(x_valid_with_label)\nextra_dataset_3 = transform_dataset(x_valid_with_label)\nextra_dataset_4 = transform_dataset(x_valid_with_label)\nextra_dataset_5 = transform_dataset(x_valid_with_label)\n\nrepopulate_rows(extra_dataset_1)\nrepopulate_rows(extra_dataset_2)\nrepopulate_rows(extra_dataset_3)\nrepopulate_rows(extra_dataset_4)\nrepopulate_rows(extra_dataset_5)\n\nx_train_with_label = x_train_with_label.append(x_valid_with_label)\nx_train_with_label = x_train_with_label.append(extra_dataset_1)\nx_train_with_label = x_train_with_label.append(extra_dataset_2)\nx_train_with_label = x_train_with_label.append(extra_dataset_3)\nx_train_with_label = x_train_with_label.append(extra_dataset_4)\nx_train_with_label = x_train_with_label.append(extra_dataset_5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we've got all of our data back in the training dataset, we can go ahead and build the classifier one more time."},{"metadata":{"trusted":true},"cell_type":"code","source":"new_x_train = x_train_with_label[features]\nnew_y_train = x_train_with_label[\"label\"]\nclassifier = SVC(kernel=\"rbf\", random_state=2020)\nclassifier.fit(new_x_train, new_y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the classifier built, let's make some predictions, and generate a new submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction = classifier.predict(test)\nsub_df = pd.DataFrame({'ImageId': [index + 1 for index, _ in enumerate(prediction)], 'Label': prediction})\nsub_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Next Steps\n\nWe have explored how we can augment our data to give us more training examples to learn from, and how those training examples improved our classifier. The next step is to take this augmented data, and perform parameter tuning on the classifier. It may be that using different hyper-parameters yields a better classifier. We can also examine other classifiers to see how well they perform.\n\nIf you found this notebook useful, please feel free to experiment with it, drop a comment, or upvote it!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}