{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Introduction\n\nIn this notebook, we're going to create a support vector machine classifier to understand each digit provided in the dataset, along with it's given label (0-9). The goal of this notebook is to provide a gentle introduction to get you up and running with easy to understand code and comments. We'll show you how to do the following:\n\n1. Read the input data into a Python structure we can use for analysis.\n2. Display an image so we can see what it looks like.\n3. Create a simple classifier, and perform basic parameter tuning.\n4. Run a set of predictions on the testing data and submit the results."},{"metadata":{},"cell_type":"markdown","source":"# 2. Load the Data\n\nIn this section, we'll go ahead and load the data into a Pandas DataFrame class. DataFrames are nice to work with because they provide all sorts of useful features to examine, analyze, and summarize data that is loaded into them. The code below will do two things:\n\n1. It will import the Pandas library.\n2. It will create two DataFrames for us to look at: one that includes the training data, and the other that includes our testing data."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntrain","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Simple Data Exploration\n\nWe can see from the above that we have 785 columns. The first column is the class `label` - this is what tells us what the number is. The remaining 784 columns contain pixel data. Values can range between 0 and 255 - with values near 0 being black, and values of 255 being white. We can perform a simple visualization of some of the numbers contained in the dataframe very easily. Let's see the image in row 0, which is supposed to be the number `1`."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\nfrom matplotlib.pyplot import imshow\n\nfeatures = [\"pixel{}\".format(pixel_num) for pixel_num in range(0, 784)]\nrow_to_examine = 0\nimage_data = np.reshape(train[features][row_to_examine:row_to_examine+1].to_numpy(), (28, 28))\n_ = imshow(image_data, cmap=\"gray\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, we have a little bit of a challenge ahead of us if we want nice results. Some of the images such as this one are skewed and rotated. This means we're going to need many examples of skewed and rotated numbers to learn from. For now, let's see how many examples of each class (digits 0-9) we have to work with. What we want to see is equal numbers from each class so that we don't have too much _skew_ from one class to the next, which will impact how well our classifier will perform."},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\n\ncounts = pd.DataFrame(train[\"label\"].value_counts())\ncounts[\"Digit\"] = counts.index\ncounts.rename(columns={\"label\": \"Number of Examples\"}, inplace=True)\n_ = sns.barplot(x=\"Digit\", y=\"Number of Examples\", data=counts)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see here that we have the largest number of examples for the digit `1`, and the fewest examples for the digit `5`. We'll want to keep that in mind, as we're probably going to need to augment our dataset in the future to provide more training examples for the digit `5`."},{"metadata":{},"cell_type":"markdown","source":"# 3. Creating a Classifier\n\nIn this step below, we'll use a Support Vector Machine (SVM) classifier called SVC. One of the choices we'll have to make is what `kernel` to use. To figure out what works best, we'll build a series of classifiers, and then choose the one that works the best. To do that, we're going to iterate through all the options (`linear`, `poly`, `rbf`, and `sigmoid`), slice up our test data into a _test_ set and a _validation_ set, and test to see which model is best. For each model, we do this building 3 times so that we can do what is called a _cross-validation_. Once we find the best parameter for `kernel`, we'll build a model using the entire dataset."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import gc\n\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.svm import SVC\n\nimport matplotlib.pyplot as plt \n%matplotlib inline\n\nkernels = [\"linear\", \"poly\", \"rbf\", \"sigmoid\"]\nscores = []\n\nfor kernel in kernels:\n    print(\"--> Building model for {} kernel\".format(kernel))\n    model = make_pipeline(StandardScaler(), SVC(kernel=kernel, random_state=2020))\n    model_scores = cross_val_score(model, train[features], train[\"label\"], cv=3, n_jobs=-1, scoring='accuracy')\n    print(\": Accuracy scores {}\".format(model_scores))\n    scores.append(model_scores.mean())\n    del model\n    gc.collect()\n\nplt.plot(kernels, scores)\nplt.xlabel('Kernel Type')\nplt.ylabel('Cross-Validated Accuracy')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some notes about the above code:\n\n1. We used a `random_state` variable above and set it to a specific number. This is so we can ensure the same results each time we run the kernel. \n2. The `cross_val_score` function performs our cross validation 3 times as specified by the `cv=3` option. \n3. We call `del model` and `gc.collect()` to keep our memory usage down - remember we only have 16 GB to work with (it may sound like a lot, but for some competitions 16 GB is not enough). \n\nFrom the results, it looks like our best model is an SVC classifier with an `rbf` kernel. Let's go ahead and build a final model using all of the data, and then submit a result."},{"metadata":{"trusted":true},"cell_type":"code","source":"classifier = SVC(kernel=\"rbf\", random_state=2020)\nclassifier.fit(train[features], train['label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Submit Predictions\n\nNow that the model is built, we'll run the model on our test data, and submit our predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"prediction = classifier.predict(test)\nsub_df = pd.DataFrame({'ImageId': [index + 1 for index, _ in enumerate(prediction)], 'Label': prediction})\nsub_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Next Steps\n\nNow that we've created a simple classifier, there are many other things we should explore.\n\n1. *What the classifier got wrong* - we'll want to take a look at examples of how the classifier is misbehaving. Specifically, we want to see what image examples the classifier is mis-labelling. \n2. *Augment the testing set* - the results from step 2 showed us we had fewer training examples for the digit `5`. There are two things we can try and do to create more examples to work from: 1) rotate our images slightly to give us more training examples; and 2) move the images around in the 28x28 image area. We can do this for every digit, and for `5` in particular until we have roughly equal numbers of the digits to work with.\n3. *Try other classifiers* - while our simple classifier here provided decent results, we probably want to see if there are other classifier types that are more appropriate for the task at hand.\n4. *Further tune the classifier* - we did rudimentary classifier tuning to get us decent results. We probably want to do more tuning to see if we can improve our predictions. For the `SVC` classifier, there are a number of parameters such as `degree` and `gamma` that we can try and tune. Using scikit learn's `GridSearchCV` will help us iterate through may examples.\n\nThat's it for a simple introduction! If you found this notebook useful, please feel free to experiment with it, drop a comment, or upvote it!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}