{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST - EDA & Baseline Linear Regression Model\n\nGreetings! In this notebook we will perform basic exploratory data analysis on the MNIST handwritten digit dataset.\n\nThen, we will examine the performance of a baseline logistic regression model, which will serve as the the basis of comparison for future model development.\n\nPlease leave any suggestions for improvement in the comments section below.\n\nLet's get into it!"},{"metadata":{},"cell_type":"markdown","source":"### Load the Good Stuff"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.pylab as pylab\nfrom sklearn.model_selection import cross_validate, train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import confusion_matrix\nimport keras\n\nparams = {'legend.fontsize': 'large',\n          'figure.figsize': (10, 8),\n         'axes.labelsize': 'large',\n         'axes.titlesize':'large',\n         'xtick.labelsize':'large',\n         'ytick.labelsize':'large'}\npylab.rcParams.update(params)\n\n# Prevent Pandas from truncating displayed dataframes\npd.set_option('display.max_rows', None)\npd.set_option('display.max_columns', None)\n\nsns.set(style=\"white\")\n\nSEED = 42","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load & Preview Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load master copies of data - these remain pristine\ntrain_ = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_ = pd.read_csv(\"../input/digit-recognizer/test.csv\")\nsample_submission = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\n\n# Take copies of the master dataframes\ntrain = train_.copy()\ntest = test_.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Previewing the training data as a dataframe, we can see that there are 785 features: 784 pixel values and 1 label. The 784 pixel values correspond to 28x28 px images that have been \"flattened\" from 2D into 1D. The label is our target value, i.e. the correct digit value represented by the flattened vector."},{"metadata":{"trusted":false},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Separate the target variable from the digits\ny = train.pop(\"label\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Obviously, viewing images as vectors of numbers is not very helpful. Instead, we can reverse engineer our recgonizable MNIST digits."},{"metadata":{"trusted":false},"cell_type":"code","source":"n_preview = 10\nfig, ax = plt.subplots()\n\nfor i in range(n_preview):\n    plt.subplot(2, 5, i+1)\n    image = train.iloc[i].values.reshape((28,28))\n    plt.imshow(image, cmap=\"Greys\")\n    plt.axis(\"off\")\n\nplt.suptitle(\"The First 10 MNIST Handwritten Digits\", y=0.9)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Likewise, we can look at the corresponding true labels for the first 10 digits."},{"metadata":{"trusted":false},"cell_type":"code","source":"y[0:10].values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Explore the Target Distribution"},{"metadata":{},"cell_type":"markdown","source":"The target variable is reasonably well-distributed between the 10 digits. This is good news - we won't have to deal with an imbalanced dataset."},{"metadata":{"trusted":false},"cell_type":"code","source":"digit_frequency = y.value_counts(normalize=True).to_frame()\nunique_digits= np.sort(y.unique())\nunique_digits_str = [str(d) for d in unique_digits]\n\nplt.bar(digit_frequency.index, digit_frequency[\"label\"].values)\nplt.title(\"Training Digit Frequency\")\nplt.xticks(unique_digits, unique_digits_str)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Preprocessing\n\nWe need to scale pixel values to range from 0 to 1. This prevents some data points from having a disproportionate impact on our model."},{"metadata":{"trusted":false},"cell_type":"code","source":"train = train / np.max(np.max(train))\ntest = test / np.max(np.max(test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Baseline Logistic Regression Model\n\nLet's compare two methods of establishing a baseline model: `train_test_split` and `cross_validate`.\n\nFirst, we'll split the data into training and validation sets. Then a Logistic Regression model is fit on the training set and scored on the validation set."},{"metadata":{"trusted":false},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(train, y, test_size=0.2, random_state=SEED)\nX_train.shape, X_valid.shape, y_train.shape, y_valid.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"lr = LogisticRegression(max_iter=1000)\n\nlr.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"yhat = lr.predict(X_valid)\nscore = lr.score(X_valid, y_valid)\nprint(\"Baseline score: {:.1%}\".format(score))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So our basic Logistic Regression model scores in the low 90s! Where is it going wrong?\n\nWhen we create a confusion matrix of actual vs. predicted labels, we see that the model's errors are reasonable. For example, `3` is often mis-classified as `8`. Likewise, there is some confusion between `7` and `9`."},{"metadata":{"trusted":false},"cell_type":"code","source":"c_matrix = confusion_matrix(y_valid, yhat, normalize=\"true\")\n\nplt.figure()\nsns.heatmap(c_matrix, annot=True, fmt=\".3f\", linewidths=.5, square = True, cmap = \"Spectral_r\")\nplt.ylabel(\"Actual Label\")\nplt.xlabel(\"Predicted Label\")\nplt.title(\"Accuracy Score: {:.1%}\".format(score), size = 15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**But**, before we get too comfortable with our 91.9% accuracy, let's check to see how the target distributions vary between the training and validation sets."},{"metadata":{"trusted":false},"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize=(15,6))\n\nsns.countplot(y_train, ax=ax[0])\nsns.countplot(y_valid, ax=ax[1])\nax[0].set_title(\"Training Labels\")\nax[1].set_title(\"Validation Labels\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The validation set appears to have disproportionately fewer `5`s. Interesting.\n\nLet's see if performing cross validation gives us better results on the validation set."},{"metadata":{"trusted":false},"cell_type":"code","source":"lr = LogisticRegression(max_iter=1000)\n\ncv_results = cross_validate(lr, train, y, cv=5, return_train_score=True)\ncv_results.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(\"Train: {}, Validation: {}\".format(cv_results[\"train_score\"].mean(), cv_results[\"test_score\"].mean()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Appears that the model does indeed score in the low 90s, regardless of the particular split that we use for the data."},{"metadata":{},"cell_type":"markdown","source":"### Previewing Misclassified Digits\n\nI think we can cut the model some slack! Some of these digits would be challenging to identify, even for a human."},{"metadata":{"trusted":false},"cell_type":"code","source":"misclassified = []\nfor i, (pred, actual) in enumerate(zip(yhat, y_valid)):\n    if pred != actual:\n        misclassified.append(i)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"n_preview = 15\nsamples = X_valid.iloc[misclassified]\nfig, ax = plt.subplots(figsize=(18,10))\n\nfor i in range(n_preview):\n    plt.subplot(3, 5, i+1)\n    image = samples.iloc[i].values.reshape((28,28))\n    plt.imshow(image, cmap=\"Greys\")\n    plt.title(\"Predicted: {} | Actual: {}\".format(yhat[misclassified[i]], y_valid.values[misclassified[i]]))\n    plt.axis(\"off\")\n\nplt.suptitle(\"10 Misclassified Digits: Predicted & Actual Labels\", y=0.99)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Generate Submission\n\nJust in case you choose to submit baseline results to see how they stack up relative to other entrants on the leaderboard (answer: not very well!)."},{"metadata":{"trusted":false},"cell_type":"code","source":"lr = LogisticRegression(max_iter=1000)\n\nlr.fit(X_train, y_train)\n\npreds = lr.predict(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sample_submission[\"Label\"] = preds\n# sample_submission.to_csv(\"baseline-logistic-regression.csv\", index=False)\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How is the predicted target variable distributed?"},{"metadata":{"trusted":false},"cell_type":"code","source":"sns.countplot(preds)\nplt.title(\"Count of Predicted Digits\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Conclusion\n\nThanks very much for reading. I hope this notebook is a helpful starting point that enables you to perform your own analyses on the MNIST dataset.\n\nStay tuned - in a follow up notebook I will experiment with various neural network configurations to see how they compare with the baseline logistic regression model.\n\nUntil next time, happy coding :)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":4}