{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":0,"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"926d417b-80c0-114f-4ca8-9251c4cbc94d","_active":false},"source":"### My first time dealing with images in Python. How exciting! \n\nThe goal of today is build a classifier which will take an image of a number and decide if it is a 0, 1, 2....9. Unfortunately, this sounds easier than it will actually be (this probably goes for most classification tasks) seeing as how I have to run through a few different algorithms, each with different tuning parameters to control for complexity for each model.","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0d9000f0-8c19-0846-5ad4-1026e48b1091","_active":false},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a26ef26b-5ca5-ac93-2149-9630defeb157","_active":false},"outputs":[],"source":"image_train = pd.read_csv('../input/train.csv')\nimage_test = pd.read_csv('../input/test.csv')","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8a7cb267-2c48-4998-fc5b-e1cd9ad08969","_active":false,"collapsed":false},"outputs":[],"source":"# check for missing data\nsum(image_train.isnull().any())","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"eb90dc5e-3f86-7adb-3294-1acf8ae49bb5","_active":false},"source":"No  missing data, which is nice. Gonna check out a few rows and look at size to get a feel for what I might be in for as far as future analysis goes. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c4e673c8-1fd5-d2de-7c91-cdb2fbff488a","_active":false},"outputs":[],"source":"print (image_train.sample(5))\nprint (\"=============================\")\nprint (image_train.shape)","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"d9b97133-8f6a-3a40-c24f-3d3b69cc1d29","_active":false},"source":"Ok, so we have a fairly sparse data frame, which is pretty nice (especially given the amount of data). I should probably check to see if classes are balanced. Any imbalances could mess things up for the models trained. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"cbb7bc40-7128-023a-c639-86dc179f036c","_active":false},"outputs":[],"source":"image_train.label.value_counts(normalize = True)","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"514fc5ad-47cb-38d9-ad70-78ba0577f94a","_active":false},"source":"So it would appear the classes (i.e. digits 0, 1, ..., 9) are fairly balanced. They all seem to comprise about 10% of the data respectively. I want to see what these bad boys look like. I am not aware of any other methods, but one method would be to reshape each row of the data so it is a square matrix, and then use a heatmap to visualize the pixel intensities. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"94f98be8-8ebc-5ec9-6c03-f668d75cfe6a","_active":false},"outputs":[],"source":"# handy function to produce an image from the dataframe\ndef show_image(image_data, image_index, axis = None):\n    image = np.array(image_data.iloc[image_index,1:]).reshape(28,28)\n    sns.heatmap(image, \n                cbar = False,\n                xticklabels = False, \n                yticklabels = False,\n                ax = axis)","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"a36c1210-1a0f-b344-28d3-fc02f2f051a5","_active":false},"source":"A couple of images. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"206a8288-dbb3-9bda-38fa-8f73b09d3337","_active":false},"outputs":[],"source":"fig, (ax1, ax2) = plt.subplots(1,2)\nshow_image(image_train, 4, axis = ax1)\nshow_image(image_train, 20, axis = ax2)\nfig.tight_layout()","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f5456c4e-1165-5252-277d-cd5b2b7feda5","_active":false},"outputs":[],"source":"# setup plot for a nxn grid of random images from the dataset, using n = 4 for now\n# Also included are individual title labels for each plot\nnrows = 4\nncols = 4\nfig, axes = plt.subplots(nrows, ncols)\nfor i in range(0, nrows):\n    for j in range(0,ncols):\n        random_index = np.random.randint(42000)\n        show_image(image_train, random_index, axis = axes[i,j])\n        actual_value = image_train.iloc[random_index,0]\n        axes[i,j].set_title('Number {}'.format(actual_value))\nfig.tight_layout()","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4a90d934-aa33-6a07-f05a-74e4ced138f7","_active":false},"outputs":[],"source":"# size of testing data\nprint (image_test.shape)\nprint (\"=======================\")\nprint (image_test.sample(3))","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"88fa9014-7b6d-88b4-9012-08c448ed1a7f","_active":false},"source":"So, interestingly enough, the test set doesn't have labels associated with the pictures. For now I'll create a validation set from the original training data (15% of the data), and then under-sample the testing data by a different percentage so the test set and validation set are of the same size. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"9afbe994-ad5f-40d1-2490-d070ff119dbb","_active":false},"outputs":[],"source":"# handy dandy dataset splitting tool\nfrom sklearn.model_selection import train_test_split\n# split frame into predictors (pixel intensity) and target (labels)\npredictors = image_train.iloc[:,1:]\ntarget = image_train.iloc[:,0]\n# get predictors for training and validation data from 'predictors'\n# get labels for training and validation data from 'target'\ntrain_pred, val_pred, train_tar, val_tar = train_test_split(predictors, target, test_size = 0.15)\na, test = train_test_split(image_test, test_size = len(val_pred)/float(len(image_test)))\n\n# check to see if lengths are good\nprint (len(train_pred), len(train_tar))\nprint (len(val_pred), len(val_tar))\nprint (len(test))","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"24a0f5fa-8163-2f1d-7cb3-88c84b9b1b23","_active":false,"collapsed":false},"outputs":[],"source":"\"\"\" Import a preprocessing tool, useful for standardizing, normalizing, etc etc. \"\"\"\n# import all of it, just to be safe\nfrom sklearn.preprocessing import * ","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ac3325c6-606c-f4b3-f693-d5e5f4d534b2","_active":false},"outputs":[],"source":"# scale training and validation data, to speed up computations\n# also saves the mean and standard deviation for future scaling\nscaler = StandardScaler(with_mean = True, with_std=True).fit(train_pred)\ntrain_scaled = scaler.transform(train_pred)\nval_scaled = scaler.transform(val_pred)\ntest_scaled = scaler.transform(test)","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"563002c1-a718-82b2-c147-0e65f64250fe","_active":false},"outputs":[],"source":"# let's look at an original image, say image 30, then one produced by the transform\nfig, (ax1,ax2) = plt.subplots(2,1)\nsns.heatmap(train_scaled[30].reshape(28,28),\n           cbar = False,\n           xticklabels = False, \n           yticklabels = False,\n           ax = ax1)\nax1.set_title(\"Scaled Image\")\nsns.heatmap(np.array(train_pred.iloc[30,:]).reshape(28,28),\n            cbar = False, \n            xticklabels = False, \n            yticklabels = False,\n            ax = ax2)\nax2.set_title(\"Original Image\")\nfig.tight_layout()","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"7fe33550-8292-d704-1e03-f8099f204c28","_active":false},"outputs":[],"source":"fig, (ax1,ax2) = plt.subplots(2,1)\nsns.heatmap(val_scaled[30].reshape(28,28),\n           cbar = False,\n           xticklabels = False, \n           yticklabels = False,\n           ax = ax1)\nax1.set_title(\"Scaled Image from Validation Set\")\nsns.heatmap(np.array(val_pred.iloc[30,:]).reshape(28,28),\n            cbar = False, \n            xticklabels = False, \n            yticklabels = False,\n            ax = ax2)\nax2.set_title(\"Original Image from Validation Set\")\nfig.tight_layout()","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4965002b-db5e-edd7-010a-9f5cdb7ba368","_active":false},"outputs":[],"source":"fig, (ax1,ax2) = plt.subplots(2,1)\nsns.heatmap(test_scaled[30].reshape(28,28),\n           cbar = False,\n           xticklabels = False, \n           yticklabels = False,\n           ax = ax1)\nax1.set_title(\"Scaled Image from Test Set\")\nsns.heatmap(np.array(test.iloc[30,:]).reshape(28,28),\n            cbar = False, \n            xticklabels = False, \n            yticklabels = False,\n            ax = ax2)\nax2.set_title(\"Original Image from Test Set\")\nfig.tight_layout()","execution_state":"idle"},{"metadata":{"_cell_guid":"6a39f1ac-a1c2-adf4-b9e3-14b8969765ec","_active":false,"collapsed":false},"source":"Scaling the pixel intensities has the benefit of speeding up computations, lowering the risk of computational instability as well as having retained the shape properties of the original images. Unfortunately, this still results in each image being comprised of 784 features. I'll save the scaled data for now and run PCA on the scaled values for further analysis. ","execution_count":null,"cell_type":"markdown","outputs":[],"execution_state":"idle"},{"metadata":{"_cell_guid":"6fe059a8-92f0-90bc-c66c-cd2c49886ddb","_active":false,"collapsed":false},"source":"from sklearn.decomposition import * \npca = PCA().fit(train_pred)\nreduced_train = pca.fit_transform(train_pred)","execution_count":null,"cell_type":"code","outputs":[],"execution_state":"idle"},{"metadata":{"_cell_guid":"577b76ff-bebf-9883-a540-04f311e3f682","_active":false,"collapsed":false},"source":"cum_var_exp = pca.explained_variance_ratio_\neigens = range(1,len(var_exp)+1)\n\nplt.figure()\nplt.plot(eigens, cum_var_exp)\nplt.tight_layout()","execution_count":null,"cell_type":"code","outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"17f6c4dc-751b-864f-89ab-08c1d29e9817","_active":false},"outputs":[],"source":"# import logistic regression module\nfrom sklearn.linear_model import LogisticRegression\n# import time module, need to benchmark how long these algorithms will take\nimport time","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"35f6f977-8677-bfc0-d47e-a8b893076951","_active":false},"outputs":[],"source":"log_reg = LogisticRegression()\n\nstart = time.time()\nmodel = log_reg.fit(train_scaled, train_tar)\nend = time.time()\nelapsed = end - start\nif elapsed < 60:\n    print (\"Model took {} s to fit.\".format(elapsed))\nelse:\n    print (\"Model took {} minutes to fit.\".format(elapsed/60.0))","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"81d62922-bece-cbc7-d5be-1a72a0608152","_active":false},"source":"After much anticipation, the gradient descent algorithm (the algorithm used to calculate the weights to minimize the least squares objective) has converged. Unfortunately it has taken an unacceptably long time to do so...AND this was on a small dataset. I am under the suspicion that regularization will help, but I am not sure to what degree. I could also try using a PCA transformed dataset to see if that helps alleviate the problem. \n\nFor now, let's see how well the model actually performed on the validation set. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ec3b566d-529b-7d24-3dcc-88e2a59e797d","_active":false},"outputs":[],"source":"from sklearn.metrics import *\npredictions = model.predict(val_scaled)\nprint (classification_report(predictions, val_tar))","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"e68271f4-503e-e541-e61e-16a12fee4db9","_active":false},"source":"The F1 score is 0.91, so that's actually pretty good! Unfortunately, the test set doesn't have labels associated with it, but what I can do is print out the values of say, 20 test images, and their predicted values and work from there. For this, I'll need to modify the code above which produced the initial 4x4 grid of images.","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d212959e-b445-4b40-bd5b-97e9b767dfcd","_active":false},"outputs":[],"source":"# produce a predicted value for each example in the test set\ntest_predictions = model.predict(test_scaled)","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0947d97f-4fdd-3537-f9e3-84780f4aa46e","_active":false},"outputs":[],"source":"nrows = 5\nncols = 4\nfig, axes = plt.subplots(nrows, ncols)\nfig.set_size_inches(15,10)\nfor i in range(0, nrows):\n    for j in range(0,ncols):\n        random_index = np.random.randint(6300)\n        sns.heatmap(np.array(test.iloc[random_index,:]).reshape(28,28),\n                    cbar = False,\n                    xticklabels = False,\n                    yticklabels = False, \n                    ax = axes[i,j])\n        axes[i,j].set_title(\"Predicted: {}\".format(test_predictions[random_index]))","execution_state":"idle"},{"cell_type":"markdown","metadata":{"_cell_guid":"5fb6a625-ef18-2846-67f6-dd97d2fa056f","_active":false},"source":"Ho-k, so the model is doing somewhat ok on the 20 images from the test set, though not spectacular. From the output above, the F1 score for the model as it pertains to predicting the value of 5 is the lowest across all the classes. The F1 score is a tie when it comes to predicting 0's and 1's, more distinct shapes. Unfortunately, the model took far too long to train and took some time to do predictions on 6300 images in the scaled validation set. Oddly enough, it didn't take much time to predict values on the scaled test set. \n\nHow about using a different solver? The solver refers to the algorithm used to calculate the weights, which was a gradient ascent algorith. Stochastic gradient descent works much faster, at the cost of simply hovering around the global optimum. ","execution_count":null,"outputs":[],"execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"21c6cf4a-a55b-c351-fdb9-c3f2f697363c","_active":false},"outputs":[],"source":"def make_model(train_predictions, train_target, solver = None, max_iter = None):\n    if solver == None:\n        log_reg = LogisticRegression()\n    else:\n        log_reg = LogisticRegression(solver = solver, max_iter = max_iter)\n    start = time.time()  \n    model = log_reg.fit(train_predictions, train_target)\n    end = time.time()\n    elapsed = end - start\n    if end - start < 60:\n        print (\"Model took {} s to train {} examples.\".format(elapsed, len(train_predictions)))\n    else:\n        print (\"Model took {} minutes to train {} examples\".format(elapsed/60., len(train_predictions)))\n    return model ","execution_state":"idle"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8eaf842c-1659-f2bf-bdf9-6d3cd7c6821f","_active":false},"outputs":[],"source":null,"execution_state":"idle"}]}