{"metadata": {"_change_revision": 0, "_is_fork": false, "language_info": {"mimetype": "text/x-python", "pygments_lexer": "ipython3", "codemirror_mode": {"name": "ipython", "version": 3}, "nbconvert_exporter": "python", "name": "python", "file_extension": ".py", "version": "3.6.1"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat": 4, "nbformat_minor": 0, "cells": [{"metadata": {"_cell_guid": "00b63778-674d-0a2d-e5b0-8956c5ce1770", "_uuid": "4801364525981259561f599dea43f491aff1411c"}, "execution_count": null, "outputs": [], "source": "# Introduction\n\nThis uses SVM to classify the digits as described in [A beginner\u2019s approach to classification](https://www.kaggle.com/archaeocharlie/digit-recognizer/a-beginner-s-approach-to-classification).  This uses the same method as the original approach where the grey scale images are converted to black-and-white, which was shown to increase the accuracy from 10% to 88.7% when using 5000 test images.  \n\nThe first goal is to see how the accuracy changes as a function of the number of training images used.  The second goal is to investigate how further image processing will increase the accuracy.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "d0abbfc5-e04e-98d3-8c5c-37d114daa804", "_uuid": "1843c5606aa375124811a57fd7b9a60a87a918d1", "trusted": false, "_execution_state": "idle"}, "execution_count": null, "outputs": [], "source": "import pandas as pd\nimport matplotlib.pyplot as plt, matplotlib.image as mpimg\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import svm\nimport time\nimport scipy\nimport numpy as np\n\n%matplotlib inline", "cell_type": "code"}, {"metadata": {"_cell_guid": "bbc96018-d0d5-c647-14f6-e2e1f3834791", "_uuid": "1ead6369dcdd55667562682a5556842178eaa1d7"}, "execution_count": null, "outputs": [], "source": "Read in the data and setup the classifier.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "9277f8d2-e2a3-58fa-620c-c780128957cc", "_uuid": "5e84da74f97cf08ed4fcf58e2d75911b20ce1c9b", "trusted": false, "_execution_state": "idle"}, "execution_count": null, "outputs": [], "source": "labeled_images = pd.read_csv('../input/train.csv')\nclf = svm.SVC()", "cell_type": "code"}, {"metadata": {"_cell_guid": "fa6bc268-eaad-644c-3079-d0e4c805aab0", "_uuid": "25cb2c3fe0ff9b0ce1b4bb1d65b037f5fedfbfb6"}, "execution_count": null, "outputs": [], "source": "Use the same number of training and testing images as SVM example for a check.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "40ea409e-229f-d5d7-8dd5-e34faf6d7789", "_uuid": "865536ace89fd84851f7648ba7e4cca9a438bbb6", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": ["images = labeled_images.iloc[0:5000,1:].copy()\n", "labels = labeled_images.iloc[0:5000,:1].copy()\n", "#make black-and-white images\n", "images[images>0] = 1\n", "\n", "train_images, test_images,train_labels, test_labels = \\\n", "    train_test_split(images, labels, train_size=0.8, random_state=0)\n", "clf.fit(train_images, train_labels.values.ravel())\n", "print(clf.score(test_images,test_labels))"], "cell_type": "code"}, {"metadata": {"_cell_guid": "8b233b7d-dd9e-ae15-2251-2564cc43288b", "_uuid": "d33f8882c637b68ff7e7360675bda216ba4b5cf5"}, "execution_count": null, "outputs": [], "source": "Accuracy is 88.7% as expected, so we can start trying new ideas.\n\n## Number of Training Images\n\nTo test the number of optimal training images, I'll test the accuracy for training images ranging from 200 to 10000.  This takes a while, so I'll print out the number of images and the accuracy so I can see the progress.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "a7e0c88d-10aa-90ac-459e-faceac42b186", "_uuid": "a99c5d8afd39f854b27d6f4d26cf601b7a79df86", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "score = []\nclf_time = []\nnips = 400\nnimages = range(nips,10001,nips)\nlabels = labeled_images.iloc[0:10000,:1].copy()\nimages = labeled_images.iloc[0:10000,1:].copy()\nimages[images>0] = 1\n\nfor i in nimages:\n    start = time.time()\n    train_images, test_images,train_labels, test_labels = \\\n        train_test_split(images[0:i], labels[0:i], train_size=0.8, random_state=0)\n    clf.fit(train_images, train_labels.values.ravel())\n    clf_time.append(time.time()-start)\n    s = clf.score(test_images,test_labels)\n    score.append(s)\n    print(i,s)", "cell_type": "code"}, {"metadata": {"_cell_guid": "57928837-43a2-c755-3f94-530c9583c3ef", "_uuid": "fa9c59b3ec425b8d4c1be342c1882b4a53063f75", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "t = np.array(clf_time)\ns = np.array(score)\nn = np.array(nimages)\nplt.plot(n, s)\nplt.ylabel('Score')\nplt.xlabel('Training Images Used')\nplt.show()", "cell_type": "code"}, {"metadata": {"_cell_guid": "a115dfd6-d79e-d845-56f1-1ffbf8f4721e", "_uuid": "00687146bab077b1e20e361322b69b61f1a2b6b0"}, "execution_count": null, "outputs": [], "source": "It appears about 3000 images is a good combination of accuracy and time efficiency, so it's a good number of images to use for testing new ideas before increasing to more images.  \n\n\n# Image Processing\n\nNext, I'm going to try cropping the images to be tight on the numbers, and centering on the numbers in the cropped image.  First, create the cropping function.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "b0a11981-722d-3e9b-147c-eecfbf92ad37", "_uuid": "08f77aad0a661c97599561ccad338f848cd50d67", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "def crop_img(in_img):\n    sq_img = np.array(in_img.values.reshape((28,28)))\n    \n    x = sq_img.sum(axis=1)\n    xn = np.array(np.where(x>0))\n    xm = int((xn.max()-xn.min())/2)\n    xc = int((xn.max()+xn.min())/2)+1\n\n    y = sq_img.sum(axis=0)\n    yn = np.array(np.where(y>0))\n    ym = int((yn.max()-yn.min())/2)\n    yc = int((yn.max()+yn.min())/2)+1\n\n    hnis = np.max([xm,ym])+1\n    cimg = scipy.misc.imresize(sq_img[xc-hnis:xc+hnis,yc-hnis:yc+hnis],[28,28])\n    cimg[cimg>0]=1\n    return cimg.reshape(784)\n\n#th = threshold level for converting to black-and-white\ndef fast_crop(in_img,th):\n    nipc = 25 #nipc = number of images per crop\n    nc = int(in_img.shape[0]/nipc)\n    out_img = in_img.copy()\n    for i in range(nc):\n        im = in_img.iloc[i*nipc:(i+1)*nipc,:].copy()\n        im[im/256 < th] = 0\n        im[im > 0] = 1\n        for j in range(nipc):\n            im.iloc[j] = crop_img(im.iloc[j])\n        out_img.iloc[i*nipc:(i+1)*nipc,:] = im\n    return out_img", "cell_type": "code"}, {"metadata": {"_cell_guid": "d660bf5c-4bbc-d843-118b-6378237ad799", "_uuid": "d85eadf4f5cc612044ea6e9d014b10345c4cfc8f"}, "execution_count": null, "outputs": [], "source": "Next, crop 5000 images to see if this increases the accuracy above the 88.7% accuracy for uncropped images.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "dc4e0fc7-9f58-745d-7a1f-ed4d3991e866", "_uuid": "8416355419475e80524ba53f8fd4f362aefc6840", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "images = labeled_images.iloc[0:5000,1:].copy()\nlabels = labeled_images.iloc[0:5000,:1].copy()\nimages = fast_crop(images,0)\nplt.imshow(np.array(images.iloc[0]).reshape((28,28)))\nplt.title('Centered and Cropped Image')\n\ntrain_images, test_images,train_labels, test_labels = \\\n    train_test_split(images, labels, train_size=0.8, random_state=0)\nclf.fit(train_images, train_labels.values.ravel())", "cell_type": "code"}, {"metadata": {"_cell_guid": "bea63231-4e23-de7d-abec-eb6826174f4c", "_uuid": "6b885772cdb0225e7fff16dee8bd1d49f7321660", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "print(clf.score(test_images,test_labels))", "cell_type": "code"}, {"metadata": {"_cell_guid": "4d5e75e1-b3ba-1112-30dd-df25c52e950a", "_uuid": "48ad9e23f2ddfbeabef90a2682f56839d4958465"}, "execution_count": null, "outputs": [], "source": "Cropping and centering the numbers increased the accuracy from 88.7% to 91.3%.  However, the function locates the numbers based on where the pixels are non-zero.  If an image doesn't have the background zeroed out everywhere, it won't work correctly.  For example, the image in index number 6889 has two pixels on the left side that are non-zero, creating a problem with finding the center and edges of the actual number.\n\nThe non-zero background pixels can be ignored by increasing the threshold of what pixels would be considered part of the number.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "6f8157e4-69ea-a1e4-8e43-f71cc4622795", "_uuid": "d4bddbef069f577b1c30eefac9e3ec9976cf3b30", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "im = labeled_images.iloc[6889,1:].copy()\nplt.imshow(np.array(im).reshape((28,28)))\nplt.title('Original Image[6889]')\nplt.show()\n\nimport copy\nim2 = im.copy()\nim2[im2 > 0] = 1\nplt.title('Black-and-White Image[6889]')\nplt.imshow(np.array(im2).reshape((28,28)))\nplt.show()\n\nim3 = im.copy()\nim3[im3/256. < 0.15] = 0\nim3[im3 > 0] = 1\nplt.title('Black-and-White Image[6889] \\nwith 15% Threshold')\nplt.imshow(np.array(im3).reshape((28,28)))\nplt.show()", "cell_type": "code"}, {"metadata": {"_cell_guid": "3a53e095-6884-37cf-0c59-628b7891ae73", "_uuid": "c7cd3a23584422312f49d3837ce74a258379fc9d", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "images = labeled_images.iloc[0:5000,1:].copy()\nimages = fast_crop(images,0.15)\nplt.imshow(np.array(images.iloc[4000]).reshape((28,28)))\nplt.title('Centered and Cropped Image')\nplt.show()\n\ntrain_images, test_images,train_labels, test_labels = \\\n    train_test_split(images, labels, train_size=0.8, random_state=0)\nclf.fit(train_images, train_labels.values.ravel())", "cell_type": "code"}, {"metadata": {"_cell_guid": "77b1b89d-7073-b7b4-7ce7-b927ae41f52d", "_uuid": "0d3d17a554c91e65dd2fe2def306a35e03f0f801", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "print(clf.score(test_images,test_labels))", "cell_type": "code"}, {"metadata": {"_cell_guid": "64b6e776-5bf0-102b-7ead-aa7d32cccaf2", "_uuid": "8da1137cfbb7fb10ac033e0518376f4985617abb"}, "execution_count": null, "outputs": [], "source": "By increasing the threshold to 15% max, the accuracy increased to 92.4%.  This just begs the question of what the optimal threshold would be:", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "ea9334a9-6b8f-7374-8a57-a0d3196757f8", "_uuid": "c3446b0283428b223d98d73be04d72c1ba2afb92", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "th = np.arange(0.2,1,0.1)\nimages = labeled_images.iloc[0:3000,1:].copy()\nlabels = labeled_images.iloc[0:3000,:1].copy()\n\nfor i in range(8):\n    cc_im = fast_crop(images,th[i])\n    train_images, test_images,train_labels, test_labels = \\\n        train_test_split(cc_im, labels, train_size=0.8, random_state=0)\n    clf.fit(train_images, train_labels.values.ravel())\n    plt.imshow(np.array(cc_im.iloc[106]).reshape((28,28)))\n    title_str = 'Threshold: '+str(th[i])+'\\nAccuracy: '+str(clf.score(test_images,test_labels))\n    plt.title(title_str)\n    plt.show()\n\n    ", "cell_type": "code"}, {"metadata": {"_cell_guid": "2ceafb82-0784-3e13-a988-876cae485abf", "_uuid": "a7c55ee187237547957eec12844c9e481e33e078"}, "execution_count": null, "outputs": [], "source": "\nThe accuracy increases until 40% as the lower valued pixels at the edges of the numbers are removed, making the numbers more defined.  Between 40% and 60% the accuracy stays constant at 93.2%.  Above 60%, the accuracy decreases as pixels that are inward from the edge of the numbers are removed.  The optimal threshold would thus be 50%.  \n\nCrop and center the images, and run the predictor on them.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "f7e2ba18-444d-310b-ffe1-ffb573b6f107", "_uuid": "e361947e04ace9ae763461693b376291fcddad47", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "#cropped/centered images\ncc_images = fast_crop(labeled_images.iloc[:,1:].copy(),0.5)\n\n#save images\ncc_images.to_csv('cropped.csv', index=False)", "cell_type": "code"}, {"metadata": {"_cell_guid": "1485a319-f29c-ce75-5f9b-5a00638830f8", "_uuid": "4cf53117b22d25c2d0099708c62a5c16ffa26914", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "labels = labeled_images.iloc[:,:1].copy()\n\ntrain_images, test_images,train_labels, test_labels = \\\n    train_test_split(cc_images, labels, train_size=0.8, random_state=0)\nclf.fit(train_images, train_labels.values.ravel())", "cell_type": "code"}, {"metadata": {"_cell_guid": "274e6b8d-e62b-50cf-ea65-b3f6d417a464", "_uuid": "d9dde75c2bb215c411d7575a9f58c393f5934e10", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "print(clf.score(test_images,test_labels))", "cell_type": "code"}, {"metadata": {"_cell_guid": "053be6e5-c276-d3c2-3499-298dd09c02ef", "_uuid": "29260f14668f8ba6d57d974f1694f0353fac50e6"}, "execution_count": null, "outputs": [], "source": "\nBy centering the numbers in the images and cropping tight on them, the prediction accuracy for the set has been increased to 95.2%.\n\n\n## Labeling the Test Data\n\nFor a competition submission, we upload the images from test.csv, then we can crop and center the numbers in the images before making a prediction, which is output to results.csv.  Similar to the Beginner's Guide, we'll do this for 5000 images.", "cell_type": "markdown"}, {"metadata": {"_cell_guid": "1831672b-d1dc-8d17-38e8-9f0f7cd5d859", "_uuid": "12b7f1d8d2daba23a554c7b5327b82268cb283d9", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "test_data = pd.read_csv('../input/test.csv')\ncc_testdata = fast_crop(test_data.iloc[0:5000].copy(),0.5)\n\nresults=clf.predict(cc_testdata)", "cell_type": "code"}, {"metadata": {"_cell_guid": "3cf7d84a-314f-58ab-1d6f-41f91437439a", "_uuid": "258a33472b7b89f32d38deefc85a14e0cbb7ffa8", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "results", "cell_type": "code"}, {"metadata": {"_cell_guid": "07667710-3b2f-ae74-0849-3d0e7f4a7c1d", "_uuid": "234c70ab92e99f5457a2e0f836b8935280be85ea", "trusted": false, "_execution_state": "busy"}, "execution_count": null, "outputs": [], "source": "df = pd.DataFrame(results)\ndf.index.name='ImageId'\ndf.index+=1\ndf.columns=['Label']\ndf.to_csv('results.csv', header=True)", "cell_type": "code"}]}