{"cells":[{"metadata":{"_cell_guid":"fe5ca418-2951-4f1a-891c-259c5c143af9","_uuid":"638ca26e690bcca75d5f582dbf62c902eb569f2e"},"cell_type":"markdown","source":"# Digits Recognizer\n## Introduction\nThis kernel is a humble start to learn the beautiful world of machine learning. The dataset doesn't require much introduction for individuals who have been introduced to machine learning. My intention here is not to describe the dataset. More information can be obtained online. A quick reading can be done on wikipedia - https://en.wikipedia.org/wiki/MNIST_database.\n\nBut no efforts are worth without a citation to Modified National Institute of Standards and Technology for this wonderful dataset. This kernel will initally focus on how to do a first pass of prediction using knn and I shall try to develop the kernel along the way. Request the reader to correct or point out my mistakes and provide suggestions that could help to improve the kernel. "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # for visualization\nimport seaborn as sns # personal choise for viz\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint('List of files in the input directory:')\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.\n\n# Set seaborn\nsns.set()","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"## Load data and inspect it\nWe see that there are 2 datasets in csv format. As is always the case, the train dataset has labels and we need to predict the labels for the test dataset. Let us try to import into a pandas dataset and have a look at the structure of the data."},{"metadata":{"_cell_guid":"02f8f6fa-7c6e-4a4f-b07a-70b03c7a1aa2","_uuid":"cc8585afa8b3ce495d2635e9e9ffd11923ef41ad","collapsed":true,"trusted":true},"cell_type":"code","source":"# Load train and test dataset\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"3c40d51c-65c2-4d31-ae9e-d3652f46bcc9","_uuid":"8104d508abfd9f731fbd209ebb88a8e145592e96","trusted":true},"cell_type":"code","source":"# Inspect the shape of the datasets\nprint('Shape of train dataset - Rows: %d; Columns: %d' % train.shape)\nprint('Shape of test dataset - Rows: %d; Columns: %d' % test.shape)","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"92917221-f367-4f5b-b20e-61e7d73af640","_uuid":"cd51dd7e612f769d8100c4530d7c3ec957788394"},"cell_type":"markdown","source":"There is one column less in the train dataset which is the column that needs to be predicted. Let us now look at the column names of these datasets."},{"metadata":{"_cell_guid":"894e8284-e096-445e-a1e6-e8215a5d9aa2","_uuid":"a60c4570f7bd328951671c56aa1bd279d32f3686","trusted":true},"cell_type":"code","source":"print('Training dataset columns:')\nprint(train.columns)\nprint('\\nTesting dataset columns:')\nprint(test.columns)","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"a387b9dc-6a3b-44db-97f4-4243c63f205a","_uuid":"55b74ef43433e1835e826fdec55d1076d7026e73"},"cell_type":"markdown","source":"The first column in the training dataset is the label and followed by 784 columns viz. pixel0, pixel1, ... pixel783. The testing dataset is the same except that it doesn't have the label column which is the way it should be. Else there is no fun. :)\n\nLets us print out a few records of the training and testing datasets for 8 arbitrary columns to visually examine the content in the pixel columns."},{"metadata":{"_cell_guid":"78be06ce-28c3-489e-95c0-94f40b8df2bd","_uuid":"b9fde977add9a9b877b4b77bcb2c6748d151ead2","trusted":true},"cell_type":"code","source":"print('8 arbitrary columns of first 5 records in the training dataset:')\nprint(train[['label', 'pixel0', 'pixel234', 'pixel75', 'pixel387', 'pixel412', 'pixel111', 'pixel782', 'pixel623']].head())\nprint('8 arbitrary columns of first 5 records in the testing dataset:')\nprint(test[['pixel0', 'pixel234', 'pixel75', 'pixel387', 'pixel412', 'pixel111', 'pixel782', 'pixel623']].head())","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"ba5e94e3-9e85-419c-9ade-17bb86e3c5a9","_uuid":"824cc9522fac9d07f30d3697c9f6f92678e6b1f2"},"cell_type":"markdown","source":"Let us look at the values of these columns with the statistical summaries to understand more."},{"metadata":{"_cell_guid":"a2d84529-8ec3-41ff-ab36-7b41e60df9a7","_uuid":"811ad01165137e9a812039a84096695898ebdf61","trusted":true},"cell_type":"code","source":"print('Training dataset - describe:')\nprint(train[['label', 'pixel0', 'pixel234', 'pixel75', 'pixel387', 'pixel412', 'pixel111', 'pixel782', 'pixel623']].describe())\nprint('\\nTesting dataset - describe:')\nprint(test[['pixel0', 'pixel234', 'pixel75', 'pixel387', 'pixel412', 'pixel111', 'pixel782', 'pixel623']].describe())","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"ecc8e920-f845-4a49-a1d0-5960218dec61","_uuid":"892c7b3c8d236875e54e90d69353c6f77f343e1f"},"cell_type":"markdown","source":"The label column ranges from 0 to 9. But it doesn't give us clue whether all digits are represented in the training dataset. We need to analyze that. The pixel values range from 0 to 255. There are some pixels that have only 0 values which could be the case depending on whether that pixel stores any handwriting on that pixel. A couple of things that we need to look out for \n1. Are there any NaN values in the datasets\n1. Are all the digits represented in the training dataset"},{"metadata":{"_cell_guid":"44037f4f-ddba-4f3b-9065-b5216ceaeb46","_uuid":"5305a09c37666250543ee48372f096e383242ca1","trusted":true},"cell_type":"code","source":"# Are there any NaNs?\nprint('No. of null values in the training dataset: %d' % np.sum(np.sum(train.isnull())))\nprint('No. of null values in the testing dataset: %d' % np.sum(np.sum(test.isnull())))","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"5061441d-eb6a-46d2-b8f2-283f013105ab","_uuid":"b1c53d4655b32be07f8b267d2229159a11165b68","trusted":true},"cell_type":"code","source":"sns.countplot(train['label'])\nplt.show()","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"83da41c9-3416-4c92-add0-172d5d9dddc4","_uuid":"bfe116a8b28fc82f09a45405eb95af53b1411017"},"cell_type":"markdown","source":"So good news. No NaNs and all digits are represented fairly in the training dataset."},{"metadata":{"_cell_guid":"93042a35-dfc7-4ff9-8fe2-be68037e2821","_uuid":"8ede9c8c4843c5a857851a8c9d56819d0e185122"},"cell_type":"markdown","source":"## So what the heck is pixels?\nA good exercise is always to read descriptions in detail before you jump into the problem. Below are some details about the data provided as part of the competition.\n\n***Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.***\n\n***The training data set, (train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.***\n\n***Each pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).***\n\nShould have read this initially. Anyways, lets reshape some of these images to 28 x 28 and visualize it using pyplot.imshow() and see if it relates to the label. There is no reason why it shouldn't. :)"},{"metadata":{"_cell_guid":"7767f5a3-7a09-4cdb-be80-42b32cbe956f","_uuid":"42be1dd86349a8260973e38048cdbedaa629522a","trusted":true},"cell_type":"code","source":"# pick 8 random digits in training dataset\nnp.random.seed(12345)\nidx = np.random.randint(0, len(train), 8)\n\nfor n, i in enumerate(idx):\n    ax = plt.subplot(2, 4, n+1)\n    tmp = train.iloc[i]\n    lbl = tmp['label']\n    tmp = tmp.drop(['label'])\n    tmp = tmp.values.reshape(28, 28)\n    ax.imshow(tmp, cmap='gray', interpolation='gaussian')\n    ax.set_title('Label: %d' % lbl)\n    ax.axis('off')\n\nplt.tight_layout()\nplt.show()","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"52aff794-3765-4c96-8b16-509d21ae21d6","_uuid":"cc14ff9fc54ccd9649299cce1965ee0199e4ebbb"},"cell_type":"markdown","source":"That's cool. We picked up 8 random digits from the training dataset. We then reshaped the digits to 28 x 28 matrix and plotted them with the titles as their corresponding labels. And all in all it matches perfectly fine. Let's do the same for the test dataset and see how it goes. Offcourse in this case we do not have the labels."},{"metadata":{"_cell_guid":"130d8f8a-4556-42d0-95aa-c5e4a657dca1","_uuid":"46c03106d362545c9eb80d558ecf9e0205cc73b0","trusted":true},"cell_type":"code","source":"# pick 4 random digits in testing dataset\nnp.random.seed(12345)\nidx = np.random.randint(0, len(test), 4)\n\nfor n, i in enumerate(idx):\n    ax = plt.subplot(2, 2, n+1)\n    tmp = test.iloc[i]\n    tmp = tmp.values.reshape(28, 28)\n    ax.imshow(tmp, cmap='gray', interpolation='gaussian')\n    ax.set_title('Label: ???')\n    ax.axis('off')\n\nplt.tight_layout()\nplt.show()","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"e2a0f1ff-111a-4ba8-be5d-67c615f49630","_uuid":"f1baf71d602842a2e9aa7434eee956b3698b810b","collapsed":true},"cell_type":"markdown","source":"So that's 0, 2, 2 and 3. But our job is not plot it out and see the values and then predict. But hope the above illustrated how the pixels are reshaped and makes sense when plotted."},{"metadata":{"_cell_guid":"9ae008a8-b3ef-469e-960a-a0dd8f9822f9","_uuid":"2e68e065bbb8ddc4f395ec8ef1d6ce6d01f96d64"},"cell_type":"markdown","source":"## Let's do some learning now\nI am assuming at this point we do not need any data cleansing or feature engineering.\n\n*\"Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work. Feature engineering is fundamental to the application of machine learning, and is both difficult and expensive. The need for manual feature engineering can be obviated by automated feature learning.\"* - https://en.wikipedia.org/wiki/Feature_engineering\n\nSo let's jump straight onto training a model using knn. We shall look are hyperparameter tuning using GridSearchCV."},{"metadata":{"_cell_guid":"ecc6e456-ef2e-4f42-add8-25f3dc49a181","_uuid":"a46f471ad81989f01d1ef71c78406d0cb37bc3bd","trusted":true},"cell_type":"code","source":"# import necessary modules\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.model_selection import train_test_split\n\n# split the training dataset to check for model performance\nX = train.drop(['label'], axis=1)\ny = train['label']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=12345)\n\nknn = KNeighborsClassifier(n_neighbors=5)\n_ = knn.fit(X_train, y_train)\nprint('The accurancy of the model: %f' % knn.score(X_test, y_test))","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"dfe5710d9c9eb3855cc16d98fa0a0ab8f76d416f"},"cell_type":"markdown","source":"The model is built and the score is a decent one for the first pass of ML.\n\n## Submission file\nThe submission file should be a csv file with the below format. \n\n|ImageId|Label|\n|------|------|\n|1|3|\n|2|7|\n|3|8|\n\nSo we need to run the prediction on the dataset provided and save it as csv file. Let us it as submission.csv for submission."},{"metadata":{"_cell_guid":"07099b48-3bed-41b1-8c5d-78b2bdca0bd9","_uuid":"7c333fb84fbcaed8c258237cf609734c4d188fc5","collapsed":true,"trusted":true},"cell_type":"code","source":"# make prediction on the set provided\nsubmitLabel = knn.predict(test)\nsubmitLabelDF = pd.DataFrame({'ImageId': range(1, len(submitLabel) + 1), 'Label': submitLabel})\n\n# Let us print the first few lines of the submission file\nprint(submitLabelDF.head())","execution_count":26,"outputs":[]},{"metadata":{"_cell_guid":"411dfb69-2bb8-474a-9d30-3f60ade138d3","_uuid":"48ecb01f76a8cbd75be918eb38a741fe95e5bce5","trusted":true},"cell_type":"code","source":"# write the submission file\nsubmitLabelDF.to_csv('submission.csv', index=False)","execution_count":29,"outputs":[]},{"metadata":{"_uuid":"acb36bead2634160f0dcb1fd78de1be2cb08eafa"},"cell_type":"markdown","source":"## Conclusion\nHope you enjoyed the kernel and was useful. Please post your comment for any feedbacks or questions. It will be great to know your feedback. Also please upvote if you liked the kernel."}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}