{"cells":[{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d8dc8b6f-4313-41af-9bc8-2e291e31f1e2","collapsed":true},"outputs":[],"source":"%matplotlib inline"},{"cell_type":"markdown","metadata":{"_cell_guid":"d3abbd27-73b9-4941-9799-c4d01711e27b"},"source":"# Digit Recognizer in Python using Convolutional Neural Nets\nby [Koba Khitalishvili](http://www.kobakhit.com/)\n## Introduction\nI used Mathematica and a standard Neural Network model to get ~0.98 accuracy score after 40 minutes of computing. Starting with a brief benchmark Python code [found in the forums](https://www.kaggle.com/c/digit-recognizer/forums/t/2299/getting-started-python-sample-code-random-forest) one can jump into solving the Digit Recognizer\nproblem right away. Below code that uses Random Forest algorithm to classify images as digits will give you \naround 0.96 accuracy score in less than a minute which is great. However, this score will put you lower than the 200th place. \nAccording to [MNIST web page](http://yann.lecun.com/exdb/mnist/), convolutional neural networks algorithm yields good results.\nI will try a simple neural network algorithm out in Python, expand it into a convolutional neural network and see if I can break into the top 100."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c6fb9907-885b-404f-83bf-3df455efa9f9"},"outputs":[],"source":"from sklearn.ensemble import RandomForestClassifier\nimport numpy as np\nimport pandas as pd\n\n# create the training & test sets, skipping the header row with [1:]\ndataset = pd.read_csv(\"../input/train.csv\")\ntarget = dataset[[0]].values.ravel()\ntrain = dataset.iloc[:,1:].values\ntest = pd.read_csv(\"../input/test.csv\").values\n\n# create and train the random forest\n# multi-core CPUs can use: rf = RandomForestClassifier(n_estimators=100, n_jobs=2)\nrf = RandomForestClassifier(n_estimators=100)\nrf.fit(train, target)\npred = rf.predict(test)\n\nnp.savetxt('submission_rand_forest.csv', np.c_[range(1,len(test)+1),pred], delimiter=',', header = 'ImageId,Label', comments = '', fmt='%d')"},{"cell_type":"markdown","metadata":{"_cell_guid":"83df8b0c-2ecf-459f-ac41-145a713dd18c"},"source":"## Table of contents\n1. Data Preprocessing\n2. Train, Predict and Save\n3. Conclusion\n\n## Data Preprocessing\nFirst, lets prepare the data. The `train.csv` has 42k rows. The first column is the digit labels. The rest 784 columns\nare pixel color values that go from 0 to 255. After loading in the csv files in code section above, I saved the digit labels in the `target` variable and rows of pixel color values  in the `train` variable. \nThe `test.csv` contains 28k rows of just the pixel color values which we need to classify as digits. Here is the preview of the complete MNIST dataset."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1cf115c6-1184-4fbb-8b32-3afbfe85589a"},"outputs":[],"source":"print(dataset.head())"},{"cell_type":"markdown","metadata":{"_cell_guid":"d55ce476-4451-45bf-a1dd-336fe921a4b1"},"source":"I convert each data variable from a python list into a numpy array. For the `target` array I specify the integer data type.\nThe train set has 42k rows and 784 columns, so its shape is `(42k,784)`. Each row is a 28 by 28 pixel\npicture. I will reshape the train set to have `(42k,1)` shape, i.e. each row will contain a 28 by 28 matrix of pixel color values. Same for the test set."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4ccd377f-e9fa-4af9-978c-97721b638cbc"},"outputs":[],"source":"# convert to array, specify data type, and reshape\ntarget = target.astype(np.uint8)\ntrain = np.array(train).reshape((-1, 1, 28, 28)).astype(np.uint8)\ntest = np.array(test).reshape((-1, 1, 28, 28)).astype(np.uint8)"},{"cell_type":"markdown","metadata":{"_cell_guid":"04c5d36b-8a77-4bc3-8c30-6add80d3e74e"},"source":"Now, we can actually plot those pixel color values and see what a sample picture of a digit looks like. Below is the picture of a digit in the 1729th row:"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a96b3557-2537-43d4-b206-c39758ca2ee3"},"outputs":[],"source":"import matplotlib.pyplot as plt\nimport matplotlib.cm as cm\n\nplt.imshow(train[1729][0], cmap=cm.binary) # draw the picture"},{"cell_type":"markdown","metadata":{"_cell_guid":"9ec8132e-2f2c-4ffd-bfe8-f16245067641"},"source":"## Train, Predict and Save\nBelow is a simple NN set up. Supposedly, it should give >0.9 accuracy score. I had trouble with figuring out\nthe training part in that the accuracy I was getting would not change during the training process. All I had to \ndo was to decrease the learning rate from 0.01 to 0.0001, nota bene."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"5487db80-6ead-4924-af4a-70cf5482a94d"},"outputs":[],"source":"import lasagne\nfrom lasagne import layers\nfrom lasagne.updates import nesterov_momentum\nfrom nolearn.lasagne import NeuralNet\nfrom nolearn.lasagne import visualize\n\nnet1 = NeuralNet(\n        layers=[('input', layers.InputLayer),\n                ('hidden', layers.DenseLayer),\n                ('output', layers.DenseLayer),\n                ],\n        # layer parameters:\n        input_shape=(None,1,28,28),\n        hidden_num_units=1000, # number of units in 'hidden' layer\n        output_nonlinearity=lasagne.nonlinearities.softmax,\n        output_num_units=10,  # 10 target values for the digits 0, 1, 2, ..., 9\n\n        # optimization method:\n        update=nesterov_momentum,\n        update_learning_rate=0.0001,\n        update_momentum=0.9,\n\n        max_epochs=15,\n        verbose=1,\n        )"},{"cell_type":"markdown","metadata":{"_cell_guid":"75911524-6af4-4c97-9502-2cff7df40f32"},"source":"Now, lets train the model. "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"7154bd1b-2af7-4959-8383-2b0d827419e4"},"outputs":[],"source":"# Train the network\nnet1.fit(train, target)"},{"cell_type":"markdown","metadata":{"_cell_guid":"35d35a0d-43f4-4864-a393-83746b942dcd"},"source":"You can see the output associated with the training process. Right off the bat this set up gives us ~0.95 accuracy\nscore in just 15 epochs which completes in less 3 minuts. Unfortunately, one layer neural network does not improve \nbeyond 0.96 accuracy score ragardless of how many neurons in a layer is specified (1000 in case above).\n\nI will try out the convolutional neural network. To set up the CNN I added two convolutional layers and one pooling layer. \nI would add another pooling layer and a dropout layer, but training such a model would last for over 20 minutes and kaggle notebook is only allowed to run for 1200 seconds (20 minutes). \nAs a result, below is the CNN model I will use for demonstration."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a4888868-fc52-4fcf-98bf-3120162505d1"},"outputs":[],"source":"def CNN(n_epochs):\n    net1 = NeuralNet(\n        layers=[\n        ('input', layers.InputLayer),\n        ('conv1', layers.Conv2DLayer),      #Convolutional layer.  Params defined below\n        ('pool1', layers.MaxPool2DLayer),   # Like downsampling, for execution speed\n        ('conv2', layers.Conv2DLayer),\n        ('hidden3', layers.DenseLayer),\n        ('output', layers.DenseLayer),\n        ],\n\n    input_shape=(None, 1, 28, 28),\n    conv1_num_filters=7, \n    conv1_filter_size=(3, 3), \n    conv1_nonlinearity=lasagne.nonlinearities.rectify,\n        \n    pool1_pool_size=(2, 2),\n        \n    conv2_num_filters=12, \n    conv2_filter_size=(2, 2),    \n    conv2_nonlinearity=lasagne.nonlinearities.rectify,\n        \n    hidden3_num_units=1000,\n    output_num_units=10, \n    output_nonlinearity=lasagne.nonlinearities.softmax,\n\n    update_learning_rate=0.0001,\n    update_momentum=0.9,\n\n    max_epochs=n_epochs,\n    verbose=1,\n    )\n    return net1\n\ncnn = CNN(15).fit(train,target) # train the CNN model for 15 epochs\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"7a3225b3-9f25-4394-bc68-ea0afde9b734"},"source":"Looks like this CNN model produces a slightly better result than a simple NN model for the same number of epochs and neurons in the hidden layer. \nThe CNN model was more time consuming though. If you think that it is not worth it to use the CNN model over the NN you are wrong.\nNN model like any other has an upper bound on the best accuracy score it can produce. After 20 epochs \nNN model does not improve beyond ~0.97 whereas a CNN model gets closer to one. I was able to break into\ntop 100 and get the 94th place by using two convolutional layers and two pooling layers which gave me 0.98614 accuracy\nin about 5 hours. I will provide the code in the post scriptum section.\n\nSo there you go. You have a starting point for using neural nets for image classification. \nIf you expand on the info here and reach a score greater than 0.99 please drop a comment.\nNow, lets use it on the test set and save the results.\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e44eb7bf-2eaf-432b-a9c6-92213399a2db"},"outputs":[],"source":"# use the NN model to classify test data\npred = cnn.predict(test)\n\n# save results\nnp.savetxt('submission_cnn.csv', np.c_[range(1,len(test)+1),pred], delimiter=',', header = 'ImageId,Label', comments = '', fmt='%d')\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"ceda803c-ba94-43fb-a8f1-37788818b8e4"},"source":"## Conclusion\nPython really stands out when it comes to solving data problems. Its quick, intuitive, well documented and has a big community.\nHowever, its biggest drawback in my opinion is setting up the environment. My set up is [Jupyter Notebook](http://jupyter.org/) coupled \nwith [Anaconda](https://www.continuum.io/downloads). Both are great tools, howeverm I ended up spending \ncouple hours taking care of the dependencies theano and lasagne and the Windows environment variables. In contrast to Mathematica which \nhas superb report generating options, setting up Python environment can be and was for me a tiring experience. Nevertheless, it pays off because \nPython framework is well developed for solving data problems. For instance, Mathematica does not even have a CNN\nimplementation available as of 11/4/2015 and everything is done under the hood whereas in a Python framework one \ncan find almost any algorithm imaginable.\n\nAmong Python, R and Julia I beleive Python and R are most competitive data science technologies with Julia being \nin the process of maturing. Choosing Python over R and vica versa really has to do with either individual preference or\nthe suitability of the technology for the problem at hand. Python is more efficient than R. But Julia is more \nefficient than both Python and R.\n"},{"cell_type":"markdown","metadata":{"_cell_guid":"ea45af9e-1126-4166-b74d-db8e09012794"},"source":"### Resources Used:\n[Convolutional Neural Networks (LeNet)](http://deeplearning.net/tutorial/lenet.html)\n\n[CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.github.io/convolutional-networks/)\n\n[Tutorial: Training convolutional neural networks with nolearn](http://nbviewer.ipython.org/github/dnouri/nolearn/blob/master/docs/notebooks/CNN_tutorial.ipynb)\n\n[Using convolutional neural nets to detect facial keypoints tutorial](http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/)\n\n[Deep learning – Convolutional neural networks and feature extraction with Python](http://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/)"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}