{"cells":[{"metadata":{"_uuid":"220032b3497ea93938f371de2fa3a5c2dcd21ad4"},"cell_type":"markdown","source":"# Handwritten Digit Recognition\n\nIn this tutorial, we'll give you a step by step walk-through of how to build a hand-written digit classifier using the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset. For someone new to deep learning, this exercise is arguably the \"Hello World\" equivalent.\n\nMNIST is a widely used dataset for the hand-written digit classification task. It consists of 70,000 labeled 28x28 pixel grayscale images of hand-written digits. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits). The task at hand is to train a model using the 60,000 training images and subsequently test its classification accuracy on the 10,000 test images.\n\n![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/example/mnist.png)\n\n**Figure 1:** Sample images from the MNIST dataset."},{"metadata":{"trusted":true,"_uuid":"57b004ad65b463f0961f934b07519d49158fd0d0"},"cell_type":"code","source":"import mxnet as mx\nimport pandas as pd\nimport numpy as np\nimport logging\nfrom sklearn.model_selection import train_test_split\n\nlogging.getLogger().setLevel(logging.DEBUG)  # logging to stdout\npath = '../input/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21eec690f9264a80260a0c84affabceb4ee7c111"},"cell_type":"code","source":"# Fix the seed\nmx.random.seed(7)\n\n# Set the compute context, GPU is available otherwise CPU\nctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9167a54f87a9dc95eb815655dde0ad3ac93b86f"},"cell_type":"code","source":"print(\"model extract train init\")\ndf_train = pd.read_csv(path + 'train.csv')\ny = (np.array(df_train['label'].values.tolist()).astype(np.int)).copy()\ndf_train = df_train.drop(columns=['label'])\nX = (np.array(df_train.values.tolist()).astype(np.float)).copy()\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\nprint(\"model extract train end\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24d3302fe9da64139ce272c8adaf861469dc0b87"},"cell_type":"code","source":"print(\"model extract test init\")\ndf_test_final = pd.read_csv(path + 'test.csv')\nX_test_final = np.array(df_test_final.values.tolist()).astype(np.float)\nprint(\"model extract train end\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def reshare_array(array, dim):\n    return np.reshape(array, (-1, 1, dim, dim))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = reshare_array(X_train, 28)\nX_test = reshare_array(X_test, 28)\nX_test_final = reshare_array(X_test_final, 28)\ny_train = np.array(y_train)\ny_test = np.array(y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bb6bbaccc0c42e290557e9ff9dd5f813ff9f43d1"},"cell_type":"code","source":"batch_size = 100\ntrain_iter = mx.io.NDArrayIter(X_train, y_train, batch_size)\nval_iter = mx.io.NDArrayIter(X_test, y_test, batch_size)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42aa7754a0ac4067d255675ff67bba1d718e9a27"},"cell_type":"markdown","source":"### Convolutional Neural Network\n\nEarlier, we briefly touched on a drawback of MLP when we said we need to discard the input image's original shape and flatten it as a vector before we can feed it as input to the MLP's first fully connected layer. Turns out this is an important issue because we don't take advantage of the fact that pixels in the image have natural spatial correlation along the horizontal and vertical axes. A convolutional neural network (CNN) aims to address this problem by using a more structured weight representation. Instead of flattening the image and doing a simple matrix-matrix multiplication, it employs one or more convolutional layers that each performs a 2-D convolution on the input image.\n\nA single convolution layer consists of one or more filters that each play the role of a feature detector. During training, a CNN learns appropriate representations (parameters) for these filters. Similar to MLP, the output from the convolutional layer is transformed by applying a non-linearity. Besides the convolutional layer, another key aspect of a CNN is the pooling layer. A pooling layer serves to make the CNN translation invariant: a digit remains the same even when it is shifted left/right/up/down by a few pixels. A pooling layer reduces a *n x m* patch into a single value to make the network less sensitive to the spatial location. Pooling layer is always included after each conv (+ activation) layer in the CNN.\n\nThe following source code defines a convolutional neural network architecture called LeNet. LeNet is a popular network known to work well on digit classification tasks. We will use a slightly different version from the original LeNet implementation, replacing the sigmoid activations with tanh activations for the neurons"},{"metadata":{"trusted":true,"_uuid":"6389295e5d9ef182b18fc55b5d488a1dbf2834af"},"cell_type":"code","source":"data = mx.sym.var('data')\n# first conv layer\nconv1 = mx.sym.Convolution(data=data, kernel=(5,5), num_filter=20)\ntanh1 = mx.sym.Activation(data=conv1, act_type=\"tanh\")\npool1 = mx.sym.Pooling(data=tanh1, pool_type=\"max\", kernel=(2,2), stride=(2,2))\n# second conv layer\nconv2 = mx.sym.Convolution(data=pool1, kernel=(5,5), num_filter=50)\ntanh2 = mx.sym.Activation(data=conv2, act_type=\"tanh\")\npool2 = mx.sym.Pooling(data=tanh2, pool_type=\"max\", kernel=(2,2), stride=(2,2))\n# first fullc layer\nflatten = mx.sym.flatten(data=pool2)\nfc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=500)\ntanh3 = mx.sym.Activation(data=fc1, act_type=\"tanh\")\n# second fullc\nfc2 = mx.sym.FullyConnected(data=tanh3, num_hidden=10)\n# softmax loss\nlenet = mx.sym.SoftmaxOutput(data=fc2, name='softmax')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc712f78a138976fa29377d86e69456d60583b30"},"cell_type":"markdown","source":"![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/conv_mnist.png)\n\n**Figure 3:** First conv + pooling layer in LeNet.\n\nNow we train LeNet with the same hyper-parameters as before. Note that, if a GPU is available, we recommend using it. This greatly speeds up computation given that LeNet is more complex and compute-intensive than the previous multilayer perceptron. To do so, we only need to change `mx.cpu()` to `mx.gpu()` and MXNet takes care of the rest. Just like before, we'll stop training after 10 epochs."},{"metadata":{"trusted":true,"_uuid":"4aead5e1780998b969768fd7bb29b847535e4777"},"cell_type":"code","source":"lenet_model = mx.mod.Module(symbol=lenet, context=ctx)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21bf81321f2228920764f480be366dcffb7a0eb5"},"cell_type":"code","source":"# train with the same\nlenet_model.fit(train_iter,\n                eval_data=val_iter,\n                optimizer='sgd',\n                optimizer_params={'learning_rate':0.005},\n                eval_metric='acc',\n                batch_end_callback = mx.callback.Speedometer(batch_size, 100),\n                num_epoch=150)","execution_count":null,"outputs":[{"output_type":"stream","text":"WARNING:root:Already bound, ignoring bind()\n/opt/conda/lib/python3.6/site-packages/mxnet/module/base_module.py:503: UserWarning: Parameters already initialized and force_init=False. init_params call ignored.\n  allow_missing=allow_missing, force_init=force_init)\nWARNING:root:optimizer already initialized, ignoring...\n","name":"stderr"}]},{"metadata":{"_uuid":"d2d6f3a58b7e988ea9c96f49632b4d2cb74c9159"},"cell_type":"markdown","source":"### Prediction\n\nFinally, we'll use the trained LeNet model to generate predictions for the test data."},{"metadata":{"trusted":true,"_uuid":"79a0c78db5c4635268f747cc15730423eac46260"},"cell_type":"code","source":"test_iter = mx.io.NDArrayIter(X_test_final, None, batch_size)\nprob = lenet_model.predict(test_iter)\n\ny = []\nfor p in list(prob):\n   y.append(list(p).index(np.max(p)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_iter = mx.io.NDArrayIter(X_test, y_test, batch_size)\n# predict accuracy of mlp\nacc = mx.metric.Accuracy()\nlenet_model.score(test_iter, acc)\nprint(acc)\nassert acc.get()[1] > 0.96, \"Achieved accuracy (%f) is lower than expected (0.96)\" % acc.get()[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame({'ImageId': [x for x in range(1, len(y) + 1)], 'Label': y})\ndf.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cea6b6938934f5a7a7a3126c03ba001593d856af"},"cell_type":"markdown","source":"If all went well, we should see a higher accuracy metric for predictions made using LeNet. With CNN we should be able to correctly predict around 98% of all test images.\n\n## Summary\n\nIn this tutorial, we have learned how to use MXNet, based on [MXNet Examples](https://mxnet.incubator.apache.org/versions/master/tutorials/python/mnist.html)."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"display_name":"","name":"","language":"python","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}