{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1d11b71f-50b4-333e-a4f9-df2fb192a40a"
      },
      "source": [
        "# Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1e0a81f5-8486-c1ca-f5c5-bc723111da99"
      },
      "source": [
        "The goal in this competition is to take an image of a handwritten single digit, and determine what that digit is.  As the competition progresses, we will release tutorials which explain different machine learning algorithms and help you to get started.\n",
        "\n",
        "\n",
        "The data for this competition were taken from the MNIST dataset. The MNIST (\"Modified National Institute of Standards and Technology\") dataset is a classic within the Machine Learning community that has been extensively studied.  More detail about the dataset, including Machine Learning algorithms that have been tried on it and their levels of success, can be found [here][1].\n",
        "\n",
        "\n",
        "  [1]: http://yann.lecun.com/exdb/mnist/index.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c7313b40-f504-5443-a468-008df68f59bb"
      },
      "source": [
        "# Loading the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ca99747a-0397-57e6-7129-375b1a05bd37"
      },
      "outputs": [],
      "source": [
        "import numpy as np # Array manipulation\n",
        "import pandas as pd # Dataframe manipulation\n",
        "\n",
        "# Multilayer perceptron Neural Network\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Dropout\n",
        "from keras.utils import np_utils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fa85554d-254d-fa59-4d9b-1499d77ef248"
      },
      "outputs": [],
      "source": [
        "# Load data\n",
        "train = pd.read_csv('../input/train.csv')\n",
        "test = pd.read_csv('../input/test.csv')\n",
        "\n",
        "# fix random seed for reproducibility\n",
        "seed = 7\n",
        "np.random.seed(seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7af5f2b3-0855-5978-7d04-4da5974cf9d2"
      },
      "source": [
        "Extract the features matrix X and transform it to an array of float numbers. And also extract the labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5f0ab393-676b-74e8-a898-c3b32954b5c8"
      },
      "outputs": [],
      "source": [
        "# Extract images pixels\n",
        "images = train.iloc[:,1:].values\n",
        "images = images.astype(np.float)\n",
        "\n",
        "# Extract numbers Labels\n",
        "labels = train.iloc[:,0].values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "64ce8b1d-bc53-14ea-bd99-e9dc559c3283"
      },
      "source": [
        "# Multilayer Perceptron"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "888840e0-cddb-2173-13f4-5f2b07c4b296"
      },
      "source": [
        "## Preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7f7709ce-9b5f-f45d-c583-1a49f7f733df"
      },
      "source": [
        "The pixel values are gray scale between 0 and 255. It is almost always a good idea to perform some scaling of input values when using neural network models. Because the scale is well known and well behaved, we can very quickly **normalize** the pixel values to the range 0 and 1 by dividing each value by the maximum of 255.\n",
        "\n",
        "Also, the output variable is an integer from 0 to 9. This is a multi-class classification problem. As such, it is good practice to use a **one hot encoding** of the class values, transforming the vector of class integers into a binary matrix. We can easily do this using the built-in np_utils.to_categorical() helper function in Keras."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1dcb5c59-bd89-9fbc-e3e4-275ba0030cf0"
      },
      "outputs": [],
      "source": [
        "# Normalize input from 0-255 to 0-1\n",
        "images = images / 255.0\n",
        "num_pixels =  images.shape[1]\n",
        "\n",
        "# one hot encode outputs\n",
        "labels = np_utils.to_categorical(labels)\n",
        "num_classes = labels.shape[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2570ea91-a644-a6f0-d943-f9e6d6b66783"
      },
      "source": [
        "We are now ready to create our simple neural network model. We will define our model in a function. This is handy if you want to extend the example later and try and get a better score.\n",
        "\n",
        "The model is a **simple neural network** with **one hidden layer** with the same **number of neurons as there are inputs (784)**. A **rectifier activation function** is used for the neurons in the hidden layer.\n",
        "\n",
        "A **softmax activation function** is used on the output layer to turn the outputs into probability-like values and allow one class of the 10 to be selected as the model\u2019s output prediction. **Logarithmic loss** is used as the loss function (called **categorical_crossentropy** in Keras) and the efficient **ADAM gradient descent algorithm** is used to **learn the weights**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5e27495d-5a10-f00c-2e73-25507f928d6b"
      },
      "source": [
        "## Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "31980f83-aaf8-a086-9494-de28ea2a9e54"
      },
      "outputs": [],
      "source": [
        "# define baseline model\n",
        "def mlp_model():\n",
        "\t# create model\n",
        "\tmodel = Sequential()\n",
        "\tmodel.add(Dense(num_pixels, input_dim=num_pixels, init='normal', activation='relu'))\n",
        "\tmodel.add(Dense(num_classes, init='normal', activation='softmax'))\n",
        "\t# Compile model\n",
        "\tmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "\treturn model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7c0ebc00-beb4-e159-3657-3b84579a8a1e"
      },
      "source": [
        "We can now fit and evaluate the model. The model is fit **over 10 epochs with updates every 200 images**. A verbose value of 2 is used to reduce the output to one line for each training epoch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6961e1b4-bbac-c951-d3d4-9a36684d6d6a"
      },
      "outputs": [],
      "source": [
        "# build the model\n",
        "model = mlp_model()\n",
        "# Fit the model\n",
        "model.fit(images, labels, nb_epoch=10, batch_size=200, verbose=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7ea4fac4-5b4e-427c-cd7c-255c87f128f4"
      },
      "source": [
        "Finally, we predict the model, we change our one hot encoded (binary matrix) results to a vector of labels from 0 to 9, and we save our results in a submission file"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1a9e8575-f459-a7e2-2a0d-0cfe7998ea48"
      },
      "source": [
        "## Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "44e2295b-00dd-6439-b1b2-a393c35ecf97"
      },
      "outputs": [],
      "source": [
        "# use the NN model to classify test data\n",
        "pred = model.predict(test.values)\n",
        "pred = pred.argmax(1) # transform the binary matrix to an array of 0 to 9 labels\n",
        "# save results\n",
        "np.savetxt('submission_MLP.csv', np.c_[range(1,len(test)+1),pred], delimiter=',', header = 'ImageId,Label', comments = '', fmt='%d')"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}