{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b6a03ea0-0790-bb3e-1b98-67de61de04b8"
      },
      "source": [
        "#Neural Network Classification Model for Handwritten Digit Recognition \n",
        "I am using this data to work an example of neural network classification. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1d09a305-ae97-cedc-ab9e-ffd895b00195"
      },
      "outputs": [],
      "source": [
        "# load modules\n",
        "import numpy as np \n",
        "import pandas as pd \n",
        "np.random.seed(1)\n",
        "# get data\n",
        "test  = pd.read_csv('../input/test.csv')\n",
        "train = pd.read_csv('../input/train.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0ade9a88-758a-1d76-cc36-1b798ce25e59"
      },
      "source": [
        "## Exploratory Analysis\n",
        "Summary statistics and visualizations of data. The data is first preprocessed by visualizing the sample size for each digit in the dataset, plotting a few of the digits using the data provided to get an understanding of exactly what the data represents, normalizing the data, and reducing the features using PCA. It is important to note that this dataset is sparse, meaning that there are mostly 0's in the feature matrix. Some pixels carry a lot of information about the digit written, while other pixel features such as the edges and usually 0 and not very informative. The scale of the data used to represent the pixels is not numerically meaningful, which leads to the need to normalize the data so that the values do not contribute to the model in an improper way. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "61fe6cdc-4d25-87e7-2ad2-105ffac58d1c"
      },
      "outputs": [],
      "source": [
        "print(train.shape)\n",
        "print(test.shape)\n",
        "train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "121eaf21-72a7-95b5-fa50-709289f0c606"
      },
      "source": [
        "This simple histogram shows the count of digits in the training data for each number. This graphic is used to visualize if there is an unequal sample size among the digits. The sample size for each digit appear to be comparable. There is no issue of unequal sampling. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ed57e3c7-4a4b-9eae-03b5-f769271b5e98"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.hist(train[\"label\"])\n",
        "plt.title(\"Frequency Histogram of Numbers in Training Data\")\n",
        "plt.xlabel(\"Number Value\")\n",
        "plt.ylabel(\"Frequency\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c799fb06-3456-e8c3-116d-07e7dc24b9ad"
      },
      "source": [
        "This graphic plots the first few digits in the training set to show how this pixel data is representing digits, and to also show how the handwriting varies. Not all digits are written the same. For example, there is a lot of variation in how people write 4s and 9s. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "783f23f6-3bf8-9bfa-919c-4878314ad88e"
      },
      "outputs": [],
      "source": [
        "import math\n",
        "# plot the first 25 digits in the training set. \n",
        "f, ax = plt.subplots(5, 5)\n",
        "# plot some 4s as an example\n",
        "for i in range(1,26):\n",
        "    # Create a 1024x1024x3 array of 8 bit unsigned integers\n",
        "    data = train.iloc[i,1:785].values #this is the first number\n",
        "    nrows, ncols = 28, 28\n",
        "    grid = data.reshape((nrows, ncols))\n",
        "    n=math.ceil(i/5)-1\n",
        "    m=[0,1,2,3,4]*5\n",
        "    ax[m[i-1], n].imshow(grid)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d3441104-ec16-4309-04ec-40ac63f9b821"
      },
      "source": [
        "# PCA\n",
        "There are many features in this data resulting in high dimensionality. PCA is used to compress the features into a small but informative set of features before using the data in a machine learning model. Data is normalized before PCA is applied. This is so the scale of the data does not throw of the PCA, and so the 0's are represented meaningfully.  There is unequal variance in this data, and features with larger variance will influence the PCA more, creating bias. This is why the data is normalized. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "48e61f68-21e2-d101-d299-701433beab97"
      },
      "outputs": [],
      "source": [
        "## normalize data ##\n",
        "label_train=train['label']\n",
        "train=train.drop('label', axis=1)\n",
        "\n",
        "#normalize data\n",
        "train = train / 255\n",
        "test = test / 255\n",
        "train['label'] = label_train"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "527791e7-3a05-09a2-970a-51dc3dfd3b5e"
      },
      "outputs": [],
      "source": [
        "from sklearn import decomposition\n",
        "from sklearn import datasets\n",
        "\n",
        "## PCA decomposition\n",
        "pca = decomposition.PCA(n_components=200) #Finds first 200 PCs\n",
        "pca.fit(train.drop('label', axis=1))\n",
        "plt.plot(pca.explained_variance_ratio_)\n",
        "plt.ylabel('% of variance explained')\n",
        "#plot reaches asymptote at around 50, which is optimal number of PCs to use. \n",
        "\n",
        "## PCA decomposition with optimal number of PCs\n",
        "#decompose train data\n",
        "pca = decomposition.PCA(n_components=50) #use first 3 PCs (update to 100 later)\n",
        "pca.fit(train.drop('label', axis=1))\n",
        "PCtrain = pd.DataFrame(pca.transform(train.drop('label', axis=1)))\n",
        "PCtrain['label'] = train['label']\n",
        "\n",
        "#decompose test data\n",
        "#pca.fit(test)\n",
        "PCtest = pd.DataFrame(pca.transform(test))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "15f0b27d-a2e7-f836-f613-64c997145bb1"
      },
      "source": [
        "This plot shows the separation of classes (digits) based on the first PCAs. Theoretically, these PCs should explain most of the variance in the data, enough to show separation in the groups of digits."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2476c607-38f5-cd63-6169-063aef8626a8"
      },
      "outputs": [],
      "source": [
        "from mpl_toolkits.mplot3d import Axes3D\n",
        "\n",
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(111, projection='3d')\n",
        "#ax = fig.add_subplot(111)\n",
        "\n",
        "x =PCtrain[0]\n",
        "y =PCtrain[1]\n",
        "z =PCtrain[2]\n",
        "\n",
        "colors = [int(i % 9) for i in PCtrain['label']]\n",
        "ax.scatter(x, y, z, c=colors, marker='o', label=colors)\n",
        "\n",
        "ax.set_xlabel('PC1')\n",
        "ax.set_ylabel('PC2')\n",
        "ax.set_zlabel('PC3')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cab42f00-3379-e7e5-2163-f6944ad24378"
      },
      "source": [
        "#Neural Network"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "89fbf7fd-06b8-dc26-1dd4-7025100fbb5b"
      },
      "outputs": [],
      "source": [
        "from sklearn.neural_network import MLPClassifier\n",
        "y = PCtrain['label'][0:20000]\n",
        "X=PCtrain.drop('label', axis=1)[0:20000]\n",
        "clf = MLPClassifier(solver='lbfgs', alpha=1e-5,\n",
        "                    hidden_layer_sizes=(2500,), random_state=1)\n",
        "clf.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "de8a400f-46e5-3d10-593c-0e03b3f15ed1"
      },
      "outputs": [],
      "source": [
        "from sklearn import  metrics\n",
        "#accuracy and confusion matrix\n",
        "predicted = clf.predict(PCtrain.drop('label', axis=1)[20001:420000])\n",
        "expected = PCtrain['label'][20001:42000]\n",
        "\n",
        "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
        "      % (clf, metrics.classification_report(expected, predicted)))\n",
        "print(\"Confusion matrix:\\n%s\" % metrics.confusion_matrix(expected, predicted))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9f081a14-7a1b-c6ae-c85d-31daa26d349f"
      },
      "outputs": [],
      "source": [
        "output = pd.DataFrame(clf.predict(PCtest), columns =['Label'])\n",
        "output.reset_index(inplace=True)\n",
        "output.rename(columns={'index': 'ImageId'}, inplace=True)\n",
        "output['ImageId']=output['ImageId']+1\n",
        "output.to_csv('output.csv', index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "85b71ffc-e756-5ebe-d094-6b928d6150d5"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}