{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bb9d6f84-2458-0bcc-cde4-50b8e09212e6"
      },
      "source": [
        "# Goal\n",
        "\n",
        "I try to train a SVM on the MNIST dataset reducing the dimensions with PCA to a number that I can manage in some minutes with my notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f27c11b2-5a51-b4a9-db94-9aa4451f32f2"
      },
      "outputs": [],
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "\n",
        "import seaborn as sns\n",
        "import matplotlib.image as mpimg\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib\n",
        "%matplotlib inline\n",
        "\n",
        "from time import time\n",
        "\n",
        "from sklearn.manifold import TSNE\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.svm import SVC\n",
        "\n",
        "import math"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d73612d1-17a8-3669-9087-28c9c8e1c18b"
      },
      "source": [
        "# Some helper functions first \n",
        "\n",
        "I prepared a couple of functions to see the pictures before and after the dimensionality reduction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "43e619b7-6558-885a-a6c9-b1756bf25319"
      },
      "outputs": [],
      "source": [
        "# Plots the image represented by a row\n",
        "def plot_number(row, w=28, h=28, labels=True):\n",
        "    if labels:\n",
        "        # the first column contains the label\n",
        "        label = row[0]\n",
        "        # The rest of columns are pixels\n",
        "        pixels = row[1:]\n",
        "    else:\n",
        "        label = ''\n",
        "        # The rest of columns are pixels\n",
        "        pixels = row[0:]\n",
        "    \n",
        "#    print(row.shape, pixels.shape)\n",
        "        \n",
        "\n",
        "    # Make those columns into a array of 8-bits pixels\n",
        "    # This array will be of 1D with length 784\n",
        "    # The pixel intensity values are integers from 0 to 255\n",
        "    pixels = 255-np.array(pixels, dtype='uint8')\n",
        "\n",
        "    # Reshape the array into 28 x 28 array (2-dimensional array)\n",
        "    pixels = pixels.reshape((w, h))\n",
        "\n",
        "    # Plot\n",
        "    if labels:\n",
        "        plt.title('Label is {label}'.format(label=label))\n",
        "    plt.imshow(pixels, cmap='gray')\n",
        "\n",
        "# Plots a whole slice of pictures\n",
        "def plot_slice(rows, size_w=28, size_h=28, labels=True):\n",
        "    num = rows.shape[0]\n",
        "    w = 4\n",
        "    h = math.ceil(num / w)\n",
        "    fig, plots = plt.subplots(h, w)\n",
        "    fig.tight_layout()\n",
        "\n",
        "    for n in range(0, num):\n",
        "        s = plt.subplot(h, w, n+1)\n",
        "        s.set_xticks(())\n",
        "        s.set_yticks(())\n",
        "        plot_number(rows.ix[n], size_w, size_h, labels)\n",
        "    plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "93c2a603-2549-ee69-0e8f-61c438532647"
      },
      "source": [
        "### Loading the training set\n",
        "Let's load the dataset and see some basic info."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7cf05f91-746a-6112-41a8-7fc6dba7fa31"
      },
      "outputs": [],
      "source": [
        "train = pd.read_csv('../input/train.csv')\n",
        "#train = pd.read_csv('train.csv')\n",
        "print(train.shape)\n",
        "train.sample(5)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "43a69e95-2e0e-4bae-29b6-a4c059047df8"
      },
      "outputs": [],
      "source": [
        "train.describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0b2113c5-60fb-01bd-1a61-5a24a89c7c1d"
      },
      "outputs": [],
      "source": [
        "train.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6213007c-5646-06ca-ae06-7ebd87c707fa"
      },
      "source": [
        "### Visualizing some rows\n",
        "\n",
        "Plotting a slice to see correspondence between labels and pictures."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e1657f83-dbe9-0395-5e96-e883c719d82e"
      },
      "outputs": [],
      "source": [
        "# Plotting first few rows\n",
        "plot_slice(train[0:12])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1e9fd815-83dd-4cca-b163-3432fdd9bd80"
      },
      "source": [
        "## Dimensionality reduction\n",
        "\n",
        "I try to see if a PCA can help to reduce the number of features without affecting too much information. First of all I split the whole dataset in features and class, then I prepare the split of both sets to training and test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8289d032-d043-541b-e0bc-8f17bb1a5f84"
      },
      "outputs": [],
      "source": [
        "X_train = train.drop(['label'], axis='columns', inplace=False)\n",
        "y_train = train['label']\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "X_tr, X_ts, y_tr, y_ts = train_test_split(X_train, y_train, test_size=0.30, random_state=4)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a8a7deb0-1e72-46fd-6943-6ba5a5292212"
      },
      "source": [
        "The `n_components` variable here is crucial. With a value of 16 the model can be trained in seconds and still achieve a `0.97229` score in the Leaderboard. I use a perfect square value because I want to see how it looks the reduced-dimension image.\n",
        "\n",
        "I've seen that using 49 components resulted in a smaller score, maybe that means that the final model is more over-fitted than the 16 components one."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "90322e39-736d-52fe-52dc-8945e2390c32"
      },
      "outputs": [],
      "source": [
        "n_components = 16\n",
        "t0 = time()\n",
        "pca = PCA(n_components=n_components, svd_solver='randomized',\n",
        "          whiten=True).fit(X_train)\n",
        "print(\"done in %0.3fs\" % (time() - t0))\n",
        "\n",
        "X_train_pca = pca.transform(X_train)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c991d53d-54e7-47d1-21f5-821faccacae5"
      },
      "source": [
        "Let's see the variance histogram."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a78b604e-a14f-a931-754c-4473ff2fcab7"
      },
      "outputs": [],
      "source": [
        "plt.hist(pca.explained_variance_ratio_, bins=n_components, log=True)\n",
        "pca.explained_variance_ratio_.sum()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "55a7fca2-2114-f710-17c6-bf4cac5584c1"
      },
      "source": [
        "Even though 16 components collect 59% of variance and 49 components get 82%, the former performs better at submission."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "03c21243-e112-ab71-be39-ba13503b6d69"
      },
      "source": [
        "Let's see how the smaller pictures look."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c50af376-879a-efbc-9522-c7db85d08f8b"
      },
      "outputs": [],
      "source": [
        "plot_slice(pd.DataFrame(data=X_train_pca[0:12]), size_w=4, size_h=4, labels=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7e787beb-9d61-4055-c4dc-f29435c9f267"
      },
      "source": [
        "Hope someone could explain these spots :D"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bba18b90-e721-355a-cd3a-c631914cebc2"
      },
      "source": [
        "I've removed the real grid search parameters here because it takes too long to make it online"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4ea943ed-c9b8-296b-c9ff-6eebd243f0c7"
      },
      "outputs": [],
      "source": [
        "param_grid = { \"C\" : [0.1]\n",
        "              , \"gamma\" : [0.1]}\n",
        "rf = SVC()\n",
        "gs = GridSearchCV(estimator=rf, param_grid=param_grid, scoring='accuracy', cv=2, n_jobs=-1, verbose=1)\n",
        "gs = gs.fit(X_train_pca, y_train)\n",
        "\n",
        "print(gs.best_score_)\n",
        "print(gs.best_params_)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "48af6341-1d64-0d95-2bea-5e9b38b53fe9"
      },
      "outputs": [],
      "source": [
        "bp = gs.best_params_\n",
        "\n",
        "t0 = time()\n",
        "clf = SVC(C=bp['C'], kernel='rbf', gamma=bp['gamma'])\n",
        "clf = clf.fit(X_train_pca, y_train)\n",
        "print(\"done in %0.3fs\" % (time() - t0))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d344e58a-72bd-8f24-41d3-d47ce0ea9e52"
      },
      "outputs": [],
      "source": [
        "clf.score(pca.transform(X_ts), y_ts)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0971f04d-218e-8d68-53d7-6939794a82eb"
      },
      "source": [
        "Nice score with a little effort."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a55c86e5-180c-2a6f-8cb8-7a964353fac5"
      },
      "outputs": [],
      "source": [
        "#val = pd.read_csv('test.csv')\n",
        "val = pd.read_csv('../input/test.csv')\n",
        "pred = clf.predict(pca.transform(val))\n",
        "# ImageId,Label\n",
        "\n",
        "val['Label'] = pd.Series(pred)\n",
        "val['ImageId'] = val.index +1\n",
        "sub = val[['ImageId','Label']]\n",
        "\n",
        "sub.to_csv('submission.csv', index=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4b1afe57-d1dc-4e62-a698-018770e921fa"
      },
      "source": [
        "## Conclusions\n",
        "\n",
        "It's been a very interesting experiment on how to reduce the features number to train the model in a decent time and still keep a goode score. It's good to see that I could reduce the number of features from __784 to 16__ with a __97%__ of overall score.\n",
        "\n",
        "A next step could be using a Neural Network to see if there are improvements in the training time or score.\n",
        "\n",
        "Any comment is appreciated!"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}