{"nbformat": 4, "nbformat_minor": 1, "cells": [{"cell_type": "markdown", "source": ["## A basic classifier on MNIST using XgBoost, no pruning, only regularisation :) "], "metadata": {"_cell_guid": "af9a48aa-99d5-4e42-b664-9123d0e4e5d2", "_uuid": "633c20f21aacb4cea652e4319c0ec6d93b116d6e"}}, {"cell_type": "code", "source": ["# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import xgboost as xgb\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "train_df = pd.read_csv(\"../input/train.csv\")\n", "test_df = pd.read_csv(\"../input/test.csv\")\n", "print (train_df.shape, test_df.shape)\n", "# Any results you write to the current directory are saved as output."], "outputs": [], "execution_count": null, "metadata": {"_cell_guid": "9d1d0aa3-14a5-4f45-a38b-78c94eed525e", "collapsed": true, "_uuid": "903296cb580a815ac5a9defc29381f74af1671c6"}}, {"cell_type": "markdown", "source": ["## Standardise features (pixel values) using StandardScaler with mean 0 and var 1.0"], "metadata": {"_cell_guid": "005d8de7-f71e-4d12-8be8-7d5f70788d2c", "_uuid": "76d09d48835ee19e07215bf7acf79937b86789f4"}}, {"cell_type": "code", "source": ["from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "\n", "sc = StandardScaler()\n", "X_std = sc.fit_transform(train_df.values[:, 1:])\n", "y = train_df.values[:, 0]\n", "\n", "test_std = sc.fit_transform(test_df.values)\n", "\n", "print (X_std.shape, y.shape)\n", "print (test_std.shape)"], "outputs": [], "execution_count": null, "metadata": {"_cell_guid": "2297bca9-6cef-4e8f-b350-35fac8d474a8", "collapsed": true, "_uuid": "24a47ad6d785aa85f6fc212a19c34b2295fc0fee"}}, {"cell_type": "markdown", "source": ["## Split the train set into train and validation set using train_test_split"], "metadata": {"_cell_guid": "0a4d14e4-30d2-4388-b0d8-67510e7de965", "_uuid": "9a610e06569ea9379544c68f3eafe381fd1bda28"}}, {"cell_type": "code", "source": ["X_train, X_valid, y_train, y_valid = train_test_split(X_std, y, test_size=0.1)\n", "print (X_train.shape, y_train.shape)\n", "print (X_valid.shape, y_valid.shape)"], "outputs": [], "execution_count": null, "metadata": {"_cell_guid": "e03922a5-a565-4b9b-82ff-9d2abd54e9af", "collapsed": true, "_uuid": "bb61f0b89a7bd74eb91dc414fda297345b972329"}}, {"cell_type": "markdown", "source": ["## Define all the parameters that will be used by XgBoost classifier, num_rounds it will run and early_stopping if any. Note that I have not used pruning as it was reducing tree sooner. The best validation accuracy is 97%"], "metadata": {"_cell_guid": "b6b899da-bc6d-4c69-be04-18ffa070e873", "_uuid": "b7b13b7725ef436a1d90f4e0db85876b95a309c1"}}, {"cell_type": "code", "source": ["param_list = [(\"eta\", 0.08), (\"max_depth\", 6), (\"subsample\", 0.8), (\"colsample_bytree\", 0.8), (\"objective\", \"multi:softmax\"), (\"eval_metric\", \"merror\"), (\"alpha\", 8), (\"lambda\", 2), (\"num_class\", 10)]\n", "n_rounds = 600\n", "early_stopping = 50\n", "    \n", "d_train = xgb.DMatrix(X_train, label=y_train)\n", "d_val = xgb.DMatrix(X_valid, label=y_valid)\n", "eval_list = [(d_train, \"train\"), (d_val, \"validation\")]\n", "bst = xgb.train(param_list, d_train, n_rounds, evals=eval_list, early_stopping_rounds=early_stopping, verbose_eval=True)"], "outputs": [], "execution_count": null, "metadata": {"_cell_guid": "0622a58e-74d5-4c8f-90d9-ecac5f27aaf1", "collapsed": true, "_uuid": "5959cf55440b2e0cb228f5ae1be12c0989e05659"}}, {"cell_type": "markdown", "source": ["## Time to make predictions on the test_set"], "metadata": {"_cell_guid": "e558793d-f1e8-4fa8-9865-b275d1b6e979", "_uuid": "9a60cb1f196fb52c00b2e30316003bc11d0468b8"}}, {"cell_type": "code", "source": ["d_test = xgb.DMatrix(data=test_std)\n", "y_pred = bst.predict(d_test)"], "outputs": [], "execution_count": null, "metadata": {"_cell_guid": "e93bd006-0ef8-4936-b36d-a4c8215e75c7", "collapsed": true, "_uuid": "d33792fc993b9d65d7b3edf1bf3d3aa3e4f66bd8"}}, {"cell_type": "markdown", "source": ["## The above code gives a prediction accuracy of 90% on test-set. The split of original MNIST dataset produced an accuracy of 96% using exact same parameter values. But the values here can also be played with further to improve the test accuracy. Cheers."], "metadata": {"_cell_guid": "916822cd-feff-4a4e-8b30-f302c83a9818", "collapsed": true, "_uuid": "45cbed1444b5db46ced05e1c7b0f48dddb8fe2d3"}}], "metadata": {"language_info": {"name": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "version": "3.6.3", "file_extension": ".py", "pygments_lexer": "ipython3", "mimetype": "text/x-python", "nbconvert_exporter": "python"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}}