{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "b18a50d5-b0b3-cfae-aca1-76f9e1b3c54a" }, "source": [ "Introduction\n", "------------\n", "I wanted to compare the prediction accuracies using h2o's deep learning algorithm by varying some of its parameters. To begin with, I compared a range of epochs from 15 to 300 for the train data set and also the binarized train data set derived from https://www.kaggle.com/davidprakash/digit-recognizer/simplifying-the-data-by-binarizing-the-values\n", "The base R script used for the two different sets of train data was *(need to ensure that h2o is started via java -jar h2o.jar prior to running the codes below):*\n", "\n", "\n", "\n", " library(\"h2o\")\n", " localH2O = h2o.init(max_mem_size = '6g',nthreads = -1)\n", " train <- read.csv(\"../input/train.csv\")\n", " train[,1] = as.factor(train[,1]) # convert digit labels to factor for classification\n", " train_h2o = as.h2o(train)\n", "\n", " time=c()\n", " acc=c()\n", " for(i in seq(15,300,5))\n", " {\n", " ## set timer\n", " s <- proc.time()\n", " \n", " ## train model\n", " model =\n", " h2o.deeplearning(x = 2:785, # column numbers for predictors\n", " y = 1, # column number for label\n", " training_frame = train_h2o, # data in H2O format\n", " activation = \"RectifierWithDropout\", # algorithm\n", " input_dropout_ratio = 0.2, # % of inputs dropout\n", " hidden_dropout_ratios = c(0.5,0.5), # % for nodes dropout\n", " balance_classes = TRUE, \n", " hidden = c(100,100), # two layers of 100 nodes\n", " momentum_stable = 0.99,\n", " nesterov_accelerated_gradient = T, # use it for speed\n", " epochs = i) # no. of epochs\n", " ## print confusion matrix\n", " x=h2o.confusionMatrix(model)\n", " print(i)\n", " print(x)\n", " time[length(time)+1]=as.numeric(proc.time()[3]-s[3])\n", " print(time[length(time)])\n", " acc[length(acc)+1]=1-x$Error[dim(x)[1]]\n", " print(acc[length(acc)])\n", " }\n", "\n", "The accuracies plot against number of epochs used is manually created below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3f2d1baf-17bf-ff16-a01c-087abe64cf1b" }, "outputs": [], "source": [ "accuracy_train=c(0.953922161726155,0.96492114840275,0.964855909227834,0.966420147911253,0.968960321446509,0.969243221367867,0.972138705151333,0.969633403256418,0.972428105544026,0.971890719005819,0.972069825436409,0.975233737815795,0.976387366743051,0.975731548986318,0.9736529318542,0.974630443467839,0.97815631262525,0.971529530540131,0.97723884305735,0.972470681773007,0.972913762198765,0.972310126582278,0.979360063203634,0.97353291038858,0.974381977671451,0.975357071011869,0.976653696498054,0.97499495866102,0.978795085216013,0.9762165389591,0.979331003494758,0.97646240937531,0.976422193237684,0.977274972828772,0.973772763262074,0.972338470960498,0.978382706164932,0.975248020843772,0.975594997983058,0.974818353737434,0.974459527400704,0.977654186115801,0.977800304105423,0.974445659739485,0.975173783515392,0.97583081570997,0.975014991005397,0.979795938983736,0.974868792894631,0.974779942636732,0.974486180014174,0.975309866137829,0.97870425914817,0.977723774807614,0.974604743083004,0.977685212734305,0.976022986228079,0.977159385597447)\n", "accuracy_binarized_train=c(0.957388939256573,0.957340276386942,0.963513109360981,0.965179652805814,0.966963660026029,0.968250798722045,0.969615192403798,0.968007948335817,0.968910871101242,0.970803662258393,0.968935496636328,0.974020823004462,0.971422861710632,0.975711311589174,0.971214392803598,0.974404585433343,0.97444279346211,0.972116729962023,0.974522292993631,0.97292472478429,0.9735,0.974493396511745,0.977297729772977,0.977304964539007,0.974932044699487,0.975573130443488,0.977539457709429,0.97423461655047,0.974150070408369,0.977389207114863,0.979322000198827,0.976588961944611,0.976653306613226,0.9746573316241,0.975943302056299,0.97379475895179,0.974133626285828,0.975069527214938,0.974483717235902,0.977640247554402,0.976486998107004,0.977965260545906,0.975710127471645,0.975936095856216,0.977359629704166,0.975360158966716,0.976416045152187,0.977186690575812,0.978102189781022,0.977038235889136,0.979036264282166,0.976102389761024,0.979210605604098,0.97437715211667,0.976472933904895,0.978230337078652,0.975,0.974820143884892)\n", "plot(seq(15,300,5),accuracy_train,col=\"blue\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "lines(seq(15,300,5),accuracy_train,col=\"blue\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "points(seq(15,300,5),accuracy_binarized_train,col=\"red\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "lines(seq(15,300,5),accuracy_binarized_train,col=\"red\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "\n", " legend(\"bottom\", legend = c(\"original train data\",\"binarized train data\"), col = c(\"blue\",\"red\"),\n", " ncol = 1, cex = 1, text.col = c(\"blue\",\"red\"), bty=\"n\", lwd = 3 )" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c8650dc6-7236-4ec2-403c-9fa996f98f53" }, "source": [ "The graph above shows that the binarized train data achieves better accuracy at 15 epochs but after 100 epochs, both datasets seem to plateau to an accuracy close to 0.979. Perhaps, the number of nodes in the hidden layers could be increased to further improve on accuracy. Since the number of variables in the dataset was 784, we tried to **increase the number of nodes in each hidden layer to 500 and 700** to compare the accuracies.\n", "\n", "The base R script used for both of the train datasets for 500 nodes per hidden layer is shown below. The code for using 700 nodes per hidden layer requires only changing line \"hidden = c(500,500),\" to \"hidden = c(700,700),\". *(need to ensure that h2o is started via java -jar h2o.jar prior to running the codes below):*\n", "\n", "\n", "\n", " library(\"h2o\")\n", " localH2O = h2o.init(max_mem_size = '6g',nthreads = -1)\n", " train <- read.csv(\"../input/train.csv\")\n", " train[,1] = as.factor(train[,1]) # convert digit labels to factor for classification\n", " train_h2o = as.h2o(train)\n", "\n", " time=c()\n", " acc=c()\n", " for(i in seq(15,25,5))\n", " {\n", " ## set timer\n", " s <- proc.time()\n", " \n", " ## train model\n", " model =\n", " h2o.deeplearning(x = 2:785, # column numbers for predictors\n", " y = 1, # column number for label\n", " training_frame = train_h2o, # data in H2O format\n", " activation = \"RectifierWithDropout\", # algorithm\n", " input_dropout_ratio = 0.2, # % of inputs dropout\n", " hidden_dropout_ratios = c(0.5,0.5), # % for nodes dropout\n", " balance_classes = TRUE, \n", " hidden = c(500,500),# two layers of 500 nodes\n", " momentum_stable = 0.99,\n", " nesterov_accelerated_gradient = T, # use it for speed\n", " epochs = i) # no. of epochs\n", " ## print confusion matrix\n", " x=h2o.confusionMatrix(model)\n", " print(i)\n", " print(x)\n", " time[length(time)+1]=as.numeric(proc.time()[3]-s[3])\n", " print(time[length(time)])\n", " acc[length(acc)+1]=1-x$Error[dim(x)[1]]\n", " print(acc[length(acc)])\n", " }\n", "\n", "The results of this evaluation is manually plot below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ba77dce1-6419-4f47-69f2-1a0fb53a3c65" }, "outputs": [], "source": [ "accuracy_binarized_train_15to25ep_700n=c(0.98942896660739,0.992475992475992,0.993840651698788)\n", "accuracy_train_15to25ep_700n=c(0.990341602277349,0.989817103311913,0.992404294105732)\n", "accuracy_binarized_train_15to25ep_500n=c(0.987717195925704,0.989770582977456,0.990738876585464)\n", "accuracy_train_15to25ep_500n=c(0.986090111883883,0.988831874434048,0.991534707698436)\n", "\n", "plot(seq(15,25,5),accuracy_train_15to25ep_700n,col=\"blue\",xlab=\"Epochs\",ylab=\"Accuracy\",ylim=c(0.98,1))\n", "lines(seq(15,25,5),accuracy_train_15to25ep_700n,col=\"blue\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "points(seq(15,25,5),accuracy_binarized_train_15to25ep_700n,col=\"red\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "lines(seq(15,25,5),accuracy_binarized_train_15to25ep_700n,col=\"red\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "\n", "points(seq(15,25,5),accuracy_train_15to25ep_500n,col=\"green\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "lines(seq(15,25,5),accuracy_train_15to25ep_500n,col=\"green\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "points(seq(15,25,5),accuracy_binarized_train_15to25ep_500n,col=\"purple\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "lines(seq(15,25,5),accuracy_binarized_train_15to25ep_500n,col=\"purple\",xlab=\"Epochs\",ylab=\"Accuracy\")\n", "\n", " legend(\"bottom\", legend = c(\"original train data (700 nodes per hidden layer)\"\n", " ,\"binarized train data (700 nodes per hidden layer)\",\n", " \"original train data (500 nodes per hidden layer)\"\n", " ,\"binarized train data (500 nodes per hidden layer)\"), col = c(\"blue\",\"red\",\"green\",\"purple\"),\n", " ncol = 1, cex = 1, lwd = 3, text.col = c(\"blue\",\"red\",\"green\",\"purple\"), bty=\"n\")" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "84f60532-d32e-d401-c49c-1d590282c179" }, "source": [ "Closing Remarks\n", "----------\n", "The 700 nodes per hidden layer for an epoch of 25 hit an accuracy of 0.9938, the highest accuracy so far. Increasing the number of nodes for the hidden layer did produce a more significant improvement in prediction accuracy compared to increasing the number of epochs. I believe there is much more that can be performed to further increase the accuracy such as the use of ensemble learning. *(h2o currently does support ensemble learning for multi class datasets.)*\n" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "3.3.2" } }, "nbformat": 4, "nbformat_minor": 0 }