{"nbformat_minor": 0, "cells": [{"cell_type": "markdown", "source": "

In this simple kernel, I'll fit a neural network with a single hidden layer by using the function \"nnet\" provided by nnet package and I'll revise some of the options that it has to offer. The neural network will be tested with the \"Digit recognizer\" dataset.

\n

Let's start by opening the train dataset and preparing it for the model:

", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "95074421-6c6e-4279-8260-c763c13916e8", "collapsed": false, "_uuid": "81227c1fbee2c3ce684054accff94c01e2708cd0"}}, {"cell_type": "code", "source": "library(data.table)\ndata <- fread(\"../input/train.csv\")\ndim(data)\nnames(data)[1:10]", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "0b6c0c3c-0f8d-4a19-9717-f6366c94726c", "collapsed": false, "_uuid": "216ca21bd03dabf57cf321b4b6b1346b0302b6d9"}}, {"cell_type": "markdown", "source": "

We have a dataset with 42.000 records and 785 features, the first one being the \"label\" and the other 784 containing the values for the 28x28 matrix that defines every digit.

", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "7488991b-11de-415e-b36c-73ae858dbd7a", "collapsed": false, "_uuid": "e48997c7bbcd53378c71d8908fd9b2f1430ec043"}}, {"cell_type": "code", "source": "plotDigit <- function(X){\n m <- matrix(unlist(X),nrow = 28,byrow = T)\n m <- t(apply(m, 2, rev))\n image(m, col=grey.colors(255))\n}\nplotDigit(data[4, -1])", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "1488d8b0-ad30-46b2-99c7-190a99b411cc", "collapsed": false, "_uuid": "d24056ed7b1fd1c7b2ccf0e91c97e1ea87f7f259"}}, {"cell_type": "markdown", "source": "# Remove near zero value features\n\n

If we take a look a the dataset, we can see that many features are zero or near-zero variance:

\n
\n

[Near-zero variance are those for which the] fraction of unique values over the sample size is low (say 10%) [...] [and the] ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20). If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model.

\n

Kuhn, M., & Johnson, K. (2013). Applied predictive modeling, New York, NY: Springer.

\n
\n

Though the removal of zero and near-zero variance predictors might not always be the best solution, let's do it here in order to reduce the time it takes for the script to be executed, and let's chose a low cutoff for the ratio of the most common value to the second most common value, so as to end up with a low number of features:

", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "8763da3b-c29e-416c-8e3c-733df6a5a955", "collapsed": false, "_uuid": "628f9c471bccdc49b0ccca3c961fa9fae34a5f07"}}, {"cell_type": "code", "source": "library(caret)\nnzv <- nearZeroVar(data, freqCut = 10)\nlength(nzv)", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "a08e2738-7a80-4aaa-864d-0f69b288a7aa", "collapsed": false, "_uuid": "ee905942b5232d3e18e50ad212605118be3859fa"}}, {"cell_type": "markdown", "source": "

The function has found 581 with zero or near-zero predictors (let's remember we're using an unusually low cutoff). Let's remove them from our dataset and normalize it by dividing the structure by 255 (pixel values vary between 0 and 255). This way all entries to the neural network will have a value between 0 and 1:

\n\n# Normalization", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "b2f96171-37fe-4b60-b468-0a0fd043a19f", "collapsed": false, "_uuid": "ce92bc6a1f23b56036fbc504b433bed360a279a7"}}, {"cell_type": "code", "source": "data_norm <- as.matrix(data) # So as to remove columns by name\ndata_norm <- data_norm[, -nzv] / 255 # We normalize the data\ndim(data_norm)", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "46366177-f6c7-4655-b653-15dd580ae738", "collapsed": false, "_uuid": "102a0362a1eceb59158810510a0087f56420e25e"}}, {"cell_type": "markdown", "source": "

Our dataset has been reduced to 204 features. Now we split the dataset in two blocks: one to train the model and the second one for its evaluation. We'll do it by using the sample function, selecting 75% of our original dataset for training and the rest for validation:

\n\n# Training and validation datasets", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "5311b241-505a-460c-9a24-99a79a772696", "collapsed": false, "_uuid": "3fbc98dba2f1e5e3640f10561e31a0351059302d"}}, {"cell_type": "code", "source": "train_recs <- sample(1:nrow(data_norm), round(0.75 * nrow(data_norm)))\ntrain_norm <- data_norm[train_recs, ]\nvalidation_norm <- data_norm[-train_recs, ]\ndim(train_norm)\ndim(validation_norm)", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "89af8c24-29eb-4a99-b4d5-9d8de5c4cdf3", "collapsed": false, "_uuid": "1b9731c0c0dfe2f9d43da947a8337eb06a0a7f8f"}}, {"cell_type": "markdown", "source": "

We have 31.500 records for training and other 10.500 for validation.

\n\n# Training the model\n\n

As we'll use the \"nnet\" function provided by nnet package, let's now take a look at its documentation:

\n
\nFit single-hidden-layer neural network, possibly with skip-layer connections.\n
\n

This function has two different methods:

\n\n
\n

S3 method for class 'formula'

\n

nnet(formula, data, weights, ..., subset, na.action, contrasts = NULL)

\n\n

Default S3 method:

\n

nnet(x, y, weights, size, Wts, mask, linout = FALSE, entropy = FALSE, softmax = FALSE, censored = FALSE, skip = FALSE, rang = 0.7, decay = 0, maxit = 100, Hess = FALSE, trace = TRUE, MaxNWts = 1000, abstol = 1.0e-4, reltol = 1.0e-8, ...)

\n
\n\n

The first method has a formula and a dataset (matrix or data frame) as inputs, while the second one has matrices or data frames with the X values and target values as inputs. I'll use the second one.

\n

The second argument (y) needs to be a matrix or data frame of target values for examples. This means a structure of ten columns (one for each possible target) filled with zeros except for the right column for every row, that needs to contain a \"1\". I'll use the class.ind provided by nnet for this.

\n

Some other basic though interesting arguments are:

\n\n

Let's then create a model with 10 units in the hidden layer:

", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "f6ea2e2e-0b83-4c24-ab8b-a0a2c9003d46", "collapsed": false, "_uuid": "a2b7bbbe7253a5cf1634fd614b859a998328754c"}}, {"cell_type": "code", "source": "library(nnet)\nX <- train_norm[, -1]\nY <- train_norm[, 1]\ndim(X)\nlength(Y)\ny <- class.ind(unlist(Y))\nnn <- nnet(X, y, size=10, softmax=TRUE, maxit=130, MaxNWts =80000)", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "ddeafaaf-8e00-4c3d-af28-4d9c94dcb0ec", "collapsed": false, "_uuid": "c21e450a585f6ec510d61fc64e56db51a01d71b1"}}, {"cell_type": "markdown", "source": "# Validation\n\n

We can now estimate the accuracy of our model by applying the predict function to our validation dataset. For this purpose I create a \"results\" data frame, with the labels and predictions, we compare them and return the number of differences:

", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "13edbf6d-b9d1-4ca8-93f4-2efcd127000e", "collapsed": false, "_uuid": "cf47d1182ae6659577ea81fad3c38e89cca0d53e"}}, {"cell_type": "code", "source": "prediction <- predict(nn, validation_norm, type=\"class\")\nresults <- data.frame(cbind(label = as.data.frame(validation_norm)$label, prediction = prediction))\nhits <- sum(as.vector(results$label) == as.vector(results$prediction))\n(accuracy <- hits * 100 / nrow(validation_norm))", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "trusted": false, "_cell_guid": "64462efe-5457-4380-a265-c081da5a8468", "collapsed": false, "_uuid": "3e3016ffdb8969bd191d376c2a11af1cadaed37f"}}, {"cell_type": "markdown", "source": "

The accurary is quite low, though, as it was the number of features considered and the number of units in the hidden layer.

", "outputs": [], "execution_count": null, "metadata": {"_execution_state": "idle", "_cell_guid": "2c6c7611-6d08-4802-ada7-500eff0d2775", "collapsed": false, "_uuid": "ddd4f25b62a6f52f0fd7acd1bfda92513ced4e8d"}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "R", "name": "ir", "language": "R"}, "language_info": {"name": "R", "version": "3.4.1", "codemirror_mode": "r", "mimetype": "text/x-r-source", "file_extension": ".r", "pygments_lexer": "r"}}}