{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n# Introduction\n\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike. \n\nIn this tutorial, your goal is to correctly identify digits from the Kaggle MNIST dataset of tens of thousands of handwritten images. We will walk through the development of several standard deep learning pipelines using PyTorch that are capable of correctly identifying digits from the MNIST dataset. We will then see how to customize standard deep learning pipelines to improve model performance. After successfully training a custom model, we will see how to submit the model's predictions to Kaggle for scoring.\n\nThis tutorial assumes some basic knowledge of neural networks. If you’re not already familiar with neural networks, then you can learn the basics concepts behind neural networks (and PyTorch!) at [course.fast.ai](https://course.fast.ai/). Much of this kernel was derived from [_What is torch.nn really?_](https://pytorch.org/tutorials/beginner/nn_tutorial.html) by Jeremy Howard, Rachel Thomas, Francisco Ingham."},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\nfrom sklearn import decomposition, ensemble, manifold, metrics, model_selection, pipeline, preprocessing\nimport torch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load the MNIST data"},{"metadata":{"trusted":true},"cell_type":"code","source":"!head ../input/train.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mnist_arr = np.loadtxt(\"../input/train.csv\", delimiter=',', skiprows=1, dtype=np.int64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# raw features are between 0 and 255\nmnist_arr.min(), mnist_arr.max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Rescale the raw data\n\nData for individual pixels is stored as integers between 0 and 255. Neural network models work best when numerical features are scaled. To rescale the raw features we can use tools from the [Scikit-Learn preprocessing module](https://scikit-learn.org/stable/modules/preprocessing.html)."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_target, training_features = mnist_arr[:, 0], mnist_arr[:, 1:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"min_max_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))\nscaled_training_features = (min_max_scaler.fit_transform(training_features)\n                                          .astype(np.float32))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, ax = plt.subplots(1,1)\n_ = ax.imshow(scaled_training_features[0].reshape((28, 28)), cmap=\"gray\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing training samples using PCA\n\n[Principal Components Analysis (PCA)](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) can be used as a visualization tool to see if there are any obvious patterns in the training samples."},{"metadata":{"trusted":true},"cell_type":"code","source":"_prng = np.random.RandomState(42)\npca = decomposition.PCA(n_components=2, random_state=_prng)\ntransformed_training_features = pca.fit_transform(scaled_training_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1)\n_ = ax.scatter(transformed_training_features[:,0], transformed_training_features[:,1], c=training_target, alpha=0.05)\nax.set_xlabel(\"Component 1\")\nax.set_ylabel(\"Component 2\")\nax.set_title(\"PCA\", fontsize=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing training sample using t-SNE\n\n[t-distributed Stochastic Neighbor Embedding (t-SNE)](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html#sklearn.manifold.TSNE) is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. For more details on t-SNE including other use cases see this excellent *Toward Data Science* [blog post](https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1)\n\nIt is highly recommended to use another dimensionality reduction method (e.g. PCA) to reduce the number of dimensions to a reasonable amount if the number of features is very high. This will suppress some noise and speed up the computation of pairwise distances between samples."},{"metadata":{"trusted":true},"cell_type":"code","source":"# initial PCA to determine n_components that capture 95% of sample variance\n_prng = np.random.RandomState(42)\npca = decomposition.PCA(random_state=_prng)\npca.fit_transform(scaled_training_features)\nn_components = np.sum(pca.explained_variance_ratio_.cumsum() < 0.95)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As suggested above we will go ahead and use an initial PCA step in our embedding pipeline to reduce the overall number of features which will speed up convergence of the t-SNE algorithm."},{"metadata":{"trusted":true},"cell_type":"code","source":"_prng = np.random.RandomState(42)\n\nembedding_pipeline = pipeline.make_pipeline(\n    decomposition.PCA(n_components, random_state=_prng),\n    manifold.TSNE(n_components=2, random_state=_prng)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"transformed_training_features = embedding_pipeline.fit_transform(scaled_training_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1)\n_ = ax.scatter(transformed_training_features[:,0], transformed_training_features[:,1], c=training_target, alpha=0.05)\nax.set_xlabel(\"Component 1\")\nax.set_ylabel(\"Component 2\")\nax.set_title(\"t-SNE\", fontsize=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Classical ML Benchmark Model\n\nTo provide a point of comparison for our neural network models, let's use PCA to peform dimensionality reduction and then train a Random Forest Classifier."},{"metadata":{},"cell_type":"markdown","source":"## Dimensionality Reduction"},{"metadata":{"trusted":true},"cell_type":"code","source":"# initial PCA to determine n_components that capture 95% of sample variance\n_prng = np.random.RandomState(42)\npca = decomposition.PCA(random_state=_prng)\npca.fit_transform(scaled_training_features)\npca.set_params(n_components=np.sum(pca.explained_variance_ratio_.cumsum() < 0.95))\n\n# second PCA fit to compute the transformed features\ntransformed_training_features = pca.fit_transform(scaled_training_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Classifier Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"# evaluate a reasonable classifier using CV\nclf = ensemble.RandomForestClassifier(n_estimators=100, random_state=_prng)\naccuracy_scores = model_selection.cross_val_score(clf,\n                                                  transformed_training_features,\n                                                  training_target,\n                                                  scoring=\"accuracy\",\n                                                  cv=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy_scores.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# retrain the classifier using the entire dataset\nclf.fit(transformed_training_features, training_target)\n\n# load the testing features (note we use transform method and NOT fit_transform!)\n_testing_features = np.loadtxt(\"../input/test.csv\", delimiter=',', skiprows=1, dtype=np.int64)\n_scaled_testing_features = min_max_scaler.transform(_testing_features)\n_transformed_testing_features = pca.transform(_scaled_testing_features)\n\n# make predictions \nclf_predictions = clf.predict(_transformed_testing_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submit to Kaggle!\n\nIn order to submit our predictions to Kaggle we first need to produce a correctly formatted submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission format for kaggle\n!head ../input/sample_submission.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\nimport pandas as pd\n\ntimestamp = time.strftime(\"%Y%m%d-%H%M%S\")\nnumber_predictions, = clf_predictions.shape\ndf = pd.DataFrame({\"ImageId\": range(1, number_predictions + 1), \"Label\": clf_predictions})\ndf.to_csv(f\"random-forest-submission-{timestamp}.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once you have successfully submited your predictions then you can check the [Digit-Recognizer competition](https://www.kaggle.com/c/digit-recognizer) website and see how well your best model compares to your peers."},{"metadata":{},"cell_type":"markdown","source":"# Neural network from scratch\n\n## Split the MNIST data into training and validation sets\n\nSince Kaggle has already split the MNIST data set into training and testing data sets, we only need to split our training data set into training and validation data. We will use the validation data to make sure that we are not over-fitting our models."},{"metadata":{"trusted":true},"cell_type":"code","source":"_prng = np.random.RandomState(42)\nscaled_training_features, scaled_validation_features, training_target, validation_target = (\n    model_selection.train_test_split(scaled_training_features,\n                                     training_target,\n                                     test_size=0.10,\n                                     random_state=_prng)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scaled_training_features.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scaled_validation_features.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nNext let's create a simple model using nothing but [PyTorch tensor operations](https://pytorch.org/docs/stable/tensors.html). PyTorch uses `torch.tensor` rather than `numpy.ndarray` so we need to convert data."},{"metadata":{"trusted":true},"cell_type":"code","source":"training_target_tensor = torch.from_numpy(training_target)\nscaled_training_features_tensor = torch.from_numpy(scaled_training_features)\n\nvalidation_target_tensor = torch.from_numpy(validation_target)\nscaled_validation_features_tensor = torch.from_numpy(scaled_validation_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scaled_training_features_tensor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_target_tensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"PyTorch provides methods to create random or zero-filled tensors, which we will use to create our weights and bias for a simple linear model. These are just regular tensors, with one very special addition: we tell PyTorch that they require a gradient. This causes PyTorch to record all of the operations done on the tensor, so that it can calculate the gradient during back-propagation automatically!\n\nFor the weights, we set `requires_grad` after the initialization, since we don’t want that step included in the gradient. (Note that a trailling `_` in PyTorch signifies that the operation is performed _in-place_.)"},{"metadata":{"trusted":true},"cell_type":"code","source":"number_samples, number_features = scaled_training_features_tensor.shape\n\n# using Xavier initialization (divide weights by sqrt(number_features))\nweights = torch.randn(number_features, 10) / number_features**0.5\nweights.requires_grad_() # trailing underscore indicates in-place operation\nbias = torch.zeros(10, requires_grad=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks to PyTorch’s ability to calculate gradients automatically, we can use any standard Python function (or callable object) in a model! So we will start by writing a function to peform matrix multiplication and broadcasted addition called `linear_transformation`. We will also need an activation function, so we’ll write a function called `log_softmax_activation` and use it. \n\n**N.B.** Although PyTorch provides lots of pre-written loss functions, activation functions, and so forth, you can easily write your own using plain python. PyTorch will even create fast GPU or vectorized CPU code for your function automatically."},{"metadata":{"trusted":true},"cell_type":"code","source":"def linear_transformation(X):\n    return X @ weights + bias\n\ndef log_softmax_activation(X):\n    return X - X.exp().sum(-1).log().unsqueeze(-1)\n    \ndef logistic_regression(X):\n    Z = linear_transformation(X)\n    return log_softmax_activation(Z)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the above, the `@` stands for the dot product operation. We will call our function on one batch of data (in this case, 64 images). Note that our predictions won’t be any better than random at this stage, since we start with random weights."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 64\noutput = logistic_regression(scaled_training_features_tensor[:batch_size])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you see, the `output` tensor contains not only the tensor values, but also a gradient function, `grad_fn`. We’ll use this later to do back propagation to update the model parameters.\n\nLet’s implement `negative_log_likelihood` to use as the loss function. Again, we can just use standard Python code."},{"metadata":{"trusted":true},"cell_type":"code","source":"def negative_log_likelihood(output, target):\n    m, _ = output.shape\n    return -output[range(m), target].mean()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"negative_log_likelihood(output, training_target_tensor[:batch_size])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let’s also implement a function to calculate the `accuracy` of our model: for each prediction, if the index with the largest value matches the target value, then the prediction was correct."},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy(output, target):\n    predictions = torch.argmax(output, dim=1)\n    return (predictions == target).float().mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For comparison purposes we can compute the accuracy of our model with randomly initialized parameters."},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy(output, training_target_tensor[:batch_size])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now run a training loop. For each iteration, we will:\n\n* select a mini-batch of data (of size `batch_size`)\n* use the model to make predictions\n* calculate the loss\n* `loss.backward()` updates the gradients of the model.\n\nWe now use these gradients to update the weights and bias (i.e., model parameters). We do this within the `torch.no_grad()` context manager, because we do not want these actions to be recorded for our next calculation of the gradient. You can read more about how PyTorch’s Autograd records operations [here](https://pytorch.org/docs/stable/notes/autograd.html).\n\nWe then set the gradients to zero, so that we are ready for the next loop. Otherwise, our gradients would record a running tally of all the operations that had happened (i.e. loss.backward() adds the gradients to whatever is already stored, rather than replacing them)."},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = logistic_regression\nloss_fn = negative_log_likelihood\n\nnumber_epochs = 2\nnumber_batches = (number_samples - 1) // batch_size + 1\n\nlearning_rate = 0.5\nfor epoch in range(number_epochs):\n    for batch in range(number_batches):\n        \n        # forward pass\n        start = batch * batch_size\n        X = scaled_training_features_tensor[start:(start + batch_size)]\n        y = training_target_tensor[start:(start + batch_size)]\n        loss = loss_fn(model_fn(X), y)\n        \n        # back propagation\n        loss.backward()\n        with torch.no_grad():\n            weights -= learning_rate * weights.grad\n            bias -= learning_rate * bias.grad\n            weights.grad.zero_()\n            bias.grad.zero_()\n            ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That’s it: we’ve created and trained a minimal neural network (in this case, a logistic regression, since we have no hidden layers) entirely from scratch! Let’s check the loss and accuracy and compare those to what we got earlier. We expect that the loss will have decreased and accuracy to have increased, and they have."},{"metadata":{"trusted":true},"cell_type":"code","source":"loss, accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactor using `torch.nn.functional`\n\nWe will now refactor our code using [torch.nn](https://pytorch.org/docs/stable/nn.html) modules to make it more concise and flexible. The first and easiest step is to make our code shorter by replacing our hand-written activation and loss functions with those from [torch.nn.functional](https://pytorch.org/docs/stable/nn.html#torch-nn-functional).\n\nSince we are using negative log likelihood loss and log softmax activation in this tutorial, we can use [torch.nn.functional.cross_entropy](https://pytorch.org/docs/stable/nn.html#cross-entropy) which combines the two."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = F.cross_entropy(linear_transformation(scaled_training_features_tensor), training_target_tensor),\n_accuracy = accuracy(linear_transformation(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactor using `torch.nn.Module`\n\nNext up, we’ll use [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#module) and [torch.nn.Parameter](https://pytorch.org/docs/stable/nn.html#parameters), for a clearer and more concise training loop. In this case, we want to create a class that holds our weights, bias, and method for the forward step. `torch.nn.Module` has a number of attributes and methods (such as `parameters()` and `zero_grad()`) which we will be using."},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch import nn\n\n\nclass LogisticRegression(nn.Module):\n    \n    def __init__(self):\n        super().__init__()\n        self._weights = nn.Parameter(torch.randn(784, 10) / 784**0.5)\n        self._bias = nn.Parameter(torch.zeros(10))\n        \n    def forward(self, X):\n        return X @ self._weights + self._bias\n    \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we’re now using an object instead of just using a function, we first have to instantiate our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can calculate the loss in the same way as before. Note that `torch.nn.Module` objects are used as if they are functions (i.e they are callable), but behind the scenes Pytorch will call the `forward` method."},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = F.cross_entropy(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Previously in our training loop we had to update the values for each parameter by name and manually zero out the grads for each parameter separately.  With our refactoring we can take advantage of `model_fn.parameters()` and `model_fn.zero_grad()` (which are both defined by PyTorch for `torch.nn.Module` base class!) to make those steps more concise and less prone to the error of forgetting some of our parameters, particularly if we had a more complicated model.\n\nIn order to facilitate re-use and continued refactoring, we can encapsulate the logic of our deep learning pipeline in the following functions. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def partial_fit(model_fn, loss_fn, X_batch, y_batch):\n    # forward pass\n    loss = loss_fn(model_fn(X_batch), y_batch)\n\n    # back propagation\n    loss.backward()\n    with torch.no_grad():\n        for parameter in model_fn.parameters():\n            parameter -= learning_rate * parameter.grad\n        model_fn.zero_grad()\n\n\ndef fit(model_fn, loss_fn, X, y, number_epochs=2, batch_size=64):\n    number_samples, _ = X.shape \n    number_batches = (number_samples - 1) // batch_size + 1\n    for epoch in range(number_epochs):\n        for batch in range(number_batches):\n            start = batch * batch_size\n            X_batch = X[start:(start + batch_size)]\n            y_batch = y[start:(start + batch_size)]\n            partial_fit(model_fn, loss_fn, X_batch, y_batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()\nloss_fn = F.cross_entropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(model_fn, loss_fn, scaled_training_features_tensor, training_target_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactoring using `torch.nn.Linear`\n\nInstead of defining and initializing `self._weights` and `self._bias`, and calculating `X  @ self._weights + self._bias`, we will instead use the Pytorch class [torch.nn.Linear](https://pytorch.org/docs/stable/nn.html#linear) to define a linear layer which does all that for us. Pytorch has many types of predefined layers that can greatly simplify our code, and since the library code is highly optimized using PyTorch's predefined layers often makes our code faster too."},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch import nn\n\n\nclass LogisticRegression(nn.Module):\n    \n    def __init__(self):\n        super().__init__()\n        self._linear_layer = nn.Linear(784, 10)\n        \n    def forward(self, X):\n        return self._linear_layer(X)\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()\nloss_fn = F.cross_entropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(model_fn, loss_fn, scaled_training_features_tensor, training_target_tensor)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactoring using `torch.optim`\n\nPytorch also has a package with various optimization algorithms, [torch.optim](https://pytorch.org/docs/stable/optim.html). We can use the step method from our optimizer to take a forward step, instead of manually updating each parameter."},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch import optim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def partial_fit(model_fn, loss_fn, X_batch, y_batch, opt):\n    # forward pass\n    loss = loss_fn(model_fn(X_batch), y_batch)\n\n    # back propagation\n    loss.backward()\n    opt.step()\n    opt.zero_grad() # don't forget to reset the gradient after each batch!\n\n        \ndef fit(model_fn, loss_fn, X, y, opt, number_epochs=2, batch_size=64):\n    number_samples, _ = X.shape \n    number_batches = (number_samples - 1) // batch_size + 1\n    for epoch in range(number_epochs):\n        for batch in range(number_batches):\n            start = batch * batch_size\n            X_batch = X[start:(start + batch_size)]\n            y_batch = y[start:(start + batch_size)]\n            partial_fit(model_fn, loss_fn, X_batch, y_batch, opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()\nloss_fn = F.cross_entropy\nopt = optim.SGD(model_fn.parameters(), lr=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(model_fn, loss_fn, scaled_training_features_tensor, training_target_tensor, opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactor using `torch.utils.data.TensorDataSet`\n\nThe [torch.utils.data](https://pytorch.org/docs/stable/data.html#module-torch.utils.data) module contains a number of useful classes that we can use to further simplify our code. PyTorch has an abstract `Dataset` class. A Dataset can be anything that has a `__len__` function (called by Python’s standard `len` function) and a `__getitem__` function as a way of indexing into it.\n\nPyTorch’s `TensorDataset` is a `Dataset` wrapping tensors. By defining a length and way of indexing, this also gives us a way to iterate, index, and slice along the first dimension of a tensor. This will make it easier to access both the independent and dependent variables in the same line as we train.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils import data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit(model_fn, loss_fn, data_set, number_samples, opt, number_epochs=2, batch_size=64):\n    number_batches = (number_samples - 1) // batch_size + 1\n    for epoch in range(number_epochs):\n        for batch in range(number_batches):\n            start = batch * batch_size\n            X_batch, y_batch = data_set[start:(start + batch_size)]\n            partial_fit(model_fn, loss_fn, X_batch, y_batch, opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()\nloss_fn = F.cross_entropy\ntraining_data_set = data.TensorDataset(scaled_training_features_tensor, training_target_tensor)\nopt = optim.SGD(model_fn.parameters(), lr=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# note the annoying dependence on number of samples!\nfit(model_fn, loss_fn, training_data_set, number_samples, opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactor using `torch.utils.data.DataLoader`\n\nPytorch’s `DataLoader` is responsible for managing batches. You can create a `DataLoader` from any `Dataset`. `DataLoader` makes it easier to iterate over batches. Rather than having to use `data_set[start:(start + batch_size)]`, the `DataLoader` gives us each minibatch automatically."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit(model_fn, loss_fn, data_loader, opt, number_epochs=2, batch_size=64):\n    for epoch in range(number_epochs):\n        for X_batch, y_batch in data_loader:\n            partial_fit(model_fn, loss_fn, X_batch, y_batch, opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()\nloss_fn = F.cross_entropy\ntraining_data_loader = data.DataLoader(training_data_set, batch_size=batch_size, shuffle=True)\nopt = optim.SGD(model_fn.parameters(), lr=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# now we no longer have the annoying dependency on number of samples!\nfit(model_fn, loss_fn, training_data_loader, opt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks to Pytorch’s `torch.nn.Module`, `torch.nn.Parameter`, `Dataset`, and `DataLoader`, our training loop is now dramatically smaller and easier to understand. Let’s now try to add the basic features necessary to create effecive models in practice."},{"metadata":{},"cell_type":"markdown","source":"# Adding Validation\n\nIn the first part of this tutorial, we were just trying to get a reasonable training loop set up for use on our training data. In reality, you always should also have a validation set, in order to identify if you are overfitting.\n\nShuffling the training data is important to prevent correlation between batches and overfitting. On the other hand, the validation loss will be identical whether we shuffle the validation set or not. Since shuffling takes extra time, it makes no sense to shuffle the validation data.\n\nWe’ll use a batch size for the validation set that is twice as large as that for the training set. This is because the validation set does not need backpropagation and thus takes less memory (it doesn’t need to store the gradients). We take advantage of this to use a larger batch size and compute the loss more quickly."},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit(model_fn, loss_fn, training_data_loader, opt, validation_data_loader=None, number_epochs=2):\n    \n    for epoch in range(number_epochs):\n        model_fn.train()\n        for X_batch, y_batch in training_data_loader:\n            partial_fit(model_fn, loss_fn, X_batch, y_batch, opt)\n        \n        # compute validation loss after each training epoch\n        if validation_data_loader is not None:\n            model_fn.eval()\n            with torch.no_grad():\n                batch_losses, batch_sizes = zip(*[(loss_fn(model_fn(X), y), len(X)) for X, y in validation_data_loader])\n                validation_loss = np.sum(np.multiply(batch_losses, batch_sizes)) / np.sum(batch_sizes)\n            print(f\"Training epoch: {epoch}, Validation loss: {validation_loss}\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LogisticRegression()\nloss_fn = F.cross_entropy\ntraining_data_loader = data.DataLoader(training_data_set, batch_size=80, shuffle=True)\nopt = optim.Adam(model_fn.parameters())\n\n_validation_data_set = data.TensorDataset(scaled_validation_features_tensor, validation_target_tensor)\nvalidation_data_loader = data.DataLoader(_validation_data_set, batch_size=2*80)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(model_fn, loss_fn, training_data_loader, opt, validation_data_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Switching to CNN\n\nWe are now going to build our neural network with three convolutional-subsampling layers. Because none of the functions in the previous section assume anything about the model form, we’ll be able to use them to train a CNN without any modification!\n\nWe will use Pytorch’s predefined [torch.nn.Conv2d](https://pytorch.org/docs/stable/nn.html#conv2d) class as our convolutional layer. We define a CNN with 3 convolutional layers. Each convolution is followed by a [torch.nn.functional.relu](https://pytorch.org/docs/stable/nn.html#id26) non-linear activation function and max pooling using [torch.nn.functional.max_pool2d](https://pytorch.org/docs/stable/nn.html#max-pool2d).\n\nAfter the three convolutional-subsampling layers, we add a couple of densely connected linear layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LeNet5(nn.Module):\n    \n    def __init__(self):\n        super().__init__()\n        self._conv1 = nn.Conv2d(1, 50, kernel_size=3, stride=1, padding=1)\n        self._conv2 = nn.Conv2d(50, 100, kernel_size=3, stride=1, padding=1)\n        self._dense1 = nn.Linear(4900, 100)  # 4900 = 100 * 7 * 7\n        self._dense2 = nn.Linear(100, 10)\n        \n    def forward(self, X):\n        X = X.view(-1, 1, 28, 28) # implicit knowledge of MNIST data shape!\n        X = F.max_pool2d(F.relu(self._conv1(X)), 2)\n        X = F.max_pool2d(F.relu(self._conv2(X)), 2)\n        X = X.view(X.size(0), -1) # equivalent to Keras flatten layer!\n        X = F.relu(self._dense1(X))\n        X = self._dense2(X)\n        return X\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = LeNet5()\nopt = optim.Adam(model_fn.parameters())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# note that we can re-use the loss function as well as trainig and validation data loaders\nfit(model_fn, loss_fn, training_data_loader, opt, validation_data_loader, number_epochs=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Refactor using `torch.nn.Sequential`\n\nPyTorch has another handy class we can use to simply our code: [torch.nn.Sequential](https://pytorch.org/docs/stable/nn.html#sequential). A `Sequential` object runs each of the modules contained within it, in a sequential manner. This is a simpler way of writing our neural network.\n\nTo take advantage of this, we need to be able to easily define a custom layer from a given function. For instance, PyTorch doesn’t have a view layer, and we need to create one for our network. `LambdaLayer` will create a layer that we can then use when defining a network with `Sequential`."},{"metadata":{"trusted":true},"cell_type":"code","source":"class LambdaLayer(nn.Module):\n    \n    def __init__(self, f):\n        super().__init__()\n        self._f = f\n        \n    def forward(self, X):\n        return self._f(X)\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = nn.Sequential(\n    LambdaLayer(lambda X: X.view(-1, 1, 28, 28)),\n    nn.Conv2d(1, 50, kernel_size=3, stride=1, padding=1),\n    nn.ReLU(),\n    nn.MaxPool2d(2),\n    nn.Conv2d(50, 100, kernel_size=3, stride=1, padding=1),\n    nn.ReLU(),\n    nn.MaxPool2d(2),\n    LambdaLayer(lambda X: X.view(X.size(0), -1)),\n    nn.Linear(4900, 100),\n    nn.ReLU(),\n    nn.Linear(100, 10)\n)\n\nopt = optim.Adam(model_fn.parameters())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(model_fn,\n    loss_fn,\n    training_data_loader,\n    opt,\n    validation_data_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generalize our pipeline by wrapping our DataLoader\n\nOur CNN is fairly concise, but it only works with MNIST, because it assumes the input is a 28*28 long vector. Let’s get rid of this assumption, so our model works with any 2d single channel image. First, we can remove the initial Lambda layer by moving the data preprocessing into a generator."},{"metadata":{"trusted":true},"cell_type":"code","source":"class WrappedDataLoader:\n    \n    def __init__(self, data_loader, f):\n        self._data_loader = data_loader\n        self._f = f\n        \n    def __len__(self):\n        return len(self._data_loader)\n    \n    def __iter__(self):\n        for batch in iter(self._data_loader):\n            yield self._f(*batch)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_fn = nn.Sequential(\n    nn.Conv2d(1, 50, kernel_size=3, stride=1, padding=1),\n    nn.ReLU(),\n    nn.MaxPool2d(2),\n    nn.Conv2d(50, 100, kernel_size=3, stride=1, padding=1),\n    nn.ReLU(),\n    nn.MaxPool2d(2),\n    LambdaLayer(lambda X: X.view(X.size(0), -1)),\n    nn.Linear(4900, 100),\n    nn.ReLU(),\n    nn.Linear(100, 10)\n)\n\nopt = optim.Adam(model_fn.parameters())\n\n_preprocess = lambda X, y: (X.view(-1, 1, 28, 28), y)\ntraining_data_loader = WrappedDataLoader(training_data_loader, _preprocess)\nvalidation_data_loader = WrappedDataLoader(validation_data_loader, _preprocess)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(model_fn,\n    loss_fn,\n    training_data_loader,\n    opt,\n    validation_data_loader,\n    number_epochs=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_loss = loss_fn(model_fn(scaled_training_features_tensor), training_target_tensor)\n_accuracy = accuracy(model_fn(scaled_training_features_tensor), training_target_tensor)\nf\"Loss: {_loss}; accuracy: {_accuracy}\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using GPU(s) to accelerate training\n\nGPUs can significantly speedup training of deep neural networks. To take advantage of GPU acceleration click on the `Settings` in the Kernel sidebar and then turn on `GPU` support. **Enabling GPU support will reset the kernel and require you to re-run the cells above to load the data.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define your deep learning model and make it available to the GPU\nlenet5 = nn.Sequential(\n    nn.Conv2d(1, 6, kernel_size=5),\n    nn.ReLU(),\n    nn.MaxPool2d(2),\n    nn.Conv2d(6, 16, kernel_size=5),\n    nn.ReLU(),\n    nn.MaxPool2d(2),\n    LambdaLayer(lambda X: X.view(X.size(0), -1)),\n    nn.Linear(256, 120),\n    nn.ReLU(),\n    nn.Linear(120, 84),\n    nn.ReLU(),\n    nn.Linear(84, 10)\n)\nlenet5.to(device)\n\n# define a loss\nloss_fn = nn.CrossEntropyLoss()\n\n# define an optimizer\nopt = optim.Adam(lenet5.parameters())\n\n# create the data loaders\nbatch_size = 2048\n_training_dataset = data.TensorDataset(scaled_training_features_tensor, training_target_tensor)\n_training_data_loader = data.DataLoader(_training_dataset, batch_size=batch_size, shuffle=True)\n\n_validation_dataset = data.TensorDataset(scaled_validation_features_tensor, validation_target_tensor)\n_validation_data_loader = data.DataLoader(_validation_dataset, batch_size=batch_size)\n\n_preprocess = lambda X, y: (X.view(-1, 1, 28, 28).to(device), y.to(device))\ntraining_data_loader = WrappedDataLoader(_training_data_loader, _preprocess)\nvalidation_data_loader = WrappedDataLoader(_validation_data_loader, _preprocess)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fit(lenet5,\n    loss_fn,\n    training_data_loader,\n    opt,\n    validation_data_loader,\n    number_epochs=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submitting to Kaggle\n\nIf you have created a Kaggle account, then you can submit your model's predictions to Kaggle and see how you stack up against your peers."},{"metadata":{},"cell_type":"markdown","source":"## Use trained model to make predictions using the test data"},{"metadata":{"trusted":false},"cell_type":"code","source":"# note we use transform method and NOT fit_transform!\n_testing_features = np.loadtxt(\"../data/raw/mnist/test.csv\", delimiter=',', skiprows=1, dtype=np.int64)\n_scaled_testing_features = (min_max_scaler.transform(_testing_features)\n                                          .astype(np.float32))\nscaled_testing_features_tensor = torch.from_numpy(_scaled_testing_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"output = model_fn(scaled_testing_features_tensor.view(-1, 1, 28, 28).to(device))\npredictions = torch.argmax(output, dim=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"predictions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visually check model predictions"},{"metadata":{"trusted":false},"cell_type":"code","source":"_, ax = plt.subplots(1,1)\n_ = ax.imshow(scaled_testing_features_tensor[0].reshape((28, 28)), cmap=\"gray\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reformat predictions"},{"metadata":{"trusted":false},"cell_type":"code","source":"# submission format for kaggle\n!head ../data/raw/mnist/sample_submission.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"import time\n\nimport pandas as pd\n\ntimestamp = time.strftime(\"%Y%m%d-%H%M%S\")\nnumber_predictions, = predictions.shape\ndf = pd.DataFrame({\"ImageId\": range(1, number_predictions + 1), \"Label\": predictions.cpu()})\ndf.to_csv(f\"lenet5-submission-{timestamp}.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submit to Kaggle!\n\nOnce you have successfully submited your predictions then you can check the [Digit-Recognizer competition](https://www.kaggle.com/c/digit-recognizer) website and see how well your best model compares to your peers."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":1}