{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Understanding Neural Network Hyperparameters\n\nGreetings! In this notebook we will create a variety of neural networks to classify MNIST handwritten digits. In doing so, we will explore and visualize the impact of adjusting important hyperparameters, including:\n\n* Batch Size\n* Learning Rate\n* Activation Functions\n\nWe will also experiment with adding more layers to the network.\n\nThis exploration is perfect for beginners who want to learn how to create neural networks using Keras.\n\nLet's get started!"},{"metadata":{},"cell_type":"markdown","source":"### Load the Good Stuff"},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras\nfrom keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.optimizers import SGD\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nSEED = 42","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the Data\n\nIn the MNIST dataset, each sample is a 28 x 28 px image that has been flattened into a 1D array of length 784.\n\n* Train: 42,000 samples\n* Test: 28,000 samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ny = train.pop(\"label\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Split the training data into training and validation sets."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(train, y, test_size=0.2, random_state=SEED)\nX_train.shape, y_train.shape, X_valid.shape, y_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we know, the `X` dataframes contain images of handwritten digits represented by vectors of pixel values.\n\nWe can preview `X_train`, but it isn't easy to see what's going on."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Instead, we can convert each row of the data frame into a 28x28 image. Much easier to visualize!"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(12):\n    plt.subplot(3, 4, i+1)\n    image = X_train.iloc[i].values.reshape((28, 28))\n    plt.imshow(image, cmap=\"Greys\")\n    plt.axis(\"off\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `y` arrays contain the actual labels (1-10) for the samples. These are the values we are trying to predict."},{"metadata":{"trusted":true},"cell_type":"code","source":"list(y_train[0:12])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Preprocessing\n\nNormalizing the inputs aids the training process (increasing the speed and helping the neural network avoid local minima).\n\nHere, pixel values are scaled by their max value (255), so that they range from 0 to 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train / 255\nX_valid = X_valid / 255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also need to address the target (aka label), which currently is a singled column vector containing the value (1-10) of each corresponding input image.\n\nHere, we use Keras' built-in `to_categorical` method to convert the target into a one-hot encoded vector of length 10."},{"metadata":{"trusted":true},"cell_type":"code","source":"n = 10\ny_train = keras.utils.to_categorical(y_train, num_classes=n)\ny_valid = keras.utils.to_categorical(y_valid, num_classes=n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.shape, y_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Modeling\n\nKeras offers two methods for creating models: Sequential and Functional. \n\nHere, we use the Sequential method to design a shallow neural network with:\n\n* One dense hidden layer with 64 sigmoid neurons and input size (784,1)\n* One dense output layer with 10 softmax neurons"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nmodel.add(Dense(10, activation=\"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of parameters in the hidden layer depends on the number of neurons in the hidden layer and in the shape of the input layer: $n_{param} = n_{weights} + n_{biases}$.\n\nDo the number of parameters in the model summary make sense?"},{"metadata":{"trusted":false},"cell_type":"code","source":"784 * 64 + 64 # 1 parameter for each of the 784 input neurons * 64 hidden neurons + 64 additional bias terms","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"64 * 10 + 10 # 1 parameter for each of the 64 hidden layer neurons * 10 output neurons + 10 additional bias terms","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"(784 * 64 + 64) + (64 * 10 + 10) # total number of parameters","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, compile the model. Indicate the desired loss (cost) function and optimizer as well as an other additional metrics that should be calcualted during training."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss=\"mean_squared_error\", optimizer=SGD(lr=0.01), metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, fit the model to the training data. Passing in validation data enables the model to estimate performance on new observations."},{"metadata":{"trusted":true},"cell_type":"code","source":"hist = model.fit(x=X_train,\n          y=y_train,\n          batch_size=128,\n          epochs=50, # Adjust to see whether validation metrics continue to improve or start to overfit\n          verbose=0, # Adjust to see training progress\n          validation_data=(X_valid, y_valid))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluation\n\nHow did the model do? The learning curves illustrate how loss and accuracy changed over the course of training."},{"metadata":{"trusted":true},"cell_type":"code","source":"val_loss = hist.history[\"val_loss\"]\nval_accuracy = hist.history[\"val_accuracy\"]\n\nfig, ax = plt.subplots(1, 2, figsize=(15, 5))\n\nsns.lineplot(x=range(0,len(val_accuracy)), y=val_accuracy, ax=ax[0], label=\"Validation Accuracy\")\nsns.lineplot(x=range(0,len(val_loss)), y=val_loss, ax=ax[1], label=\"Validation Loss\")\n\nax[0].set_xlabel(\"# of Epochs\")\nax[1].set_xlabel(\"# of Epochs\")\n\nplt.suptitle(\"Learning Curves\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To wrap things up, we can evaluate the model on the validation data to estimate test performance."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(X_valid, y_valid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### An Aside: Saving Models\n\nAs an aside, Keras makes it easy to save and load models to/from files, thus saving you from having to re-train!\n\nUncomment below to utilize this functionality."},{"metadata":{"trusted":false},"cell_type":"code","source":"# model.save(\"model.hd5\")\n# model = keras.models.load_model(\"model.hd5\")\n# model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Experimentation\n\nDespite a lot going on under the hood, Keras makes it simple to create and apply a neural network.\n\nNow, let's experiment with alternative model configurations, including:\n\n* Different batch sizes\n* Increased or decreased learning rates\n* Additional hidden layers\n* Alternative neuronal activation functions"},{"metadata":{},"cell_type":"markdown","source":"### Helper Functions\n\nTo reduce code duplication, the following wrapper class bundles Keras and plotting functionality together. Very handy :)"},{"metadata":{"trusted":false},"cell_type":"code","source":"class NeuralNetwork():\n    def __init__(self, name, batch_size, epochs, learning_rate, verbose):\n        self.name = name\n        self.batch_size = batch_size\n        self.epochs = epochs\n        self.learning_rate = learning_rate\n        self.verbose = verbose\n        self.model = Sequential()\n        \n    def add_(self, layer):\n        self.model.add(layer)\n\n    def compile_and_fit(self):\n        self.model.compile(loss=\"mean_squared_error\", optimizer=SGD(lr=self.learning_rate), metrics=[\"accuracy\"])\n        self.history = self.model.fit(x=X_train,\n                                      y=y_train,\n                                      batch_size=self.batch_size,\n                                      epochs=self.epochs,\n                                      verbose=self.verbose,\n                                      validation_data=(X_valid, y_valid))\n        self.val_loss = self.history.history[\"val_loss\"]\n        self.val_accuracy = self.history.history[\"val_accuracy\"]\n    \n    def plot_learning_curves(self):\n        fig, ax = plt.subplots(1, 2, figsize=(15, 5))\n        \n        sns.lineplot(x=range(0,len(self.val_accuracy)), y=self.val_accuracy, ax=ax[0], label=\"Validation Accuracy\")\n        sns.lineplot(x=range(0,len(self.val_loss)), y=self.val_loss, ax=ax[1], label=\"Validation Loss\")\n\n        ax[0].set_xlabel(\"# of Epochs\")\n        ax[1].set_xlabel(\"# of Epochs\")\n\n        plt.suptitle(\"Learning Curves: {}\".format(self.name))\n        plt.show()\n\n    def evaluate_(self):\n        return self.model.evaluate(X_valid, y_valid)\n    \n    def save(self, filename):\n        self.model.save(\"working/\"+filename+\".hd5\")\n        \n    def summary_(self):\n        return self.model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def compare_learning_curves(models):\n    fig, ax = plt.subplots(1, 2, figsize=(20, 8))\n    \n    for model in models:\n        sns.lineplot(x=range(0,len(model.val_accuracy)), y=model.val_accuracy, ax=ax[0], label=model.name)\n        sns.lineplot(x=range(0,len(model.val_loss)), y=model.val_loss, ax=ax[1], label=model.name)\n    \n    ax[0].set_xlabel(\"# of Epochs\")\n    ax[1].set_xlabel(\"# of Epochs\")\n\n    ax[0].set_title(\"Validation Accuracy\")\n    ax[1].set_title(\"Validation Loss\")\n\n    plt.suptitle(\"Learning Curves\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Batch Size\n\nBatch Size is a critical hyperparameter that controls how many training samples are passed through the model at a time. These inputs are processed via forward propagation to produce an output, which is then evaluated by the cost function. Finally, the weights and biases of all neurons are updated via gradient descent to generate better and better estimates.\n\nChanging the batch size changes the number of training samples that are used to calculate the cost function. Too big a batch size can be computationally expensive and tends to create models that do not generalize as well. Too small a batch size can lead to lengthy training processes and wasteful use of resources.\n\nHere we compare learning curves for a handful of models utilizing different batch sizes."},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_sizes = [8, 16, 32, 64, 128, 256]\nn_epochs = 50","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"accuracy = pd.DataFrame(columns=batch_sizes, index=range(n_epochs))\nloss = pd.DataFrame(columns=batch_sizes, index=range(n_epochs))\naccuracy[\"Epoch\"] = range(n_epochs)\nloss[\"Epoch\"] = range(n_epochs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for batch_size in batch_sizes:\n    model = Sequential()\n    model.add(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\n    model.add(Dense(10, activation=\"softmax\"))\n    model.compile(loss=\"mean_squared_error\", optimizer=SGD(lr=0.01), metrics=[\"accuracy\"])\n    \n    hist = model.fit(x=X_train,\n              y=y_train,\n              batch_size=batch_size,\n              epochs=n_epochs,\n              verbose=0,\n              validation_data=(X_valid, y_valid))\n    \n    accuracy[batch_size] = hist.history[\"val_accuracy\"]\n    loss[batch_size] = hist.history[\"val_loss\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"accuracy_melt = accuracy.melt(value_name=\"Accuracy\", var_name=\"Batch Size\", id_vars=[\"Epoch\"])\nloss_melt = loss.melt(value_name=\"Loss\", var_name=\"Batch Size\", id_vars=[\"Epoch\"])\n\naccuracy_melt[\"Batch Size\"] = accuracy_melt[\"Batch Size\"].astype(object)\nloss_melt[\"Batch Size\"] = loss_melt[\"Batch Size\"].astype(object)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"accuracy_melt = accuracy.melt(value_name=\"Accuracy\", var_name=\"Batch Size\", id_vars=[\"Epoch\"])\nloss_melt = loss.melt(value_name=\"Loss\", var_name=\"Batch Size\", id_vars=[\"Epoch\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20, 8))\n\nsns.lineplot(x=\"Epoch\", y=\"Accuracy\", hue=\"Batch Size\", data=accuracy_melt, ax=ax[0], legend=\"full\")\nsns.lineplot(x=\"Epoch\", y=\"Loss\", hue=\"Batch Size\", data=loss_melt, ax=ax[1], legend=\"full\")\n\nax[0].set_title(\"Validation Accuracy\")\nax[1].set_title(\"Validation Loss\")\n\nax[0].set_xlabel(\"# of Epochs\")\nax[1].set_xlabel(\"# of Epochs\")\n\nplt.suptitle(\"Learning Curves\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observations**\n\nLarger batch sizes are more stable, but slower to learn. In contrast, smaller batch sizes exhibit faster, but more erratic, learning."},{"metadata":{},"cell_type":"markdown","source":"### Learning Rate\n\nThe learning rate, $\\eta$, controls the step size of gradient descent. In other words, it controls the magnitude of the adjustments made to model weights and biases\n\nLarge learning rates require fewer training epochs, but can cause unstable training and failure to converge to the optimal solution. On the other hand, small learning rates are more stable, but require more training epochs and are computationally inefficient.\n\nLet's tweak the learning rate and observe the impact."},{"metadata":{"trusted":false},"cell_type":"code","source":"n_epochs = 100\nbatch_size = 128\nverbose = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"learning_rate = 100\nnn_lr_100 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_100.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_100.add_(Dense(10, activation=\"softmax\"))\n\nlearning_rate = 1000\nnn_lr_1000 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_1000.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_1000.add_(Dense(10, activation=\"softmax\"))\n\nlearning_rate = 10\nnn_lr_10 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_10.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_10.add_(Dense(10, activation=\"softmax\"))\n\nlearning_rate = 1\nnn_lr_1 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_1.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_1.add_(Dense(10, activation=\"softmax\"))\n\nlearning_rate = 0.1\nnn_lr_p1 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_p1.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_p1.add_(Dense(10, activation=\"softmax\"))\n\nlearning_rate = 0.01 # default\nnn_lr_p01 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_p01.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_p01.add_(Dense(10, activation=\"softmax\"))\n\nlearning_rate = 0.001\nnn_lr_p001 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_p001.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_p001.add_(Dense(10, activation=\"softmax\"))\n\nnn_lr_100.compile_and_fit()\nnn_lr_1000.compile_and_fit()\nnn_lr_10.compile_and_fit()\nnn_lr_1.compile_and_fit()\nnn_lr_p1.compile_and_fit()\nnn_lr_p01.compile_and_fit()\nnn_lr_p001.compile_and_fit()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"learning_rate = 0.1\nnn_lr_p1 = NeuralNetwork(\"LR = {}\".format(learning_rate), batch_size, n_epochs, learning_rate, verbose)\nnn_lr_p1.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_lr_p1.add_(Dense(10, activation=\"softmax\"))\nnn_lr_p1.compile_and_fit()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"compare_learning_curves([nn_lr_1000, nn_lr_100, nn_lr_10, nn_lr_1, nn_lr_p1, nn_lr_p01, nn_lr_p001])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In particular, notice the training instability visible at higher LR values, for example `LR=10`:"},{"metadata":{"trusted":false},"cell_type":"code","source":"nn_lr_10.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In contrast, there is minimal training instability at `LR<10`, however models take more epochs to reach their full potential:"},{"metadata":{"trusted":false},"cell_type":"code","source":"nn_lr_p1.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observations**\n\nLearning rates greater than 10 were unable to converge to a solution (at least not within 100 epochs), resulting in poor accuracy and high loss. A learning rate of 10 was the fastest to reach optimal accuracy and loss. This is as expected: large learning rates increase the magnitude of parameter updates and therefore speed up training. There were, however, slight oscillations in the learning curves for `LR=10`, illustrating some training instability. This instability is not visible at smaller learning rates, at the cost of slower training."},{"metadata":{},"cell_type":"markdown","source":"### Additional Hidden Layers\n\nIn practice, neural networks rarely consist of only 3 layers. Additional hidden layers are added to enable models to extract increasingly high level representations from the data, suitable for complex applications. \n\nHere, we compare models with one, two and three hidden layers. Of course, you can also adjust the number of neurons in each layer, but for simplicity, we've stuck with a static 64 units."},{"metadata":{"trusted":false},"cell_type":"code","source":"n_epochs = 100\nbatch_size = 128\nlearning_rate = 0.01\nverbose = 0","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"nn_l1 = NeuralNetwork(\"1 Hidden Layer\", batch_size, n_epochs, learning_rate, verbose)\nnn_l1.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_l1.add_(Dense(10, activation=\"softmax\"))\n\nnn_l1.summary_()\nnn_l1.compile_and_fit()\nnn_l1.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"nn_l2 = NeuralNetwork(\"2 Hidden Layers\", batch_size, n_epochs, learning_rate, verbose)\nnn_l2.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_l2.add_(Dense(64, activation=\"sigmoid\"))\nnn_l2.add_(Dense(10, activation=\"softmax\"))\n\nnn_l2.summary_()\nnn_l2.compile_and_fit()\nnn_l2.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"nn_l3 = NeuralNetwork(\"3 Hidden Layers\", batch_size, n_epochs, learning_rate, verbose)\nnn_l3.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_l3.add_(Dense(64, activation=\"sigmoid\"))\nnn_l3.add_(Dense(64, activation=\"sigmoid\"))\nnn_l3.add_(Dense(10, activation=\"softmax\"))\n\nnn_l3.summary_()\nnn_l3.compile_and_fit()\nnn_l3.plot_learning_curves()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"compare_learning_curves([nn_l1, nn_l2, nn_l3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observations**\n\nIn this instance, additional hidden layers do not improve accuracy or decrease loss, at least not in the first 100 epochs. In general, if the number of hidden layers is inconsistent with the complexity of the problem, the network tends to overfit the training data. Trial and error is often necesary to determine the optimal architecture for a given problem."},{"metadata":{},"cell_type":"markdown","source":"### Activation Functions\n\nAn activation function translates the weighted sum of a neuron's inputs and bias terms into an output that is passed further along the network. There are many different activation functions, each with varying applications. The key is for the activation function to be non-linear; this gives deep learning models the power of universal function approximators - a fancy way of saying that they can estimate any output `y` given some input `x`.\n\nThe last thing we'll look at is what happens when we substitute the `sigmoid` activation function with two popular alternatives: `tanh` and `relu`."},{"metadata":{"trusted":false},"cell_type":"code","source":"n_epochs = 100\nbatch_size = 128\nlearning_rate = 0.01\nverbose = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"nn_sigmoid = NeuralNetwork(\"Sigmoid\", batch_size, n_epochs, learning_rate, verbose)\nnn_sigmoid.add_(Dense(64, activation=\"sigmoid\", input_shape=(784,)))\nnn_sigmoid.add_(Dense(10, activation=\"softmax\"))\nnn_sigmoid.compile_and_fit()\n\nnn_tanh = NeuralNetwork(\"Tanh\", batch_size, n_epochs, learning_rate, verbose)\nnn_tanh.add_(Dense(64, activation=\"tanh\", input_shape=(784,)))\nnn_tanh.add_(Dense(10, activation=\"softmax\"))\nnn_tanh.compile_and_fit()\n\nnn_relu = NeuralNetwork(\"ReLU\", batch_size, n_epochs, learning_rate, verbose)\nnn_relu.add_(Dense(64, activation=\"relu\", input_shape=(784,)))\nnn_relu.add_(Dense(10, activation=\"softmax\"))\nnn_relu.compile_and_fit()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"compare_learning_curves([nn_sigmoid, nn_tanh, nn_relu])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Observations**\n\nThe `tanh` and `relu` activation functions easily outperform `sigmoid` in terms of more quickly reaching peak accuracy and minimum loss. It should come as no surprise that `relu` is the most widely used activation function - when in doubt, use `relu`!"},{"metadata":{},"cell_type":"markdown","source":"### Conclusion\n\nI hope you have enjoyed learning more about neural network hyperparameters in the context of the MNIST dataset.\n\nI'd love to hear about your process for tuning hyperparameters - please leave me a comment below.\n\nThanks for reading and, until next time, happy coding :)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.3"}},"nbformat":4,"nbformat_minor":4}