{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# TensorFlow 2 Tutorial: Get Started in Deep Learning With tf.keras\n\n\n\n## Credit: [Jason Brownlee: Original Tutorial](https://machinelearningmastery.com/tensorflow-tutorial-deep-learning-with-tf-keras/)","metadata":{}},{"cell_type":"markdown","source":"Predictive modeling with deep learning is a skill that modern developers need to know.\n\n`TensorFlow` is the premier open-source deep learning framework developed and maintained by Google. Although using TensorFlow directly can be challenging, the modern `tf.keras API` beings the simplicity and ease of use of `Keras` to the `TensorFlow` project.\n\nUsing `tf.keras` allows you to design, fit, evaluate, and use deep learning models to make predictions in just a few lines of code. It makes common deep learning tasks, such as classification and regression predictive modeling, accessible to average developers looking to get things done.\n\nIn this tutorial, you will discover a step-by-step guide to developing deep learning models in TensorFlow using the `tf.keras API`.\n***\nAfter completing this tutorial, you will know:\n\n- The difference between `keras` and `tf.keras` and how to install and confirm `TensorFlow` is working.\n- The 5-step life-cycle of `tf.keras` models and how to use the `sequential` and `functional APIs`.\n- How to develop `MLP`, `CNN`, and `RNN` models with `tf.keras` for regression, classification, and time series forecasting.\n- How to use the advanced features of the `tf.keras API` to inspect and diagnose your model.\n- How to improve the performance of your `tf.keras` model by reducing overfitting and accelerating training.\n***","metadata":{}},{"cell_type":"markdown","source":"# TensorFlow Tutorial Overview\n\nThis tutorial is designed to be your complete introduction to `tf.keras` for your deep learning project.\n\nThe focus is on using the API for common deep learning model development tasks; we will not be diving into the math and theory of deep learning.\n\nThe best way to learn deep learning in python is by doing. Dive in. You can circle back for more theory later.\n\nI have designed each code example to use best practices and to be standalone so that you can copy and paste it directly into your project and adapt it to your specific needs. This will give you a massive head start over trying to figure out the API from official documentation alone.\n\n1. Install TensorFlow and tf.keras\n    1. What Are Keras and tf.keras?\n    2. How to Install TensorFlow\n    3. How to Confirm TensorFlow Is Installed\n2. Deep Learning Model Life-Cycle\n    1. The 5-Step Model Life-Cycle\n    2. Sequential Model API (Simple)\n    3. Functional Model API (Advanced)\n3. How to Develop Deep Learning Models\n    1. Develop Multilayer Perceptron Models\n    1. Develop Convolutional Neural Network Models\n    1. Develop Recurrent Neural Network Models\n4. How to Use Advanced Model Features\n    1. How to Visualize a Deep Learning Model\n    1. How to Plot Model Learning Curves\n    1. How to Save and Load Your Model\n5. How to Get Better Model Performance\n    1. How to Reduce Overfitting With Dropout\n    1. How to Accelerate Training With Batch Normalization\n    1. How to Halt Training at the Right Time With Early Stopping","metadata":{}},{"cell_type":"markdown","source":"# 1. Install TensorFlow and tf.keras\n\nIn this section, you will discover what `tf.keras` is, how to install it, and how to confirm that it is installed correctly.\n\n## 1.1 What Are `Keras` and `tf.keras`?\n`Keras` is an open-source deep learning library written in Python.\n\n`Keras` was popular because the API was clean and simple, allowing standard deep learning models to be defined, fit, and evaluated in just a few lines of code.\n\nA secondary reason `Keras` took-off was because it allowed you to use any one among the range of popular deep learning mathematical libraries as the backend (e.g. used to perform the computation), such as `TensorFlow`, `Theano`, and later, `CNTK`. This allowed the power of these libraries to be harnessed (e.g. GPUs) with a very clean and simple interface.\n\nIn 2019, Google released a new version of their `TensorFlow` deep learning library (`TensorFlow 2`) that integrated the `Keras API` directly and promoted this interface as the default or standard interface for deep learning development on the platform.\n\nThis integration is commonly referred to as the `tf.keras` interface or `API` (“`tf`” is short for “`TensorFlow`“). This is to distinguish it from the so-called standalone `Keras` open source project.\n\n- **Standalone `Keras`**. The standalone open source project that supports `TensorFlow`, `Theano` and `CNTK` backends.\n- **`tf.keras`**. The `Keras API` integrated into `TensorFlow 2`.\n\nThe `Keras API` implementation in `Keras` is referred to as “`tf.keras`” because this is the Python idiom used when referencing the `API`. First, the `TensorFlow` module is imported and named “`tf`“; then, `Keras API` elements are accessed via calls to `tf.keras`; for example:","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nmodel = tf.keras.Sequential()","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:47.315914Z","iopub.execute_input":"2021-09-03T19:23:47.316514Z","iopub.status.idle":"2021-09-03T19:23:54.716362Z","shell.execute_reply.started":"2021-09-03T19:23:47.316464Z","shell.execute_reply":"2021-09-03T19:23:54.715525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2 How to Install TensorFlow\n\nBefore installing `TensorFlow`, ensure that you have Python installed, such as Python 3.6 or higher.\n\nThere are many ways to install the TensorFlow open-source deep learning library.\n\nThe most common, and perhaps the simplest, way to install TensorFlow on your workstation is by using pip.\n\nFor example, on the command line, you can type:\n`sudo pip install tensorflow`\n\nThere is no need to set up the `GPU` now.\n\nAll examples in this tutorial will work just fine on a modern `CPU`. If you want to configure `TensorFlow` for your `GPU`, you can do that after completing this tutorial. Don’t get distracted!","metadata":{}},{"cell_type":"markdown","source":"## 1.3 How to Confirm TensorFlow Is Installed\nOnce `TensorFlow` is installed, it is important to confirm that the library was installed successfully and that you can start using it.\n\nIf TensorFlow is not installed correctly or raises an error on this step, you won’t be able to run the examples later.","metadata":{}},{"cell_type":"code","source":"import tensorflow\ntensorflow.__version__","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.720927Z","iopub.execute_input":"2021-09-03T19:23:54.722366Z","iopub.status.idle":"2021-09-03T19:23:54.728879Z","shell.execute_reply.started":"2021-09-03T19:23:54.722296Z","shell.execute_reply":"2021-09-03T19:23:54.728275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This confirms that TensorFlow is installed correctly and that we are all using the same version.\n****","metadata":{}},{"cell_type":"markdown","source":"# 2. Deep Learning Model Life-Cycle\n\nIn this section, you will discover the life-cycle for a deep learning model and the two `tf.keras` APIs that you can use to define models.\n\n## 2.1 The 5-Step Model Life-Cycle\nA model has a life-cycle, and this very simple knowledge provides the backbone for both modeling a dataset and understanding the `tf.keras API`.\n\nThe five steps in the life-cycle are as follows:\n\n1. Define the model.\n1. Compile the model.\n1. Fit the model.\n1. Evaluate the model.\n1. Make predictions.\nLet’s take a closer look at each step in turn.\n\n**Define the Model**\n\nDefining the model requires that you first select the type of model that you need and then choose the architecture or network topology.\n\nFrom an `API` perspective, this involves defining the layers of the model, configuring each layer with a `number of nodes` and `activation function`, and connecting the layers together into a cohesive model.\n\nModels can be defined either with the `Sequential API` or the `Functional API`, and we will take a look at this in the next section.\n\n**Compile the Model**\n\nCompiling the model requires that you first select a `loss function` that you want to optimize, such as `mean squared error` or `cross-entropy`.\n\nIt also requires that you select an algorithm to perform the optimization procedure, typically `stochastic gradient descent`, or a `modern variation`, such as `Adam`. It may also require that you select any `performance metrics` to keep track of during the model training process.\n\nFrom an `API` perspective, this involves calling a function to compile the model with the chosen configuration, which will prepare the appropriate data structures required for the efficient use of the model you have defined.\n\nThe optimizer can be specified as a string for a known optimizer class, e.g. ‘`sgd`‘ for `stochastic gradient descent`, or you can configure an instance of an optimizer class and use that.\n\nFor a list of supported optimizers, see this: [tf.keras Optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)","metadata":{}},{"cell_type":"code","source":"# compile the model\nopt = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)\nmodel.compile(optimizer=opt, loss='binary_crossentropy')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.72986Z","iopub.execute_input":"2021-09-03T19:23:54.730227Z","iopub.status.idle":"2021-09-03T19:23:54.741854Z","shell.execute_reply.started":"2021-09-03T19:23:54.730189Z","shell.execute_reply":"2021-09-03T19:23:54.741158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The three most common loss functions are:\n\n- ‘`binary_crossentropy`‘ for binary classification.\n- ‘`sparse_categorical_crossentropy`‘ for multi-class classification.\n- ‘`mse`‘ (mean squared error) for regression.","metadata":{}},{"cell_type":"code","source":"# compile the model\nmodel.compile(optimizer='sgd', loss='mse')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.742827Z","iopub.execute_input":"2021-09-03T19:23:54.743179Z","iopub.status.idle":"2021-09-03T19:23:54.754985Z","shell.execute_reply.started":"2021-09-03T19:23:54.743142Z","shell.execute_reply":"2021-09-03T19:23:54.754257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For a list of supported loss functions, see: [tf.keras Loss Functions](https://www.tensorflow.org/api_docs/python/tf/keras/losses)\n\nMetrics are defined as a list of strings for known metric functions or a list of functions to call to evaluate predictions.\n\nFor a list of supported metrics, see: [tf.keras Metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)","metadata":{}},{"cell_type":"code","source":"# compile the model\nmodel.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.757481Z","iopub.execute_input":"2021-09-03T19:23:54.757858Z","iopub.status.idle":"2021-09-03T19:23:54.769544Z","shell.execute_reply.started":"2021-09-03T19:23:54.75782Z","shell.execute_reply":"2021-09-03T19:23:54.768175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Fit the Model**\n\nFitting the model requires that you first select the training configuration, such as the number of `epochs` (loops through the training dataset) and the `batch size` (number of samples in an epoch used to estimate model error).\n\nTraining applies the chosen optimization algorithm to minimize the chosen loss function and updates the model using the `backpropagation` of error algorithm.\n\nFitting the model is the slow part of the whole process and can take seconds to hours to days, depending on the complexity of the model, the hardware you’re using, and the size of the training dataset.\n\nFrom an `API` perspective, this involves calling a function to perform the training process. This function will block (not return) until the training process has finished.","metadata":{}},{"cell_type":"code","source":"# fit the model\n# model.fit(X, y, epochs=100, batch_size=32)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.773175Z","iopub.execute_input":"2021-09-03T19:23:54.773922Z","iopub.status.idle":"2021-09-03T19:23:54.780346Z","shell.execute_reply.started":"2021-09-03T19:23:54.773781Z","shell.execute_reply":"2021-09-03T19:23:54.779426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While fitting the model, a progress bar will summarize the status of each `epoch` and the overall training process. This can be simplified to a simple report of model performance each `epoch` by setting the “`verbose`” argument to 2. All output can be turned off during training by setting “`verbose`” to 0.","metadata":{}},{"cell_type":"code","source":"...\n# fit the model\n# model.fit(X, y, epochs=100, batch_size=32, verbose=0)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.782124Z","iopub.execute_input":"2021-09-03T19:23:54.782839Z","iopub.status.idle":"2021-09-03T19:23:54.793847Z","shell.execute_reply.started":"2021-09-03T19:23:54.78277Z","shell.execute_reply":"2021-09-03T19:23:54.793187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Evaluate the Model**\n\nEvaluating the model requires that you first choose a holdout dataset used to evaluate the model. This should be data not used in the training process so that we can get an unbiased estimate of the performance of the model when making predictions on new data.\n\nThe speed of model evaluation is proportional to the amount of data you want to use for the evaluation, although it is much faster than training as the model is not changed.\n\nFrom an `API` perspective, this involves calling a function with the holdout dataset and getting a loss and perhaps other metrics that can be reported.","metadata":{}},{"cell_type":"code","source":"...\n# evaluate the model\n# loss = model.evaluate(X, y, verbose=0)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.794826Z","iopub.execute_input":"2021-09-03T19:23:54.795202Z","iopub.status.idle":"2021-09-03T19:23:54.805666Z","shell.execute_reply.started":"2021-09-03T19:23:54.795164Z","shell.execute_reply":"2021-09-03T19:23:54.804751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Make a Prediction**\n\nMaking a prediction is the final step in the life-cycle. It is why we wanted the model in the first place.\n\nIt requires you have new data for which a prediction is required, e.g. where you do not have the target values.\n\nFrom an `API` perspective, you simply call a function to make a prediction of a class label, probability, or numerical value: whatever you designed your model to predict.\n\nYou may want to save the model and later load it to make predictions. You may also choose to fit a model on all of the available data before you start using it.\n\nNow that we are familiar with the model life-cycle, let’s take a look at the two main ways to use the `tf.keras API` to build models: `sequential` and `functional`.\n***","metadata":{}},{"cell_type":"code","source":"...\n# make a prediction\n# yhat = model.predict(X)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.807849Z","iopub.execute_input":"2021-09-03T19:23:54.809097Z","iopub.status.idle":"2021-09-03T19:23:54.819675Z","shell.execute_reply.started":"2021-09-03T19:23:54.809044Z","shell.execute_reply":"2021-09-03T19:23:54.81869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Sequential Model API (Simple)\n\nThe `sequential` model `API` is the simplest and is the `API` that I recommend, especially when getting started.\n\nIt is referred to as “`sequential`” because it involves defining a `Sequential` class and adding `layers` to the `model` one by one in a linear manner, from input to output.\n\nThe example below defines a `Sequential MLP model` that accepts eight inputs, has one hidden layer with 10 nodes and then an output layer with one node to predict a numerical value.","metadata":{}},{"cell_type":"code","source":"# example of a model defined with the sequential api\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\n# define the model\nmodel = Sequential()\nmodel.add(Dense(10, input_shape=(8,)))\nmodel.add(Dense(1))","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.821648Z","iopub.execute_input":"2021-09-03T19:23:54.822066Z","iopub.status.idle":"2021-09-03T19:23:54.931213Z","shell.execute_reply.started":"2021-09-03T19:23:54.821964Z","shell.execute_reply":"2021-09-03T19:23:54.930196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that the visible `layer` of the network is defined by the “`input_shape`” argument on the first hidden layer. That means in the above example, the model expects the input for one sample to be a vector of eight numbers.\n\nThe `sequential API` is easy to use because you keep calling `model.add()` until you have added all of your layers.\n\nFor example, here is a deep MLP with five hidden layers.","metadata":{}},{"cell_type":"code","source":"# example of a model defined with the sequential api\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\n# define the model\nmodel = Sequential()\nmodel.add(Dense(100, input_shape=(8,)))\nmodel.add(Dense(80))\nmodel.add(Dense(30))\nmodel.add(Dense(10))\nmodel.add(Dense(5))\nmodel.add(Dense(1))","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.93284Z","iopub.execute_input":"2021-09-03T19:23:54.933216Z","iopub.status.idle":"2021-09-03T19:23:54.990128Z","shell.execute_reply.started":"2021-09-03T19:23:54.93315Z","shell.execute_reply":"2021-09-03T19:23:54.989137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Functional Model API (Advanced)\n\nThe `functional API` is more complex but is also more flexible.\n\nIt involves explicitly connecting the output of one layer to the input of another layer. Each connection is specified.\n\nFirst, an input layer must be defined via the Input class, and the shape of an input sample is specified. We must retain a reference to the input layer when defining the model.","metadata":{}},{"cell_type":"code","source":"...\n# define the layers\n# x_in = Input(shape=(8,))","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.991403Z","iopub.execute_input":"2021-09-03T19:23:54.991773Z","iopub.status.idle":"2021-09-03T19:23:54.996572Z","shell.execute_reply.started":"2021-09-03T19:23:54.991627Z","shell.execute_reply":"2021-09-03T19:23:54.995729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, a fully connected layer can be connected to the input by calling the layer and passing the input layer. This will return a reference to the output connection in this new layer.","metadata":{}},{"cell_type":"code","source":"...\n# x = Dense(10)(x_in)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:54.998155Z","iopub.execute_input":"2021-09-03T19:23:54.998514Z","iopub.status.idle":"2021-09-03T19:23:55.010961Z","shell.execute_reply.started":"2021-09-03T19:23:54.998465Z","shell.execute_reply":"2021-09-03T19:23:55.010188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can then connect this to an output layer in the same manner.","metadata":{}},{"cell_type":"code","source":"...\n# x_out = Dense(1)(x)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:55.012651Z","iopub.execute_input":"2021-09-03T19:23:55.012938Z","iopub.status.idle":"2021-09-03T19:23:55.025088Z","shell.execute_reply.started":"2021-09-03T19:23:55.012887Z","shell.execute_reply":"2021-09-03T19:23:55.024169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Once connected, we define a Model object and specify the input and output layers. The complete example is listed below.","metadata":{}},{"cell_type":"code","source":"# example of a model defined with the functional api\nfrom tensorflow.keras import Model\nfrom tensorflow.keras import Input\nfrom tensorflow.keras.layers import Dense\n# define the layers\nx_in = Input(shape=(8,))\nx = Dense(10)(x_in)\nx_out = Dense(1)(x)\n# define the model\nmodel = Model(inputs=x_in, outputs=x_out)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:55.027136Z","iopub.execute_input":"2021-09-03T19:23:55.028176Z","iopub.status.idle":"2021-09-03T19:23:55.052596Z","shell.execute_reply.started":"2021-09-03T19:23:55.027629Z","shell.execute_reply":"2021-09-03T19:23:55.051883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As such, it allows for more complicated model designs, such as models that may have multiple input paths (separate vectors) and models that have multiple output paths (e.g. a word and a number).\n\nThe `functional API` can be a lot of fun when you get used to it.\n\nFor more on the functional API, see: [The Keras functional API in TensorFlow](https://www.tensorflow.org/guide/keras/functional)\n\nNow that we are familiar with the model life-cycle and the two APIs that can be used to define models, let’s look at developing some standard models.\n***","metadata":{}},{"cell_type":"markdown","source":"# 3. How to Develop Deep Learning Models\n\nIn this section, you will discover how to develop, evaluate, and make predictions with standard deep learning models, including `Multilayer Perceptrons (MLP)`, `Convolutional Neural Networks (CNNs)`, and `Recurrent Neural Networks (RNNs)`.\n\n## 3.1 Develop Multilayer Perceptron Models\n\nA `Multilayer Perceptron model`, or `MLP` for short, is a standard fully connected neural network model.\n\nIt is comprised of layers of nodes where each node is connected to all outputs from the previous layer and the output of each node is connected to all inputs for nodes in the next layer.\n\nAn `MLP` is created by with one or more `Dense` layers. This model is appropriate for tabular data, that is data as it looks in a table or spreadsheet with one column for each variable and one row for each variable. There are three predictive modeling problems you may want to explore with an `MLP`; they are binary classification, multiclass classification, and regression.\n\nLet’s fit a model on a real dataset for each of these cases.\n\nNote, the models in this section are effective, but not optimized. See if you can improve their performance. Post your findings in the comments below.\n\n**`MLP` for Binary Classification**\n\nWe will use the Ionosphere binary (two-class) classification dataset to demonstrate an `MLP` for binary classification.\n\nThis dataset involves predicting whether a structure is in the atmosphere or not given radar returns.\n\nThe dataset will be downloaded automatically using Pandas, but you can learn more about it here.\n- [Ionosphere Dataset (csv)](https://raw.githubusercontent.com/jbrownlee/Datasets/master/ionosphere.csv).\n- [Ionosphere Dataset Description (csv)](https://raw.githubusercontent.com/jbrownlee/Datasets/master/ionosphere.names).\n\nWe will use a `LabelEncoder` to encode the string labels to integer values 0 and 1. The model will be fit on 67 percent of the data, and the remaining 33 percent will be used for evaluation, split using the `train_test_split()` function.\n\nIt is a good practice to use ‘`relu`‘ activation with a ‘`he_normal`‘ weight initialization. This combination goes a long way to overcome the problem of vanishing gradients when training deep neural network models. For more on `ReLU`, see the tutorial:\n\n- [A Gentle Introduction to the Rectified Linear Unit (ReLU)](https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/)\n\nThe model predicts the probability of class 1 and uses the sigmoid activation function. The model is optimized using the adam version of stochastic gradient descent and seeks to minimize the cross-entropy loss.","metadata":{}},{"cell_type":"code","source":"# mlp for binary classification\nfrom pandas import read_csv\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\n\n# load the dataset\npath = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/ionosphere.csv'\ndf = read_csv(path, header=None)\n\n# split into input and output columns\nX, y = df.values[:, :-1], df.values[:, -1]\n\n# ensure all data are floating point values\nX = X.astype('float32')\n\n# encode strings to integer\ny = LabelEncoder().fit_transform(y)\n\n# split into train and test datasets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)\nprint(X_train.shape, X_test.shape, y_train.shape, y_test.shape)\n\n# determine the number of input features\nn_features = X_train.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dense(8, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\n# fit the model\nmodel.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)\n\n# evaluate the model\nloss, acc = model.evaluate(X_test, y_test, verbose=0)\nprint(f'Test Accuracy: {acc:.3f}')\n\n# make a prediction\nrow = [1,0 ,0.99539, -0.05889, 0.85243, 0.02306, 0.83398, -0.37708, 1, 0.03760, \n       0.85243, -0.17755, 0.59755, -0.44945, 0.60536, -0.38223, 0.84356, -0.38542,\n       0.58212, -0.32192, 0.56971, -0.29674, 0.36946, -0.47357, 0.56811, -0.51171,\n       0.41078, -0.46168, 0.21266, -0.34090, 0.42267, -0.54487, 0.18641, -0.45300]\nyhat = model.predict([row])\nprint(f'Predicted: {yhat}')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:55.053886Z","iopub.execute_input":"2021-09-03T19:23:55.054332Z","iopub.status.idle":"2021-09-03T19:23:59.405836Z","shell.execute_reply.started":"2021-09-03T19:23:55.054289Z","shell.execute_reply":"2021-09-03T19:23:59.404781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example first reports the shape of the dataset, then fits the model and evaluates it on the test dataset. Finally, a prediction is made for a single row of data.\n\nYour specific results will vary given the stochastic nature of the learning algorithm. Try running the example a few times.\n\n**What results did you get? Can you change the model to do better?**\n\nIn this case, we can see that the model achieved a classification accuracy of about 94 percent and then predicted a probability of 0.9 that the one row of data belongs to class 1.\n***","metadata":{}},{"cell_type":"markdown","source":"**MLP for Multiclass Classification**\n\nWe will use the Iris flowers multiclass classification dataset to demonstrate an `MLP` for multiclass classification.\n\nThis problem involves predicting the species of iris flower given measures of the flower.\n\nGiven that it is a multiclass classification, the model must have one node for each class in the output layer and use the `softmax activation function`. The loss function is the ‘`sparse_categorical_crossentropy`‘, which is appropriate for integer encoded class labels (e.g. 0 for one class, 1 for the next class, etc.)","metadata":{}},{"cell_type":"code","source":"# mlp for multiclass classification\nfrom numpy import argmax\nfrom pandas import read_csv\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\n\n# load the dataset\npath = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/iris.csv'\ndf = read_csv(path, header=None)\n\n# split into input and output columns\nX, y = df.values[:, :-1], df.values[:, -1]\n\n# ensure all data are floating point values\nX = X.astype('float32')\n\n# encode strings to integer\ny = LabelEncoder().fit_transform(y)\n\n# split into train and test datasets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)\nprint(X_train.shape, X_test.shape, y_train.shape, y_test.shape)\n\n# determine the number of input features\nn_features = X_train.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dense(8, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(3, activation='softmax'))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n\n# fit the model\nmodel.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)\n\n# evaluate the model\nloss, acc = model.evaluate(X_test, y_test, verbose=0)\nprint(f'Test Accuracy: {acc:.3f}')\n\n# make a prediction\nrow = [5.1, 3.5, 1.4, 0.2]\nyhat = model.predict([row])\nprint(f'Predicted: {yhat} (class={argmax(yhat)})')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:23:59.407328Z","iopub.execute_input":"2021-09-03T19:23:59.40758Z","iopub.status.idle":"2021-09-03T19:24:01.176326Z","shell.execute_reply.started":"2021-09-03T19:23:59.407544Z","shell.execute_reply":"2021-09-03T19:24:01.175244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example first reports the shape of the dataset, then fits the model and evaluates it on the test dataset. Finally, a prediction is made for a single row of data.\n\nYour specific results will vary given the stochastic nature of the learning algorithm. Try running the example a few times.\n\nIn this case, we can see that the model achieved a classification accuracy of about 98 percent and then predicted a probability of a row of data belonging to each class, although class 0 has the highest probability.\n***","metadata":{}},{"cell_type":"markdown","source":"**MLP for Regression**\n\nWe will use the Boston housing regression dataset to demonstrate an MLP for regression predictive modeling.\n\nThis problem involves predicting house value based on properties of the house and neighborhood.\n\nThis is a regression problem that involves predicting a single numerical value. As such, the output layer has a single node and uses the default or linear activation function (no activation function). The mean squared error (mse) loss is minimized when fitting the model.\n\nRecall that this is a regression, not classification; therefore, we cannot calculate classification accuracy. For more on this, see the tutorial:\n\n[Difference Between Classification and Regression in Machine Learning](https://machinelearningmastery.com/classification-versus-regression-in-machine-learning/)","metadata":{}},{"cell_type":"code","source":"# mlp for regression\nfrom numpy import sqrt\nfrom pandas import read_csv\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\n\n# load the dataset\npath = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.csv'\ndf = read_csv(path, header=None)\n\n# split into input and output columns\nX, y = df.values[:, :-1], df.values[:, -1]\n\n# split into train and test datasets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)\nprint(X_train.shape, X_test.shape, y_train.shape, y_test.shape)\n\n# determine the number of input features\nn_features = X_train.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dense(8, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(1))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='mse')\n\n# fit the model\nmodel.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)\n\n# evaluate the model\nerror = model.evaluate(X_test, y_test, verbose=0)\nprint(f'MSE: {error:.3f}, RMSE: {sqrt(error):.3f}')\n# make a prediction\nrow = [0.00632, 18.00, 2.310, 0, 0.5380, 6.5750, 65.20, 4.0900, 1, 296.0, 15.30, 396.90, 4.98]\nyhat = model.predict([row])\nprint(f'Predicted: {yhat}')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:24:01.178173Z","iopub.execute_input":"2021-09-03T19:24:01.178809Z","iopub.status.idle":"2021-09-03T19:24:04.672708Z","shell.execute_reply.started":"2021-09-03T19:24:01.178748Z","shell.execute_reply":"2021-09-03T19:24:04.671787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example first reports the shape of the dataset then fits the model and evaluates it on the test dataset. Finally, a prediction is made for a single row of data.\n\nYour specific results will vary given the stochastic nature of the learning algorithm. Try running the example a few times\n\nIn this case, we can see that the model achieved an MSE of about 60 which is an RMSE of about 7 (units are thousands of dollars). A value of about 30 is then predicted for the single example.\n***","metadata":{}},{"cell_type":"markdown","source":"## 3.2 Develop Convolutional Neural Network Models\n\n`Convolutional Neural Networks`, or `CNNs` for short, are a type of network designed for image input.\n\nThey are comprised of models with convolutional layers that extract features (called feature maps) and pooling layers that distill features down to the most salient elements.\n\n`CNNs` are most well-suited to image classification tasks, although they can be used on a wide array of tasks that take images as input.\n\nA popular image classification task is the `MNIST handwritten digit classification`. It involves tens of thousands of handwritten digits that must be classified as a number between 0 and 9.\n\nThe `tf.keras API` provides a convenience function to download and load this dataset directly.","metadata":{}},{"cell_type":"code","source":"# example of loading and plotting the mnist dataset\nfrom tensorflow.keras.datasets.mnist import load_data\nfrom matplotlib import pyplot\n%matplotlib inline\n\n# load dataset\n(trainX, trainy), (testX, testy) = load_data()\n\n# summarize loaded dataset\nprint(f'Train: X={trainX.shape}, y={trainy.shape}')\nprint(f'Test: X={testX.shape}, y={testy.shape}')\n\npyplot.figure(figsize=(12,12))\n# plot first few images\nfor i in range(25):\n\t# define subplot\n\tpyplot.subplot(5, 5, i+1)\n\t# plot raw pixel data\n\tpyplot.imshow(trainX[i], cmap=pyplot.get_cmap('gray'))\n# show the figure\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:24:04.674397Z","iopub.execute_input":"2021-09-03T19:24:04.674761Z","iopub.status.idle":"2021-09-03T19:24:08.501926Z","shell.execute_reply.started":"2021-09-03T19:24:04.674693Z","shell.execute_reply":"2021-09-03T19:24:08.500921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example loads the `MNIST dataset`, then summarizes the default train and test datasets.\n\nA plot is then created showing a grid of examples of handwritten images in the training dataset.\n\nWe can train a `CNN model` to classify the images in the MNIST dataset.\n\nNote that the images are arrays of grayscale pixel data; therefore, we must add a channel dimension to the data before we can use the images as input to the model. The reason is that `CNN models` expect images in a channels-last format, that is each example to the network has the dimensions `[rows, columns, channels]`, where channels represent the color channels of the image data.\n\nIt is also a good idea to scale the pixel values from the default range of `0-255` to `0-1` when training a `CNN`. ","metadata":{}},{"cell_type":"code","source":"# example of a cnn for image classification\nfrom numpy import unique\nfrom numpy import argmax\nfrom tensorflow.keras.datasets.mnist import load_data\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Conv2D\nfrom tensorflow.keras.layers import MaxPooling2D\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.layers import Dropout\n\n# load dataset\n(x_train, y_train), (x_test, y_test) = load_data()\n\n# reshape data to have a single channel\nx_train = x_train.reshape((x_train.shape[0], x_train.shape[1], x_train.shape[2], 1))\nx_test = x_test.reshape((x_test.shape[0], x_test.shape[1], x_test.shape[2], 1))\n\n# determine the shape of the input images\nin_shape = x_train.shape[1:]\n\n# determine the number of classes\nn_classes = len(unique(y_train))\nprint(in_shape, n_classes)\n\n# normalize pixel values\nx_train = x_train.astype('float32') / 255.0\nx_test = x_test.astype('float32') / 255.0\n\n# define model\nmodel = Sequential()\nmodel.add(Conv2D(32, (3,3), activation='relu', kernel_initializer='he_uniform', input_shape=in_shape))\nmodel.add(MaxPooling2D((2, 2)))\nmodel.add(Flatten())\nmodel.add(Dense(100, activation='relu', kernel_initializer='he_uniform'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(n_classes, activation='softmax'))\n\n# define loss and optimizer\nmodel.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n\n# fit the model\nmodel.fit(x_train, y_train, epochs=10, batch_size=128, verbose=0)\n\n# evaluate the model\nloss, acc = model.evaluate(x_test, y_test, verbose=0)\nprint(f'Accuracy: {acc:.3f}')\n\n# make a prediction\nimage = x_train[0]\nyhat = model.predict([[image]])\nprint(f'Predicted: class={argmax(yhat)}')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:24:08.503638Z","iopub.execute_input":"2021-09-03T19:24:08.504209Z","iopub.status.idle":"2021-09-03T19:26:19.064072Z","shell.execute_reply.started":"2021-09-03T19:24:08.504148Z","shell.execute_reply":"2021-09-03T19:26:19.063173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example first reports the shape of the dataset, then fits the model and evaluates it on the test dataset. Finally, a prediction is made for a single image.\n\nYour specific results will vary given the stochastic nature of the learning algorithm. Try running the example a few times.\n\nFirst, the shape of each image is reported along with the number of classes; we can see that each image is `28×28` pixels and there are `10` classes as we expected.\n\nIn this case, we can see that the model achieved a classification accuracy of about `98` percent on the test dataset. We can then see that the model predicted class `5` for the first image in the training set.\n***","metadata":{}},{"cell_type":"markdown","source":"## 3.3 Develop Recurrent Neural Network Models\n\n`Recurrent Neural Networks`, or `RNNs` for short, are designed to operate upon sequences of data.\n\nThey have proven to be very effective for natural language processing problems where sequences of text are provided as input to the model. `RNNs` have also seen some modest success for time series forecasting and speech recognition.\n\nThe most popular type of `RNN` is the Long Short-Term Memory network, or LSTM for short. LSTMs can be used in a model to accept a sequence of input data and make a prediction, such as assign a class label or predict a numerical value like the next value or values in the sequence.\n\nWe will use the car sales dataset to demonstrate an LSTM RNN for univariate time series forecasting.\n\nThis problem involves predicting the number of car sales per month.\n\nWe will frame the problem to take a window of the last five months of data to predict the current month’s data.\n\nTo achieve this, we will define a new function named `split_sequence()` that will split the input sequence into windows of data appropriate for fitting a supervised learning model, like an LSTM.\n\nWe will use the last `12 months` of data as the test dataset.\n\nLSTMs expect each sample in the dataset to have two dimensions; the first is the number of time steps (in this case it is `5`), and the second is the number of observations per time step (in this case it is `1`).\n\nBecause it is a regression type problem, we will use a linear activation function (no activation\nfunction) in the output layer and optimize the `mean squared error loss` function. We will also evaluate the model using the `mean absolute error (MAE)` metric.","metadata":{}},{"cell_type":"code","source":"# lstm for time series forecasting\nfrom numpy import sqrt\nfrom numpy import asarray\nfrom pandas import read_csv\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import LSTM\n\n# split a univariate sequence into samples\ndef split_sequence(sequence, n_steps):\n\tX, y = list(), list()\n\tfor i in range(len(sequence)):\n\t\t# find the end of this pattern\n\t\tend_ix = i + n_steps\n\t\t# check if we are beyond the sequence\n\t\tif end_ix > len(sequence)-1:\n\t\t\tbreak\n\t\t# gather input and output parts of the pattern\n\t\tseq_x, seq_y = sequence[i:end_ix], sequence[end_ix]\n\t\tX.append(seq_x)\n\t\ty.append(seq_y)\n\treturn asarray(X), asarray(y)\n\n# load the dataset\npath = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/monthly-car-sales.csv'\ndf = read_csv(path, header=0, index_col=0, squeeze=True)\n\n# retrieve the values\nvalues = df.values.astype('float32')\n\n# specify the window size\nn_steps = 5\n\n# split into samples\nX, y = split_sequence(values, n_steps)\n\n# reshape into [samples, timesteps, features]\nX = X.reshape((X.shape[0], X.shape[1], 1))\n\n# split into train/test\nn_test = 12\nX_train, X_test, y_train, y_test = X[:-n_test], X[-n_test:], y[:-n_test], y[-n_test:]\nprint(X_train.shape, X_test.shape, y_train.shape, y_test.shape)\n\n# define model\nmodel = Sequential()\nmodel.add(LSTM(100, activation='relu', kernel_initializer='he_normal', input_shape=(n_steps,1)))\nmodel.add(Dense(50, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(50, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(1))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='mse', metrics=['mae'])\n\n# fit the model\nmodel.fit(X_train, y_train, epochs=350, batch_size=32, verbose=0, validation_data=(X_test, y_test))\n\n# evaluate the model\nmse, mae = model.evaluate(X_test, y_test, verbose=0)\nprint(f'MSE: {mse:.3f}, RMSE: {sqrt(mse):.3f}, MAE: {mae:.3f}')\n\n# make a prediction\nrow = asarray([18024.0, 16722.0, 14385.0, 21342.0, 17180.0]).reshape((1, n_steps, 1))\nyhat = model.predict(row)\nprint(f'Predicted: {yhat}')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:19.06545Z","iopub.execute_input":"2021-09-03T19:26:19.065707Z","iopub.status.idle":"2021-09-03T19:26:29.854123Z","shell.execute_reply.started":"2021-09-03T19:26:19.065664Z","shell.execute_reply":"2021-09-03T19:26:29.853053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example first reports the shape of the dataset, then fits the model and evaluates it on the test dataset. Finally, a prediction is made for a single example.\n\nYour specific results will vary given the stochastic nature of the learning algorithm. Try running the example a few times.\n\nFirst, the shape of the train and test datasets is displayed, confirming that the last 12 examples are used for model evaluation.\n\nIn this case, the model achieved an MAE of about 2,800 and predicted the next value in the sequence from the test set as 13,199, where the expected value is 14,577 (pretty close).\n***","metadata":{}},{"cell_type":"markdown","source":"# 4. How to Use Advanced Model Features\n\nIn this section, you will discover how to use some of the slightly more advanced model features, such as reviewing learning curves and saving models for later use.\n\n## 4.1 How to Visualize a Deep Learning Model\nThe architecture of deep learning models can quickly become large and complex.\n\nAs such, it is important to have a clear idea of the connections and data flow in your model. This is especially important if you are using the functional API to ensure you have indeed connected the layers of the model in the way you intended.\n\nThere are two tools you can use to visualize your model: a text description and a plot.\n\n**Model Text Description**\n\nA text description of your model can be displayed by calling the `summary()` function on your model.","metadata":{}},{"cell_type":"code","source":"# example of summarizing a model\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(8,)))\nmodel.add(Dense(8, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# summarize the model\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:29.855483Z","iopub.execute_input":"2021-09-03T19:26:29.855731Z","iopub.status.idle":"2021-09-03T19:26:29.892468Z","shell.execute_reply.started":"2021-09-03T19:26:29.855695Z","shell.execute_reply":"2021-09-03T19:26:29.89135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example prints a summary of each layer, as well as a total summary.\n\nThis is an invaluable diagnostic for checking the output shapes and number of parameters (weights) in your model.\n\n**Model Architecture Plot**\n\nYou can create a plot of your model by calling the plot_model() function.\n\nThis will create an image file that contains a box and line diagram of the layers in your model.\n\nThe example below creates a small three-layer model and saves a plot of the model architecture to ‘model.png‘ that includes input and output shapes.","metadata":{}},{"cell_type":"code","source":"# example of plotting a model\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.utils import plot_model\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(8,)))\nmodel.add(Dense(8, activation='relu', kernel_initializer='he_normal'))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# summarize the model\nplot_model(model, 'model.png', show_shapes=True)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:29.893656Z","iopub.execute_input":"2021-09-03T19:26:29.893907Z","iopub.status.idle":"2021-09-03T19:26:30.299982Z","shell.execute_reply.started":"2021-09-03T19:26:29.893865Z","shell.execute_reply":"2021-09-03T19:26:30.298952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 How to Plot Model Learning Curves\nLearning curves are a plot of neural network model performance over time, such as calculated at the end of each training epoch.\n\nPlots of learning curves provide insight into the learning dynamics of the model, such as whether the model is learning well, whether it is underfitting the training dataset, or whether it is overfitting the training dataset.\n\nYou can easily create learning curves for your deep learning models.\n\nFirst, you must update your call to the fit function to include reference to a validation dataset. This is a portion of the training set not used to fit the model, and is instead used to evaluate the performance of the model during training.\n\nYou can split the data manually and specify the validation_data argument, or you can use the `validation_split` argument and specify a percentage split of the training dataset and let the API perform the split for you. The latter is simpler for now.\n\nThe fit function will return a history object that contains a trace of performance metrics recorded at the end of each training epoch. This includes the chosen loss function and each configured metric, such as accuracy, and each loss and metric is calculated for the training and validation datasets.\n\nA learning curve is a plot of the loss on the training dataset and the validation dataset. We can create this plot from the history object using the `Matplotlib` library.\n\nThe example below fits a small `neural network` on a synthetic binary classification problem. A validation split of `30` percent is used to evaluate the model during training and the `cross-entropy loss` on the train and validation datasets are then graphed using a line plot.","metadata":{}},{"cell_type":"code","source":"# example of plotting learning curves\nfrom sklearn.datasets import make_classification\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.optimizers import SGD\nfrom matplotlib import pyplot\n\n# create the dataset\nX, y = make_classification(n_samples=1000, n_classes=2, random_state=1)\n\n# determine the number of input features\nn_features = X.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nsgd = SGD(learning_rate=0.001, momentum=0.8)\nmodel.compile(optimizer=sgd, loss='binary_crossentropy')\n\n# fit the model\nhistory = model.fit(X, y, epochs=100, batch_size=32, verbose=0, validation_split=0.3)\n\n# plot learning curves\npyplot.title('Learning Curves')\npyplot.xlabel('Epoch')\npyplot.ylabel('Cross Entropy')\npyplot.plot(history.history['loss'], label='train')\npyplot.plot(history.history['val_loss'], label='val')\npyplot.legend()\npyplot.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:30.302055Z","iopub.execute_input":"2021-09-03T19:26:30.302381Z","iopub.status.idle":"2021-09-03T19:26:35.616703Z","shell.execute_reply.started":"2021-09-03T19:26:30.302323Z","shell.execute_reply":"2021-09-03T19:26:35.613873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example fits the model on the dataset. At the end of the run, the history object is returned and used as the basis for creating the line plot.\n\nThe `cross-entropy loss` for the training dataset is accessed via the ‘`loss`‘ key and the loss on the validation dataset is accessed via the ‘`val_loss`‘ key on the history attribute of the history object.\n***","metadata":{}},{"cell_type":"markdown","source":"## 4.3 How to Save and Load Your Model\n\nTraining and evaluating models is great, but we may want to use a model later without retraining it each time.\n\nThis can be achieved by saving the model to file and later loading it and using it to make predictions.\n\nThis can be achieved using the `save()` function on the model to save the model. It can be loaded later using the `load_model()` function.\n\nThe model is saved in `H5 format`, an efficient array storage format. As such, you must ensure that the `h5py` library is installed on your workstation. This can be achieved using pip; for example:","metadata":{}},{"cell_type":"code","source":"pip install h5py","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:35.618267Z","iopub.execute_input":"2021-09-03T19:26:35.618639Z","iopub.status.idle":"2021-09-03T19:26:43.054067Z","shell.execute_reply.started":"2021-09-03T19:26:35.618562Z","shell.execute_reply":"2021-09-03T19:26:43.052925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# example of saving a fit model\nfrom sklearn.datasets import make_classification\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.optimizers import SGD\n\n# create the dataset\nX, y = make_classification(n_samples=1000, n_features=4, n_classes=2, random_state=1)\n\n# determine the number of input features\nn_features = X.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nsgd = SGD(learning_rate=0.001, momentum=0.8)\nmodel.compile(optimizer=sgd, loss='binary_crossentropy')\n\n# fit the model\nmodel.fit(X, y, epochs=100, batch_size=32, verbose=0, validation_split=0.3)\n\n# save model to file\nmodel.save('model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:43.055586Z","iopub.execute_input":"2021-09-03T19:26:43.055845Z","iopub.status.idle":"2021-09-03T19:26:48.360589Z","shell.execute_reply.started":"2021-09-03T19:26:43.055801Z","shell.execute_reply":"2021-09-03T19:26:48.35965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example fits the model and saves it to file with the name ‘`model.h5`‘.\n\nWe can then load the model and use it to make a prediction, or continue training it, or do whatever we wish with it.\n\nThe example below loads the model and uses it to make a prediction.","metadata":{}},{"cell_type":"code","source":"# example of loading a saved model\nfrom sklearn.datasets import make_classification\nfrom tensorflow.keras.models import load_model\n\n# create the dataset\nX, y = make_classification(n_samples=1000, n_features=4, n_classes=2, random_state=1)\n\n# load the model from file\nmodel = load_model('model.h5')\n\n# make a prediction\nrow = [1.91518414, 1.14995454, -1.52847073, 0.79430654]\nyhat = model.predict([row])\nprint(f'Predicted: {yhat}')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:48.361698Z","iopub.execute_input":"2021-09-03T19:26:48.36194Z","iopub.status.idle":"2021-09-03T19:26:48.698318Z","shell.execute_reply.started":"2021-09-03T19:26:48.3619Z","shell.execute_reply":"2021-09-03T19:26:48.697618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Running the example loads the image from file, then uses it to make a prediction on a new row of data and prints the result.\n***","metadata":{}},{"cell_type":"markdown","source":"# 5. How to Get Better Model Performance\n\nIn this section, you will discover some of the techniques that you can use to improve the performance of your deep learning models.\n\nA big part of improving deep learning performance involves avoiding overfitting by slowing down the learning process or stopping the learning process at the right time.\n\n## 5.1 How to Reduce Overfitting With Dropout\nDropout is a clever regularization method that reduces overfitting of the training dataset and makes the model more robust.\n\nThis is achieved during training, where some number of layer outputs are randomly ignored or “`dropped out`.” This has the effect of making the layer look like – and be treated like – a layer with a different number of nodes and connectivity to the prior layer.\n\nDropout has the effect of making the training process noisy, forcing nodes within a layer to probabilistically take on more or less responsibility for the inputs.\n\nYou can add dropout to your models as a new layer prior to the layer that you want to have input connections `dropped-out`.\n\nThis involves adding a layer called `Dropout()` that takes an argument that specifies the probability that each output from the previous to drop. E.g. `0.4` means `40%` percent of inputs will be dropped each update to the model.\n\nYou can add Dropout layers in `MLP`, `CNN`, and `RNN` models, although there are also specialized versions of dropout for use with `CNN` and `RNN` models that you might also want to explore.\n\nThe example below fits a small `neural network` model on a synthetic binary classification problem.\n\nA dropout layer with `50` percent dropout is inserted between the first hidden layer and the output layer.","metadata":{}},{"cell_type":"code","source":"# example of using dropout\nfrom sklearn.datasets import make_classification\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Dropout\nfrom matplotlib import pyplot\n\n# create the dataset\nX, y = make_classification(n_samples=1000, n_classes=2, random_state=1)\n\n# determine the number of input features\nn_features = X.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy')\n\n# fit the model\nmodel.fit(X, y, epochs=100, batch_size=32, verbose=0)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:48.699536Z","iopub.execute_input":"2021-09-03T19:26:48.699953Z","iopub.status.idle":"2021-09-03T19:26:53.768701Z","shell.execute_reply.started":"2021-09-03T19:26:48.699904Z","shell.execute_reply":"2021-09-03T19:26:53.7675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 How to Accelerate Training With Batch Normalization\n\nThe scale and distribution of inputs to a layer can greatly impact how easy or quickly that layer can be trained.\n\nThis is generally why it is a good idea to scale input data prior to modeling it with a neural network model.\n\nBatch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.\n\nYou can use batch normalization in your network by adding a batch normalization layer prior to the layer that you wish to have standardized inputs. You can use batch normalization with MLP, CNN, and RNN models.\n\nThis can be achieved by adding the `BatchNormalization` layer directly.\n\nThe example below defines a small MLP network for a binary classification prediction problem with a batch normalization layer between the first hidden layer and the output layer.","metadata":{}},{"cell_type":"code","source":"# example of using batch normalization\nfrom sklearn.datasets import make_classification\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import BatchNormalization\nfrom matplotlib import pyplot\n\n# create the dataset\nX, y = make_classification(n_samples=1000, n_classes=2, random_state=1)\n\n# determine the number of input features\nn_features = X.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(BatchNormalization())\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy')\n\n# fit the model\nmodel.fit(X, y, epochs=100, batch_size=32, verbose=0)","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:26:53.77035Z","iopub.execute_input":"2021-09-03T19:26:53.770729Z","iopub.status.idle":"2021-09-03T19:26:59.311222Z","shell.execute_reply.started":"2021-09-03T19:26:53.77066Z","shell.execute_reply":"2021-09-03T19:26:59.310209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3 How to Halt Training at the Right Time With Early Stopping\n\nNeural networks are challenging to train. Too little training and the model is underfit; too much training and the model overfits the training dataset. Both cases result in a model that is less effective than it could be.\n\nOne approach to solving this problem is to use early stopping. This involves monitoring the loss on the training dataset and a validation dataset (a subset of the training set not used to fit the model). As soon as loss for the validation set starts to show signs of overfitting, the training process can be stopped.\n\nEarly stopping can be used with your model by first ensuring that you have a validation dataset. You can define the validation dataset manually via the `validation_data` argument to the `fit()` function, or you can use the `validation_split` and specify the amount of the training dataset to hold back for validation.\n\nYou can then define an `EarlyStopping` and instruct it on which performance measure to monitor, such as ‘`val_loss`‘ for loss on the validation dataset, and the number of epochs to observed overfitting before taking action, e.g. `5`.\n\nThis configured `EarlyStopping` callback can then be provided to the `fit()` function via the “`callbacks`” argument that takes a list of callbacks.\n\nThis allows you to set the number of epochs to a large number and be confident that training will end as soon as the model starts overfitting. You might also like to create a learning curve to discover more insights into the learning dynamics of the run and when training was halted.\n\nThe example below demonstrates a small neural network on a synthetic binary classification problem that uses early stopping to halt training as soon as the model starts overfitting (after about 50 epochs).","metadata":{}},{"cell_type":"code","source":"# example of using early stopping\nfrom sklearn.datasets import make_classification\nfrom tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.callbacks import EarlyStopping\n\n# create the dataset\nX, y = make_classification(n_samples=1000, n_classes=2, random_state=1)\n\n# determine the number of input features\nn_features = X.shape[1]\n\n# define model\nmodel = Sequential()\nmodel.add(Dense(10, activation='relu', kernel_initializer='he_normal', input_shape=(n_features,)))\nmodel.add(Dense(1, activation='sigmoid'))\n\n# compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy')\n\n# configure early stopping\nes = EarlyStopping(monitor='val_loss', patience=5)\n\n# fit the model\nhistory = model.fit(X, y, epochs=200, batch_size=32, validation_split=0.3, callbacks=[es])","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-03T19:26:59.312845Z","iopub.execute_input":"2021-09-03T19:26:59.313224Z","iopub.status.idle":"2021-09-03T19:27:02.193719Z","shell.execute_reply.started":"2021-09-03T19:26:59.313153Z","shell.execute_reply":"2021-09-03T19:27:02.192769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.plot(history.history['loss'], label='Loss')\nplt.plot(history.history['val_loss'], label='val_Loss')\nplt.legend()\nplt.title('Loss evolution')","metadata":{"execution":{"iopub.status.busy":"2021-09-03T19:27:02.194997Z","iopub.execute_input":"2021-09-03T19:27:02.195272Z","iopub.status.idle":"2021-09-03T19:27:02.498616Z","shell.execute_reply.started":"2021-09-03T19:27:02.195228Z","shell.execute_reply":"2021-09-03T19:27:02.497251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Summary\nIn this tutorial, you discovered a step-by-step guide to developing deep learning models in TensorFlow using the tf.keras API.\n\nSpecifically, you learned:\n\n- The difference between Keras and tf.keras and how to install and confirm TensorFlow is working.\n- The 5-step life-cycle of tf.keras models and how to use the sequential and functional APIs.\n- How to develop MLP, CNN, and RNN models with tf.keras for regression, classification, and time series forecasting.\n- How to use the advanced features of the tf.keras API to inspect and diagnose your model.\n- How to improve the performance of your tf.keras model by reducing overfitting and accelerating training.","metadata":{}}]}