{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AUTOENCODERS","metadata":{}},{"cell_type":"markdown","source":"## Introduction\n\nAn autoencoder, also known as autoassociator or Diabolo networks, is an artificial neural network employed to recreate the given input. It takes a set of <b>unlabeled</b> inputs, encodes them and then tries to extract the most valuable information from them. They are used for feature extraction, learning generative models of data, dimensionality reduction and can be used for compression. \n\nAutoencoders are based on Restricted Boltzmann Machines, are employed in some of the largest deep learning applications. They are the building blocks of Deep Belief Networks (DBN).\n\n<img src=\"https://ibm.box.com/shared/static/xlkv9v7xzxhjww681dq3h1pydxcm4ktp.png\" width=400>","metadata":{}},{"cell_type":"markdown","source":"## Autoencoder Structure \n\n<img src=\"https://res.cloudinary.com/dpyleb8ih/image/upload/v1560249044/deep_autoencoder.png\">\n\nAn autoencoder can be divided in two parts, the <b>encoder</b> and the <b>decoder</b>.\n\nThe encoder needs to compress the representation of an input. In this case we are going to reduce the dimension. The decoder works like encoder network in reverse. It works to recreate the input, as closely as possible. This plays an important role during training, because it forces the autoencoder to select the most important features in the compressed representation.","metadata":{}},{"cell_type":"markdown","source":"<h2>Training: Loss function</h2>\n\nAn autoencoder uses the Loss function to properly train the network. The Loss function will calculate the differences between our output and the expected results. After that, we can minimize this error with gradient descent. There are more than one type of Loss function, it depends on the type of data.\n\n**Binary Values:**\n$$l(f(x)) = - \\sum_{k} (x_k log(\\hat{x}_k) + (1 - x_k) \\log (1 - \\hat{x}_k) \\ )$$\n\nFor binary values, we can use an equation based on the sum of Bernoulli's cross-entropy.  $x_k$ is one of our inputs and $\\hat{x}_k$ is the respective output. We use this function so that if $x_k$ equals to one, we want to push $\\hat{x}_k$ as close as possible to one. The same if $x_k$ equals to zero. If the value is one, we just need to calculate the first part of the formula, that is, $- x_k log(\\hat{x}_k)$. Which, turns out to just calculate $- log(\\hat{x}_k)$. And if the value is zero, we need to calculate just the second part, $(1 - x_k) \\log (1 - \\hat{x}_k) \\ )$ - which turns out to be $log (1 - \\hat{x}_k) $.\n\n**Real values:**\n$$l(f(x)) = - \\frac{1}{2}\\sum_{k} (\\hat{x}_k- x_k \\ )^2$$\n\nAs the above function would behave badly with inputs that are not 0 or 1, we can use the sum of squared differences for our Loss function.\n\nAs it was with the above example, $x_k$ is one of our inputs and $\\hat{x}_k$ is the respective output, and we want to make our output as similar as possible to our input.","metadata":{}},{"cell_type":"markdown","source":"## MNIST","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport tensorflow.keras as kr\n%matplotlib inline\ntf.__version__","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:03:18.574286Z","iopub.execute_input":"2022-05-20T13:03:18.574814Z","iopub.status.idle":"2022-05-20T13:03:19.546889Z","shell.execute_reply.started":"2022-05-20T13:03:18.57475Z","shell.execute_reply":"2022-05-20T13:03:19.546029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's give the parameters that are going to be used by our NN.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# MNIST Dataset parameters.\nnum_features = 784 # data features (img shape: 28*28).\n\n# Training parameters.\nbatch_size = 128\nepochs = 50\n\n# Network Parameters\nhidden_1 = 128 # 1st layer num features.\nhidden_2 = 64 # 2nd layer num features (the latent dim).","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:03:19.548158Z","iopub.execute_input":"2022-05-20T13:03:19.548425Z","iopub.status.idle":"2022-05-20T13:03:19.552538Z","shell.execute_reply.started":"2022-05-20T13:03:19.548372Z","shell.execute_reply":"2022-05-20T13:03:19.551894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.datasets import mnist, fashion_mnist\n\ndef load_data(choice='mnist', labels=False):\n    if choice not in ['mnist', 'fashion_mnist']:\n        raise ('Choices are mnist and fashion_mnist')\n    \n    if choice is 'mnist':\n        (X_train, y_train), (X_test, y_test) = mnist.load_data()\n    else:\n        (X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()\n    \n    X_train, X_test = X_train / 255., X_test / 255.\n    X_train, X_test = X_train.reshape([-1, 784]), X_test.reshape([-1, 784])\n    X_train = X_train.astype(np.float32, copy=False)\n    X_test = X_test.astype(np.float32, copy=False)\n    \n    if labels:\n        return (X_train, y_train), (X_test, y_test)\n    \n    return X_train, X_test\n\n\ndef plot_predictions(y_true, y_pred):    \n    f, ax = plt.subplots(2, 10, figsize=(15, 4))\n    for i in range(10):\n        ax[0][i].imshow(np.reshape(y_true[i], (28, 28)), aspect='auto')\n        ax[1][i].imshow(np.reshape(y_pred[i], (28, 28)), aspect='auto')\n    plt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:03:19.553308Z","iopub.execute_input":"2022-05-20T13:03:19.553762Z","iopub.status.idle":"2022-05-20T13:03:19.5663Z","shell.execute_reply.started":"2022-05-20T13:03:19.553724Z","shell.execute_reply":"2022-05-20T13:03:19.565267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_digits(X, y, encoder, batch_size=128):\n    \"\"\"Plots labels and MNIST digits as function of 2D latent vector\n\n    Parameters:\n    ----------\n    encoder: Model\n        A Keras Model instance\n    X: np.ndarray\n        Test data\n    y: np.ndarray\n        Test data labels\n    batch_size: int\n        Prediction batch size\n    \"\"\"\n    # display a 2D plot of the digit classes in the latent space\n    z_mean, _, _ = encoder.predict(X, batch_size=batch_size)\n    plt.figure(figsize=(12, 10))\n    plt.scatter(z_mean[:, 0], z_mean[:, 1], c=y)\n    plt.colorbar()\n    plt.xlabel(\"z[0] Latent Dimension\")\n    plt.ylabel(\"z[1] Latent Dimension\")\n    plt.show()\n    \n    \ndef generate_manifold(decoder):\n    \"\"\"Generates a manifold of MNIST digits from a random noisy data.\n\n    Parameters:\n    ----------\n    decoder: Model\n        A Keras Model instance\n    \"\"\"\n    \n    # display a 30x30 2D manifold of digits\n    n = 30\n    digit_size = 28\n    figure = np.zeros((digit_size * n, digit_size * n))\n    \n    # linearly spaced coordinates corresponding to the 2D plot\n    # of digit classes in the latent space\n    grid_x = np.linspace(-4, 4, n)\n    grid_y = np.linspace(-4, 4, n)[::-1]\n    \n    for i, yi in enumerate(grid_y):\n        for j, xi in enumerate(grid_x):\n            z_sample = np.array([[xi, yi]])\n            x_decoded = decoder.predict(z_sample)\n            digit = x_decoded[0].reshape(digit_size, digit_size)\n            figure[i * digit_size: (i + 1) * digit_size,\n                   j * digit_size: (j + 1) * digit_size] = digit        \n    \n    plt.figure(figsize=(10, 10))\n    start_range = digit_size // 2\n    end_range = n * digit_size + start_range + 1\n    pixel_range = np.arange(start_range, end_range, digit_size)\n    sample_range_x = np.round(grid_x, 1)\n    sample_range_y = np.round(grid_y, 1)\n    \n    plt.xticks(pixel_range, sample_range_x)\n    plt.yticks(pixel_range, sample_range_y)\n    plt.xlabel(\"z[0] Latent Dimension\")\n    plt.ylabel(\"z[1] Latent Dimension\")\n    plt.imshow(figure, cmap='Greys_r')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:03:19.568125Z","iopub.execute_input":"2022-05-20T13:03:19.56856Z","iopub.status.idle":"2022-05-20T13:03:19.579221Z","shell.execute_reply.started":"2022-05-20T13:03:19.568376Z","shell.execute_reply":"2022-05-20T13:03:19.578428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to create our encoder/decoder. For this, we are going to use sigmoidal functions. Sigmoidal functions delivers great results with this type of network. This is due to having a good derivative that is well-suited to backpropagation.","metadata":{}},{"cell_type":"markdown","source":"**Encoder**","metadata":{}},{"cell_type":"code","source":"inputs = kr.Input(shape=(num_features, ))\nencoder = kr.layers.Dense(hidden_1, activation='sigmoid')(inputs)\nencoder = kr.layers.Dense(hidden_1, activation='sigmoid')(encoder)\nencoder = kr.layers.Dense(hidden_1, activation='sigmoid')(encoder)\nencoder = kr.layers.Dense(hidden_2, activation='softmax')(encoder)\nencoder_model = kr.Model(inputs, encoder, name='encoder')\nencoder_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:24:23.502119Z","iopub.execute_input":"2022-05-20T13:24:23.502744Z","iopub.status.idle":"2022-05-20T13:24:23.585548Z","shell.execute_reply.started":"2022-05-20T13:24:23.502687Z","shell.execute_reply":"2022-05-20T13:24:23.584289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Decoder**","metadata":{}},{"cell_type":"code","source":"latent_dim = kr.Input(shape=(hidden_2, ))\ndecoder = kr.layers.Dense(hidden_1, activation='sigmoid')(latent_dim)\ndecoder = kr.layers.Dense(hidden_1, activation='sigmoid')(decoder)\ndecoder = kr.layers.Dense(hidden_1, activation='sigmoid')(decoder)\ndecoder = kr.layers.Dense(num_features, activation='sigmoid')(decoder)\ndecoder_model = kr.Model(latent_dim, decoder, name='decoder')\ndecoder_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:24:25.716436Z","iopub.execute_input":"2022-05-20T13:24:25.716751Z","iopub.status.idle":"2022-05-20T13:24:25.794999Z","shell.execute_reply.started":"2022-05-20T13:24:25.716692Z","shell.execute_reply":"2022-05-20T13:24:25.794121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Autoencoder Model**","metadata":{}},{"cell_type":"code","source":"outputs = decoder_model(encoder_model(inputs))\nmnist_model = kr.Model(inputs, outputs )\nmnist_model.compile(optimizer='adam', loss='mse')\nmnist_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:24:30.837696Z","iopub.execute_input":"2022-05-20T13:24:30.838008Z","iopub.status.idle":"2022-05-20T13:24:30.954985Z","shell.execute_reply.started":"2022-05-20T13:24:30.837954Z","shell.execute_reply":"2022-05-20T13:24:30.95381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test = load_data('mnist')\nmnist_model.fit(x=X_train, y=X_train, batch_size=batch_size, shuffle=False, epochs=110)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-05-20T15:42:46.985844Z","iopub.execute_input":"2022-05-20T15:42:46.986947Z","iopub.status.idle":"2022-05-20T15:50:09.923544Z","shell.execute_reply.started":"2022-05-20T15:42:46.986593Z","shell.execute_reply":"2022-05-20T15:50:09.922792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:41:21.423412Z","iopub.execute_input":"2022-05-20T13:41:21.423721Z","iopub.status.idle":"2022-05-20T13:41:24.775626Z","shell.execute_reply.started":"2022-05-20T13:41:21.42367Z","shell.execute_reply":"2022-05-20T13:41:24.774331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T13:58:25.484987Z","iopub.execute_input":"2022-05-20T13:58:25.485329Z","iopub.status.idle":"2022-05-20T13:58:28.255542Z","shell.execute_reply.started":"2022-05-20T13:58:25.48525Z","shell.execute_reply":"2022-05-20T13:58:28.254618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's simply visualize our graphs!","metadata":{}},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T14:05:59.26712Z","iopub.execute_input":"2022-05-20T14:05:59.267332Z","iopub.status.idle":"2022-05-20T14:06:02.488376Z","shell.execute_reply.started":"2022-05-20T14:05:59.267299Z","shell.execute_reply":"2022-05-20T14:06:02.487376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T14:16:43.4468Z","iopub.execute_input":"2022-05-20T14:16:43.447157Z","iopub.status.idle":"2022-05-20T14:16:46.502294Z","shell.execute_reply.started":"2022-05-20T14:16:43.447102Z","shell.execute_reply":"2022-05-20T14:16:46.501207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T14:26:28.043698Z","iopub.execute_input":"2022-05-20T14:26:28.044216Z","iopub.status.idle":"2022-05-20T14:26:31.381644Z","shell.execute_reply.started":"2022-05-20T14:26:28.04416Z","shell.execute_reply":"2022-05-20T14:26:31.380675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T14:35:52.393257Z","iopub.execute_input":"2022-05-20T14:35:52.393557Z","iopub.status.idle":"2022-05-20T14:35:55.500166Z","shell.execute_reply.started":"2022-05-20T14:35:52.393504Z","shell.execute_reply":"2022-05-20T14:35:55.498837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T14:43:51.530787Z","iopub.execute_input":"2022-05-20T14:43:51.530997Z","iopub.status.idle":"2022-05-20T14:43:54.595398Z","shell.execute_reply.started":"2022-05-20T14:43:51.530964Z","shell.execute_reply":"2022-05-20T14:43:54.594531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T14:52:30.915519Z","iopub.execute_input":"2022-05-20T14:52:30.915833Z","iopub.status.idle":"2022-05-20T14:52:34.054745Z","shell.execute_reply.started":"2022-05-20T14:52:30.915774Z","shell.execute_reply":"2022-05-20T14:52:34.053162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T15:02:10.946035Z","iopub.execute_input":"2022-05-20T15:02:10.946678Z","iopub.status.idle":"2022-05-20T15:02:14.383277Z","shell.execute_reply.started":"2022-05-20T15:02:10.94661Z","shell.execute_reply":"2022-05-20T15:02:14.381662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T15:10:51.341886Z","iopub.execute_input":"2022-05-20T15:10:51.342222Z","iopub.status.idle":"2022-05-20T15:10:54.659159Z","shell.execute_reply.started":"2022-05-20T15:10:51.342165Z","shell.execute_reply":"2022-05-20T15:10:54.658092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T15:19:37.816449Z","iopub.execute_input":"2022-05-20T15:19:37.816773Z","iopub.status.idle":"2022-05-20T15:19:41.131526Z","shell.execute_reply.started":"2022-05-20T15:19:37.816721Z","shell.execute_reply":"2022-05-20T15:19:41.130314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T15:30:40.247823Z","iopub.execute_input":"2022-05-20T15:30:40.248196Z","iopub.status.idle":"2022-05-20T15:30:43.798433Z","shell.execute_reply.started":"2022-05-20T15:30:40.248152Z","shell.execute_reply":"2022-05-20T15:30:43.797293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T15:40:17.179316Z","iopub.execute_input":"2022-05-20T15:40:17.179627Z","iopub.status.idle":"2022-05-20T15:40:20.393587Z","shell.execute_reply.started":"2022-05-20T15:40:17.179569Z","shell.execute_reply":"2022-05-20T15:40:20.392524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[10:20]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T15:32:19.149631Z","iopub.execute_input":"2022-05-20T15:32:19.149947Z","iopub.status.idle":"2022-05-20T15:32:22.653941Z","shell.execute_reply.started":"2022-05-20T15:32:19.149909Z","shell.execute_reply":"2022-05-20T15:32:22.652996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[10:20]\ny_pred = mnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T16:05:54.823195Z","iopub.execute_input":"2022-05-20T16:05:54.823604Z","iopub.status.idle":"2022-05-20T16:05:58.430637Z","shell.execute_reply.started":"2022-05-20T16:05:54.823558Z","shell.execute_reply":"2022-05-20T16:05:58.429667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"out = encoder_model.predict(y_true)\nout[0]","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:28:35.552099Z","iopub.execute_input":"2022-05-20T10:28:35.552658Z","iopub.status.idle":"2022-05-20T10:28:35.594942Z","shell.execute_reply.started":"2022-05-20T10:28:35.552594Z","shell.execute_reply":"2022-05-20T10:28:35.594109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fashion MNIST\n\nLet's try our network with fashion mnist to see how well it performs.","metadata":{}},{"cell_type":"code","source":"# Encoder\ninputs = kr.Input(shape=(num_features, ))\nencoder = kr.layers.Dense(hidden_1, activation='sigmoid')(inputs)\nencoder = kr.layers.Dense(hidden_2, activation='sigmoid')(encoder)\nencoder_model = kr.Model(inputs, encoder, name='encoder')\nencoder_model.summary()\n\n# Decoder\nlatent_dim = kr.Input(shape=(hidden_2, ))\ndecoder = kr.layers.Dense(hidden_1, activation='sigmoid')(latent_dim)\ndecoder = kr.layers.Dense(num_features, activation='sigmoid')(decoder)\ndecoder_model = kr.Model(latent_dim, decoder, name='decoder')\ndecoder_model.summary()\n\n# AE\noutputs = decoder_model(encoder_model(inputs))\nfmnist_model = kr.Model(inputs, outputs )\nfmnist_model.compile(optimizer='adam', loss='mse')\nfmnist_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:28:35.596477Z","iopub.execute_input":"2022-05-20T10:28:35.596843Z","iopub.status.idle":"2022-05-20T10:28:35.802658Z","shell.execute_reply.started":"2022-05-20T10:28:35.596746Z","shell.execute_reply":"2022-05-20T10:28:35.801794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test = load_data('fashion_mnist')\nfmnist_model.fit(x=X_train, y=X_train, batch_size=batch_size, shuffle=False, epochs=epochs)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-05-20T10:28:35.805615Z","iopub.execute_input":"2022-05-20T10:28:35.805944Z","iopub.status.idle":"2022-05-20T10:31:19.497066Z","shell.execute_reply.started":"2022-05-20T10:28:35.805889Z","shell.execute_reply":"2022-05-20T10:31:19.496014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = X_test[:10]\ny_pred = fmnist_model.predict(y_true)\nplot_predictions(y_true, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:19.498313Z","iopub.execute_input":"2022-05-20T10:31:19.498539Z","iopub.status.idle":"2022-05-20T10:31:22.910505Z","shell.execute_reply.started":"2022-05-20T10:31:19.498498Z","shell.execute_reply":"2022-05-20T10:31:22.909462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As can be seen construction was okay not perfect due to some noise in the images. Increasing layer sizes may help.","metadata":{}},{"cell_type":"markdown","source":"## Variational Autoencoder\n\n<img src=\"https://res.cloudinary.com/dpyleb8ih/image/upload/v1560249262/vae-gaussian.png\">","metadata":{}},{"cell_type":"markdown","source":"We're trying to build a generative model here, not just a fuzzy data structure that can \"memorize\" images. We can't generate anything yet, since we don't know how to create latent vectors other than encoding them from images.\n\nThere's a simple solution here. We add a constraint on the encoding network, that forces it to generate latent vectors that roughly follow a unit gaussian distribution. It is this constraint that separates a variational autoencoder from a standard one.\n\nGenerating new images is now easy: all we need to do is sample a latent vector from the unit gaussian and pass it into the decoder.\n\nIn practice, there's a tradeoff between how accurate our network can be and how close its latent variables can match the unit gaussian distribution.\n\nWe let the network decide this itself. For our loss term, we sum up two separate losses: the generative loss, which is a mean squared error that measures how accurately the network reconstructed the images, and a latent loss, which is the KL divergence that measures how closely the latent variables match a unit gaussian.\n我们在这里尝试建立一个生成模型，而不仅仅是一个可以“记忆”图像的模糊数据结构。我们还不能生成任何东西，因为除了从图像中编码它们之外，我们不知道如何创建潜在向量。\n\n这里有一个简单的解决方案。我们在编码网络上添加一个约束，强制它生成大致遵循单位高斯分布的潜在向量。正是这种约束将变分自动编码器与标准自动编码器区分开来。\n\n生成新图像现在很容易：我们需要做的就是从单位高斯中采样一个潜在向量并将其传递给解码器。\n\n在实践中，我们的网络可以有多准确，以及它的潜在变量与单位高斯分布的匹配程度有多接近，需要权衡取舍。\n\n我们让网络自己决定。对于我们的损失项，我们总结了两个单独的损失：生成损失，它是衡量网络重建图像准确程度的均方误差，以及潜在损失，它是衡量潜在变量匹配程度的 KL 散度单位高斯。[ref](http://kvfrans.com/variational-autoencoders-explained/)","metadata":{}},{"cell_type":"code","source":"def sampling(args):\n    \"\"\"Reparameterization trick. Instead of sampling from Q(z|X), \n    sample eps = N(0,I) z = z_mean + sqrt(var)*eps.\n\n    Parameters:\n    -----------\n    args: list of Tensors\n        Mean and log of variance of Q(z|X)\n\n    Returns\n    -------\n    z: Tensor\n        Sampled latent vector\n    \"\"\"\n\n    z_mean, z_log_var = args\n    eps = tf.random_normal(tf.shape(z_log_var), dtype=tf.float32, mean=0., stddev=1.0, name='epsilon')\n    z = z_mean + tf.exp(z_log_var / 2) * eps\n    return z","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.912288Z","iopub.execute_input":"2022-05-20T10:31:22.913002Z","iopub.status.idle":"2022-05-20T10:31:22.920867Z","shell.execute_reply.started":"2022-05-20T10:31:22.912899Z","shell.execute_reply":"2022-05-20T10:31:22.920178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hidden_dim = 512\nlatent_dim = 2  # The bigger this is, more accurate the network is but 2 is for illustration purposes.","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.924075Z","iopub.execute_input":"2022-05-20T10:31:22.924517Z","iopub.status.idle":"2022-05-20T10:31:22.93318Z","shell.execute_reply.started":"2022-05-20T10:31:22.924473Z","shell.execute_reply":"2022-05-20T10:31:22.932435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tt = tf.random.normal([4], 0, 1, tf.float32)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.934418Z","iopub.execute_input":"2022-05-20T10:31:22.934881Z","iopub.status.idle":"2022-05-20T10:31:22.951789Z","shell.execute_reply.started":"2022-05-20T10:31:22.934835Z","shell.execute_reply":"2022-05-20T10:31:22.950748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.random.set_seed(5);\ntf.random.normal([2,2], 0, 1, tf.float32, seed=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.953146Z","iopub.execute_input":"2022-05-20T10:31:22.953639Z","iopub.status.idle":"2022-05-20T10:31:23.058011Z","shell.execute_reply.started":"2022-05-20T10:31:22.953594Z","shell.execute_reply":"2022-05-20T10:31:22.972908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tt[1]","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.973635Z","iopub.status.idle":"2022-05-20T10:31:22.974262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Encoder**","metadata":{}},{"cell_type":"code","source":"inputs = kr.layers.Input(shape=(num_features, ), name='input')\nx = kr.layers.Dense(hidden_dim, activation='relu')(inputs)\nz_mean = kr.layers.Dense(latent_dim, name='z_mean')(x)\nz_log_var = kr.layers.Dense(latent_dim, name='z_log_var')(x)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.97525Z","iopub.status.idle":"2022-05-20T10:31:22.975921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Sampling Layer**  \nUse reparameterization trick to push the sampling out as input","metadata":{}},{"cell_type":"code","source":"z = kr.layers.Lambda(sampling, name='z')([z_mean, z_log_var])\n\n# instantiate encoder model\nencoder = kr.Model(inputs, [z_mean, z_log_var, z], name='encoder')\nencoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.976857Z","iopub.status.idle":"2022-05-20T10:31:22.977398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Decoder**","metadata":{}},{"cell_type":"code","source":"latent_inputs = kr.layers.Input(shape=(latent_dim,), name='z_sampling')\nx = kr.layers.Dense(hidden_dim, activation='relu')(latent_inputs)\noutputs = kr.layers.Dense(num_features, activation='sigmoid')(x)\n\n# instantiate decoder model\ndecoder = kr.Model(latent_inputs, outputs, name='decoder')\ndecoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.978049Z","iopub.status.idle":"2022-05-20T10:31:22.978559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # VAE model = encoder + decoder\noutputs = decoder(encoder(inputs)[2])  # Select the Z value from outputs of the encoder\nvae = kr.Model(inputs, outputs, name='vae')","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.9792Z","iopub.status.idle":"2022-05-20T10:31:22.979708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Define VAE Loss**","metadata":{}},{"cell_type":"code","source":"# Reconstruction loss\nreconstruction_loss = tf.losses.mean_squared_error(inputs, outputs)\nreconstruction_loss = reconstruction_loss * num_features\n\n# KL Divergence loss\nkl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)\nkl_loss = -0.5 * tf.reduce_sum(kl_loss, axis=-1)\nvae_loss = tf.reduce_mean(reconstruction_loss + kl_loss)\n\nvae.add_loss(vae_loss)\nvae.compile(optimizer='adam')\nvae.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.98038Z","iopub.status.idle":"2022-05-20T10:31:22.980915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(X_train, _),  (X_test, y) = load_data('mnist', labels=True)\nvae.fit(X_train, epochs=epochs, batch_size=batch_size, validation_data=(X_test, None))","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-05-20T10:31:22.981552Z","iopub.status.idle":"2022-05-20T10:31:22.982087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generate_manifold(decoder)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.982713Z","iopub.status.idle":"2022-05-20T10:31:22.983253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_digits(X_test, y, encoder)  # y for label coloring","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.983923Z","iopub.status.idle":"2022-05-20T10:31:22.984447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like by using the decoder on noisy data we can generate a completely new data! Now how amazing is that? <3.\nBut this is an image data so the question we have in our mind that would it possible to use with convolutions so we can get better results? Of course! Now let's try convolution version of VAE.","metadata":{}},{"cell_type":"markdown","source":"## Convolutional VAE","metadata":{}},{"cell_type":"markdown","source":"Although, we don't use maxpooling in our model, feel free to use it when you want to test and see if it helps(both for decreasing amount of trainable parameters and faster training.)","metadata":{}},{"cell_type":"markdown","source":"<img src=\"https://res.cloudinary.com/dpyleb8ih/image/upload/v1560252253/1-s2.0-S0009250918300460-gr3.jpg\" width=\"900\" height=\"600\">","metadata":{}},{"cell_type":"code","source":"# new network parameters\ninput_shape = (28, 28, 1)\nbatch_size = 128\nfilters = 32\nlatent_dim = 8\nepochs = 30","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.985083Z","iopub.status.idle":"2022-05-20T10:31:22.985603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Encoder**","metadata":{}},{"cell_type":"code","source":"inputs = kr.Input(shape=input_shape, name='input')\n\nx = kr.layers.Conv2D(filters, (3, 3), activation='relu', strides=2, padding='same')(inputs)\nx = kr.layers.Conv2D(filters*2, (3, 3), activation='relu', strides=2, padding='same')(x)\n\n# shape info needed to build decoder model\nshape = x.get_shape().as_list()\n\n# generate latent vector Q(z|X)\nx = kr.layers.Flatten()(x)\nx = kr.layers.Dense(16, activation='relu')(x)\nz_mean = kr.layers.Dense(latent_dim, name='z_mean')(x)\nz_log_var = kr.layers.Dense(latent_dim, name='z_log_var')(x)\n\nz = kr.layers.Lambda(sampling, name='z')([z_mean, z_log_var])\n\n# instantiate encoder model\nencoder = kr.Model(inputs, [z_mean, z_log_var, z], name='encoder')\nencoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.986343Z","iopub.status.idle":"2022-05-20T10:31:22.987064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Decoder**","metadata":{}},{"cell_type":"code","source":"# build decoder model\nlatent_inputs = kr.Input(shape=(latent_dim,), name='z_sampling')\nx = kr.layers.Dense(shape[1] * shape[2] * shape[3], activation='relu')(latent_inputs)\nx = kr.layers.Reshape((shape[1], shape[2], shape[3]))(x)\nx = kr.layers.Conv2DTranspose(filters*2, (3, 3), activation='relu', strides=2, padding='same')(x)\nx = kr.layers.Conv2DTranspose(filters, (3, 3), activation='relu', strides=2, padding='same')(x)\noutputs = kr.layers.Conv2DTranspose(1, (3, 3), activation='sigmoid', padding='same', name='decoder_output')(x)\n\n# instantiate decoder model\ndecoder = kr.Model(latent_inputs, outputs, name='decoder')\ndecoder.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.987932Z","iopub.status.idle":"2022-05-20T10:31:22.98834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# instantiate VAE model\noutputs = decoder(encoder(inputs)[2])\nvae = kr.Model(inputs, outputs, name='vae')","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.989354Z","iopub.status.idle":"2022-05-20T10:31:22.989862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reconstruction loss\nreconstruction_loss = tf.losses.mean_squared_error(inputs, outputs)\nreconstruction_loss = reconstruction_loss * num_features\n\n# KL Divergence loss\nkl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)\nkl_loss = -0.5 * tf.reduce_sum(kl_loss, axis=-1)\nvae_loss = tf.reduce_mean(reconstruction_loss + kl_loss)\n\nvae.add_loss(vae_loss)\nvae.compile(optimizer='adam')\nvae.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.990718Z","iopub.status.idle":"2022-05-20T10:31:22.991337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(X_train, _),  (X_test, y) = load_data('mnist', labels=True)\nX_train, X_test = X_train.reshape([-1, 28, 28, 1]), X_test.reshape([-1, 28, 28, 1])\nvae.fit(X_train, epochs=epochs, batch_size=batch_size, validation_data=(X_test, None))","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.992233Z","iopub.status.idle":"2022-05-20T10:31:22.993033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test[:1].shape,y[0]","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.9936Z","iopub.status.idle":"2022-05-20T10:31:22.994279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ttt = encoder.predict(X_test[:1])\nttt","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.995118Z","iopub.status.idle":"2022-05-20T10:31:22.995746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test[0].reshape(28,28).shape","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.996376Z","iopub.status.idle":"2022-05-20T10:31:22.996896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(X_test[0].reshape(28,28))","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.99762Z","iopub.status.idle":"2022-05-20T10:31:22.998107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dttt = decoder.predict(ttt[2])\nplt.imshow(dttt[0].reshape(28,28))","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:22.99873Z","iopub.status.idle":"2022-05-20T10:31:22.999615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.losses.mean_squared_error(dttt, X_test[:1])","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:23.000188Z","iopub.status.idle":"2022-05-20T10:31:23.000654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_features","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:23.001483Z","iopub.status.idle":"2022-05-20T10:31:23.002091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test[:1].shape","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:23.003039Z","iopub.status.idle":"2022-05-20T10:31:23.003608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generate_manifold(decoder)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:23.004575Z","iopub.status.idle":"2022-05-20T10:31:23.00515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_digits(X_test, y, encoder)","metadata":{"execution":{"iopub.status.busy":"2022-05-20T10:31:23.00573Z","iopub.status.idle":"2022-05-20T10:31:23.006488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusions\nBy changing the size of filters or layers, it's possible to get much more better results. Give it a try if you want to. And also if you like my kernel, please upvote it :) ","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}