{"cells": [{"source": "Adversarial Autoencoders (as explained [here][1]) are a great way to use unsupervised learning for finding latent space representations of a given dataset and to generate images similar to this dataset. In the following, I will briefly explain the principle behind AAE and give the Code for programming one in Keras.\n\nThe Autoencoder will take an Image x and will \"encode\" it to a latent space z. This is done by the encoder part of the Network. The decoder will then take the vector z and try to restore the original image as close as possible (x'). This means the encoder imperfectly \"compresses\" the image x and the decoder imperfectly restores the image from z.\n\nTraining an Autoencoder is fairly easy: Take the decoder network,  put it directly on top of the encoder network and then train with x as input and x as desired output.  After training, the encoder can be used to find the representation of a given image as z. If we then change z around a little bit, it is likely, that this will not lead to an image similar to the dataset we started out with (e.g. not a valid number in the case of MNIST). We would like to be able to change around z and get meaningful changes in x' though.\n\nThat is a problem that is solved by trying to force z to take on a certain probability-distribution (e.g. normal distribution or others). I know 2 ways of how to do that:\na) Use the Kullback\u2013Leibler divergence as an additional loss term during training (This is called a variational autoencoder) or b) Use an adversarial network as an additional training partner for the encoder.\n\nIn this Kernel, I opted for Option b and thus built an Adversarial Autoencoder (AAE).\n\nHere are the generation results of running the code below for 100 Epochs:\nhttp://imgur.com/a/1pNdM\n\n  [1]: http://hjweide.github.io/adversarial-autoencoders", "outputs": [], "cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "72eb210b-0176-4659-8a38-307620a63ef5", "_uuid": "d850e73dfa187d7b7572d4fc315a9ad541763ab9"}, "execution_count": null}, {"source": "First, the MNIST Dataset is imported", "outputs": [], "cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "f209be28-f592-447b-a5f2-e9942261d7fd", "_uuid": "be458551e641814287e0c7072674f4ab8ea60b2a"}, "execution_count": null}, {"source": "import numpy as np\nimport keras as ke\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\ntrain = pd.read_csv(\"../input/train.csv\").values\n\nx_train = train[:, 1:].reshape(train.shape[0], 28, 28, 1)\nx_train = x_train.astype(float)\nx_train /= 255.0\n\n#Use Entire Train-Set when trying this code on your own machine\nx_train = x_train[:2500]", "outputs": [], "cell_type": "code", "metadata": {"trusted": false, "_cell_guid": "37acd364-c9b0-434c-a93d-21ab206f38de", "_uuid": "27714080a91e6fd2c9c6363c2c11c92fc3a7a7e4"}, "execution_count": null}, {"source": "Next, the models for Encoder, Decoder and Discriminator (the adversarial opponent) are defined:", "outputs": [], "cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "ee9c1919-6bb4-429d-a94a-81e58e158e28", "_uuid": "9c5dd117dc69e2d3692478500993322a1aba87ae"}, "execution_count": null}, {"source": "def build_model_enc():\n    model = ke.models.Sequential()\n    model.add(ke.layers.Conv2D(32, (5,5), padding=\"same\", activation=\"relu\", input_shape=(28, 28, 1)))\n    model.add(ke.layers.Conv2D(64, (5,5), strides=(2,2), activation=\"relu\", padding=\"same\"))\n    model.add(ke.layers.Conv2D(128, (5,5), strides=(2,2), activation=\"relu\", padding=\"same\"))\n    model.add(ke.layers.Flatten())\n    model.add(ke.layers.Dense(2, activation=\"linear\"))\n\n    return model\n\ndef build_model_dec():\n    model = ke.models.Sequential()\n    model.add(ke.layers.Dense(6272, input_shape=(2,)))\n    model.add(ke.layers.Reshape((7, 7, 128)))\n    model.add(ke.layers.Conv2D(64, (5,5), activation=\"relu\", padding=\"same\"))\n    model.add(ke.layers.UpSampling2D())\n    model.add(ke.layers.Conv2D(32, (5,5), activation=\"relu\", padding=\"same\"))\n    model.add(ke.layers.UpSampling2D())\n    model.add(ke.layers.Conv2D(1, (5,5), activation=\"sigmoid\", padding=\"same\"))\n\n    return model\n\ndef build_model_disc():\n    model = ke.models.Sequential()\n    model.add(ke.layers.Dense(32, activation=\"relu\", input_shape=(2,)))\n    model.add(ke.layers.Dense(32, activation=\"relu\"))\n    model.add(ke.layers.Dense(1, activation=\"sigmoid\"))\n    return model", "outputs": [], "cell_type": "code", "metadata": {"collapsed": false, "trusted": false, "_execution_state": "idle", "_cell_guid": "c47e32b0-5a8c-4733-9207-6f384acce7c5", "_uuid": "3df08e698e0efb2d776dc390e66cd4dac46af57e"}, "execution_count": null}, {"source": "The Models are put together in the required combinations and compiled", "outputs": [], "cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "9f5bee87-8cf5-4ea6-85c2-a0a9b71aa6c4", "_uuid": "bca6ec8ea09c7a53c2a0a243eab26a9ee970948b"}, "execution_count": null}, {"source": "def build_model_aae():\n    model_enc = build_model_enc()\n    model_dec = build_model_dec()\n    model_disc = build_model_disc()\n    \n    model_ae = ke.models.Sequential()\n    model_ae.add(model_enc)\n    model_ae.add(model_dec)\n    \n    model_enc_disc = ke.models.Sequential()\n    model_enc_disc.add(model_enc)\n    model_enc_disc.add(model_disc)\n    \n    return model_enc, model_dec, model_disc, model_ae, model_enc_disc\n\nmodel_enc, model_dec, model_disc, model_ae, model_enc_disc = build_model_aae()\n\nmodel_enc.summary()\nmodel_dec.summary()\nmodel_disc.summary()\nmodel_ae.summary()\nmodel_enc_disc.summary()\n\nmodel_disc.compile(optimizer=ke.optimizers.Adam(lr=1e-4), loss=\"binary_crossentropy\")\nmodel_enc_disc.compile(optimizer=ke.optimizers.Adam(lr=1e-4), loss=\"binary_crossentropy\")\nmodel_ae.compile(optimizer=ke.optimizers.Adam(lr=1e-3), loss=\"binary_crossentropy\")", "outputs": [], "cell_type": "code", "metadata": {"collapsed": false, "trusted": false, "_execution_state": "idle", "_cell_guid": "f91989f2-efbd-4299-ba30-40260e17a8db", "_uuid": "c40c97a7d45a3eca6e37756fbc3a196222b53eeb"}, "execution_count": null}, {"source": "Some helper functions to facilitate training and give a nice overview of the 2 dimensional latent space are defined.", "outputs": [], "cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "654b824e-2837-4b3a-93af-498a1de63e35", "_uuid": "78623bcea7d6389fb2671dc5c19c499cec55b0fe"}, "execution_count": null}, {"source": "\ndef imagegrid(dec, epochnumber):        \n        fig = plt.figure(figsize=[20, 20])\n        \n        for i in range(-5, 5):\n            for j in range(-5,5):\n                topred = np.array((i*0.5,j*0.5))\n                topred = topred.reshape((1, 2))\n                img = dec.predict(topred)\n                img = img.reshape((28, 28))\n                ax = fig.add_subplot(10, 10, (i+5)*10+j+5+1)\n                ax.set_axis_off()\n                ax.imshow(img, cmap=\"gray\")\n        \n        fig.savefig(str(epochnumber)+\".png\")\n        plt.show()\n        plt.close(fig)\n        \ndef settrainable(model, toset):\n    for layer in model.layers:\n        layer.trainable = toset\n    model.trainable = toset", "outputs": [], "cell_type": "code", "metadata": {"collapsed": false, "trusted": false, "_execution_state": "idle", "_cell_guid": "ddd785ca-4806-4da1-8b7b-1379aa1b1068", "_uuid": "dc2ea1d1fd9873f447dcce175615d6b6c0920779"}, "execution_count": null}, {"source": "The Model is trained in the following steps for each minibatch:\n    1) The Autoencoder is trained\n    2) The Discriminator is trained to differentiate between z from images and the distribution we want z to have (normal distribution in this case)\n    3) The Encoder is trained to fool the parameter-fixed discriminator into thinking the z are real", "outputs": [], "cell_type": "markdown", "metadata": {"collapsed": false, "_execution_state": "idle", "_cell_guid": "b5457c9e-37e0-4274-a882-2996555bf7d7", "_uuid": "a2605cec6e1984c429cec3ed300d02e603fb6d72"}, "execution_count": null}, {"source": "batchsize=50\n#Set Number of Epochs to 10-20 or higher.\nfor epochnumber in range(1):\n    np.random.shuffle(x_train)\n    \n    for i in range(int(len(x_train) / batchsize)):\n        settrainable(model_ae, True)\n        settrainable(model_enc, True)\n        settrainable(model_dec, True)\n        \n        batch = x_train[i*batchsize:i*batchsize+batchsize]\n        model_ae.train_on_batch(batch, batch)\n        \n        settrainable(model_disc, True)\n        batchpred = model_enc.predict(batch)\n        fakepred = np.random.standard_normal((batchsize,2))\n        discbatch_x = np.concatenate([batchpred, fakepred])\n        discbatch_y = np.concatenate([np.zeros(batchsize), np.ones(batchsize)])\n        model_disc.train_on_batch(discbatch_x, discbatch_y)\n        \n        settrainable(model_enc_disc, True)\n        settrainable(model_enc, True)\n        settrainable(model_disc, False)\n        model_enc_disc.train_on_batch(batch, np.ones(batchsize))\n    \n    print (\"Reconstruction Loss:\", model_ae.evaluate(x_train, x_train, verbose=0))\n    print (\"Adverserial Loss:\", model_enc_disc.evaluate(x_train, np.ones(len(x_train)), verbose=0))\n    \n    \n    imagegrid(model_dec, epochnumber)     \n        ", "outputs": [], "cell_type": "code", "metadata": {"collapsed": false, "trusted": false, "_execution_state": "idle", "_cell_guid": "aaf9381f-cb3a-405f-8983-faa56bd8370a", "_uuid": "fb8d32a4a7c2a727c19f180a6310ce533a6a84a5"}, "execution_count": null}], "nbformat": 4, "metadata": {"language_info": {"name": "python", "version": "3.6.1", "mimetype": "text/x-python", "file_extension": ".py", "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "codemirror_mode": {"name": "ipython", "version": 3}}, "kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3"}}, "nbformat_minor": 0}