{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Assignment - GANs\n\nHi there! In this assignment, you will use GANs (or something else, if you want) to produce \"super-resolution\" versions of images.\n\nUnlike last assignment, data is no longer paired, which makes this **much more difficult** (but also realistic).\n\nThat is, you have access to the low-resolution data and the high-resolution data, but you do not have the associated high-resolution version of a low-resolution image.\n\nSpecifically, you must upscale 32x32 images in color to 64x64 images in color (i.e. 4 times as many pixels).\n\nI have normalized the images in [0, 1] (for all color channels).\n\nTo get you started, I have provided a complete working example. This is much more \"creative\" than the examples I usually provide, and just reading through it and figuring out what it does might be fun/helpful for you.\n\nYou should not take the example too seriously. It is meant as something you may draw inspiration from, showing how you can use \"domain knowledge\" to help GANs achieve the goal you aim for. It is not a hugely successful example, but quite informative nonetheless.\n\n**Note**: This is quite computationally expensive. This will take time to train, even on Colab!\n\n**Remark**: This assignment is very difficult - I realize this. However, I wanted to do something special for the last assignment. This is beyond what is expected for the exam (as it is beyond the required reading for the course). It is most related to and inspired by CycleGAN, although not quite the same (not actually mapping back). \n\nWhen you are done, submit your results on the Kaggle webpage for this competition. If you do not like to show your score to everyone, you may either:\n\n1. Send your  .csv  to me, and I will then do it for you.\n1. Use an anonymous username on Kaggle.\n\nHowever, I suggest you use your real name, after all it is just meant as an exercise and it is more fun that way. You can submit 5 times every day, so you can experiment with some stuff without being \"locked in\"."},{"metadata":{},"cell_type":"markdown","source":"# Details\n\nThe metric used to score this assignment MSE (on the pixel level).\n\nMaking use of convolutional layers and GANs is heavily encouraged.\n\n**Remark**: Note that there is no way you are able to validate your results internally (but you can of course use Kaggle), as you do not have paired data. So you cannot perform hyperparameter search as you might usually do, but it would most likely be too slow even if you could."},{"metadata":{},"cell_type":"markdown","source":"# Kaggle\n\nThe assignment uses Kaggle. You can use it to submit your predictions. It will then give you a score (based on some of the test data). The final score will be calculated using the other part of the test data (so you have no change of looking at the test data for the competition).\n\nYou can visit the webpage here: https://www.kaggle.com/c/aml-gan. You need to create an account (free, just need an email)."},{"metadata":{"trusted":false},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"# Load data (must be in same folder as this file, which it will be if you simply unzip the assignment).\n# Note that we don't have any y_test! This way you cannot \"cheat\"!\n\nx_train = np.load('x_train.npy').astype('float32')\nx_test = np.load('x_test.npy').astype('float32')\ny_train = np.load('y_train.npy').astype('float32')\n\nprint(x_train.shape, x_test.shape, y_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"BATCH_SIZE = 128\n\nsmall_images_dataset = tf.data.Dataset.from_tensor_slices(x_train).shuffle(len(x_train)).batch(BATCH_SIZE)\nlarge_images_dataset = tf.data.Dataset.from_tensor_slices(y_train).shuffle(len(x_train)).batch(BATCH_SIZE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n\ndef discriminator_loss(real_output, fake_output):\n    real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)\n    total_loss = real_loss + fake_loss\n    return total_loss\n\ndef generator_loss(fake_output, small_images, large_predicted_images):\n    gan_loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n    struct_loss = structural_loss(small_images, large_predicted_images)\n    total_loss = gan_loss + struct_loss\n    return total_loss\n\ndef structural_loss(small_images, large_predicted_images):\n    # Try to figure out what I try to accomplish with this part of the generator loss!\n    resized = tf.image.resize(large_predicted_images, (32, 32))\n    return tf.reduce_mean(tf.abs(small_images - resized))\n\ndef eval_generator(generator, discriminator, sample):\n    gen_predictions = generator(sample, training=False)\n    \n    dis_predictions = discriminator.predict(gen_predictions)\n    dis_acc = round(np.mean(dis_predictions < 0.5) * 100, 2)\n    \n    print(f'Generator \"structural\" loss = {structural_loss(sample, gen_predictions)}.')\n    print(f'Generator total loss = {generator_loss(dis_predictions, sample, gen_predictions)}.')\n    print(f'Discriminator accuracy on fake images: {dis_acc}%.')\n    \n    fig = plt.figure(figsize=(4, 4))\n\n    for i in range(16):\n        plt.subplot(4, 4, i+1)\n        plt.imshow(sample[i].numpy())\n        plt.axis('off')\n        \n    plt.show()\n    \n    fig = plt.figure(figsize=(4, 4))\n\n    for i in range(16):\n        plt.subplot(4, 4, i+1)\n        plt.imshow(gen_predictions[i])\n        plt.axis('off')\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"generator = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2DTranspose(64, 3, 1, padding='same', input_shape=(32, 32, 3)),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.Conv2DTranspose(128, 4, 2, padding='same'),\n    tf.keras.layers.BatchNormalization(),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.Conv2DTranspose(3, 4, 1, padding='same', activation='sigmoid'),\n])\ngenerator.summary()\n\ndiscriminator = tf.keras.models.Sequential([\n    tf.keras.layers.Conv2D(32, 4, 1, padding='same', input_shape=(64, 64, 3)),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.Conv2D(64, 4, 2, padding='same'),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.Conv2D(64, 4, 2, padding='same'),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.Conv2D(128, 4, 2, padding='same'),\n    tf.keras.layers.LeakyReLU(),\n    tf.keras.layers.Dropout(0.3),\n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(1),\n])\ndiscriminator.summary()\n\ngenerator_optimizer = tf.keras.optimizers.Adam(1e-4)\ndiscriminator_optimizer = tf.keras.optimizers.Adam(1e-4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"@tf.function\ndef train_step(small_images, large_images):\n    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n        generated_images = generator(small_images, training=True)\n\n        real_output = discriminator(large_images, training=True)\n        fake_output = discriminator(generated_images, training=True)\n\n        gen_loss = generator_loss(fake_output, small_images, generated_images)\n        disc_loss = discriminator_loss(real_output, fake_output)\n\n    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n\n    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":false},"cell_type":"code","source":"for epoch in range(40):\n    for train_x, train_y in zip(small_images_dataset, large_images_dataset):\n        train_step(train_x, train_y)\n        \n    print(f'Epoch: {epoch}')\n    eval_generator(generator, discriminator, train_x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"y_test_hat = generator.predict(x_test)\ny_test_hat_flat = y_test_hat.reshape(-1) # For Kaggle evaluation, we flatten the data.\ny_test_hat_pd = pd.DataFrame({\n    'Id': list(range(len(y_test_hat_flat))),\n    'Predicted': y_test_hat_flat,\n})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us for fun check how we perform (visually)! I think it looks quite cool, even if it obviously isn't perfect."},{"metadata":{"scrolled":false,"trusted":false},"cell_type":"code","source":"for i in range(10):\n    plt.imshow(x_test[i]); plt.title('32x32'); plt.show()\n    plt.imshow(y_test_hat[i]); plt.title('Upscaled to 64x64'); plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# After you make your predictions, you should submit them on the Kaggle webpage for our competition.\n# You may also (and I recommend you do it) send your code to me (at tsdj@sam.sdu.dk).\n# Then I can provide feecback if you'd like (so ask away!).\n# Even better, you can upload your notebook on Kaggle!\n\n# Below is a small check that your output has the right type and shape\nassert isinstance(y_test_hat_pd, pd.DataFrame)\nassert all(y_test_hat_pd.columns == ['Id', 'Predicted'])\nassert len(y_test_hat_pd) == 10715136\n\n# If you pass the checks, the file is saved.\ny_test_hat_pd.to_csv('y_test_hat.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}