{"cells":[{"metadata":{"_uuid":"d9ff5a5f4d2aadd848852609a891626137ae6974"},"cell_type":"markdown","source":"# Reminder\nEnable the GPU in Settings."},{"metadata":{"_cell_guid":"9c2453df-b948-46c8-b6cf-ff1456e7dae4","_uuid":"b1b821adb6c3640bb8f5bc5e8f2c1e5748c903c1"},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport time\nimport tensorflow as tf\n\nfrom tensorflow.python.keras.layers import Dense, Flatten, Conv2D, Reshape, Conv2DTranspose, LeakyReLU\nfrom keras.utils.np_utils import to_categorical   \n\n# Main TFGAN library.\ntfgan = tf.contrib.gan\n\ntf.set_random_seed(0)\ntf.reset_default_graph()","execution_count":1,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"},"cell_type":"markdown","source":"# Utility Functions"},{"metadata":{"collapsed":true,"_cell_guid":"0607aa78-0fe7-49b1-97ad-92efd28343cf","_uuid":"771f3a60a24be3557d667263af750889f9edb4bb","trusted":true},"cell_type":"code","source":"#from learntools.gans.generators import basic_generator\n#from learntools.gans.discriminators import basic_discriminator\n#from learntools.gans.gan_utils import visualize_training_generator, dataset_to_stream\n\nimport sys\nsys.path.append('/kaggle/input/python-utility-code-for-deep-learning-exercises/utils/gans/')\nfrom generators import basic_generator\nfrom discriminators import basic_discriminator\nfrom gan_utils import visualize_training_generator, dataset_to_stream","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"eb596b23-4356-44fe-af37-975e4369d635","_uuid":"4ffbe0622213fb2fb851d7f632d60cb85426c603"},"cell_type":"markdown","source":"# Data Input Pipeline"},{"metadata":{"collapsed":true,"_cell_guid":"08f6b566-637c-46bd-a3b1-3f24b69dc7e7","_uuid":"2fb640eef4dd5592d6e145aefcf017f207594b1c","trusted":true},"cell_type":"code","source":"train_fname = '../input/digit-recognizer/train.csv'\n# Size of each digit\nimg_rows, img_cols = 28, 28\n# Target has 10 values corresponding to 10 numbers (0, 1, 2 ... 9)\nnum_classes = 10\n# Choice of batch size is not critical\nbatch_size = 40\n\nraw = pd.read_csv(train_fname)\nnum_images = raw.shape[0]\nx_as_array = raw.values[:,1:]\n# Reshape from 1 vector into an image. Last dimension shows it is greyscale, which is 1 channel\nx_shaped_array = x_as_array.reshape(num_images, img_rows, img_cols, 1)\n# Optimization with default params is better when vals scaled to [-1, 1]\nimage_array = ((x_shaped_array - 128)/ 128).astype(np.float32)\n# set up target\nlabels_array = to_categorical(raw.values[:,0], num_classes=10)\n\n# following 2 lines create the iterator/stream of tensors consumed in model training\nmy_dataset = tf.data.Dataset.from_tensor_slices((image_array))\nbatched_dataset = dataset_to_stream(my_dataset, batch_size)","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"4fcb2852-fd3f-4e82-8a53-f921cfcdfa41","_uuid":"e229b6867fa493f2d639f02791815c8f5009e6c5"},"cell_type":"markdown","source":"# Model\nDefine the GANModel tuple using the TFGAN library function. For the simplest case, we need the following:\n\n- A generator function that takes input noise and outputs generated images\n- A discriminator function that takes images and outputs a probability of being real or fake\n- Real images\n- A noise vector to pass to the generator"},{"metadata":{"collapsed":true,"_cell_guid":"2d885804-9ee9-4d4b-8018-33f331a337a9","_uuid":"6c42b85436e666dcd4cf0d4011c7a8ed065baf24","trusted":true},"cell_type":"code","source":"noise_dims = 64\ngan_model = tfgan.gan_model(\n    basic_generator,\n    basic_discriminator,\n    real_data=batched_dataset,\n    generator_inputs=tf.random_normal([batch_size, noise_dims]))","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"9e299f46-6187-4070-95e1-94152adff518","_uuid":"09aefd117df20e97dc15eef81cbe5709c752f9cf"},"cell_type":"markdown","source":"# Losses and Optimization\nWe next set up the GAN model losses.\n\nLoss functions are an active area of research. The losses library provides some well-known or successful loss functions, such as the original minimax, Wasserstein, and improved Wasserstein losses."},{"metadata":{"collapsed":true,"_cell_guid":"eb643e1f-ef8a-41e8-9742-60a682c44083","_uuid":"fe52b5b328b7c193511c9b209d99a6c490864ddb","trusted":true},"cell_type":"code","source":"# Example of classical loss function.\n#vanilla_gan_loss = tfgan.gan_loss(\n#    gan_model,\n#    generator_loss_fn=tfgan.losses.minimax_generator_loss,\n#    discriminator_loss_fn=tfgan.losses.minimax_discriminator_loss)\n\n# Wasserstein loss (https://arxiv.org/abs/1701.07875) with the \n# gradient penalty from the improved Wasserstein loss paper \n# (https://arxiv.org/abs/1704.00028).\nimproved_wgan_loss = tfgan.gan_loss(\n    gan_model,\n    generator_loss_fn=tfgan.losses.wasserstein_generator_loss,\n    discriminator_loss_fn=tfgan.losses.wasserstein_discriminator_loss,\n    gradient_penalty_weight=1.0)","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"960a4020-4311-48ed-88fd-9b060d4f263f","_uuid":"c981b4c035fc20ddecf5aec93b1589b69d977758"},"cell_type":"markdown","source":"### Optimizer Settings\n\nThe choice of optimizer and settings has been the subject of a lot of guesswork and iteration. When getting started, it's likely not a good use of time to fiddle with these. This also may be "},{"metadata":{"collapsed":true,"_cell_guid":"815eb2de-edff-437b-8d2c-12dde30a956e","_uuid":"60578b3581cc51bbcb65e6886537af85f61cec37","trusted":true},"cell_type":"code","source":"generator_optimizer = tf.train.AdamOptimizer(0.001, beta1=0.5)\ndiscriminator_optimizer = tf.train.AdamOptimizer(0.0001, beta1=0.5)\ngan_train_ops = tfgan.gan_train_ops(\n    gan_model,\n    improved_wgan_loss,\n    generator_optimizer,\n    discriminator_optimizer)","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"8e529222-f631-4cfe-97b3-045d3bf15f42","_uuid":"9257de1ee52b590b4b3d278645fd370a41c56ae2"},"cell_type":"markdown","source":"# Set Up Progress Tracking\n\nThis helps us see the evolution in image quality as the GAN is being trained. Specifically, the code below takes a sample of images and shapes them into something that can be viewed."},{"metadata":{"collapsed":true,"_cell_guid":"74329701-aee6-4468-8086-b1477ed8e38d","_uuid":"04d64ddf85b293774c5f25e8b132311a4760d125","trusted":true},"cell_type":"code","source":"images_to_eval = 20\n\n# For variables to load, use the same variable scope as in the train job.\nwith tf.variable_scope('Generator', reuse=True):\n    eval_images = gan_model.generator_fn(tf.random_normal([images_to_eval, noise_dims]))\n\n# Reshape eval images for viewing.\ngenerated_data_to_visualize = tfgan.eval.image_reshaper(eval_images[:images_to_eval,...], num_cols=10)","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"7914ffbb-2498-4681-a433-c9cac891d597","_uuid":"353addee802cf54083ae0d9b97226fb362fa213c"},"cell_type":"markdown","source":"# Train Steps\n\nNow we're ready to train. TFGAN handles the alternating training scheme. There is a gan_train function which provies more slightly more automation, but we will run the training in a for-loop so we can introspect more."},{"metadata":{"_cell_guid":"4875c66a-0256-4e32-94e3-f9e56f3f4eca","_uuid":"799290718abf7f434faa5bb397bc7fde508b3023","trusted":true},"cell_type":"code","source":"g_d_updates_per_step = tfgan.GANTrainSteps(1,2)  # do 1 gen step, then 2 disc steps.  \ntrain_step_fn = tfgan.get_sequential_train_steps(g_d_updates_per_step)\n\nglobal_step = tf.train.get_or_create_global_step()\n\nn_batches = 1501\nwith tf.train.SingularMonitoredSession() as sess:\n    start_time = time.time()\n    for i in range(n_batches):\n        train_step_fn(sess, gan_train_ops, global_step, train_step_kwargs={})\n        if i % 100 == 0:\n            digits_np = sess.run([generated_data_to_visualize])\n            visualize_training_generator(i, start_time, digits_np)","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}