{"cells":[{"metadata":{},"cell_type":"markdown","source":"# CycleGAN painter!\n\nThis notebook inspired by Amy Jang's tutorial notebook.\n"},{"metadata":{},"cell_type":"markdown","source":"# Introduction and Setup\n\nThis notebook utilizes a CycleGAN architecture to add Monet-style to photos. For this tutorial, we will be using the TFRecord dataset. Import the following packages and change the accelerator to TPU.\n\nFor more information, check out [TensorFlow](https://www.tensorflow.org/tutorials/generative/cyclegan) and [Keras](https://keras.io/examples/generative/cyclegan/) CycleGAN documentation pages."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa # addons ???\nimport math,random,cv2\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport PIL\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load in the data\n\nWe want to keep our photo dataset and our Monet dataset separate. First, load in the filenames of the TFRecords."},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()\n# google cloud storage path of the dataset. like some data root in linux something.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n# Monet's paint \n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))\n# What we want to change with the sytle like Monet paint.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Datanum\nhttps://www.kaggle.com/swepat/cyclegan-to-generate-monet-style-images"},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\ndef count_data_items(filenames):\n    n = [int(re.compile(r'-([0-9]*)\\.').search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_monet_samples = count_data_items(MONET_FILENAMES)\nn_photo_samples = count_data_items(PHOTO_FILENAMES)\nprint('number of monet tfrecord files :', len(MONET_FILENAMES))\nprint('number of photo tfrecord files:', len(PHOTO_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('n_monet_samples : ',n_monet_samples)\nprint('n_photo_samples : ',n_photo_samples)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization jpg_iamge\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import glob\nimport os\n\n# 要找出input下有什么文件，比较稳妥的方法就是一步步的os.path.list 比较好。否则直接看右边并不准确。比如 ： 现在就没有什么gan-getting-started 文件夹。\nBASE_PATH = '../input/monet-gan-getting-started/'\nmonet_path = glob.glob(str(BASE_PATH+'monet_jpg/*jpg'))\nphoto_path = glob.glob(str(BASE_PATH+'photo_jpg/*jpg'))\n\n# print(os.getcwd())\n# print(os.listdir('../input/monet-gan-getting-started/monet_jpg'))\n\n\n\ndef visualization_images(imagepath, n_images, is_random=True, figsize=(16, 16)):\n    plt.figure(figsize=figsize)\n\n    rows = int(n_images ** 0.5)\n    cols = math.ceil(n_images / rows)\n\n    image_names = imagepath[:n_images]\n\n    if is_random:\n        image_names = random.sample(image_names,n_images)\n        # 返回从image_names 重新排列的n_images.\n\n    for i, image_name in enumerate(image_names):\n        img = cv2.imread(image_name)\n        # cv2 中图像现实的bgr形式，需要进行转换。\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        plt.subplot(rows,cols,i+1) # 因为一共rows 行，cols列。\n        plt.imshow(img)\n        plt.axis('off')\n    plt.tight_layout()\n    plt.show()\n\nvisualization_images(monet_path, 12, is_random=True, figsize=(23, 23))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualization_images(photo_path, 12, is_random=True, figsize=(23, 23))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All the images for the competition are already sized to 256x256. As these images are RGB images, set the channel to 3. Additionally, we need to scale the images to a [-1, 1] scale. Because we are building a generative model, we don't need the labels or the image id so we'll only return the image from the TFRecord."},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1 # pixel value is in (-1,1)\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define the function to extract the image from the files."},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load in our datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's  visualize a photo example and a Monet example."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataaugment"},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nimport tensorflow as tf\nimport math\nimport tensorflow.keras.backend as K\n\ndef transform_rotation(image,height,rotation):\n    '''random rotate the image'''\n    dim = height\n    xdim = height % 2\n    rotation = rotation * tf.random.uniform([1], dtype='float32')\n    rotation = math.pi / 180. * rotation\n    # rotation matrix\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1], dtype='float32')\n    zero = tf.constant([0], dtype='float32')\n    rotation_matrix = tf.reshape(tf.concat([c1,s1,zero,-s1,c1,zero,zero,zero,one],axis=0),shape=[3,3])\n\n    # destination pixel indices :\n    x = tf.repeat(tf.range(dim//2,-dim//2,-1), dim) # shape like : [111222333]\n    y = tf.tile(tf.range(-dim//2,dim//2), [dim]) #shape like: [123123123]\n    z = tf.ones([dim*dim], dtype='int32')\n    idx = tf.stack([x,y,z]) # shape : 3 x (dim*dim*)\n\n    # rotate destination pixels onto origin pixels :\n    idx2 = K.dot(rotation_matrix, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2,-dim//2+1+xdim, dim//2)\n\n    # find original pixel values:\n    # in image because the coordinate is different\n    idx3 = tf.stack([dim//2-idx2[0,], dim//2-1+idx2[1,]])\n\n    d = tf.gather_nd(image, tf.transpose(idx3))\n    return tf.reshape(d, shape=[dim,dim,3])\n\n\ndef data_augument(image):\n    p_rotate = tf.random.uniform([],0,1.0,dtype=tf.float32)\n    # 旋转的概率\n    p_spatial = tf.random.uniform([],0,1.0,dtype=tf.float32)\n    # 空间增强\n    p_crop = tf.random.uniform([],0,1.0,dtype=tf.float32)\n    # 随机裁剪增强：\n    if p_crop > 0.5 :\n        # 随机扩大分辨率。\n        temp_size = random.randint(260,290)\n        image = tf.image.resize(image,[temp_size,temp_size])\n        # 然后再转换回来。\n        image = tf.image.random_crop(image,size=[256,256,3])\n        if p_crop >0.9 :\n            temp_size2 = random.randint(290,310)\n            image = tf.image.resize(image,[temp_size2,temp_size2])\n            image = tf.image.random_crop(image,size=[256,256,3])\n\n    # 随机的旋转。\n    if p_rotate > 0.75:\n        image = tf.image.rot90(image,k=3)\n    elif p_rotate > .5:\n        image = tf.image.rot90(image,k=2)\n    elif p_rotate > 0.25:\n        image = tf.image.rot90(image,k=1)\n\n    if p_rotate >=0.3:\n        image = transform_rotation(image, height=256,rotation=45.)\n\n    # 随机翻转：\n    if p_spatial >0.6:\n        image = tf.image.random_flip_left_right(image)\n        image = tf.image.random_flip_up_down(image)\n        if p_spatial >0.85:\n            image = tf.image.transpose(image)\n\n    return image\n\n    # 随机进行旋转。\n\n    # tf.image.transpose 将image的高和宽进行转换。\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### traindataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\ndef get_gan_dataset(monet_files, photo_files, augment=None,repeat=True,shuffle=True,batch_size=1):\n    monet_ds = load_dataset(monet_files)\n    photo_ds = load_dataset(photo_files)\n    \n    if augment:\n        monet_ds = monet_ds.map(augment, num_parallel_calls=AUTO)\n        photo_ds = photo_ds.map(augment, num_parallel_calls=AUTO)\n    if repeat:\n        monet_ds = monet_ds.repeat()\n        photo_ds = photo_ds.repeat()\n    if shuffle:\n        monet_ds = monet_ds.shuffle(2048)\n        photo_ds = photo_ds.shuffle(2048)\n    monet_ds = monet_ds.batch(batch_size, drop_remainder=True)   \n    photo_ds = photo_ds.batch(batch_size, drop_remainder=True)\n    monet_ds = monet_ds.cache()\n    photo_ds = photo_ds.cache()\n    monet_ds = monet_ds.prefetch(AUTO)\n    photo_ds = photo_ds.prefetch(AUTO)\n    \n    gan_ds = tf.data.Dataset.zip((monet_ds,photo_ds))\n    \n    return gan_ds\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gan_dataset = get_gan_dataset(MONET_FILENAMES,PHOTO_FILENAMES,augment=data_augument,repeat=True,shuffle=True,batch_size=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# example_monet,example_photo = next(iter(gan_dataset)) \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# show augment iamges "},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_images(example, nrows=1,ncols=5):\n    ds_iter = iter(example)\n    fig = plt.figure(figsize=(25, nrows*5.05))\n    for i in range(ncols*nrows):\n        image = next(ds_iter)\n        image = image.numpy()\n        ax = fig.add_subplot(nrows,ncols,i+1,xticks=[],yticks=[])\n        ax.imshow(image[0]*0.5+.5)\n        \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_images(monet_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_images(photo_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_images_2(aug_example,nrows=2,ncols=5):\n    ds_iter = iter(aug_example)\n    fig = plt.figure(figsize=(25,nrows*5.05))\n    for i in range((ncols)*(nrows//2)):\n        monetimage,photoimage = next(ds_iter)\n        monetimage = monetimage.numpy()\n        photoimage = photoimage.numpy()\n        ax = fig.add_subplot(nrows,ncols,i+1,xticks=[],yticks=[])\n        ax.imshow(monetimage[0]*0.5+0.5)\n        ax2 = fig.add_subplot(nrows,ncols,i+ncols+1,xticks=[],yticks=[])\n        ax2.imshow(photoimage[0]*0.5+0.5)\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_images_2(gan_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the generator\n\nWe'll be using a UNET architecture for our CycleGAN. To build our generator, let's first define our `downsample` and `upsample` methods.\n\nThe `downsample`, as the name suggests, reduces the 2D dimensions, the width and height, of the image by the stride. The stride is the length of the step the filter takes. Since the stride is 2, the filter is applied to every other pixel, hence reducing the weight and height by 2.\n\nWe'll be using an instance normalization instead of batch normalization. As the instance normalization is not standard in the TensorFlow API, we'll use the layer from TensorFlow Add-ons."},{"metadata":{"trusted":true},"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    result.add(layers.LeakyReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Upsample` does the opposite of downsample and increases the dimensions of the of the image. `Conv2DTranspose` does basically the opposite of a `Conv2D` layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's build our generator!\n\nThe generator first downsamples the input image and then upsample while establishing long skip connections. Skip connections are a way to help bypass the vanishing gradient problem by concatenating the output of a layer to multiple layers instead of only one. Here we concatenate the output of the downsample layer to the upsample layer in a symmetrical fashion.\n生成器将下采样层与对应的上采样曾进行连接，预防梯度消失。"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024) 因为输出是512，concate 之后就是 1024 \n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x) # 将中间的layer的输出保存到skips中，之后进行连接。\n\n    skips = reversed(skips[:-1]) #前边的最后一个和和后边的第一个连接，所以需要反向。最后一个不算。因为切片的特性。\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the discriminator\n\nThe discriminator takes in the input image and classifies it as real or fake (generated). Instead of outputing a single node, the discriminator outputs a smaller 2D image with higher pixel values indicating a real classification and lower values indicating a fake classification.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n    # 为什么InstanceNormalizaion 初始化用gama_init 。而不用普通的呢？其实是一样的，但是tfa.layers.InstanceNormalization只是在一个通道内进行normalization.\n    # batchnormalization 则是在一个batch的相应通道内进行normalization。\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since our generators are not trained yet, the generated Monet-esque photo does not show what is expected at this point."},{"metadata":{"trusted":true},"cell_type":"code","source":"to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Monet-esque Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the CycleGAN model\n\nWe will subclass a `tf.keras.Model` so that we can run `fit()` later to train our model. During the training step, the model transforms a photo to a Monet painting and then back to a photo. The difference between the original photo and the twice-transformed photo is the cycle-consistency loss. We want the original photo and the twice-transformed photo to be similar to one another.\n\nThe losses are defined in the next section."},{"metadata":{"trusted":true},"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        # 传递一些新的loss_fn, 和 optimizer.\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        # 优化器要apply gradient，而gradient需要loss来求。\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # 然后用判别器对生成的和真实的进行评分。\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # 三种loss 生成器loss，循环loss，辨别器的loss\n            # 生成器loss 1：generator_loss: 从照片生成作品，从作品生成照片，2 identity_loss: 从照片生成照片，从作品生成作品，3 cycle_loss:真作品生成假作品，从真照片生成假照片。\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # 最终得到生成作品，生成照片，判别作品，判别照片四个损失loss。\n            # 分别对应两个生成器两个判别器，然后用他们来优化这四个子网络\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # 判别器loss\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet) #真实作品和照片生成的作品。\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo) #真实照片和从作品生成的假照片。\n\n        # 计算梯度，然后应用梯度进行反向传播。\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define loss functions\n\nThe discriminator loss function below compares real images to a matrix of 1s and fake images to a matrix of 0s. The perfect discriminator will output all 1s for real images and all 0s for fake images. The discriminator loss outputs the average of the real and generated loss."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n        # 这里的real_loss 为什么这样设置？？\n        # 因为如果是真的话，那么他们的空间cos距离接近于1，否则接近于0？\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n        # 为什么？？？\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The generator wants to fool the discriminator into thinking the generated image is real. The perfect generator will have the discriminator output only 1s. Thus, it compares the generated image to a matrix of 1s to find the loss."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated) #生成器生成的作品的距离需要接近于1\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We want our original photo and the twice transformed photo to be similar to one another. Thus, we can calculate the cycle consistency loss be finding the average of their difference."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA): #这里输入图像而不是图像的距离\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image)) # reduce_mean()没有指定axis，则取全部的平均值。\n\n        return LAMBDA * loss1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The identity loss compares the image with its generator (i.e. photo with photo generator). If given a photo as input, we want it to generate the same image as the image was originally a photo. The identity loss compares the input with the output of the generator."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the CycleGAN\n\nLet's compile our model. Since we used `tf.keras.Model` to build our CycleGAN, we can just ude the `fit` function to train our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.utils.plot_model(cycle_gan_model, show_shapes=True, dpi=100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class EarlyStopping_custom(tf.keras.callbacks.EarlyStopping):\n\n    def __init__(self,monitor,mode,patience,\n                 restore_best_weights=True):\n        super(EarlyStopping_custom, self).__init__()\n        self.restore_best_weights = restore_best_weights\n        self.monitor = monitor,\n        self.mode = mode,\n        self.patience,\n        \n    def on_epoch_end(self, epoch, logs=None):\n        current = self.get_monitor_value(logs)\n        if current is None:\n            return\n        if self.monitor_op(current.mean() - self.min_delta, self.best):\n            self.best = current.mean()\n            self.wait = 0\n            if self.restore_best_weights:\n                self.best_weights = self.model.get_weights()\n        else:\n            self.wait += 1\n            if self.wait >= self.patience:\n                self.stopped_epoch = epoch\n                self.model.stop_training = True\n                if self.restore_best_weights:\n                    if self.verbose > 0:\n                        print('Restoring model weights from the end of the best epoch.')\n                    self.model.set_weights(self.best_weights)\n\n\nearly_stoping = EarlyStopping_custom(monitor='monet_disc_loss',\n                                                 patience=5,\n                                                 mode = 'min',\n                                                 restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 2\nsteps_per_epoch = (n_photo_samples//BATCH_SIZE)\ngan_dataset = get_gan_dataset(MONET_FILENAMES,PHOTO_FILENAMES,augment=data_augument,repeat=True,shuffle=True,batch_size=BATCH_SIZE)\n\n\nhistory = cycle_gan_model.fit(\n    gan_dataset,\n    epochs=31,\n    steps_per_epoch = steps_per_epoch,\n#     callbacks = [early_stoping],\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.asarray(history.history['photo_disc_loss']).shape   #为什么是这个形状？因为discriminator输出就是 batch 30 30 1\n# 而BinaryCrossentropy，输出则是 batch 30 30 ，2表示两个epoch，4代表什么？？？0000","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize our Monet-esque photos"},{"metadata":{"trusted":true},"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL \n! mkdir ../images ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# i = 1\n# for img in photo_ds:\n#     prediction = monet_generator(img, training=False)[0].numpy()\n#     prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n#     im = PIL.Image.fromarray(prediction)\n#     im.save(\"../images/\" + str(i) + \".jpg\")\n#     i += 1\n    \n    \nimport PIL\ndef predict_and_save(input_ds, generator_model, output_path):\n    i = 1\n    for img in input_ds:\n        prediction = generator_model(img, training=False)[0].numpy() # make predition\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)   # re-scale\n        im = PIL.Image.fromarray(prediction)\n        im.save(f'{output_path}{str(i)}.jpg')\n        i += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict_and_save(load_dataset(PHOTO_FILENAMES).batch(1), monet_generator, '../images/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")\n\n# print(f'number of generated samples :{len([name for name in os.listdir('/kaggle/images/') if os.path.isfile(os.path.join('/kaggle/images/', name))])}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}