{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"},{"sourceId":1567182,"sourceType":"datasetVersion","datasetId":925704},{"sourceId":6934993,"sourceType":"datasetVersion","datasetId":3949255}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## CLIPTraVeLGAN https://ceur-ws.org/Vol-3387/paper1.pdf","metadata":{"papermill":{"duration":0.01019,"end_time":"2023-11-09T16:06:08.802112","exception":false,"start_time":"2023-11-09T16:06:08.791922","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"This is a novel approach for semantically robust unpaired image translation. CLIPTraVeLGAN replaces the Siamese network in TraVeLGAN with a contrastively pretrained language-image model (CLIP) with frozen weights. This approach significantly simplifies the model selection and training process of TraVeLGAN, making it more robust and easier to use. \n\n","metadata":{"papermill":{"duration":0.008198,"end_time":"2023-11-09T16:06:08.818998","exception":false,"start_time":"2023-11-09T16:06:08.810800","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## TraVeLGAN\n\nThe authors [1] introduce the concept of a transformation vector between two points. In natural language processing tasks, words are represented by points in a space in which if a certain vector would transform the word \"man\" into the word \"woman\", then the word \"king\" into the word \"queen\" would be transformed by a very similar vector. A Siamese network S represents an image by points in a certain space. In the image translation task, instead of changing the gender of the word, the transformation vector can change the background colour, size or shape of the image. But the main idea is that the vector of transformation of a point obtained from the Siamese network from one image S(xi) (\"man\") to a point of another original image S(xj) (\"woman\") will also transform the point of the generated image S (GXY (xi)) \"king\" to the point of the generated S(GXY(xj)) \"queen\".\nTraVeLGAN used an additional Siamese network to encode high-level semantics between the source and target domains. This idea seemed like a breakthrough as the Siamese network was believed to outperform CycleGAN in terms of translation quality. However, TraVeLGAN has not received much development due to the difficulties in choosing the architecture of the Siamese network and the parameters of its training. This results in a large set of possible solutions and makes it difficult to determine the effectiveness of each of them. ","metadata":{"papermill":{"duration":0.009507,"end_time":"2023-11-09T16:06:08.836760","exception":false,"start_time":"2023-11-09T16:06:08.827253","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## CLIP\n\nCLIP [2] was introduced as a language-image model for the transfer of knowledge without any further training. Encoders learn the internal representation of images in a shared space with the internal representation of natural language texts. \n\nAfter pretraining the model, it can be used for any purpose with any images without any tuning. Trained on a dataset of billions of image-caption pairs from the Web (WIT), the model can successfully classify images with text class labels for a wide range of tasks, even quite far from its training set: geolocation, car brands. CLIP trained on WIT shows better accuracy on ImageNet than ResNet50 trained on ImageNet. The worst performance of knowledge transfer without additional training is shown on very specialized data sets, such as classification of satellite images, medical images, and object counting in synthetic images.\n\nNatural language encodes semantic content and hierarchical relations between concepts with words. Contrastive learning of a visual model using natural language texts as a learning cue led to the learning and generalization of such special knowledge about image elements as expressed in image-relevant texts in human language. The extent to which the visual model learns the hierarchy of concepts that exists in a human language requires separate research. Currently, CLIP reflects the meaning of the image in hidden representation most effectively among other well-known models.\nThe vector into which CLIP transforms images is the best choice for finding similar images. Other options for using CLIP in the search task are finding images that are most relevant to the content of some text and finding the text that most relevantly describes the image. \nOverall, CLIP's powerful internal representation of images and text make it a valuable tool for a wide range of applications, with potential future uses that have yet to be imagined.\n\n","metadata":{"papermill":{"duration":0.0087,"end_time":"2023-11-09T16:06:08.853906","exception":false,"start_time":"2023-11-09T16:06:08.845206","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Semantic robustness\nIn [4], the concept of semantic stability of unpaired translation of images was introduced and the reasons for the conflict between compliance with the subject area and accuracy of the translation, and the reasons for hallucinating objects that are absent in the input image were highlighted. SRUNIT model is proposed to provide translation semantically robust, which is simultaneously trained with a generator and a discriminator similar to TraVeLGAN's Siamese network. CLIP is not used. In [5], the use of Vector Symbolic Architectures was proposed to improve the semantic robustness of unpaired image translation, which showed even better indicators of semantic translation accuracy than SRUNIT. CLIP is not used also.\n","metadata":{"papermill":{"duration":0.00802,"end_time":"2023-11-09T16:06:08.869718","exception":false,"start_time":"2023-11-09T16:06:08.861698","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Our method\n\nThe core of the CLIPTraVeLGAN approach is the use of a pre-trained language-image model (CLIP) as a Siamese network in TraVeLGAN setup. The proposed CLIPTraVeLGAN model is composed of a generator, a discriminator and pretrained CLIP model. The generator takes an image from one domain and generates an image that belongs to the other domain. The discriminator is responsible for distinguishing between real and fake images. In CLIPTraVeLGAN, we replace the Siamese network in TraVeLGAN with the pre-trained language-image model CLIP. The CLIP model is used to encode the high-level semantics of the input and target domains. \n\nWe train the CLIPTraVeLGAN model using the adversarial loss and the TraVeL loss. The adversarial loss is used to ensure that the generated image belongs to the target domain, while the TraVeL loss encourages the generator to preserve the high-level semantics of the input image. Thus, the final objective terms of the generator are: \n\nLg=Ladv+λLtravel,\t(1)\n\nwhere λ controls the relative importance of TraVeL loss.\n\nTraVeL loss is the same as in TraVeLGAN: \n\nLtravel=ΣΣi≠j Dist[ S(Xi)-S(Xj),S(G(Xi))- S(G(Xj))] ,(2)\n\nwhere Dist is a distance metric, such as cosine similarity.\n\nOne advantage of our approach is that it eliminates the need for choosing and training a Siamese network, which can be complex and time-consuming. Instead, the transfer of knowledge from CLIP to CLIPTraVeLGAN enables the generator to understand the relationships between images without any additional training. This makes our approach simpler and more straightforward, while still ensuring the high-level semantics are captured in the generated image.\n\nIn this context, the use of CLIP in CLIPTraVeLGAN adds the ability to preserve high-level semantics between the source and target domains, making the translations semantically robust. Therefore, our work builds upon the idea of TraVeLGAN and leverages the advantages of CLIP to improve the quality of unpaired image translation while maintaining semantic robustness.\n","metadata":{"papermill":{"duration":0.008027,"end_time":"2023-11-09T16:06:08.885809","exception":false,"start_time":"2023-11-09T16:06:08.877782","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Experiment \n\nIn this notebook we evaluate CLIPTraVeLGAN on GTA (Grand Theft Auto) [6] to Cityscapes dataset [7] which is a benchmark dataset for unpaired image translation because it involves translating images from one domain to another, where the source and target domains are vastly different. \nThe GTA and Cityscapes datasets represent two different domains of real-world urban environments. The GTA dataset consists of images of urban scenes generated from a video game, while the Cityscapes dataset comprises real-world urban scenes captured by a camera mounted on a car. The images in these datasets differ in terms of lighting conditions, weather, time of day, and many other factors. The main problem is that GTA images have more sky than Cityscapes. The discriminator can easily distinct fake image by that criterion. Cityscapes images have more vegetation instead. Thus, models may hallucinate vegetation in open sky regions that is semantic mistake.\n","metadata":{"papermill":{"duration":0.007752,"end_time":"2023-11-09T16:06:08.901895","exception":false,"start_time":"2023-11-09T16:06:08.894143","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## References\n[1]\tAmodio, M., Krishnaswamy, S.: Travelgan: Image-to-image translation by transformation vector learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8983–8992 (2019). https://arxiv.org/abs/1902.09631\n\n[2]\tRadford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning. https://arxiv.org/abs/2103.00020\n\n[3]\tZhu, J., Park, T., Isola, P., & Efros, A.A. (2017). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. 2017 IEEE International Conference on Computer Vision (ICCV), 2242-2251. \n\n[4]\tZhiwei Jia et al. “Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 14273–14283. https://arxiv.org/abs/2012.04932\n\n[5]\tTheiss, Justin D. et al. “Unpaired Image Translation via Vector Symbolic Architectures.” European Conference on Computer Vision (2022). https://arxiv.org/abs/2209.02686\n\n[6]\tStephan R Richter et al. “Playing for data: Ground truth from computer games”. In: European conference on computer vision. Springer. 2016, pp. 102–118.\n\n[7]\tM. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. \n","metadata":{"papermill":{"duration":0.008034,"end_time":"2023-11-09T16:06:08.917910","exception":false,"start_time":"2023-11-09T16:06:08.909876","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Import","metadata":{"papermill":{"duration":0.007902,"end_time":"2023-11-09T16:06:08.933797","exception":false,"start_time":"2023-11-09T16:06:08.925895","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install transformers -U","metadata":{"_kg_hide-output":true,"papermill":{"duration":14.316414,"end_time":"2023-11-09T16:06:23.257986","exception":false,"start_time":"2023-11-09T16:06:08.941572","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:07:21.868498Z","iopub.execute_input":"2023-11-20T18:07:21.868785Z","iopub.status.idle":"2023-11-20T18:07:35.912803Z","shell.execute_reply.started":"2023-11-20T18:07:21.868737Z","shell.execute_reply":"2023-11-20T18:07:35.911629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE = 128 # change in siamis loss too\n\nimport tensorflow as tf\nimport tensorflow_hub as hub\nimport random, json, PIL, shutil, re, gc\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\nfrom scipy import linalg\nimport matplotlib.pyplot as plt\nimport numpy as np\n\n%matplotlib inline\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"on TPU\")\nexcept tf.errors.NotFoundError:\n print(\"not on TPU\")\n strategy = tf.distribute.MirroredStrategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTO = tf.data.AUTOTUNE\nprint(tf.__version__)\n\nAUTOTUNE = tf.data.AUTOTUNE\n\n","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":false,"_kg_hide-output":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":50.69577,"end_time":"2023-11-09T16:07:13.963629","exception":false,"start_time":"2023-11-09T16:06:23.267859","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:07:35.914740Z","iopub.execute_input":"2023-11-20T18:07:35.915105Z","iopub.status.idle":"2023-11-20T18:07:59.715609Z","shell.execute_reply.started":"2023-11-20T18:07:35.915073Z","shell.execute_reply":"2023-11-20T18:07:59.714860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset","metadata":{"papermill":{"duration":0.010397,"end_time":"2023-11-09T16:07:13.985059","exception":false,"start_time":"2023-11-09T16:07:13.974662","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"One of the competition's problems is that 300 of the images provided for training are not the same images used to compute the competition metric. Also, if we take all 1193 Monet paintings, it turns out that they correspond even less to the set on which the competition metric is based. Here, I am loading the Monet paintings numbers that I have selected, which, in my opinion, are the most suitable for achieving a high result. This aspect is not related to the implementation of CLIPTraVeLGAN, and you can use your own set of images (or the one provided in the competition).","metadata":{"papermill":{"duration":0.01033,"end_time":"2023-11-09T16:07:14.005748","exception":false,"start_time":"2023-11-09T16:07:13.995418","status":"completed"},"tags":[]}},{"cell_type":"code","source":"best_seq = [True] * 400 + [False] * 793 # this line will work if you'll remove the next line\nbest_seq = list(np.load('/kaggle/input/monettrainingset/best_seq.npy', allow_pickle=True)) # remove this line to avoid error message","metadata":{"papermill":{"duration":0.026745,"end_time":"2023-11-09T16:07:14.042870","exception":false,"start_time":"2023-11-09T16:07:14.016125","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:07:59.716589Z","iopub.execute_input":"2023-11-20T18:07:59.716890Z","iopub.status.idle":"2023-11-20T18:07:59.729481Z","shell.execute_reply.started":"2023-11-20T18:07:59.716861Z","shell.execute_reply":"2023-11-20T18:07:59.728646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PHOTO_PATH = '/kaggle/input/gan-getting-started'\nPHOTO_FILENAMES = tf.io.gfile.glob(str(PHOTO_PATH + '/photo_tfrec/*.tfrec'))\n\nMONET_PATH1 = '/kaggle/input/monet2photo/testA'\nMONET_PATH2 = '/kaggle/input/monet2photo/trainA'\nMONET_FILENAMES = tf.io.gfile.glob(str(MONET_PATH1 + '/*.*'))\nMONET_FILENAMES_2 = tf.io.gfile.glob(str(MONET_PATH2 + '/*.*'))\nMONET_FILENAMES = MONET_FILENAMES + MONET_FILENAMES_2\nprint('Number of images in monet2photo: ', len(MONET_FILENAMES))\n\nMONET_FILENAMES = np.array(MONET_FILENAMES)\nMONET_FILENAMES = list(MONET_FILENAMES[best_seq])\nprint('Number of choosen images: ', len(MONET_FILENAMES))\n","metadata":{"papermill":{"duration":0.505934,"end_time":"2023-11-09T16:07:14.559156","exception":false,"start_time":"2023-11-09T16:07:14.053222","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:07:59.731129Z","iopub.execute_input":"2023-11-20T18:07:59.731381Z","iopub.status.idle":"2023-11-20T18:08:00.073911Z","shell.execute_reply.started":"2023-11-20T18:07:59.731355Z","shell.execute_reply":"2023-11-20T18:08:00.072999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n image = tf.image.decode_jpeg(image, channels=3)\n image = (tf.cast(image, tf.float32) / 127.5) - 1\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image\n\ndef read_tfrecord(example):\n tfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_image(example['image'])\n return image\n\ndef load_im(filename):\n file = tf.io.read_file(filename) \n image = tf.image.decode_jpeg(file, channels=3)\n image = tf.image.convert_image_dtype(image, tf.float32)\n image = image * 2 - 1\n return image","metadata":{"papermill":{"duration":0.022219,"end_time":"2023-11-09T16:07:14.594312","exception":false,"start_time":"2023-11-09T16:07:14.572093","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:00.074899Z","iopub.execute_input":"2023-11-20T18:08:00.075149Z","iopub.status.idle":"2023-11-20T18:08:00.081363Z","shell.execute_reply.started":"2023-11-20T18:08:00.075122Z","shell.execute_reply":"2023-11-20T18:08:00.080615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n# Differentiable Augmentation for Data-Efficient GAN Training\n# Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han\n# https://arxiv.org/pdf/2006.10738\n# from https://github.com/mit-han-lab/data-efficient-gans/blob/master/DiffAugment_tf.py\n\n\n\n def DiffAugment(x, policy='', channels_first=False):\n if policy:\n if channels_first:\n x = tf.transpose(x, [0, 2, 3, 1])\n for p in policy.split(','):\n for f in AUGMENT_FNS[p]:\n x = f(x)\n if channels_first:\n x = tf.transpose(x, [0, 3, 1, 2])\n return x\n\n\n def rand_brightness(x):\n magnitude = tf.random.uniform([tf.shape(x)[0], 1, 1, 1]) - 0.5\n x = x + magnitude\n return x\n\n\n def rand_saturation(x):\n magnitude = tf.random.uniform([tf.shape(x)[0], 1, 1, 1]) * 2\n x_mean = tf.reduce_sum(x, axis=3, keepdims=True) * 0.3333333333333333333\n x = (x - x_mean) * magnitude + x_mean\n return x\n\n\n def rand_contrast(x):\n magnitude = tf.random.uniform([tf.shape(x)[0], 1, 1, 1]) + 0.5\n x_mean = tf.reduce_sum(x, axis=[1, 2, 3], keepdims=True) * 5.086e-6\n x = (x - x_mean) * magnitude + x_mean\n return x\n\n def rand_translation(x, ratio=0.125):\n batch_size = tf.shape(x)[0]\n image_size = tf.shape(x)[1:3]\n shift = tf.cast(tf.cast(image_size, tf.float32) * ratio + 0.5, tf.int32)\n translation_x = tf.random.uniform([batch_size, 1], -shift[0], shift[0] + 1, dtype=tf.int32)\n translation_y = tf.random.uniform([batch_size, 1], -shift[1], shift[1] + 1, dtype=tf.int32)\n grid_x = tf.clip_by_value(tf.expand_dims(tf.range(image_size[0], dtype=tf.int32), 0) + translation_x + 1, 0, image_size[0] + 1)\n grid_y = tf.clip_by_value(tf.expand_dims(tf.range(image_size[1], dtype=tf.int32), 0) + translation_y + 1, 0, image_size[1] + 1)\n x = tf.gather_nd(tf.pad(x, [[0, 0], [1, 1], [0, 0], [0, 0]]), tf.expand_dims(grid_x, -1), batch_dims=1)\n x = tf.transpose(tf.gather_nd(tf.pad(tf.transpose(x, [0, 2, 1, 3]), [[0, 0], [1, 1], [0, 0], [0, 0]]), tf.expand_dims(grid_y, -1), batch_dims=1), [0, 2, 1, 3])\n return x\n\n\n def rand_cutout(x, ratio=0.5):\n batch_size = tf.shape(x)[0]\n image_size = tf.shape(x)[1:3]\n cutout_size = tf.cast(tf.cast(image_size, tf.float32) * ratio + 0.5, tf.int32)\n offset_x = tf.random.uniform([tf.shape(x)[0], 1, 1], maxval=image_size[0] + (1 - cutout_size[0] % 2), dtype=tf.int32)\n offset_y = tf.random.uniform([tf.shape(x)[0], 1, 1], maxval=image_size[1] + (1 - cutout_size[1] % 2), dtype=tf.int32)\n grid_batch, grid_x, grid_y = tf.meshgrid(tf.range(batch_size, dtype=tf.int32), tf.range(cutout_size[0], dtype=tf.int32), tf.range(cutout_size[1], dtype=tf.int32), indexing='ij')\n cutout_grid = tf.stack([grid_batch, grid_x + offset_x - cutout_size[0] // 2, grid_y + offset_y - cutout_size[1] // 2], axis=-1)\n mask_shape = tf.stack([batch_size, image_size[0], image_size[1]])\n cutout_grid = tf.maximum(cutout_grid, 0)\n cutout_grid = tf.minimum(cutout_grid, tf.reshape(mask_shape - 1, [1, 1, 1, 3]))\n mask = tf.maximum(1 - tf.scatter_nd(cutout_grid, tf.ones([batch_size, cutout_size[0], cutout_size[1]], dtype=tf.float32), mask_shape), 0)\n x = x * tf.expand_dims(mask, axis=3)\n return x\n\n\n AUGMENT_FNS = {\n 'color': [rand_brightness, rand_saturation, rand_contrast],\n 'translation': [rand_translation],\n 'cutout': [rand_cutout],\n}\n def aug_fn(image):\n return DiffAugment(image,\"color,translation,cutout\")","metadata":{"papermill":{"duration":0.035568,"end_time":"2023-11-09T16:07:14.640632","exception":false,"start_time":"2023-11-09T16:07:14.605064","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:00.082319Z","iopub.execute_input":"2023-11-20T18:08:00.082570Z","iopub.status.idle":"2023-11-20T18:08:00.106842Z","shell.execute_reply.started":"2023-11-20T18:08:00.082544Z","shell.execute_reply":"2023-11-20T18:08:00.106031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_augment_flip(image):\n image = tf.image.random_flip_left_right(image)\n return image\n\n\ndef augment_image(image): # input data augmentation\n x = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n y = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n \n if y > .5: # random crop image\n image = tf.image.resize(image, [286, 286])\n image = tf.image.random_crop(image, size=[256, 256, 3])\n \n if x > .6: # random flip image\n image = tf.image.random_flip_left_right(image)\n \n return image\n\n","metadata":{"papermill":{"duration":0.023584,"end_time":"2023-11-09T16:07:14.677244","exception":false,"start_time":"2023-11-09T16:07:14.653660","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:00.107791Z","iopub.execute_input":"2023-11-20T18:08:00.108045Z","iopub.status.idle":"2023-11-20T18:08:00.125260Z","shell.execute_reply.started":"2023-11-20T18:08:00.108018Z","shell.execute_reply":"2023-11-20T18:08:00.124434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dataset(filenames):\n dataset = tf.data.TFRecordDataset(filenames)\n dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n return dataset\n\nphoto_ds = load_dataset(PHOTO_FILENAMES)\nmonet_ds = tf.data.Dataset.from_tensor_slices(MONET_FILENAMES).map(load_im)\nmonet_ds = monet_ds.repeat()\nphoto_ds = photo_ds.repeat()\n\ngan_ds = tf.data.Dataset.zip((monet_ds, photo_ds)).batch(BATCH_SIZE, drop_remainder=True).prefetch(AUTOTUNE)\n\n\nfast_photo_ds = load_dataset(PHOTO_FILENAMES).batch(8*strategy.num_replicas_in_sync).prefetch(4)\nfid_photo_ds = load_dataset(PHOTO_FILENAMES).take(2048).batch(8*strategy.num_replicas_in_sync, drop_remainder=True).prefetch(4)\nfid_monet_ds = tf.data.Dataset.from_tensor_slices(MONET_FILENAMES).map(load_im).batch(8*strategy.num_replicas_in_sync, drop_remainder=True).prefetch(4)","metadata":{"papermill":{"duration":0.264961,"end_time":"2023-11-09T16:07:14.955143","exception":false,"start_time":"2023-11-09T16:07:14.690182","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:00.126260Z","iopub.execute_input":"2023-11-20T18:08:00.126612Z","iopub.status.idle":"2023-11-20T18:08:00.371602Z","shell.execute_reply.started":"2023-11-20T18:08:00.126576Z","shell.execute_reply":"2023-11-20T18:08:00.370779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## FID Estimate","metadata":{"papermill":{"duration":0.013765,"end_time":"2023-11-09T16:07:14.982357","exception":false,"start_time":"2023-11-09T16:07:14.968592","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print('1 in')\n# This is not real FID from Kaggle competition. Real FID function is impemented in Tensorflow 1 and may be too slow. This is my fast but not precise version\nwith strategy.scope():\n\n inception_layer = tf.keras.applications.inception_v3.InceptionV3(input_shape=(256,256,3),pooling=\"avg\",include_top=False)\n\n mix3 = inception_layer.get_layer(\"mixed9\").output\n f0 = tf.keras.layers.GlobalMaxPooling2D()(mix3)\n\n inception_model = tf.keras.Model(inputs=inception_layer.input, outputs=f0)\n inception_model.trainable = False\n\n \n def calculate_activation_statistics_mod(images,fid_model):\n act=fid_model.predict(images)\n mu = np.mean(act, axis=0)\n\n sigma = np.cov(act, rowvar=False)\n\n return mu, sigma\n\nmyFID_mu2, myFID_sigma2 = calculate_activation_statistics_mod(fid_monet_ds,inception_model)\nprint('end')","metadata":{"papermill":{"duration":10.09646,"end_time":"2023-11-09T16:07:25.091411","exception":false,"start_time":"2023-11-09T16:07:14.994951","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:00.372512Z","iopub.execute_input":"2023-11-20T18:08:00.372769Z","iopub.status.idle":"2023-11-20T18:08:10.339497Z","shell.execute_reply.started":"2023-11-20T18:08:00.372732Z","shell.execute_reply":"2023-11-20T18:08:10.338306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n def calculate_frechet_distance(mu1,sigma1,mu2,sigma2):\n fid_epsilon = 1e-14\n mu1 = np.atleast_1d(mu1)\n mu2 = np.atleast_1d(mu2)\n sigma1 = np.atleast_2d(sigma1)\n sigma2 = np.atleast_2d(sigma2)\n\n assert mu1.shape == mu2.shape, 'Training and test mean vectors have different lengths'\n assert sigma1.shape == sigma2.shape, 'Training and test covariances have different dimensions'\n\n # product might be almost singular\n covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n \n\n if not np.isfinite(covmean).all():\n msg = f'fid calculation produces singular product; adding {fid_epsilon} to diagonal of cov estimates'\n warnings.warn(msg)\n offset = np.eye(sigma1.shape[0]) * fid_epsilon\n covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n \n # numerical error might give slight imaginary component\n if np.iscomplexobj(covmean):\n# if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n# m = np.max(np.abs(covmean.imag))\n# raise ValueError(f'Imaginary component {m}')\n covmean = covmean.real\n\n tr_covmean = np.trace(covmean)\n\n return (mu1 - mu2).dot(mu1 - mu2) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n\n\n \n \n def FID(images,gen_model,inception_model=inception_model,myFID_mu2=myFID_mu2, myFID_sigma2=myFID_sigma2):\n inp = layers.Input(shape=[256, 256, 3], name='input_image')\n x = monet_generator(inp, training = False)\n x=inception_model(x)\n fid_model = tf.keras.Model(inputs=inp, outputs=x)\n \n mu1, sigma1= calculate_activation_statistics_mod(images,fid_model)\n\n fid_value = calculate_frechet_distance(mu1, sigma1,myFID_mu2, myFID_sigma2)\n\n return fid_value","metadata":{"papermill":{"duration":0.054333,"end_time":"2023-11-09T16:07:25.192702","exception":false,"start_time":"2023-11-09T16:07:25.138369","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.342855Z","iopub.execute_input":"2023-11-20T18:08:10.343147Z","iopub.status.idle":"2023-11-20T18:08:10.355910Z","shell.execute_reply.started":"2023-11-20T18:08:10.343117Z","shell.execute_reply":"2023-11-20T18:08:10.354734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CLIP model and preprocessing according to CLIPConfig","metadata":{"papermill":{"duration":0.011651,"end_time":"2023-11-09T16:07:25.216045","exception":false,"start_time":"2023-11-09T16:07:25.204394","status":"completed"},"tags":[]}},{"cell_type":"code","source":" \n def get_scale_layer():\n mean = np.array([0.48145466,0.4578275,0.40821073]) * 2 - 1 \n std = np.array([0.26862954,0.26130258,0.27577711]) * 2 \n scaling_layer = keras.layers.Lambda(lambda x: ( tf.cast(x, tf.float32) - mean) / std )\n\n return scaling_layer\n \n \n def get_clip_model():\n\n layer_scaling = get_scale_layer()\n layer_permute = tf.keras.layers.Permute((3,1,2))\n backbone = TFCLIPVisionModel.from_pretrained(\"openai/clip-vit-large-patch14\")\n \n inp = tf.keras.layers.Input(shape = [256, 256, 3]) # [B, C, H, W]\n x = inp[:,16:240,16:240,:]\n x = layer_scaling(x)\n x = layer_permute(x)\n \n output = backbone({'pixel_values':x}).pooler_output\n\n return tf.keras.Model(inputs=[inp], outputs=[output])","metadata":{"papermill":{"duration":0.021718,"end_time":"2023-11-09T16:07:25.249075","exception":false,"start_time":"2023-11-09T16:07:25.227357","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.356940Z","iopub.execute_input":"2023-11-20T18:08:10.357194Z","iopub.status.idle":"2023-11-20T18:08:10.380395Z","shell.execute_reply.started":"2023-11-20T18:08:10.357166Z","shell.execute_reply":"2023-11-20T18:08:10.379412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generator and Discriminator","metadata":{"papermill":{"duration":0.011278,"end_time":"2023-11-09T16:07:25.272242","exception":false,"start_time":"2023-11-09T16:07:25.260964","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\nOUTPUT_CHANNELS = 3\n\ndef down_sample(filters, size, apply_instancenorm=True):\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n layer = keras.Sequential()\n layer.add(layers.Conv2D(filters, size, strides=2, padding='same', kernel_initializer=initializer, use_bias=False))\n\n if apply_instancenorm:\n layer.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n layer.add(layers.LeakyReLU())\n\n return layer","metadata":{"papermill":{"duration":0.020042,"end_time":"2023-11-09T16:07:25.303795","exception":false,"start_time":"2023-11-09T16:07:25.283753","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.381658Z","iopub.execute_input":"2023-11-20T18:08:10.381941Z","iopub.status.idle":"2023-11-20T18:08:10.398298Z","shell.execute_reply.started":"2023-11-20T18:08:10.381913Z","shell.execute_reply":"2023-11-20T18:08:10.397164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def up_sample(filters, size, apply_dropout=False):\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n layer = keras.Sequential()\n layer.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same', kernel_initializer=initializer,use_bias=False))\n layer.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n if apply_dropout:\n layer.add(layers.Dropout(0.5))\n\n layer.add(layers.ReLU())\n\n return layer","metadata":{"papermill":{"duration":0.019849,"end_time":"2023-11-09T16:07:25.335338","exception":false,"start_time":"2023-11-09T16:07:25.315489","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.399413Z","iopub.execute_input":"2023-11-20T18:08:10.399690Z","iopub.status.idle":"2023-11-20T18:08:10.411920Z","shell.execute_reply.started":"2023-11-20T18:08:10.399662Z","shell.execute_reply":"2023-11-20T18:08:10.410961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Generator():\n inputs = tf.keras.layers.Input(shape=(256, 256, 3))\n initializer = tf.random_normal_initializer(0., 0.02) \n x = tf.keras.layers.Conv2D(64, 3, padding='same', kernel_initializer=initializer,activation='relu')(inputs)\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Conv2D(64, 3, padding='same', kernel_initializer=initializer,activation='relu')(x)\n x = tf.keras.layers.BatchNormalization()(x) # 256*256*64\n \n x1 = tf.keras.layers.MaxPooling2D(padding='same')(x) # 128*128*64\n \n x1 = tf.keras.layers.Conv2D(128, 3, padding='same', kernel_initializer=initializer,activation='relu')(x1)\n x1 = tf.keras.layers.BatchNormalization()(x1)\n x1 = tf.keras.layers.Conv2D(128, 3, padding='same', kernel_initializer=initializer,activation='relu')(x1)\n x1 = tf.keras.layers.BatchNormalization()(x1) # 128*128*128\n \n x2 = tf.keras.layers.MaxPooling2D(padding='same')(x1) # 64*64*128\n \n x2 = tf.keras.layers.Conv2D(256, 3, padding='same', kernel_initializer=initializer,activation='relu')(x2)\n x2 = tf.keras.layers.BatchNormalization()(x2)\n x2 = tf.keras.layers.Conv2D(256, 3, padding='same', kernel_initializer=initializer,activation='relu')(x2)\n x2 = tf.keras.layers.BatchNormalization()(x2) # 64*64*256\n \n x3 = tf.keras.layers.MaxPooling2D(padding='same')(x2) # 32*32*256\n \n x3 = tf.keras.layers.Conv2D(512, 3, padding='same', kernel_initializer=initializer,activation='relu')(x3)\n x3 = tf.keras.layers.BatchNormalization()(x3)\n x3 = tf.keras.layers.Conv2D(512, 3, padding='same', kernel_initializer=initializer,activation='relu')(x3)\n x3 = tf.keras.layers.BatchNormalization()(x3) # 32*32*512\n \n x4 = tf.keras.layers.MaxPooling2D(padding='same')(x3) # 16*16*512\n \n x4 = tf.keras.layers.Conv2D(1024, 3, padding='same', kernel_initializer=initializer,activation='relu')(x4)\n x4 = tf.keras.layers.BatchNormalization()(x4)\n x4 = tf.keras.layers.Conv2D(1024, 3, padding='same', kernel_initializer=initializer,activation='relu')(x4)\n x4 = tf.keras.layers.BatchNormalization()(x4) # 16*16*1024\n \n x14 = tf.keras.layers.MaxPooling2D(padding='same')(x4) # 16*16*512\n \n x14 = tf.keras.layers.Conv2D(2048, 3, padding='same', kernel_initializer=initializer,activation='relu')(x14)\n x14 = tf.keras.layers.BatchNormalization()(x14)\n x14 = tf.keras.layers.Conv2D(2048, 3, padding='same', kernel_initializer=initializer,activation='relu')(x14)\n x14 = tf.keras.layers.BatchNormalization()(x14) # 8*8*2048\n \n x15 = tf.keras.layers.Conv2DTranspose(1024, 2, strides=2, padding='same', kernel_initializer=initializer,activation='relu')(x14)\n x15 = tf.keras.layers.BatchNormalization()(x15) # 32*32*512\n \n x16 = tf.concat([x4, x15], axis=-1) # 32*32*1024\n \n x16 = tf.keras.layers.Conv2D(1024, 3, padding='same', kernel_initializer=initializer,activation='relu')(x16)\n x16 = tf.keras.layers.BatchNormalization()(x16)\n x16 = tf.keras.layers.Conv2D(1024, 3, padding='same', kernel_initializer=initializer,activation='relu')(x16)\n x16 = tf.keras.layers.BatchNormalization()(x16) # 32*32*512\n \n x5 = tf.keras.layers.Conv2DTranspose(1024, 2, strides=2, padding='same', kernel_initializer=initializer,activation='relu')(x16)\n x5 = tf.keras.layers.BatchNormalization()(x5) # 32*32*512\n \n x6 = tf.concat([x3, x5], axis=-1) # 32*32*1024\n \n x6 = tf.keras.layers.Conv2D(512, 3, padding='same', kernel_initializer=initializer,activation='relu')(x6)\n x6 = tf.keras.layers.BatchNormalization()(x6)\n x6 = tf.keras.layers.Conv2D(512, 3, padding='same', kernel_initializer=initializer,activation='relu')(x6)\n x6 = tf.keras.layers.BatchNormalization()(x6) # 32*32*512\n \n x7 = tf.keras.layers.Conv2DTranspose(256, 2, strides=2, padding='same', kernel_initializer=initializer,activation='relu')(x6)\n x7 = tf.keras.layers.BatchNormalization()(x7) # 64*64*256\n \n x8 = tf.concat([x2, x7], axis=-1) # 64*64*512\n \n x8 = tf.keras.layers.Conv2D(256, 3, padding='same', kernel_initializer=initializer,activation='relu')(x8)\n x8 = tf.keras.layers.BatchNormalization()(x8)\n x8 = tf.keras.layers.Conv2D(256, 3, padding='same', kernel_initializer=initializer,activation='relu')(x8)\n x8 = tf.keras.layers.BatchNormalization()(x8) # 64*64*256\n \n x9 = tf.keras.layers.Conv2DTranspose(128, 2, strides=2, padding='same', kernel_initializer=initializer,activation='relu')(x8)\n x9 = tf.keras.layers.BatchNormalization()(x9) # 128*128*128\n \n x10 = tf.concat([x1, x9], axis=-1) # 128*128*256\n \n x10 = tf.keras.layers.Conv2D(128, 3, padding='same', kernel_initializer=initializer,activation='relu')(x10)\n x10 = tf.keras.layers.BatchNormalization()(x10)\n x10 = tf.keras.layers.Conv2D(128, 3, padding='same', kernel_initializer=initializer,activation='relu')(x10)\n x10 = tf.keras.layers.BatchNormalization()(x10) # 128*128*128\n \n x11 = tf.keras.layers.Conv2DTranspose(64, 2, strides=2, padding='same', kernel_initializer=initializer,activation='relu')(x10)\n x11 = tf.keras.layers.BatchNormalization()(x11) # 256*256*64\n \n x12 = tf.concat([x, x11], axis=-1) # 256*256*128\n \n x12 = tf.keras.layers.Conv2D(64, 3, padding='same', kernel_initializer=initializer,activation='relu')(x12)\n x12 = tf.keras.layers.BatchNormalization()(x12)\n x12 = tf.keras.layers.Conv2D(64, 3, padding='same', kernel_initializer=initializer,activation='relu')(x12)\n x12 = tf.keras.layers.BatchNormalization()(x12) # 256*256*64\n \n\n outputs = tf.keras.layers.Conv2D(3, 1,kernel_initializer=initializer, activation='tanh')(x12) # 256*256*3\n \n return tf.keras.Model(inputs=inputs, outputs=outputs)","metadata":{"papermill":{"duration":0.041056,"end_time":"2023-11-09T16:07:25.387869","exception":false,"start_time":"2023-11-09T16:07:25.346813","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.413473Z","iopub.execute_input":"2023-11-20T18:08:10.413743Z","iopub.status.idle":"2023-11-20T18:08:10.438439Z","shell.execute_reply.started":"2023-11-20T18:08:10.413716Z","shell.execute_reply":"2023-11-20T18:08:10.437466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Discriminator():\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n \n inp = layers.Input(shape=[256, 256, 3], name='input_image')\n x = inp\n \n down1 = down_sample(64, 4, False)(x) # (size, 128, 128, 64)\n down2 = down_sample(128, 4)(down1) # (size, 64, 64, 128)\n down3 = down_sample(256, 4)(down2) # (size, 32, 32, 256)\n\n zero_pad1 = layers.ZeroPadding2D()(down3) # (size, 34, 34, 256)\n conv = layers.Conv2D(512, 4, strides=1, kernel_initializer=initializer, use_bias=False)(zero_pad1) # (size, 31, 31, 512)\n\n norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n leaky_relu = layers.LeakyReLU()(norm1)\n zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (size, 33, 33, 512)\n last = layers.Conv2D(1, 4, strides=1, kernel_initializer=initializer)(zero_pad2) # (size, 30, 30, 1)\n\n return tf.keras.Model(inputs=inp, outputs=last)\n\n","metadata":{"papermill":{"duration":0.021423,"end_time":"2023-11-09T16:07:25.421035","exception":false,"start_time":"2023-11-09T16:07:25.399612","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.439527Z","iopub.execute_input":"2023-11-20T18:08:10.439798Z","iopub.status.idle":"2023-11-20T18:08:10.454388Z","shell.execute_reply.started":"2023-11-20T18:08:10.439771Z","shell.execute_reply":"2023-11-20T18:08:10.453733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CLIP model","metadata":{"papermill":{"duration":0.01294,"end_time":"2023-11-09T16:07:25.446948","exception":false,"start_time":"2023-11-09T16:07:25.434008","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\nfrom transformers import CLIPProcessor, TFCLIPVisionModel, CLIPFeatureExtractor\n","metadata":{"papermill":{"duration":22.030447,"end_time":"2023-11-09T16:07:47.489931","exception":false,"start_time":"2023-11-09T16:07:25.459484","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:10.455168Z","iopub.execute_input":"2023-11-20T18:08:10.455390Z","iopub.status.idle":"2023-11-20T18:08:32.298884Z","shell.execute_reply.started":"2023-11-20T18:08:10.455367Z","shell.execute_reply":"2023-11-20T18:08:32.297926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n monet_generator = Generator() # transforms photos to Monet-esque paintings\n monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n clip_net = get_clip_model()\n clip_net.trainable = False","metadata":{"_kg_hide-output":true,"papermill":{"duration":55.763372,"end_time":"2023-11-09T16:08:43.266622","exception":false,"start_time":"2023-11-09T16:07:47.503250","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:08:32.299957Z","iopub.execute_input":"2023-11-20T18:08:32.300667Z","iopub.status.idle":"2023-11-20T18:09:26.837611Z","shell.execute_reply.started":"2023-11-20T18:08:32.300632Z","shell.execute_reply":"2023-11-20T18:09:26.836430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CLIPTraVeLGan Class","metadata":{"papermill":{"duration":0.015649,"end_time":"2023-11-09T16:08:43.297935","exception":false,"start_time":"2023-11-09T16:08:43.282286","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class CLIPTraVeLGan(keras.Model):\n def __init__(\n self,\n monet_generator,\n monet_discriminator,\n siames_net,\n lambda_id=0.00001 # balance between adversarial loss and clip loss\n ):\n super(CLIPTraVeLGan, self).__init__()\n self.m_gen = monet_generator\n self.m_disc = monet_discriminator\n self.siames_net = siames_net\n self.lambda_id = lambda_id\n\n \n def compile(\n self,\n m_gen_optimizer,\n m_disc_optimizer,\n gen_loss_fn,\n disc_loss_fn,\n siames_loss_fn\n ):\n super(CLIPTraVeLGan, self).compile()\n self.m_gen_optimizer = m_gen_optimizer\n self.m_disc_optimizer = m_disc_optimizer\n self.gen_loss_fn = gen_loss_fn\n self.disc_loss_fn = disc_loss_fn\n self.siames_loss_fn = siames_loss_fn\n \n \n def train_step(self, batch_data):\n real_monet, real_photo = batch_data\n batch_size = tf.shape(real_monet)[0]\n semantic_real = self.siames_net(real_photo, training=False) # CLIP embedding of real\n with tf.GradientTape(persistent=True) as tape:\n\n \n fake_monet = self.m_gen(real_photo, training=True)\n semantic_fake = self.siames_net(fake_monet, training=False) # CLIP embedding of fake\n \n \n ################## My code #####################\n \n both_monet = tf.concat([real_monet, fake_monet], axis=0) \n \n aug_monet = aug_fn(both_monet)\n \n aug_real_monet = aug_monet[:batch_size]\n aug_fake_monet = aug_monet[batch_size:]\n \n ################ End of my code #################\n \n # discriminator used to check, inputing real images\n disc_real_monet = self.m_disc(aug_real_monet, training=True) # aug_real_monet\n\n # discriminator used to check, inputing fake images\n disc_fake_monet = self.m_disc(aug_fake_monet, training=True) # aug_fake_monet\n\n \n\n # evaluates generator loss\n monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n\n # evaluates discriminator loss\n monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n \n\n # travel loss on CLIP embeddings\n\n monet_travel_loss = self.siames_loss_fn(semantic_real,semantic_fake)\n\n # evaluates generator loss\n total_monet_gen_loss = self.gen_loss_fn(disc_fake_monet) + monet_travel_loss*self.lambda_id\n \n\n \n # Calculate the gradients for generator and discriminator\n monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n self.m_disc.trainable_variables)\n\n monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n self.m_gen.trainable_variables)\n\n\n self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n self.m_gen.trainable_variables))\n\n\n self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n self.m_disc.trainable_variables))\n\n\n \n return {\n \"disc_real_monet\":disc_real_monet,\n \"disc_fake_monet\": disc_fake_monet,\n \"monet_disc_loss\": monet_disc_loss,\n \"monet_travel_loss\" : monet_travel_loss,\n }\n\n ","metadata":{"papermill":{"duration":0.031145,"end_time":"2023-11-09T16:08:43.344020","exception":false,"start_time":"2023-11-09T16:08:43.312875","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:09:26.838928Z","iopub.execute_input":"2023-11-20T18:09:26.839224Z","iopub.status.idle":"2023-11-20T18:09:26.851263Z","shell.execute_reply.started":"2023-11-20T18:09:26.839195Z","shell.execute_reply":"2023-11-20T18:09:26.850276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loss functions","metadata":{"papermill":{"duration":0.01515,"end_time":"2023-11-09T16:08:43.374576","exception":false,"start_time":"2023-11-09T16:08:43.359426","status":"completed"},"tags":[]}},{"cell_type":"code","source":"with strategy.scope():\n def discriminator_loss(real, generated):\n real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n total_disc_loss = real_loss + generated_loss\n\n return total_disc_loss * 0.5\n","metadata":{"papermill":{"duration":0.023556,"end_time":"2023-11-09T16:08:43.413210","exception":false,"start_time":"2023-11-09T16:08:43.389654","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:09:26.852191Z","iopub.execute_input":"2023-11-20T18:09:26.852435Z","iopub.status.idle":"2023-11-20T18:09:26.870779Z","shell.execute_reply.started":"2023-11-20T18:09:26.852402Z","shell.execute_reply":"2023-11-20T18:09:26.870031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n def generator_loss(generated):\n return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n","metadata":{"papermill":{"duration":0.022225,"end_time":"2023-11-09T16:08:43.450596","exception":false,"start_time":"2023-11-09T16:08:43.428371","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:09:26.871742Z","iopub.execute_input":"2023-11-20T18:09:26.871985Z","iopub.status.idle":"2023-11-20T18:09:26.882411Z","shell.execute_reply.started":"2023-11-20T18:09:26.871961Z","shell.execute_reply":"2023-11-20T18:09:26.881695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n def siames_loss(s1_x, s1_g):\n orders = np.array([list(range(i, 128)) + list(range(i)) for i in range(1, 128)]) # change 128 to batch_size\n orders = tf.constant(orders)\n \n orders2 = np.array([list(range(0, 128)) for i in range(1, 128)]) # change 128 to batch_size\n orders2 = tf.constant(orders2)\n\n\n dists_within_x1 = tf.gather(s1_x, orders2) - tf.gather(s1_x, orders)\n dists_within_g1 = tf.gather(s1_g, orders2) - tf.gather(s1_g, orders)\n \n cosine_loss = tf.keras.losses.CosineSimilarity(axis = 1,reduction=tf.keras.losses.Reduction.NONE)\n losses_travel_1 = tf.reduce_sum(cosine_loss(dists_within_x1, dists_within_g1) + 1)\n\n \n return losses_travel_1","metadata":{"papermill":{"duration":0.024829,"end_time":"2023-11-09T16:08:43.490408","exception":false,"start_time":"2023-11-09T16:08:43.465579","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:09:26.883367Z","iopub.execute_input":"2023-11-20T18:09:26.883613Z","iopub.status.idle":"2023-11-20T18:09:26.897540Z","shell.execute_reply.started":"2023-11-20T18:09:26.883589Z","shell.execute_reply":"2023-11-20T18:09:26.896791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create a model","metadata":{"papermill":{"duration":0.015471,"end_time":"2023-11-09T16:08:43.520910","exception":false,"start_time":"2023-11-09T16:08:43.505439","status":"completed"},"tags":[]}},{"cell_type":"code","source":"with strategy.scope():\n gan_model = CLIPTraVeLGan(monet_generator, monet_discriminator, clip_net)\n\n monet_generator.built = True\n monet_discriminator.built = True\n","metadata":{"papermill":{"duration":0.03452,"end_time":"2023-11-09T16:08:43.570378","exception":false,"start_time":"2023-11-09T16:08:43.535858","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-11-20T18:09:26.898433Z","iopub.execute_input":"2023-11-20T18:09:26.898647Z","iopub.status.idle":"2023-11-20T18:09:26.921084Z","shell.execute_reply.started":"2023-11-20T18:09:26.898625Z","shell.execute_reply":"2023-11-20T18:09:26.920356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{"papermill":{"duration":0.015542,"end_time":"2023-11-09T16:08:43.600980","exception":false,"start_time":"2023-11-09T16:08:43.585438","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nfids=[]\ndisc_m_r=[]\ndisc_m_f=[]\ndisc_m_loss=[]\nbest_fid=999999999\n\nwith strategy.scope():\n for (lr, stg, ep) in [(2e-4, 7, 1),(1e-4, 5, 1),(3e-5, 2, 1)]:\n print(f\"Learnning rate = {lr}\")\n monet_generator_optimizer = tf.keras.optimizers.Adam(lr, beta_1=0.5)\n monet_discriminator_optimizer = tf.keras.optimizers.Adam(lr * 2, beta_1=0.5) \n\n gan_model.compile(\n m_gen_optimizer = monet_generator_optimizer,\n m_disc_optimizer = monet_discriminator_optimizer,\n gen_loss_fn = generator_loss,\n disc_loss_fn = discriminator_loss,\n siames_loss_fn = siames_loss,\n )\n\n for stage in range(1,stg+1):\n print(\"Stage = \", stage)\n hist = gan_model.fit(gan_ds,steps_per_epoch=1400, epochs=ep).history\n disc_m_loss.append(hist[\"monet_disc_loss\"][0])\n# cur_fid = FID(fid_photo_ds, monet_generator)\n# fids.append(cur_fid)\n# print(\"After stage #{} FID = {} \\n\".format(stage, cur_fid))\n \n# if cur_fid