{"cells":[{"metadata":{},"cell_type":"markdown","source":"# PPGAN notebook version for MLiP\n\nA lot of code is reused from the Patch Permutation GAN implementation by to i-evi (https://github.com/i-evi/p2gan)\nTheir code is not a notebook and only works on Tensorflow 1, so most of our adjustments to this code are to make it work with TF2 as a notebook.\nCredits to the authors of the Patch Permuation GAN paper (https://arxiv.org/abs/2001.07466)"},{"metadata":{},"cell_type":"markdown","source":"## Imports\nWe use the tf_slim package and tensorflow.compat.v1 library to make the TF1 code from the paper work in TF2 (which is what Kaggle uses).\nFurthermore, we also download the vgg16.npy weights file which you normally would manually place in the project folder."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Required slim API for tf2 and vgg16 weights file\n!pip install tf_slim\n!wget https://www.dropbox.com/s/8a8rei66f72um4i/vgg16.npy\n    \n!wget https://www.b4rt.nl/files/SUN397Landscape-ish_256x256.zip\n!unzip -qq SUN397Landscape-ish_256x256.zip\n\n# Libraries\nimport os\nimport random\nimport time\nimport json\nimport cv2\nimport inspect\nimport shutil\nimport numpy as np\nfrom matplotlib import pyplot as plt\n\n# Compatibility imports (to run the tf1 code in tf2)\nimport tf_slim as slim\nimport tensorflow.compat.v1 as tf\ntf.disable_v2_behavior()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Parameters\nOriginally, many of these parameters were set in a JSON config file, or placed in some of the Python modules.\nSince we made a notebook version, we placed all configurable parameters here and removed the JSON configuration."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Most important parameters for our experiments\nBATCH_SIZE   = 8\nLAMBDA       = 0.000005 # \"hyper parameter to balance content and style\" default: 0.000005, higher == more content\nTRAIN_EPOCHS = 10\n\n# Image size\nPSI_D_SIZE   = 256 \nG_IMG_SIZE   = 256\nPATCH_SIZE   = 16 # Size of patches\n\n# VGG params\nVGG_L        = 1\nVGG_FEATURES = 64\n\n# Gen/Discrim params\nDISCRIMINATOR_MAX_ITER = 1\nGENERATOR_MAX_ITER = 2\n\n# Training paths\nSTYLE_PATH      = \"/kaggle/input/gan-getting-started/monet_jpg/\" # Do not forget trailing slash\nTRAINSET_PATH   = \"/kaggle/working/SUN397/\" #\"/kaggle/input/gan-getting-started/photo_jpg/\" # Do not forget trailing slash\n\n# Render paths\nIMGSRC_PATH = \"/kaggle/working/SUN397/\" #\"/kaggle/input/gan-getting-started/photo_jpg/\" # path of input images to convert. Do not forget trailing slash\nOUT_PATH    = \"/kaggle/working/images/\" # path of output images. Do not forget trailing slash\n\ngpu_options = tf.GPUOptions(allow_growth=True)\n\n# List of style images we want to use\n#STYLE_IMAGE_LIST = [n for n in os.listdir(STYLE_PATH)]\n\n# Custom list of hand picked style images\n#STYLE_IMAGE_LIST = [\"0bd913dbc7.jpg\", \"3b262c6726.jpg\", \"4c74254ad3.jpg\", \"6f0b9df5c5.jpg\", \"19dc36ccb2.jpg\", \"4c0e35882c.jpg\", \"5cb895b722.jpg\", \"92c0ba8c0d.jpg\"]\n#STYLE_IMAGE_LIST = [\"e753318d04.jpg\"]\nSTYLE_IMAGE_LIST = [\"000c1e3bff.jpg\", \"0bd913dbc7.jpg\", \"0e3b3292da.jpg\", \"1a127acf4d.jpg\", \"1e4e4e63c5.jpg\", \"1f9667f2a7.jpg\", \"1f22663e72.jpg\", \"2acfbab228.jpg\",\n                    \"2c00f5147f.jpg\", \"2cca56415e.jpg\", \"2e0d0e6e19.jpg\", \"2f90c99e10.jpg\", \"2f20944b6a.jpg\", \"3b262c6726.jpg\", \"3c341ff93e.jpg\", \"3d13fe022e.jpg\",\n                    \"3deea9f4a4.jpg\", \"3eaef3ee43.jpg\", \"4ab2583fe2.jpg\", \"4ad8b366c1.jpg\", \"4b0adf7c6f.jpg\", \"4c0e35882c.jpg\", \"4c74254ad3.jpg\", \"4e05523825.jpg\",\n                    \"4f4de0bbba.jpg\", \"4f7e01f097.jpg\", \"4f045779b0.jpg\", \"05b493ff42.jpg\", \"5aa514ffac.jpg\", \"5c79cfe0b3.jpg\", \"5cb895b722.jpg\", \"5ffbfe68d4.jpg\",\n                    \"6bbe0e63c6.jpg\", \"6bfbd1df5b.jpg\", \"6c6cc46498.jpg\", \"6d0e87f557.jpg\", \"6e0429f92e.jpg\", \"6ee7c39dbc.jpg\", \"6f0b9df5c5.jpg\", \"07fcaee35f.jpg\",\n                    \"7cb36714d0.jpg\", \"7d64c3100c.jpg\", \"8b54448a07.jpg\", \"8c48e112d0.jpg\", \"8c8011c291.jpg\", \"8e5ff15568.jpg\", \"8f02369f42.jpg\", \"9f409e3376.jpg\",\n                    \"9fc868e864.jpg\", \"10c555c1b1.jpg\", \"11ab570c5e.jpg\", \"19dc36ccb2.jpg\", \"22b1ac6b44.jpg\", \"23f0fbd77e.jpg\", \"24af733334.jpg\", \"25c9904782.jpg\",\n                    \"26b66eb819.jpg\", \"32cc820303.jpg\", \"32e33792cc.jpg\", \"40d7d18ad3.jpg\", \"049e293b93.jpg\", \"052a77c020.jpg\", \"52aed0f943.jpg\", \"52d12dc627.jpg\",\n                    \"52fc351abf.jpg\", \"058f878b7c.jpg\", \"59df696966.jpg\", \"61e735361a.jpg\", \"066fe4cbaa.jpg\", \"66a144f547.jpg\", \"68b60c04b7.jpg\", \"68d60af226.jpg\",\n                    \"69f4b75a37.jpg\", \"73f33a12c5.jpg\", \"74e452fb31.jpg\", \"76cc7181f8.jpg\", \"77b37629f2.jpg\", \"82b9fd68b1.jpg\", \"89d970411d.jpg\", \"89fcbf2f76.jpg\",\n                    \"92c0ba8c0d.jpg\", \"95a53d7b0b.jpg\", \"95b5f01a85.jpg\", \"99a51d3e25.jpg\", \"99d94af5dd.jpg\", \"118da0690c.jpg\", \"133b42e498.jpg\", \"184d6c66cd.jpg\",\n                    \"252d9a4abc.jpg\", \"281b73fb5e.jpg\", \"344d1829bb.jpg\", \"417e77e946.jpg\", \"429e382095.jpg\", \"512cd171a9.jpg\", \"526a110636.jpg\", \"536aa87152.jpg\",\n                    \"565f19268b.jpg\", \"586acab7c5.jpg\", \"593db29cce.jpg\", \"608ee0d370.jpg\", \"632ddbc784.jpg\", \"661e374153.jpg\", \"676a5a4c2e.jpg\", \"730f325b64.jpg\",\n                    \"853f8d711f.jpg\", \"893db2701d.jpg\", \"932d0dd808.jpg\", \"990ed28f62.jpg\", \"1814cc6632.jpg\", \"2759c1ed37.jpg\", \"3843e221cc.jpg\", \"4995c04b1a.jpg\",\n                    \"05144e306f.jpg\", \"5185e8c56a.jpg\", \"5926f85cbf.jpg\", \"6043aadea0.jpg\", \"7017e6caa1.jpg\", \"7239ba0b55.jpg\", \"7341d96c1d.jpg\", \"7960adbd50.jpg\",\n                    \"8114fa2607.jpg\", \"8314acfd35.jpg\", \"9843bc25c5.jpg\", \"9908d1daa9.jpg\", \"011835cfbf.jpg\", \"14162de938.jpg\", \"17557a29cb.jpg\", \"23832dead5.jpg\",\n                    \"29696b4455.jpg\", \"49337b68f4.jpg\", \"064487d630.jpg\", \"66226e18fc.jpg\", \"68729aac07.jpg\", \"79224da51f.jpg\", \"79292e1434.jpg\", \"82991e742a.jpg\",\n                    \"89964efa86.jpg\", \"93132f89ee.jpg\", \"106757e5d8.jpg\", \"463835bbc6.jpg\", \"488600cb75.jpg\", \"695897bd4a.jpg\", \"718445ebe3.jpg\", \"910610e827.jpg\",\n                    \"910729e0ce.jpg\", \"1078363ff0.jpg\", \"2581464ddc.jpg\", \"4660310c3e.jpg\", \"7952021d2f.jpg\", \"85580214be.jpg\", \"88402296cc.jpg\", \"158740962c.jpg\",\n                    \"599098859e.jpg\", \"815624563e.jpg\", \"3545597386.jpg\", \"6742294320.jpg\", \"7054793632.jpg\", \"a4e4a61fb2.jpg\", \"a8fbbe3eb1.jpg\", \"a030bc32e6.jpg\",\n                    \"a59f3f5b89.jpg\", \"a96b79a93f.jpg\", \"a210ceedc7.jpg\", \"a885da7b52.jpg\", \"a6291c2a1c.jpg\", \"a7977705be.jpg\", \"ad8ce41fc0.jpg\", \"b1ea5d5a7d.jpg\",\n                    \"b2ce76c750.jpg\", \"b3adc75e7d.jpg\", \"b5c2fe7c4c.jpg\", \"b13c0973ee.jpg\", \"b44f24c048.jpg\", \"b76d52e05a.jpg\", \"b256e61a5d.jpg\", \"b1310da865.jpg\",\n                    \"b99546090b.jpg\", \"ba52f976af.jpg\", \"baf6efabfe.jpg\", \"bbc5ac4564.jpg\", \"bc4b364a44.jpg\", \"bf6db09354.jpg\", \"c1dc1a85a4.jpg\", \"c6c88ce9c4.jpg\",\n                    \"c6c360756c.jpg\", \"c7d8142152.jpg\", \"c67ba2060c.jpg\", \"c68c52e8fc.jpg\", \"c4622e3fb6.jpg\", \"c14505c1da.jpg\", \"c2576267d4.jpg\", \"cb9c553ded.jpg\",\n                    \"cc2bb659f4.jpg\", \"cd6623d07d.jpg\", \"ceb6cf5f31.jpg\", \"cfc6fce7b5.jpg\", \"d05cab011d.jpg\", \"d6d6e625bd.jpg\", \"d14c1abdd4.jpg\", \"d239dae42d.jpg\",\n                    \"d087730b76.jpg\", \"d754850d01.jpg\", \"d88482796d.jpg\", \"da72006ef5.jpg\", \"dc33f0edbe.jpg\", \"dd46691bd7.jpg\", \"de6f71b00f.jpg\", \"df64ac2dcb.jpg\",\n                    \"e9f686534b.jpg\", \"e88d9de918.jpg\", \"e510a74d3c.jpg\", \"e568f84fad.jpg\", \"e2253b87a0.jpg\", \"e9580cd500.jpg\", \"e37407c747.jpg\", \"e753318d04.jpg\",\n                    \"e3112413b1.jpg\", \"eb3cc5c559.jpg\", \"ec78d80dbd.jpg\", \"ec3398cef9.jpg\", \"ed597655a0.jpg\", \"ede9769cb3.jpg\", \"ee7adac58f.jpg\", \"f0d789c4bc.jpg\",\n                    \"f486c1655f.jpg\", \"f0884db067.jpg\", \"f821791c85.jpg\", \"fb3b06dcb2.jpg\", \"fb806a2a1c.jpg\", \"fb93438ff9.jpg\", \"fba982625d.jpg\", \"fc11d52502.jpg\",\n                    \"fd63a333f1.jpg\", \"ffd74c77ea.jpg\"]\n\nSTYLE_IMAGE_COUNT = len(STYLE_IMAGE_LIST)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## UTIL functions\nContain functions for loading images, creating patches, etc.\nMostly unchanged from util.py from the original code. At the end of this code block we made a special function that allows the model to work with multiple style images (instead of just one)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def imbatch_read(path, file_list, shape):\n    if len(file_list) <= 0:\n        return None\n    sls = []\n    batch = None\n    for i, item in enumerate(file_list):\n        img = cv2.imread(path +  item)\n        sls.append((img.shape[1], img.shape[0]))\n        img = cv2.resize(img, shape)\n        img = (img / 255.0 - 0.5) * 2.0\n        img = np.expand_dims(img, axis=0)\n        if not isinstance(batch, np.ndarray):\n            batch = img\n        else:\n            batch = np.vstack((batch, img))\n    return batch, sls\n\ndef img_write(img, path, shape):\n    if len(img.shape) != 3 or img.shape[2] != 3:\n        return False # Only support RGB img\n    img = (((img + 1) / 2) * 255)#.astype(np.uint8)\n    img = cv2.resize(img, shape)\n    cv2.imwrite(path, img)\n    return True\n\ndef background(shape, ch=3):\n    return np.zeros(shape[0] * shape[1] * ch).\\\n        astype(np.uint8).reshape((shape[0], shape[1], ch))\n\ndef crop(img, x, y, xl, yl):\n    if (x + xl) > img.shape[1] or (y + yl) > img.shape[0]:\n        return None\n    return img[y:y+yl, x:x+xl,:]\n\ndef merge(bg, img, x, y):\n    bg[y:y+img.shape[0] if y+img.shape[0] < bg.shape[0] else bg.shape[0],\n        x:x+img.shape[1] if x+img.shape[1] < bg.shape[1] else bg.shape[1],:] =\\\n            img[:img.shape[0] if y+img.shape[0] < bg.shape[0] else bg.shape[0] - y,\n        :img.shape[1] if x+img.shape[1] < bg.shape[1] else bg.shape[1] - x,:]\n\ndef random_crop(img, xl, yl):\n    y = int((img.shape[0] - yl) * random.random())\n    x = int((img.shape[1] - xl) * random.random())\n    return crop(img, x, y, xl, yl)\n\ndef patch_fill(bg, img, offset_x, offset_y, gap, s):\n    if bg.shape[0] != bg.shape[1]:\n        exit('Error filling patches, background or image must be square')\n    # if s % 2 != 1:\n    #   exit('Patch size must be odd, not %d'%s)\n    jump = s + gap\n    cnt = int(bg.shape[0] / (jump))\n    for j in range(cnt):\n        for i in range(cnt):\n            merge(bg, random_crop(img, s, s),\n                i * jump + offset_x, j * jump + offset_y)\n    return bg\n\ndef patch_fill_multi(bg, imgls, offset_x, offset_y, gap, s):\n    if bg.shape[0] != bg.shape[1]:\n        exit('Error filling patches, background or image must be square')\n    # if s % 2 != 1:\n    #   exit('Patch size must be odd, not %d'%s)\n    jump = s + gap\n    cnt = int(bg.shape[0] / (jump))\n    for j in range(cnt):\n        for i in range(cnt):\n            merge(bg, \n                random_crop(imgls[int(random.random() * len(imgls))], s, s),\n                i * jump + offset_x, j * jump + offset_y)\n\n\ndef pickup_list(ls, cnt, begin):\n    ls_len = len(ls)\n    if begin >= ls_len:\n        return None\n    if (begin + cnt) >= ls_len:\n        begin = ls_len - cnt\n    return ls[begin:begin+cnt]\n\n\ndef pickup_list_random(ls, batchSize):\n    return random.sample(ls, batchSize)\n\n\ndef make_input_batch(img, bs, h, w, ps):\n    (_, _, c) = img.shape\n    bat = np.zeros(bs * h * w * c).astype(np.float32).reshape((bs, h, w, c))\n    for i in range(bs):\n        patch_fill(bat[i], img, 0, 0, 0, ps)\n    return bat\n\n\ndef make_input_batch_multi(imgls, bs, h, w, ps):\n    (_, _, c) = imgls[0].shape\n    bat = np.zeros(bs * h * w * c).astype(np.float32).reshape((bs, h, w, c))\n    for i in range(bs):\n        patch_fill_multi(bat[i], imgls, 0, 0, 0, ps)\n    return bat\n\ndef ls_files_to_json(path, jname=None, reverse=False, ext=[]):\n    ls = []\n    for x in os.listdir(path):\n        if os.path.isfile(os.path.join(path, x)):\n            if len(ext) == 0:\n                ls.append(x)\n            else:\n                if x.split('.')[-1] in ext:\n                    ls.append(x)\n    if reverse == None:\n        pass\n    elif reverse == False:\n        ls.sort(reverse=False)\n    elif reverse == True:\n        ls.sort(reverse=True)\n    if isinstance(jname, str):\n        with open(jname, 'w+') as fp:\n            fp.write(json.dumps(ls))\n    return ls\n\ndef open_img(img_path):\n    img = cv2.imread(img_path)\n    img = (img / 255.0 - 0.5) * 2.0\n    return img\n\ndef load_bat_img(img_path, shape, prep=True, singleCh=False, gray=False, remove_pad=False):\n    img = cv2.imread(img_path)\n    if gray:\n        img= cv2.cvtColor(img,cv2.COLOR_RGB2GRAY).reshape((img.shape[0], img.shape[1], 1))\n        timg = np.copy(img)\n        img = np.concatenate((img, timg), axis=2)\n        img = np.concatenate((img, timg), axis=2)\n    if remove_pad:\n        y = img.shape[0]\n        x = img.shape[1]\n        img = img[int(y/8):y - int(y/8), int(x/8):x-int(x/8),:]\n    if isinstance(shape, tuple):\n        img = cv2.resize(img, shape)\n    img = np.expand_dims(img, axis=0)\n    if prep:\n        img = (img / 255.0 - 0.5) * 2.0\n    if singleCh:\n        img = np.mean(img, axis=3)\n        img = np.expand_dims(img, axis=3)\n    return img\n\ndef random_file_list(path, batch_size, rand=True):\n    images_path =[n for n in os.listdir(path)]\n    if (batch_size > len(images_path) or batch_size <= 0):\n        print('N/A')\n        return None\n    if rand:\n        random.shuffle(images_path)\n    return images_path[0:batch_size]\n\ndef images_batch(folder, file_list, shape, prep=True, singleCh=False, gray=False, remove_pad=False):\n    if len(file_list) <= 0:\n        return None\n    batch = load_bat_img(folder+'/'+file_list[0], prep=prep, shape=shape, singleCh=False, gray=gray, remove_pad=remove_pad)\n    for i in range(len(file_list) - 1):\n        batch = np.vstack((batch, load_bat_img(folder+'/'+file_list[i + 1], prep=prep, shape=shape, singleCh=False, gray=gray, remove_pad=remove_pad)))\n    return batch\n\ndef save_as_rgb_img(img, path):\n    if len(img.shape) != 3 or img.shape[2] != 3:\n        return False # Only support RGB img\n    img = (((img + 1) / 2) * 255)\n    cv2.imwrite(path, img)\n    return True\n\ndef save_batch_as_rgb_img(bat, path, prefix=None):\n    code = 0\n    for item in bat:\n        if prefix == None:\n            save_as_rgb_img(item, path+'/%d.jpg'%code)\n        else:\n            save_as_rgb_img(item, path+'/%s%d.jpg'%(prefix, code))\n        code = code + 1\n\ndef random_sub_set(set, n):\n    if n <= 1:\n        return set\n    set_index = np.arange(1, set.shape[0] + 1)\n    subset_index = np.random.choice(set_index, n)\n    batch_data = set10[set10_index[0]]\n    batch_data = np.expand_dims(batch_data, axis=0)\n    for item in subset_index[1:]:\n        batch_data = np.vstack((batch_data, np.expand_dims(set10[item], axis=0)))\n    return batch_data\n\ndef sub_set(inset, index_tab):\n    if len(index_tab) <= 1:\n        return inset\n    batch_data = inset[index_tab[0]]\n    batch_data = np.expand_dims(batch_data, axis=0)\n    for item in index_tab[1:]:\n        batch_data = np.vstack((batch_data, np.expand_dims(inset[item], axis=0)))\n    return batch_data\n\ndef silent_mkdir(path):\n    try:\n        os.makedirs(path)\n    except:\n        pass\n\n############################\n# CUSTOM MULTI STYLE VERSION\n############################\n# Adapted code to use multiple style images instead of one\n    \nGlobalStyleImageCounter = 0\n    \n# Chooses a new style image every time.\n# When all style images are used, it starts over\ndef make_input_batch_path(imgpath, bs, h, w, ps):\n    images_path =[n for n in os.listdir(imgpath)]\n    \n    (_, _, c) = open_img(imgpath + images_path[0]).shape\n    \n    global GlobalStyleImageCounter\n    \n    bat = np.zeros(bs * h * w * c).astype(np.float32).reshape((bs, h, w, c))\n    for i in range(bs):\n        img = open_img(imgpath + STYLE_IMAGE_LIST[GlobalStyleImageCounter]) #random.randint(0,STYLE_IMAGE_COUNT-1)\n        patch_fill(bat[i], img, 0, 0, 0, ps)\n        \n        # Update style image counter\n        GlobalStyleImageCounter += 1\n        if GlobalStyleImageCounter == STYLE_IMAGE_COUNT:\n            GlobalStyleImageCounter = 0\n    return bat\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model\nThe model code is mostly unchanged. We removed the configuration for other image and patch sizes, since our images are always 256x256."},{"metadata":{"trusted":true},"cell_type":"code","source":"def leaky_relu(x, lk = 0.2):\n    return tf.maximum(x, x * lk)\n\ndef _fixed_padding(inputs, kernel_size, rate=1):\n    kernel_size_effective = [kernel_size[0] + (kernel_size[0] - 1) * (rate - 1),\n                            kernel_size[0] + (kernel_size[0] - 1) * (rate - 1)]\n    pad_total = [kernel_size_effective[0] - 1, kernel_size_effective[1] - 1]\n    pad_beg = [pad_total[0] // 2, pad_total[1] // 2]\n    pad_end = [pad_total[0] - pad_beg[0], pad_total[1] - pad_beg[1]]\n    padded_inputs = tf.pad(inputs, [[0, 0], [pad_beg[0], pad_end[0]],\n                        [pad_beg[1], pad_end[1]], [0, 0]], mode='SYMMETRIC')\n    return padded_inputs\n\n\nDL1C = 256\nDL2C = 512\n\ndiscriminator_cfg_p16 = {\n    'l_num': 2,\n    'l0_c': 3,                           # 256\n    'l1_c': DL1C,   'l1_k': 4, 'l1_s': 4, # 64\n    'l2_c': DL2C,   'l2_k': 4, 'l2_s': 4, # 16\n}\n\nbatch_norm_decay=0.95\nbatch_norm_epsilon=0.001\nbatch_norm_updates_collections=tf.GraphKeys.UPDATE_OPS\n\ndef build_discriminator(inp, patch_size=16, is_training=True,\n    name='discriminator', reuse=False):\n    d_state = inp\n    batch_norm_params = {\n        'center': True,\n        'scale': True,\n        'decay': batch_norm_decay,\n        'epsilon': batch_norm_epsilon,\n        'updates_collections': batch_norm_updates_collections,\n        'is_training': is_training\n    }\n    with slim.arg_scope([slim.batch_norm], **batch_norm_params):\n        with tf.variable_scope(name, reuse=reuse):\n            # conv s1\n            cfg = discriminator_cfg_p16\n            with slim.arg_scope([slim.conv2d],\n                activation_fn=leaky_relu,\n                normalizer_fn=slim.batch_norm,\n                padding='VALID'):\n                for l in range(1, cfg['l_num'] + 1):\n                    d_state = slim.conv2d(d_state,\n                        cfg['l%d_c'%l],\n                        [cfg['l%d_k'%l], cfg['l%d_k'%l]],\n                        stride=cfg['l%d_s'%l], scope='s1_%d'%l)\n            d_state = slim.conv2d(d_state, 1, [1, 1], stride=1,\n                activation_fn=tf.nn.sigmoid, scope='patch_mat')\n    return d_state\n\ng_encoder_cfg = {\n    'l_num': 3,\n    'l0_c': 32,   'l0_k': 3, 'l0_s': 2, # 1/2  --------> sc1\n    'l1_c': 64,   'l1_k': 3, 'l1_s': 2, # 1/4  --------> sc2\n    'l2_c': 128,  'l2_k': 3, 'l2_s': 2  # 1/8  --------> sc3 -L\n}\n\ng_residual_cfg = {\n    'l_num': 1,\n    'c': 128, 'k': 3\n}\n\ng_decoder_cfg = {\n    'l_num': 3,\n    'l0_c': 64,  'l0_k': 3, 'l0_s': 2, # --- x2\n    'l1_c': 32,  'l1_k': 3, 'l1_s': 2, # --- x4\n    'l2_c': 16,  'l2_k': 3, 'l2_s': 2, # --- x8\n}\n\ng_skip_conn_cfg = {\n    'l_num' : 2\n}\n\ndef build_generator(inp, name='generator', reuse=False):\n    g_state = inp   # No prep\n    with tf.variable_scope(name, reuse=reuse):\n        cfg = g_encoder_cfg\n        skip_conn = []\n        with slim.arg_scope([slim.conv2d, slim.separable_conv2d], activation_fn=tf.nn.relu,\n                    normalizer_fn=slim.instance_norm, padding='VALID'):\n            for index in range(cfg['l_num']):\n                g_state = _fixed_padding(g_state, [cfg['l%d_k'%index]])\n                g_state = slim.separable_conv2d(g_state, None, [cfg['l%d_k'%index], cfg['l%d_k'%index]],\n                    depth_multiplier=1, stride=cfg['l%d_s'%index], scope='enc_%d_dw'%index)\n                g_state = slim.conv2d(g_state, cfg['l%d_c'%index], [1, 1], stride=1, scope='enc_%d_pw'%index)\n                skip_conn.append(g_state)\n\n        cfg = g_residual_cfg\n        with slim.arg_scope([slim.conv2d, slim.separable_conv2d], activation_fn=tf.nn.relu,\n                    normalizer_fn=slim.instance_norm, padding='VALID'):\n            for index in range(cfg['l_num']):\n                res_g = g_state\n                g_state = _fixed_padding(g_state, [cfg['k']])\n                g_state = slim.separable_conv2d(g_state, None, [cfg['k'], cfg['k']],\n                    depth_multiplier=1, stride=1, scope='res_%d_dw'%index)\n                g_state = slim.conv2d(g_state, cfg['c'], [1, 1], stride=1,\n                    activation_fn=None, scope='res_%d_pw'%index)\n                g_state = tf.nn.relu(g_state + res_g)\n\n        cfg = g_decoder_cfg\n        with slim.arg_scope([slim.conv2d, slim.separable_conv2d], activation_fn=None,\n                    normalizer_fn=slim.instance_norm, padding='VALID'):\n            for index in range(cfg['l_num']):\n                g_state = tf.image.resize_images(g_state,\n                    (g_state.shape[1]*2, g_state.shape[2]*2), method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n                g_state = _fixed_padding(g_state, [cfg['l%d_k'%index]])\n                g_state = slim.separable_conv2d(g_state, None,\n                    [cfg['l%d_k'%index], cfg['l%d_k'%index]], depth_multiplier=1, stride=1, scope='dec_%d_dw'%index)\n                g_state = slim.conv2d(g_state, cfg['l%d_c'%index], [1, 1], stride=1, scope='dec_%d_pw'%index)\n                # sc = slim.conv2d(skip_conn[?], 128, [1, 1], stride=1, scope='sc')\n                if index < g_skip_conn_cfg['l_num']:\n                    g_state = tf.nn.relu(g_state + skip_conn[g_skip_conn_cfg['l_num'] - index - 1])\n                else:\n                    g_state = tf.nn.relu(g_state)\n        g_state = _fixed_padding(g_state, [3])\n        g_state = slim.conv2d(g_state, 3, [3, 3], stride=1, padding='VALID',\n            activation_fn=tf.nn.tanh, normalizer_fn=None, scope='output')\n    return g_state\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## VGG16\nFor calculating the content loss by performing image classification\nMostly unchanged. Except for a fix to an error while loading the weights file. We also removed some commented out code."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Required fix to properly load vgg16.npy weights\nnp_load_old = np.load # save np.load\nnp.load = lambda *a,**k: np_load_old(*a, allow_pickle=True, **k) # modify the default parameters of np.load\n\nVGG_MEAN = [103.939, 116.779, 123.68]\n\nclass Vgg16:\n    def __init__(self, vgg16_npy_path=None):\n        self.data_dict = np.load(\"vgg16.npy\", encoding='latin1').item()\n\n    def build(self, rgb):\n        \"\"\"\n        load variable from npy to build the VGG\n\n        :param rgb: rgb image [batch, height, width, 3] values scaled [0, 1]\n        \"\"\"\n\n        start_time = time.time()\n        print(\"build model started\")\n        rgb_scaled = rgb * 255.0\n\n        # Convert RGB to BGR\n        red, green, blue = tf.split(axis=3, num_or_size_splits=3, value=rgb_scaled)\n        \n        bgr = tf.concat(axis=3, values=[\n            blue - VGG_MEAN[0],\n            green - VGG_MEAN[1],\n            red - VGG_MEAN[2],\n        ])\n\n        self.conv1_1 = self.conv_layer(bgr, \"conv1_1\")\n        self.conv1_2 = self.conv_layer(self.conv1_1, \"conv1_2\")\n      \n        self.prob = self.conv1_2\n\n    def avg_pool(self, bottom, name):\n        return tf.nn.avg_pool(bottom, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name=name)\n\n    def max_pool(self, bottom, name):\n        return tf.nn.max_pool(bottom, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name=name)\n\n    def conv_layer(self, bottom, name):\n        with tf.variable_scope(name):\n            filt = self.get_conv_filter(name)\n\n            conv = tf.nn.conv2d(bottom, filt, [1, 1, 1, 1], padding='SAME')\n\n            conv_biases = self.get_bias(name)\n            bias = tf.nn.bias_add(conv, conv_biases)\n\n            relu = tf.nn.relu(bias)\n            return relu\n\n    def fc_layer(self, bottom, name):\n        with tf.variable_scope(name):\n            shape = bottom.get_shape().as_list()\n            dim = 1\n            for d in shape[1:]:\n                dim *= d\n            x = tf.reshape(bottom, [-1, dim])\n\n            weights = self.get_fc_weight(name)\n            biases = self.get_bias(name)\n\n            # Fully connected layer. Note that the '+' operation automatically\n            # broadcasts the biases.\n            fc = tf.nn.bias_add(tf.matmul(x, weights), biases)\n\n            return fc\n\n    def get_conv_filter(self, name):\n        return tf.constant(self.data_dict[name][0], name=\"filter\")\n\n    def get_bias(self, name):\n        return tf.constant(self.data_dict[name][1], name=\"biases\")\n\n    def get_fc_weight(self, name):\n        return tf.constant(self.data_dict[name][0], name=\"weights\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training\nTraining the GAN on the style and photo images.\nWe modified this part to keep the TF session variable to prevent having to save it and load it again in the render part.\nWe also changed it to use our own multi-style batch generation function instead of the single style image one."},{"metadata":{"trusted":true},"cell_type":"code","source":"input_ls = ls_files_to_json(TRAINSET_PATH, ext=['png', 'bmp', 'jpg', 'jpeg'])\nTRAIN_SET = len(input_ls)\n\nsess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))\n\ninput_s = tf.placeholder(tf.float32, shape=[BATCH_SIZE, PSI_D_SIZE, PSI_D_SIZE, 3], name='inps')\ninput_c = tf.placeholder(tf.float32, shape=[BATCH_SIZE, G_IMG_SIZE, G_IMG_SIZE, 3], name='inpc')\n\nvgg_c = Vgg16()\nwith tf.name_scope(\"content_vgg\"):\n    vgg_c.build(input_c)\n\ng_state = build_generator(input_c, name='generator')\n\nvgg_g = Vgg16()\nwith tf.name_scope(\"content_vgg\"):\n    vgg_g.build(g_state)\n\ndp_real = build_discriminator(input_s, patch_size=PATCH_SIZE, name='discriminator')\ndp_fake = build_discriminator(g_state, patch_size=PATCH_SIZE, name='discriminator', reuse=True)\n\nd_raw = vgg_c.prob # 128 * 128 * 64\nd_gen = vgg_g.prob # 128 * 128 * 64\n\nd_real_d = tf.reduce_mean(dp_real)\nd_fake_d = tf.reduce_mean(dp_fake)\n\nmean_d_fake = tf.reduce_mean(dp_fake)\nd_fake_g = tf.reduce_mean((dp_fake) ** (1.0 - (dp_fake - mean_d_fake)))\n\nd_loss = -(tf.log(d_real_d) + tf.log(1 - d_fake_d))\ng_loss = (tf.norm(d_raw - d_gen) ** 2)*LAMBDA /(BATCH_SIZE*((G_IMG_SIZE/VGG_L)*(G_IMG_SIZE/VGG_L))*VGG_FEATURES)-tf.log(d_fake_g)\n\nd_var_ls = tf.trainable_variables(scope='discriminator')\nupdate_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\nwith tf.control_dependencies(update_ops):\n    train_step_d = tf.train.RMSPropOptimizer(5e-4).minimize(d_loss, var_list=d_var_ls)\n\ng_var_ls = tf.trainable_variables(scope='generator')\ntrain_step_g = tf.train.RMSPropOptimizer(5e-4).minimize(g_loss, var_list=g_var_ls)\n\n\nsess.run(tf.global_variables_initializer())\nvar_ls = g_var_ls.append(d_var_ls)\nepoch = 0\n\nwhile epoch < TRAIN_EPOCHS:\n    random.shuffle(input_ls)\n    travx = int(TRAIN_SET / BATCH_SIZE) + (1 if (TRAIN_SET % BATCH_SIZE) != 0 else 0)\n    for offset in range(travx):\n        sub_ls  = pickup_list(input_ls, BATCH_SIZE, offset * BATCH_SIZE) # sub_ls  = pickup_list_random(input_ls, BATCH_SIZE)\n        sub_img = images_batch(TRAINSET_PATH, sub_ls, prep=True,\n                            shape=(G_IMG_SIZE, G_IMG_SIZE), singleCh=False, remove_pad=True)    \n        for td in range(DISCRIMINATOR_MAX_ITER):\n            sess.run(train_step_d, feed_dict={\n                    input_s: make_input_batch_path(STYLE_PATH, BATCH_SIZE, PSI_D_SIZE, PSI_D_SIZE, PATCH_SIZE),\n                    input_c: sub_img\n                })\n        for tg in range(GENERATOR_MAX_ITER):\n            sess.run(train_step_g, feed_dict={\n                    input_c: sub_img\n                })\n        # optional progress print\n        #print('epoch %04d'%epoch, 'InnerProcess: %d/%d'%(offset, travx))\n\n\n    cur_d_real = sess.run(d_real_d, feed_dict={\n            input_s: make_input_batch_path(STYLE_PATH, BATCH_SIZE, PSI_D_SIZE, PSI_D_SIZE, PATCH_SIZE),\n            input_c: sub_img\n        })\n    cur_d_fake = sess.run(d_fake_d, feed_dict={\n            input_c: sub_img\n        })\n    print('\\33[1;32mEpoch %d D_TURN D real\\33[0m = '%epoch, cur_d_real.mean())\n    print('\\33[1;31mEpoch %d D_TURN D fake\\33[0m = '%epoch, cur_d_fake.mean())\n    epoch = epoch + 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Render\nUse trained model to generate output images.\nInstead of loading the saved model, we directly use the TF session of the previous code block.\nThis allows us to run the trained model without having to load and \"warm up\" the network, making the code a lot smaller and a bit more efficient.\nHowever, this caused some issues with the batch size, so we included some fixes for that as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"if (not os.path.isdir(OUT_PATH)):\n    os.makedirs(OUT_PATH)\n\ninput_ls = ls_files_to_json(IMGSRC_PATH, ext=['png', 'bmp', 'jpg', 'jpeg'])\nCNT = len(input_ls)\n\nfor i in range(CNT):\n    sub_ls  = pickup_list(input_ls, 1, i)\n    #print(\"Processing:\", sub_ls[0])\n    sub_img, sls = imbatch_read(IMGSRC_PATH, sub_ls, (PSI_D_SIZE, PSI_D_SIZE))\n    sub_img2  = np.tile(sub_img,(BATCH_SIZE, 1,1,1))\n    time_start = time.time()\n    render_batch = sess.run(g_state, feed_dict={input_c: sub_img2})\n    img_write(render_batch[0],\n            OUT_PATH+'/'+sub_ls[0],\n        sls[0])# (PSI_D_SIZE, PSI_D_SIZE)) #","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate output images.zip and clean\nFinally, we added these two lines to generate the images.zip file for submission and to clean all temp files from the project which we do not want to (or cannot, because of the large number of files) save."},{"metadata":{"trusted":true},"cell_type":"code","source":"shutil.make_archive('/kaggle/working/images/', 'zip', 'images')\n# remove all files except the output zip file\n!rm -rf images preview vgg16.npy SUN* monet*","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}