{"cells":[{"metadata":{"_cell_guid":"e61ef2d8-f315-4f7f-b07e-1de0f4e8441a","_uuid":"1677fddbb95f7545b6540e9201f3339a0fdbfc5d"},"cell_type":"markdown","source":"# Intro\nHi, \n\nI haven't seen many Tensorflow kernels that shows how to use Estimators and Datasets, so here is one let me know what you think. It should be a good start for anyone that would like to experiment with the new tensorflow API and hopefully a place to discuss the best practices.\n\nThe kernel is build based on the [Keras U-Net starter](https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277). Which I extended in the following ways:\n- I'm using image padding and cropping to avoid need for scaling, which improves the ootb. (LB 0.277).\n- I followed UNet paper more closely and added weights for borders and mask erosion to separate contiguous masks. (LB 0.341)\n- I'm representing the samples in a way that lets me add random transformations without the need to store seeds. The Image, Mask & Weights are put together as one image with 5 channels (Red, Green, Blue, Mask, Weight).\n- I'm splitting the training examples into training and validation that ensures that the Validation set is always separated from Training, no matter what random seed is being used. I'm using sha1 of file names to split the samples. \n\nThings that are left to do:\n- To fully implement the UNet paper I should change the convolution to 'valid' and add an image augmentation described in the paper,\n- The Dataset needs caching of samples  as it takes longer to load a batch than to train on GPU ([I have a question how to do it on slack, If i get the answer I will add it here. ](https://stackoverflow.com/questions/49082590/how-to-build-external-cache-for-tf-data-dataset))\n\nLet me walk you through the notebook if this sparked your interest or you think you could help me by suggesting some improvements to the way I write Tensorflow models."},{"metadata":{"_uuid":"a29b41ee8ec03439acbb8f8854f9f2a8d918c07e"},"cell_type":"markdown","source":"# Imports & Some paramters"},{"metadata":{"trusted":true,"_uuid":"c16ceb563f881014fb4a53cb32286ee3a33220b7"},"cell_type":"code","source":"# I have 3 gpus this way I ask Tensorflow to use only one\n%env CUDA_DEVICE_ORDER=PCI_BUS_ID\n%env CUDA_VISIBLE_DEVICES=0\n\nMODEL_NAME = 'unet-same'\nIMG_MIN_SIZE = 256\nIMG_CHANNELS = 3\n\nSTEPS_IN_TRAINING=3000\nSTEPS_IN_EVALUATION=None\n\n# Crippled version so that it can run as Kernel, comment that out and use the settings above.\nSTEPS_IN_TRAINING=1\nSTEPS_IN_EVALUATION=1\n\n# The use of Weights and Erosion is really slow as It uses py_func so it is disabled, but it gives better models LB 0.341 if set to True\nUSE_WEIGHTS_N_EROSION = False\n\n#I was trying to use the new pathlib module, not sure if that was good approach\nfrom pathlib import Path\nTRAIN_PATH = Path('../input/stage1_train/')\nTEST_PATH = Path('../input/stage1_test/')","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"c332549b-8d23-4bb5-8497-e7a8eb8b21d2","_uuid":"5c38504af3a84bee68c66d3cde74443c58df422f","collapsed":true,"trusted":false},"cell_type":"code","source":"%matplotlib inline  \nimport os\nimport sys\nimport random\nimport warnings\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom skimage import io, transform, morphology, filters\nfrom scipy import ndimage\nimport tensorflow as tf\n\nwarnings.filterwarnings('ignore', category=UserWarning, module='skimage')","execution_count":29,"outputs":[]},{"metadata":{"_cell_guid":"59c4a25d-645f-4b74-9c53-145ac78cc481","_uuid":"875af74f980236825de3a650825b46e25632422c"},"cell_type":"markdown","source":"# Split the labeled training samples\nInstead of relying on random seeds to keep our validation set separate from training, this code uses SHA1 to convert a file name to a random number between 1 and 100. The number is more or less uniformly distributed so we can use it to split the files in desired proportion, (10% in my case). I've found this super cool idea in (tensorflow example: speech_commands)[https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/input_data.py#L61]. I wonder why it is not being used more widely. (The imports are kept close to facilitate copy and paste)."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"5c34e9c97123cf4ec8e588dceb35dd00a2dd620a"},"cell_type":"code","source":"from collections import namedtuple\nimport hashlib \n\ndef filename_to_uniform_number(filepath, MAX_ITEMS_PER_CLASS=2 ** 27 - 1 ):\n    \"\"\"Associate the `filepath` to a number [0.0 - 1.0] based on file name.\n        \n    The numbers are generated based on sha1 of the file name, so should be uniformly distributed \n    and can be used to separate a validation set from training set. It let's you later add more training examples\n    and keep the training and validation set separate.\n\n    Parameters\n    ----------\n          filepath - pathlib.Path object with a path to file name\n          MAX_ITEMS_PER_CLASS - helper constant defining a maximum number elements in class\n    Returns\n    -------\n          a number random number from 0.0-1.0 to upper_bound, that depends only on a file name\n    \"\"\"\n    hash_name = filepath.name.split('_nohash_')[0]\n    hash_name_hashed = hashlib.sha1(hash_name.encode(\"utf-8\")).hexdigest()\n    return ((int(hash_name_hashed, 16) % (MAX_ITEMS_PER_CLASS + 1)) *\n           (1 / MAX_ITEMS_PER_CLASS))\n\ndef which_set(fn, validation_size=0.10):\n    if filename_to_uniform_number(fn) < validation_size:\n         return tf.estimator.ModeKeys.EVAL\n    return tf.estimator.ModeKeys.TRAIN\n# based on: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/input_data.py#L61","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"74f2de9f6523b00939609f62e0e82e8a7e97e37b"},"cell_type":"markdown","source":"Here is the most pythonic way I could find to write equivalent to this scala cde: `paths.groupBy(which_set(_, 0.10)` (I miss scala a bit). Any one knowing about better way please [post your answer to this stackoveflow question](https://stackoverflow.com/questions/49016362/how-to-create-a-dict-of-list-from-list-of-dicts-in-a-functional-way-in-python?noredirect=1#comment85042460_49016362)."},{"metadata":{"_cell_guid":"ca0cc34b-c26f-41ee-88d7-975aebdb634e","_uuid":"9e389ba8bdb5b6fc03b231b6a6c84a8bde634053","collapsed":true,"scrolled":false,"trusted":false},"cell_type":"code","source":"TE = namedtuple('TE', [tf.estimator.ModeKeys.TRAIN, tf.estimator.ModeKeys.EVAL])\npaths = TE([],[])\nfor x in TRAIN_PATH.glob(\"*\"):\n    getattr(paths, which_set(x, 0.10)).append(x)\n\nprint(\"Validation set precentage:\", len(paths.eval)/(len(paths.train)+len(paths.eval)))","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"d7a06fb540c080c03dab43d7597a71e05f94c781"},"cell_type":"markdown","source":"# Read and encode the training samples\n\nWe are going to represent a sample as one tensor and put mask and weight as two additional channels at the end of the image. It will let us make random transformations on such sample without a need to fix random seeds.\n\nThanks to this slices we can write `sample[IMG]` to get an image or `sample[MASK]` to get mask tensor."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"0996d5869d27ece3b93591eb30507bc3e728d70f"},"cell_type":"code","source":"IMG = (slice(None), slice(None), slice(0,3))\nMASK = (slice(None), slice(None), slice(3,4))\nWEIGHTS = (slice(None), slice(None), slice(4,5))","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"9cecbf6f854fd3c6d715c7054b1735385c787f12"},"cell_type":"markdown","source":"A bit of code to display a sample, quite useful to inspect the code later."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"b2d9f40f6c69f519e92e119a19e83833d2571817"},"cell_type":"code","source":"def show_sample(sample):   \n    if type (sample) == tf.Tensor:\n        print(\"Unable to display tensor that was not executed, run_in_sess(sample)\", sample)\n        return\n    images = [sample[IMG], sample[MASK], sample[WEIGHTS]]\n    show_image_list(images)\n\ndef show_image_list(images):\n    if len(images) == 1:\n        im = plt.imshow(np.squeeze(images[0]))\n    else:\n        fig, axs = plt.subplots(1, len(images), figsize=(20,20))\n        for img, ax in zip(images, axs):\n            im = ax.imshow(np.squeeze(img))\n        fig.colorbar(im, ax=axs.ravel().tolist(), shrink=0.2)\n    plt.show()","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"af602aea-5e56-42a8-9331-54b4b2650593","_uuid":"5fcee2b9aee2fba5c60d43ad48a14139e9c1318c"},"cell_type":"markdown","source":"### The code to calculate the background pixels weight & to introduce borders between masks\n\nThe code below is not being used unless `USE_WEIGHTS_N_EROSION` is set to True. It is too slow to be used as part of tensorflow `input_fn`. I've kept it to use when I find a way to add caching to the Datasets that is persistent (about that later)."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"76f47bda66ba9569ce52d3be384d2219d5bbff4d"},"cell_type":"code","source":"def calculate_unet_background_weight(merged_mask, masks, w0=10, q=5,):\n    weight = np.zeros(merged_mask.shape)\n    # calculate weight for important pixels\n    distances = np.array([ndimage.distance_transform_edt(m==0) for m in masks])\n    shortest_dist = np.sort(distances, axis=0)\n    # distance to the border of the nearest cell \n    d1 = shortest_dist[0]\n    # distance to the border of the second nearest cell\n    d2 = shortest_dist[1] if len(shortest_dist) > 1 else np.zeros(d1.shape)\n\n    weight = w0 * np.exp(-(d1+d2)**2/(2*q**2)).astype(np.float32)\n    weight = 1 + (merged_mask == 0) * weight\n    return weight\n\ndef binary_erosion_tf(mask):\n    def binary_erosion(img):\n        img = ndimage.morphology.binary_erosion((img > 0 ), border_value=1).astype(np.uint8)\n        return img\n    return tf.py_func(binary_erosion, [mask], tf.uint8)\n\ndef calculate_weights_tf(merged_mask, masks):\n    return tf.py_func(calculate_unet_background_weight, [merged_mask, masks], tf.float32)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"02261b3c85d715117dfb8e1348420efb839b51ef"},"cell_type":"markdown","source":"# Input pipline\n\n## Functions to load a single sample\n\nIt was quite a challenge to get this piece of code right. Tensorflow does not give you a lot of ways to read files or list directories.  Here is the only way I have found that let me do this without escaping to python. Reading files this way should be faster and allow us minimise the amount of data passed from python to Tensorflow on each execution of `input_fn`.  Moreover, it let us do shuffling on file names instead of on fully loaded files that should be a bit faster.\n\nAlthough the code has some issues. for example, I still don't know how to handle exceptions, like the lack of `/masks` directory in test folder set, or if to ignore files is not readable. If you have some ideas how to add the exception handling, please let me know in comments. \n\nBtw. I have a convention that if the function ends with `_tf` it means that it builds a tensorflow graph, and should be executed in a session. Do you have a better / more standardized way to make this distinction? My convention is a bit like the [Hungarin naming convention](https://en.wikipedia.org/wiki/Hungarian_notation), which is hard to read.\n"},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"59a7f770d6640ac9c3e5d4810cb12a5922f3f273"},"cell_type":"code","source":"def load_mask_tf(sample_path, use_weights_n_erosion=USE_WEIGHTS_N_EROSION):\n    mask_ds = (tf.data.Dataset.list_files(sample_path+\"/masks*/*.png\")\n                .map(lambda x: tf.image.decode_image(tf.read_file(x), channels=1), num_parallel_calls=4))\n    \n    if use_weights_n_erosion:\n        mask_ds = mask_ds.map(binary_erosion_tf)\n\n    masks = tf.contrib.data.get_single_element(mask_ds.batch(1024))\n    masks = tf.clip_by_value(tf.cast(masks, dtype=tf.float32), 0, 1) # convert to binary mask (it was 0, 255)\n    colors = tf.cast(tf.range(1, tf.shape(masks)[0]+1), dtype=tf.float32)\n    colors = tf.reshape(colors, shape=[-1,1,1,1])\n    merged_mask = tf.reduce_max(masks * colors, axis=0)\n    \n    if use_weights_n_erosion:\n        weights = calculate_weights_tf(merged_mask, masks)\n    else:\n        weights = tf.ones_like(merged_mask)\n        \n    return merged_mask, weights\n\ndef load_sample_tf(sample_path, load_mask=True, use_weights_n_erosion=USE_WEIGHTS_N_EROSION):  \n    image = tf.contrib.data.get_single_element(tf.data.Dataset.list_files(sample_path+\"/images/*.png\")\n            .map(lambda x: tf.image.decode_image(tf.read_file(x), channels=3)))\n    image = tf.cast(image, dtype=tf.float32)  \n    if load_mask:\n        merged_mask, weights = load_mask_tf(sample_path, use_weights_n_erosion=use_weights_n_erosion)\n        sample = tf.concat([image, merged_mask, weights], axis=2)\n    else:\n        img_shape = tf.shape(image)\n        mask_shape = [img_shape[0],img_shape[1],1]\n        sample = tf.concat([image, tf.zeros(mask_shape), tf.ones(mask_shape)], axis=2)\n\n    return sample\n\ndef run_in_sess(fn, *args, **kwargs):\n    with tf.Graph().as_default():\n        x = fn(*args, **kwargs)\n        init = (tf.global_variables_initializer(), tf.local_variables_initializer())\n        with tf.Session() as sess:\n            sess.run(init)\n            ret = sess.run(x)\n    return ret\n\ndef load_sample(sample_path, load_mask=True, use_weights_n_erosion=USE_WEIGHTS_N_EROSION):\n    return run_in_sess(load_sample_tf, str(sample_path), load_mask, use_weights_n_erosion)\n\nsample = load_sample(paths.train[350], use_weights_n_erosion=True)\nprint(\"A slow version of load_sample that calculate weights and erosion, normally the weigts are set to an array of ones\")\nprint(\"Sample size:\", sample.shape)\nshow_sample(sample)\nassert sample.dtype == 'float32'\nassert sample.shape[2] == 5, sample.shape[2]\nprint(\"Maximum weight set\", np.max(sample[WEIGHTS]))","execution_count":56,"outputs":[]},{"metadata":{"_uuid":"7c1d68b6ca3f5f43db43c123dd4ba54cd1f9bb1a"},"cell_type":"markdown","source":"## Transformations to normalize the size of the samples\n\nWe know the locations of the nucleus in the images so we can cut the images as much as we need to fit them into our network. There is no need to use image resize that may introduce unwanted distortions, besides since the original U-net is designed to work with fragments of a large image so this approach fits better in this case."},{"metadata":{"trusted":false,"_uuid":"5fd5cbe8c198559154232cb10036a88f510fe2e5"},"cell_type":"code","source":"def pad_to_min_size_tf(img, batch=False):\n    \"\"\"Pads image so that it's height and width is divisable by IMG_MIN_SIZE\n    \n    It handles both single images or batches of images.\n    \"\"\"\n    if batch:\n        print (\"Batch mode\")\n        img_shape=slice(1,3)\n        samples_dim = [[0,0]]\n    else:\n        img_shape=slice(0,2)\n        samples_dim = []\n        \n    shape = tf.shape(img)    \n    desired_shape = tf.cast(tf.ceil(tf.cast(shape[img_shape], dtype=tf.float32) / IMG_MIN_SIZE) * IMG_MIN_SIZE, dtype=tf.int32)\n    pad = (desired_shape - shape[img_shape]) \n    padding = samples_dim+[[0, pad[0]], [0, pad[1]], [0, 0]]\n    return tf.pad(img, padding, mode='SYMMETRIC', name='test'), shape\n\n#sample2, orig_shape = run_in_sess(pad_to_min_size_tf, np.expand_dims(sample,0), batch=False)\nsample2, orig_shape = run_in_sess(pad_to_min_size_tf, sample, batch=False)\nprint(\"Orignal shape:\", sample.shape, \"new padded shape:\", sample2.shape)\nshow_sample(sample2)","execution_count":60,"outputs":[]},{"metadata":{"_uuid":"fcb9d26764e3dad56b222716c6cd17bb747bc6be"},"cell_type":"markdown","source":"The code to randomly crop images will be more complicated if we represent sample as a `dict` or a `tuple` of images.\nImagine we would have to generate a random seed for each call and then pass it to 3 identical calls to `tf.random_crop`. "},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"e57707fa3b112e5c5d9fe11380647a87373d204d"},"cell_type":"code","source":"def random_crop_tf(sample):\n    return tf.random_crop(sample, size=[IMG_MIN_SIZE, IMG_MIN_SIZE, tf.shape(sample)[2]])\n\nsample2 = run_in_sess(random_crop_tf, sample)\nprint(\"Croping\", sample[IMG].shape, \"to\", sample2[IMG].shape)\nprint(\"Orignal\")\nshow_sample(sample)\nprint(\"Cropped version\")\nshow_sample(sample2)\n","execution_count":61,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"d60e5f3a639a22fdcc1a5036a1d9aea57213da58"},"cell_type":"code","source":"def cut_to_many_tf(sample, return_dataset=True):\n    \"\"\"Cut a padded sample to many images of size IMG_MIN_SIZExIMG_MIN_SIZE.\n    \n    Used for validation set where we don't want to use random_crop.\n    \"\"\"\n    even_sample, orig_size = pad_to_min_size_tf(sample)\n    shape = tf.shape(even_sample)\n    \n    ch = shape[2]\n    y = shape[1]\n    split_y = tf.reshape(even_sample, [-1, IMG_MIN_SIZE, y, ch])\n    split_num = tf.cast(tf.shape(even_sample)[0]//IMG_MIN_SIZE, dtype=tf.int64)\n\n    def split_in_x(i):\n\n        y0 = tf.cast(i*IMG_MIN_SIZE, dtype=tf.int32)\n        y1 = tf.cast((i+1)*IMG_MIN_SIZE, dtype=tf.int32)\n        \n        img = split_y[:, :, y0:y1, :]\n        return tf.data.Dataset.from_tensor_slices(img)\n    \n    ds = tf.data.Dataset.range(split_num).flat_map(split_in_x)\n    if return_dataset:\n        return ds\n    return tf.contrib.data.get_single_element(ds.batch(1024))\n\n\nnew_samples = run_in_sess(cut_to_many_tf, sample, False)\nprint(\"3 first samples extracted from the large sample\", sample.shape)\nprint(\"Total amount of samples extracted:\",len(new_samples))\nfor s in new_samples[:3]:\n    show_sample(s)","execution_count":69,"outputs":[]},{"metadata":{"_uuid":"181b6799daed5aa2e26e2a24fd95342e92230d69"},"cell_type":"markdown","source":"## The input_fn used for training with estimators\nHere is our definition of the input pipeline, this is where all the functions above are joined together using `Dataset.map` . It is the perfect place to introduce data augmentation later.\n\nA few remarks:\n- The `input_fn` will return a bound function with bound parameters so that it can be executed by estimator .\n\n- You may notice that the dataset has `cache` function, unfortunately, this does not solve our problem of slow loading sample_paths as the cache is cleared each time the estimator switches from training to evaluation on the valuation data set.\n"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"a08a52ad3cbd752869c3355163a9251bef518cdf"},"cell_type":"code","source":"def input_fn(sample_paths, batch_size=1, shuffle=False, num_epochs=1, take=None, load_labels=True):\n    \"\"\"Reads all samples form a sample_paths list, then tranforms them so that it can be used in training.\n    \n    Parameters\n    ----------\n        sample_paths - list of string or pathlib.Path objects pointing to directories with samples\n        batch_size   - size of a single batch\n        shuffle      - if True shuffles the order of samples _and_ use `random_crop_tf` to crop images to IMG_MIN_SIZExIMG_MIN_SIZE\n                       else it uses `cut_to_man_tf` which cut's large images to images matching our required size.\n        num_epochs   - if None it will iterate over the dataset forever, otherwise it returns that number of epochs. \n                       Make sure you don't set it to None for your evaluation set. \n        take         - returns 'take' amount of batches\n        load_labels  - can be set to False if you load the test data set using this function\n    \n    Returns\n    -------\n        input_fn to be used with `estimator.train` or `.evaluate`\n    \"\"\"\n    \n    # Estimator API except a different representation of samples, as tuple of dicts one for features one for labels.\n    # so we have this small conversion utility\n    def sample_to_features_n_labels(sample): \n        \"\"\"Conver our sample representation to match tensorflow API\n        \"\"\"\n        return {'image':sample[IMG]}, {'mask': sample[MASK], 'weights': sample[WEIGHTS]}\n    \n    sample_paths_tensor = tf.constant(list(map(str, sample_paths)), dtype=tf.string)\n    \n    def input_fn():\n        dataset = tf.data.Dataset.from_tensor_slices(sample_paths_tensor)\n\n        # As you see Dataset.shuffle except a buffer that can hold all samples to correctly shuffle them.\n        # Since we are starting from paths the buffer does not have to be that large.\n        if shuffle:\n            dataset = dataset.shuffle(buffer_size=len(sample_paths))\n \n        dataset = (dataset\n            .map(lambda x: load_sample_tf(x, load_mask=load_labels), num_parallel_calls=4)\n            .cache()) # this does not work that well if the evaluation is being done on the same GPU as training.\n\n        if shuffle:\n            dataset = dataset.map(random_crop_tf)\n        else:\n            dataset = dataset.flat_map(cut_to_many_tf)            \n\n        dataset = (dataset.map(sample_to_features_n_labels)\n            .repeat(num_epochs)\n            .batch(batch_size)\n            .prefetch(1)\n        )\n\n        if take is not None:\n            dataset = dataset.take(take)\n            \n        iterator = dataset.make_one_shot_iterator()\n        # `features` is a dictionary in which each value is a batch of values for\n        # that feature; `labels` is a batch of labels.\n        features, labels = iterator.get_next()\n        return features, labels\n    return input_fn\n\nimport time\nstart_time = time.time()\n\nwith tf.Graph().as_default():\n    it = input_fn(paths.eval, batch_size=1024)()\n    with tf.Session() as s:    \n        all_eval_samples = (s.run((it)))\n        print(\"First batch fetched, now the ds should stop\")\n        try:\n            nothing = (s.run((it)))\n        except tf.errors.OutOfRangeError:\n            print(\"The iterator correctly let us know that it is empty.\")\n    \nimages_loaded = all_eval_samples[0]['image'].shape[0]\ntotal_time = (time.time() - start_time)\nprint(\"Total execution time %s sec, performance %s samples / sec\" % (total_time, images_loaded/total_time))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3ac86ec4293dd5ca2017fcc1b76bfa5b442b02f"},"cell_type":"markdown","source":"Below is a small helper function that let's us display results returned from input_fn, in case we need it."},{"metadata":{"trusted":false,"_uuid":"ff507bfc364db6330fd46aa059bd1fe4b924b7a6"},"cell_type":"code","source":"def show_features_n_labels(features, labels, max_samples=1):\n    features = np.array([i for i in features.values()]) \n    labels = np.array([i for i in labels.values()])\n\n    if len(features.shape) not in [4,5]:\n        raise AttributeError(\"Wrong shape of images\", features.shape)\n\n    if len(features.shape) == 4:\n        features = np.expand_dims(features)\n        labels = np.expand_dims(labels)\n\n    ## TODO: make it more efficient at trimming the batch\n    samples_to_show = min(max_samples, features.shape[1])        \n    for sample_idx in range(0, samples_to_show):\n        sample = [f for f in features[:,sample_idx,:,:,:]] + [l for l in labels[:,sample_idx,:,:,:]]\n        show_image_list(sample)\n    \nshow_features_n_labels(*all_eval_samples, max_samples=2)","execution_count":73,"outputs":[]},{"metadata":{"_uuid":"988c3045940072c5621a7a92d183c8e39a0fcde7"},"cell_type":"markdown","source":"# The Model\n\nThis is a rewritten network from [Keras U-Net starter](https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277). It isn't exactly U-Net as it uses 'same' convolutions instead of 'valid' but it is good enough as a starter.\n\nThe code of the model (`get_model`) is a bit nicer to read in Tensorflow than in Keras, but that is subjective.\nWhat is really cool is the wrapping function `model_fn` that let us play with the training and prediction process, \nwhich is way more flexible than the approach taken by keras. Thanks to this:\n\n- we can convert the `MASK` to binary image (normally it has different number for each object so it displays nicer)\n- for predictions it let us pad images and then cut the mask back to the orignal shape (that is hard to do in keras afaik) "},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"b39e9c9bde399158406d78cec11baae86e54eca9"},"cell_type":"code","source":"from tensorflow.python.ops import array_ops\n\ndef conv2d_3x3(filters):\n    return tf.layers.Conv2D(filters, kernel_size=(3,3), activation=tf.nn.relu, padding='same')\n\ndef max_pool():\n    return tf.layers.MaxPooling2D((2,2), strides=2, padding='same') \n\ndef conv2d_transpose_2x2(filters):\n    return tf.layers.Conv2DTranspose(filters, kernel_size=(2, 2), strides=(2, 2), padding='same')\n\ndef concatenate(branches):\n    return array_ops.concat(branches, 3)\n\ndef get_model(features, mode, params):\n    x = features['image']\n    x = tf.placeholder_with_default(x, [None, None, None, IMG_CHANNELS], name='input_image_placeholder')\n    \n    s = x / 255 # convert image to 0 .. 1.0\n\n    c1 = conv2d_3x3(8) (s)\n    c1 = conv2d_3x3(8) (c1)\n    p1 = max_pool() (c1)\n\n    c2 = conv2d_3x3(16) (p1)\n    c2 = conv2d_3x3(16) (c2)\n    p2 = max_pool() (c2)\n\n    c3 = conv2d_3x3(32) (p2)\n    c3 = conv2d_3x3(32) (c3)\n    p3 = max_pool() (c3)\n\n    c4 = conv2d_3x3(64) (p3)\n    c4 = conv2d_3x3(64) (c4)\n    p4 = max_pool() (c4)\n\n    c5 = conv2d_3x3(128) (p4)\n    c5 = conv2d_3x3(128) (c5)\n\n    u6 = conv2d_transpose_2x2(64) (c5)\n    u6 = concatenate([u6, c4])\n    c6 = conv2d_3x3(64) (u6)\n    c6 = conv2d_3x3(64) (c6)\n\n    u7 = conv2d_transpose_2x2(32) (c6)\n    u7 = concatenate([u7, c3])\n    c7 = conv2d_3x3(32) (u7)\n    c7 = conv2d_3x3(32) (c7)\n\n    u8 = conv2d_transpose_2x2(16) (c7)\n    u8 = concatenate([u8, c2])\n    c8 = conv2d_3x3(16) (u8)\n    c8 = conv2d_3x3(16) (c8)\n\n    u9 = conv2d_transpose_2x2(8) (c8)\n    u9 = concatenate([u9, c1])\n    c9 = conv2d_3x3(8) (u9)\n    c9 = conv2d_3x3(8) (c9)\n\n    logits = tf.layers.Conv2D(1, (1, 1)) (c9)\n    return logits","execution_count":76,"outputs":[]},{"metadata":{"_cell_guid":"c1dbc57c-b497-4ccb-b077-2053203ab7ed","_uuid":"0aa97d66c29f45dfac9b0f45fcf74ba0e778ba5d","collapsed":true,"trusted":false},"cell_type":"code","source":"def model_fn(features, labels, mode, params={}):\n    if mode == tf.estimator.ModeKeys.PREDICT:\n        f, sh = pad_to_min_size_tf(features['image'], batch=True)\n        logits = get_model({'image': f}, mode, params)\n\n        mask = tf.nn.sigmoid(logits, name='sigmoid_tensor')        \n        predictions = {\n            'mask': mask[:,0:sh[1],0:sh[2],:], # we remove pixels added in pad_to_min_size_tf\n            'image': features['image']         # we return the image as well to simplify testing\n        }\n    \n        export_outputs={'generate' : tf.estimator.export.PredictOutput(predictions)}\n        return tf.estimator.EstimatorSpec(mode=mode, \n                                          predictions=predictions,\n                                          export_outputs=export_outputs)\n    logits = get_model(features, mode, params)\n    predictions = {\n        'mask': tf.nn.sigmoid(logits, name='sigmoid_tensor'),\n    }\n\n    true_mask = tf.reshape(tf.clip_by_value(labels['mask'], 0.0 ,1.0), [-1, 256,256,1])\n    weights = labels['weights']\n    loss = tf.losses.sigmoid_cross_entropy(multi_class_labels=true_mask, \n                                           logits=logits,\n                                           weights=weights)\n\n    # Configure the training op\n    if mode == tf.estimator.ModeKeys.TRAIN:\n        optimizer =  tf.train.AdamOptimizer(learning_rate=1e-4)\n        update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n        with tf.control_dependencies(update_ops):\n            train_op = optimizer.minimize(loss, tf.train.get_or_create_global_step())\n\n    else:\n        train_op = None\n\n\n    with tf.variable_scope('mean_iou_calc'):\n        prec = []\n        up_opts = []\n        for t in np.arange(0.5, 1.0, 0.05):\n            predicted_mask = tf.to_int32(predictions['mask'] > t)\n            score, up_opt = tf.metrics.mean_iou(true_mask, predicted_mask, 2)\n            up_opts.append(up_opt)\n            prec.append(score)\n        mean_iou = tf.reduce_mean(tf.stack(prec), axis=0), tf.stack(up_opts)\n\n    eval_metrics = {'mean_iou': mean_iou}\n\n    return tf.estimator.EstimatorSpec(\n        mode=mode,\n        predictions=predictions,\n        loss=loss,\n        train_op=train_op,\n        eval_metric_ops=eval_metrics)\n\nrun_config = tf.estimator.RunConfig(keep_checkpoint_max=25)\nestimator = tf.estimator.Estimator(\n    model_fn=model_fn,\n    model_dir=\"./data/logs/\"+MODEL_NAME,\n    config=run_config)\n\n# In case you want to profile this model here is a profile hook. To use it add hooks=[profiler_hook] to TrainSpec.\ntf.gfile.MakeDirs('timelines/'+MODEL_NAME)\nprofiler_hook = tf.train.ProfilerHook(\n    save_steps=1000,\n    save_secs=None,\n    output_dir='timelines/'+MODEL_NAME,\n    show_dataflow=True,\n    show_memory=True)\n","execution_count":78,"outputs":[]},{"metadata":{"_cell_guid":"72330944-6ce7-4070-b276-c3c4b20c4fe5","_uuid":"92350b6e18cc50f3fa7b6e9a02d39fcbff8238f7"},"cell_type":"markdown","source":"# Training\n\nHere is where we train our model. I'm not sure how to add early stopping here so that I've set `max_steps` to something that gives me a good enough result.\nThe default API does quite a bit:\n- it logs loss and mean_iou in `estimator.model_dir` so that it can be viewed by tensorflow, really useful feature\n- it saves a check point from the training so that you can stop it and resume it later\n- it also reads the last check point, it is very handy if your notebook crashes and you have to restart your training, life saver if you use GPU in a cloud."},{"metadata":{"collapsed":true,"scrolled":false,"trusted":false,"_uuid":"94f124fb5fa09386af9aa2f5308b457531f70833"},"cell_type":"code","source":"input_fns = TE(input_fn(paths.train, \n                           batch_size=128, \n                           shuffle=False, \n                           num_epochs=None),\n               input_fn(paths.eval, batch_size=10, shuffle=False),)\nspecs = TE(tf.estimator.TrainSpec(input_fn=input_fns.train, max_steps=STEPS_IN_TRAINING),\n          tf.estimator.EvalSpec(input_fn=input_fns.eval, steps=STEPS_IN_EVALUATION, throttle_secs=600))\n\ntf.estimator.train_and_evaluate(estimator, specs.train, specs.eval)","execution_count":79,"outputs":[]},{"metadata":{"_cell_guid":"1f381f5b-1b71-4daa-a417-e02f4894540b","_uuid":"bb15226ea617cf91ed8f43179fccb5a15809e5a0"},"cell_type":"markdown","source":"# Predictions\n\nThis part is very similar to the one provided in keras kernel, we look at the predictions then convert them to RLE.\n\nWhat is worth noting is that we put whole images from Test set without scaling to the model one by one.\nThis is possible because UNet is fully convolutional network without fully connected layers so it does not care about the size of the image as long as it is divisible by the `IMG_MIN_SIZE`. Otherwise it will break on `concanation` layers.\n"},{"metadata":{"_uuid":"7272b6c3ce8b9d660dcd478fbfe379cb6ac3d3f4"},"cell_type":"markdown","source":"## How the predicted masks look for Training and Validation sets"},{"metadata":{"trusted":false,"_uuid":"6861664685a1f96b9c71fb5241b2b9bf0cc84893"},"cell_type":"code","source":"train_pred = next(iter(estimator.predict(input_fns.train)))\nshow_image_list([train_pred['image'], train_pred['mask']])","execution_count":81,"outputs":[]},{"metadata":{"_cell_guid":"2daa48d5-ac98-4e18-af3f-a582baaa44f0","_uuid":"f841760b4abca1a25cb750822f88268bd79bf2ce","trusted":false},"cell_type":"code","source":"eval_pred = next(iter(estimator.predict(input_fns.eval)))\nshow_image_list([eval_pred['image'], eval_pred['mask']])","execution_count":82,"outputs":[]},{"metadata":{"_uuid":"51cfd2a30495e1b6779d9fed27b1282d96bbcdf1"},"cell_type":"markdown","source":"## Prediction for the test samples\n### Prepare the input function\nLet's first load the samples and see how they look"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"5fe03605b5b0dd9d9446d178edab976982fbab87"},"cell_type":"code","source":"test_paths = list(sorted(TEST_PATH.glob(\"*\")))","execution_count":85,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b7316997b8f945f7fd85a8e524b29fad4a343f07"},"cell_type":"code","source":"sample = load_sample(test_paths[10], load_mask=False)\n\nprint(sample.shape)\nshow_image_list([sample[IMG]])","execution_count":88,"outputs":[]},{"metadata":{"_uuid":"48a4fb51035afcd85e4267b8194cfe9d0a51bb32"},"cell_type":"markdown","source":"We have added `pad_to_min_size_tf` to our model_fn during prediction phase so that the sample above will be presented as padded to the network, and then the prediction will be cut to the correct size."},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"c48024c5c310668798ffa8ebe6afffae407cabf8"},"cell_type":"code","source":"padded, sh = run_in_sess(pad_to_min_size_tf, sample[IMG])\nprint(sh, \"->\", padded.shape)\nshow_image_list([padded])","execution_count":89,"outputs":[]},{"metadata":{"_uuid":"676049269fa7ebb3914da2ae51f0d61a4e0774d3"},"cell_type":"markdown","source":"We want to keep the images in original size for the model to analyze; this cannot be done in batches, so we have a custom version of `input_fn`."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"292512654d515fc6ba54a61547cb1df07f60d465"},"cell_type":"code","source":"def pred_input_fn(paths):\n    paths=list(map(str, paths))\n    def input_fn():\n        ds = (tf.data.Dataset.from_tensor_slices(paths)\n                .map(lambda x: load_sample_tf(x, load_mask=False))\n                .map(lambda x: ({\"image\": x[IMG]}, {\"mask\": tf.zeros_like(x[IMG])})).batch(1))          \n        iterator = ds.make_one_shot_iterator()\n        return iterator.get_next()\n    return input_fn","execution_count":92,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5020390b868d74e5b841e6531923da58d43fca69"},"cell_type":"code","source":"thr=0.5\npred = next(iter(estimator.predict(pred_input_fn(test_paths))))\nshow_image_list([pred['image'], pred['mask'], (pred['mask'] > thr)])","execution_count":96,"outputs":[]},{"metadata":{"_uuid":"47e6575d51adc55b3692cd41659b983ba6d06820"},"cell_type":"markdown","source":"### Run the predictions"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"8318685faf904a1d7bd545ad2e41557f7729f23c"},"cell_type":"code","source":"preds_test = []\npreds_test_t = []\nfor path, pred in zip(test_paths, estimator.predict(pred_input_fn(test_paths))):\n    print(path)\n    mask = np.squeeze(pred['mask'])\n    preds_test_t.append((mask > thr).astype(np.uint8))\n    preds_test.append(mask)","execution_count":97,"outputs":[]},{"metadata":{"_cell_guid":"a6690535-b2e4-49ac-98d9-7191bfabfb6f","_uuid":"6a34c98de7c6ae473f676a34fe7e099b46764eca"},"cell_type":"markdown","source":"# Encode and submit our results\n\nThis part is almost 1-1 copy of the [Keras U-Net starter](https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277). I've extended it to add some post processing like adding `dilation` in case we use `erosion` on masks, and added few test but it stays the same."},{"metadata":{"_uuid":"8f3d47a0bafb015b421df0656921a90bbf308953"},"cell_type":"markdown","source":"## RLE Encoding"},{"metadata":{"_cell_guid":"59a0af60-a7d7-41ef-a6fe-9e3c72defa07","_uuid":"4f99c1bf852e82b60bd4f982ca0df293f712cdf0","collapsed":true,"trusted":false},"cell_type":"code","source":"# Run-length encoding stolen from https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\ndef rle_encoding(x):\n    dots = np.where(x.T.flatten() == 1)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b>prev+1): run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return run_lengths\n","execution_count":100,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"cbd7d265b9d46fa19563df2a669adef70488f170"},"cell_type":"code","source":"# ref.: https://www.kaggle.com/paulorzp/run-length-encode-and-decode\n# ref.: https://www.kaggle.com/stainsby/fast-tested-rle\n\ndef rle_decode(mask_rle, shape):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    if type(mask_rle) == str:\n        s = mask_rle.split()\n    else:\n        s = mask_rle\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T","execution_count":101,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"911ca1f61476cdab56e0376a1e8694075de4dc46"},"cell_type":"code","source":"def prob_to_rles(x, cutoff=0.5, debug=False, dilation=USE_WEIGHTS_N_EROSION):\n    lab_img = morphology.label(x > cutoff) # split of components goes here\n    if debug:\n        plt.imshow(lab_img)\n        plt.show() \n        lab_img2=lab_img\n    if dilation:\n        for i in range(1, lab_img.max() + 1):    \n            lab_img = np.maximum(lab_img, ndimage.morphology.binary_dilation(lab_img==i)*i)\n        if debug:\n            plt.imshow(lab_img)\n            plt.show()    \n    for i in range(1, lab_img.max() + 1):\n        img = lab_img == i\n        yield rle_encoding(img)","execution_count":105,"outputs":[]},{"metadata":{"_cell_guid":"22fe24a1-7659-4cc9-9d23-211f38e5b99f","_uuid":"089587843ed6a3955fdcb9b23a6ec3bf5d703688","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"new_test_ids = []\nrles = []\nfor path, pred_mask in zip(test_paths, preds_test):\n    id_ = path.name\n    rle = list(prob_to_rles(pred_mask))\n    rles.extend(rle)\n    new_test_ids.extend([id_] * len(rle))","execution_count":109,"outputs":[]},{"metadata":{"_uuid":"0b09b229871b32d6681f37c887b023b8ee37fa5d"},"cell_type":"markdown","source":"Let see if it is working for a particular example"},{"metadata":{"trusted":false,"_uuid":"974d8fc2029bb7bb1d9b64a7eb5405bfdf438372"},"cell_type":"code","source":"def check_encoding(id_):\n    ix = test_paths.index(TEST_PATH / id_)\n    thresh_val = 0.5 #threshold_otsu(preds_test[ix])\n    sample = load_sample(test_paths[ix], load_mask=False)\n    print(\"Original image & predicted path\")\n    show_image_list([sample[IMG], preds_test[ix],  preds_test[ix]> thresh_val ])\n    shape = preds_test[ix].shape\n    pos = new_test_ids.index(id_)\n    imgs = []\n    while new_test_ids[pos] == id_:\n        imgs.append(rle_decode(rles[pos], shape))\n        pos += 1\n    print(\"List of extracted masks that are going to be submitted to kaggle\")\n    show_image_list(imgs)\n    plt.show()\n\ncheck_encoding('e17b7aedd251a016c01ef9158e6e4aa940d9f1b35942d86028dc1222192a9258')","execution_count":112,"outputs":[]},{"metadata":{"_cell_guid":"20b6b627-0fd6-425d-888f-da7f39efb124","_uuid":"849184a40a2c9c21506d8b8eb10ad9155fa229e8"},"cell_type":"markdown","source":"and finally, create a submission file (exact copy from Keras kernel)"},{"metadata":{"_cell_guid":"1ba0ee3a-cca0-4349-83f6-09a1ac6fcb44","_uuid":"ba589f56f5be1e6886bc88f5bf9e7d0a408e4048","collapsed":true,"trusted":false},"cell_type":"code","source":"# Create submission DataFrame\nsub = pd.DataFrame()\nsub['ImageId'] = new_test_ids\nsub['EncodedPixels'] = pd.Series(rles).apply(lambda x: ' '.join(str(y) for y in x))\n\nfname = MODEL_NAME+'-tf.csv'\nprint(\"Submission file: \"+fname)\nsub.to_csv(fname, index=False)","execution_count":114,"outputs":[]},{"metadata":{"_cell_guid":"222475b9-3171-461a-90f0-a820a6bd2634","_uuid":"fb5e6f8cca872f1bd7036f6d9ac2ed2cab615536"},"cell_type":"markdown","source":"This model gave me LB 0.276. Setting `USE_WEIGHTS_N_EROSION=True` should give you LB 0.349, at least it is what I've got for a model written in keras where the caching of the training data weren't an issue."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"8ccdaf6ecb363c2e7418099ffa4cf35ad6216ea0"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}