{"cells":[{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:24.481119Z","start_time":"2018-02-23T18:03:24.475116Z"},"collapsed":true,"trusted":false,"_uuid":"698536e674127642cbd73a2e112e57a9449a3316"},"cell_type":"code","source":"# Deformable convolutions https://arxiv.org/pdf/1703.06211.pdf & https://github.com/felixlaumon/deform-conv\n# Please, download the https://github.com/felixlaumon/deform-conv project and add the /deform-conv folder to you project files.","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"dbb7f019f62e2bd0b4ff18f5301a1a155935fab2"},"cell_type":"markdown","source":"# The Deformable Convolutions\nThe Deformable convolutions described [in this paper](https://arxiv.org/pdf/1703.06211.pdf).\nI didn't find the examples of usage the deformable convolutions in the autoencoders. Hope, this example help with it. The standard convolutions use the box, unchangeable shape of the filters. The deformable convolutions learn the filter shapes and adjust shapes to the most frequent cases. I thought that would be helpful in this particular case when the nucleus have very similar shapes. I didn't run a whole bunch of the experiments to make sure the deformable convolutions work better in the most cases. In my tests, the deformable convolutions work at last on a par or a little bit better comparing with standard convolutions."},{"metadata":{"_cell_guid":"01e3b8c5-7e10-450b-a663-08c829eb793e","_uuid":"9ae1e73615b56d28a06645128661dfa9e817f9c7"},"cell_type":"markdown","source":"This code is inspirated by\n\nhttps://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277 and \n\nhttps://www.kaggle.com/takuok/keras-generator-starter-lb-0-326?scriptVersionId=2209692\n"},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:24.514142Z","start_time":"2018-02-23T18:03:24.486124Z"},"collapsed":true,"trusted":false,"_uuid":"6754ddb67b9d9358eadfdbd3210033feaaee89f7"},"cell_type":"code","source":"import time\n\ndef log(text):\n    time_int = time.time() - start_time\n    print('{}:{:02.0f} {}.'.format(int(time_int/60), time_int%60, text))","execution_count":2,"outputs":[]},{"metadata":{"heading_collapsed":true,"_uuid":"4f971a2c7fbb8ce4df3547453e02a7e8a937bf77"},"cell_type":"markdown","source":"# Data Preparation"},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:26.059464Z","start_time":"2018-02-23T18:03:24.517145Z"},"_cell_guid":"a3a5212a-bc80-485a-9109-e5be9b05f546","_uuid":"bcb8165daaa0a560b300be8bcf4e4f9b31bfa472","collapsed":true,"hidden":true,"trusted":false},"cell_type":"code","source":"import cv2\nimport os\nimport numpy as np\nimport pandas as pd\n\n\ndef make_df(train_path, test_path, img_size):\n    train_ids = next(os.walk(train_path))[1]\n    test_ids = next(os.walk(test_path))[1]\n    X_train = np.zeros((len(train_ids), img_size, img_size, 3), dtype=np.uint8)\n    Y_train = np.zeros((len(train_ids), img_size, img_size, 1), dtype=np.bool)\n    for i, id_ in enumerate(train_ids):\n        path = train_path + id_\n        img = cv2.imread(path + '/images/' + id_ + '.png')\n        img = cv2.resize(img, (img_size, img_size))\n        X_train[i] = img\n        mask = np.zeros((img_size, img_size, 1), dtype=np.bool)\n        for mask_file in next(os.walk(path + '/masks/'))[2]:\n            mask_ = cv2.imread(path + '/masks/' + mask_file, 0)\n            mask_ = cv2.resize(mask_, (img_size, img_size))\n            mask_ = mask_[:, :, np.newaxis]\n            mask = np.maximum(mask, mask_)\n        Y_train[i] = mask\n    X_test = np.zeros((len(test_ids), img_size, img_size, 3), dtype=np.uint8)\n    sizes_test = []\n    for i, id_ in enumerate(test_ids):\n        path = test_path + id_\n        img = cv2.imread(path + '/images/' + id_ + '.png')\n        sizes_test.append([img.shape[0], img.shape[1]])\n        img = cv2.resize(img, (img_size, img_size))\n        X_test[i] = img\n\n    return X_train, Y_train, X_test, sizes_test","execution_count":3,"outputs":[]},{"metadata":{"heading_collapsed":true,"_uuid":"38c036d7ef115a46da08d2509558b6ea23e21559"},"cell_type":"markdown","source":"# Model Definition"},{"metadata":{"_cell_guid":"27f0fe04-cd5b-4042-a2e0-eb9e301933f0","_uuid":"bf6df4dfbbaddbd3d40293f3b05bf68ee51d4475","hidden":true},"cell_type":"markdown","source":"We use the autoencoder **Unet** model. We add the **ConvOffset2D** layer which performs the deformable convolution."},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:38.965656Z","start_time":"2018-02-23T18:03:26.062464Z"},"_cell_guid":"1960ba32-cdec-4aec-b39b-86e832940976","_uuid":"8318ccbe407bf32eeba9723a597556cc634465ff","hidden":true,"trusted":false,"collapsed":true},"cell_type":"code","source":"from keras.layers.convolutional import Conv2D, Conv2DTranspose\nfrom keras.layers.pooling import MaxPooling2D\nfrom keras.layers.merge import concatenate\nfrom keras.models import Model\nfrom keras.layers import Input, BatchNormalization\nfrom keras.layers.core import Dropout, Lambda, Dense\nfrom deform_conv.layers import ConvOffset2D\n\n\ndef conv_block(input, num_filters, filter_size=(3,3), dropout=0.3, kernel_initializer='he_normal', deform=True):\n    #c1 = BatchNormalization()(input)\n    c1 = Conv2D(num_filters, filter_size, activation='elu', padding='same', kernel_initializer=kernel_initializer) (input)\n    c1 = BatchNormalization()(c1)\n    #c1 = Dropout(dropout)(c1)\n    c1 = ConvOffset2D(num_filters)(c1)\n    c2 = Conv2D(num_filters, filter_size, strides=(2,2), activation='elu', padding='same', \n                kernel_initializer=kernel_initializer) (c1)  if deform else c1\n    return c2\n\ndef conv_pool_block(input, num_filters, filter_size=(3,3), pool_size=(2,2), dropout=0.3):\n    conv = conv_block(input, num_filters, filter_size=filter_size)\n#     pool = MaxPooling2D(pool_size) (conv)\n#     return conv, pool\n    return conv, conv\n\ndef transp_conv_block(input, input_res, num_filters, filter_size=(3,3), transp_size=(2,2),\n                      dropout=0.3, kernel_initializer='he_normal', deform=False):\n    transp = Conv2DTranspose(num_filters, transp_size, strides=transp_size, padding='same',\n                             kernel_initializer=kernel_initializer) (input)\n    cnc = concatenate([transp, input_res])\n    cb = conv_block(cnc, num_filters, filter_size=filter_size, deform=deform)\n    return cb\n\ndef Unet(img_size):\n    inputs = Input((img_size, img_size, 3))\n    s = Lambda(lambda x: x / 255)(inputs)\n\n    c1, cp1 = conv_pool_block(s, 8, dropout=0.1)\n    c2, cp2 = conv_pool_block(cp1, 16, dropout=0.1)\n    c3, cp3 = conv_pool_block(cp2, 32, dropout=0.1)\n    c4, cp4 = conv_pool_block(cp3, 64, dropout=0.2)\n    c5, cp5 = conv_pool_block(cp4, 128, dropout=0.2)\n    c6, cp6 = conv_pool_block(cp5, 256, dropout=0.3)\n\n    c_middle = conv_block(cp6, 512, dropout=0.3)\n\n    tc6 = transp_conv_block(c_middle, c6, 256, dropout=0.3)\n    tc5 = transp_conv_block(tc6, c5, 128, dropout=0.2)\n    tc4 = transp_conv_block(tc5, c4, 64, dropout=0.2)\n    tc3 = transp_conv_block(tc4, c3, 32, dropout=0.1)\n    tc2 = transp_conv_block(tc3, c2, 16, dropout=0.1)\n    tc1 = transp_conv_block(tc2, c1, 8, dropout=0.1)\n\n    #outputs = Conv2D(1, (1, 1), activation='sigmoid') (tc1)\n    tc1 = Conv2DTranspose(4, (2,2), strides=(2,2), padding='same',\n                             kernel_initializer='he_normal') (tc1)\n    outputs = Dense(1, activation='sigmoid') (tc1)\n\n    model = Model(inputs=[inputs], outputs=[outputs])\n    #model.summary()\n    return model","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"475c0fc2-ca32-4db9-9008-05f675696a0e","_uuid":"bebc9fc8e555aaba633741c0452fcfa0925664b4","heading_collapsed":true},"cell_type":"markdown","source":"# Data Generator\nWe use the keras ImageDataGenerator.\nYou can change the method of data augmentation by changing data_gen_args."},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:39.109753Z","start_time":"2018-02-23T18:03:38.969660Z"},"_cell_guid":"b2a782ca-1656-4c4b-b320-4212989c530d","_uuid":"487865a51b4d4c28bb69b88d4e5e20bc6b7bba80","collapsed":true,"hidden":true,"trusted":false},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n\ndef generator(xtr, xval, ytr, yval, batch_size):\n    data_gen_args = dict(horizontal_flip=True,\n                         vertical_flip=True,\n                         rotation_range=90.,\n                         width_shift_range=0.1,\n                         height_shift_range=0.1,\n                         zoom_range=0.1)\n    val_gen_args = dict()\n\n    def get_gen(data, args):\n        gen = ImageDataGenerator(**args)\n        gen.fit(data, seed=7)\n        gen = gen.flow(data, batch_size=batch_size, seed=7)\n        return gen\n    \n    train_generator = zip(get_gen(xtr, data_gen_args), get_gen(ytr, data_gen_args))\n    val_generator = zip(get_gen(xval, val_gen_args), get_gen(yval, val_gen_args))\n    return train_generator, val_generator","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"c9d9d672-adfc-4e79-a3fa-26d206e5e6d2","_uuid":"fa55995ed9b64862fd28a6454172ad7ed74bae49","heading_collapsed":true},"cell_type":"markdown","source":"# Define metrics and loss function."},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:39.174798Z","start_time":"2018-02-23T18:03:39.112756Z"},"_cell_guid":"23672e7c-a61a-40cb-9221-5d52a3893cb1","_uuid":"bfaf03f19a053db96548e4973a26060072c3ccce","collapsed":true,"hidden":true,"trusted":false},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nfrom keras import backend as K\nfrom keras.losses import binary_crossentropy\n\n\ndef mean_iou(y_true, y_pred):\n    prec = []\n    for t in np.arange(0.5, 1.0, 0.05):\n        y_pred_ = tf.to_int32(y_pred > t)\n        score, up_opt = tf.metrics.mean_iou(y_true, y_pred_, 2)\n        K.get_session().run(tf.local_variables_initializer())\n        with tf.control_dependencies([up_opt]):\n            score = tf.identity(score)\n        prec.append(score)\n    return K.mean(K.stack(prec))\n\ndef dice_coef(y_true, y_pred):\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n\ndef bce_dice_loss(y_true, y_pred):\n    return 0.5 * binary_crossentropy(y_true, y_pred) - dice_coef(y_true, y_pred)","execution_count":6,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T18:03:41.466997Z","start_time":"2018-02-23T18:03:39.182803Z"},"_cell_guid":"af47f36d-64d0-4f71-a012-8761055e0ea2","_uuid":"fa38f24d8a66d3017332927be0f350a00b389558","collapsed":true,"hidden":true,"trusted":false},"cell_type":"code","source":"from skimage.morphology import label\n\ndef rle_encoding(x):\n    dots = np.where(x.T.flatten() == 1)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b>prev+1): run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return run_lengths\n\ndef prob_to_rles(x, cutoff=0.4): # 0.5\n    lab_img = label(x > cutoff)\n    for i in range(1, lab_img.max() + 1):\n        yield rle_encoding(lab_img == i)","execution_count":7,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T05:48:43.045147Z","start_time":"2018-02-23T05:48:43.040146Z"},"_uuid":"ad2296a6ef09d2724783ed65d2391abeb2987888"},"cell_type":"markdown","source":"# Execute"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"1eeff62d0acafe1f5720c39ca5c4aa307e416a0a"},"cell_type":"code","source":"def add_empty_ids(sub, test_ids, new_test_ids): \n    empty_ids = list(set(test_ids) - set(new_test_ids))\n    print('debug: images without masks:', len(empty_ids))\n    if len(empty_ids) == 0:\n        return sub\n    add_df = pd.DataFrame(empty_ids,columns=['ImageId'])\n    add_df['EncodedPixels'] = ''\n    return pd.concat([sub,add_df])\n\n\ndef save_submission(file_name, preds, img_path, img_sizes):\n    img_ids = next(os.walk(img_path))[1]\n    \n    preds_upsampled = []\n    for i in range(len(preds)):\n        preds_upsampled.append(cv2.resize(preds[i], (img_sizes[i][1], img_sizes[i][0])))\n        \n    new_img_ids = []\n    rles = []\n    for n, id_ in enumerate(img_ids):\n        rle = list(prob_to_rles(preds_upsampled[n]))\n        rles.extend(rle)\n        new_img_ids.extend([id_] * len(rle))\n    sub = pd.DataFrame()\n    sub['ImageId'] = new_img_ids\n    sub['EncodedPixels'] = pd.Series(rles).apply(lambda x: ' '.join(str(y) for y in x))\n    sub = add_empty_ids(sub, img_ids, new_img_ids)\n    \n    print('debug: len(img_ids), len(sub.ImageId.unique())', len(img_ids), len(sub.ImageId.unique()))\n    sub.to_csv(file_name, index=False)\n\n","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"a8450496a0d15062403c78880b5ba73ed61fa891"},"cell_type":"markdown","source":"Unfortunately, I dind't find how to use the Keras EarlyStopping and ModelCheckpoint collbacks with the ConvOffset2D layer. Please, let me know if you find it out."},{"metadata":{"ExecuteTime":{"end_time":"2018-02-23T20:13:37.674416Z","start_time":"2018-02-23T18:03:41.466997Z"},"_cell_guid":"9b52df95-5431-4765-a304-fb3192fb25ac","_uuid":"1b752a957409a3242f07436af05572f8bfe5ad2d","collapsed":true,"trusted":false},"cell_type":"code","source":"import cv2\nimport os\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\nfrom keras.models import load_model\n\n\nif __name__ == \"__main__\":\n    start_time = time.time()\n    img_size = 256\n    batch_size = 8 # 32\n    epochs = 100\n    \n    data_path = 'data/'    \n    train_path = data_path +'stage1/train/'\n    test_path = data_path +'stage2/test/'\n    \n    log('Process Started')\n    X_train, Y_train, X_test, sizes_test = make_df(train_path, test_path, img_size)\n    log('Read data')\n    xtr, xval, ytr, yval = train_test_split(X_train, Y_train, test_size=0.1, random_state=7)\n    train_generator, val_generator = generator(xtr, xval, ytr, yval, batch_size)\n    log('Data prepared')\n    \n    model = Unet(img_size)\n    model.compile(optimizer='adam', loss=bce_dice_loss, metrics=[mean_iou])\n    \n    log('Model Fitting started...')\n    earlystopper = EarlyStopping(patience=25, verbose=1)\n    checkpointer = ModelCheckpoint('model.v3.h5', verbose=1, save_best_only=True)\n    model.fit_generator(train_generator, steps_per_epoch=len(xtr)/6, epochs=epochs,\n                        validation_data=val_generator, validation_steps=len(xval)/batch_size,\n  #                     callbacks=[earlystopper, checkpointer]\n                       )\n    \n   # model = load_model('model.v3.h5', custom_objects={'mean_iou': mean_iou, 'bce_dice_loss': bce_dice_loss})\n   #                    #,'ConvOffset2D': ConvOffset2D})\n    log('  Model Fitting finished')\n    \n    preds_test = model.predict(X_test, verbose=1)\n    log('Test Predicted')\n\n    save_submission('submission.v3.csv', preds_test, test_path, sizes_test)    \n    log('Sibmission Created')\n","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"984604c4f436c6bcae16aa875df5eb1514c1415c"},"cell_type":"code","source":"model.save('model.v3.h5')","execution_count":11,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ff36bd477fd362df290b16425aed3ebb717dd97c"},"cell_type":"code","source":"# a visual ad-hock verification of the submission.\ndf = pd.read_csv('submission.v3.csv', nrows=2000)\ndf.head()","execution_count":16,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"432bef9ac7c0626821d8ad830f2e7151003a23ba"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}