{"nbformat_minor": 1, "cells": [{"source": ["# Intro\n", "Hello! This rather quick and dirty kernel shows how to get started on segmenting nuclei using a neural network in Keras. \n", "\n", "The architecture used is the so-called [U-Net](https://arxiv.org/abs/1505.04597), which is very common for image segmentation problems such as this. I believe they also have a tendency to work quite well even on small datasets.\n", "\n", "Let's get started importing everything we need!"], "metadata": {"_cell_guid": "e61ef2d8-f315-4f7f-b07e-1de0f4e8441a", "_uuid": "1677fddbb95f7545b6540e9201f3339a0fdbfc5d"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["import os\n", "import sys\n", "import random\n", "import warnings\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "import matplotlib.pyplot as plt\n", "\n", "from tqdm import tqdm\n", "from itertools import chain\n", "from skimage.io import imread, imshow, imread_collection, concatenate_images\n", "from skimage.transform import resize\n", "from skimage.morphology import label\n", "\n", "from keras.models import Model, load_model\n", "from keras.layers import Input\n", "from keras.layers.core import Dropout, Lambda\n", "from keras.layers.convolutional import Conv2D, Conv2DTranspose\n", "from keras.layers.pooling import MaxPooling2D\n", "from keras.layers.merge import concatenate\n", "from keras.callbacks import EarlyStopping, ModelCheckpoint\n", "from keras import backend as K\n", "\n", "import tensorflow as tf\n", "\n", "# Set some parameters\n", "IMG_WIDTH = 128\n", "IMG_HEIGHT = 128\n", "IMG_CHANNELS = 3\n", "TRAIN_PATH = '../input/stage1_train/'\n", "TEST_PATH = '../input/stage1_test/'\n", "\n", "warnings.filterwarnings('ignore', category=UserWarning, module='skimage')\n", "seed = 42\n", "random.seed = seed\n", "np.random.seed = seed"], "outputs": [], "metadata": {"_cell_guid": "c332549b-8d23-4bb5-8497-e7a8eb8b21d2", "_uuid": "5c38504af3a84bee68c66d3cde74443c58df422f"}, "cell_type": "code"}, {"execution_count": null, "source": ["# Get train and test IDs\n", "train_ids = next(os.walk(TRAIN_PATH))[1]\n", "test_ids = next(os.walk(TEST_PATH))[1]"], "outputs": [], "metadata": {"collapsed": true, "_cell_guid": "ffa0caf0-2d1b-40f2-865b-8e6db88526b6", "_uuid": "3fb9d6530fbbd0e22e41fc4fd9fd9fc0bff027ac"}, "cell_type": "code"}, {"source": ["# Get the data\n", "Let's first import all the images and associated masks. I downsample both the training and test images to keep things light and manageable, but we need to keep a record of the original sizes of the test images to upsample our predicted masks and create correct run-length encodings later on. There are definitely better ways to handle this, but it works fine for now!"], "metadata": {"_cell_guid": "59c4a25d-645f-4b74-9c53-145ac78cc481", "_uuid": "875af74f980236825de3a650825b46e25632422c"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Get and resize train images and masks\n", "X_train = np.zeros((len(train_ids), IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS), dtype=np.uint8)\n", "Y_train = np.zeros((len(train_ids), IMG_HEIGHT, IMG_WIDTH, 1), dtype=np.bool)\n", "print('Getting and resizing train images and masks ... ')\n", "sys.stdout.flush()\n", "for n, id_ in tqdm(enumerate(train_ids), total=len(train_ids)):\n", "    path = TRAIN_PATH + id_\n", "    img = imread(path + '/images/' + id_ + '.png')[:,:,:IMG_CHANNELS]\n", "    img = resize(img, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True)\n", "    X_train[n] = img\n", "    mask = np.zeros((IMG_HEIGHT, IMG_WIDTH, 1), dtype=np.bool)\n", "    for mask_file in next(os.walk(path + '/masks/'))[2]:\n", "        mask_ = imread(path + '/masks/' + mask_file)\n", "        mask_ = np.expand_dims(resize(mask_, (IMG_HEIGHT, IMG_WIDTH), mode='constant', \n", "                                      preserve_range=True), axis=-1)\n", "        mask = np.maximum(mask, mask_)\n", "    Y_train[n] = mask\n", "\n", "# Get and resize test images\n", "X_test = np.zeros((len(test_ids), IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS), dtype=np.uint8)\n", "sizes_test = []\n", "print('Getting and resizing test images ... ')\n", "sys.stdout.flush()\n", "for n, id_ in tqdm(enumerate(test_ids), total=len(test_ids)):\n", "    path = TEST_PATH + id_\n", "    img = imread(path + '/images/' + id_ + '.png')[:,:,:IMG_CHANNELS]\n", "    sizes_test.append([img.shape[0], img.shape[1]])\n", "    img = resize(img, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True)\n", "    X_test[n] = img\n", "\n", "print('Done!')"], "outputs": [], "metadata": {"_cell_guid": "ca0cc34b-c26f-41ee-88d7-975aebdb634e", "_uuid": "9e389ba8bdb5b6fc03b231b6a6c84a8bde634053"}, "cell_type": "code"}, {"source": ["Let's see if things look all right by drawing some random images and their associated masks."], "metadata": {"_cell_guid": "c0523b03-1fc5-4505-a1b8-eb35ee617c8a", "_uuid": "d4f8327802a1ec6139ce0585953986272ba62ce1"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Check if training data looks all right\n", "ix = random.randint(0, len(train_ids))\n", "imshow(X_train[ix])\n", "plt.show()\n", "imshow(np.squeeze(Y_train[ix]))\n", "plt.show()"], "outputs": [], "metadata": {"_cell_guid": "88829b53-50ce-45d9-9540-77dd7384ad4c", "_uuid": "283af26f0860b7069bdfd133c746e5d20971542c"}, "cell_type": "code"}, {"source": ["Seems good!\n", "\n", "# Create our Keras metric\n", "\n", "Now we try to define the *mean average precision at different intersection over union (IoU) thresholds* metric in Keras. TensorFlow has a mean IoU metric, but it doesn't have any native support for the mean over multiple thresholds, so I tried to implement this. **I'm by no means certain that this implementation is correct, though!** Any assistance in verifying this would be most welcome! \n", "\n", "*Update: This implementation is most definitely not correct due to the very large discrepancy between the results reported here and the LB results. It also seems to just increase over time no matter what when you train ... *"], "metadata": {"_cell_guid": "2574ffe9-b911-4bfd-a00f-9ba5c25f45de", "_uuid": "938648da705689a0f940ff462477c801db3f0737"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Define IoU metric\n", "def mean_iou(y_true, y_pred):\n", "    prec = []\n", "    for t in np.arange(0.5, 1.0, 0.05):\n", "        y_pred_ = tf.to_int32(y_pred > t)\n", "        score, up_opt = tf.metrics.mean_iou(y_true, y_pred_, 2)\n", "        K.get_session().run(tf.local_variables_initializer())\n", "        with tf.control_dependencies([up_opt]):\n", "            score = tf.identity(score)\n", "        prec.append(score)\n", "    return K.mean(K.stack(prec), axis=0)"], "outputs": [], "metadata": {"collapsed": true, "_cell_guid": "c1df6f3a-d58f-434b-9216-ef7be38637d4", "_uuid": "5abd38950ae99b60f8afec7656eb654a48d449fe"}, "cell_type": "code"}, {"source": ["# Build and train our neural network\n", "Next we build our U-Net model, loosely based on [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/pdf/1505.04597.pdf) and very similar to [this repo](https://github.com/jocicmarko/ultrasound-nerve-segmentation) from the Kaggle Ultrasound Nerve Segmentation competition.\n", "\n", "![](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/u-net-architecture.png)"], "metadata": {"_cell_guid": "c3b9f148-1dba-4b6a-981b-6cdbf394fc3c", "_uuid": "986488a4c5223576be370e224426a30431911eb2"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Build U-Net model\n", "inputs = Input((IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS))\n", "s = Lambda(lambda x: x / 255) (inputs)\n", "\n", "c1 = Conv2D(16, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (s)\n", "c1 = Dropout(0.1) (c1)\n", "c1 = Conv2D(16, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c1)\n", "p1 = MaxPooling2D((2, 2)) (c1)\n", "\n", "c2 = Conv2D(32, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (p1)\n", "c2 = Dropout(0.1) (c2)\n", "c2 = Conv2D(32, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c2)\n", "p2 = MaxPooling2D((2, 2)) (c2)\n", "\n", "c3 = Conv2D(64, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (p2)\n", "c3 = Dropout(0.2) (c3)\n", "c3 = Conv2D(64, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c3)\n", "p3 = MaxPooling2D((2, 2)) (c3)\n", "\n", "c4 = Conv2D(128, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (p3)\n", "c4 = Dropout(0.2) (c4)\n", "c4 = Conv2D(128, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c4)\n", "p4 = MaxPooling2D(pool_size=(2, 2)) (c4)\n", "\n", "c5 = Conv2D(256, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (p4)\n", "c5 = Dropout(0.3) (c5)\n", "c5 = Conv2D(256, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c5)\n", "\n", "u6 = Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same') (c5)\n", "u6 = concatenate([u6, c4])\n", "c6 = Conv2D(128, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (u6)\n", "c6 = Dropout(0.2) (c6)\n", "c6 = Conv2D(128, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c6)\n", "\n", "u7 = Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same') (c6)\n", "u7 = concatenate([u7, c3])\n", "c7 = Conv2D(64, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (u7)\n", "c7 = Dropout(0.2) (c7)\n", "c7 = Conv2D(64, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c7)\n", "\n", "u8 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c7)\n", "u8 = concatenate([u8, c2])\n", "c8 = Conv2D(32, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (u8)\n", "c8 = Dropout(0.1) (c8)\n", "c8 = Conv2D(32, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c8)\n", "\n", "u9 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c8)\n", "u9 = concatenate([u9, c1], axis=3)\n", "c9 = Conv2D(16, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (u9)\n", "c9 = Dropout(0.1) (c9)\n", "c9 = Conv2D(16, (3, 3), activation='elu', kernel_initializer='he_normal', padding='same') (c9)\n", "\n", "outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n", "\n", "model = Model(inputs=[inputs], outputs=[outputs])\n", "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[mean_iou])\n", "model.summary()"], "outputs": [], "metadata": {"_cell_guid": "c1dbc57c-b497-4ccb-b077-2053203ab7ed", "_uuid": "0aa97d66c29f45dfac9b0f45fcf74ba0e778ba5d"}, "cell_type": "code"}, {"source": ["*Update: Changed to ELU units, added dropout.*\n", "\n", "Next we fit the model on the training data, using a validation split of 0.1. We use a small batch size because we have so little data. I recommend using checkpointing and early stopping when training your model. I won't do it here to make things a bit more reproducible (although it's very likely that your results will be different anyway). I'll just train for 10 epochs, which takes around 10 minutes in the Kaggle kernel with the current parameters. \n", "\n", "*Update: Added early stopping and checkpointing and increased to 30 epochs.*"], "metadata": {"_cell_guid": "72330944-6ce7-4070-b276-c3c4b20c4fe5", "_uuid": "92350b6e18cc50f3fa7b6e9a02d39fcbff8238f7"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Fit model\n", "earlystopper = EarlyStopping(patience=5, verbose=1)\n", "checkpointer = ModelCheckpoint('model-dsbowl2018-1.h5', verbose=1, save_best_only=True)\n", "results = model.fit(X_train, Y_train, validation_split=0.1, batch_size=16, epochs=50, \n", "                    callbacks=[earlystopper, checkpointer])"], "outputs": [], "metadata": {"_cell_guid": "9415b1c4-aa69-41b9-a1e3-d6053dbd4f64", "_uuid": "c060db22daa2abf12b28240cd81bbcbf1ce1bf87"}, "cell_type": "code"}, {"source": ["All right, looks good! Loss seems to be a bit erratic, though. I'll leave it to you to improve the model architecture and parameters! \n", "\n", "# Make predictions\n", "\n", "Let's make predictions both on the test set, the val set and the train set (as a sanity check). Remember to load the best saved model if you've used early stopping and checkpointing."], "metadata": {"_cell_guid": "1f381f5b-1b71-4daa-a417-e02f4894540b", "_uuid": "bb15226ea617cf91ed8f43179fccb5a15809e5a0"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Predict on train, val and test\n", "model = load_model('model-dsbowl2018-1.h5', custom_objects={'mean_iou': mean_iou})\n", "preds_train = model.predict(X_train[:int(X_train.shape[0]*0.9)], verbose=1)\n", "preds_val = model.predict(X_train[int(X_train.shape[0]*0.9):], verbose=1)\n", "preds_test = model.predict(X_test, verbose=1)\n", "\n", "# Threshold predictions\n", "preds_train_t = (preds_train > 0.5).astype(np.uint8)\n", "preds_val_t = (preds_val > 0.5).astype(np.uint8)\n", "preds_test_t = (preds_test > 0.5).astype(np.uint8)\n", "\n", "# Create list of upsampled test masks\n", "preds_test_upsampled = []\n", "for i in range(len(preds_test)):\n", "    preds_test_upsampled.append(resize(np.squeeze(preds_test[i]), \n", "                                       (sizes_test[i][0], sizes_test[i][1]), \n", "                                       mode='constant', preserve_range=True))"], "outputs": [], "metadata": {"_cell_guid": "2daa48d5-ac98-4e18-af3f-a582baaa44f0", "_uuid": "f841760b4abca1a25cb750822f88268bd79bf2ce"}, "cell_type": "code"}, {"execution_count": null, "source": ["# Perform a sanity check on some random training samples\n", "ix = random.randint(0, len(preds_train_t))\n", "imshow(X_train[ix])\n", "plt.show()\n", "imshow(np.squeeze(Y_train[ix]))\n", "plt.show()\n", "imshow(np.squeeze(preds_train_t[ix]))\n", "plt.show()"], "outputs": [], "metadata": {"_cell_guid": "649248cd-a1fb-4da6-ade2-4bebad44bcab", "_uuid": "7e06242a50870e07a080064a4912b761775990fa"}, "cell_type": "code"}, {"source": ["The model is at least able to fit to the training data! Certainly a lot of room for improvement even here, but a decent start. How about the validation data?"], "metadata": {"_cell_guid": "af602aea-5e56-42a8-9331-54b4b2650593", "_uuid": "5fcee2b9aee2fba5c60d43ad48a14139e9c1318c"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Perform a sanity check on some random validation samples\n", "ix = random.randint(0, len(preds_val_t))\n", "imshow(X_train[int(X_train.shape[0]*0.9):][ix])\n", "plt.show()\n", "imshow(np.squeeze(Y_train[int(Y_train.shape[0]*0.9):][ix]))\n", "plt.show()\n", "imshow(np.squeeze(preds_val_t[ix]))\n", "plt.show()"], "outputs": [], "metadata": {"_cell_guid": "4f66b75c-c694-41a1-8c91-34bb6595837b", "_uuid": "d4ccbb559375bc2777ffb692a20adc313159f2cc"}, "cell_type": "code"}, {"source": ["Not too shabby! Definitely needs some more training and tweaking.\n", "\n", "# Encode and submit our results\n", "\n", "Now it's time to submit our results. I've stolen [this](https://www.kaggle.com/rakhlin/fast-run-length-encoding-python) excellent implementation of run-length encoding."], "metadata": {"_cell_guid": "a6690535-b2e4-49ac-98d9-7191bfabfb6f", "_uuid": "6a34c98de7c6ae473f676a34fe7e099b46764eca"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Run-length encoding stolen from https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\n", "def rle_encoding(x):\n", "    dots = np.where(x.T.flatten() == 1)[0]\n", "    run_lengths = []\n", "    prev = -2\n", "    for b in dots:\n", "        if (b>prev+1): run_lengths.extend((b + 1, 0))\n", "        run_lengths[-1] += 1\n", "        prev = b\n", "    return run_lengths\n", "\n", "def prob_to_rles(x, cutoff=0.5):\n", "    lab_img = label(x > cutoff)\n", "    for i in range(1, lab_img.max() + 1):\n", "        yield rle_encoding(lab_img == i)"], "outputs": [], "metadata": {"collapsed": true, "_cell_guid": "59a0af60-a7d7-41ef-a6fe-9e3c72defa07", "_uuid": "4f99c1bf852e82b60bd4f982ca0df293f712cdf0"}, "cell_type": "code"}, {"source": ["Let's iterate over the test IDs and generate run-length encodings for each seperate mask identified by skimage ..."], "metadata": {"_cell_guid": "31133f8c-3f40-4dff-8e1d-898d56672332", "_uuid": "2e07f6afc4787b068ba714428145dcb3951d718f"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["new_test_ids = []\n", "rles = []\n", "for n, id_ in enumerate(test_ids):\n", "    rle = list(prob_to_rles(preds_test_upsampled[n]))\n", "    rles.extend(rle)\n", "    new_test_ids.extend([id_] * len(rle))"], "outputs": [], "metadata": {"collapsed": true, "_cell_guid": "22fe24a1-7659-4cc9-9d23-211f38e5b99f", "_uuid": "089587843ed6a3955fdcb9b23a6ec3bf5d703688"}, "cell_type": "code"}, {"source": ["... and then finally create our submission!"], "metadata": {"_cell_guid": "20b6b627-0fd6-425d-888f-da7f39efb124", "_uuid": "849184a40a2c9c21506d8b8eb10ad9155fa229e8"}, "cell_type": "markdown"}, {"execution_count": null, "source": ["# Create submission DataFrame\n", "sub = pd.DataFrame()\n", "sub['ImageId'] = new_test_ids\n", "sub['EncodedPixels'] = pd.Series(rles).apply(lambda x: ' '.join(str(y) for y in x))\n", "sub.to_csv('sub-dsbowl2018-1.csv', index=False)"], "outputs": [], "metadata": {"collapsed": true, "_cell_guid": "1ba0ee3a-cca0-4349-83f6-09a1ac6fcb44", "_uuid": "ba589f56f5be1e6886bc88f5bf9e7d0a408e4048"}, "cell_type": "code"}, {"source": ["This scored 0.233 on the LB for me. That was with version 2 of this notebook; be aware that the results from the neural network are extremely erratic and vary greatly from run to run (version 3 is significantly worse, for example). Version 7 scores 0.277!\n", "\n", "You should easily be able to stabilize and improve the results just by changing a few parameters, tweaking the architecture a little bit and training longer with early stopping.\n", "\n", "**Have fun!**\n", "\n", "LB score history:\n", "- Version 7: 0.277 LB"], "metadata": {"collapsed": true, "_cell_guid": "222475b9-3171-461a-90f0-a820a6bd2634", "_uuid": "fb5e6f8cca872f1bd7036f6d9ac2ed2cab615536"}, "cell_type": "markdown"}, {"execution_count": null, "source": [], "outputs": [], "metadata": {"collapsed": true, "_cell_guid": "3f5e5a47-6133-4870-976a-a8e4fa7bf46c", "_uuid": "2a83eab66bf55194f300953bea5534b6a043130f"}, "cell_type": "code"}], "metadata": {"language_info": {"mimetype": "text/x-python", "version": "3.6.4", "name": "python", "codemirror_mode": {"version": 3, "name": "ipython"}, "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "file_extension": ".py"}, "kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3"}}, "nbformat": 4}