{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Segmentation with U-Net","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we will be working the 2018 Data Science Bowl competetion dataset for cell nuclei segmentation. The original dataset can be found at [2018 Data Science Bowl](https://www.kaggle.com/c/data-science-bowl-2018). In the original dataset, the masks of individual nucleus segementations for same image are saved as different png files. Since we will be performing binary segmentation, for convenience, the masks of different nulcei have been merged for each image and the new dataset is available at [Nuclei Images Masks from DSB 2018](https://www.kaggle.com/sinjoysaha/nucleiimagesmasksfromdsb2018).\nThe new folder structure is as follows:\n- nucleiimagesmasksfromdsb2018\n    - data-science-bowl-2018\n        - stage1_test\n            - <id_like_0dfg21dfg1...>\n                - images\n                    - \\<same_id>.png\n                    \n        - stage1_train\n            - same_id\n                - images\n                    - \\<same_id>.png\n                - masks\n                    - \\<same_id>.png\n                    \n        - stage1_train_combinedmasks\n            - \\<same_id>.png\n           \nFor entirety, the code for merging of masks is also provided in comment blocks. Understanding it is upto the reader.","metadata":{}},{"cell_type":"code","source":"# Importing necessary libraries\nimport os\nimport numpy as np\nfrom tqdm import tqdm\nfrom skimage.io import imread, imshow\nfrom skimage.transform import resize\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.layers import (Input, Lambda, Conv2D, Dropout, MaxPooling2D, \n                                    Conv2DTranspose, concatenate)\n\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_PATH = '../input/nucleiimagesmasksfromdsb2018/data-science-bowl-2018/stage1_train/'\nTRAINMASKS_PATH = '../input/nucleiimagesmasksfromdsb2018/data-science-bowl-2018/stage1_train_combinedmasks/'\nTEST_PATH = '../input/nucleiimagesmasksfromdsb2018/data-science-bowl-2018/stage1_test/'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Params\nIMG_WIDTH = 128\nIMG_HEIGHT = 128\nIMG_CHANNELS = 3","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{}},{"cell_type":"code","source":"train_ids = next(os.walk(TRAIN_PATH))[1]\ntest_ids = next(os.walk(TEST_PATH))[1]\nprint(len(train_ids), len(test_ids))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = np.zeros((len(train_ids), IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS), dtype=np.uint8)\ny_train = np.zeros((len(train_ids), IMG_HEIGHT, IMG_WIDTH, 1), dtype=np.bool)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train images\nprint('Resizing training images and masks')\nfor n, id_ in tqdm(enumerate(train_ids), total=len(train_ids)):\n    path = TRAIN_PATH + id_\n    img = imread(path+'/images/'+id_+'.png')[:,:,:IMG_CHANNELS]\n    img = resize(img, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True)\n    X_train[n] = img\n    '''\n    # This code is only needed to combine masks of diff cells into one mask\n    # Adding diff masks for diff cells into one mask\n    mask = np.zeros((IMG_HEIGHT, IMG_WIDTH, 1), dtype=np.bool)\n    for mask_file in next(os.walk(path+'/masks/'))[2]:\n        mask_ = imread(path+'/masks/'+mask_file)\n        mask_ = np.expand_dims(resize(\n                mask_, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True),\n                              axis=-1)\n        mask = np.maximum(mask, mask_)\n        \n    y_train[n] = mask'''\n    mask = imread(TRAINMASKS_PATH+id_+'.png')\n    mask = np.expand_dims(resize(\n                mask, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True),\n                              axis=-1)\n    y_train[n] = mask","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\n# Code to Save the masks and zip it for downloading\nos.makedirs('../outputs/train_masks')\ndef img_frombytes(data):\n    size = (data.shape[0],data.shape[1])\n    databytes = np.packbits(data, axis=1)\n    return Image.frombytes(mode='1', size=size, data=databytes)\n\nfor n, id_ in tqdm(enumerate(train_ids), total=len(train_ids)):\n    print(train_ids[n],end=' ')\n    im = img_frombytes((y_train[n]*255).astype(np.uint8))\n    im.save('../outputs/train_masks/'+train_ids[n]+'.png')\n    print(' Saved to '+'../outputs/train_masks/'+train_ids[n]+'.png\\n')\n    \nimport shutil\nOUTPUT_NAME = 'download_folder'\nDIRECTORY_TO_ZIP = '../outputs/train_masks/'\nshutil.make_archive(OUTPUT_NAME, 'zip', DIRECTORY_TO_ZIP)\n'''","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 10\nplt.subplot(121)\nimshow(X_train[i])\nplt.title('Image')\nplt.subplot(122)\nimshow(np.squeeze(y_train[i]))\nplt.title('Mask')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = np.zeros((len(test_ids), IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS), dtype=np.uint8)\nsizes_test = []","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Test images\nprint('Resizing test images and masks')\nfor n, id_ in tqdm(enumerate(test_ids), total=len(test_ids)):\n    path = TEST_PATH + id_\n    img = imread(path+'/images/'+id_+'.png')[:,:,:IMG_CHANNELS]\n    img = resize(img, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True)\n    X_test[n] = img","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the U-Net Model","metadata":{}},{"cell_type":"code","source":"# Inputs\ninputs = Input((IMG_WIDTH, IMG_HEIGHT, IMG_CHANNELS))\n# Change integer to float and also scale pixel values\ns = Lambda(lambda x: x/255.0)(inputs)\n\n# Contraction/Encoder path\n# Block 1\nc1 = Conv2D(filters=16, kernel_size=(3,3),\n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(s)\nc1 = Dropout(0.1)(c1)\nc1 = Conv2D(filters=16, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c1)\np1 = MaxPooling2D(pool_size=(2,2))(c1)\n# Block 2\nc2 = Conv2D(filters=32, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(p1)\nc2 = Dropout(0.1)(c2)\nc2 = Conv2D(filters=32, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c2)\np2 = MaxPooling2D(pool_size=(2,2))(c2)\n# Block 3\nc3 = Conv2D(filters=64, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(p2)\nc3 = Dropout(0.2)(c3)\nc3 = Conv2D(filters=64, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c3)\np3 = MaxPooling2D(pool_size=(2,2))(c3)\n# Block 4\nc4 = Conv2D(filters=128, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(p3)\nc4 = Dropout(0.2)(c4)\nc4 = Conv2D(filters=128, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c4)\np4 = MaxPooling2D(pool_size=(2,2))(c4)\n# Block 5\nc5 = Conv2D(filters=256, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(p4)\nc5 = Dropout(0.3)(c5)\nc5 = Conv2D(filters=256, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c5)\n\n# Expansion/Decoder path\n# Block 6\nu6 = Conv2DTranspose(filters=128, kernel_size=(2,2), strides = (2,2), padding='same')(c5)\nu6 = concatenate([u6, c4])\nc6 = Conv2D(filters=128, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(u6)\nc6 = Dropout(0.2)(c6)\nc6 = Conv2D(filters=128, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c6)\n\n# Block 7\nu7 = Conv2DTranspose(filters=64, kernel_size=(2,2), strides = (2,2), padding='same')(c6)\nu7 = concatenate([u7, c3])\nc7 = Conv2D(filters=64, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(u7)\nc7 = Dropout(0.2)(c7)\nc7 = Conv2D(filters=64, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c7)\n\n# Block 8\nu8 = Conv2DTranspose(filters=32, kernel_size=(2,2), strides = (2,2), padding='same')(c7)\nu8 = concatenate([u8, c2])\nc8 = Conv2D(filters=32, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(u8)\nc8 = Dropout(0.1)(c8)\nc8 = Conv2D(filters=32, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c8)\n\n# Block 9\nu9 = Conv2DTranspose(filters=16, kernel_size=(2,2), strides = (2,2), padding='same')(c8)\nu9 = concatenate([u9, c1])\nc9 = Conv2D(filters=16, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(u9)\nc9 = Dropout(0.1)(c9)\nc9 = Conv2D(filters=16, kernel_size=(3,3), \n                            activation='relu', kernel_initializer='he_normal',\n                           padding='same')(c9)\n# Outputs\noutputs = Conv2D(filters=1, kernel_size=(1,1), \n                            activation='sigmoid')(c9)\n\nmodel = Model(inputs=[inputs], outputs=[outputs])\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training","metadata":{}},{"cell_type":"code","source":"# Callbacks\ncallbacks_list = [ModelCheckpoint('nuclei_model.h5', verbose=1, save_best_only=True),\n                  EarlyStopping(patience=2, monitor='val_loss'),\n                  TensorBoard(log_dir='logs')]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_results = model.fit(X_train, y_train, validation_split=0.1, batch_size=32, \n                          epochs=25, callbacks=callbacks_list)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=[10,6])\nfor key in model_results.history.keys():\n    plt.plot(model_results.history[key], label=key)\n    \nplt.legend()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Further visualizations can be done using the `logs` in `output` folder. The `logs` was created using `TensorBoard`.","metadata":{}},{"cell_type":"markdown","source":"# Model Evaluation","metadata":{}},{"cell_type":"code","source":"preds_train = model.predict(X_train[:int(X_train.shape[0]*0.9)], verbose=1)\ny_true_train = y_train[:int(y_train.shape[0]*0.9)]\npreds_val = model.predict(X_train[int(X_train.shape[0]*0.9):], verbose=1)\ny_true_val = y_train[int(y_train.shape[0]*0.9):]\npreds_test = model.predict(X_test, verbose=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Thresholding\npreds_train_t = (preds_train > 0.5).astype(np.uint8)\npreds_val_t = (preds_val > 0.5).astype(np.uint8)\npreds_test_t = (preds_test > 0.5).astype(np.uint8)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show images\ndef show_images(i, ti, orgimg, y_true, preds, preds_t):\n    plt.figure(figsize=(8,8))\n    plt.subplot(221)\n    imshow(orgimg[i])\n    plt.title('Image to be Segmented')\n    plt.subplot(222)\n    imshow(y_true[ti])\n    plt.title('Segmentation Ground Truth')\n    plt.subplot(223)\n    imshow(preds[ti])\n    plt.title('Predicted Segmentation')\n    plt.subplot(224)\n    imshow(preds_t[ti])\n    plt.title('Thresholded Segmentation')\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# On Train\n# train max 602\ni = 602\nshow_images(i, i, X_train, y_true_train, preds_train, preds_train_t)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# On Val\n# i = 603:669\ni = 660\nshow_images(i, i-603,  X_train, y_true_val, preds_val, preds_val_t)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# On Test\n# Ground Truths Not Available\ni = 0\nplt.figure(figsize=(8,8))\nplt.subplot(221)\nimshow(X_test[i])\nplt.title('Image to be Segmented')\nplt.subplot(222)\nplt.title('Segmentation Ground Truth NA')\nplt.subplot(223)\nimshow(preds_test[i])\nplt.title('Predicted Segmentation')\nplt.subplot(224)\nimshow(preds_test_t[i])\nplt.title('Thresholded Segmentation')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Evaluate on val data\")\nresults = model.evaluate(X_train[int(X_train.shape[0]*0.9):], y_train[int(y_train.shape[0]*0.9):], batch_size=128)\nprint(\"Test Loss:\", results[0])\nprint(\"Test Acc :\", results[1]*100, \"%\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It would have been much better to evaluate on new test set other than validation set but here the data is too small to split into train, validation and test sets. One can extend this work by using techniques of data augmentation which may be considered in the next version of this notebook.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}