{"cells":[{"metadata":{"_cell_guid":"fae00f4b-81e7-45c8-b3ef-6ad64b5d2f1b","_uuid":"50ed7d03c138e2929326cc9f5746669150235d1d"},"cell_type":"markdown","source":"# Overview\nThe notebook uses the [CONX package](https://github.com/Calysto/conx) from [Douglas Blank](https://cs.brynmawr.edu/~dblank/) and the Calysto team to show how the basic U-Net architecture works on segmentation problems. We take the Kaggle Data Science Bowl for 2018 as an exciting example of real-world segmentation and study how our MiniUNET architecture handles the problem. This kernel/notebook is best run interactively (fork it and run it a in an interactive Kaggle Session or download the IPython notebook and run it using mybinder on \n[![imagetool](https://img.shields.io/badge/launch-UNET_Demo-yellow.svg)](http://mybinder.org/v2/gh/Quantitative-Big-Imaging/conx/master?urlpath=%2Fapps%2Fseg_notebooks%2FUNetDemo.ipynb)\n\n## Note\nThis is not 'real' U-Net since it does not have the proper cropping layers nor the correct size and depth (Ronneberger trained the original model using 512x512 images and having many more layers of max-pooling and upsampling).  The cropping layers are important as well since edges can skew the weights in the convolutions and cause the algorithm to converge slowly or with small enough windows incorrectly."},{"metadata":{"_cell_guid":"c89468e7-dfb4-4881-99a4-5caa6e75414c","_uuid":"e44fa262bfdc5a5d32bab9584524c5bdbf274aa5","collapsed":true,"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom skimage.io import imread","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"592aeb1f-f5c2-404c-9816-1516eb7e2767","_uuid":"f2249202fd2760db4144ea4bc670581fb3e61f98"},"cell_type":"markdown","source":"# Loading Training Data\nHere we load the images using the tools from Keegil in his [kernel](https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277?scriptVersionId=2164855). It is a very simplified way of loading the data in and downsampling everything, it is probably worthwhile to investigate better approaches for handling high-resolution data well. As other kernels show the image sizes vary wildly. We also apply the normalizing [code](https://www.kaggle.com/kmader/normalizing-brightfield-stained-and-fluorescence) to make the images more consistent "},{"metadata":{"_cell_guid":"a81a0da6-3a0f-446f-8aa7-36fb3ba58a43","_uuid":"d48a98990eda9df26890f219bed19414d7c6bfc1","trusted":true},"cell_type":"code","source":"import os\nimport sys\nimport random\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nfrom itertools import chain\nfrom skimage.io import imshow, imread_collection, concatenate_images\nfrom skimage.io import imread as imread_raw\nimport cv2\nfrom skimage.transform import resize\nfrom skimage.morphology import label\ndef imread(in_path, grid_size = 8): \n    in_rgb_img = imread_raw(in_path)\n    if len(in_rgb_img.shape)>2:\n        bgr = in_rgb_img[:,:,[2,1,0]] # flip r and b\n        lab = cv2.cvtColor(bgr, cv2.COLOR_BGR2LAB)[:,:,0]\n    else:\n        lab = in_rgb_img\n    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(grid_size,grid_size))\n    out_img = np.expand_dims(clahe.apply(lab), -1)\n    return 255-out_img if out_img.mean()>127 else out_img\n\n# Set some parameters\nIMG_WIDTH = 128\nIMG_HEIGHT = 128\nIMG_CHANNELS = 1\nTRAIN_PATH = '../input/stage1_train/'\nTEST_PATH = '../input/stage1_test/'\n\nwarnings.filterwarnings('ignore', category=UserWarning, module='skimage')\nseed = 42\nrandom.seed = seed\n# Get train and test IDs\ntrain_ids = next(os.walk(TRAIN_PATH))[1]\ntest_ids = next(os.walk(TEST_PATH))[1]\n# Get and resize train images and masks\nX_train = np.zeros((len(train_ids), IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS), dtype=np.uint8)\nY_train = np.zeros((len(train_ids), IMG_HEIGHT, IMG_WIDTH, 1), dtype=np.bool)\n\nprint('Getting and resizing train images and masks ... ')\nsys.stdout.flush()\nfor n, id_ in tqdm(enumerate(train_ids), total=len(train_ids)):\n    path = TRAIN_PATH + id_\n    img = imread(path + '/images/' + id_ + '.png')[:,:,:IMG_CHANNELS]\n    img = resize(img, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True)\n    X_train[n] = img\n    mask = np.zeros((IMG_HEIGHT, IMG_WIDTH, 1), dtype=np.bool)\n    for mask_file in next(os.walk(path + '/masks/'))[2]:\n        mask_ = imread_raw(path + '/masks/' + mask_file)\n        mask_ = np.expand_dims(resize(mask_, (IMG_HEIGHT, IMG_WIDTH), mode='constant', \n                                      preserve_range=True), axis=-1)\n        mask = np.maximum(mask, mask_)\n    Y_train[n] = mask\n\n# Get and resize test images\nX_test = np.zeros((len(test_ids), IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS), dtype=np.uint8)\nsizes_test = []\nprint('Getting and resizing test images ... ')\nsys.stdout.flush()\nfor n, id_ in tqdm(enumerate(test_ids), total=len(test_ids)):\n    path = TEST_PATH + id_\n    img = imread(path + '/images/' + id_ + '.png')[:,:,:IMG_CHANNELS]\n    sizes_test.append([img.shape[0], img.shape[1]])\n    img = resize(img, (IMG_HEIGHT, IMG_WIDTH), mode='constant', preserve_range=True)\n    X_test[n] = img\n\nprint('Done!')","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"bc1d1874-ee17-46da-8422-5113d3ab0854","_uuid":"dbc52617cee42aa3c5bafe6157359b5fbc06e293","trusted":true},"cell_type":"code","source":"t_img = X_train[1]/255.0\nm_img = Y_train[1]\nfig, (ax_img, ax_mask) = plt.subplots(1,2, figsize = (12, 6))\nax_img.imshow(np.clip(255*t_img, 0, 255).astype(np.uint8) if t_img.shape[2]==3 else t_img[:,:,0],\n              interpolation = 'none', cmap = 'bone')\nax_mask.imshow(m_img[:,:,0], cmap = 'bone')","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"c627d414-4322-46cb-93f7-f77efa39d175","_uuid":"b1311eea4202c3d99ee6d1d8a48d3fa7a129e6d4","trusted":true},"cell_type":"code","source":"import conx as cx","execution_count":11,"outputs":[]},{"metadata":{"_cell_guid":"6818dd0a-684a-4c91-adc6-2580ca7172fd","_uuid":"d620a0bd2ae2197f331ed6f95f6927bc0ede41de","collapsed":true,"trusted":true},"cell_type":"code","source":"net = cx.Network(name = \"MiniUNet\")\nbase_depth = 8","execution_count":12,"outputs":[]},{"metadata":{"_cell_guid":"ece23fb8-e589-46e0-8cd6-f72e3efee30e","_uuid":"85ee48bebd728e0ea0127531fba09434baf4bdc1","collapsed":true,"trusted":true},"cell_type":"code","source":"net.add(cx.ImageLayer(\"input\", (IMG_WIDTH, IMG_HEIGHT) , IMG_CHANNELS)) \nnet.add(cx.BatchNormalizationLayer(\"bnorm\"))\nc2 = lambda i, j, act = \"relu\": cx.Conv2DLayer(\"conv_{}\".format(i, j), j, (3, 3), padding='same', activation=act)\nnet.add(c2(0, base_depth))\nnet.add(c2(1, base_depth))\nnet.add(cx.MaxPool2DLayer(\"pool1\", pool_size=(2, 2), dropout=0.25))\nnet.add(c2(2, 2*base_depth))\nnet.add(c2(3, 2*base_depth))\nnet.add(cx.MaxPool2DLayer(\"pool2\", pool_size=(2, 2), dropout=0.25))\nnet.add(c2(4, 4*base_depth))\nnet.add(c2(5, 4*base_depth))\nnet.add(cx.UpSampling2DLayer(\"up2\", size = (2,2)))\nnet.add(cx.ConcatenateLayer(\"cat2\"))\nnet.add(c2(6, 2*base_depth))\nnet.add(c2(7, 2*base_depth))\nnet.add(cx.UpSampling2DLayer(\"up1\", size = (2,2)))\nnet.add(cx.ConcatenateLayer(\"cat1\"))\nnet.add(c2(8, 2*base_depth))\nnet.add(cx.Conv2DLayer(\"output\", 1, (1, 1), padding='same', activation='sigmoid'));","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"a46bdc74-463c-4693-93a6-0f7d2f4dbd63","_uuid":"9eafa153da05815e14283e927636c11e108483e5"},"cell_type":"markdown","source":"# Connections\nWe have to connect all of the layers together in a U-Net style. The tricky part is the skip connections that skip over the max pooling layers and go directly to the concatenate to combine the higher resolution information with the lower resolution feature space"},{"metadata":{"_cell_guid":"15e74050-dd18-43bd-bf28-6eacfed5366d","_uuid":"2b113d0c5d4d92f94dbdf7cc74b5614268cc7620","collapsed":true,"trusted":true},"cell_type":"code","source":"net.connect('input', 'bnorm')\nnet.connect('bnorm', 'conv_0')\nnet.connect('bnorm', 'cat1')\nnet.connect('conv_0', 'conv_1')","execution_count":14,"outputs":[]},{"metadata":{"_cell_guid":"e0585195-ea6c-4f86-89f8-c7077cdc7ef9","_uuid":"03793ee79a25d179d924f487bc4622a6dee42247","collapsed":true,"trusted":true},"cell_type":"code","source":"net.connect('conv_1', 'pool1')\nnet.connect('pool1', 'conv_2')\nnet.connect('conv_2', 'conv_3')\nnet.connect('conv_3', 'pool2')\nnet.connect('pool2', 'conv_4')\nnet.connect('conv_4', 'conv_5')\nnet.connect('conv_5', 'up2')\nnet.connect('up2', 'cat2')\nnet.connect('conv_3', 'cat2')\nnet.connect('cat2', 'conv_6')\nnet.connect('conv_6', 'conv_7')\nnet.connect('conv_7', 'up1')\nnet.connect('up1', 'cat1')\nnet.connect('cat1', 'conv_8')\nnet.connect('conv_8', 'output')","execution_count":15,"outputs":[]},{"metadata":{"_cell_guid":"56073f09-26ca-4226-bd3c-9515f6c53564","_uuid":"d8a7c5f474823d00eb193c572272d6d4fdf94c45","trusted":true},"cell_type":"code","source":"net.compile(error=\"binary_crossentropy\", optimizer=\"adam\")","execution_count":16,"outputs":[]},{"metadata":{"_cell_guid":"467c3b13-b64e-42aa-bcdc-7d80f21c5a82","_uuid":"46bb3a7337f1c5c65e72bae343c2a05026b3c979","trusted":true},"cell_type":"code","source":"net.picture(t_img, dynamic = True, rotate = True, show_targets = True, show_errors=True, scale = 1.0)","execution_count":17,"outputs":[]},{"metadata":{"_cell_guid":"9d4a6ddc-ac82-4928-9820-a929e8b4ea53","_uuid":"3f8c9e7dcf7ece15ce552a3a96cbdf4494f07286","trusted":true},"cell_type":"code","source":"net.dataset.clear()\nip_pairs = [(x/255.0,y*1.0) for x,y in zip(X_train, Y_train)]\nnet.dataset.append(ip_pairs)\nnet.dataset.split(0.25)","execution_count":18,"outputs":[]},{"metadata":{"_cell_guid":"547176c3-578b-4240-8a7a-631b0e38136c","_uuid":"cc389126da5eb1d9b15011b1ac653339879b2961","trusted":true},"cell_type":"code","source":"net.propagate_to_image(\"conv_5\", t_img, scale = 4)","execution_count":20,"outputs":[]},{"metadata":{"_cell_guid":"3a8a4b8f-8844-45e6-b607-5bc14be0b4f8","_uuid":"8cdf30932e458b62c8ea0f3f1ea19cea32f9d730","trusted":true},"cell_type":"code","source":"net.train(epochs=25, record=True)","execution_count":21,"outputs":[]},{"metadata":{"_cell_guid":"856f9916-7078-4815-987e-10cb8ca907b1","_uuid":"bd2c11919b18a864b52f2fb2cec716e471d84d5a","trusted":true},"cell_type":"code","source":"net.propagate_to_image(\"conv_5\", t_img, scale = 5)","execution_count":22,"outputs":[]},{"metadata":{"_cell_guid":"b04ef32f-9a07-4999-b419-4a8b2a20cdc2","_uuid":"74726830e1d13bb282d42571e880da8d9c275f48","scrolled":false,"trusted":true},"cell_type":"code","source":"net.picture(t_img, dynamic = True, rotate = True, show_targets = True, show_errors=True, scale = 1.25)","execution_count":23,"outputs":[]},{"metadata":{"_cell_guid":"e27b56a1-c519-40f7-8a59-6e94d569784a","_uuid":"905f020b0edbbc06314a51b7e1a3003ac805564d","scrolled":false,"trusted":true},"cell_type":"code","source":"net.dashboard()","execution_count":24,"outputs":[]},{"metadata":{"_cell_guid":"a66f04c9-8c58-4fd6-afa4-18ae6bd53519","_uuid":"f3deef6ceebd4ecf03dc7ebeb82af81202e83e73","trusted":true},"cell_type":"code","source":"net.movie(lambda net, epoch: net.propagate_to_image(\"conv_5\", t_img, scale = 3), \n                'mid_conv.gif', mp4 = False)","execution_count":25,"outputs":[]},{"metadata":{"_cell_guid":"d95eb35a-7b1d-49c7-9ab7-b71ced2f1b9c","_uuid":"d9f98df2dc3eee586506afdcd2ac0b5e954824c5","trusted":true},"cell_type":"code","source":"net.movie(lambda net, epoch: net.propagate_to_image(\"conv_8\", t_img, scale = 3), \n                'hr_conv.gif', mp4 = False)","execution_count":26,"outputs":[]},{"metadata":{"_cell_guid":"3a6c3a08-2d36-4bb6-a39a-2104c012c490","_uuid":"693b3809ea6686abda0a3d0e6c9b53aabc7251c0","trusted":true},"cell_type":"code","source":"net.movie(lambda net, epoch: net.propagate_to_image(\"output\", t_img, scale = 3), \n                'output.gif', mp4 = False)","execution_count":27,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"2a0be8ea094f958a0d4515ba66affd970f47debe"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}