{"cells":[{"metadata":{"_uuid":"04d19b5b984566d7c01dbfe5c2960b69fa8808bf"},"cell_type":"markdown","source":"# In this exploratory notebook, we wil get an idea for the types of images in the training and test set and apply some basic image segmenting to predict nuclei. With this notebook we can obtain an accuracy of 0.243 on the test set. See the sci-kit image lecture notes for details: http://www.scipy-lectures.org/packages/scikit-image/. This notebook is also inspired in part by https://www.kaggle.com/stkbailey/teaching-notebook-for-total-imaging-newbies","outputs":[],"execution_count":null},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"bdfaa5dfad35518fc4b04a6180286c447396e957"},"cell_type":"code","source":"from skimage.io import imread\nfrom skimage import color, filters, measure\nimport matplotlib.pyplot as plt\nimport glob\nimport numpy as np\nimport pandas as pd\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5a7d36d006516c6d8c92e0053d2cddeff592f1d"},"cell_type":"markdown","source":"## Part 1 Examine the data: First let's look at what sizes of images there are in the training and test set, and visualize one image from each class.","outputs":[],"execution_count":null},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"15b83b64baab4628a95c19ef774dcb4742ff2ce4"},"cell_type":"code","source":"#glob all of the training images\nimg_list = glob.glob(\"/home/john/Data_Sci/COMP_540/Project/train_images/*/images/*\")\nprint(len(img_list))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"00e9ff47dea869444cdc8c50c6864d981ab97fdd"},"cell_type":"code","source":"#Let's determine how many different image sizes we have, and look at an example of each image size\n\nnum_images=len(img_list)\nimg_shapes = {}\navg_image = {}\nsample_image = {}\nfor i in range(num_images):\n #  img_index = np.random.randint(len(img_list)-1)\n    img_index = i\n    img_path = img_list[img_index]\n    img = imread(img_path)\n#    print(\"Image \", img_index)\n    if img.shape not in img_shapes:\n        img_shapes[img.shape] = 1\n        sample_image[img.shape] = np.copy(img)\n#        avg_image[img.shape] = np.copy(img)\n    else:\n        img_shapes[img.shape] += 1\n#        avg_image[img.shape] += np.copy(img)\n\n#We tried to plot the average image for each image size, but got essentially static, due\n#to the non-uniformity of the locations of the nuclei\nprint(\"There are %d shapes\" % len(img_shapes))\nfor shape, num_images in img_shapes.items():\n#    avg_image[shape] =  avg_image[shape]/num_images\n    print(shape,num_images)\n    print(\"sample image with shape \", shape)\n#    plt.imshow(avg_image[shape])\n    plt.imshow(sample_image[shape])\n\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccebd5a992a59567a026f81be00b14e0bdab134d"},"cell_type":"markdown","source":"There are images of 9 different sizes, and we'll eventually want to be able to predict on images sizes we've never seen before. Even with just plotting 1 example of each image size, we can see there's also a lot of variability in the type of image, e.g.  fluorescence microscopy, various staining/light microscopy, some are colored and some are essentially black/white. We will eventually also want to  predict on the image regardless of what type of microscope was used to capture it.","outputs":[],"execution_count":null},{"metadata":{"_uuid":"c3690455ef10f14f8bfe92e9b4cdca3c27176fcc"},"cell_type":"markdown","source":"Let's repeat the above procedure on the test images, to get a feel for the diversity among the samples there.","outputs":[],"execution_count":null},{"metadata":{"trusted":false,"_uuid":"bfe0b935e0a6e5d482ca9b8c5c7d62c9426cd40d"},"cell_type":"code","source":"#glob all of the test images\nimg_list = glob.glob(\"/home/john/Data_Sci/COMP_540/Project/test_images/*/images/*\")\n#Let's determine how many different image sizes we have, and look at an example of each image size\n\nnum_images=len(img_list)\nimg_shapes = {}\nsample_image = {}\nfor i in range(num_images):\n    img_index = i\n    img_path = img_list[img_index]\n    image_id = img_path.split(\"/\")[-1].split('.')[0] #extract image id from path\n    img = imread(img_path)\n\n    if img.shape not in img_shapes:\n        img_shapes[img.shape] = 1\n        sample_image[img.shape] = np.copy(img)\n\n    else:\n        img_shapes[img.shape] += 1\n\n\nprint(\"There are %d shapes\" % len(img_shapes))\nfor shape, num_images in img_shapes.items():\n    print(shape,num_images)\n    print(\"sample image with shape \", shape)\n    plt.imshow(sample_image[shape])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33887ed5eb3fada4b306af55b2873d14e3376baf"},"cell_type":"markdown","source":"The variability of the test images is even higher than the training image. Our model will need to be able to predict on images of sizes it has never seen.","outputs":[],"execution_count":null},{"metadata":{"_uuid":"52301af57a32bfccdd9b67d10e0cb3766b971f61"},"cell_type":"markdown","source":"## Part 2. Predict: For now, we will try some basic thresholding to separate nuclei (foreground) from the rest of the image (background). We will use this procedure to predict nuclei on the test set after first converting the images to gray scale.","outputs":[],"execution_count":null},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"cc59d250b863a4e603f8c7892823ae2da7cf4a0f"},"cell_type":"code","source":"def rle_encoding(dots): #this function modified from https://www.kaggle.com/rakhlin/fast-run-length-encoding-python\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b>prev+1): run_lengths.extend((b+1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return [str(i) for i in run_lengths]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"collapsed":true,"_uuid":"5cc49b040261cad1af772bfb8b0049848202ca49"},"cell_type":"code","source":"#Now let's use basic otsu thresholding to identify nuclei in the images\n\nimg_list = sorted(glob.glob(\"/home/john/Data_Sci/COMP_540/Project/test_images/*/images/*\"))\n\nnum_images = len(img_list)\ndata = pd.DataFrame(columns=[\"ImageId\", \"EncodedPixels\"])\nfor i in range(num_images):\n    img_index = i\n    img_path = img_list[img_index]\n    image_id = img_path.split(\"/\")[-1].split('.')[0] #extract image id from path\n    img = color.rgb2gray(imread(img_path)) #read as black and white for simple thresholding segmentation\n    val = filters.threshold_otsu(img)\n    nuc_mask = img > val\n    nuc_labels = measure.label(nuc_mask,background=0) #label individual objects (nuclei)\n    #Get RLE encoding for EACH nucleus in a given image\n    for label in np.unique(nuc_labels)[1:]:\n        nuc_pix = np.where(nuc_labels.T.flatten() == label)[0]  #np.where returns a tuple, \n       #so we have to take the first element, which is the array we want\n #       rle = rle_from_mask(nuc_pix)\n        rle = rle_encoding(nuc_pix)\n        #only keep rle's that are clearly not noise, e.g. more than a minimum number of pixels\n        min_pix = 20 #Perhaps we could cross-validate the min_pix value on the training set.\n        num_pix = sum([float(num) for num in rle[1::2]])\n        if num_pix > min_pix:\n            data = data.append({\"ImageId\":image_id, \"EncodedPixels\": \" \".join(rle)}, ignore_index=True)\ndata.to_csv(\"submission.csv\", index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"661fc4657800ebb04ace7941dbd39ea0906700d9"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"d2b8d8b7a61252fde3a87c92fcb5772bbae6ae39"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.3"}},"nbformat":4,"nbformat_minor":1}