{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport skimage.io\n\nfrom tqdm import tqdm\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":11,"outputs":[]},{"metadata":{"_cell_guid":"7bab6cf9-e9fb-4c56-aabe-f3560ce904da","_uuid":"b4bdf956c5c638b223a967947d6962ba2d54f4d5"},"cell_type":"markdown","source":"### Data\n\nThis dataset contains a large number of segmented nuclei images. The images were acquired under a variety of conditions and vary in the cell type, magnification, and imaging modality (brightfield vs. fluorescence). The dataset is designed to challenge an algorithm's ability to generalize across these variations.\n\nEach image is represented by an associated ImageId. Files belonging to an image are contained in a folder with this ImageId. Within this folder are two subfolders:\n\n**images **contains the image file.\n**masks** contains the segmented masks of each nucleus. This folder is only included in the training set. Each mask contains one nucleus. Masks are not allowed to overlap (no pixel belongs to two masks).\n\nYou'll notice the images and masks are in a zip archive. Kaggle says: \nZIP archive files will be uncompressed and their contents available at the root folder when this dataset is used in Kernels.\nYou can see this by listing the archive (Showing the first 5 names here).\nBut what you wanna do is \"walking\" the archive - this does not return a value, it's a generator object. Navigate with __next__() method."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"dataTrainPath=\"../input/stage1_train/\"\ndataTestPath=\"../input/stage1_test/\"\n\nroot,dirs,files=os.walk(dataTrainPath).__next__()","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"c89e8dcc-d99d-4588-8cec-d695087fa916","_uuid":"22a0c5d54c55d2efc1009d50bffb8fbccd7a354f"},"cell_type":"markdown","source":"Your folder names look like this:"},{"metadata":{"_cell_guid":"1ab1ea65-4d7a-409a-8987-8695bbc809e6","_uuid":"1ba2304e38898aa5bb37640cb66e9869ecbee123","trusted":false,"collapsed":true},"cell_type":"code","source":"print(dirs[0:5],\"\\n\")\nprint(\"And there are {} of them\".format(len(dirs)))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"310fc382-84de-4374-ac10-894e4c9fcbea","_uuid":"1a20a114ebe8fe3b4bccbd47e983d61247b07323"},"cell_type":"markdown","source":"Each directory contains the following directories:"},{"metadata":{"_cell_guid":"7ed027b8-46fb-496f-855a-94cdf1a7b3fa","_uuid":"c7ec001033185f0d9be63fe3cf67bb11be4419c3","collapsed":true,"trusted":false},"cell_type":"code","source":"print(os.listdir(root+dirs[0]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9322bfd5-f527-449b-91e9-092a9467e526","_uuid":"067ae925210a898f065dcee125a268c86c943bfd"},"cell_type":"markdown","source":"And what's inside of them? Here's an example:"},{"metadata":{"_cell_guid":"1cfb1ec2-5b45-45ea-ad4e-b5190f21ffbb","_uuid":"943cada46ffef3e8d2d96eeaf0c8913e5842b058","trusted":true},"cell_type":"code","source":"rIm,dIm,fIm=os.walk(root+dirs[0]+\"/images/\").__next__()\nprint(\"This is an example image file name: \",fIm)\nrM,dM,fM=os.walk(root+dirs[0]+\"/masks/\").__next__()\nprint(\"Example mask file name: \\n \",fM)\nprint(\"Well, look at that, multiple masks for one image file. \\nAll .png format.\")","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"1c50f524-2e39-4a72-a9cb-c57ed9a89c13","_uuid":"434c5880ef78652ba7acaf444c2f1fce46068342"},"cell_type":"markdown","source":"Okay, so what do they look like?"},{"metadata":{"_cell_guid":"842ba542-f48e-4401-88bc-7e73a36c6edf","_uuid":"7dbceba8c22404f35c2e0870a8c0989882141adb","trusted":true},"cell_type":"code","source":"print(\"This is an image\")\nim=mpimg.imread(rIm+fIm[0])\nplt.imshow(im)\nplt.show()","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"205fac88-7861-45d3-a7b2-f9a686b95b6b","_uuid":"392ac32e500df41c50fcba7bdc379347dbf09a2e","trusted":true},"cell_type":"code","source":"print(\"These are the corresponding masks\")\nfor mask in fM:\n    iMask=mpimg.imread(rM+mask)\n    plt.imshow(iMask)\n    plt.show()","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"12543d9f-b335-443b-851c-1eea3d5d5036","_uuid":"4dfd599ec78a4d8e54ee8e2d1a8a4b87b1765599"},"cell_type":"markdown","source":"So um...all masks are saved separately :/ Can we view them aggregated?\nApparently, scikitlearn has a useful image processing collection of algorithms: skimage (you'll find it loaded in the beginning)"},{"metadata":{"_cell_guid":"08e11212-dcbf-4ac4-b92d-37ebb79b1ba8","_uuid":"a9f3c0c3cfcbb56d0797555613baf897cdee23dc","trusted":true},"cell_type":"code","source":"maskList=[rM+mask for mask in fM]\nmasks=skimage.io.imread_collection(maskList).concatenate()\ns=masks.shape\nprint(\"Shape of this squashed masks ndarray: \",s) #so number of masks, dimension1, dimension2\n\n# Start with a matrix of size the same as the image (or masks)\nlabels=np.zeros((s[1],s[2]),np.uint16)\n\nfor i in range(s[0]):\n    labels[masks[i]>0]=i+1\n\n# And there you have it\nprint(\"MASKS all together:\")\nplt.imshow(labels)\nplt.show()\n\n# Just to compare again with the original image:\nprint(\"IMAGE:\")\nplt.imshow(im)\nplt.show()","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"ad67f645-fbc7-46d9-8512-8822fa141e10","_uuid":"931144b0b3bd440bc9bbbe14101c7415a50250fa"},"cell_type":"markdown","source":"### How many masks (nuclei) per image?"},{"metadata":{"_cell_guid":"d6516a04-695d-4b82-b99f-40f86504b7e3","_uuid":"0b119c497f2b73d4556656fbf3eb11cc8fb5519b","trusted":true},"cell_type":"code","source":"numMasks=np.zeros(len(dirs),np.uint16)\nfor i,dirName in enumerate(dirs):\n    tempR,tempD,tempF=os.walk(root+dirName+\"/masks/\").__next__()\n    numMasks[i]=len(tempF)\nplt.hist(numMasks,40)\nplt.xlabel(\"Number of masks per image\")\nplt.show()\n\ndfNumMasks=pd.DataFrame(numMasks,columns=[\"NumNuclei\"])\ndfNumMasks.describe()","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"beb0a1f8-5b26-472e-9d5a-e660d32100c6","_uuid":"85ef1930b0c264c8580082ed87de4539359847cc"},"cell_type":"markdown","source":"Okay, all directories contain at least one mask, so no need to worry about that. \nMost of them have below 54, but some have as many as 375! I wanna see those."},{"metadata":{"_cell_guid":"e00b240a-6f1a-4692-8e20-586883a2ed25","_uuid":"9bfab543496f5a166f392ceaddd5f7f0f20cdc65","collapsed":true,"trusted":true},"cell_type":"code","source":"# Let me adapt the above code for merging masks into a function\ndef mergeNPlotMasks(imageRoot,imageDirName):\n    imageFileName=imageRoot+imageDirName+\"/images/\"+imageDirName+\".png\"\n    image=skimage.io.imread(imageFileName)\n    \n    rM,dM,fM=os.walk(imageRoot+imageDirName+\"/masks/\").__next__()\n    maskList=[rM+mask for mask in fM]\n    masks=skimage.io.imread_collection(maskList).concatenate()\n    s=masks.shape\n    \n   # Start with a matrix of size the same as the image (or masks)\n    labels=np.zeros((s[1],s[2]),np.uint16)\n    for i in range(s[0]):\n        labels[masks[i]>0]=i+1\n\n    # And there you have it\n    print(\"MASKS all together:\")\n    plt.imshow(labels)\n    plt.show()\n\n    # Just to compare again with the original image:\n    print(\"IMAGE:\")\n    plt.imshow(image)\n    plt.show()","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"4fbb9fcf-3a77-4a93-9479-19d988954184","_uuid":"d31d637070ffe4dc33398dc861d8410c5515d123","trusted":true},"cell_type":"code","source":"for i,dirName in enumerate(dirs):\n    tempR,tempD,tempF=os.walk(root+dirName+\"/masks/\").__next__()\n    if len(tempF)>300:\n        print(\"#\",i)\n        mergeNPlotMasks(root,dirName)","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"fd62d9b0-6c0a-4268-b6ec-89450f9caa74","_uuid":"388b199902b75f06f42fc606df491fcd7393e423"},"cell_type":"markdown","source":"Look at that nasty little rascal at #198!"},{"metadata":{"_cell_guid":"cd7d51df-4f06-4636-8192-dbf5143a1d87","_uuid":"dea30f75c5737357ae0df02950317769d7e31986"},"cell_type":"markdown","source":"### What about image sizes?"},{"metadata":{"_cell_guid":"7ace3f71-5270-4362-bf4f-b1f8093e55f2","_uuid":"d668435cee40b020e0d2c84ca163040109c499c1","trusted":true},"cell_type":"code","source":"imageAreas=[]\nimageLevels=[]\nfor dirName in dirs:\n    tempImage=skimage.io.imread(root+dirName+\"/images/\"+dirName+\".png\")\n    imageAreas.append(tempImage.shape[0]*tempImage.shape[1])\n    imageLevels.append(tempImage.shape[2])\nplt.hist(imageAreas,20)\nplt.xlabel(\"Image area [pixels]\")\nplt.show()\n\nplt.hist(imageLevels,20)\nplt.xlabel(\"Levels\")\nplt.show()\n\ndfImageSizes=pd.DataFrame(imageAreas,columns=[\"image_area_pxls\"])\ndfImageSizes.describe()\n\n","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"bd290bf0-f673-4e28-ab98-bbd5b5c69664","_uuid":"cd163f7ace12acb422b0380b4e69494c1ee8c7ba"},"cell_type":"markdown","source":"So there are images of substantially different sizes, though most of them are small\n\nAnd all of them have 4 levels. Hm, four? Sanity check? \n\nRGBA, I suppose. Let's see the values of this 4th item."},{"metadata":{"_cell_guid":"2e4915d5-20aa-4e98-9c36-705eaed55caa","_uuid":"40c906ea8924193bc8fbf63c8a0494053e901445","trusted":true},"cell_type":"code","source":"alphaMeans=[]\nfor dirName in tqdm(dirs):\n    tempImage=skimage.io.imread(root+dirName+\"/images/\"+dirName+\".png\")\n    tempAlpha=[tempImage[i,j,3] for i in range(0,tempImage.shape[0])for j in range(0,tempImage.shape[1])]\n    alphaMeans.append(np.mean(tempAlpha))\ndfAlphaMeans=pd.DataFrame(alphaMeans,columns=[\"Alpha levels\"])\ndfAlphaMeans.describe()","execution_count":12,"outputs":[]},{"metadata":{"_cell_guid":"fafa44c7-5098-4204-9326-96b372f1b87b","_uuid":"794fecac5515b7bc46b9677b205b23578654ba6f"},"cell_type":"markdown","source":"Okay, apparently there is no other value of this fourth parameter than 255, so we should not take it into consideration when training a model, as it contains no useful info."}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}