{ "id": 56326, "title": "5th place solution (based only on Mask-RCNN)", "url": "/competitions/data-science-bowl-2018/discussion/56326", "author_name": "Inom Mirzaev", "post_date": "2018-05-08T15:49:39.634000", "votes": 81, "comment_count": 35, "views": 0, "content": "

Model overview

\n\n

Codes now available on GitHub: https://github.com/mirzaevinom/data_science_bowl_2018

\n\n

For this competition, I modified Matterport's implementation of Mask-RCNN deep neural network for object instance segmentation. I adapted the existing model configurations to detect small nuclei in images with varying size and modality. To ensure that the model doesn't overfit, I used an external dataset and relied heavily on image augmentation. Moreover, generated mosaics from train images based on this notebook. To improve generalizability of the model, I split (using stratification) the stage1_train dataset into train and validation sets based on 5 image modalities provided by Allen Goodman. After training the model using Resnet101 as a backbone encoder and Adam as an optimizer, I improved prediction accuracy by test time augmentation and post-processing the masks.

\n\n

Huge thanks to Heng CherKeng!!! I learned a lot from your posts.

\n\n

Training Method(s)

\n\n

Pre-processing

\n\n\n\n

Model and Training

\n\n\n\n

Post-processing

\n\n\n\n

Interesting findings

\n\n\n\n

Unsuccessful approaches tried

\n\n", "messages": [ { "id": 325591, "postDate": "2018-05-08T15:49:39.633Z", "content": "

Model overview

\n\n

Codes now available on GitHub: https://github.com/mirzaevinom/data_science_bowl_2018

\n\n

For this competition, I modified Matterport's implementation of Mask-RCNN deep neural network for object instance segmentation. I adapted the existing model configurations to detect small nuclei in images with varying size and modality. To ensure that the model doesn't overfit, I used an external dataset and relied heavily on image augmentation. Moreover, generated mosaics from train images based on this notebook. To improve generalizability of the model, I split (using stratification) the stage1_train dataset into train and validation sets based on 5 image modalities provided by Allen Goodman. After training the model using Resnet101 as a backbone encoder and Adam as an optimizer, I improved prediction accuracy by test time augmentation and post-processing the masks.

\n\n

Huge thanks to Heng CherKeng!!! I learned a lot from your posts.

\n\n

Training Method(s)

\n\n

Pre-processing

\n\n\n\n

Model and Training

\n\n\n\n

Post-processing

\n\n\n\n

Interesting findings

\n\n\n\n

Unsuccessful approaches tried

\n\n", "rawMarkdown": "\n## Model overview ##\n\nCodes now available on GitHub: [https://github.com/mirzaevinom/data_science_bowl_2018](https://github.com/mirzaevinom/data_science_bowl_2018)\n\nFor this competition, I modified [Matterport's][1] implementation of [Mask-RCNN][3] deep neural network for object instance segmentation. I adapted the existing model configurations to detect small nuclei in images with varying size and modality. To ensure that the model doesn't overfit, I used an [external dataset][4] and relied heavily on image augmentation. Moreover, generated mosaics from train images based on [this notebook][8]. To improve generalizability of the model, I split (using stratification) the `stage1_train` dataset into train and validation sets based on 5 image modalities provided by [Allen Goodman][5]. After training the model using Resnet101 as a backbone encoder and Adam as an optimizer, I improved prediction accuracy by test time augmentation and post-processing the masks.\n\n*__Huge thanks to Heng CherKeng!!! I learned a lot from your posts.__*\n\n## Training Method(s) ##\n\n### Pre-processing\n- I noticed some issues with the provided masks. Therefore, used the annotations and mask provided by [Konstantin Lopuhin][15] in [data quality issues][14] thread.\n- Removed the alpha channel from the images.\n- Filled holes in the masks\n- Splitted (using stratification) the `stage1_train` dataset into 90% train and 10% validation sets based on 5 image modalities provided by [Allen Goodman][5].\n- Used an [external dataset][4] provided in the forum. Divided the images and the masks into 4 pieces due their large sizes. External dataset [download links][19].\n- Generated mosaics from train images based on [Emil's][8] notebook.\n\n\n### Model and Training\n* Modified [Matterport's][1] implementation of [Mask-RCNN][3] deep neural network for object instance segmentation.\n* Tuned hyperparameters to detect small nuclei from the images. (I found [this tutorial][7] very useful for understanding the model hyperparameters)\n + Original Matterport implementation was validating only on one image so fixed this [validation issue][20].\n + Reduced RPN (region proposal network) anchor sizes since the nuclei are mostly small.\n + Increased number of anchors to be used since the nuclei are small and can be found anywhere on an image.\n + Increased maximum number of predicted objects since an image can contain 300 or more nuclei.\n + Increased `POST_NMS_ROIS_TRAINING` to get more region proposals during training.\n + Added extra parameter `DETECTION_MASK_THRESHOLD` to model configuration. Default was hardcoded in the model as 0.5 but setting it to 0.35 helped in detection of small nuclei boundaries.\n + Resized images and masks to 512x512\n* Relied heavily on image augmentation due to small training set:\n - Random horizontal or vertical flips\n - Random 90 or -90 degrees rotation\n - [Random rotations][18] in the range of (-15, 15) degrees\n - [Random cropping][18] of bigger images and masks to 256x256x3.\n - [Random scaling][18] of image and mask scaling in the range (0.5, 2.0)\n\n* Used Resnet101 architecture as a backbone encoder but initialized the first 50 layers of the model with pre-trained Resnet50 weights from [ImageNet competition][16].\n* Trained the model with [Adam][17] optimizer for 75 epochs:\n - 25 epochs with learning rate 1e-4\n - 25 epochs with learning rate 1e-5\n - 25 epochs with learning rate 1e-6\n* Did not retrain the model with stage1 test data during stage 2 as I was afraid of overfitting on black and white images.\n\n\n### Post-processing\n\n- Combined predictions on actual image and horizontally flipped image: took unions of masks with maximum overlap and removed false positive masks with small overlap.\n- Due to configured RPN anchor sizes, the model predicts small nuclei very well. However, it struggles at predicting large nuclei. Therefore, if a model predicts no masks for an image then I scale down the image and predict once again.\n- Removed overlaps between predicted nuclei based on their objectness score. In other words, removed intersections from the masks with lower scores.\n - If this intersection removal results in multiple objects in that mask, then removing all the small pieces.\n- Closing small holes inside the masks using morphological operations (dilation followed by erosion).\n\n## Interesting findings ##\n\n- Mask-RCNN model overfits easily without image augmentation.\n- Removing false positive mask predictions improves the overall score significantly.\n- Since images are on different scales, predicting masks on scaled images helps with the model generalizability.\n- Dilating and then eroding individual masks helped me achieve slightly better result.\n- Matterport's original implementation was only [validating on only one image][6]. Fixing this issue made the training process reproducible.\n- I found that the model reaches a local minima faster when trained using Adam optimizer compared to default SGD optimizer.\n\n## Unsuccessful approaches tried\n- Trained the model with Dice Coefficient Loss instead of default binary cross-entropy loss for the masks heads. Although got comparable results couldn't beat prediction accuracy of binary cross-entropy on my validation set.\n- Trained with random Gaussian and Poisson (or shot) noise for image augmentation. It actually hurt overall model performance.\n- Tried ensembling actual image predictions with horizontal and vertical flip predictions. Used non-maximum suppression for removing overlaps. Did not improve prediction accuracy on the validation set.\n- Trained end-to-end without initializing with pre-trained ImageNet weights. Mostly got to Mean IoU score of 0.35 on stage1 test set.\n- Trained on preprocessed images with adaptive histogram equalization (CLAHE). The model performed way worse.\n\n[1]: https://github.com/matterport/Mask_RCNN\n[2]: https://www.osc.edu/\n[3]: https://arxiv.org/abs/1703.06870\n[4]: https://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations\n[5]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48130\n[6]: https://github.com/matterport/Mask_RCNN/issues/89\n[7]: https://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46\n[8]: https://www.kaggle.com/bonlime/train-test-image-mosaic\n[10]: https://nucleisegmentationbenchmark.weebly.com/dataset.html\n[11]: http://mbi.osu.edu\n[12]: http://www.lerner.ccf.org/thor/scott/lab/\n[13]: https://promise12.grand-challenge.org\n[14]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572\n[15]: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes\n[16]: https://github.com/fchollet/deep-learning-models/releases/\n[17]: https://arxiv.org/abs/1412.6980\n[18]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/49692\n[19]: https://nucleisegmentationbenchmark.weebly.com/dataset.html\n[20]: https://github.com/matterport/Mask_RCNN/issues/89", "votes": 81 }, { "id": 1383586, "postDate": "2021-07-11T04:21:55.653Z", "content": "

Hello,i am Chinese and i'm a new deep learninger.I want to know how to relize the model in kaggle,I want to run in my own computer but it's too rubblish.I cannot do it.Can you help me that how to realize this model?Think you.My english is not very good,if someplace offend you,please forgive me.

", "rawMarkdown": "Hello,i am Chinese and i'm a new deep learninger.I want to know how to relize the model in kaggle,I want to run in my own computer but it's too rubblish.I cannot do it.Can you help me that how to realize this model?Think you.My english is not very good,if someplace offend you,please forgive me.", "votes": 1, "replies": [ { "id": 1396873, "postDate": "2021-07-22T14:55:22.760Z", "content": "

你好,
\n可以用中文敘述你的問題嗎?
\n或許我可以幫你

", "rawMarkdown": "你好,\n可以用中文敘述你的問題嗎?\n或許我可以幫你" } ] }, { "id": 354678, "postDate": "2018-07-10T03:45:53.353Z", "content": "

Hey! Thank you for this amazing topic and for the effort you put on it.\nI'd like to apply a mas r-cnn for nuclei segmentation but using Whole Slide Images. I'm thinking about patch sampling my .svs files and then use a Multi Instance Learning method. \nDo you it would be possible to use your model for something like that?\nKind Regards

", "rawMarkdown": "Hey! Thank you for this amazing topic and for the effort you put on it.\nI'd like to apply a mas r-cnn for nuclei segmentation but using Whole Slide Images. I'm thinking about patch sampling my .svs files and then use a Multi Instance Learning method. \nDo you it would be possible to use your model for something like that?\nKind Regards", "votes": 1, "replies": [ { "id": 354981, "postDate": "2018-07-10T16:13:38.007Z", "content": "

Hi! I think it is definitely possible. I would even split large .svs files into smaller ones to get more training data.

", "rawMarkdown": "Hi! I think it is definitely possible. I would even split large .svs files into smaller ones to get more training data. ", "votes": 1 }, { "id": 355009, "postDate": "2018-07-10T16:58:32.433Z", "content": "

Well, I'll give it a try then. \nThis would be my MSc dissertation so I'm really hyped. I'll test your method and let you know the results. \nDo you advise any other method btw? Like UNet or something?

", "rawMarkdown": "Well, I'll give it a try then. \nThis would be my MSc dissertation so I'm really hyped. I'll test your method and let you know the results. \nDo you advise any other method btw? Like UNet or something?" }, { "id": 355010, "postDate": "2018-07-10T17:04:17.213Z", "content": "

I think U-net based solution could work too. A lot of participants have used U-net for this competition. For instance, see this post. Good luck with your MSc dissertation!

", "rawMarkdown": "I think U-net based solution could work too. A lot of participants have used U-net for this competition. For instance, see [this post][1]. Good luck with your MSc dissertation!\n\n\n [1]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/54426" } ] }, { "id": 857767, "postDate": "2020-05-22T22:43:07.430Z", "content": "

Hi. Thank you for your work. In your code, how can you print RPN region proposal bounding boxes?

", "rawMarkdown": "Hi. Thank you for your work. In your code, how can you print RPN region proposal bounding boxes?" }, { "id": 593204, "postDate": "2019-08-06T09:54:44.160Z", "content": "

thanks for this topic.

\n\n

But i want to inquire is i can implement this model to detect object when it has similar histogram with background.

", "rawMarkdown": "thanks for this topic.\n\nBut i want to inquire is i can implement this model to detect object when it has similar histogram with background." }, { "id": 402538, "postDate": "2018-10-11T20:57:46.100Z", "content": "

Hey, I wanted to check how you did end-to-end training without using the pre-trained imagenet weights. Currently, the code uses imagenet if no other model is specified.

", "rawMarkdown": "Hey, I wanted to check how you did end-to-end training without using the pre-trained imagenet weights. Currently, the code uses imagenet if no other model is specified. " }, { "id": 377711, "postDate": "2018-08-29T16:23:38.310Z", "content": "

Congrats and thanks a lot for the sharing. I would like to ask, what have you done with \"Removing false positive mask predictions improves the overall score significantly.\"?

", "rawMarkdown": "Congrats and thanks a lot for the sharing. I would like to ask, what have you done with \"Removing false positive mask predictions improves the overall score significantly.\"?", "replies": [ { "id": 377965, "postDate": "2018-08-30T00:25:32.770Z", "content": "

Thanks, Bing! It's the first bullet of post-processing section.

\n\n", "rawMarkdown": "Thanks, Bing! It's the first bullet of post-processing section. \n\n - Combined predictions on actual image and horizontally flipped image: took unions of masks with maximum overlap and removed false positive masks with small overlap.", "votes": 1 } ] }, { "id": 357697, "postDate": "2018-07-16T16:25:29.113Z", "content": "

@lvguofeng and @JubaerHossain. Sorry for the late response. Glad that you guys have sorted out the issue. The .csv files are now available on the same repository.

", "rawMarkdown": "@lvguofeng and @JubaerHossain. Sorry for the late response. Glad that you guys have sorted out the issue. The .csv files are now available on the same repository.", "replies": [ { "id": 378633, "postDate": "2018-08-30T12:36:47.220Z", "content": "

Hi!

\n\n

I'd really like to try out your solution but unfortunately the problem with the non-existing '.h5' file is still open for me (the one @lvguofeng pointed out above). I really want to execute every single step of your instructions, therefore I'd really appreciate your help on how to fix this problem.

\n\n

Thanks!

", "rawMarkdown": "Hi!\n\nI'd really like to try out your solution but unfortunately the problem with the non-existing '.h5' file is still open for me (the one @lvguofeng pointed out above). I really want to execute every single step of your instructions, therefore I'd really appreciate your help on how to fix this problem. \n\nThanks!" }, { "id": 379185, "postDate": "2018-08-31T01:32:21.387Z", "content": "

Hi,\nIn augment_preprocess.py file, if you only have the original data, the original data .h5 file is not exist, you should use the function 'data_to_array()' to change the data to .h format first（the reason change to .h5 format is save masks as h5 files to make trainig faster）\nso you should change the sequence in main() function, run make_n_save_mosaic() after data_to_array()\nhope my suggestion will help you

", "rawMarkdown": "Hi,\nIn augment_preprocess.py file, if you only have the original data, the original data .h5 file is not exist, you should use the function 'data_to_array()' to change the data to .h format first（the reason change to .h5 format is save masks as h5 files to make trainig faster）\nso you should change the sequence in main() function, run make_n_save_mosaic() after data_to_array()\nhope my suggestion will help you" }, { "id": 379434, "postDate": "2018-08-31T11:05:24.107Z", "content": "

Thanks for your reply! I think you even have to put data_to_array() before the preprocess_external_data() statement, otherwise the data_to_array() method will fail due to some items that will be created in preprocess_external_data() in the stage1_train folder which the data_to_array() cannot handle. Other than that, it works as you suggested!

", "rawMarkdown": "Thanks for your reply! I think you even have to put data_to_array() before the preprocess_external_data() statement, otherwise the data_to_array() method will fail due to some items that will be created in preprocess_external_data() in the stage1_train folder which the data_to_array() cannot handle. Other than that, it works as you suggested! " }, { "id": 379502, "postDate": "2018-08-31T13:25:15.537Z", "content": "

Agh, I see what you guys are saying. Yes, data_to_array function should be executed before functions preprocess_external_data and make_n_save_mosaic. I pushed an update to github repo. Thanks for reporting that issue.

", "rawMarkdown": "Agh, I see what you guys are saying. Yes, `data_to_array` function should be executed before functions `preprocess_external_data` and `make_n_save_mosaic`. I pushed an update to github repo. Thanks for reporting that issue." }, { "id": 381250, "postDate": "2018-09-04T11:13:42.727Z", "content": "

Hi Inom!

\n\n

I got one further question regarding your solution and code. Is it possible to provide images for prediction that are much bigger than the images of the DSB dataset? Let's say like 4000x4000 pixels?\nI tried your model on other WSI (Whole Slide Images) that are h&e stained (thus in raw-format they are colored) but have a larger size. Unfortunately the model doesn't predict anything and saves the prediction as a very downscaled image (small in size and very pixelated). A grayscale version of the same WSI didn't help either.

\n\n

Thanks for your reply!

", "rawMarkdown": "Hi Inom!\n\nI got one further question regarding your solution and code. Is it possible to provide images for prediction that are much bigger than the images of the DSB dataset? Let's say like 4000x4000 pixels?\nI tried your model on other WSI (Whole Slide Images) that are h&e stained (thus in raw-format they are colored) but have a larger size. Unfortunately the model doesn't predict anything and saves the prediction as a very downscaled image (small in size and very pixelated). A grayscale version of the same WSI didn't help either. \n\nThanks for your reply!" }, { "id": 382978, "postDate": "2018-09-07T13:24:23.010Z", "content": "

Hi Chrisk,

\n\n

I would suggest dividing up the images into smaller pieces, make predictions for each piece and then patch the predictions into one piece. Easiest you can divide the image into 4 pieces and do the predictions. Hope it helps!

", "rawMarkdown": "Hi Chrisk,\n\nI would suggest dividing up the images into smaller pieces, make predictions for each piece and then patch the predictions into one piece. Easiest you can divide the image into 4 pieces and do the predictions. Hope it helps!\n\n" }, { "id": 383006, "postDate": "2018-09-07T14:16:18.723Z", "content": "

Hi Inom Mirzaev

\n\n

i have the same question like Chrisk,.\nWhen the model predict the image which is too big, its prediction result is very poor.\nFor example, in the dataset of stage2_test_final, this image shape is 1024*1360.\n\"enter\nfirst, i think the model cant predict the big image, but after i split the image into blocks of 256*256, the result was well. i have the two plan to merge the separate blocks.\nfirst is add all blocks into one piece and calculate their Connected domain， but this plan will identify adjacent different nuclei into one.\n\"enter\nsecond is keep each blocks mask , but this plan will separate the nucleus in the block edge.\n\"enter

\n\n

Do you have any great plan?

\n\n

Thanks for your reply！

", "rawMarkdown": "Hi Inom Mirzaev\n\ni have the same question like Chrisk,.\nWhen the model predict the image which is too big, its prediction result is very poor.\nFor example, in the dataset of stage2_test_final, this image shape is 1024*1360.\n![enter image description here][1]\nfirst, i think the model cant predict the big image, but after i split the image into blocks of 256*256, the result was well. i have the two plan to merge the separate blocks.\nfirst is add all blocks into one piece and calculate their Connected domain， but this plan will identify adjacent different nuclei into one.\n![enter image description here][2]\nsecond is keep each blocks mask , but this plan will separate the nucleus in the block edge.\n![enter image description here][3]\n\nDo you have any great plan?\n\nThanks for your reply！\n [1]: https://note.youdao.com/yws/public/resource/7e98eafe0f1de9fe57789e62479e70f2/xmlnote/WEBRESOURCE6a1d2ab2e8c8042348901cd442a611e4/5762\n [2]: https://note.youdao.com/yws/public/resource/7e98eafe0f1de9fe57789e62479e70f2/xmlnote/WEBRESOURCEa363eb98744b2f92d8b7394fc288ec4d/5764\n [3]: https://note.youdao.com/yws/public/resource/7e98eafe0f1de9fe57789e62479e70f2/xmlnote/WEBRESOURCE3d36c0ad75db4810e8a9d29d6713e512/5766" }, { "id": 383075, "postDate": "2018-09-07T16:23:56.790Z", "content": "

That sounds like a solid plan. For calculating connected domains and merging, I would suggest looking at making mosaics functions in the preprocessing module.

", "rawMarkdown": "That sounds like a solid plan. For calculating connected domains and merging, I would suggest looking at making mosaics functions in the preprocessing module." } ] }, { "id": 357116, "postDate": "2018-07-15T10:20:53.473Z", "content": "

@Inom Hi, Great thread! I organized my folders as you mentioned in github. But there is no file named ../data/share_train_df.csv in your mention. So, when I run augment_preprocess.py, it show error; because you have declared df = pd.read_csv('../data/share_train_df.csv') in save_n_mosaic function. Can your please help me in this regard?

", "rawMarkdown": "@Inom Hi, Great thread! I organized my folders as you mentioned in github. But there is no file named ../data/share_train_df.csv in your mention. So, when I run augment_preprocess.py, it show error; because you have declared df = pd.read_csv('../data/share_train_df.csv') in save_n_mosaic function. Can your please help me in this regard?", "replies": [ { "id": 357138, "postDate": "2018-07-15T11:21:48.217Z", "content": "

https://www.kaggle.com/c/data-science-bowl-2018/discussion/51184\nhere，you can download your need two .csv file

", "rawMarkdown": "https://www.kaggle.com/c/data-science-bowl-2018/discussion/51184\nhere，you can download your need two .csv file" }, { "id": 357160, "postDate": "2018-07-15T12:06:47.503Z", "content": "

Thank you very much! I have another problem, I think you also might have faced it. There is no file named ../data/classes.csv in train.py How did you solve it?\nThanks again for your quick reply :)

", "rawMarkdown": "Thank you very much! I have another problem, I think you also might have faced it. There is no file named ../data/classes.csv in train.py How did you solve it?\nThanks again for your quick reply :)" }, { "id": 357176, "postDate": "2018-07-15T12:46:08.887Z", "content": "

here\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/273531/8366/classes.csv

", "rawMarkdown": "here\nhttps://kaggle2.blob.core.windows.net/forum-message-attachments/273531/8366/classes.csv" }, { "id": 357182, "postDate": "2018-07-15T12:51:47.100Z", "content": "

but you still will have the same problem as me, without a .h5 file

", "rawMarkdown": "but you still will have the same problem as me, without a .h5 file" }, { "id": 357184, "postDate": "2018-07-15T12:56:47.010Z", "content": "

Can you please tell me how did you run the complete code? Did you follow any specific kernel or something?

", "rawMarkdown": "Can you please tell me how did you run the complete code? Did you follow any specific kernel or something?" }, { "id": 357205, "postDate": "2018-07-15T13:28:27.573Z", "content": "

i skip this step(run augment_preprocess.py), and i can run the complete code follw the blogger Instruction, and i change the config.py IMAGE_MIN_DIM/IMAGE_MAX_DIM = 256 (before is 512) ,i think 512 is too big, i havent enough memory

", "rawMarkdown": "i skip this step(run augment_preprocess.py), and i can run the complete code follw the blogger Instruction, and i change the config.py IMAGE_MIN_DIM/IMAGE_MAX_DIM = 256 (before is 512) ,i think 512 is too big, i havent enough memory" }, { "id": 357308, "postDate": "2018-07-15T17:56:19.240Z", "content": "

Thanks man!

", "rawMarkdown": "Thanks man! " } ] }, { "id": 355899, "postDate": "2018-07-12T13:55:59.790Z", "content": "

Thank you for making your code public and let me learn.\nbut i download your code and follow your Instruction，i met some issues\nwhen i run

\n\n

python augment_preprocess.py

\n\n

i met some issues:

\n\n

Traceback (most recent call last):\nFile \"augment_preprocess.py\", line 473, in \nmake_n_save_mosaic()\nFile \"augment_preprocess.py\", line 361, in make_n_save_mosaic\nwith h5py.File(path, \"r\") as hf:\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 312, in init\nfid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 142, in make_fid\nfid = h5f.open(name, flags, fapl=fapl)\nFile \"h5py/_objects.pyx\", line 54, in h5py._objects.with_phil.wrapper\nFile \"h5py/_objects.pyx\", line 55, in h5py._objects.with_phil.wrapper\nFile \"h5py/h5f.pyx\", line 78, in h5py.h5f.open\nOSError: Unable to open file (unable to open file: name = '../data/stage1_train/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d/masks/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

\n\n

i want know what is this .h5 file，and how can i fix this problem to run your code

", "rawMarkdown": "Thank you for making your code public and let me learn.\nbut i download your code and follow your Instruction，i met some issues\nwhen i run\n\n`python augment_preprocess.py`\n\ni met some issues:\n\nTraceback (most recent call last):\nFile \"augment_preprocess.py\", line 473, in \nmake_n_save_mosaic()\nFile \"augment_preprocess.py\", line 361, in make_n_save_mosaic\nwith h5py.File(path, \"r\") as hf:\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 312, in init\nfid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 142, in make_fid\nfid = h5f.open(name, flags, fapl=fapl)\nFile \"h5py/_objects.pyx\", line 54, in h5py._objects.with_phil.wrapper\nFile \"h5py/_objects.pyx\", line 55, in h5py._objects.with_phil.wrapper\nFile \"h5py/h5f.pyx\", line 78, in h5py.h5f.open\nOSError: Unable to open file (unable to open file: name = '../data/stage1_train/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d/masks/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)\n\ni want know what is this .h5 file，and how can i fix this problem to run your code" }, { "id": 349132, "postDate": "2018-06-27T21:02:47.550Z", "content": "

Codes with some comments are now available on GitHub: https://github.com/mirzaevinom/data_science_bowl_2018. Hope you find it helpful!

", "rawMarkdown": "Codes with some comments are now available on GitHub: [https://github.com/mirzaevinom/data_science_bowl_2018][1]. Hope you find it helpful!\n\n[1]: https://github.com/mirzaevinom/data_science_bowl_2018" }, { "id": 326333, "postDate": "2018-05-09T15:21:28.663Z", "content": "

The link to Allen Goodman does not work. (404)

", "rawMarkdown": "The link to Allen Goodman does not work. (404)", "replies": [ { "id": 326339, "postDate": "2018-05-09T15:27:43.317Z", "content": "

Fixed! Thanks for pointing out.

", "rawMarkdown": "Fixed! Thanks for pointing out." } ] }, { "id": 326255, "postDate": "2018-05-09T13:40:28.943Z", "content": "

Thanks for the nice details and congrats on a strong finish!

", "rawMarkdown": "Thanks for the nice details and congrats on a strong finish!", "replies": [ { "id": 326283, "postDate": "2018-05-09T14:05:26.137Z", "content": "

Thanks, Russ! Congrats on the 4th place too!

", "rawMarkdown": "Thanks, Russ! Congrats on the 4th place too!" } ] }, { "id": 489778, "postDate": "2019-03-14T04:20:54.087Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1383586, "author_name": "Cdong121", "author_url": "", "post_date": "2021-07-11T04:21:55.653000", "content": "

Hello,i am Chinese and i'm a new deep learninger.I want to know how to relize the model in kaggle,I want to run in my own computer but it's too rubblish.I cannot do it.Can you help me that how to realize this model?Think you.My english is not very good,if someplace offend you,please forgive me.

", "votes": 1, "replies": [ { "id": 1396873, "author_name": "Bruce Chen", "author_url": "", "post_date": "2021-07-22T14:55:22.760000", "content": "

你好,
\n可以用中文敘述你的問題嗎?
\n或許我可以幫你

", "votes": 0, "replies": [] } ] }, { "id": 354678, "author_name": "luistelmocosta", "author_url": "", "post_date": "2018-07-10T03:45:53.353000", "content": "

Hey! Thank you for this amazing topic and for the effort you put on it.\nI'd like to apply a mas r-cnn for nuclei segmentation but using Whole Slide Images. I'm thinking about patch sampling my .svs files and then use a Multi Instance Learning method. \nDo you it would be possible to use your model for something like that?\nKind Regards

", "votes": 1, "replies": [ { "id": 354981, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-07-10T16:13:38.007000", "content": "

Hi! I think it is definitely possible. I would even split large .svs files into smaller ones to get more training data.

", "votes": 1, "replies": [] }, { "id": 355009, "author_name": "luistelmocosta", "author_url": "", "post_date": "2018-07-10T16:58:32.433000", "content": "

Well, I'll give it a try then. \nThis would be my MSc dissertation so I'm really hyped. I'll test your method and let you know the results. \nDo you advise any other method btw? Like UNet or something?

", "votes": 0, "replies": [] }, { "id": 355010, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-07-10T17:04:17.213000", "content": "

I think U-net based solution could work too. A lot of participants have used U-net for this competition. For instance, see this post. Good luck with your MSc dissertation!

", "votes": 0, "replies": [] } ] }, { "id": 857767, "author_name": "uzunper", "author_url": "", "post_date": "2020-05-22T22:43:07.430000", "content": "

Hi. Thank you for your work. In your code, how can you print RPN region proposal bounding boxes?

", "votes": 0, "replies": [] }, { "id": 593204, "author_name": "mahmoud", "author_url": "", "post_date": "2019-08-06T09:54:44.160000", "content": "

thanks for this topic.

\n\n

But i want to inquire is i can implement this model to detect object when it has similar histogram with background.

", "votes": 0, "replies": [] }, { "id": 402538, "author_name": "codergli", "author_url": "", "post_date": "2018-10-11T20:57:46.100000", "content": "

Hey, I wanted to check how you did end-to-end training without using the pre-trained imagenet weights. Currently, the code uses imagenet if no other model is specified.

", "votes": 0, "replies": [] }, { "id": 377711, "author_name": "Bing Liu", "author_url": "", "post_date": "2018-08-29T16:23:38.310000", "content": "

Congrats and thanks a lot for the sharing. I would like to ask, what have you done with \"Removing false positive mask predictions improves the overall score significantly.\"?

", "votes": 0, "replies": [ { "id": 377965, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-08-30T00:25:32.770000", "content": "

Thanks, Bing! It's the first bullet of post-processing section.

\n\n", "votes": 1, "replies": [] } ] }, { "id": 357697, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-07-16T16:25:29.113000", "content": "

@lvguofeng and @JubaerHossain. Sorry for the late response. Glad that you guys have sorted out the issue. The .csv files are now available on the same repository.

", "votes": 0, "replies": [ { "id": 378633, "author_name": "chrisk", "author_url": "", "post_date": "2018-08-30T12:36:47.220000", "content": "

Hi!

\n\n

I'd really like to try out your solution but unfortunately the problem with the non-existing '.h5' file is still open for me (the one @lvguofeng pointed out above). I really want to execute every single step of your instructions, therefore I'd really appreciate your help on how to fix this problem.

\n\n

Thanks!

", "votes": 0, "replies": [] }, { "id": 379185, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-08-31T01:32:21.387000", "content": "

Hi,\nIn augment_preprocess.py file, if you only have the original data, the original data .h5 file is not exist, you should use the function 'data_to_array()' to change the data to .h format first（the reason change to .h5 format is save masks as h5 files to make trainig faster）\nso you should change the sequence in main() function, run make_n_save_mosaic() after data_to_array()\nhope my suggestion will help you

", "votes": 0, "replies": [] }, { "id": 379434, "author_name": "chrisk", "author_url": "", "post_date": "2018-08-31T11:05:24.107000", "content": "

Thanks for your reply! I think you even have to put data_to_array() before the preprocess_external_data() statement, otherwise the data_to_array() method will fail due to some items that will be created in preprocess_external_data() in the stage1_train folder which the data_to_array() cannot handle. Other than that, it works as you suggested!

", "votes": 0, "replies": [] }, { "id": 379502, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-08-31T13:25:15.537000", "content": "

Agh, I see what you guys are saying. Yes, data_to_array function should be executed before functions preprocess_external_data and make_n_save_mosaic. I pushed an update to github repo. Thanks for reporting that issue.

", "votes": 0, "replies": [] }, { "id": 381250, "author_name": "chrisk", "author_url": "", "post_date": "2018-09-04T11:13:42.727000", "content": "

Hi Inom!

\n\n

I got one further question regarding your solution and code. Is it possible to provide images for prediction that are much bigger than the images of the DSB dataset? Let's say like 4000x4000 pixels?\nI tried your model on other WSI (Whole Slide Images) that are h&e stained (thus in raw-format they are colored) but have a larger size. Unfortunately the model doesn't predict anything and saves the prediction as a very downscaled image (small in size and very pixelated). A grayscale version of the same WSI didn't help either.

\n\n

Thanks for your reply!

", "votes": 0, "replies": [] }, { "id": 382978, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-09-07T13:24:23.010000", "content": "

Hi Chrisk,

\n\n

I would suggest dividing up the images into smaller pieces, make predictions for each piece and then patch the predictions into one piece. Easiest you can divide the image into 4 pieces and do the predictions. Hope it helps!

", "votes": 0, "replies": [] }, { "id": 383006, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-09-07T14:16:18.723000", "content": "

Hi Inom Mirzaev

\n\n

i have the same question like Chrisk,.\nWhen the model predict the image which is too big, its prediction result is very poor.\nFor example, in the dataset of stage2_test_final, this image shape is 1024*1360.\n\"enter\nfirst, i think the model cant predict the big image, but after i split the image into blocks of 256*256, the result was well. i have the two plan to merge the separate blocks.\nfirst is add all blocks into one piece and calculate their Connected domain， but this plan will identify adjacent different nuclei into one.\n\"enter\nsecond is keep each blocks mask , but this plan will separate the nucleus in the block edge.\n\"enter

\n\n

Do you have any great plan?

\n\n

Thanks for your reply！

", "votes": 0, "replies": [] }, { "id": 383075, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-09-07T16:23:56.790000", "content": "

That sounds like a solid plan. For calculating connected domains and merging, I would suggest looking at making mosaics functions in the preprocessing module.

", "votes": 0, "replies": [] } ] }, { "id": 357116, "author_name": "JubaerHossain", "author_url": "", "post_date": "2018-07-15T10:20:53.473000", "content": "

@Inom Hi, Great thread! I organized my folders as you mentioned in github. But there is no file named ../data/share_train_df.csv in your mention. So, when I run augment_preprocess.py, it show error; because you have declared df = pd.read_csv('../data/share_train_df.csv') in save_n_mosaic function. Can your please help me in this regard?

", "votes": 0, "replies": [ { "id": 357138, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-07-15T11:21:48.217000", "content": "

https://www.kaggle.com/c/data-science-bowl-2018/discussion/51184\nhere，you can download your need two .csv file

", "votes": 0, "replies": [] }, { "id": 357160, "author_name": "JubaerHossain", "author_url": "", "post_date": "2018-07-15T12:06:47.503000", "content": "

Thank you very much! I have another problem, I think you also might have faced it. There is no file named ../data/classes.csv in train.py How did you solve it?\nThanks again for your quick reply :)

", "votes": 0, "replies": [] }, { "id": 357176, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-07-15T12:46:08.887000", "content": "

here\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/273531/8366/classes.csv

", "votes": 0, "replies": [] }, { "id": 357182, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-07-15T12:51:47.100000", "content": "

but you still will have the same problem as me, without a .h5 file

", "votes": 0, "replies": [] }, { "id": 357184, "author_name": "JubaerHossain", "author_url": "", "post_date": "2018-07-15T12:56:47.010000", "content": "

Can you please tell me how did you run the complete code? Did you follow any specific kernel or something?

", "votes": 0, "replies": [] }, { "id": 357205, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-07-15T13:28:27.573000", "content": "

i skip this step(run augment_preprocess.py), and i can run the complete code follw the blogger Instruction, and i change the config.py IMAGE_MIN_DIM/IMAGE_MAX_DIM = 256 (before is 512) ,i think 512 is too big, i havent enough memory

", "votes": 0, "replies": [] }, { "id": 357308, "author_name": "JubaerHossain", "author_url": "", "post_date": "2018-07-15T17:56:19.240000", "content": "

Thanks man!

", "votes": 0, "replies": [] } ] }, { "id": 355899, "author_name": "lvguofeng", "author_url": "", "post_date": "2018-07-12T13:55:59.790000", "content": "

Thank you for making your code public and let me learn.\nbut i download your code and follow your Instruction，i met some issues\nwhen i run

\n\n

python augment_preprocess.py

\n\n

i met some issues:

\n\n

Traceback (most recent call last):\nFile \"augment_preprocess.py\", line 473, in \nmake_n_save_mosaic()\nFile \"augment_preprocess.py\", line 361, in make_n_save_mosaic\nwith h5py.File(path, \"r\") as hf:\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 312, in init\nfid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 142, in make_fid\nfid = h5f.open(name, flags, fapl=fapl)\nFile \"h5py/_objects.pyx\", line 54, in h5py._objects.with_phil.wrapper\nFile \"h5py/_objects.pyx\", line 55, in h5py._objects.with_phil.wrapper\nFile \"h5py/h5f.pyx\", line 78, in h5py.h5f.open\nOSError: Unable to open file (unable to open file: name = '../data/stage1_train/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d/masks/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

\n\n

i want know what is this .h5 file，and how can i fix this problem to run your code

", "votes": 0, "replies": [] }, { "id": 349132, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-06-27T21:02:47.550000", "content": "

Codes with some comments are now available on GitHub: https://github.com/mirzaevinom/data_science_bowl_2018. Hope you find it helpful!

", "votes": 0, "replies": [] }, { "id": 326333, "author_name": "Bruce Chen", "author_url": "", "post_date": "2018-05-09T15:21:28.663000", "content": "

The link to Allen Goodman does not work. (404)

", "votes": 0, "replies": [ { "id": 326339, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-05-09T15:27:43.317000", "content": "

Fixed! Thanks for pointing out.

", "votes": 0, "replies": [] } ] }, { "id": 326255, "author_name": "Russ Wolfinger", "author_url": "", "post_date": "2018-05-09T13:40:28.943000", "content": "

Thanks for the nice details and congrats on a strong finish!

", "votes": 0, "replies": [ { "id": 326283, "author_name": "Inom Mirzaev", "author_url": "", "post_date": "2018-05-09T14:05:26.137000", "content": "

Thanks, Russ! Congrats on the 4th place too!

", "votes": 0, "replies": [] } ] }, { "id": 489778, "author_name": "", "author_url": "", "post_date": "2019-03-14T04:20:54.087000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "325591": "\n## Model overview ##\n\nCodes now available on GitHub: [https://github.com/mirzaevinom/data_science_bowl_2018](https://github.com/mirzaevinom/data_science_bowl_2018)\n\nFor this competition, I modified [Matterport's][1] implementation of [Mask-RCNN][3] deep neural network for object instance segmentation. I adapted the existing model configurations to detect small nuclei in images with varying size and modality. To ensure that the model doesn't overfit, I used an [external dataset][4] and relied heavily on image augmentation. Moreover, generated mosaics from train images based on [this notebook][8]. To improve generalizability of the model, I split (using stratification) the `stage1_train` dataset into train and validation sets based on 5 image modalities provided by [Allen Goodman][5]. After training the model using Resnet101 as a backbone encoder and Adam as an optimizer, I improved prediction accuracy by test time augmentation and post-processing the masks.\n\n*__Huge thanks to Heng CherKeng!!! I learned a lot from your posts.__*\n\n## Training Method(s) ##\n\n### Pre-processing\n- I noticed some issues with the provided masks. Therefore, used the annotations and mask provided by [Konstantin Lopuhin][15] in [data quality issues][14] thread.\n- Removed the alpha channel from the images.\n- Filled holes in the masks\n- Splitted (using stratification) the `stage1_train` dataset into 90% train and 10% validation sets based on 5 image modalities provided by [Allen Goodman][5].\n- Used an [external dataset][4] provided in the forum. Divided the images and the masks into 4 pieces due their large sizes. External dataset [download links][19].\n- Generated mosaics from train images based on [Emil's][8] notebook.\n\n\n### Model and Training\n* Modified [Matterport's][1] implementation of [Mask-RCNN][3] deep neural network for object instance segmentation.\n* Tuned hyperparameters to detect small nuclei from the images. (I found [this tutorial][7] very useful for understanding the model hyperparameters)\n + Original Matterport implementation was validating only on one image so fixed this [validation issue][20].\n + Reduced RPN (region proposal network) anchor sizes since the nuclei are mostly small.\n + Increased number of anchors to be used since the nuclei are small and can be found anywhere on an image.\n + Increased maximum number of predicted objects since an image can contain 300 or more nuclei.\n + Increased `POST_NMS_ROIS_TRAINING` to get more region proposals during training.\n + Added extra parameter `DETECTION_MASK_THRESHOLD` to model configuration. Default was hardcoded in the model as 0.5 but setting it to 0.35 helped in detection of small nuclei boundaries.\n + Resized images and masks to 512x512\n* Relied heavily on image augmentation due to small training set:\n - Random horizontal or vertical flips\n - Random 90 or -90 degrees rotation\n - [Random rotations][18] in the range of (-15, 15) degrees\n - [Random cropping][18] of bigger images and masks to 256x256x3.\n - [Random scaling][18] of image and mask scaling in the range (0.5, 2.0)\n\n* Used Resnet101 architecture as a backbone encoder but initialized the first 50 layers of the model with pre-trained Resnet50 weights from [ImageNet competition][16].\n* Trained the model with [Adam][17] optimizer for 75 epochs:\n - 25 epochs with learning rate 1e-4\n - 25 epochs with learning rate 1e-5\n - 25 epochs with learning rate 1e-6\n* Did not retrain the model with stage1 test data during stage 2 as I was afraid of overfitting on black and white images.\n\n\n### Post-processing\n\n- Combined predictions on actual image and horizontally flipped image: took unions of masks with maximum overlap and removed false positive masks with small overlap.\n- Due to configured RPN anchor sizes, the model predicts small nuclei very well. However, it struggles at predicting large nuclei. Therefore, if a model predicts no masks for an image then I scale down the image and predict once again.\n- Removed overlaps between predicted nuclei based on their objectness score. In other words, removed intersections from the masks with lower scores.\n - If this intersection removal results in multiple objects in that mask, then removing all the small pieces.\n- Closing small holes inside the masks using morphological operations (dilation followed by erosion).\n\n## Interesting findings ##\n\n- Mask-RCNN model overfits easily without image augmentation.\n- Removing false positive mask predictions improves the overall score significantly.\n- Since images are on different scales, predicting masks on scaled images helps with the model generalizability.\n- Dilating and then eroding individual masks helped me achieve slightly better result.\n- Matterport's original implementation was only [validating on only one image][6]. Fixing this issue made the training process reproducible.\n- I found that the model reaches a local minima faster when trained using Adam optimizer compared to default SGD optimizer.\n\n## Unsuccessful approaches tried\n- Trained the model with Dice Coefficient Loss instead of default binary cross-entropy loss for the masks heads. Although got comparable results couldn't beat prediction accuracy of binary cross-entropy on my validation set.\n- Trained with random Gaussian and Poisson (or shot) noise for image augmentation. It actually hurt overall model performance.\n- Tried ensembling actual image predictions with horizontal and vertical flip predictions. Used non-maximum suppression for removing overlaps. Did not improve prediction accuracy on the validation set.\n- Trained end-to-end without initializing with pre-trained ImageNet weights. Mostly got to Mean IoU score of 0.35 on stage1 test set.\n- Trained on preprocessed images with adaptive histogram equalization (CLAHE). The model performed way worse.\n\n[1]: https://github.com/matterport/Mask_RCNN\n[2]: https://www.osc.edu/\n[3]: https://arxiv.org/abs/1703.06870\n[4]: https://www.kaggle.com/voglinio/external-h-e-data-with-mask-annotations\n[5]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/48130\n[6]: https://github.com/matterport/Mask_RCNN/issues/89\n[7]: https://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46\n[8]: https://www.kaggle.com/bonlime/train-test-image-mosaic\n[10]: https://nucleisegmentationbenchmark.weebly.com/dataset.html\n[11]: http://mbi.osu.edu\n[12]: http://www.lerner.ccf.org/thor/scott/lab/\n[13]: https://promise12.grand-challenge.org\n[14]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/47572\n[15]: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes\n[16]: https://github.com/fchollet/deep-learning-models/releases/\n[17]: https://arxiv.org/abs/1412.6980\n[18]: https://www.kaggle.com/c/data-science-bowl-2018/discussion/49692\n[19]: https://nucleisegmentationbenchmark.weebly.com/dataset.html\n[20]: https://github.com/matterport/Mask_RCNN/issues/89", "1383586": "Hello,i am Chinese and i'm a new deep learninger.I want to know how to relize the model in kaggle,I want to run in my own computer but it's too rubblish.I cannot do it.Can you help me that how to realize this model?Think you.My english is not very good,if someplace offend you,please forgive me.", "354678": "Hey! Thank you for this amazing topic and for the effort you put on it.\nI'd like to apply a mas r-cnn for nuclei segmentation but using Whole Slide Images. I'm thinking about patch sampling my .svs files and then use a Multi Instance Learning method. \nDo you it would be possible to use your model for something like that?\nKind Regards", "857767": "Hi. Thank you for your work. In your code, how can you print RPN region proposal bounding boxes?", "593204": "thanks for this topic.\n\nBut i want to inquire is i can implement this model to detect object when it has similar histogram with background.", "402538": "Hey, I wanted to check how you did end-to-end training without using the pre-trained imagenet weights. Currently, the code uses imagenet if no other model is specified. ", "377711": "Congrats and thanks a lot for the sharing. I would like to ask, what have you done with \"Removing false positive mask predictions improves the overall score significantly.\"?", "357697": "@lvguofeng and @JubaerHossain. Sorry for the late response. Glad that you guys have sorted out the issue. The .csv files are now available on the same repository.", "357116": "@Inom Hi, Great thread! I organized my folders as you mentioned in github. But there is no file named ../data/share_train_df.csv in your mention. So, when I run augment_preprocess.py, it show error; because you have declared df = pd.read_csv('../data/share_train_df.csv') in save_n_mosaic function. Can your please help me in this regard?", "355899": "Thank you for making your code public and let me learn.\nbut i download your code and follow your Instruction，i met some issues\nwhen i run\n\n`python augment_preprocess.py`\n\ni met some issues:\n\nTraceback (most recent call last):\nFile \"augment_preprocess.py\", line 473, in \nmake_n_save_mosaic()\nFile \"augment_preprocess.py\", line 361, in make_n_save_mosaic\nwith h5py.File(path, \"r\") as hf:\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 312, in init\nfid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)\nFile \"/home/lgf/anaconda3/envs/tensorflow-gpu/lib/python3.6/site-packages/h5py/_hl/files.py\", line 142, in make_fid\nfid = h5f.open(name, flags, fapl=fapl)\nFile \"h5py/_objects.pyx\", line 54, in h5py._objects.with_phil.wrapper\nFile \"h5py/_objects.pyx\", line 55, in h5py._objects.with_phil.wrapper\nFile \"h5py/h5f.pyx\", line 78, in h5py.h5f.open\nOSError: Unable to open file (unable to open file: name = '../data/stage1_train/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d/masks/ad473063dab4bf4f2461d9a99a9c0166d4871f156516d9e0a523484e7cf2258d.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)\n\ni want know what is this .h5 file，and how can i fix this problem to run your code", "349132": "Codes with some comments are now available on GitHub: [https://github.com/mirzaevinom/data_science_bowl_2018][1]. Hope you find it helpful!\n\n[1]: https://github.com/mirzaevinom/data_science_bowl_2018", "326333": "The link to Allen Goodman does not work. (404)", "326255": "Thanks for the nice details and congrats on a strong finish!", "489778": "" }, "complete": true, "source": "existing_discussion", "competition_id": 8089, "competition_slug": "data-science-bowl-2018", "team_id": 1309638, "solution_url": "/competitions/data-science-bowl-2018/writeups/inom-mirzaev-5th-place-solution-based-only-on-mask" }