Hi all, \nI'm currently trying to implement the Mask R-CNN model from Matterport's repo. Thanks to the generosity of Konstantin Maksimov, I have this code:
\n\nclass NucleiDataset(utils.Dataset):
\n\ndef load_nuclei(self, root_path, mode='train', filter_ids=None):\n\n self.add_class(\"nuclei\", 1, \"nucleus\")\n\n if mode == 'test':\n files = os.listdir(os.path.join(root_path, 'test_images'))\n else:\n files = os.listdir(os.path.join(root_path, 'training_imgs'))\n\n if filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids]\n\n for i, ffile in enumerate(tqdm(files)):\n if mode == 'test':\n data_path = os.path.join(root_path, 'test_images', ffile)\n mask_path = 'No masks'\n else:\n data_path = os.path.join(root_path, 'training_imgs', ffile)\n\n bg_color = random.randint(0, 255)\n original_id = ffile.split('.')[0]\n # TODO change later maybe\n\n data = cv2.imread(data_path)[:, :, 0]\n height, width = data.shape\n\n self.add_image(\"nuclei\", image_id=i, path=data_path,\n width=width, height=height,\n bg_color=bg_color, original_id=original_id,\n data_path=data_path,\n root_path=root_path)\n\n image = cv2.imread(data_path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n\n\ndef load_image(self, image_id):\n info = self.get_info(image_id)\n path = info['data_path']\n image = cv2.imread(path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n return image\n\ndef get_info(self, image_id):\n return self.image_info[image_id]\n\ndef load_mask(self, image_id):\n info = self.get_info(image_id)\n original_id = info['original_id']\n root_path = info['root_path']\n width = info['width']\n height = info['height']\n return self.add_mask_data(original_id, root_path, width=width, height=height)[:]\n\ndef add_mask_data(self, original_id, root_path, width=256, height=256):\n # info = self.image_info[image_id]\n orifinal_id = original_id\n root_path = root_path\n\n all_masks = os.listdir(os.path.join(root_path, 'Original_masks3'))\n all_masks = [element for element in all_masks if element.split('_')[0] == orifinal_id]\n\n num_labels = 0\n #mask_ids = next(os.walk(all_masks))[1]\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n if np.sum(data) != 0:\n num_labels += 1\n\n mask = np.zeros([height,width,num_labels], dtype=np.bool)\n num_labels = 0\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n\n if np.sum(data) != 0:\n data = (data != 0)\n mask[:, :, num_labels] = data\n num_labels += 1\n\n class_ids = np.array([1] * num_labels)\n class_ids = class_ids.astype(np.int32)\n return mask, class_ids\nFurther, based on 'training_shapes.ipynb' file on the repo, I have this code:
\n\nmodel = modellib.MaskRCNN(mode=\"training\", config=config,\n model_dir=MODEL_DIR)\n\ninit_with = \"coco\"\n # Load weights trained on MS COCO, but skip layers that\n # are different due to the different number of classes\nmodel.load_weights(COCO_MODEL_PATH, by_name=True,\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])\n\nfilter_ids = os.listdir(os.path.join(ROOT_DIR, 'training_imgs'))\nfiles = [item.split('.')[0] for item in filter_ids]\noriginal_id = os.listdir(os.path.join(ROOT_DIR, 'Original_masks3'))\nall_masks = [element.split('_')[0] for element in original_id]\n\n# Training dataset\ndataset_train = NucleiDataset()\ndataset_train.load_nuclei(ROOT_DIR, mode='train',filter_ids = files[:603] )\ndataset_train.prepare()\n\n# Validation dataset\ndataset_val = NucleiDataset()\ndataset_val.load_nuclei(ROOT_DIR, mode = 'train', filter_ids = files[603:] )\ndataset_val.prepare()\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=15, \n layers='heads')\nI continue to get this error after running the last four lines: \n'TypeError: Can't pickle _thread.lock objects'
\n\nDoes anyone know where I'm going wrong?
", "messages": [ { "id": "289023", "postDate": "02/28/2018 19:08:25", "content": "Hi all, \nI'm currently trying to implement the Mask R-CNN model from Matterport's repo. Thanks to the generosity of Konstantin Maksimov, I have this code:
\n\nclass NucleiDataset(utils.Dataset):
\n\ndef load_nuclei(self, root_path, mode='train', filter_ids=None):\n\n self.add_class(\"nuclei\", 1, \"nucleus\")\n\n if mode == 'test':\n files = os.listdir(os.path.join(root_path, 'test_images'))\n else:\n files = os.listdir(os.path.join(root_path, 'training_imgs'))\n\n if filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids]\n\n for i, ffile in enumerate(tqdm(files)):\n if mode == 'test':\n data_path = os.path.join(root_path, 'test_images', ffile)\n mask_path = 'No masks'\n else:\n data_path = os.path.join(root_path, 'training_imgs', ffile)\n\n bg_color = random.randint(0, 255)\n original_id = ffile.split('.')[0]\n # TODO change later maybe\n\n data = cv2.imread(data_path)[:, :, 0]\n height, width = data.shape\n\n self.add_image(\"nuclei\", image_id=i, path=data_path,\n width=width, height=height,\n bg_color=bg_color, original_id=original_id,\n data_path=data_path,\n root_path=root_path)\n\n image = cv2.imread(data_path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n\n\ndef load_image(self, image_id):\n info = self.get_info(image_id)\n path = info['data_path']\n image = cv2.imread(path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n return image\n\ndef get_info(self, image_id):\n return self.image_info[image_id]\n\ndef load_mask(self, image_id):\n info = self.get_info(image_id)\n original_id = info['original_id']\n root_path = info['root_path']\n width = info['width']\n height = info['height']\n return self.add_mask_data(original_id, root_path, width=width, height=height)[:]\n\ndef add_mask_data(self, original_id, root_path, width=256, height=256):\n # info = self.image_info[image_id]\n orifinal_id = original_id\n root_path = root_path\n\n all_masks = os.listdir(os.path.join(root_path, 'Original_masks3'))\n all_masks = [element for element in all_masks if element.split('_')[0] == orifinal_id]\n\n num_labels = 0\n #mask_ids = next(os.walk(all_masks))[1]\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n if np.sum(data) != 0:\n num_labels += 1\n\n mask = np.zeros([height,width,num_labels], dtype=np.bool)\n num_labels = 0\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n\n if np.sum(data) != 0:\n data = (data != 0)\n mask[:, :, num_labels] = data\n num_labels += 1\n\n class_ids = np.array([1] * num_labels)\n class_ids = class_ids.astype(np.int32)\n return mask, class_ids\nFurther, based on 'training_shapes.ipynb' file on the repo, I have this code:
\n\nmodel = modellib.MaskRCNN(mode=\"training\", config=config,\n model_dir=MODEL_DIR)\n\ninit_with = \"coco\"\n # Load weights trained on MS COCO, but skip layers that\n # are different due to the different number of classes\nmodel.load_weights(COCO_MODEL_PATH, by_name=True,\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])\n\nfilter_ids = os.listdir(os.path.join(ROOT_DIR, 'training_imgs'))\nfiles = [item.split('.')[0] for item in filter_ids]\noriginal_id = os.listdir(os.path.join(ROOT_DIR, 'Original_masks3'))\nall_masks = [element.split('_')[0] for element in original_id]\n\n# Training dataset\ndataset_train = NucleiDataset()\ndataset_train.load_nuclei(ROOT_DIR, mode='train',filter_ids = files[:603] )\ndataset_train.prepare()\n\n# Validation dataset\ndataset_val = NucleiDataset()\ndataset_val.load_nuclei(ROOT_DIR, mode = 'train', filter_ids = files[603:] )\ndataset_val.prepare()\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=15, \n layers='heads')\nI continue to get this error after running the last four lines: \n'TypeError: Can't pickle _thread.lock objects'
\n\nDoes anyone know where I'm going wrong?
", "rawMarkdown": "Hi all, \nI'm currently trying to implement the Mask R-CNN model from Matterport's [repo][1]. Thanks to the generosity of Konstantin Maksimov, I have this code: \n\n\nclass NucleiDataset(utils.Dataset):\n \n def load_nuclei(self, root_path, mode='train', filter_ids=None):\n \n self.add_class(\"nuclei\", 1, \"nucleus\")\n \n if mode == 'test':\n files = os.listdir(os.path.join(root_path, 'test_images'))\n else:\n files = os.listdir(os.path.join(root_path, 'training_imgs'))\n\n if filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids]\n \n for i, ffile in enumerate(tqdm(files)):\n if mode == 'test':\n data_path = os.path.join(root_path, 'test_images', ffile)\n mask_path = 'No masks'\n else:\n data_path = os.path.join(root_path, 'training_imgs', ffile)\n \n bg_color = random.randint(0, 255)\n original_id = ffile.split('.')[0]\n # TODO change later maybe\n\n data = cv2.imread(data_path)[:, :, 0]\n height, width = data.shape\n\n self.add_image(\"nuclei\", image_id=i, path=data_path,\n width=width, height=height,\n bg_color=bg_color, original_id=original_id,\n data_path=data_path,\n root_path=root_path)\n\n image = cv2.imread(data_path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n \n \n def load_image(self, image_id):\n info = self.get_info(image_id)\n path = info['data_path']\n image = cv2.imread(path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n return image\n\n def get_info(self, image_id):\n return self.image_info[image_id]\n \n def load_mask(self, image_id):\n info = self.get_info(image_id)\n original_id = info['original_id']\n root_path = info['root_path']\n width = info['width']\n height = info['height']\n return self.add_mask_data(original_id, root_path, width=width, height=height)[:]\n\n def add_mask_data(self, original_id, root_path, width=256, height=256):\n # info = self.image_info[image_id]\n orifinal_id = original_id\n root_path = root_path\n \n all_masks = os.listdir(os.path.join(root_path, 'Original_masks3'))\n all_masks = [element for element in all_masks if element.split('_')[0] == orifinal_id]\n\n num_labels = 0\n #mask_ids = next(os.walk(all_masks))[1]\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n if np.sum(data) != 0:\n num_labels += 1\n\n mask = np.zeros([height,width,num_labels], dtype=np.bool)\n num_labels = 0\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n\n if np.sum(data) != 0:\n data = (data != 0)\n mask[:, :, num_labels] = data\n num_labels += 1\n \n class_ids = np.array([1] * num_labels)\n class_ids = class_ids.astype(np.int32)\n return mask, class_ids\n\nFurther, based on 'training_shapes.ipynb' file on the repo, I have this code: \n\n model = modellib.MaskRCNN(mode=\"training\", config=config,\n model_dir=MODEL_DIR)\n\n init_with = \"coco\"\n # Load weights trained on MS COCO, but skip layers that\n # are different due to the different number of classes\n model.load_weights(COCO_MODEL_PATH, by_name=True,\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])\n \n filter_ids = os.listdir(os.path.join(ROOT_DIR, 'training_imgs'))\n files = [item.split('.')[0] for item in filter_ids]\n original_id = os.listdir(os.path.join(ROOT_DIR, 'Original_masks3'))\n all_masks = [element.split('_')[0] for element in original_id]\n \n # Training dataset\n dataset_train = NucleiDataset()\n dataset_train.load_nuclei(ROOT_DIR, mode='train',filter_ids = files[:603] )\n dataset_train.prepare()\n \n # Validation dataset\n dataset_val = NucleiDataset()\n dataset_val.load_nuclei(ROOT_DIR, mode = 'train', filter_ids = files[603:] )\n dataset_val.prepare()\n \n model.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=15, \n layers='heads')\n\nI continue to get this error after running the last four lines: \n'TypeError: Can't pickle _thread.lock objects'\n\nDoes anyone know where I'm going wrong? \n\n\n [1]: https://github.com/matterport/Mask_RCNN", "votes": null }, { "id": "289080", "postDate": "02/28/2018 22:02:15", "content": "Hey Andrew,
\n\nI haven't come across your error, but you could have a look at Yan Wangs code:
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/50610\nhttps://github.com/YanWang2014/Mask_RCNN\nThat worked for me!
", "rawMarkdown": "Hey Andrew,\n\nI haven't come across your error, but you could have a look at Yan Wangs code:\n\n\n https://www.kaggle.com/c/data-science-bowl-2018/discussion/50610\n https://github.com/YanWang2014/Mask_RCNN\n\nThat worked for me!", "votes": null }, { "id": "289148", "postDate": "03/01/2018 01:27:10", "content": "Thank you for your reply FabSchreiber. I ran his code and got the same error after one epoch of training. Maybe something's wrong with the software package in the pre-installed virtual environment on this AWS gpu.
", "rawMarkdown": "Thank you for your reply FabSchreiber. I ran his code and got the same error after one epoch of training. Maybe something's wrong with the software package in the pre-installed virtual environment on this AWS gpu.", "votes": null }, { "id": "289174", "postDate": "03/01/2018 03:11:00", "content": "Hi FabSchreiber, what is LB score did you achieved use YanWang code? I only got 0.33 instead of expected 0.41
", "rawMarkdown": "Hi FabSchreiber, what is LB score did you achieved use YanWang code? I only got 0.33 instead of expected 0.41", "votes": null }, { "id": "289239", "postDate": "03/01/2018 08:10:09", "content": "Hey John,
\n\nI have run the code for ~80 epochs and get 0.39. I used val_test1.py to generate predictions/submissions.\nval_test.py gave worse results and I couldn't get val_test2.py to work.
\n\nBtw. I wonder if there is a more sophisticated way to change learning rates.
", "rawMarkdown": "Hey John,\n\nI have run the code for ~80 epochs and get 0.39. I used val_test1.py to generate predictions/submissions.\nval_test.py gave worse results and I couldn't get val_test2.py to work.\n\nBtw. I wonder if there is a more sophisticated way to change learning rates.", "votes": null }, { "id": "289245", "postDate": "03/01/2018 08:30:09", "content": "I see. If you want to call val_test2.py, you have to delete the line
\n\n\n # semantic = dataset.load_semantic(image_id)\n # for j in range(r['masks'].shape[-1]-1):\n # r['masks'][:,:,j]= dilation(r['masks'][:,:,j], 2)\n # r['masks'][:,:,j] = r['masks'][:,:,j] * semantic\n\nFor that, I can achieved 0.41 LB as the author expected\nFor learning rate, it used poly schedule. You also can use step but it need to change the code
Nice!
\n\nI haven't heard of poly scheduling before. Do you have a link?
\n\nThx!
", "rawMarkdown": "Nice! \n\nI haven't heard of poly scheduling before. Do you have a link?\n\nThx!", "votes": null }, { "id": "289253", "postDate": "03/01/2018 08:44:35", "content": "You are welcome. Also try the uncomment the lines in the function numpy2encoding_no_overlap2:
\n\n\npredicts = np.apply_along_axis(refineMasks, 2, predicts)\n for i in range(predicts.shape[2]-1):\n predicts[:,:,i] = refineMasks(predicts[:,:,i])\n
This is link for learning rate\nhttps://stackoverflow.com/questions/30033096/what-is-lr-policy-in-caffe\nLet me know the performance when you use refineMasks.
", "rawMarkdown": "You are welcome. Also try the uncomment the lines in the function numpy2encoding_no_overlap2:\n\n```\npredicts = np.apply_along_axis(refineMasks, 2, predicts)\n for i in range(predicts.shape[2]-1):\n predicts[:,:,i] = refineMasks(predicts[:,:,i])\n```\n\nThis is link for learning rate\nhttps://stackoverflow.com/questions/30033096/what-is-lr-policy-in-caffe\nLet me know the performance when you use refineMasks.", "votes": null }, { "id": "289479", "postDate": "03/01/2018 19:16:35", "content": "I did not encounter this error, and I am not running in AWS. But in resolving a related error I noted this currently open issue: Issue #13 which sounds like what you are experiencing. If that does not resolve your problem, you can certainly try what I did to get this code base to run. See it here: Issue #278. I think that the data generator is not threadsafe in more environments than just NT. I may be wrong, but you will see from Issue #278 what worked for me.
\n\nGood luck!
", "rawMarkdown": "I did not encounter this error, and I am not running in AWS. But in resolving a related error I noted this currently open issue: [Issue #13][1] which sounds like what you are experiencing. If that does not resolve your problem, you can certainly try what I did to get this code base to run. See it here: [Issue #278][2]. I think that the data generator is not threadsafe in more environments than just NT. I may be wrong, but you will see from Issue #278 what worked for me.\n\nGood luck!\n\n\n [1]: https://github.com/matterport/Mask_RCNN/issues/13\n [2]: https://github.com/matterport/Mask_RCNN/issues/278", "votes": null }, { "id": "289841", "postDate": "03/02/2018 12:10:00", "content": "Hi! Try to tun off multiprocessing in model.py
", "rawMarkdown": "Hi! Try to tun off multiprocessing in model.py", "votes": null }, { "id": "289893", "postDate": "03/02/2018 15:11:16", "content": "Not sure if this is also your problem, but I got that error when trying to save the model. If you're running it with callbacks and have save_weights_only = False or you try to execute model.save() then this will happen. The issue is in the gt_boxes Lambda layer: keras can't make a deepcopy of the layer configuration because there is a tensor in it (see https://github.com/keras-team/keras/issues/8343). If you save weights only then you can avoid this problem.
", "rawMarkdown": "Not sure if this is also your problem, but I got that error when trying to save the model. If you're running it with callbacks and have save_weights_only = False or you try to execute model.save() then this will happen. The issue is in the gt_boxes Lambda layer: keras can't make a deepcopy of the layer configuration because there is a tensor in it (see https://github.com/keras-team/keras/issues/8343). If you save weights only then you can avoid this problem.", "votes": null }, { "id": "290156", "postDate": "03/03/2018 05:35:48", "content": "Hi Konstantin, Could you tell why is better to tun off multiprocessing if we have multiple GPUs? I found the speed is slower when it turned off. Thanks.
", "rawMarkdown": "Hi Konstantin, Could you tell why is better to tun off multiprocessing if we have multiple GPUs? I found the speed is slower when it turned off. Thanks.", "votes": null }, { "id": "290177", "postDate": "03/03/2018 07:12:41", "content": "Hi! in my case on Ubuntu 16 04 multiprocessing cause segfault too often. Multiprocessing is used to prepare data, multi gpu model is in another place. My suggestion is to turn of multiprocessing in fit_generator call. For me it made code execution stable without any segfaults.
", "rawMarkdown": "Hi! in my case on Ubuntu 16 04 multiprocessing cause segfault too often. Multiprocessing is used to prepare data, multi gpu model is in another place. My suggestion is to turn of multiprocessing in fit_generator call. For me it made code execution stable without any segfaults.", "votes": null }, { "id": "290178", "postDate": "03/03/2018 07:14:18", "content": "Thank you all (FabSchreiber, Konstantin Maksimov, Brian Farrar, fergusoci, and john1231983) for you comments. I resolved the issue. The source of the error: I changed line 2315 in model.py from 'save_weights_only=True' to 'save_best_only=True' with 'monitor=val_loss'. I changed it back, then the model trained successfully using Yan Wang's implementation of Matterport Mask R-CNN code.
", "rawMarkdown": "Thank you all (FabSchreiber, Konstantin Maksimov, Brian Farrar, fergusoci, and john1231983) for you comments. I resolved the issue. The source of the error: I changed line 2315 in model.py from 'save_weights_only=True' to 'save_best_only=True' with 'monitor=val_loss'. I changed it back, then the model trained successfully using Yan Wang's implementation of Matterport Mask R-CNN code.", "votes": null }, { "id": "290179", "postDate": "03/03/2018 07:17:39", "content": "Good job. Let me know your LB when you ran your code. I achieved 0.41 LB. Let me know if you need my help
", "rawMarkdown": "Good job. Let me know your LB when you ran your code. I achieved 0.41 LB. Let me know if you need my help", "votes": null }, { "id": "290182", "postDate": "03/03/2018 07:50:55", "content": "I see. Thanks.
", "rawMarkdown": "I see. Thanks.", "votes": null }, { "id": "290183", "postDate": "03/03/2018 07:58:54", "content": "Thank you. I got 0.376 from 40 epochs, running 'heads'. My next aim is to run it for 60 epochs, then to implant a ResNeXt-101 backbone, then to work on pre and post processing.
", "rawMarkdown": "Thank you. I got 0.376 from 40 epochs, running 'heads'. My next aim is to run it for 60 epochs, then to implant a ResNeXt-101 backbone, then to work on pre and post processing.", "votes": null }, { "id": "290185", "postDate": "03/03/2018 08:03:38", "content": "Too low. I think you can use:\n1. The fixed dataset of someone in github\n2. Use Adam for fast converge. I tried SGD but it is not so good\n3. Do not run heads, just run all with step learning rate schedule. \nIf you consider the three points above, you can achive 0.4+ as baseline
I will try Adam as an optimizer, and I am aware that if I used the poly learning-rate schedule in Yan Wang's code, I would likely achieve an LB score of 0.41. He does have 'all' instead of 'heads' for every epoch. If my score improves from 20 more epochs using 'heads', I will shift focus to improving the backbone cnn of this model, and then implement the entire poly schedule with 'all', and Adam.
\n\nThough, I'm not sure about the 'fixed' dataset. Could you elaborate more on how the default dataset is not 'fixed'?
\n\nThank you again.
", "rawMarkdown": "I will try Adam as an optimizer, and I am aware that if I used the poly learning-rate schedule in Yan Wang's code, I would likely achieve an LB score of 0.41. He does have 'all' instead of 'heads' for every epoch. If my score improves from 20 more epochs using 'heads', I will shift focus to improving the backbone cnn of this model, and then implement the entire poly schedule with 'all', and Adam. \n\nThough, I'm not sure about the 'fixed' dataset. Could you elaborate more on how the default dataset is not 'fixed'?\n\nThank you again.", "votes": null }, { "id": "290191", "postDate": "03/03/2018 08:29:58", "content": "This one: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes\nFor Adam, he used step learning rate schedule, instead of poly.
", "rawMarkdown": "This one: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes\nFor Adam, he used step learning rate schedule, instead of poly.", "votes": null }, { "id": "290278", "postDate": "03/03/2018 15:28:36", "content": "Hi john, I uncommented the lines in the function numpy2encoding_no_overlap2 as you mentioned but met the error message like raise RuntimeError('structure and input must have same dimensionality'). Did you see the same error? Thanks.
", "rawMarkdown": "Hi john, I uncommented the lines in the function numpy2encoding_no_overlap2 as you mentioned but met the error message like raise RuntimeError('structure and input must have same dimensionality'). Did you see the same error? Thanks.", "votes": null }, { "id": "294080", "postDate": "03/11/2018 08:19:03", "content": "@john1231983,\nhow did you solve the data augmentation problem here? \nI initially thought setting Mini_Mask to False would solve the problem, but resulted in this:
\n\n/home/ec2-user/model.py:1597: RuntimeWarning: divide by zero encountered in log\n np.log(gt_h / a_h),\n/home/ec2-user/model.py:1598: RuntimeWarning: divide by zero encountered in log\n np.log(gt_w / a_w)\nwith loss=nan.
", "rawMarkdown": "john1231983,\nhow did you solve the data augmentation problem [here][1]? \nI initially thought setting Mini_Mask to False would solve the problem, but resulted in this: \n\n /home/ec2-user/model.py:1597: RuntimeWarning: divide by zero encountered in log\n np.log(gt_h / a_h),\n /home/ec2-user/model.py:1598: RuntimeWarning: divide by zero encountered in log\n np.log(gt_w / a_w)\n\nwith loss=nan. \n [1]: https://github.com/matterport/Mask_RCNN/pull/301", "votes": null }, { "id": "598660", "postDate": "08/13/2019 21:09:45", "content": "@wolfman30 I am facing the same problem. Do you by chance remember why this problem occurred in your code?
", "rawMarkdown": "wolfman30 I am facing the same problem. Do you by chance remember why this problem occurred in your code?", "votes": null } ], "comments": [ { "id": 289080, "author_name": "fabsta", "author_url": "", "post_date": "02/28/2018 22:02:15", "content": "Hey Andrew,
\n\nI haven't come across your error, but you could have a look at Yan Wangs code:
\n\nhttps://www.kaggle.com/c/data-science-bowl-2018/discussion/50610\nhttps://github.com/YanWang2014/Mask_RCNN\nThat worked for me!
", "votes": null, "replies": [ { "id": 289148, "author_name": "wolfman30", "author_url": "", "post_date": "03/01/2018 01:27:10", "content": "Thank you for your reply FabSchreiber. I ran his code and got the same error after one epoch of training. Maybe something's wrong with the software package in the pre-installed virtual environment on this AWS gpu.
", "votes": null, "replies": [] }, { "id": 289174, "author_name": "john1231983", "author_url": "", "post_date": "03/01/2018 03:11:00", "content": "Hi FabSchreiber, what is LB score did you achieved use YanWang code? I only got 0.33 instead of expected 0.41
", "votes": null, "replies": [] }, { "id": 289239, "author_name": "fabsta", "author_url": "", "post_date": "03/01/2018 08:10:09", "content": "Hey John,
\n\nI have run the code for ~80 epochs and get 0.39. I used val_test1.py to generate predictions/submissions.\nval_test.py gave worse results and I couldn't get val_test2.py to work.
\n\nBtw. I wonder if there is a more sophisticated way to change learning rates.
", "votes": null, "replies": [] }, { "id": 289245, "author_name": "john1231983", "author_url": "", "post_date": "03/01/2018 08:30:09", "content": "I see. If you want to call val_test2.py, you have to delete the line
\n\n\n # semantic = dataset.load_semantic(image_id)\n # for j in range(r['masks'].shape[-1]-1):\n # r['masks'][:,:,j]= dilation(r['masks'][:,:,j], 2)\n # r['masks'][:,:,j] = r['masks'][:,:,j] * semantic\n\nFor that, I can achieved 0.41 LB as the author expected\nFor learning rate, it used poly schedule. You also can use step but it need to change the code
Nice!
\n\nI haven't heard of poly scheduling before. Do you have a link?
\n\nThx!
", "votes": null, "replies": [] }, { "id": 289253, "author_name": "john1231983", "author_url": "", "post_date": "03/01/2018 08:44:35", "content": "You are welcome. Also try the uncomment the lines in the function numpy2encoding_no_overlap2:
\n\n\npredicts = np.apply_along_axis(refineMasks, 2, predicts)\n for i in range(predicts.shape[2]-1):\n predicts[:,:,i] = refineMasks(predicts[:,:,i])\n
This is link for learning rate\nhttps://stackoverflow.com/questions/30033096/what-is-lr-policy-in-caffe\nLet me know the performance when you use refineMasks.
", "votes": null, "replies": [] }, { "id": 290278, "author_name": "ybwu01", "author_url": "", "post_date": "03/03/2018 15:28:36", "content": "Hi john, I uncommented the lines in the function numpy2encoding_no_overlap2 as you mentioned but met the error message like raise RuntimeError('structure and input must have same dimensionality'). Did you see the same error? Thanks.
", "votes": null, "replies": [] } ] }, { "id": 289479, "author_name": "jbfarrar", "author_url": "", "post_date": "03/01/2018 19:16:35", "content": "I did not encounter this error, and I am not running in AWS. But in resolving a related error I noted this currently open issue: Issue #13 which sounds like what you are experiencing. If that does not resolve your problem, you can certainly try what I did to get this code base to run. See it here: Issue #278. I think that the data generator is not threadsafe in more environments than just NT. I may be wrong, but you will see from Issue #278 what worked for me.
\n\nGood luck!
", "votes": null, "replies": [] }, { "id": 289841, "author_name": "maksimovka", "author_url": "", "post_date": "03/02/2018 12:10:00", "content": "Hi! Try to tun off multiprocessing in model.py
", "votes": null, "replies": [ { "id": 290156, "author_name": "ybwu01", "author_url": "", "post_date": "03/03/2018 05:35:48", "content": "Hi Konstantin, Could you tell why is better to tun off multiprocessing if we have multiple GPUs? I found the speed is slower when it turned off. Thanks.
", "votes": null, "replies": [] }, { "id": 290177, "author_name": "maksimovka", "author_url": "", "post_date": "03/03/2018 07:12:41", "content": "Hi! in my case on Ubuntu 16 04 multiprocessing cause segfault too often. Multiprocessing is used to prepare data, multi gpu model is in another place. My suggestion is to turn of multiprocessing in fit_generator call. For me it made code execution stable without any segfaults.
", "votes": null, "replies": [] }, { "id": 290182, "author_name": "ybwu01", "author_url": "", "post_date": "03/03/2018 07:50:55", "content": "I see. Thanks.
", "votes": null, "replies": [] } ] }, { "id": 289893, "author_name": "fergusoci", "author_url": "", "post_date": "03/02/2018 15:11:16", "content": "Not sure if this is also your problem, but I got that error when trying to save the model. If you're running it with callbacks and have save_weights_only = False or you try to execute model.save() then this will happen. The issue is in the gt_boxes Lambda layer: keras can't make a deepcopy of the layer configuration because there is a tensor in it (see https://github.com/keras-team/keras/issues/8343). If you save weights only then you can avoid this problem.
", "votes": null, "replies": [] }, { "id": 290178, "author_name": "wolfman30", "author_url": "", "post_date": "03/03/2018 07:14:18", "content": "Thank you all (FabSchreiber, Konstantin Maksimov, Brian Farrar, fergusoci, and john1231983) for you comments. I resolved the issue. The source of the error: I changed line 2315 in model.py from 'save_weights_only=True' to 'save_best_only=True' with 'monitor=val_loss'. I changed it back, then the model trained successfully using Yan Wang's implementation of Matterport Mask R-CNN code.
", "votes": null, "replies": [ { "id": 290179, "author_name": "john1231983", "author_url": "", "post_date": "03/03/2018 07:17:39", "content": "Good job. Let me know your LB when you ran your code. I achieved 0.41 LB. Let me know if you need my help
", "votes": null, "replies": [] }, { "id": 290183, "author_name": "wolfman30", "author_url": "", "post_date": "03/03/2018 07:58:54", "content": "Thank you. I got 0.376 from 40 epochs, running 'heads'. My next aim is to run it for 60 epochs, then to implant a ResNeXt-101 backbone, then to work on pre and post processing.
", "votes": null, "replies": [] }, { "id": 290185, "author_name": "john1231983", "author_url": "", "post_date": "03/03/2018 08:03:38", "content": "Too low. I think you can use:\n1. The fixed dataset of someone in github\n2. Use Adam for fast converge. I tried SGD but it is not so good\n3. Do not run heads, just run all with step learning rate schedule. \nIf you consider the three points above, you can achive 0.4+ as baseline
I will try Adam as an optimizer, and I am aware that if I used the poly learning-rate schedule in Yan Wang's code, I would likely achieve an LB score of 0.41. He does have 'all' instead of 'heads' for every epoch. If my score improves from 20 more epochs using 'heads', I will shift focus to improving the backbone cnn of this model, and then implement the entire poly schedule with 'all', and Adam.
\n\nThough, I'm not sure about the 'fixed' dataset. Could you elaborate more on how the default dataset is not 'fixed'?
\n\nThank you again.
", "votes": null, "replies": [] }, { "id": 290191, "author_name": "john1231983", "author_url": "", "post_date": "03/03/2018 08:29:58", "content": "This one: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes\nFor Adam, he used step learning rate schedule, instead of poly.
", "votes": null, "replies": [] }, { "id": 294080, "author_name": "wolfman30", "author_url": "", "post_date": "03/11/2018 08:19:03", "content": "@john1231983,\nhow did you solve the data augmentation problem here? \nI initially thought setting Mini_Mask to False would solve the problem, but resulted in this:
\n\n/home/ec2-user/model.py:1597: RuntimeWarning: divide by zero encountered in log\n np.log(gt_h / a_h),\n/home/ec2-user/model.py:1598: RuntimeWarning: divide by zero encountered in log\n np.log(gt_w / a_w)\nwith loss=nan.
", "votes": null, "replies": [] }, { "id": 598660, "author_name": "rituyadav92", "author_url": "", "post_date": "08/13/2019 21:09:45", "content": "@wolfman30 I am facing the same problem. Do you by chance remember why this problem occurred in your code?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "289023": "Hi all, \nI'm currently trying to implement the Mask R-CNN model from Matterport's [repo][1]. Thanks to the generosity of Konstantin Maksimov, I have this code: \n\n\nclass NucleiDataset(utils.Dataset):\n \n def load_nuclei(self, root_path, mode='train', filter_ids=None):\n \n self.add_class(\"nuclei\", 1, \"nucleus\")\n \n if mode == 'test':\n files = os.listdir(os.path.join(root_path, 'test_images'))\n else:\n files = os.listdir(os.path.join(root_path, 'training_imgs'))\n\n if filter_ids is not None:\n files = [item for item in files if item.split('.')[0] in filter_ids]\n \n for i, ffile in enumerate(tqdm(files)):\n if mode == 'test':\n data_path = os.path.join(root_path, 'test_images', ffile)\n mask_path = 'No masks'\n else:\n data_path = os.path.join(root_path, 'training_imgs', ffile)\n \n bg_color = random.randint(0, 255)\n original_id = ffile.split('.')[0]\n # TODO change later maybe\n\n data = cv2.imread(data_path)[:, :, 0]\n height, width = data.shape\n\n self.add_image(\"nuclei\", image_id=i, path=data_path,\n width=width, height=height,\n bg_color=bg_color, original_id=original_id,\n data_path=data_path,\n root_path=root_path)\n\n image = cv2.imread(data_path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n \n \n def load_image(self, image_id):\n info = self.get_info(image_id)\n path = info['data_path']\n image = cv2.imread(path)\n # If grayscale. Convert to RGB for consistency.\n if image.ndim != 3:\n image = skimage.color.gray2rgb(image)\n return image\n\n def get_info(self, image_id):\n return self.image_info[image_id]\n \n def load_mask(self, image_id):\n info = self.get_info(image_id)\n original_id = info['original_id']\n root_path = info['root_path']\n width = info['width']\n height = info['height']\n return self.add_mask_data(original_id, root_path, width=width, height=height)[:]\n\n def add_mask_data(self, original_id, root_path, width=256, height=256):\n # info = self.image_info[image_id]\n orifinal_id = original_id\n root_path = root_path\n \n all_masks = os.listdir(os.path.join(root_path, 'Original_masks3'))\n all_masks = [element for element in all_masks if element.split('_')[0] == orifinal_id]\n\n num_labels = 0\n #mask_ids = next(os.walk(all_masks))[1]\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n if np.sum(data) != 0:\n num_labels += 1\n\n mask = np.zeros([height,width,num_labels], dtype=np.bool)\n num_labels = 0\n for i, ffile in enumerate(all_masks): \n subpath = os.path.join(root_path, 'Original_masks3', ffile)\n data = imread(subpath)[:, :]\n\n if np.sum(data) != 0:\n data = (data != 0)\n mask[:, :, num_labels] = data\n num_labels += 1\n \n class_ids = np.array([1] * num_labels)\n class_ids = class_ids.astype(np.int32)\n return mask, class_ids\n\nFurther, based on 'training_shapes.ipynb' file on the repo, I have this code: \n\n model = modellib.MaskRCNN(mode=\"training\", config=config,\n model_dir=MODEL_DIR)\n\n init_with = \"coco\"\n # Load weights trained on MS COCO, but skip layers that\n # are different due to the different number of classes\n model.load_weights(COCO_MODEL_PATH, by_name=True,\n exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n \"mrcnn_bbox\", \"mrcnn_mask\"])\n \n filter_ids = os.listdir(os.path.join(ROOT_DIR, 'training_imgs'))\n files = [item.split('.')[0] for item in filter_ids]\n original_id = os.listdir(os.path.join(ROOT_DIR, 'Original_masks3'))\n all_masks = [element.split('_')[0] for element in original_id]\n \n # Training dataset\n dataset_train = NucleiDataset()\n dataset_train.load_nuclei(ROOT_DIR, mode='train',filter_ids = files[:603] )\n dataset_train.prepare()\n \n # Validation dataset\n dataset_val = NucleiDataset()\n dataset_val.load_nuclei(ROOT_DIR, mode = 'train', filter_ids = files[603:] )\n dataset_val.prepare()\n \n model.train(dataset_train, dataset_val, \n learning_rate=config.LEARNING_RATE, \n epochs=15, \n layers='heads')\n\nI continue to get this error after running the last four lines: \n'TypeError: Can't pickle _thread.lock objects'\n\nDoes anyone know where I'm going wrong? \n\n\n [1]: https://github.com/matterport/Mask_RCNN", "289080": "Hey Andrew,\n\nI haven't come across your error, but you could have a look at Yan Wangs code:\n\n\n https://www.kaggle.com/c/data-science-bowl-2018/discussion/50610\n https://github.com/YanWang2014/Mask_RCNN\n\nThat worked for me!", "289148": "Thank you for your reply FabSchreiber. I ran his code and got the same error after one epoch of training. Maybe something's wrong with the software package in the pre-installed virtual environment on this AWS gpu.", "289174": "Hi FabSchreiber, what is LB score did you achieved use YanWang code? I only got 0.33 instead of expected 0.41", "289239": "Hey John,\n\nI have run the code for ~80 epochs and get 0.39. I used val_test1.py to generate predictions/submissions.\nval_test.py gave worse results and I couldn't get val_test2.py to work.\n\nBtw. I wonder if there is a more sophisticated way to change learning rates.", "289245": "I see. If you want to call val_test2.py, you have to delete the line\n\n```\n # semantic = dataset.load_semantic(image_id)\n # for j in range(r['masks'].shape[-1]-1):\n # r['masks'][:,:,j]= dilation(r['masks'][:,:,j], 2)\n # r['masks'][:,:,j] = r['masks'][:,:,j] * semantic\n```\nFor that, I can achieved 0.41 LB as the author expected\nFor learning rate, it used poly schedule. You also can use step but it need to change the code", "289247": "Nice! \n\nI haven't heard of poly scheduling before. Do you have a link?\n\nThx!", "289253": "You are welcome. Also try the uncomment the lines in the function numpy2encoding_no_overlap2:\n\n```\npredicts = np.apply_along_axis(refineMasks, 2, predicts)\n for i in range(predicts.shape[2]-1):\n predicts[:,:,i] = refineMasks(predicts[:,:,i])\n```\n\nThis is link for learning rate\nhttps://stackoverflow.com/questions/30033096/what-is-lr-policy-in-caffe\nLet me know the performance when you use refineMasks.", "289479": "I did not encounter this error, and I am not running in AWS. But in resolving a related error I noted this currently open issue: [Issue #13][1] which sounds like what you are experiencing. If that does not resolve your problem, you can certainly try what I did to get this code base to run. See it here: [Issue #278][2]. I think that the data generator is not threadsafe in more environments than just NT. I may be wrong, but you will see from Issue #278 what worked for me.\n\nGood luck!\n\n\n [1]: https://github.com/matterport/Mask_RCNN/issues/13\n [2]: https://github.com/matterport/Mask_RCNN/issues/278", "289841": "Hi! Try to tun off multiprocessing in model.py", "289893": "Not sure if this is also your problem, but I got that error when trying to save the model. If you're running it with callbacks and have save_weights_only = False or you try to execute model.save() then this will happen. The issue is in the gt_boxes Lambda layer: keras can't make a deepcopy of the layer configuration because there is a tensor in it (see https://github.com/keras-team/keras/issues/8343). If you save weights only then you can avoid this problem.", "290156": "Hi Konstantin, Could you tell why is better to tun off multiprocessing if we have multiple GPUs? I found the speed is slower when it turned off. Thanks.", "290177": "Hi! in my case on Ubuntu 16 04 multiprocessing cause segfault too often. Multiprocessing is used to prepare data, multi gpu model is in another place. My suggestion is to turn of multiprocessing in fit_generator call. For me it made code execution stable without any segfaults.", "290178": "Thank you all (FabSchreiber, Konstantin Maksimov, Brian Farrar, fergusoci, and john1231983) for you comments. I resolved the issue. The source of the error: I changed line 2315 in model.py from 'save_weights_only=True' to 'save_best_only=True' with 'monitor=val_loss'. I changed it back, then the model trained successfully using Yan Wang's implementation of Matterport Mask R-CNN code.", "290179": "Good job. Let me know your LB when you ran your code. I achieved 0.41 LB. Let me know if you need my help", "290182": "I see. Thanks.", "290183": "Thank you. I got 0.376 from 40 epochs, running 'heads'. My next aim is to run it for 60 epochs, then to implant a ResNeXt-101 backbone, then to work on pre and post processing.", "290185": "Too low. I think you can use:\n1. The fixed dataset of someone in github\n2. Use Adam for fast converge. I tried SGD but it is not so good\n3. Do not run `heads`, just run `all` with step learning rate schedule. \nIf you consider the three points above, you can achive 0.4+ as baseline", "290189": "I will try Adam as an optimizer, and I am aware that if I used the poly learning-rate schedule in Yan Wang's code, I would likely achieve an LB score of 0.41. He does have 'all' instead of 'heads' for every epoch. If my score improves from 20 more epochs using 'heads', I will shift focus to improving the backbone cnn of this model, and then implement the entire poly schedule with 'all', and Adam. \n\nThough, I'm not sure about the 'fixed' dataset. Could you elaborate more on how the default dataset is not 'fixed'?\n\nThank you again.", "290191": "This one: https://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes\nFor Adam, he used step learning rate schedule, instead of poly.", "290278": "Hi john, I uncommented the lines in the function numpy2encoding_no_overlap2 as you mentioned but met the error message like raise RuntimeError('structure and input must have same dimensionality'). Did you see the same error? Thanks.", "294080": "john1231983,\nhow did you solve the data augmentation problem [here][1]? \nI initially thought setting Mini_Mask to False would solve the problem, but resulted in this: \n\n /home/ec2-user/model.py:1597: RuntimeWarning: divide by zero encountered in log\n np.log(gt_h / a_h),\n /home/ec2-user/model.py:1598: RuntimeWarning: divide by zero encountered in log\n np.log(gt_w / a_w)\n\nwith loss=nan. \n [1]: https://github.com/matterport/Mask_RCNN/pull/301", "598660": "wolfman30 I am facing the same problem. Do you by chance remember why this problem occurred in your code?" }, "source": "meta" }