Hey guys,
\n\nThis can save you a day or two to start with the competition in case if you struggle to make your first submit. PRs are more than welcome!
\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline
", "messages": [ { "id": 289913, "postDate": "2018-03-02T15:37:19.217Z", "content": "Hey guys,
\n\nThis can save you a day or two to start with the competition in case if you struggle to make your first submit. PRs are more than welcome!
\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline
", "rawMarkdown": "Hey guys,\n\nThis can save you a day or two to start with the competition in case if you struggle to make your first submit. PRs are more than welcome!\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline", "votes": 53 }, { "id": 308907, "postDate": "2018-04-04T10:04:47.130Z", "content": "Thanks for sharing @Nikolay Shebanov.
\n\nI just took a look at your code and I could not find where you did the train/validation split. I can only see that you passed the stage1_train directory to get both train and validation data. I assumed you have split them up in another function right? If so where?
", "rawMarkdown": "Thanks for sharing @Nikolay Shebanov. \n\nI just took a look at your code and I could not find where you did the train/validation split. I can only see that you passed the stage1_train directory to get both train and validation data. I assumed you have split them up in another function right? If so where?" }, { "id": 296196, "postDate": "2018-03-14T22:19:28.640Z", "content": "I'm looking for a description of the following parameters in config
\n\nMASK_POOL_SIZE = 14\nMASK_SHAPE = [28, 28]
\n\nDoes it make sense to adapt them to our task?
", "rawMarkdown": "I'm looking for a description of the following parameters in config\n\nMASK_POOL_SIZE = 14\nMASK_SHAPE = [28, 28]\n\nDoes it make sense to adapt them to our task?\n", "replies": [ { "id": 297123, "postDate": "2018-03-16T10:40:00.090Z", "content": "I'm still not sure about the MASK_SHAPE since it's role in the original implementation is kinda similar to MINI_MASK_SHAPE, although it's used only during training. I suppose you will not be able to use pretrained weights if you change this.
\n\nAs for the MASK_POOL_SIZE, that's The width of the square feature map generated from ROI Pooling., i.e. the size of the input feature of the masking branch.
It definitely makes sense to tune these. This paper can give you an idea how to approach this:
\n\nhttp://ieeexplore.ieee.org/document/8019550/
\n\nBut be ready for everything to stop working :)
", "rawMarkdown": "I'm still not sure about the MASK_SHAPE since it's role in the original implementation is kinda similar to MINI_MASK_SHAPE, although it's used only during training. I suppose you will not be able to use pretrained weights if you change this. \n\nAs for the MASK_POOL_SIZE, that's `The width of the square feature map generated from ROI Pooling.`, i.e. the size of the input feature of the masking branch. \n\nIt definitely makes sense to tune these. This paper can give you an idea how to approach this:\n\nhttp://ieeexplore.ieee.org/document/8019550/\n\nBut be ready for everything to stop working :)" } ] }, { "id": 295620, "postDate": "2018-03-13T23:24:58.860Z", "content": "When running CUDA_VISIBLE_DEVICES=\"0\" python train.py, I get this error,
Traceback (most recent call last):
File \"train.py\", line 68, in <module>
layers=\"all\")\n File \"/content/kaggle-ds-bowl-2018-baseline/model.py\", line 2237, in train\n validation_data=next(
File \"/content/kaggle-ds-bowl-2018-baseline/model.py\", line 1736, in data_generator
dataset.image_info[image_id]))
UnboundLocalError: local variable 'image_id' referenced before assignment
Do you know why this is happening? Thanks
", "rawMarkdown": "When running `CUDA_VISIBLE_DEVICES=\"0\" python train.py`, I get this error, \n\n`Traceback (most recent call last):`\n\n `File \"train.py\", line 68, in ", "replies": [ { "id": 295624, "postDate": "2018-03-13T23:45:09.543Z", "content": "Hard to tell, image_id should be on its place at this point.
\n\nI have not changed the code.\nI think that I have downloaded all the data but I’m not sure where to put it. Can you tell me what folders to put the data I to and what to name the folders ?
", "rawMarkdown": "I have not changed the code.\nI think that I have downloaded all the data but I’m not sure where to put it. Can you tell me what folders to put the data I to and what to name the folders ?", "votes": 1 }, { "id": 295942, "postDate": "2018-03-14T12:57:25.133Z", "content": "Check the README carefully, please. stage1_train and stage1_test should be at the root of the project.
If you use the fixed dataset, then check for the stage1_train directory at the root of Lopuhin's repo:\nhttps://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes
Thank you, it worked out.
", "rawMarkdown": "Thank you, it worked out." } ] }, { "id": 294145, "postDate": "2018-03-11T11:58:25.087Z", "content": "Hi, thanks for the code. One thing that I didn't understand, though, is why do you need/is advised to use the ResNet model? Do you use the pre-trained weights because of the RPN and mask prediction?\nOne other question is what is the least amount of memory required to run this model? I've got 2GB and I'm getting OOM even with 128x128 images and 1 image per GPU.\nThanks
", "rawMarkdown": "Hi, thanks for the code. One thing that I didn't understand, though, is why do you need/is advised to use the ResNet model? Do you use the pre-trained weights because of the RPN and mask prediction?\nOne other question is what is the least amount of memory required to run this model? I've got 2GB and I'm getting OOM even with 128x128 images and 1 image per GPU.\nThanks", "replies": [ { "id": 294193, "postDate": "2018-03-11T14:41:32.563Z", "content": "ResNet is a natural choice for the task, especially because the original paper was built using it. Yes, I use pre-trained weights. I'm not sure about the memory, but I run it on 1080ti, which has over 11 gigs of it.
\n\nNot sure if it's possible to run on 2 gigs without some weird tricks. You can cut most of the constants in the config file to make it lightweight but there's no sense in using it this way.
", "rawMarkdown": "ResNet is a natural choice for the task, especially because the original paper was built using it. Yes, I use pre-trained weights. I'm not sure about the memory, but I run it on 1080ti, which has over 11 gigs of it. \n\nNot sure if it's possible to run on 2 gigs without some weird tricks. You can cut most of the constants in the config file to make it lightweight but there's no sense in using it this way.", "votes": 2 }, { "id": 294388, "postDate": "2018-03-11T21:40:08.727Z", "content": "I may be wrong but in the orignal paper they had a lot of classes to classify and much more data, so I guess it is natural that the model needs to be deeper. However, in our case we are just using 2 classes and have less images.. \nOr are the pre-trained weights mainly useful for the RPN and mask prediction part?
", "rawMarkdown": "I may be wrong but in the orignal paper they had a lot of classes to classify and much more data, so I guess it is natural that the model needs to be deeper. However, in our case we are just using 2 classes and have less images.. \nOr are the pre-trained weights mainly useful for the RPN and mask prediction part?" }, { "id": 295623, "postDate": "2018-03-13T23:41:41.210Z", "content": "Pre-trained weights are mainly useful to compensate for the lack of training data. It's much harder to train from scratch on the data and it takes more time.
\n\nYou may be right as for the argument that there could be a better and simpler architecture for the competition, but I had no goal to alter the one from the matterport Mask R-CNN code, I just wanted to make it work. Remember, it's not exactly my code, it's just a fork :)
", "rawMarkdown": "Pre-trained weights are mainly useful to compensate for the lack of training data. It's much harder to train from scratch on the data and it takes more time. \n\nYou may be right as for the argument that there could be a better and simpler architecture for the competition, but I had no goal to alter the one from the matterport Mask R-CNN code, I just wanted to make it work. Remember, it's not exactly my code, it's just a fork :)", "votes": 1 }, { "id": 295646, "postDate": "2018-03-14T01:01:45.150Z", "content": "I see, thanks :)\nBtw, I managed to run the ResNet but now I'm getting this error:
\n\n> Exception: Invalid bounding box with area of zero
\n\nI tried for both the usual Kaggle dataset and the one with the fixes, is there any modification that must be done to the masks before training?
\n\n-Edit- I found out the error. I was resizing the images and because of that some masks were disappearing.
", "rawMarkdown": "I see, thanks :)\nBtw, I managed to run the ResNet but now I'm getting this error:\n\n> Exception: Invalid bounding box with area of zero\n\nI tried for both the usual Kaggle dataset and the one with the fixes, is there any modification that must be done to the masks before training?\n\n-Edit- I found out the error. I was resizing the images and because of that some masks were disappearing.\n" } ] }, { "id": 292949, "postDate": "2018-03-08T23:51:45.047Z", "content": "Hey everybody. Great start and thank you for the post. Was wondering if somebody could help me with an issue. Running the code gives me the following error when running the training. My platform is a Win 2016 server with 2 GPU's. I have tensorflow 1.5 and Keras 2.0.9. I have verified that all datasets are being built properly with the visualization tools. Kind of stuck. Thanks in advance!!
\n\nStopIteration Traceback (most recent call last)\n in ()\n 6 learning_rate=config.LEARNING_RATE / 10,\n 7 epochs=2,\n----> 8 layers=\"all\")
\n\n~\\model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers)\n 2236 max_queue_size=100,\n 2237 workers=workers,\n-> 2238 use_multiprocessing=False,\n 2239 )\n 2240 self.epoch = max(self.epoch, epochs)
\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\n 85 warnings.warn('Update your ' + object_name +\n 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)\n---> 87 return func(*args, **kwargs)\n 88 wrapper._original_function = func\n 89 return wrapper
C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 2044 batch_index = 0\n 2045 while steps_done < steps_per_epoch:\n-> 2046 generator_output = next(output_generator)\n 2047 \n 2048 if not hasattr(generator_output, 'len'):
\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\utils\\data_utils.py in get(self)\n 657 all_finished = all([not thread.is_alive() for thread in self._threads])\n 658 if all_finished and self.queue.empty():\n--> 659 raise StopIteration()\n 660 else:\n 661 time.sleep(self.wait_time)
\n\nStopIteration:
", "rawMarkdown": "Hey everybody. Great start and thank you for the post. Was wondering if somebody could help me with an issue. Running the code gives me the following error when running the training. My platform is a Win 2016 server with 2 GPU's. I have tensorflow 1.5 and Keras 2.0.9. I have verified that all datasets are being built properly with the visualization tools. Kind of stuck. Thanks in advance!!\n\nStopIteration Traceback (most recent call last)\n", "replies": [ { "id": 292981, "postDate": "2018-03-09T01:12:50.090Z", "content": "Do you mind reformatting the code to make it more readable? For now I can see only that it isn't a full backtrace, there should be something more.
", "rawMarkdown": "Do you mind reformatting the code to make it more readable? For now I can see only that it isn't a full backtrace, there should be something more." }, { "id": 293028, "postDate": "2018-03-09T03:29:46.807Z", "content": "Hey Nikolay.
\n\nHere is the code:
\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\scipy\\ndimage\\interpolation.py:600: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.\n \"the returned array has changed.\", UserWarning)\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\nEpoch 1/100\nTraceback (most recent call last):
\n\nFile \"\", line 1, in \n runfile('C:/kaggle-ds-bowl-2018-baseline-master/train.py', wdir='C:/kaggle-ds-bowl-2018-baseline-master')
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\spyder\\utils\\site\\sitecustomize.py\", line 710, in runfile\n execfile(filename, namespace)
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\spyder\\utils\\site\\sitecustomize.py\", line 101, in execfile\n exec(compile(f.read(), filename, 'exec'), namespace)
\n\nFile \"C:/kaggle-ds-bowl-2018-baseline-master/train.py\", line 78, in \n layers=\"all\")
\n\nFile \"C:\\kaggle-ds-bowl-2018-baseline-master\\model.py\", line 2241, in train\n use_multiprocessing=False,
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\legacy\\interfaces.py\", line 87, in wrapper\n return func(*args, **kwargs)
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\engine\\training.py\", line 2046, in fit_generator\n generator_output = next(output_generator)
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\utils\\data_utils.py\", line 659, in get\n raise StopIteration()
\n\nStopIteration
", "rawMarkdown": "Hey Nikolay.\n\nHere is the code:\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\scipy\\ndimage\\interpolation.py:600: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.\n \"the returned array has changed.\", UserWarning)\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\nEpoch 1/100\nTraceback (most recent call last):\n\n File \"" }, { "id": 293199, "postDate": "2018-03-09T11:36:55.297Z", "content": "This might be obvious, but StopIteration means that the training generator is empty.\nhttps://github.com/keras-team/keras/blob/4b74fc5418944c9f449eb88ed4b40ada280fa5ca/keras/utils/data_utils.py#L784
\n\nAre you sure you tested the datasets well? Also, have you made any notable changes to my code or you just run it as is?
", "rawMarkdown": "This might be obvious, but StopIteration means that the training generator is empty.\nhttps://github.com/keras-team/keras/blob/4b74fc5418944c9f449eb88ed4b40ada280fa5ca/keras/utils/data_utils.py#L784\n\nAre you sure you tested the datasets well? Also, have you made any notable changes to my code or you just run it as is?" }, { "id": 293260, "postDate": "2018-03-09T14:42:21.497Z", "content": "No I appreciate the info :). Yes I agree the training generator is dropping data. But i have used the verification and visualization code to check the and I can see training data in the data dataset_training, it shows masks on that data and it returns shapes of the data. Which is why I am very confused why it's dropping. I thought it might be an issue between Linux and Windows but I tested code to change the worker roles and multithreaded processing to false and that had no effect.
", "rawMarkdown": "No I appreciate the info :). Yes I agree the training generator is dropping data. But i have used the verification and visualization code to check the and I can see training data in the data dataset_training, it shows masks on that data and it returns shapes of the data. Which is why I am very confused why it's dropping. I thought it might be an issue between Linux and Windows but I tested code to change the worker roles and multithreaded processing to false and that had no effect." }, { "id": 293267, "postDate": "2018-03-09T14:53:42.707Z", "content": "Try to drop a import pdb; pdb.set_trace() on this line and debug if the data gets loaded properly https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/model.py#L1639
It feels like some windows black magic is going on here.
", "rawMarkdown": "Try to drop a `import pdb; pdb.set_trace()` on this line and debug if the data gets loaded properly https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/model.py#L1639\n\nIt feels like some windows black magic is going on here." }, { "id": 293268, "postDate": "2018-03-09T14:56:55.590Z", "content": "Ill give it a shot! Thank you sir!
", "rawMarkdown": "Ill give it a shot! Thank you sir!" }, { "id": 293295, "postDate": "2018-03-09T15:43:06.820Z", "content": "Still hunting. It looks like the dataset is getting images and masks. I am wondering if it's something in the batch. Dataset, batches, image returns, classes are all showing data. Bizzaro...
", "rawMarkdown": "Still hunting. It looks like the dataset is getting images and masks. I am wondering if it's something in the batch. Dataset, batches, image returns, classes are all showing data. Bizzaro...\n" }, { "id": 293360, "postDate": "2018-03-09T17:47:46.520Z", "content": "Might have a theory, since there is so much processing going on, I had noticed that I set my config to: GPU 2 / IMAGES_PER_GPU = 4. This value is setting the batch size. I wonder if it was too low and therefore the queue was empty before preprocessing another batch occurred.
", "rawMarkdown": "Might have a theory, since there is so much processing going on, I had noticed that I set my config to: GPU 2 / IMAGES_PER_GPU = 4. This value is setting the batch size. I wonder if it was too low and therefore the queue was empty before preprocessing another batch occurred." }, { "id": 293375, "postDate": "2018-03-09T18:39:44.677Z", "content": "Shouldn't be an issue. Anyway, I've found it slower to use multiple GPUs, at least with the current dataset setup (reading multiple files).
", "rawMarkdown": "Shouldn't be an issue. Anyway, I've found it slower to use multiple GPUs, at least with the current dataset setup (reading multiple files)." }, { "id": 293415, "postDate": "2018-03-09T19:40:22.943Z", "content": "So I went back to basics. Verified that the only change to the original code was to mod the regex to allow // on the path. Aside from that it is a straight run with. The only thing i am doing is running from spyder and not a command prompt. Out of curiosity, are you running 3.5 or 2.7 for your version?
", "rawMarkdown": "So I went back to basics. Verified that the only change to the original code was to mod the regex to allow // on the path. Aside from that it is a straight run with. The only thing i am doing is running from spyder and not a command prompt. Out of curiosity, are you running 3.5 or 2.7 for your version?" }, { "id": 293427, "postDate": "2018-03-09T20:08:58.773Z", "content": "I'm on 3.6.4 to be exact :) Is that the regex from the comment somewhere above on this page?
", "rawMarkdown": "I'm on 3.6.4 to be exact :) Is that the regex from the comment somewhere above on this page?" }, { "id": 293428, "postDate": "2018-03-09T20:10:32.407Z", "content": "Yes. I saw something about pycotools not being verified on anything past 2.7. I have verified that the queue is empty on reaching that point. It has memory allocation but nothing inside. Very strange
", "rawMarkdown": "Yes. I saw something about pycotools not being verified on anything past 2.7. I have verified that the queue is empty on reaching that point. It has memory allocation but nothing inside. Very strange " }, { "id": 293431, "postDate": "2018-03-09T20:13:06.710Z", "content": "Oh wait, I run Keras 2.1.3 / tensorflow 1.4.1, try to update yours
", "rawMarkdown": "Oh wait, I run Keras 2.1.3 / tensorflow 1.4.1, try to update yours" }, { "id": 293433, "postDate": "2018-03-09T20:20:52.080Z", "content": "Ok even the demo run fails in the same spot. Something's up :)
", "rawMarkdown": "Ok even the demo run fails in the same spot. Something's up :)" }, { "id": 294190, "postDate": "2018-03-11T14:37:36.047Z", "content": "I was getting the StopIteration raised as well, I fixed by updating both Tensorflow and Keras
", "rawMarkdown": "I was getting the StopIteration raised as well, I fixed by updating both Tensorflow and Keras", "votes": 2 }, { "id": 294312, "postDate": "2018-03-11T18:06:07.407Z", "content": "What versions of Keras and Tensorflow are you running now?
", "rawMarkdown": "What versions of Keras and Tensorflow are you running now?" }, { "id": 294387, "postDate": "2018-03-11T21:34:53.913Z", "content": "tf.__version__ : '1.4.1'\nkeras.__version__ : '2.1.4'\nHey everybody thanks for the help on this. It was indeed the Keras version needed to be updated as well as Tensorflow. But a word to my colleagues out there, if you are using Windows. If you update Tensorflow make sure you force install the gpu version as it will upgrade Tensorflow to 1.5 but it will wipe out the GPU version. So you have to force pip install into your conda environment no less than 1.3 or it will fail the version check. Training now!
", "rawMarkdown": "Hey everybody thanks for the help on this. It was indeed the Keras version needed to be updated as well as Tensorflow. But a word to my colleagues out there, if you are using Windows. If you update Tensorflow make sure you force install the gpu version as it will upgrade Tensorflow to 1.5 but it will wipe out the GPU version. So you have to force pip install into your conda environment no less than 1.3 or it will fail the version check. Training now! ", "votes": 1 }, { "id": 294809, "postDate": "2018-03-12T16:57:55.637Z", "content": "Glad to hear that. I hope it was worth the troubles!
", "rawMarkdown": "Glad to hear that. I hope it was worth the troubles!" } ] }, { "id": 291395, "postDate": "2018-03-06T03:28:42.427Z", "content": "Anyone tried the script under windows 10. Seems the path parsing under windows is different from linux (Solved)
", "rawMarkdown": "Anyone tried the script under windows 10. Seems the path parsing under windows is different from linux (Solved)", "replies": [ { "id": 291449, "postDate": "2018-03-06T06:41:12.717Z", "content": "yes. add .replace(\"\\\\\", \"/\") after mask_path in bowl_dataset.py
", "rawMarkdown": "yes. add **.replace(\"\\\\\\\", \"/\")** after **mask_path** in bowl_dataset.py", "votes": 2 }, { "id": 291543, "postDate": "2018-03-06T10:24:37.483Z", "content": "Sorry guys, I tried to make it portable, but didn't actually test it. I'd appreciate if someone will send a PR to fix the regex here https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/bowl_dataset.py#L19
", "rawMarkdown": "Sorry guys, I tried to make it portable, but didn't actually test it. I'd appreciate if someone will send a PR to fix the regex here https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/bowl_dataset.py#L19" }, { "id": 291571, "postDate": "2018-03-06T11:56:46.677Z", "content": "Thx Wei Lu and Nikolay Shebanov
\n\nNow it works with the following fix in bowl_dataset.py
\n\nfor mask_path in glob(os.path.join(base_path, \"**\", \"masks\", \"*.png\")):
`mask_path = mask_path.replace(\"\\\\\", \"/\") `\n `matches = id_extractor.match(mask_path)`\nCheers
", "rawMarkdown": "Thx Wei Lu and Nikolay Shebanov\n\nNow it works with the following fix in bowl_dataset.py \n\n`for mask_path in glob(os.path.join(base_path, \"**\", \"masks\", \"*.png\")):`\n\n `mask_path = mask_path.replace(\"\\\\\", \"/\") `\n `matches = id_extractor.match(mask_path)`\n\nCheers" }, { "id": 291589, "postDate": "2018-03-06T12:27:07.977Z", "content": "There's a couple of mistakes in your code, try this instead:
\n\nmask_path = mask_path.replace(\"\\\\\", \"/\")\nHi Nikolay, I am a windows 10 user, I am getting an error on line 20 in bowl_dataset.py:
\n\nid_extractor = re.compile(f\"{base_path}{os.sep}(?P.){os.sep}masks{os.sep}(?P.*).png\") that says \"sre_constants.error: unbalanced parenthesis at position 30\".
\n\nThe mask_path line occurs below this line, so the above fixes do not help. Any idea what could be causing this? Would appreciate any help!
", "rawMarkdown": "Hi Nikolay, I am a windows 10 user, I am getting an error on line 20 in bowl_dataset.py:\n\nid_extractor = re.compile(f\"{base_path}{os.sep}(?P.){os.sep}masks{os.sep}(?P" }, { "id": 299417, "postDate": "2018-03-20T21:10:31.157Z", "content": "What line do you pass as a base_path? i.e. what goes as an argument to load_bowl()?
If you could drop an import pdb; pdb.set_trace() directly before this line and then check what's the value of f\"{base_path}{os.sep}(?P.){os.sep}masks{os.sep}(?P.*).png\" then it would be easier to understand the problem.
This is what I get:
\n\nipdb> print(f\"{base_path}\\{os.sep}(?P<image_id>.*)\\{os.sep}masks\\{os.sep}(?P<mask_id>.*)\\.png\")\nstage1_train\\{\\(?P<image_id>.*)\\{\\masks\\{\\(?P<mask_id>.*)\\.png
There is a bracket mismatch definitely. I am fairly new to Python and I did try correcting this but no luck! Any ideas on what is going wrong?
", "rawMarkdown": "This is what I get:\n\n ipdb> print(f\"{base_path}\\{os.sep}(?P" }, { "id": 301035, "postDate": "2018-03-22T07:38:26.090Z", "content": "Which version of Python are you on?
", "rawMarkdown": "Which version of Python are you on?" }, { "id": 301038, "postDate": "2018-03-22T07:42:50.403Z", "content": "Python 3.6.1
", "rawMarkdown": "Python 3.6.1" } ] }, { "id": 290858, "postDate": "2018-03-05T03:45:32.677Z", "content": "Thank you for sharing!
\n\nWhen I running your code in model training, I encounter one error : \nTraceback (most recent call last):\n File \"train.py\", line 63, in \n layers='heads')\n File \"/home/hkueee/kaggle/DataScienceBowl/dsbowl/model.py\", line 2237, in train\n validation_data=next(val_generator),\n File \"/home/hkueee/kaggle/DataScienceBowl/dsbowl/model.py\", line 1736, in data_generator\n dataset.image_info[image_id]))\nUnboundLocalError: local variable 'image_id' referenced before assignment
\n\nDo you know how to fix that? Thanks!
", "rawMarkdown": "Thank you for sharing!\n\nWhen I running your code in model training, I encounter one error : \nTraceback (most recent call last):\n File \"train.py\", line 63, in ", "replies": [ { "id": 290920, "postDate": "2018-03-05T07:18:24.523Z", "content": "Fixed. Need to specify the path of data first.
", "rawMarkdown": "Fixed. Need to specify the path of data first. ", "votes": 1 }, { "id": 372571, "postDate": "2018-08-19T18:32:52.597Z", "rawMarkdown": "", "isDeleted": true }, { "id": 372594, "postDate": "2018-08-19T20:19:08.023Z", "content": "Hey Kate, did you experience the same error? I think it's easy to fix by providing relative paths to data as an argument to the load_bowl function. Let me know if that's worked.
", "rawMarkdown": "Hey Kate, did you experience the same error? I think it's easy to fix by providing relative paths to data as an argument to the load_bowl function. Let me know if that's worked." }, { "id": 372620, "postDate": "2018-08-19T22:12:28.327Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 290650, "postDate": "2018-03-04T16:33:25.020Z", "content": "What graphics card do you use? I'm trying to start train on NVidia 1070 (8 GB), but I got a out of memory error on start like this:
\n\nResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1200,256,14,14]\n [[Node: training/SGD/gradients/mrcnn_mask_deconv/conv2d_transpose_grad/Conv2D = Conv2D[T=DT_FLOAT, _class=[\"loc:@mrcnn_mask_deconv/conv2d_transpose\"], data_format=\"NHWC\", padding=\"VALID\", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](training/SGD/gradients/mrcnn_mask_deconv/Relu_grad/ReluGrad, mrcnn_mask_deconv/kernel/read)]]\n [[Node: training/SGD/gradients/mrcnn_mask_conv1/convolution_grad/Conv2DBackpropInput/_2931 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_7871_training/SGD/gradients/mrcnn_mask_conv1/convolution_grad/Conv2DBackpropInput\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\nA couple of 1080TIs actually, so that's more memory than you have. Try to change this variable to 1 locally: https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/bowl_config.py#L14
\n\nTell me if it helps.
", "rawMarkdown": "A couple of 1080TIs actually, so that's more memory than you have. Try to change this variable to 1 locally: https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/bowl_config.py#L14\n\nTell me if it helps.", "votes": 2 }, { "id": 290707, "postDate": "2018-03-04T18:01:32.783Z", "content": "Yes, it works with \"IMAGES_PER_GPU = 1\". Thank you.
", "rawMarkdown": "Yes, it works with \"IMAGES_PER_GPU = 1\". Thank you." }, { "id": 291447, "postDate": "2018-03-06T06:38:11.680Z", "content": "Hello Roman, would you mind telling us how you reach 49th place? Did you use this mask-rcnn method? Thanks.
", "rawMarkdown": "Hello Roman, would you mind telling us how you reach 49th place? Did you use this mask-rcnn method? Thanks." }, { "id": 291597, "postDate": "2018-03-06T12:37:04.953Z", "content": "No, I'm using UNet-based CNN with additional post-processing.
", "rawMarkdown": "No, I'm using UNet-based CNN with additional post-processing." }, { "id": 291692, "postDate": "2018-03-06T17:16:42.933Z", "content": "I see. Thanks
", "rawMarkdown": "I see. Thanks" } ] }, { "id": 290505, "postDate": "2018-03-04T10:11:26.223Z", "content": "Hi Nikolay, thanks for the code and the steps.
\n\nI am running your code on Google cloud with Tesla K80 GPU but it took over 5 hours to complete 1 epoch (total 100 epochs). It is taking approximately 30 seconds for each image to run (total 664 images). After going through the forums, I think that cudnn might not be installed properly. \nCan you please guide me in the proper direction so that I do not waste a lot of time on setup.
\n\nI am new with GPUs and your help is greatly appreciated.
", "rawMarkdown": "Hi Nikolay, thanks for the code and the steps. \n\nI am running your code on Google cloud with Tesla K80 GPU but it took over 5 hours to complete 1 epoch (total 100 epochs). It is taking approximately 30 seconds for each image to run (total 664 images). After going through the forums, I think that cudnn might not be installed properly. \nCan you please guide me in the proper direction so that I do not waste a lot of time on setup.\n\nI am new with GPUs and your help is greatly appreciated.", "replies": [ { "id": 290550, "postDate": "2018-03-04T12:06:37.507Z", "content": "Easy first step would be to follow one of the tutorials specific to Google Cloud setup.
\n\nI think this should work: https://github.com/eshvk/gcp-dl
", "rawMarkdown": "Easy first step would be to follow one of the tutorials specific to Google Cloud setup.\n\nI think this should work: https://github.com/eshvk/gcp-dl" }, { "id": 290590, "postDate": "2018-03-04T13:55:02.347Z", "content": "Hi Abhishek \n the issue is using too much I/O operation when loading：\ndef load_image(self, image_id):\n info = self.image_info[image_id]\n return cv2.imread(info[\"path\"])
", "rawMarkdown": "Hi Abhishek \n the issue is using too much I/O operation when loading：\ndef load_image(self, image_id):\n info = self.image_info[image_id]\n return cv2.imread(info[\"path\"])\n" } ] }, { "id": 290500, "postDate": "2018-03-04T10:00:46.803Z", "content": "Hi Nikolay! Thanks for sharing the code. \nIn train.py I see:\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE, \n epochs=1, \n layers='heads')
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE / 10,\n epochs=100, \n layers=\"all\")
\n\nI think we can just run the second model.train (with layers==all) and skip the first one right ? \nOr is there a reason to run it this way ? \nThanks\nMehul
", "rawMarkdown": "Hi Nikolay! Thanks for sharing the code. \nIn train.py I see:\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE, \n epochs=1, \n layers='heads')\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE / 10,\n epochs=100, \n layers=\"all\")\n\nI think we can just run the second model.train (with layers==all) and skip the first one right ? \nOr is there a reason to run it this way ? \nThanks\nMehul", "replies": [ { "id": 290544, "postDate": "2018-03-04T11:49:35.293Z", "content": "I copied that from the original repo. You can run it with \"all\" right from the start.
\n\nHowever, the paper says that:
\n\n\n\n\nTo unify RPNs with Fast R-CNN [2] object detec-\n tion networks, we propose a training scheme that\n alternates between fine-tuning for the region proposal\n task and then fine-tuning for object detection, while\n keeping the proposals fixed. This scheme converges\n quickly and produces a unified network with convo-\n lutional features that are shared between both tasks.
\n
Not sure yet how to train like that using the matterport framework.
", "rawMarkdown": "I copied that from the original repo. You can run it with \"all\" right from the start.\n\nHowever, the paper says that:\n\n> To unify RPNs with Fast R-CNN [2] object detec-\ntion networks, we propose a training scheme that\nalternates between fine-tuning for the region proposal\ntask and then fine-tuning for object detection, while\nkeeping the proposals fixed. This scheme converges\nquickly and produces a unified network with convo-\nlutional features that are shared between both tasks.\n\nNot sure yet how to train like that using the matterport framework." } ] }, { "id": 290464, "postDate": "2018-03-04T06:32:09.190Z", "content": "Hi Nikolay, thank you for sharing your code.
\n\nCan you also share the versions of python packages used (including tensorflow and scipy)? I am seeing rpn_bbox_loss: nan after some point during training. Since I was running the code as is without changes, I am guessing I may have a version which causes issue but I am not sure.
", "rawMarkdown": "Hi Nikolay, thank you for sharing your code. \n\nCan you also share the versions of python packages used (including tensorflow and scipy)? I am seeing rpn_bbox_loss: nan after some point during training. Since I was running the code as is without changes, I am guessing I may have a version which causes issue but I am not sure.\n\n", "replies": [ { "id": 290547, "postDate": "2018-03-04T11:54:24.087Z", "content": "Here you go:
\n\nCython==0.27.3\nnumpy==1.14.0\nopencv-python==3.4.0.12\npycocotools==2.0\nscikit-image==0.13.1\nscikit-learn==0.19.1\nscipy==1.0.0\ntensorflow-gpu==1.4.1\ntensorflow-tensorboard==0.4.0\nI've heard of the same issue from Konstantin Maksimov. He's using his own fork, so let's ask him how he fixed that.
", "rawMarkdown": "Here you go:\n\n Cython==0.27.3\n numpy==1.14.0\n opencv-python==3.4.0.12\n pycocotools==2.0\n scikit-image==0.13.1\n scikit-learn==0.19.1\n scipy==1.0.0\n tensorflow-gpu==1.4.1\n tensorflow-tensorboard==0.4.0\n\nI've heard of the same issue from [Konstantin Maksimov][1]. He's using his own fork, so let's ask him how he fixed that.\n\n\n [1]: https://www.kaggle.com/maksimovka", "votes": 2 }, { "id": 290562, "postDate": "2018-03-04T12:41:58.927Z", "content": "https://www.kaggle.com/c/data-science-bowl-2018/discussion/49805#289277
", "rawMarkdown": "https://www.kaggle.com/c/data-science-bowl-2018/discussion/49805#289277", "votes": 2 }, { "id": 290884, "postDate": "2018-03-05T05:12:34.903Z", "content": "Makes sense. Thanks for the help!
", "rawMarkdown": "Makes sense. Thanks for the help!" } ] } ], "comments": [ { "id": 308907, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-04-04T10:04:47.130000", "content": "Thanks for sharing @Nikolay Shebanov.
\n\nI just took a look at your code and I could not find where you did the train/validation split. I can only see that you passed the stage1_train directory to get both train and validation data. I assumed you have split them up in another function right? If so where?
", "votes": 0, "replies": [] }, { "id": 296196, "author_name": "tichy ijon", "author_url": "", "post_date": "2018-03-14T22:19:28.640000", "content": "I'm looking for a description of the following parameters in config
\n\nMASK_POOL_SIZE = 14\nMASK_SHAPE = [28, 28]
\n\nDoes it make sense to adapt them to our task?
", "votes": 0, "replies": [ { "id": 297123, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-16T10:40:00.090000", "content": "I'm still not sure about the MASK_SHAPE since it's role in the original implementation is kinda similar to MINI_MASK_SHAPE, although it's used only during training. I suppose you will not be able to use pretrained weights if you change this.
\n\nAs for the MASK_POOL_SIZE, that's The width of the square feature map generated from ROI Pooling., i.e. the size of the input feature of the masking branch.
It definitely makes sense to tune these. This paper can give you an idea how to approach this:
\n\nhttp://ieeexplore.ieee.org/document/8019550/
\n\nBut be ready for everything to stop working :)
", "votes": 0, "replies": [] } ] }, { "id": 295620, "author_name": "bilal2vec", "author_url": "", "post_date": "2018-03-13T23:24:58.860000", "content": "When running CUDA_VISIBLE_DEVICES=\"0\" python train.py, I get this error,
Traceback (most recent call last):
File \"train.py\", line 68, in <module>
layers=\"all\")\n File \"/content/kaggle-ds-bowl-2018-baseline/model.py\", line 2237, in train\n validation_data=next(
File \"/content/kaggle-ds-bowl-2018-baseline/model.py\", line 1736, in data_generator
dataset.image_info[image_id]))
UnboundLocalError: local variable 'image_id' referenced before assignment
Do you know why this is happening? Thanks
", "votes": 0, "replies": [ { "id": 295624, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-13T23:45:09.543000", "content": "Hard to tell, image_id should be on its place at this point.
\n\nI have not changed the code.\nI think that I have downloaded all the data but I’m not sure where to put it. Can you tell me what folders to put the data I to and what to name the folders ?
", "votes": 1, "replies": [] }, { "id": 295942, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-14T12:57:25.133000", "content": "Check the README carefully, please. stage1_train and stage1_test should be at the root of the project.
If you use the fixed dataset, then check for the stage1_train directory at the root of Lopuhin's repo:\nhttps://github.com/lopuhin/kaggle-dsbowl-2018-dataset-fixes
Thank you, it worked out.
", "votes": 0, "replies": [] } ] }, { "id": 294145, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-03-11T11:58:25.087000", "content": "Hi, thanks for the code. One thing that I didn't understand, though, is why do you need/is advised to use the ResNet model? Do you use the pre-trained weights because of the RPN and mask prediction?\nOne other question is what is the least amount of memory required to run this model? I've got 2GB and I'm getting OOM even with 128x128 images and 1 image per GPU.\nThanks
", "votes": 0, "replies": [ { "id": 294193, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-11T14:41:32.563000", "content": "ResNet is a natural choice for the task, especially because the original paper was built using it. Yes, I use pre-trained weights. I'm not sure about the memory, but I run it on 1080ti, which has over 11 gigs of it.
\n\nNot sure if it's possible to run on 2 gigs without some weird tricks. You can cut most of the constants in the config file to make it lightweight but there's no sense in using it this way.
", "votes": 2, "replies": [] }, { "id": 294388, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-03-11T21:40:08.727000", "content": "I may be wrong but in the orignal paper they had a lot of classes to classify and much more data, so I guess it is natural that the model needs to be deeper. However, in our case we are just using 2 classes and have less images.. \nOr are the pre-trained weights mainly useful for the RPN and mask prediction part?
", "votes": 0, "replies": [] }, { "id": 295623, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-13T23:41:41.210000", "content": "Pre-trained weights are mainly useful to compensate for the lack of training data. It's much harder to train from scratch on the data and it takes more time.
\n\nYou may be right as for the argument that there could be a better and simpler architecture for the competition, but I had no goal to alter the one from the matterport Mask R-CNN code, I just wanted to make it work. Remember, it's not exactly my code, it's just a fork :)
", "votes": 1, "replies": [] }, { "id": 295646, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-03-14T01:01:45.150000", "content": "I see, thanks :)\nBtw, I managed to run the ResNet but now I'm getting this error:
\n\n> Exception: Invalid bounding box with area of zero
\n\nI tried for both the usual Kaggle dataset and the one with the fixes, is there any modification that must be done to the masks before training?
\n\n-Edit- I found out the error. I was resizing the images and because of that some masks were disappearing.
", "votes": 0, "replies": [] } ] }, { "id": 292949, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-08T23:51:45.047000", "content": "Hey everybody. Great start and thank you for the post. Was wondering if somebody could help me with an issue. Running the code gives me the following error when running the training. My platform is a Win 2016 server with 2 GPU's. I have tensorflow 1.5 and Keras 2.0.9. I have verified that all datasets are being built properly with the visualization tools. Kind of stuck. Thanks in advance!!
\n\nStopIteration Traceback (most recent call last)\n in ()\n 6 learning_rate=config.LEARNING_RATE / 10,\n 7 epochs=2,\n----> 8 layers=\"all\")
\n\n~\\model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers)\n 2236 max_queue_size=100,\n 2237 workers=workers,\n-> 2238 use_multiprocessing=False,\n 2239 )\n 2240 self.epoch = max(self.epoch, epochs)
\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\n 85 warnings.warn('Update your ' + object_name +\n 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)\n---> 87 return func(*args, **kwargs)\n 88 wrapper._original_function = func\n 89 return wrapper
C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 2044 batch_index = 0\n 2045 while steps_done < steps_per_epoch:\n-> 2046 generator_output = next(output_generator)\n 2047 \n 2048 if not hasattr(generator_output, 'len'):
\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\utils\\data_utils.py in get(self)\n 657 all_finished = all([not thread.is_alive() for thread in self._threads])\n 658 if all_finished and self.queue.empty():\n--> 659 raise StopIteration()\n 660 else:\n 661 time.sleep(self.wait_time)
\n\nStopIteration:
", "votes": 0, "replies": [ { "id": 292981, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-09T01:12:50.090000", "content": "Do you mind reformatting the code to make it more readable? For now I can see only that it isn't a full backtrace, there should be something more.
", "votes": 0, "replies": [] }, { "id": 293028, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T03:29:46.807000", "content": "Hey Nikolay.
\n\nHere is the code:
\n\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\scipy\\ndimage\\interpolation.py:600: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.\n \"the returned array has changed.\", UserWarning)\nC:\\Anaconda\\envs\\py35\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\nEpoch 1/100\nTraceback (most recent call last):
\n\nFile \"\", line 1, in \n runfile('C:/kaggle-ds-bowl-2018-baseline-master/train.py', wdir='C:/kaggle-ds-bowl-2018-baseline-master')
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\spyder\\utils\\site\\sitecustomize.py\", line 710, in runfile\n execfile(filename, namespace)
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\spyder\\utils\\site\\sitecustomize.py\", line 101, in execfile\n exec(compile(f.read(), filename, 'exec'), namespace)
\n\nFile \"C:/kaggle-ds-bowl-2018-baseline-master/train.py\", line 78, in \n layers=\"all\")
\n\nFile \"C:\\kaggle-ds-bowl-2018-baseline-master\\model.py\", line 2241, in train\n use_multiprocessing=False,
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\legacy\\interfaces.py\", line 87, in wrapper\n return func(*args, **kwargs)
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\engine\\training.py\", line 2046, in fit_generator\n generator_output = next(output_generator)
\n\nFile \"C:\\Anaconda\\envs\\py35\\lib\\site-packages\\keras\\utils\\data_utils.py\", line 659, in get\n raise StopIteration()
\n\nStopIteration
", "votes": 0, "replies": [] }, { "id": 293199, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-09T11:36:55.297000", "content": "This might be obvious, but StopIteration means that the training generator is empty.\nhttps://github.com/keras-team/keras/blob/4b74fc5418944c9f449eb88ed4b40ada280fa5ca/keras/utils/data_utils.py#L784
\n\nAre you sure you tested the datasets well? Also, have you made any notable changes to my code or you just run it as is?
", "votes": 0, "replies": [] }, { "id": 293260, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T14:42:21.497000", "content": "No I appreciate the info :). Yes I agree the training generator is dropping data. But i have used the verification and visualization code to check the and I can see training data in the data dataset_training, it shows masks on that data and it returns shapes of the data. Which is why I am very confused why it's dropping. I thought it might be an issue between Linux and Windows but I tested code to change the worker roles and multithreaded processing to false and that had no effect.
", "votes": 0, "replies": [] }, { "id": 293267, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-09T14:53:42.707000", "content": "Try to drop a import pdb; pdb.set_trace() on this line and debug if the data gets loaded properly https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/model.py#L1639
It feels like some windows black magic is going on here.
", "votes": 0, "replies": [] }, { "id": 293268, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T14:56:55.590000", "content": "Ill give it a shot! Thank you sir!
", "votes": 0, "replies": [] }, { "id": 293295, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T15:43:06.820000", "content": "Still hunting. It looks like the dataset is getting images and masks. I am wondering if it's something in the batch. Dataset, batches, image returns, classes are all showing data. Bizzaro...
", "votes": 0, "replies": [] }, { "id": 293360, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T17:47:46.520000", "content": "Might have a theory, since there is so much processing going on, I had noticed that I set my config to: GPU 2 / IMAGES_PER_GPU = 4. This value is setting the batch size. I wonder if it was too low and therefore the queue was empty before preprocessing another batch occurred.
", "votes": 0, "replies": [] }, { "id": 293375, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-09T18:39:44.677000", "content": "Shouldn't be an issue. Anyway, I've found it slower to use multiple GPUs, at least with the current dataset setup (reading multiple files).
", "votes": 0, "replies": [] }, { "id": 293415, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T19:40:22.943000", "content": "So I went back to basics. Verified that the only change to the original code was to mod the regex to allow // on the path. Aside from that it is a straight run with. The only thing i am doing is running from spyder and not a command prompt. Out of curiosity, are you running 3.5 or 2.7 for your version?
", "votes": 0, "replies": [] }, { "id": 293427, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-09T20:08:58.773000", "content": "I'm on 3.6.4 to be exact :) Is that the regex from the comment somewhere above on this page?
", "votes": 0, "replies": [] }, { "id": 293428, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T20:10:32.407000", "content": "Yes. I saw something about pycotools not being verified on anything past 2.7. I have verified that the queue is empty on reaching that point. It has memory allocation but nothing inside. Very strange
", "votes": 0, "replies": [] }, { "id": 293431, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-09T20:13:06.710000", "content": "Oh wait, I run Keras 2.1.3 / tensorflow 1.4.1, try to update yours
", "votes": 0, "replies": [] }, { "id": 293433, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-09T20:20:52.080000", "content": "Ok even the demo run fails in the same spot. Something's up :)
", "votes": 0, "replies": [] }, { "id": 294190, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-03-11T14:37:36.047000", "content": "I was getting the StopIteration raised as well, I fixed by updating both Tensorflow and Keras
", "votes": 2, "replies": [] }, { "id": 294312, "author_name": "maddocmatt", "author_url": "", "post_date": "2018-03-11T18:06:07.407000", "content": "What versions of Keras and Tensorflow are you running now?
", "votes": 0, "replies": [] }, { "id": 294387, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2018-03-11T21:34:53.913000", "content": "tf.__version__ : '1.4.1'\nkeras.__version__ : '2.1.4'\nHey everybody thanks for the help on this. It was indeed the Keras version needed to be updated as well as Tensorflow. But a word to my colleagues out there, if you are using Windows. If you update Tensorflow make sure you force install the gpu version as it will upgrade Tensorflow to 1.5 but it will wipe out the GPU version. So you have to force pip install into your conda environment no less than 1.3 or it will fail the version check. Training now!
", "votes": 1, "replies": [] }, { "id": 294809, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-12T16:57:55.637000", "content": "Glad to hear that. I hope it was worth the troubles!
", "votes": 0, "replies": [] } ] }, { "id": 291395, "author_name": "Kelvin Au", "author_url": "", "post_date": "2018-03-06T03:28:42.427000", "content": "Anyone tried the script under windows 10. Seems the path parsing under windows is different from linux (Solved)
", "votes": 0, "replies": [ { "id": 291449, "author_name": "Wei Lu", "author_url": "", "post_date": "2018-03-06T06:41:12.717000", "content": "yes. add .replace(\"\\\\\", \"/\") after mask_path in bowl_dataset.py
", "votes": 2, "replies": [] }, { "id": 291543, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-06T10:24:37.483000", "content": "Sorry guys, I tried to make it portable, but didn't actually test it. I'd appreciate if someone will send a PR to fix the regex here https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/bowl_dataset.py#L19
", "votes": 0, "replies": [] }, { "id": 291571, "author_name": "Kelvin Au", "author_url": "", "post_date": "2018-03-06T11:56:46.677000", "content": "Thx Wei Lu and Nikolay Shebanov
\n\nNow it works with the following fix in bowl_dataset.py
\n\nfor mask_path in glob(os.path.join(base_path, \"**\", \"masks\", \"*.png\")):
`mask_path = mask_path.replace(\"\\\\\", \"/\") `\n `matches = id_extractor.match(mask_path)`\nCheers
", "votes": 0, "replies": [] }, { "id": 291589, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-06T12:27:07.977000", "content": "There's a couple of mistakes in your code, try this instead:
\n\nmask_path = mask_path.replace(\"\\\\\", \"/\")\nHi Nikolay, I am a windows 10 user, I am getting an error on line 20 in bowl_dataset.py:
\n\nid_extractor = re.compile(f\"{base_path}{os.sep}(?P.){os.sep}masks{os.sep}(?P.*).png\") that says \"sre_constants.error: unbalanced parenthesis at position 30\".
\n\nThe mask_path line occurs below this line, so the above fixes do not help. Any idea what could be causing this? Would appreciate any help!
", "votes": 0, "replies": [] }, { "id": 299417, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-20T21:10:31.157000", "content": "What line do you pass as a base_path? i.e. what goes as an argument to load_bowl()?
If you could drop an import pdb; pdb.set_trace() directly before this line and then check what's the value of f\"{base_path}{os.sep}(?P.){os.sep}masks{os.sep}(?P.*).png\" then it would be easier to understand the problem.
This is what I get:
\n\nipdb> print(f\"{base_path}\\{os.sep}(?P<image_id>.*)\\{os.sep}masks\\{os.sep}(?P<mask_id>.*)\\.png\")\nstage1_train\\{\\(?P<image_id>.*)\\{\\masks\\{\\(?P<mask_id>.*)\\.png
There is a bracket mismatch definitely. I am fairly new to Python and I did try correcting this but no luck! Any ideas on what is going wrong?
", "votes": 0, "replies": [] }, { "id": 301035, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-22T07:38:26.090000", "content": "Which version of Python are you on?
", "votes": 0, "replies": [] }, { "id": 301038, "author_name": "Ankita Jha", "author_url": "", "post_date": "2018-03-22T07:42:50.403000", "content": "Python 3.6.1
", "votes": 0, "replies": [] } ] }, { "id": 290858, "author_name": "Binyu Wang", "author_url": "", "post_date": "2018-03-05T03:45:32.677000", "content": "Thank you for sharing!
\n\nWhen I running your code in model training, I encounter one error : \nTraceback (most recent call last):\n File \"train.py\", line 63, in \n layers='heads')\n File \"/home/hkueee/kaggle/DataScienceBowl/dsbowl/model.py\", line 2237, in train\n validation_data=next(val_generator),\n File \"/home/hkueee/kaggle/DataScienceBowl/dsbowl/model.py\", line 1736, in data_generator\n dataset.image_info[image_id]))\nUnboundLocalError: local variable 'image_id' referenced before assignment
\n\nDo you know how to fix that? Thanks!
", "votes": 0, "replies": [ { "id": 290920, "author_name": "Binyu Wang", "author_url": "", "post_date": "2018-03-05T07:18:24.523000", "content": "Fixed. Need to specify the path of data first.
", "votes": 1, "replies": [] }, { "id": 372571, "author_name": "", "author_url": "", "post_date": "2018-08-19T18:32:52.597000", "content": "", "votes": 0, "replies": [] }, { "id": 372594, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-08-19T20:19:08.023000", "content": "Hey Kate, did you experience the same error? I think it's easy to fix by providing relative paths to data as an argument to the load_bowl function. Let me know if that's worked.
", "votes": 0, "replies": [] }, { "id": 372620, "author_name": "", "author_url": "", "post_date": "2018-08-19T22:12:28.327000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 290650, "author_name": "Roman Chernenko", "author_url": "", "post_date": "2018-03-04T16:33:25.020000", "content": "What graphics card do you use? I'm trying to start train on NVidia 1070 (8 GB), but I got a out of memory error on start like this:
\n\nResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1200,256,14,14]\n [[Node: training/SGD/gradients/mrcnn_mask_deconv/conv2d_transpose_grad/Conv2D = Conv2D[T=DT_FLOAT, _class=[\"loc:@mrcnn_mask_deconv/conv2d_transpose\"], data_format=\"NHWC\", padding=\"VALID\", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](training/SGD/gradients/mrcnn_mask_deconv/Relu_grad/ReluGrad, mrcnn_mask_deconv/kernel/read)]]\n [[Node: training/SGD/gradients/mrcnn_mask_conv1/convolution_grad/Conv2DBackpropInput/_2931 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_7871_training/SGD/gradients/mrcnn_mask_conv1/convolution_grad/Conv2DBackpropInput\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\nA couple of 1080TIs actually, so that's more memory than you have. Try to change this variable to 1 locally: https://github.com/killthekitten/kaggle-ds-bowl-2018-baseline/blob/master/bowl_config.py#L14
\n\nTell me if it helps.
", "votes": 2, "replies": [] }, { "id": 290707, "author_name": "Roman Chernenko", "author_url": "", "post_date": "2018-03-04T18:01:32.783000", "content": "Yes, it works with \"IMAGES_PER_GPU = 1\". Thank you.
", "votes": 0, "replies": [] }, { "id": 291447, "author_name": "Wei Lu", "author_url": "", "post_date": "2018-03-06T06:38:11.680000", "content": "Hello Roman, would you mind telling us how you reach 49th place? Did you use this mask-rcnn method? Thanks.
", "votes": 0, "replies": [] }, { "id": 291597, "author_name": "Roman Chernenko", "author_url": "", "post_date": "2018-03-06T12:37:04.953000", "content": "No, I'm using UNet-based CNN with additional post-processing.
", "votes": 0, "replies": [] }, { "id": 291692, "author_name": "Wei Lu", "author_url": "", "post_date": "2018-03-06T17:16:42.933000", "content": "I see. Thanks
", "votes": 0, "replies": [] } ] }, { "id": 290505, "author_name": "Abhishek Jindal", "author_url": "", "post_date": "2018-03-04T10:11:26.223000", "content": "Hi Nikolay, thanks for the code and the steps.
\n\nI am running your code on Google cloud with Tesla K80 GPU but it took over 5 hours to complete 1 epoch (total 100 epochs). It is taking approximately 30 seconds for each image to run (total 664 images). After going through the forums, I think that cudnn might not be installed properly. \nCan you please guide me in the proper direction so that I do not waste a lot of time on setup.
\n\nI am new with GPUs and your help is greatly appreciated.
", "votes": 0, "replies": [ { "id": 290550, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-04T12:06:37.507000", "content": "Easy first step would be to follow one of the tutorials specific to Google Cloud setup.
\n\nI think this should work: https://github.com/eshvk/gcp-dl
", "votes": 0, "replies": [] }, { "id": 290590, "author_name": "Kevin Liu", "author_url": "", "post_date": "2018-03-04T13:55:02.347000", "content": "Hi Abhishek \n the issue is using too much I/O operation when loading：\ndef load_image(self, image_id):\n info = self.image_info[image_id]\n return cv2.imread(info[\"path\"])
", "votes": 0, "replies": [] } ] }, { "id": 290500, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2018-03-04T10:00:46.803000", "content": "Hi Nikolay! Thanks for sharing the code. \nIn train.py I see:\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE, \n epochs=1, \n layers='heads')
\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE / 10,\n epochs=100, \n layers=\"all\")
\n\nI think we can just run the second model.train (with layers==all) and skip the first one right ? \nOr is there a reason to run it this way ? \nThanks\nMehul
", "votes": 0, "replies": [ { "id": 290544, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-04T11:49:35.293000", "content": "I copied that from the original repo. You can run it with \"all\" right from the start.
\n\nHowever, the paper says that:
\n\n\n\n\nTo unify RPNs with Fast R-CNN [2] object detec-\n tion networks, we propose a training scheme that\n alternates between fine-tuning for the region proposal\n task and then fine-tuning for object detection, while\n keeping the proposals fixed. This scheme converges\n quickly and produces a unified network with convo-\n lutional features that are shared between both tasks.
\n
Not sure yet how to train like that using the matterport framework.
", "votes": 0, "replies": [] } ] }, { "id": 290464, "author_name": "jackkwok", "author_url": "", "post_date": "2018-03-04T06:32:09.190000", "content": "Hi Nikolay, thank you for sharing your code.
\n\nCan you also share the versions of python packages used (including tensorflow and scipy)? I am seeing rpn_bbox_loss: nan after some point during training. Since I was running the code as is without changes, I am guessing I may have a version which causes issue but I am not sure.
", "votes": 0, "replies": [ { "id": 290547, "author_name": "Nikolay Shebanov", "author_url": "", "post_date": "2018-03-04T11:54:24.087000", "content": "Here you go:
\n\nCython==0.27.3\nnumpy==1.14.0\nopencv-python==3.4.0.12\npycocotools==2.0\nscikit-image==0.13.1\nscikit-learn==0.19.1\nscipy==1.0.0\ntensorflow-gpu==1.4.1\ntensorflow-tensorboard==0.4.0\nI've heard of the same issue from Konstantin Maksimov. He's using his own fork, so let's ask him how he fixed that.
", "votes": 2, "replies": [] }, { "id": 290562, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2018-03-04T12:41:58.927000", "content": "https://www.kaggle.com/c/data-science-bowl-2018/discussion/49805#289277
", "votes": 2, "replies": [] }, { "id": 290884, "author_name": "jackkwok", "author_url": "", "post_date": "2018-03-05T05:12:34.903000", "content": "Makes sense. Thanks for the help!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "289913": "Hey guys,\n\nThis can save you a day or two to start with the competition in case if you struggle to make your first submit. PRs are more than welcome!\n\nhttps://github.com/killthekitten/kaggle-ds-bowl-2018-baseline", "308907": "Thanks for sharing @Nikolay Shebanov. \n\nI just took a look at your code and I could not find where you did the train/validation split. I can only see that you passed the stage1_train directory to get both train and validation data. I assumed you have split them up in another function right? If so where?", "296196": "I'm looking for a description of the following parameters in config\n\nMASK_POOL_SIZE = 14\nMASK_SHAPE = [28, 28]\n\nDoes it make sense to adapt them to our task?\n", "295620": "When running `CUDA_VISIBLE_DEVICES=\"0\" python train.py`, I get this error, \n\n`Traceback (most recent call last):`\n\n `File \"train.py\", line 68, in ", "294145": "Hi, thanks for the code. One thing that I didn't understand, though, is why do you need/is advised to use the ResNet model? Do you use the pre-trained weights because of the RPN and mask prediction?\nOne other question is what is the least amount of memory required to run this model? I've got 2GB and I'm getting OOM even with 128x128 images and 1 image per GPU.\nThanks", "292949": "Hey everybody. Great start and thank you for the post. Was wondering if somebody could help me with an issue. Running the code gives me the following error when running the training. My platform is a Win 2016 server with 2 GPU's. I have tensorflow 1.5 and Keras 2.0.9. I have verified that all datasets are being built properly with the visualization tools. Kind of stuck. Thanks in advance!!\n\nStopIteration Traceback (most recent call last)\n", "291395": "Anyone tried the script under windows 10. Seems the path parsing under windows is different from linux (Solved)", "290858": "Thank you for sharing!\n\nWhen I running your code in model training, I encounter one error : \nTraceback (most recent call last):\n File \"train.py\", line 63, in ", "290650": "What graphics card do you use? I'm trying to start train on NVidia 1070 (8 GB), but I got a out of memory error on start like this:\n\n ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1200,256,14,14]\n \t [[Node: training/SGD/gradients/mrcnn_mask_deconv/conv2d_transpose_grad/Conv2D = Conv2D[T=DT_FLOAT, _class=[\"loc:@mrcnn_mask_deconv/conv2d_transpose\"], data_format=\"NHWC\", padding=\"VALID\", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](training/SGD/gradients/mrcnn_mask_deconv/Relu_grad/ReluGrad, mrcnn_mask_deconv/kernel/read)]]\n \t [[Node: training/SGD/gradients/mrcnn_mask_conv1/convolution_grad/Conv2DBackpropInput/_2931 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_7871_training/SGD/gradients/mrcnn_mask_conv1/convolution_grad/Conv2DBackpropInput\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\n", "290505": "Hi Nikolay, thanks for the code and the steps. \n\nI am running your code on Google cloud with Tesla K80 GPU but it took over 5 hours to complete 1 epoch (total 100 epochs). It is taking approximately 30 seconds for each image to run (total 664 images). After going through the forums, I think that cudnn might not be installed properly. \nCan you please guide me in the proper direction so that I do not waste a lot of time on setup.\n\nI am new with GPUs and your help is greatly appreciated.", "290500": "Hi Nikolay! Thanks for sharing the code. \nIn train.py I see:\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE, \n epochs=1, \n layers='heads')\n\nmodel.train(dataset_train, dataset_val, \n learning_rate=bowl_config.LEARNING_RATE / 10,\n epochs=100, \n layers=\"all\")\n\nI think we can just run the second model.train (with layers==all) and skip the first one right ? \nOr is there a reason to run it this way ? \nThanks\nMehul", "290464": "Hi Nikolay, thank you for sharing your code. \n\nCan you also share the versions of python packages used (including tensorflow and scipy)? I am seeing rpn_bbox_loss: nan after some point during training. Since I was running the code as is without changes, I am guessing I may have a version which causes issue but I am not sure.\n\n" } }