I have been running software that gets me best scores of about .34 with 75 epochs. How many epochs are people with high scores using? It takes a very long time to run 75 epochs because I am very hardware limited at the moment.
\n\ntx
\n\nKickback
", "messages": [ { "id": "293777", "postDate": "03/10/2018 15:40:51", "content": "I have been running software that gets me best scores of about .34 with 75 epochs. How many epochs are people with high scores using? It takes a very long time to run 75 epochs because I am very hardware limited at the moment.
\n\ntx
\n\nKickback
", "rawMarkdown": "I have been running software that gets me best scores of about .34 with 75 epochs. How many epochs are people with high scores using? It takes a very long time to run 75 epochs because I am very hardware limited at the moment.\n\ntx\n\nKickback", "votes": null }, { "id": "294187", "postDate": "03/11/2018 14:31:10", "content": "At 75 epochs, I got 0.421 using Yan Wang's repo. Something like an LR of 0.0001 for the first 50, and an LR of 0.00001 for the last 25. Start with init_with='coco' and `layers='all' to train all layers of the model based on pretrained weights from coco (assuming you downloaded and placed mask_rcnn_coco.h5 where it needs to be).
To save memory, you could delete the weights that you won't use. For example, it will show where the weights are saved:
\n\ncheckpoints/nuclei20180311T1412/mask_rcnn_nuclei_{epoch:04d}.h5. \nYou could then use os.remove() to manually remove the weights prior to the weights that you need. For example: \nos.remove(checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_0001.h5) and \nos.remove(checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_0002.h5 and so on as you progress to the 50th epoch. This is annoying as it would be better to have \"save_best_only\" instead of 'save_weights_only' but I don't know how to make this work without error.
After reaching the 50th epoch, switch the init_with='coco' to init_with='last'. Change your LR to .00001, change epochs=50 to epochs=75, and run based on the last weights.
Thanks Andrew! that is really very helpful. \nFew more questions: with the Yan Wang repo, did you set: USE_MINI_MASK = True \nWhen i do that i keep getting NaN error.
\n\nOther than the learning rate did you make changes to the default config2 file \nhttps://github.com/YanWang2014/Mask_RCNN/blob/master/Spot-Nuclei2/config2.py
\n\nThanks again\nMehul
", "rawMarkdown": "Thanks Andrew! that is really very helpful. \nFew more questions: with the Yan Wang repo, did you set: USE_MINI_MASK = True \nWhen i do that i keep getting NaN error. \n\nOther than the learning rate did you make changes to the default config2 file \nhttps://github.com/YanWang2014/Mask_RCNN/blob/master/Spot-Nuclei2/config2.py\n\nThanks again\nMehul", "votes": null }, { "id": "294403", "postDate": "03/11/2018 22:45:30", "content": "I did set USE_MINI_MASK=True with MINI_MASK_SHAPE = (56, 56). I also changed the optimizer to Adam around line 2060 in model.py. If you used the learning rate 0.001, it would cause loss = nan. Make sure the max learning rate is 4 places to the right of the decimal.
", "rawMarkdown": "I did set USE_MINI_MASK=True with MINI_MASK_SHAPE = (56, 56). I also changed the optimizer to Adam around line 2060 in model.py. If you used the learning rate 0.001, it would cause loss = nan. Make sure the max learning rate is 4 places to the right of the decimal.", "votes": null }, { "id": "294942", "postDate": "03/12/2018 21:35:26", "content": "Thanks Andrew! \nfor reference with 50 epoch i got 0.399 and with 75 epoch i got 0.416 following the steps you mentioned above.
", "rawMarkdown": "Thanks Andrew! \nfor reference with 50 epoch i got 0.399 and with 75 epoch i got 0.416 following the steps you mentioned above.", "votes": null }, { "id": "295165", "postDate": "03/13/2018 07:11:16", "content": "", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 294187, "author_name": "wolfman30", "author_url": "", "post_date": "03/11/2018 14:31:10", "content": "At 75 epochs, I got 0.421 using Yan Wang's repo. Something like an LR of 0.0001 for the first 50, and an LR of 0.00001 for the last 25. Start with init_with='coco' and `layers='all' to train all layers of the model based on pretrained weights from coco (assuming you downloaded and placed mask_rcnn_coco.h5 where it needs to be).
To save memory, you could delete the weights that you won't use. For example, it will show where the weights are saved:
\n\ncheckpoints/nuclei20180311T1412/mask_rcnn_nuclei_{epoch:04d}.h5. \nYou could then use os.remove() to manually remove the weights prior to the weights that you need. For example: \nos.remove(checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_0001.h5) and \nos.remove(checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_0002.h5 and so on as you progress to the 50th epoch. This is annoying as it would be better to have \"save_best_only\" instead of 'save_weights_only' but I don't know how to make this work without error.
After reaching the 50th epoch, switch the init_with='coco' to init_with='last'. Change your LR to .00001, change epochs=50 to epochs=75, and run based on the last weights.
Thanks Andrew! that is really very helpful. \nFew more questions: with the Yan Wang repo, did you set: USE_MINI_MASK = True \nWhen i do that i keep getting NaN error.
\n\nOther than the learning rate did you make changes to the default config2 file \nhttps://github.com/YanWang2014/Mask_RCNN/blob/master/Spot-Nuclei2/config2.py
\n\nThanks again\nMehul
", "votes": null, "replies": [] }, { "id": 294403, "author_name": "wolfman30", "author_url": "", "post_date": "03/11/2018 22:45:30", "content": "I did set USE_MINI_MASK=True with MINI_MASK_SHAPE = (56, 56). I also changed the optimizer to Adam around line 2060 in model.py. If you used the learning rate 0.001, it would cause loss = nan. Make sure the max learning rate is 4 places to the right of the decimal.
", "votes": null, "replies": [] }, { "id": 294942, "author_name": "mpsampat", "author_url": "", "post_date": "03/12/2018 21:35:26", "content": "Thanks Andrew! \nfor reference with 50 epoch i got 0.399 and with 75 epoch i got 0.416 following the steps you mentioned above.
", "votes": null, "replies": [] }, { "id": 295165, "author_name": "chiaosun", "author_url": "", "post_date": "03/13/2018 07:11:16", "content": "", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "293777": "I have been running software that gets me best scores of about .34 with 75 epochs. How many epochs are people with high scores using? It takes a very long time to run 75 epochs because I am very hardware limited at the moment.\n\ntx\n\nKickback", "294187": "At 75 epochs, I got 0.421 using Yan Wang's [repo][1]. Something like an LR of 0.0001 for the first 50, and an LR of 0.00001 for the last 25. Start with `init_with='coco'` and `layers='all' to train all layers of the model based on pretrained weights from coco (assuming you downloaded and placed mask_rcnn_coco.h5 where it needs to be).\n\nTo save memory, you could delete the weights that you won't use. For example, it will show where the weights are saved: \n\n checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_{epoch:04d}.h5. \n\nYou could then use os.remove() to manually remove the weights prior to the weights that you need. For example: \n`os.remove(checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_0001.h5)` and \n`os.remove(checkpoints/nuclei20180311T1412/mask_rcnn_nuclei_0002.h5` and so on as you progress to the 50th epoch. This is annoying as it would be better to have \"save_best_only\" instead of 'save_weights_only' but I don't know how to make this work without error.\n\nAfter reaching the 50th epoch, switch the `init_with='coco'` to `init_with='last'`. Change your LR to .00001, change epochs=50 to epochs=75, and run based on the last weights. \n\n\n [1]: https://github.com/YanWang2014/Mask_RCNN/tree/master/Spot-Nuclei2", "294382": "Thanks Andrew! that is really very helpful. \nFew more questions: with the Yan Wang repo, did you set: USE_MINI_MASK = True \nWhen i do that i keep getting NaN error. \n\nOther than the learning rate did you make changes to the default config2 file \nhttps://github.com/YanWang2014/Mask_RCNN/blob/master/Spot-Nuclei2/config2.py\n\nThanks again\nMehul", "294403": "I did set USE_MINI_MASK=True with MINI_MASK_SHAPE = (56, 56). I also changed the optimizer to Adam around line 2060 in model.py. If you used the learning rate 0.001, it would cause loss = nan. Make sure the max learning rate is 4 places to the right of the decimal.", "294942": "Thanks Andrew! \nfor reference with 50 epoch i got 0.399 and with 75 epoch i got 0.416 following the steps you mentioned above.", "295165": "" }, "source": "meta" }