Hi all, I am training matterport mask rcnn, and I notice that my training loss goes to around 0.7, but then does not go further down. This is true irrespective of (a) number of epochs (b) learning rate (c) Learning algorithm (I tried SGD and Adam).
\n\nI am using some rotation as the only augmentation. I would think that such a big model should give me lower and lower losses by overfitting the training data, but that does not happen.
\n\nDoes someone else face the same problem?
", "messages": [ { "id": "309460", "postDate": "04/05/2018 11:45:34", "content": "Hi all, I am training matterport mask rcnn, and I notice that my training loss goes to around 0.7, but then does not go further down. This is true irrespective of (a) number of epochs (b) learning rate (c) Learning algorithm (I tried SGD and Adam).
\n\nI am using some rotation as the only augmentation. I would think that such a big model should give me lower and lower losses by overfitting the training data, but that does not happen.
\n\nDoes someone else face the same problem?
", "rawMarkdown": "Hi all, I am training matterport mask rcnn, and I notice that my training loss goes to around 0.7, but then does not go further down. This is true irrespective of (a) number of epochs (b) learning rate (c) Learning algorithm (I tried SGD and Adam).\n\nI am using some rotation as the only augmentation. I would think that such a big model should give me lower and lower losses by overfitting the training data, but that does not happen.\n\nDoes someone else face the same problem?", "votes": null }, { "id": "309611", "postDate": "04/05/2018 17:19:14", "content": "Hi Vivek,\nI am also facing the similar issue and my training loss for matter port mask RCNN is around 0.65. I tried with different learning rates and number of epochs. Apparently augmentation didn't help much.
\n\nAre you using any post processing for mask rcnn predicted masks?
", "rawMarkdown": "Hi Vivek,\nI am also facing the similar issue and my training loss for matter port mask RCNN is around 0.65. I tried with different learning rates and number of epochs. Apparently augmentation didn't help much. \n\nAre you using any post processing for mask rcnn predicted masks?", "votes": null }, { "id": "309619", "postDate": "04/05/2018 17:26:44", "content": "Here was my recipe:
\n\nload_image will create a random 128x128 crop; no further augmentation=> after ~30 epochs you should drop to ~.58; and it will continue to decrease as you train\n=> further augmentation in later epochs is helpful :)
", "rawMarkdown": "Here was my recipe:\n\n 1. I cleaned up a significant number of the train images manually (separated merged, removed bad training data, added missing annotations, closed holes, etc.)\n 2. Used Adam and cyclic training schedule on 40, 80, 120 epochs doing 1-e4, 1e-5, 1e-6; and then every 2 epochs it will switch the LR.\n 3. Set config to 512x512 image dimensions; steps per epoch is set to number of train instances; batch size 1.\n 4. `load_image` will create a random 128x128 crop; no further augmentation\n\n=> after ~30 epochs you should drop to ~.58; and it will continue to decrease as you train\n=> further augmentation in later epochs is helpful :)", "votes": null }, { "id": "309844", "postDate": "04/06/2018 03:45:36", "content": "Yes, I am using post processing.
", "rawMarkdown": "Yes, I am using post processing.", "votes": null } ], "comments": [ { "id": 309611, "author_name": "puneetvashisht786", "author_url": "", "post_date": "04/05/2018 17:19:14", "content": "Hi Vivek,\nI am also facing the similar issue and my training loss for matter port mask RCNN is around 0.65. I tried with different learning rates and number of epochs. Apparently augmentation didn't help much.
\n\nAre you using any post processing for mask rcnn predicted masks?
", "votes": null, "replies": [ { "id": 309844, "author_name": "thevivekpandey", "author_url": "", "post_date": "04/06/2018 03:45:36", "content": "Yes, I am using post processing.
", "votes": null, "replies": [] } ] }, { "id": 309619, "author_name": "spnrpa", "author_url": "", "post_date": "04/05/2018 17:26:44", "content": "Here was my recipe:
\n\nload_image will create a random 128x128 crop; no further augmentation=> after ~30 epochs you should drop to ~.58; and it will continue to decrease as you train\n=> further augmentation in later epochs is helpful :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "309460": "Hi all, I am training matterport mask rcnn, and I notice that my training loss goes to around 0.7, but then does not go further down. This is true irrespective of (a) number of epochs (b) learning rate (c) Learning algorithm (I tried SGD and Adam).\n\nI am using some rotation as the only augmentation. I would think that such a big model should give me lower and lower losses by overfitting the training data, but that does not happen.\n\nDoes someone else face the same problem?", "309611": "Hi Vivek,\nI am also facing the similar issue and my training loss for matter port mask RCNN is around 0.65. I tried with different learning rates and number of epochs. Apparently augmentation didn't help much. \n\nAre you using any post processing for mask rcnn predicted masks?", "309619": "Here was my recipe:\n\n 1. I cleaned up a significant number of the train images manually (separated merged, removed bad training data, added missing annotations, closed holes, etc.)\n 2. Used Adam and cyclic training schedule on 40, 80, 120 epochs doing 1-e4, 1e-5, 1e-6; and then every 2 epochs it will switch the LR.\n 3. Set config to 512x512 image dimensions; steps per epoch is set to number of train instances; batch size 1.\n 4. `load_image` will create a random 128x128 crop; no further augmentation\n\n=> after ~30 epochs you should drop to ~.58; and it will continue to decrease as you train\n=> further augmentation in later epochs is helpful :)", "309844": "Yes, I am using post processing." }, "source": "meta" }