{
  "id": 51145,
  "title": "matterport Mask RCNN: loosing accuracy after resuming checkpoints",
  "url": "/competitions/data-science-bowl-2018/discussion/51145",
  "author_name": "",
  "post_date": "2018-03-05T21:08:16.407997900Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I’m using my own implementation over matterport Mask RCNN.</p>\n\n<p>I tried to use early stopping having this very bad results.</p>\n\n<p><img src=\"https://raw.githubusercontent.com/virilo/kaggle-forums/c901e8754aa1ee2b3070136f6cd38a2980ac0a0a/images/dsb2018-resuming-checkpoints.png\" alt=\"val_loss getting lost\"></p>\n\n<p>After a training session, I recover the best checkpoint (according to the val_loss score), and start a new training session with a lower learning rate. I’m using a validation set of 133 pictures.  And measuring val_loss over all the 133 pictures on each epoch.</p>\n\n<p>The checkpoints contains only the weights.  So perhaps, after reseting the momentum, it has lost its way. <br>\nPerhaps I have a bug in the code.  Or perhaps I'm trusting too much in a val_loss for just a few observations.</p>\n\n<p>What could be happening?  How important is to preserve the momentum?\nIs it usual to start loosing model accuracy at the beggining resume?</p>",
  "messages": [
    {
      "id": "291244",
      "postDate": "03/05/2018 21:08:16",
      "content": "<p>I’m using my own implementation over matterport Mask RCNN.</p>\n\n<p>I tried to use early stopping having this very bad results.</p>\n\n<p><img src=\"https://raw.githubusercontent.com/virilo/kaggle-forums/c901e8754aa1ee2b3070136f6cd38a2980ac0a0a/images/dsb2018-resuming-checkpoints.png\" alt=\"val_loss getting lost\"></p>\n\n<p>After a training session, I recover the best checkpoint (according to the val_loss score), and start a new training session with a lower learning rate. I’m using a validation set of 133 pictures.  And measuring val_loss over all the 133 pictures on each epoch.</p>\n\n<p>The checkpoints contains only the weights.  So perhaps, after reseting the momentum, it has lost its way. <br>\nPerhaps I have a bug in the code.  Or perhaps I'm trusting too much in a val_loss for just a few observations.</p>\n\n<p>What could be happening?  How important is to preserve the momentum?\nIs it usual to start loosing model accuracy at the beggining resume?</p>",
      "rawMarkdown": "I’m using my own implementation over matterport Mask RCNN.\n\nI tried to use early stopping having this very bad results.\n\n![val_loss getting lost][1]\n\nAfter a training session, I recover the best checkpoint (according to the val_loss score), and start a new training session with a lower learning rate. I’m using a validation set of 133 pictures.  And measuring val_loss over all the 133 pictures on each epoch.\n\nThe checkpoints contains only the weights.  So perhaps, after reseting the momentum, it has lost its way.  \nPerhaps I have a bug in the code.  Or perhaps I'm trusting too much in a val_loss for just a few observations.\n\nWhat could be happening?  How important is to preserve the momentum?\nIs it usual to start loosing model accuracy at the beggining resume?\n\n\n  [1]: https://raw.githubusercontent.com/virilo/kaggle-forums/c901e8754aa1ee2b3070136f6cd38a2980ac0a0a/images/dsb2018-resuming-checkpoints.png",
      "votes": null
    },
    {
      "id": "291806",
      "postDate": "03/06/2018 22:11:51",
      "content": "<p>Hi Virilo, you are not alone, i have exactly the same issue. And the best result is not reproducable</p>",
      "rawMarkdown": "Hi Virilo, you are not alone, i have exactly the same issue. And the best result is not reproducable",
      "votes": null
    },
    {
      "id": "293638",
      "postDate": "03/10/2018 09:31:08",
      "content": "<p>I'm still stuck at this point (0.368 stage1 LB) </p>\n\n<p>Trying to answer myself:</p>\n\n<blockquote>\n  <p>How important is to preserve the <a href=\"http://cs231n.github.io/neural-networks-3/#sgd\">momentum</a>?</p>\n</blockquote>\n\n<p>A lot, since I've given a weight of 0.9 to the momentum ;-)</p>\n\n<p>I tried to save and load the optimizer weights without success.\nCurrently, <a href=\"https://github.com/matterport/Mask_RCNN/issues/277\">it is not possible with matterport RCNN as is</a>. Due to <a href=\"https://stackoverflow.com/a/47068604\">something related to Keras</a>.</p>\n\n<p>I think I'll give up saving the optimizer weights, because I think this is neither the only problem nor the only solution.</p>\n\n<p>Any idea that sheds light on this will be welcome!</p>",
      "rawMarkdown": "I'm still stuck at this point (0.368 stage1 LB) \n\nTrying to answer myself:\n\n&gt; How important is to preserve the [momentum][1]?\n\nA lot, since I've given a weight of 0.9 to the momentum ;-)\n\nI tried to save and load the optimizer weights without success.\nCurrently, [it is not possible with matterport RCNN as is][2]. Due to [something related to Keras][3].\n\nI think I'll give up saving the optimizer weights, because I think this is neither the only problem nor the only solution.\n\nAny idea that sheds light on this will be welcome!\n\n\n  [1]: http://cs231n.github.io/neural-networks-3/#sgd\n  [2]: https://github.com/matterport/Mask_RCNN/issues/277\n  [3]: https://stackoverflow.com/a/47068604",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 291806,
      "author_name": "theone4ever",
      "author_url": "",
      "post_date": "03/06/2018 22:11:51",
      "content": "<p>Hi Virilo, you are not alone, i have exactly the same issue. And the best result is not reproducable</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 293638,
      "author_name": "virilo",
      "author_url": "",
      "post_date": "03/10/2018 09:31:08",
      "content": "<p>I'm still stuck at this point (0.368 stage1 LB) </p>\n\n<p>Trying to answer myself:</p>\n\n<blockquote>\n  <p>How important is to preserve the <a href=\"http://cs231n.github.io/neural-networks-3/#sgd\">momentum</a>?</p>\n</blockquote>\n\n<p>A lot, since I've given a weight of 0.9 to the momentum ;-)</p>\n\n<p>I tried to save and load the optimizer weights without success.\nCurrently, <a href=\"https://github.com/matterport/Mask_RCNN/issues/277\">it is not possible with matterport RCNN as is</a>. Due to <a href=\"https://stackoverflow.com/a/47068604\">something related to Keras</a>.</p>\n\n<p>I think I'll give up saving the optimizer weights, because I think this is neither the only problem nor the only solution.</p>\n\n<p>Any idea that sheds light on this will be welcome!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "291244": "I’m using my own implementation over matterport Mask RCNN.\n\nI tried to use early stopping having this very bad results.\n\n![val_loss getting lost][1]\n\nAfter a training session, I recover the best checkpoint (according to the val_loss score), and start a new training session with a lower learning rate. I’m using a validation set of 133 pictures.  And measuring val_loss over all the 133 pictures on each epoch.\n\nThe checkpoints contains only the weights.  So perhaps, after reseting the momentum, it has lost its way.  \nPerhaps I have a bug in the code.  Or perhaps I'm trusting too much in a val_loss for just a few observations.\n\nWhat could be happening?  How important is to preserve the momentum?\nIs it usual to start loosing model accuracy at the beggining resume?\n\n\n  [1]: https://raw.githubusercontent.com/virilo/kaggle-forums/c901e8754aa1ee2b3070136f6cd38a2980ac0a0a/images/dsb2018-resuming-checkpoints.png",
    "291806": "Hi Virilo, you are not alone, i have exactly the same issue. And the best result is not reproducable",
    "293638": "I'm still stuck at this point (0.368 stage1 LB) \n\nTrying to answer myself:\n\n&gt; How important is to preserve the [momentum][1]?\n\nA lot, since I've given a weight of 0.9 to the momentum ;-)\n\nI tried to save and load the optimizer weights without success.\nCurrently, [it is not possible with matterport RCNN as is][2]. Due to [something related to Keras][3].\n\nI think I'll give up saving the optimizer weights, because I think this is neither the only problem nor the only solution.\n\nAny idea that sheds light on this will be welcome!\n\n\n  [1]: http://cs231n.github.io/neural-networks-3/#sgd\n  [2]: https://github.com/matterport/Mask_RCNN/issues/277\n  [3]: https://stackoverflow.com/a/47068604"
  },
  "source": "meta"
}