{
  "id": 53971,
  "title": "Metrics",
  "url": "/competitions/data-science-bowl-2018/discussion/53971",
  "author_name": "Alchemist",
  "post_date": "2018-04-07T15:54:49.207000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi guys,</p>\n\n<p>what metrics do you use to measure the model output ? </p>\n\n<p>I'm going with a U-Net, so the task is naturally split in 2:</p>\n\n<ul>\n<li><p>identify the masks with a simple U-Net</p></li>\n<li><p>use post-processing to separate the object instances (in my case, I use another U-Net output for that)</p></li>\n</ul>\n\n<p>I've come up with the following analysis:</p>\n\n<ul>\n<li><p>for each mask in an image, I calculate:</p>\n\n<ul><li><p>iou with predicted masks</p></li>\n<li><p>coverage by predicted mask (ignoring the object instance)</p></li></ul></li>\n</ul>\n\n<p>There are 4 different losses that impact the score:</p>\n\n<ul>\n<li><p>false-positives - false objects detected by the model</p></li>\n<li><p>false-negatives - real objects, not detected by the model</p></li>\n<li><p>overlaps - correctly identified mask of 2 or more adjacent objects, but not separate as different instances</p></li>\n<li><p>pixel accuracy - for object that are correctly identified and separated, the IOU</p></li>\n</ul>\n\n<p>I find the pixel accuracy extremely important - this is probably because many objects are quite small, and if you have an IOU of say 0.69, visually it looks great, but with the scoring formula you will have a true positive for IOU [0.5, 0.55, 0.6, 0.65], but you will also have a false negative and a false positive for IOUs [0.7, 0.75, 0.8, 0.85, 0.9, 0.95]. </p>\n\n<p>While the false-positives, false-negatives, overlaps are pretty obvious to spot and the idea is the same between train / test, how do you prevent overfitting on the pixel accuracy ? </p>\n\n<p>I've just tested a 'better' segmentation, with the same instances identification as previous submission, and I see a difference of -0.03 (while on my validation set I've had +0.02), which is pretty huge. </p>",
  "messages": [
    {
      "id": 310460,
      "postDate": "2018-04-07T15:54:49.207Z",
      "content": "<p>Hi guys,</p>\n\n<p>what metrics do you use to measure the model output ? </p>\n\n<p>I'm going with a U-Net, so the task is naturally split in 2:</p>\n\n<ul>\n<li><p>identify the masks with a simple U-Net</p></li>\n<li><p>use post-processing to separate the object instances (in my case, I use another U-Net output for that)</p></li>\n</ul>\n\n<p>I've come up with the following analysis:</p>\n\n<ul>\n<li><p>for each mask in an image, I calculate:</p>\n\n<ul><li><p>iou with predicted masks</p></li>\n<li><p>coverage by predicted mask (ignoring the object instance)</p></li></ul></li>\n</ul>\n\n<p>There are 4 different losses that impact the score:</p>\n\n<ul>\n<li><p>false-positives - false objects detected by the model</p></li>\n<li><p>false-negatives - real objects, not detected by the model</p></li>\n<li><p>overlaps - correctly identified mask of 2 or more adjacent objects, but not separate as different instances</p></li>\n<li><p>pixel accuracy - for object that are correctly identified and separated, the IOU</p></li>\n</ul>\n\n<p>I find the pixel accuracy extremely important - this is probably because many objects are quite small, and if you have an IOU of say 0.69, visually it looks great, but with the scoring formula you will have a true positive for IOU [0.5, 0.55, 0.6, 0.65], but you will also have a false negative and a false positive for IOUs [0.7, 0.75, 0.8, 0.85, 0.9, 0.95]. </p>\n\n<p>While the false-positives, false-negatives, overlaps are pretty obvious to spot and the idea is the same between train / test, how do you prevent overfitting on the pixel accuracy ? </p>\n\n<p>I've just tested a 'better' segmentation, with the same instances identification as previous submission, and I see a difference of -0.03 (while on my validation set I've had +0.02), which is pretty huge. </p>",
      "rawMarkdown": "Hi guys,\n\nwhat metrics do you use to measure the model output ? \n\nI'm going with a U-Net, so the task is naturally split in 2:\n\n- identify the masks with a simple U-Net\n\n- use post-processing to separate the object instances (in my case, I use another U-Net output for that)\n\n\nI've come up with the following analysis:\n\n- for each mask in an image, I calculate:\n\n   - iou with predicted masks\n\n   - coverage by predicted mask (ignoring the object instance)\n\n\nThere are 4 different losses that impact the score:\n\n- false-positives - false objects detected by the model\n\n- false-negatives - real objects, not detected by the model\n\n- overlaps - correctly identified mask of 2 or more adjacent objects, but not separate as different instances\n\n- pixel accuracy - for object that are correctly identified and separated, the IOU\n\n\nI find the pixel accuracy extremely important - this is probably because many objects are quite small, and if you have an IOU of say 0.69, visually it looks great, but with the scoring formula you will have a true positive for IOU [0.5, 0.55, 0.6, 0.65], but you will also have a false negative and a false positive for IOUs [0.7, 0.75, 0.8, 0.85, 0.9, 0.95]. \n\nWhile the false-positives, false-negatives, overlaps are pretty obvious to spot and the idea is the same between train / test, how do you prevent overfitting on the pixel accuracy ? \n\nI've just tested a 'better' segmentation, with the same instances identification as previous submission, and I see a difference of -0.03 (while on my validation set I've had +0.02), which is pretty huge. \n\n\n"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "310460": "Hi guys,\n\nwhat metrics do you use to measure the model output ? \n\nI'm going with a U-Net, so the task is naturally split in 2:\n\n- identify the masks with a simple U-Net\n\n- use post-processing to separate the object instances (in my case, I use another U-Net output for that)\n\n\nI've come up with the following analysis:\n\n- for each mask in an image, I calculate:\n\n   - iou with predicted masks\n\n   - coverage by predicted mask (ignoring the object instance)\n\n\nThere are 4 different losses that impact the score:\n\n- false-positives - false objects detected by the model\n\n- false-negatives - real objects, not detected by the model\n\n- overlaps - correctly identified mask of 2 or more adjacent objects, but not separate as different instances\n\n- pixel accuracy - for object that are correctly identified and separated, the IOU\n\n\nI find the pixel accuracy extremely important - this is probably because many objects are quite small, and if you have an IOU of say 0.69, visually it looks great, but with the scoring formula you will have a true positive for IOU [0.5, 0.55, 0.6, 0.65], but you will also have a false negative and a false positive for IOUs [0.7, 0.75, 0.8, 0.85, 0.9, 0.95]. \n\nWhile the false-positives, false-negatives, overlaps are pretty obvious to spot and the idea is the same between train / test, how do you prevent overfitting on the pixel accuracy ? \n\nI've just tested a 'better' segmentation, with the same instances identification as previous submission, and I see a difference of -0.03 (while on my validation set I've had +0.02), which is pretty huge. \n\n\n"
  }
}