{
  "id": 48490,
  "title": "Loss is minimized, but plot results are completely wrong",
  "url": "/competitions/data-science-bowl-2018/discussion/48490",
  "author_name": "",
  "post_date": "2018-01-28T16:40:53.784469300Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi all, \nI have run into this problem after trying a Unet variation with Tensorflow. My loss appears to be minimized on both Train and Val set, using binary cross-entropy as well as the IOU value increasing up to the early stop point. The problem is when I run an image through the trained model get the gibberish output seen below. My question to this awesome community is if you think this is a plotting/model running bug, or if it could be that the whole model architecture is in correct. An example is shown below, which was run live during a session epoch. Loss claimed to be .03, IOU .55(not sure if this implementation is correct). (Image order-Image, expected mask, model output)\nHere is the code for running the graphs(y = output tensor, x_= input tensor, batch[0] = 10 batched  inputs, batch[1] =10 batch of correct masks):\n<strong>UPDATE</strong>\nAfter closer examination it appears that the train-step is running the outputs all down to zero although the y_true data appears correct against the logits. When all of the outputs tesnor are zero/black loss is described as 0, using binary_crossentropy</p>\n\n<p><code>\n    pred = sess.run(y, feed_dict={x_: batch[0], keep_prob: 1.0}) <br>\n    mask = (pred &amp;gt; .5).astype(np.uint8) <br>\n    imshow(batch[0][1]) <br>\n    plt.show() <br>\n    imshow(np.squeeze(batch[1][1])) <br>\n    plt.show() <br>\n    imshow(np.squeeze(mask[1])) <br>\n    plt.show() <br>\n</code></p>",
  "messages": [
    {
      "id": "275284",
      "postDate": "01/28/2018 16:40:53",
      "content": "<p>Hi all, \nI have run into this problem after trying a Unet variation with Tensorflow. My loss appears to be minimized on both Train and Val set, using binary cross-entropy as well as the IOU value increasing up to the early stop point. The problem is when I run an image through the trained model get the gibberish output seen below. My question to this awesome community is if you think this is a plotting/model running bug, or if it could be that the whole model architecture is in correct. An example is shown below, which was run live during a session epoch. Loss claimed to be .03, IOU .55(not sure if this implementation is correct). (Image order-Image, expected mask, model output)\nHere is the code for running the graphs(y = output tensor, x_= input tensor, batch[0] = 10 batched  inputs, batch[1] =10 batch of correct masks):\n<strong>UPDATE</strong>\nAfter closer examination it appears that the train-step is running the outputs all down to zero although the y_true data appears correct against the logits. When all of the outputs tesnor are zero/black loss is described as 0, using binary_crossentropy</p>\n\n<p><code>\n    pred = sess.run(y, feed_dict={x_: batch[0], keep_prob: 1.0}) <br>\n    mask = (pred &amp;gt; .5).astype(np.uint8) <br>\n    imshow(batch[0][1]) <br>\n    plt.show() <br>\n    imshow(np.squeeze(batch[1][1])) <br>\n    plt.show() <br>\n    imshow(np.squeeze(mask[1])) <br>\n    plt.show() <br>\n</code></p>",
      "rawMarkdown": "Hi all, \nI have run into this problem after trying a Unet variation with Tensorflow. My loss appears to be minimized on both Train and Val set, using binary cross-entropy as well as the IOU value increasing up to the early stop point. The problem is when I run an image through the trained model get the gibberish output seen below. My question to this awesome community is if you think this is a plotting/model running bug, or if it could be that the whole model architecture is in correct. An example is shown below, which was run live during a session epoch. Loss claimed to be .03, IOU .55(not sure if this implementation is correct). (Image order-Image, expected mask, model output)\nHere is the code for running the graphs(y = output tensor, x_= input tensor, batch[0] = 10 batched  inputs, batch[1] =10 batch of correct masks):\n**UPDATE**\nAfter closer examination it appears that the train-step is running the outputs all down to zero although the y_true data appears correct against the logits. When all of the outputs tesnor are zero/black loss is described as 0, using binary_crossentropy\n\n``` \n    pred = sess.run(y, feed_dict={x_: batch[0], keep_prob: 1.0})  \n    mask = (pred &gt; .5).astype(np.uint8)  \n    imshow(batch[0][1])  \n    plt.show()  \n    imshow(np.squeeze(batch[1][1]))  \n    plt.show()  \n    imshow(np.squeeze(mask[1]))  \n    plt.show()  \n```",
      "votes": null
    },
    {
      "id": "275327",
      "postDate": "01/28/2018 20:41:38",
      "content": "<p>@Ian, based on the regular pattern in the results, the first thing I'd probably look for is a reshaping or broadcasting bug of some sort.  Sorry, that's not very specific, but maybe it'll help.</p>",
      "rawMarkdown": "Ian, based on the regular pattern in the results, the first thing I'd probably look for is a reshaping or broadcasting bug of some sort.  Sorry, that's not very specific, but maybe it'll help.",
      "votes": null
    },
    {
      "id": "275817",
      "postDate": "01/30/2018 03:19:10",
      "content": "<p>Thanks @Tim, i looked very closely into my reshaping and have been trying to figure out the broadcasting. It appears that AdamOptimizer  backprop train step is attempting to send all output to zero which is considers a zero 'loss' Ive looked deep into my graph and cant find any feed or session errors.</p>",
      "rawMarkdown": "Thanks @Tim, i looked very closely into my reshaping and have been trying to figure out the broadcasting. It appears that AdamOptimizer  backprop train step is attempting to send all output to zero which is considers a zero 'loss' Ive looked deep into my graph and cant find any feed or session errors.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 275327,
      "author_name": "bitsofbits",
      "author_url": "",
      "post_date": "01/28/2018 20:41:38",
      "content": "<p>@Ian, based on the regular pattern in the results, the first thing I'd probably look for is a reshaping or broadcasting bug of some sort.  Sorry, that's not very specific, but maybe it'll help.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 275817,
      "author_name": "ianrowan",
      "author_url": "",
      "post_date": "01/30/2018 03:19:10",
      "content": "<p>Thanks @Tim, i looked very closely into my reshaping and have been trying to figure out the broadcasting. It appears that AdamOptimizer  backprop train step is attempting to send all output to zero which is considers a zero 'loss' Ive looked deep into my graph and cant find any feed or session errors.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "275284": "Hi all, \nI have run into this problem after trying a Unet variation with Tensorflow. My loss appears to be minimized on both Train and Val set, using binary cross-entropy as well as the IOU value increasing up to the early stop point. The problem is when I run an image through the trained model get the gibberish output seen below. My question to this awesome community is if you think this is a plotting/model running bug, or if it could be that the whole model architecture is in correct. An example is shown below, which was run live during a session epoch. Loss claimed to be .03, IOU .55(not sure if this implementation is correct). (Image order-Image, expected mask, model output)\nHere is the code for running the graphs(y = output tensor, x_= input tensor, batch[0] = 10 batched  inputs, batch[1] =10 batch of correct masks):\n**UPDATE**\nAfter closer examination it appears that the train-step is running the outputs all down to zero although the y_true data appears correct against the logits. When all of the outputs tesnor are zero/black loss is described as 0, using binary_crossentropy\n\n``` \n    pred = sess.run(y, feed_dict={x_: batch[0], keep_prob: 1.0})  \n    mask = (pred &gt; .5).astype(np.uint8)  \n    imshow(batch[0][1])  \n    plt.show()  \n    imshow(np.squeeze(batch[1][1]))  \n    plt.show()  \n    imshow(np.squeeze(mask[1]))  \n    plt.show()  \n```",
    "275327": "Ian, based on the regular pattern in the results, the first thing I'd probably look for is a reshaping or broadcasting bug of some sort.  Sorry, that's not very specific, but maybe it'll help.",
    "275817": "Thanks @Tim, i looked very closely into my reshaping and have been trying to figure out the broadcasting. It appears that AdamOptimizer  backprop train step is attempting to send all output to zero which is considers a zero 'loss' Ive looked deep into my graph and cant find any feed or session errors."
  },
  "source": "meta"
}