{
  "id": 52466,
  "title": "Batch Normalization giving poor results",
  "url": "/competitions/data-science-bowl-2018/discussion/52466",
  "author_name": "",
  "post_date": "2018-03-20T13:26:12.145520700Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>I have implemented basic unet arch in keras with drop out and L2 reg. Tried it with and without batch normalization.</p>\n\n<p><strong>Without batch normalization</strong>: validation loss, map &amp; miou are slightly worse than their training counterparts.</p>\n\n<p><strong>With batch normalization</strong>: loss followed the same pattern as w/o batch norm. But the validation map &amp; miou are close to zero while the training map &amp; miou are around 0.7 and 0.8.</p>\n\n<p>How can we explain the difference in the training and validation map &amp; miou with batch norm? Do i need to take extra care with batch norm? Am I missing something?</p>\n\n<p>Please share ur thoughts. </p>",
  "messages": [
    {
      "id": "299099",
      "postDate": "03/20/2018 13:26:12",
      "content": "<p>Hi,</p>\n\n<p>I have implemented basic unet arch in keras with drop out and L2 reg. Tried it with and without batch normalization.</p>\n\n<p><strong>Without batch normalization</strong>: validation loss, map &amp; miou are slightly worse than their training counterparts.</p>\n\n<p><strong>With batch normalization</strong>: loss followed the same pattern as w/o batch norm. But the validation map &amp; miou are close to zero while the training map &amp; miou are around 0.7 and 0.8.</p>\n\n<p>How can we explain the difference in the training and validation map &amp; miou with batch norm? Do i need to take extra care with batch norm? Am I missing something?</p>\n\n<p>Please share ur thoughts. </p>",
      "rawMarkdown": "Hi,\n\nI have implemented basic unet arch in keras with drop out and L2 reg. Tried it with and without batch normalization.\n\n**Without batch normalization**: validation loss, map &amp; miou are slightly worse than their training counterparts.\n\n**With batch normalization**: loss followed the same pattern as w/o batch norm. But the validation map &amp; miou are close to zero while the training map &amp; miou are around 0.7 and 0.8.\n\nHow can we explain the difference in the training and validation map &amp; miou with batch norm? Do i need to take extra care with batch norm? Am I missing something?\n\nPlease share ur thoughts.",
      "votes": null
    },
    {
      "id": "299170",
      "postDate": "03/20/2018 16:15:42",
      "content": "<p>I have very little experience using batch norm but I think we use weighted loss in case of batch norm where the weight equals to batch size. I hope you are taking care of that.</p>",
      "rawMarkdown": "I have very little experience using batch norm but I think we use weighted loss in case of batch norm where the weight equals to batch size. I hope you are taking care of that.",
      "votes": null
    },
    {
      "id": "299237",
      "postDate": "03/20/2018 17:30:54",
      "content": "<p>Thanks Vijay for your input.</p>\n\n<p>No, I have not changed the loss function. In case you have it handy, can you share a link to an article/kernel that explains the usage of weighted loss with batch norm.</p>",
      "rawMarkdown": "Thanks Vijay for your input.\n\nNo, I have not changed the loss function. In case you have it handy, can you share a link to an article/kernel that explains the usage of weighted loss with batch norm.",
      "votes": null
    },
    {
      "id": "299716",
      "postDate": "03/21/2018 02:44:19",
      "content": "<p>My mistake... I meant to say loss normalisation( scaling of loss ) which may not be helpful in your case.</p>",
      "rawMarkdown": "My mistake... I meant to say loss normalisation( scaling of loss ) which may not be helpful in your case.",
      "votes": null
    },
    {
      "id": "299739",
      "postDate": "03/21/2018 03:15:29",
      "content": "<p>Like setting the weights parameter of this function to batch size.</p>\n\n<p>tf.losses.sigmoid_cross_entropy(\n    multi_class_labels,\n    logits,\n    weights=1.0,\n    label_smoothing=0,\n    scope=None,\n    loss_collection=tf.GraphKeys.LOSSES,\n    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS\n)</p>",
      "rawMarkdown": "Like setting the weights parameter of this function to batch size.\n\ntf.losses.sigmoid_cross_entropy(\n    multi_class_labels,\n    logits,\n    weights=1.0,\n    label_smoothing=0,\n    scope=None,\n    loss_collection=tf.GraphKeys.LOSSES,\n    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS\n)",
      "votes": null
    },
    {
      "id": "299740",
      "postDate": "03/21/2018 03:15:45",
      "content": "<p>oh ok.</p>",
      "rawMarkdown": "oh ok.",
      "votes": null
    },
    {
      "id": "301717",
      "postDate": "03/23/2018 06:34:28",
      "content": "<p>There was a bug in the way I was passing the inputs to the network. Because of the bug, the inputs were divided twice by 255. Once this is fixed (ie divide the inputs only once by 255), batch normalization started giving faster convergence and better results. </p>\n\n<p>Not very sure how this double division is causing problems with BN.</p>",
      "rawMarkdown": "There was a bug in the way I was passing the inputs to the network. Because of the bug, the inputs were divided twice by 255. Once this is fixed (ie divide the inputs only once by 255), batch normalization started giving faster convergence and better results. \n\nNot very sure how this double division is causing problems with BN.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 299170,
      "author_name": "vijaybj",
      "author_url": "",
      "post_date": "03/20/2018 16:15:42",
      "content": "<p>I have very little experience using batch norm but I think we use weighted loss in case of batch norm where the weight equals to batch size. I hope you are taking care of that.</p>",
      "votes": null,
      "replies": [
        {
          "id": 299237,
          "author_name": "ravivadapalli",
          "author_url": "",
          "post_date": "03/20/2018 17:30:54",
          "content": "<p>Thanks Vijay for your input.</p>\n\n<p>No, I have not changed the loss function. In case you have it handy, can you share a link to an article/kernel that explains the usage of weighted loss with batch norm.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 299716,
          "author_name": "vijaybj",
          "author_url": "",
          "post_date": "03/21/2018 02:44:19",
          "content": "<p>My mistake... I meant to say loss normalisation( scaling of loss ) which may not be helpful in your case.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 299739,
          "author_name": "vijaybj",
          "author_url": "",
          "post_date": "03/21/2018 03:15:29",
          "content": "<p>Like setting the weights parameter of this function to batch size.</p>\n\n<p>tf.losses.sigmoid_cross_entropy(\n    multi_class_labels,\n    logits,\n    weights=1.0,\n    label_smoothing=0,\n    scope=None,\n    loss_collection=tf.GraphKeys.LOSSES,\n    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS\n)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 299740,
          "author_name": "ravivadapalli",
          "author_url": "",
          "post_date": "03/21/2018 03:15:45",
          "content": "<p>oh ok.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 301717,
      "author_name": "ravivadapalli",
      "author_url": "",
      "post_date": "03/23/2018 06:34:28",
      "content": "<p>There was a bug in the way I was passing the inputs to the network. Because of the bug, the inputs were divided twice by 255. Once this is fixed (ie divide the inputs only once by 255), batch normalization started giving faster convergence and better results. </p>\n\n<p>Not very sure how this double division is causing problems with BN.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "299099": "Hi,\n\nI have implemented basic unet arch in keras with drop out and L2 reg. Tried it with and without batch normalization.\n\n**Without batch normalization**: validation loss, map &amp; miou are slightly worse than their training counterparts.\n\n**With batch normalization**: loss followed the same pattern as w/o batch norm. But the validation map &amp; miou are close to zero while the training map &amp; miou are around 0.7 and 0.8.\n\nHow can we explain the difference in the training and validation map &amp; miou with batch norm? Do i need to take extra care with batch norm? Am I missing something?\n\nPlease share ur thoughts.",
    "299170": "I have very little experience using batch norm but I think we use weighted loss in case of batch norm where the weight equals to batch size. I hope you are taking care of that.",
    "299237": "Thanks Vijay for your input.\n\nNo, I have not changed the loss function. In case you have it handy, can you share a link to an article/kernel that explains the usage of weighted loss with batch norm.",
    "299716": "My mistake... I meant to say loss normalisation( scaling of loss ) which may not be helpful in your case.",
    "299739": "Like setting the weights parameter of this function to batch size.\n\ntf.losses.sigmoid_cross_entropy(\n    multi_class_labels,\n    logits,\n    weights=1.0,\n    label_smoothing=0,\n    scope=None,\n    loss_collection=tf.GraphKeys.LOSSES,\n    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS\n)",
    "299740": "oh ok.",
    "301717": "There was a bug in the way I was passing the inputs to the network. Because of the bug, the inputs were divided twice by 255. Once this is fixed (ie divide the inputs only once by 255), batch normalization started giving faster convergence and better results. \n\nNot very sure how this double division is causing problems with BN."
  },
  "source": "meta"
}