I have implemented unet in tensor flow (using tf estimator & tf layers). The network architecture is exactly as shared in this kernel - https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277.
\n\nI see that just after 30 steps (each of batch size = 16) the sigmoid activations are getting saturated. Using he_normal kernel initialiser did not have any impact.
\n\nPlease suggest things to try out.
", "messages": [ { "id": "274393", "postDate": "01/26/2018 14:33:48", "content": "I have implemented unet in tensor flow (using tf estimator & tf layers). The network architecture is exactly as shared in this kernel - https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277.
\n\nI see that just after 30 steps (each of batch size = 16) the sigmoid activations are getting saturated. Using he_normal kernel initialiser did not have any impact.
\n\nPlease suggest things to try out.
", "rawMarkdown": "I have implemented unet in tensor flow (using tf estimator & tf layers). The network architecture is exactly as shared in this kernel - https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277.\n\nI see that just after 30 steps (each of batch size = 16) the sigmoid activations are getting saturated. Using he_normal kernel initialiser did not have any impact.\n\nPlease suggest things to try out.", "votes": null }, { "id": "274559", "postDate": "01/26/2018 19:44:23", "content": "Identified the problem. I was using tf.losses.sigmoid_cross_entropy() for loss calculation, but passing sigmoid activation instead of the logits to this function. Once i changed this and started passing logits, the sigmoid activation saturation (aka vanishing gradient) problem is resolved.
", "rawMarkdown": "Identified the problem. I was using tf.losses.sigmoid_cross_entropy() for loss calculation, but passing sigmoid activation instead of the logits to this function. Once i changed this and started passing logits, the sigmoid activation saturation (aka vanishing gradient) problem is resolved.", "votes": null }, { "id": "274762", "postDate": "01/27/2018 08:15:53", "content": "hi, I have the same problem but I didn't understand the solution. Can you please explain more? What does \"passing sigmoid activation instead of the logits\" mean?
", "rawMarkdown": "hi, I have the same problem but I didn't understand the solution. Can you please explain more? What does \"passing sigmoid activation instead of the logits\" mean?", "votes": null }, { "id": "274803", "postDate": "01/27/2018 11:17:04", "content": "hi,
\n\noutputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n\nmodel = Model(inputs=[inputs], outputs=[outputs])\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=[mean_iou])\nmodel.summary()\nThis snippet is from the above keras kernel. Here, the \"outputs\" are the sigmoid activations. The last block in unet does conv2d and then sigmoid activation. Lets hypothetically split it into 2 parts. 1st part that does just conv2d and 2nd part that does sigmoid activation. The output of the 1st part, which is passed as input to the 2nd part is called the logits. Same can be written in keras as below:
\n\nlogits = Conv2D(1, (1, 1), activation=None) (c9)\noutputs = sigmoid(logits)\nIn Keras, binary_crossentropy loss function takes sigmoid activations (aka 'outputs') as an argument. Whereas in tensorflow, tf.losses.sigmoid_cross_entropy() takes logits as an argument but not sigmoid activations. This was clear to me only after going through the keras source code. so, once I split my implementation into 2 parts as shown above and passed logits to the loss function, the above problem resolved.
\n\nhope, it helps.
", "rawMarkdown": "hi,\n\n outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n \n model = Model(inputs=[inputs], outputs=[outputs])\n model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[mean_iou])\n model.summary()\n\nThis snippet is from the above keras kernel. Here, the \"outputs\" are the sigmoid activations. The last block in unet does conv2d and then sigmoid activation. Lets hypothetically split it into 2 parts. 1st part that does just conv2d and 2nd part that does sigmoid activation. The output of the 1st part, which is passed as input to the 2nd part is called the logits. Same can be written in keras as below:\n\n logits = Conv2D(1, (1, 1), activation=None) (c9)\n outputs = sigmoid(logits)\n\nIn Keras, binary_crossentropy loss function takes sigmoid activations (aka 'outputs') as an argument. Whereas in tensorflow, tf.losses.sigmoid_cross_entropy() takes logits as an argument but not sigmoid activations. This was clear to me only after going through the keras source code. so, once I split my implementation into 2 parts as shown above and passed logits to the loss function, the above problem resolved.\n\nhope, it helps.", "votes": null }, { "id": "274812", "postDate": "01/27/2018 12:15:51", "content": "ok, I see it now. thanks for the detailed explaination.
", "rawMarkdown": "ok, I see it now. thanks for the detailed explaination.", "votes": null } ], "comments": [ { "id": 274559, "author_name": "ravivadapalli", "author_url": "", "post_date": "01/26/2018 19:44:23", "content": "Identified the problem. I was using tf.losses.sigmoid_cross_entropy() for loss calculation, but passing sigmoid activation instead of the logits to this function. Once i changed this and started passing logits, the sigmoid activation saturation (aka vanishing gradient) problem is resolved.
", "votes": null, "replies": [ { "id": 274762, "author_name": "armamut", "author_url": "", "post_date": "01/27/2018 08:15:53", "content": "hi, I have the same problem but I didn't understand the solution. Can you please explain more? What does \"passing sigmoid activation instead of the logits\" mean?
", "votes": null, "replies": [] }, { "id": 274803, "author_name": "ravivadapalli", "author_url": "", "post_date": "01/27/2018 11:17:04", "content": "hi,
\n\noutputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n\nmodel = Model(inputs=[inputs], outputs=[outputs])\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=[mean_iou])\nmodel.summary()\nThis snippet is from the above keras kernel. Here, the \"outputs\" are the sigmoid activations. The last block in unet does conv2d and then sigmoid activation. Lets hypothetically split it into 2 parts. 1st part that does just conv2d and 2nd part that does sigmoid activation. The output of the 1st part, which is passed as input to the 2nd part is called the logits. Same can be written in keras as below:
\n\nlogits = Conv2D(1, (1, 1), activation=None) (c9)\noutputs = sigmoid(logits)\nIn Keras, binary_crossentropy loss function takes sigmoid activations (aka 'outputs') as an argument. Whereas in tensorflow, tf.losses.sigmoid_cross_entropy() takes logits as an argument but not sigmoid activations. This was clear to me only after going through the keras source code. so, once I split my implementation into 2 parts as shown above and passed logits to the loss function, the above problem resolved.
\n\nhope, it helps.
", "votes": null, "replies": [] }, { "id": 274812, "author_name": "armamut", "author_url": "", "post_date": "01/27/2018 12:15:51", "content": "ok, I see it now. thanks for the detailed explaination.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "274393": "I have implemented unet in tensor flow (using tf estimator & tf layers). The network architecture is exactly as shared in this kernel - https://www.kaggle.com/keegil/keras-u-net-starter-lb-0-277.\n\nI see that just after 30 steps (each of batch size = 16) the sigmoid activations are getting saturated. Using he_normal kernel initialiser did not have any impact.\n\nPlease suggest things to try out.", "274559": "Identified the problem. I was using tf.losses.sigmoid_cross_entropy() for loss calculation, but passing sigmoid activation instead of the logits to this function. Once i changed this and started passing logits, the sigmoid activation saturation (aka vanishing gradient) problem is resolved.", "274762": "hi, I have the same problem but I didn't understand the solution. Can you please explain more? What does \"passing sigmoid activation instead of the logits\" mean?", "274803": "hi,\n\n outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n \n model = Model(inputs=[inputs], outputs=[outputs])\n model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[mean_iou])\n model.summary()\n\nThis snippet is from the above keras kernel. Here, the \"outputs\" are the sigmoid activations. The last block in unet does conv2d and then sigmoid activation. Lets hypothetically split it into 2 parts. 1st part that does just conv2d and 2nd part that does sigmoid activation. The output of the 1st part, which is passed as input to the 2nd part is called the logits. Same can be written in keras as below:\n\n logits = Conv2D(1, (1, 1), activation=None) (c9)\n outputs = sigmoid(logits)\n\nIn Keras, binary_crossentropy loss function takes sigmoid activations (aka 'outputs') as an argument. Whereas in tensorflow, tf.losses.sigmoid_cross_entropy() takes logits as an argument but not sigmoid activations. This was clear to me only after going through the keras source code. so, once I split my implementation into 2 parts as shown above and passed logits to the loss function, the above problem resolved.\n\nhope, it helps.", "274812": "ok, I see it now. thanks for the detailed explaination." }, "source": "meta" }