My best single model so far has LB score 0.304. It is a simple UNet with flipping for data augmentation. There was no dropout. I used ELU instead of ReLU for my network as well. I've tried experimenting with dropout, but it doesn't seem to increase the score. I guess I'm still underfitting then. Here's the code for my model:
\n\ninputs = Input((128, 128, 3))\ns = Lambda(lambda x: x / 255) (inputs)\n\nc1 = Conv2D(16, (3, 3), activation='elu', padding='same') (s)\nc1 = Conv2D(16, (3, 3), activation='elu', padding='same') (c1)\np1 = MaxPooling2D((2, 2)) (c1)\n\nc2 = Conv2D(32, (3, 3), activation='elu', padding='same') (p1)\nc2 = Conv2D(32, (3, 3), activation='elu', padding='same') (c2)\np2 = MaxPooling2D((2, 2)) (c2)\n\nc3 = Conv2D(64, (3, 3), activation='elu', padding='same') (p2)\nc3 = Conv2D(64, (3, 3), activation='elu', padding='same') (c3)\np3 = MaxPooling2D((2, 2)) (c3)\n\nc4 = Conv2D(64, (3, 3), activation='elu', padding='same') (p3)\nc4 = Conv2D(64, (3, 3), activation='elu', padding='same') (c4)\np4 = MaxPooling2D(pool_size=(2, 2)) (c4)\n\nc5 = Conv2D(128, (3, 3), activation='elu', padding='same') (p4)\nc5 = Conv2D(128, (3, 3), activation='elu', padding='same') (c5)\n\n#Going up U-Net\n\nu6 = Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same') (c5)\nu6 = concatenate([u6, c4])\nc6 = Conv2D(64, (3, 3), activation='elu', padding='same') (u6)\nc6 = Conv2D(64, (3, 3), activation='elu', padding='same') (c6)\n\nu7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)\nu7 = concatenate([u7, c3])\nc7 = Conv2D(64, (3, 3), activation='elu', padding='same') (u7)\nc7 = Conv2D(64, (3, 3), activation='elu', padding='same') (c7)\n\nu8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)\nu8 = concatenate([u8, c2])\nc8 = Conv2D(32, (3, 3), activation='elu', padding='same') (u8)\nc8 = Conv2D(32, (3, 3), activation='elu', padding='same') (c8)\n\nu9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)\nu9 = concatenate([u9, c1], axis=3)\nc9 = Conv2D(16, (3, 3), activation='elu', padding='same') (u9)\nc9 = Conv2D(16, (3, 3), activation='elu', padding='same') (c9)\n\noutputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\nUNet10 = Model(inputs=[inputs], outputs=[outputs])\nI trained it for 35 epochs with a batch size of 8.
\n\nHopefully everyone can learn about what doesn't work and what can improve their models. Post whatever changes have been benefiting your models if you'd like!
", "messages": [ { "id": "271916", "postDate": "01/21/2018 21:27:41", "content": "My best single model so far has LB score 0.304. It is a simple UNet with flipping for data augmentation. There was no dropout. I used ELU instead of ReLU for my network as well. I've tried experimenting with dropout, but it doesn't seem to increase the score. I guess I'm still underfitting then. Here's the code for my model:
\n\ninputs = Input((128, 128, 3))\ns = Lambda(lambda x: x / 255) (inputs)\n\nc1 = Conv2D(16, (3, 3), activation='elu', padding='same') (s)\nc1 = Conv2D(16, (3, 3), activation='elu', padding='same') (c1)\np1 = MaxPooling2D((2, 2)) (c1)\n\nc2 = Conv2D(32, (3, 3), activation='elu', padding='same') (p1)\nc2 = Conv2D(32, (3, 3), activation='elu', padding='same') (c2)\np2 = MaxPooling2D((2, 2)) (c2)\n\nc3 = Conv2D(64, (3, 3), activation='elu', padding='same') (p2)\nc3 = Conv2D(64, (3, 3), activation='elu', padding='same') (c3)\np3 = MaxPooling2D((2, 2)) (c3)\n\nc4 = Conv2D(64, (3, 3), activation='elu', padding='same') (p3)\nc4 = Conv2D(64, (3, 3), activation='elu', padding='same') (c4)\np4 = MaxPooling2D(pool_size=(2, 2)) (c4)\n\nc5 = Conv2D(128, (3, 3), activation='elu', padding='same') (p4)\nc5 = Conv2D(128, (3, 3), activation='elu', padding='same') (c5)\n\n#Going up U-Net\n\nu6 = Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same') (c5)\nu6 = concatenate([u6, c4])\nc6 = Conv2D(64, (3, 3), activation='elu', padding='same') (u6)\nc6 = Conv2D(64, (3, 3), activation='elu', padding='same') (c6)\n\nu7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)\nu7 = concatenate([u7, c3])\nc7 = Conv2D(64, (3, 3), activation='elu', padding='same') (u7)\nc7 = Conv2D(64, (3, 3), activation='elu', padding='same') (c7)\n\nu8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)\nu8 = concatenate([u8, c2])\nc8 = Conv2D(32, (3, 3), activation='elu', padding='same') (u8)\nc8 = Conv2D(32, (3, 3), activation='elu', padding='same') (c8)\n\nu9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)\nu9 = concatenate([u9, c1], axis=3)\nc9 = Conv2D(16, (3, 3), activation='elu', padding='same') (u9)\nc9 = Conv2D(16, (3, 3), activation='elu', padding='same') (c9)\n\noutputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\nUNet10 = Model(inputs=[inputs], outputs=[outputs])\nI trained it for 35 epochs with a batch size of 8.
\n\nHopefully everyone can learn about what doesn't work and what can improve their models. Post whatever changes have been benefiting your models if you'd like!
", "rawMarkdown": "My best single model so far has LB score 0.304. It is a simple UNet with flipping for data augmentation. There was no dropout. I used ELU instead of ReLU for my network as well. I've tried experimenting with dropout, but it doesn't seem to increase the score. I guess I'm still underfitting then. Here's the code for my model:\n\n inputs = Input((128, 128, 3))\n s = Lambda(lambda x: x / 255) (inputs)\n \n c1 = Conv2D(16, (3, 3), activation='elu', padding='same') (s)\n c1 = Conv2D(16, (3, 3), activation='elu', padding='same') (c1)\n p1 = MaxPooling2D((2, 2)) (c1)\n \n c2 = Conv2D(32, (3, 3), activation='elu', padding='same') (p1)\n c2 = Conv2D(32, (3, 3), activation='elu', padding='same') (c2)\n p2 = MaxPooling2D((2, 2)) (c2)\n \n c3 = Conv2D(64, (3, 3), activation='elu', padding='same') (p2)\n c3 = Conv2D(64, (3, 3), activation='elu', padding='same') (c3)\n p3 = MaxPooling2D((2, 2)) (c3)\n \n c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (p3)\n c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (c4)\n p4 = MaxPooling2D(pool_size=(2, 2)) (c4)\n \n c5 = Conv2D(128, (3, 3), activation='elu', padding='same') (p4)\n c5 = Conv2D(128, (3, 3), activation='elu', padding='same') (c5)\n \n #Going up U-Net\n \n u6 = Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same') (c5)\n u6 = concatenate([u6, c4])\n c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (u6)\n c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (c6)\n \n u7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)\n u7 = concatenate([u7, c3])\n c7 = Conv2D(64, (3, 3), activation='elu', padding='same') (u7)\n c7 = Conv2D(64, (3, 3), activation='elu', padding='same') (c7)\n \n u8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)\n u8 = concatenate([u8, c2])\n c8 = Conv2D(32, (3, 3), activation='elu', padding='same') (u8)\n c8 = Conv2D(32, (3, 3), activation='elu', padding='same') (c8)\n \n u9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)\n u9 = concatenate([u9, c1], axis=3)\n c9 = Conv2D(16, (3, 3), activation='elu', padding='same') (u9)\n c9 = Conv2D(16, (3, 3), activation='elu', padding='same') (c9)\n \n outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n UNet10 = Model(inputs=[inputs], outputs=[outputs])\n\nI trained it for 35 epochs with a batch size of 8.\n\nHopefully everyone can learn about what doesn't work and what can improve their models. Post whatever changes have been benefiting your models if you'd like!", "votes": null }, { "id": "272234", "postDate": "01/22/2018 15:38:55", "content": "Using bilinear upsampling instead of deconv worked well for me.\nAnd increasing the number of filters can work as well.
", "rawMarkdown": "Using bilinear upsampling instead of deconv worked well for me.\nAnd increasing the number of filters can work as well.", "votes": null }, { "id": "272274", "postDate": "01/22/2018 17:36:11", "content": "Is there a Keras implementation of it? I would like to try it
", "rawMarkdown": "Is there a Keras implementation of it? I would like to try it", "votes": null }, { "id": "272284", "postDate": "01/22/2018 18:04:37", "content": "In Keras, I'm afraid that Bilinear Upsampling isn't implemented officially. Although unofficially implementation, I found out this: https://github.com/JihongJu/keras-fcn/blob/master/keras_fcn/layers.py
\n\nIn another way, you can use Upsampling2D() of Keras. You can use it in place of bilinear upsampling.
", "rawMarkdown": "In Keras, I'm afraid that Bilinear Upsampling isn't implemented officially. Although unofficially implementation, I found out this: https://github.com/JihongJu/keras-fcn/blob/master/keras_fcn/layers.py\n\nIn another way, you can use Upsampling2D() of Keras. You can use it in place of bilinear upsampling.", "votes": null }, { "id": "278313", "postDate": "02/05/2018 21:49:44", "content": "Hey,
\n\nAt this point my best single model does LB 0.339. It is adapted U-Net, training took 5h.
", "rawMarkdown": "Hey,\n\nAt this point my best single model does LB 0.339. It is adapted U-Net, training took 5h.", "votes": null }, { "id": "278513", "postDate": "02/06/2018 11:19:35", "content": "I'm curious why your U-Net takes 5 hours to train. Are you training on CPU? Or are you doing something special with your architecture? Or do you train for 100s of epochs? (For comparison, I use a GPU and U-Net takes 3 seconds per epoch on this dataset.)
", "rawMarkdown": "I'm curious why your U-Net takes 5 hours to train. Are you training on CPU? Or are you doing something special with your architecture? Or do you train for 100s of epochs? (For comparison, I use a GPU and U-Net takes 3 seconds per epoch on this dataset.)", "votes": null }, { "id": "278554", "postDate": "02/06/2018 13:21:54", "content": "Hi,
\n\nI train on single GPU. I use 256x256 images, batch size is 16. U-Net is adapted in such a way that I use 4 repeat blocks and initial number of convolutions is 16. I run it for 250 epochs. You can actually easily reproduce this result using this source code.
", "rawMarkdown": "Hi,\n\nI train on single GPU. I use 256x256 images, batch size is 16. U-Net is adapted in such a way that I use 4 repeat blocks and initial number of convolutions is 16. I run it for 250 epochs. You can actually easily reproduce this result using this [source code](https://github.com/neptune-ml/data-science-bowl-2018).", "votes": null }, { "id": "278678", "postDate": "02/06/2018 18:01:34", "content": "I've also tried experimenting with dropout and it doesn't help but slows down the learning process. Batchnormalization does increase the score a little bit and stabilizes the loss decreasing while I'm using Nadam optimizer.
", "rawMarkdown": "I've also tried experimenting with dropout and it doesn't help but slows down the learning process. Batchnormalization does increase the score a little bit and stabilizes the loss decreasing while I'm using Nadam optimizer.", "votes": null }, { "id": "366106", "postDate": "08/04/2018 02:50:25", "content": "Thanks for posting this. It was helpful for me to see how others tweak their U-NET.
", "rawMarkdown": "Thanks for posting this. It was helpful for me to see how others tweak their U-NET.", "votes": null } ], "comments": [ { "id": 272234, "author_name": "lyakaap", "author_url": "", "post_date": "01/22/2018 15:38:55", "content": "Using bilinear upsampling instead of deconv worked well for me.\nAnd increasing the number of filters can work as well.
", "votes": null, "replies": [ { "id": 272274, "author_name": "arpandhatt", "author_url": "", "post_date": "01/22/2018 17:36:11", "content": "Is there a Keras implementation of it? I would like to try it
", "votes": null, "replies": [] }, { "id": 272284, "author_name": "lyakaap", "author_url": "", "post_date": "01/22/2018 18:04:37", "content": "In Keras, I'm afraid that Bilinear Upsampling isn't implemented officially. Although unofficially implementation, I found out this: https://github.com/JihongJu/keras-fcn/blob/master/keras_fcn/layers.py
\n\nIn another way, you can use Upsampling2D() of Keras. You can use it in place of bilinear upsampling.
", "votes": null, "replies": [] } ] }, { "id": 278313, "author_name": "kkaczmarek", "author_url": "", "post_date": "02/05/2018 21:49:44", "content": "Hey,
\n\nAt this point my best single model does LB 0.339. It is adapted U-Net, training took 5h.
", "votes": null, "replies": [ { "id": 278513, "author_name": "humananalog", "author_url": "", "post_date": "02/06/2018 11:19:35", "content": "I'm curious why your U-Net takes 5 hours to train. Are you training on CPU? Or are you doing something special with your architecture? Or do you train for 100s of epochs? (For comparison, I use a GPU and U-Net takes 3 seconds per epoch on this dataset.)
", "votes": null, "replies": [] }, { "id": 278554, "author_name": "kkaczmarek", "author_url": "", "post_date": "02/06/2018 13:21:54", "content": "Hi,
\n\nI train on single GPU. I use 256x256 images, batch size is 16. U-Net is adapted in such a way that I use 4 repeat blocks and initial number of convolutions is 16. I run it for 250 epochs. You can actually easily reproduce this result using this source code.
", "votes": null, "replies": [] } ] }, { "id": 278678, "author_name": "syoya1997", "author_url": "", "post_date": "02/06/2018 18:01:34", "content": "I've also tried experimenting with dropout and it doesn't help but slows down the learning process. Batchnormalization does increase the score a little bit and stabilizes the loss decreasing while I'm using Nadam optimizer.
", "votes": null, "replies": [] }, { "id": 366106, "author_name": "flagshipdynamics", "author_url": "", "post_date": "08/04/2018 02:50:25", "content": "Thanks for posting this. It was helpful for me to see how others tweak their U-NET.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "271916": "My best single model so far has LB score 0.304. It is a simple UNet with flipping for data augmentation. There was no dropout. I used ELU instead of ReLU for my network as well. I've tried experimenting with dropout, but it doesn't seem to increase the score. I guess I'm still underfitting then. Here's the code for my model:\n\n inputs = Input((128, 128, 3))\n s = Lambda(lambda x: x / 255) (inputs)\n \n c1 = Conv2D(16, (3, 3), activation='elu', padding='same') (s)\n c1 = Conv2D(16, (3, 3), activation='elu', padding='same') (c1)\n p1 = MaxPooling2D((2, 2)) (c1)\n \n c2 = Conv2D(32, (3, 3), activation='elu', padding='same') (p1)\n c2 = Conv2D(32, (3, 3), activation='elu', padding='same') (c2)\n p2 = MaxPooling2D((2, 2)) (c2)\n \n c3 = Conv2D(64, (3, 3), activation='elu', padding='same') (p2)\n c3 = Conv2D(64, (3, 3), activation='elu', padding='same') (c3)\n p3 = MaxPooling2D((2, 2)) (c3)\n \n c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (p3)\n c4 = Conv2D(64, (3, 3), activation='elu', padding='same') (c4)\n p4 = MaxPooling2D(pool_size=(2, 2)) (c4)\n \n c5 = Conv2D(128, (3, 3), activation='elu', padding='same') (p4)\n c5 = Conv2D(128, (3, 3), activation='elu', padding='same') (c5)\n \n #Going up U-Net\n \n u6 = Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same') (c5)\n u6 = concatenate([u6, c4])\n c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (u6)\n c6 = Conv2D(64, (3, 3), activation='elu', padding='same') (c6)\n \n u7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)\n u7 = concatenate([u7, c3])\n c7 = Conv2D(64, (3, 3), activation='elu', padding='same') (u7)\n c7 = Conv2D(64, (3, 3), activation='elu', padding='same') (c7)\n \n u8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)\n u8 = concatenate([u8, c2])\n c8 = Conv2D(32, (3, 3), activation='elu', padding='same') (u8)\n c8 = Conv2D(32, (3, 3), activation='elu', padding='same') (c8)\n \n u9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)\n u9 = concatenate([u9, c1], axis=3)\n c9 = Conv2D(16, (3, 3), activation='elu', padding='same') (u9)\n c9 = Conv2D(16, (3, 3), activation='elu', padding='same') (c9)\n \n outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)\n UNet10 = Model(inputs=[inputs], outputs=[outputs])\n\nI trained it for 35 epochs with a batch size of 8.\n\nHopefully everyone can learn about what doesn't work and what can improve their models. Post whatever changes have been benefiting your models if you'd like!", "272234": "Using bilinear upsampling instead of deconv worked well for me.\nAnd increasing the number of filters can work as well.", "272274": "Is there a Keras implementation of it? I would like to try it", "272284": "In Keras, I'm afraid that Bilinear Upsampling isn't implemented officially. Although unofficially implementation, I found out this: https://github.com/JihongJu/keras-fcn/blob/master/keras_fcn/layers.py\n\nIn another way, you can use Upsampling2D() of Keras. You can use it in place of bilinear upsampling.", "278313": "Hey,\n\nAt this point my best single model does LB 0.339. It is adapted U-Net, training took 5h.", "278513": "I'm curious why your U-Net takes 5 hours to train. Are you training on CPU? Or are you doing something special with your architecture? Or do you train for 100s of epochs? (For comparison, I use a GPU and U-Net takes 3 seconds per epoch on this dataset.)", "278554": "Hi,\n\nI train on single GPU. I use 256x256 images, batch size is 16. U-Net is adapted in such a way that I use 4 repeat blocks and initial number of convolutions is 16. I run it for 250 epochs. You can actually easily reproduce this result using this [source code](https://github.com/neptune-ml/data-science-bowl-2018).", "278678": "I've also tried experimenting with dropout and it doesn't help but slows down the learning process. Batchnormalization does increase the score a little bit and stabilizes the loss decreasing while I'm using Nadam optimizer.", "366106": "Thanks for posting this. It was helpful for me to see how others tweak their U-NET." }, "source": "meta" }