I'm just learning to squeeze the most out of training CNNs. Right now I'm stuck at LB 0.23. Using a 3-layer CNN is vastly better than 2-layers. Though going to 4-layers is no help at all. The model simply overfits the validation set, and gives no improvement on local test or LB score. I'm using ImageDataGenerator() for data augmentation.
\n\nBojan Tunguz says in another thread that it's possible to do better than 0.05, so I must be missing something very big here. Any suggestions? Thanks!
", "messages": [ { "id": "146812", "postDate": "11/27/2016 00:10:45", "content": "I'm just learning to squeeze the most out of training CNNs. Right now I'm stuck at LB 0.23. Using a 3-layer CNN is vastly better than 2-layers. Though going to 4-layers is no help at all. The model simply overfits the validation set, and gives no improvement on local test or LB score. I'm using ImageDataGenerator() for data augmentation.
\n\nBojan Tunguz says in another thread that it's possible to do better than 0.05, so I must be missing something very big here. Any suggestions? Thanks!
", "rawMarkdown": "I'm just learning to squeeze the most out of training CNNs. Right now I'm stuck at LB 0.23. Using a 3-layer CNN is vastly better than 2-layers. Though going to 4-layers is no help at all. The model simply overfits the validation set, and gives no improvement on local test or LB score. I'm using ImageDataGenerator() for data augmentation.\r\n\r\nBojan Tunguz says in another thread that it's possible to do better than 0.05, so I must be missing something very big here. Any suggestions? Thanks!", "votes": null }, { "id": "146827", "postDate": "11/27/2016 03:12:27", "content": "Here's my network design:
\n\n model = Sequential()\n\n filters = 32\n for i in range(4):\n model.add(Convolution2D(filters, 7, 7))\n model.add(PReLU())\n model.add(Convolution2D(filters, 7, 7))\n model.add(PReLU())\n model.add(MaxPooling2D(pool_size=(2, 2)))\n model.add(Dropout(0.2))\n filters = int(filters * 2)\n\n model.add(Flatten())\n\n for i in range(2):\n model.add(Dense(500))\n model.add(PReLU())\n model.add(Dropout(0.2))\n\n model.add(Dense(num_classes))\n model.add(Activation('softmax'))\n@kubilai Try changing the last line
\n\nfrom
\n\n model.add(Activation('softmax'))\nto
\n\n model.add(Activation('sigmoid'))\nThank you for the suggestion! I tried it on small datasets, but results are erratic, sometimes sigmoid is better, sometimes softmax is better. Will report back after running bigger datasets...
", "rawMarkdown": "Thank you for the suggestion! I tried it on small datasets, but results are erratic, sometimes sigmoid is better, sometimes softmax is better. Will report back after running bigger datasets...", "votes": null }, { "id": "149284", "postDate": "12/09/2016 06:15:23", "content": "I'm able to come to 0.1042 for a CNN trained completely from scratch. But I'm not able to improve any further. Even the training loss doesn't decrease for me (it is 0.17). Any ideas on why this is happening?
", "rawMarkdown": "I'm able to come to 0.1042 for a CNN trained completely from scratch. But I'm not able to improve any further. Even the training loss doesn't decrease for me (it is 0.17). Any ideas on why this is happening?", "votes": null }, { "id": "149290", "postDate": "12/09/2016 06:53:53", "content": "Are you using a similar architecture? I get 0.15 training loss 0.23 LB with a three-layer network. Working on a 4-layer one.
", "rawMarkdown": "Are you using a similar architecture? I get 0.15 training loss 0.23 LB with a three-layer network. Working on a 4-layer one.", "votes": null }, { "id": "149291", "postDate": "12/09/2016 06:59:46", "content": "Mine is a VGG 19 style architecture but with little bit fewer layers (12 conv layers I think) and smaller number of filters. What beats me is that training loss is higher than Val loss.
", "rawMarkdown": "Mine is a VGG 19 style architecture but with little bit fewer layers (12 conv layers I think) and smaller number of filters. What beats me is that training loss is higher than Val loss.", "votes": null }, { "id": "149292", "postDate": "12/09/2016 07:02:58", "content": "Is it necessary to divide the image pixel values by 255? I did not do any kind of image preprocessing and got the above mentioned results.
", "rawMarkdown": "Is it necessary to divide the image pixel values by 255? I did not do any kind of image preprocessing and got the above mentioned results.", "votes": null }, { "id": "149401", "postDate": "12/09/2016 23:01:21", "content": "So I've gotta add more layers then, thanks. Yes, normalizing the pixel values will help with training. Also, did you use image augmentation? I use Keras' ImageDataGenerator(), and it has a handy scale parameter.
", "rawMarkdown": "So I've gotta add more layers then, thanks. Yes, normalizing the pixel values will help with training. Also, did you use image augmentation? I use Keras' ImageDataGenerator(), and it has a handy scale parameter.", "votes": null }, { "id": "149423", "postDate": "12/10/2016 03:00:08", "content": "Yes, I used Data Augmentation. Specifically, translation and random pixel value shifts.
", "rawMarkdown": "Yes, I used Data Augmentation. Specifically, translation and random pixel value shifts.", "votes": null }, { "id": "149577", "postDate": "12/11/2016 06:01:36", "content": "Current best result is using VGG-16 weights, tuning all but the top 5 layers. Training/validation/localtest@0.03 LB@0.14. So we have different problems hehe. I suspect the resolution we feed it is important (currently 224x224), will have to try larger size and wait for days...
", "rawMarkdown": "Current best result is using VGG-16 weights, tuning all but the top 5 layers. Training/validation/localtest@0.03 LB@0.14. So we have different problems hehe. I suspect the resolution we feed it is important (currently 224x224), will have to try larger size and wait for days...", "votes": null }, { "id": "155664", "postDate": "01/12/2017 10:13:35", "content": "Have you tried using batch normalization layers?: https://keras.io/layers/normalization/#batchnormalization
\n\nAlso it might help to pretrain each layer individually, i.e.:\nFirst only use one conv layer + Dense Layer(s) and train to predict cats/dogs.
\n\nThen keep the weights for the first conv layer, add another conv layer and train, etc...
", "rawMarkdown": "Have you tried using batch normalization layers?: https://keras.io/layers/normalization/#batchnormalization\r\n\r\nAlso it might help to pretrain each layer individually, i.e.:\r\nFirst only use one conv layer + Dense Layer(s) and train to predict cats/dogs.\r\n\r\nThen keep the weights for the first conv layer, add another conv layer and train, etc...", "votes": null }, { "id": "155930", "postDate": "01/13/2017 10:39:31", "content": "You train a Conv+Dense first. \nThen you train OldConv+NewConv+Dense where OldConv is the previously trained Conv layer with its weights retained?
\n\nDid I understand it right? What is the benefit of doing such a process?
", "rawMarkdown": "You train a Conv+Dense first. \r\nThen you train OldConv+NewConv+Dense where OldConv is the previously trained Conv layer with its weights retained? \r\n\r\nDid I understand it right? What is the benefit of doing such a process?", "votes": null }, { "id": "156223", "postDate": "01/14/2017 23:54:13", "content": "Yes, that's what bestaar meant. Training a deep network at once, we get into the vanishing gradient problem, and the first layers don't train properly. So people pre-train layers a small group at a time, and build up a deep network slowly. Will try batch normalization and report back.
", "rawMarkdown": "Yes, that's what bestaar meant. Training a deep network at once, we get into the vanishing gradient problem, and the first layers don't train properly. So people pre-train layers a small group at a time, and build up a deep network slowly. Will try batch normalization and report back.", "votes": null }, { "id": "156254", "postDate": "01/15/2017 08:24:46", "content": "exactly ;)
", "rawMarkdown": "exactly ;)", "votes": null }, { "id": "162766", "postDate": "02/21/2017 02:42:56", "content": "This can happen when using dropout or other regularization methods.
", "rawMarkdown": "This can happen when using dropout or other regularization methods.", "votes": null } ], "comments": [ { "id": 146827, "author_name": "kubilai", "author_url": "", "post_date": "11/27/2016 03:12:27", "content": "Here's my network design:
\n\n model = Sequential()\n\n filters = 32\n for i in range(4):\n model.add(Convolution2D(filters, 7, 7))\n model.add(PReLU())\n model.add(Convolution2D(filters, 7, 7))\n model.add(PReLU())\n model.add(MaxPooling2D(pool_size=(2, 2)))\n model.add(Dropout(0.2))\n filters = int(filters * 2)\n\n model.add(Flatten())\n\n for i in range(2):\n model.add(Dense(500))\n model.add(PReLU())\n model.add(Dropout(0.2))\n\n model.add(Dense(num_classes))\n model.add(Activation('softmax'))\n@kubilai Try changing the last line
\n\nfrom
\n\n model.add(Activation('softmax'))\nto
\n\n model.add(Activation('sigmoid'))\nThank you for the suggestion! I tried it on small datasets, but results are erratic, sometimes sigmoid is better, sometimes softmax is better. Will report back after running bigger datasets...
", "votes": null, "replies": [] }, { "id": 149284, "author_name": "prajwalkr", "author_url": "", "post_date": "12/09/2016 06:15:23", "content": "I'm able to come to 0.1042 for a CNN trained completely from scratch. But I'm not able to improve any further. Even the training loss doesn't decrease for me (it is 0.17). Any ideas on why this is happening?
", "votes": null, "replies": [] }, { "id": 149290, "author_name": "kubilai", "author_url": "", "post_date": "12/09/2016 06:53:53", "content": "Are you using a similar architecture? I get 0.15 training loss 0.23 LB with a three-layer network. Working on a 4-layer one.
", "votes": null, "replies": [] }, { "id": 149291, "author_name": "prajwalkr", "author_url": "", "post_date": "12/09/2016 06:59:46", "content": "Mine is a VGG 19 style architecture but with little bit fewer layers (12 conv layers I think) and smaller number of filters. What beats me is that training loss is higher than Val loss.
", "votes": null, "replies": [ { "id": 162766, "author_name": "malr87", "author_url": "", "post_date": "02/21/2017 02:42:56", "content": "This can happen when using dropout or other regularization methods.
", "votes": null, "replies": [] } ] }, { "id": 149292, "author_name": "prajwalkr", "author_url": "", "post_date": "12/09/2016 07:02:58", "content": "Is it necessary to divide the image pixel values by 255? I did not do any kind of image preprocessing and got the above mentioned results.
", "votes": null, "replies": [] }, { "id": 149401, "author_name": "kubilai", "author_url": "", "post_date": "12/09/2016 23:01:21", "content": "So I've gotta add more layers then, thanks. Yes, normalizing the pixel values will help with training. Also, did you use image augmentation? I use Keras' ImageDataGenerator(), and it has a handy scale parameter.
", "votes": null, "replies": [] }, { "id": 149423, "author_name": "prajwalkr", "author_url": "", "post_date": "12/10/2016 03:00:08", "content": "Yes, I used Data Augmentation. Specifically, translation and random pixel value shifts.
", "votes": null, "replies": [] }, { "id": 149577, "author_name": "kubilai", "author_url": "", "post_date": "12/11/2016 06:01:36", "content": "Current best result is using VGG-16 weights, tuning all but the top 5 layers. Training/validation/localtest@0.03 LB@0.14. So we have different problems hehe. I suspect the resolution we feed it is important (currently 224x224), will have to try larger size and wait for days...
", "votes": null, "replies": [] }, { "id": 155664, "author_name": "bestaar", "author_url": "", "post_date": "01/12/2017 10:13:35", "content": "Have you tried using batch normalization layers?: https://keras.io/layers/normalization/#batchnormalization
\n\nAlso it might help to pretrain each layer individually, i.e.:\nFirst only use one conv layer + Dense Layer(s) and train to predict cats/dogs.
\n\nThen keep the weights for the first conv layer, add another conv layer and train, etc...
", "votes": null, "replies": [] }, { "id": 155930, "author_name": "prajwalkr", "author_url": "", "post_date": "01/13/2017 10:39:31", "content": "You train a Conv+Dense first. \nThen you train OldConv+NewConv+Dense where OldConv is the previously trained Conv layer with its weights retained?
\n\nDid I understand it right? What is the benefit of doing such a process?
", "votes": null, "replies": [] }, { "id": 156223, "author_name": "kubilai", "author_url": "", "post_date": "01/14/2017 23:54:13", "content": "Yes, that's what bestaar meant. Training a deep network at once, we get into the vanishing gradient problem, and the first layers don't train properly. So people pre-train layers a small group at a time, and build up a deep network slowly. Will try batch normalization and report back.
", "votes": null, "replies": [] }, { "id": 156254, "author_name": "bestaar", "author_url": "", "post_date": "01/15/2017 08:24:46", "content": "exactly ;)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "146812": "I'm just learning to squeeze the most out of training CNNs. Right now I'm stuck at LB 0.23. Using a 3-layer CNN is vastly better than 2-layers. Though going to 4-layers is no help at all. The model simply overfits the validation set, and gives no improvement on local test or LB score. I'm using ImageDataGenerator() for data augmentation.\r\n\r\nBojan Tunguz says in another thread that it's possible to do better than 0.05, so I must be missing something very big here. Any suggestions? Thanks!", "146827": "Here's my network design:\r\n\r\n model = Sequential()\r\n\r\n filters = 32\r\n for i in range(4):\r\n model.add(Convolution2D(filters, 7, 7))\r\n model.add(PReLU())\r\n model.add(Convolution2D(filters, 7, 7))\r\n model.add(PReLU())\r\n model.add(MaxPooling2D(pool_size=(2, 2)))\r\n model.add(Dropout(0.2))\r\n filters = int(filters * 2)\r\n\r\n model.add(Flatten())\r\n\r\n for i in range(2):\r\n model.add(Dense(500))\r\n model.add(PReLU())\r\n model.add(Dropout(0.2))\r\n\r\n model.add(Dense(num_classes))\r\n model.add(Activation('softmax'))", "147794": "kubilai Try changing the last line \r\n\r\nfrom \r\n\r\n model.add(Activation('softmax'))\r\nto \r\n\r\n model.add(Activation('sigmoid'))", "149062": "Thank you for the suggestion! I tried it on small datasets, but results are erratic, sometimes sigmoid is better, sometimes softmax is better. Will report back after running bigger datasets...", "149284": "I'm able to come to 0.1042 for a CNN trained completely from scratch. But I'm not able to improve any further. Even the training loss doesn't decrease for me (it is 0.17). Any ideas on why this is happening?", "149290": "Are you using a similar architecture? I get 0.15 training loss 0.23 LB with a three-layer network. Working on a 4-layer one.", "149291": "Mine is a VGG 19 style architecture but with little bit fewer layers (12 conv layers I think) and smaller number of filters. What beats me is that training loss is higher than Val loss.", "149292": "Is it necessary to divide the image pixel values by 255? I did not do any kind of image preprocessing and got the above mentioned results.", "149401": "So I've gotta add more layers then, thanks. Yes, normalizing the pixel values will help with training. Also, did you use image augmentation? I use Keras' ImageDataGenerator(), and it has a handy scale parameter.", "149423": "Yes, I used Data Augmentation. Specifically, translation and random pixel value shifts.", "149577": "Current best result is using VGG-16 weights, tuning all but the top 5 layers. Training/validation/localtest@0.03 LB@0.14. So we have different problems hehe. I suspect the resolution we feed it is important (currently 224x224), will have to try larger size and wait for days...", "155664": "Have you tried using batch normalization layers?: https://keras.io/layers/normalization/#batchnormalization\r\n\r\nAlso it might help to pretrain each layer individually, i.e.:\r\nFirst only use one conv layer + Dense Layer(s) and train to predict cats/dogs.\r\n\r\nThen keep the weights for the first conv layer, add another conv layer and train, etc...", "155930": "You train a Conv+Dense first. \r\nThen you train OldConv+NewConv+Dense where OldConv is the previously trained Conv layer with its weights retained? \r\n\r\nDid I understand it right? What is the benefit of doing such a process?", "156223": "Yes, that's what bestaar meant. Training a deep network at once, we get into the vanishing gradient problem, and the first layers don't train properly. So people pre-train layers a small group at a time, and build up a deep network slowly. Will try batch normalization and report back.", "156254": "exactly ;)", "162766": "This can happen when using dropout or other regularization methods." }, "source": "meta" }