Hi everyone,
\nI was wondering if anyone is interesting in sharing training strategies for convolutional neural nets (or other deep learners)? We don't have to team up or share code, I thought it may just be useful to bounce successful/unsuccessful ideas of each other.
\nIf anyone is interested (and I will share first) please let me know and we can email.
", "messages": [ { "id": "62799", "postDate": "01/16/2015 20:18:33", "content": "Hi everyone,
\nI was wondering if anyone is interesting in sharing training strategies for convolutional neural nets (or other deep learners)? We don't have to team up or share code, I thought it may just be useful to bounce successful/unsuccessful ideas of each other.
\nIf anyone is interested (and I will share first) please let me know and we can email.
", "rawMarkdown": "", "votes": null }, { "id": "62801", "postDate": "01/16/2015 20:25:41", "content": "You can share within a team or share on the forum, but it is against the rules to bounce ideas off of each other in email unless you are on the same team.
", "rawMarkdown": "", "votes": null }, { "id": "62802", "postDate": "01/16/2015 20:31:48", "content": "Sorry, I did not know that. I guess I then withdraw this forum post.
", "rawMarkdown": "", "votes": null }, { "id": "62807", "postDate": "01/17/2015 00:09:56", "content": "James,
Where does it specifically say we are unable to share ideas privately? I only see rules for code.
How are people supposed to team up without sharing any ideas prior? Seems like a poor strategy to blindly team up with another competitor with no insight into their methodology.
It's true the rules only refer to sharing code and data, which strikes me as strange since in most competitions you could easily describe your whole approach in an easy to replicate way without sharing any code or data. If someone describes their entire convnet architecture and data augmentation scheme, that's just as good as sharing the code.
\n\nAs far as blindly teaming up with no insight into each other's methodology, that's how I've always done it.
", "rawMarkdown": "", "votes": null }, { "id": "62810", "postDate": "01/17/2015 02:51:52", "content": "To stay within the rules I don't mind having the conversation here. Basically I am interested in which architecture people are using to train their CNNs. I am currently using a pure theano approach (similar to this theano CNN) but cannot seem to break the 1.2 barrier in my error score.
\nAre people using (or have used in the past) any "black box" architectures such as Caffe or cuda-ConvNet2, or any others?
\nConsidering I am still getting a score about twice that of the leading scores right now I am keen tp try some new approaches.
", "rawMarkdown": "", "votes": null }, { "id": "62811", "postDate": "01/17/2015 03:53:38", "content": "I started with the cxxnet kaggle bowl example and built on that. It's not a very complex architecture, so I had to beef it up a bit. Data augmentation helps quite a bit.
", "rawMarkdown": "", "votes": null }, { "id": "62823", "postDate": "01/17/2015 09:54:48", "content": "James, that's quite impressive that you blindly team up with people without sharing strategies prior. I doubt many other competitors do the same (especially in late game). It'd be interesting to see how effective that strategy is in practice.
\nHow would a rule on sharing ideas privately even be enforced?
From my experience with past kaggle competitions, there's always been sharing of ideas between competitors in private. Heck, I speak to competitors in private about ideas without the intention to compete myself.
In my opinion, there's a big difference between sharing strategies that may help a fellow competitor improve their skills as a data scientist, and giving away a winning strategy. (Hell, if you have a winning strategy, why would you share it in the first place?). If someone takes the time to show that they have a willingness to learn by asking questions to fellow competitors in private, then I have no qualms with helping as much as I can. I feel it's ridiculous to impose the constraint that those ideas must be made public.
Anyway, Mike, while I have not made a submission yet, I've had decent results with building an augmented dataset of the training data (CV results are ~0.9-1.09 log-loss) with a simple 2 layer CNN (100, 150 filters with max pooling) with a 500-121 hidden unit dense layer with a softmax classifier.
Augmentation of the dataset includes: random rotation (between 0-360), random cropping, and random mirroring.
Miroslaw,
\nThanks for the info. I think one of my big problems is I do no data augmentation, I just take the raw pixels as is and feed that to my net. Can you explain a bit on how to do this properly or point me to some references that go through it in detail.
\nAlso, I have seen winning solutions in other image classification competitions (kaggle-galaxies) blend multiple models together. Do you have a good idea of what they are doing here?
\nThanks!
", "rawMarkdown": "", "votes": null }, { "id": "62834", "postDate": "01/17/2015 16:26:26", "content": "See Nagadomi's post in this thread
\n\nhttp://www.kaggle.com/c/datasciencebowl/forums/t/11421/converting-images-to-standard-scale-code
\n\nfor his approach to data augmentation.
", "rawMarkdown": "", "votes": null }, { "id": "62839", "postDate": "01/17/2015 18:01:42", "content": "Mike,
I recommend checking out this course: https://class.coursera.org/neuralnets-2012-001/lecture (Hopefully you can still audit the lectures)
Geoffrey Hinton covers combining models in lecture 10.
Here are the slides from that lecture in case you are unable to access the course material: https://d396qusza40orc.cloudfront.net/neuralnets/lecture_slides/lec10.pdf
Regarding image augmentation, I built some basic augmentation tools in python using the Pillow image processing library. When I have some spare time, I'll refactor my code and post it on the forums.
", "rawMarkdown": "", "votes": null }, { "id": "62852", "postDate": "01/18/2015 06:02:46", "content": "deep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
", "rawMarkdown": "", "votes": null }, { "id": "62858", "postDate": "01/18/2015 13:54:51", "content": "[quote=nagadomi;62852]
\ndeep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
\n[/quote]
\nDo you preserve aspect ratio? I found that if you break aspect ratio then you can work with smaller resolution.
", "rawMarkdown": "", "votes": null }, { "id": "62867", "postDate": "01/18/2015 17:09:58", "content": "[quote=Ilya Kostrikov;62858]
\n[quote=nagadomi;62852]
\ndeep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
\n[/quote]
\nDo you preserve aspect ratio? I found that if you break aspect ratio then you can work with smaller resolution.
\n[/quote]
\nwow, that is very useful.
I had thought that top-20 people preserve aspect ratio.
EDIT: In preprocessing, I preserve aspect ratio. but my data augmentation layer sometimes breaks aspect ratio.
", "rawMarkdown": "", "votes": null }, { "id": "62870", "postDate": "01/18/2015 17:49:16", "content": "Likewise, I've always blindly gone and preserved aspect ratio in preprocessing.
Thanks for the tip Ilya!
I'm quite surprised too. I thought people preserve aspect ratio...or else it might hurt performance..
\n[quote=Ilya Kostrikov;62858]
\n[quote=nagadomi;62852]
\ndeep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
\n[/quote]
\nDo you preserve aspect ratio? I found that if you break aspect ratio then you can work with smaller resolution.
\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "62990", "postDate": "01/22/2015 01:21:22", "content": "My best score (1.18 or so on the leaderboard) was with 48x48 sized images, just straight resized (and thus breaking aspect ratio). I have since run many, many more configurations with 94x94 and 96x96 aditional layers, different number of nodes, etc. while preserving aspect ratio (rotate 45 degrees, 90 degrees, flip vertical and flip horizontal, while trying also to "balance out" the data set across classes), and to my surprise have never surpassed my "dumb" resize score. The best I got was an improved validation score, but it came out substantially higher on the public leaderboard (1.3, instead of the 1.14 my validation score suggested).
\n\nVery counter-intuitive to me, as I had expected the more pixel resolution, the more refined details I could pick up on, but to be fair, I am learning as I go along with this...
", "rawMarkdown": "", "votes": null }, { "id": "62991", "postDate": "01/22/2015 01:27:26", "content": "Just resizing to 48x48 and then rotating 0/90/180/270 and mirroring should get you to a better lb score than that. I would check the generated images carefully and make sure there is not some mistake.
\nYour 1.18 is very close to my best lb score at 48x48 with only a random mirror.
", "rawMarkdown": "", "votes": null }, { "id": "62992", "postDate": "01/22/2015 01:49:59", "content": "Can someone please confirm that my understanding of data augmentation is correct. Let assumes that we have 1 simple transformation of randomly rotating by 0/90/180/270 degrees and we have a total of 15 batches and 10 epochs.
\nDuring training, for each batch in each epoch, we randomly rotate that batch of data and proceed with calculating the training and validation loss based on that augmentation. Therefore, every batch in every epoch will have a random data augmentation step applied to it.
\nThanks!
", "rawMarkdown": "", "votes": null }, { "id": "62999", "postDate": "01/22/2015 09:51:17", "content": "[quote=mike1886;62992]
\nCan someone please confirm that my understanding of data augmentation is correct. Let assumes that we have 1 simple transformation of randomly rotating by 0/90/180/270 degrees and we have a total of 15 batches and 10 epochs.
\nDuring training, for each batch in each epoch, we randomly rotate that batch of data and proceed with calculating the training and validation loss based on that augmentation. Therefore, every batch in every epoch will have a random data augmentation step applied to it.
\n[/quote]
\nThat is one of many possible schemes, and I think it has a reasonable chance of improving the generalisation of any model capable of over-fitting.
\nThe general idea is to pick alterations to the input data that logically should not change the output class, and that remain representative of inputs you might expect to see in the test data.
\nYou have a few choices on how you then apply those alterations. With your scheme, as well as randomly altering items in each batch, you could pre-calculate all transformations and quadruple the size of your training set. If you had a scheme with more options, this is less feasible, but you could still just randomly generate a fixed training set size. Or you could effectively create an infinite online or semi-online data source, drawing from it on-demand.
", "rawMarkdown": "", "votes": null }, { "id": "63003", "postDate": "01/22/2015 13:43:06", "content": "thanks Neil that was very helpful!
\nFor the scheme I described, do you tend to leave a larger portion of the batches untransformed?
", "rawMarkdown": "", "votes": null }, { "id": "63004", "postDate": "01/22/2015 13:55:00", "content": "[quote=mike1886;63003]
\nthanks Neil that was very helpful!
\nFor the scheme I described, do you tend to leave a larger portion of the batches untransformed?
\n[/quote]
\nI am not sure. My personal preference is to add the transforms (including a chance of no transform) to all inputs. However, I have no measure of whether that works in general. I haven't entered this competition, I'm just in the forums trying to learn more about CNNs.
", "rawMarkdown": "", "votes": null }, { "id": "63022", "postDate": "01/22/2015 19:16:15", "content": "[quote=James King;62991]
\nJust resizing to 48x48 and then rotating 0/90/180/270 and mirroring should get you to a better lb score than that. I would check the generated images carefully and make sure there is not some mistake.
\nYour 1.18 is very close to my best lb score at 48x48 with only a random mirror.
\n[/quote]
\nThanks! I'll give that a try as well.
\nI am printing out test images (one for each class on data load), and they come out just fine. I am still running further tests on my net architecture (next up is trying 5x5 filters, rather than 3x3) and with the larger images sizes I have been "squaring" them by creating a blank image of width and height based on max(org_img.width, org_img.height), which preserves aspect ratio, as well as a 45 degree transform, expanding the size, which creates a square image, as well. I'll have to try to go back to the straight resize, aspect ratio be damned! :)
\nI am also suspecting now that selectively doing more data augmentation for those classes with fewer source images may actually be the wrong strategy...
\nNo test is wasted, though, as it is all an excellent exercise to see how this all works.
", "rawMarkdown": "", "votes": null }, { "id": "63028", "postDate": "01/22/2015 19:41:51", "content": "Is there a theoretical preferred way to combine real time augmentation (i.e. I feed 30k images into the network and flip them randomly at each epoch) and just increasing the size of the data by feeding more observation into the network (say I flip at 90, 180, 270 degrees, and feed 120k observations into the network)? Obviously the practical part is important, given the limited amount of GPU memory. Just curious to know what people is doing...
", "rawMarkdown": "", "votes": null }, { "id": "63035", "postDate": "01/22/2015 22:16:28", "content": "[quote=Giulio;63028]
\nIs there a theoretical preferred way to combine real time augmentation (i.e. I feed 30k images into the network and flip them randomly at each epoch) and just increasing the size of the data by feeding more observation into the network (say I flip at 90, 180, 270 degrees, and feed 120k observations into the network)? Obviously the practical part is important, given the limited amount of GPU memory. Just curious to know what people is doing...
\n[/quote]
\nReal-time or online augmentation doesn't affect the GPU RAM requirement for SGD. It is because SGD is online learning. The RAM requirement is mainly determined by the model's total parameters and the mini-batch size.
", "rawMarkdown": "", "votes": null }, { "id": "63036", "postDate": "01/23/2015 01:38:38", "content": "How many epochs are people using?
\nI am currently only using ~200-300 and using early stopping after 50 epochs of no improvement. However, I feel like I probably can let it run longer since it still doesn't seem to converge.
", "rawMarkdown": "", "votes": null }, { "id": "63037", "postDate": "01/23/2015 02:43:21", "content": "Here's the code I'm currently using to create the rotations of images.
\nIt's nothing special, and could probably use some cleanup, but it gets the job done.
\nhttp://pastebin.com/SjTKUBGZ
\nIt's written in python. The only extra library needed is the Pillow library. You can install that if you have pip:
\n```
pip install Pillow
```
To use the code, all you need to do is pass a regular expression to the directory holding the images. The script will create image rotations at angles [45, 90, 135, 180, 225, 270, 315]. You can easily change this by modifying the rotation_angles variable.
\nThe output data will be written in the same directory with a _rot[angle].png suffix.
\nI have some more augmentation code to share, I just haven't had much time to clean up my code.
Usage example:
python ./rotate_images.py ~/kaggle/plankton/data/train/\\*\\*.png
This will recursively rotate all images that match the wildcard expression.
Do you guys mind sharing how deep and wide are your fully connected layers? The CXXNET example uses 2 layers of 512 units and I feel that's not enough for 121 classes?
\nAlso what are your techniques to optimize the structure of the CNN? I tried visualizing the convolutional layer but it's not really useful (as it is so small).
", "rawMarkdown": "", "votes": null }, { "id": "63142", "postDate": "01/25/2015 15:23:16", "content": "My best results so far have come with
\n InputLayer (None, 1, 48, 48) produces 2304 outputs
Conv2DCCLayer (None, 192, 43, 43) produces 355008 outputs
MaxPool2DCCLayer (None, 192, 22, 22) produces 92928 outputs
DropoutLayer (None, 192, 22, 22) produces 92928 outputs
Conv2DCCLayer (None, 384, 18, 18) produces 124416 outputs
MaxPool2DCCLayer (None, 384, 9, 9) produces 31104 outputs
DropoutLayer (None, 384, 9, 9) produces 31104 outputs
DenseLayer (None, 1000) produces 1000 outputs
DropoutLayer (None, 1000) produces 1000 outputs
DenseLayer (None, 1000) produces 1000 outputs
DenseLayer (None, 121) produces 121 outputs
I have also tried adding in a third Conv2DCCLayer and also a DropoutLayer after my last DenseLayer but that seems to not help. I think I need to increase my layer size to something bigger than 192 and 384 but not sure.
\nWhat have you tried?
", "rawMarkdown": "", "votes": null }, { "id": "63145", "postDate": "01/25/2015 16:27:12", "content": "Ahh fellow nolearn user! I tried adding a 3rd convolutional layer too but I don't think it's helping. One thing that frustrates me is I don't know how to qualitatively compares 2 models. I tried visualizing the weights learned from the filter but it's hard to interpret.
\nNow I'm trying out maxout layers (layers.FeaturePoolLayer) (see http://arxiv.org/pdf/1302.4389v4.pdf) and debugging my batch iterator to make sure the data augmentation is working properly.
", "rawMarkdown": "", "votes": null }, { "id": "63149", "postDate": "01/25/2015 18:10:45", "content": "Have you experimented with different dropout values? I found that 0.5 seems to work best but didn't try to many values or scenarios?
", "rawMarkdown": "", "votes": null }, { "id": "63151", "postDate": "01/25/2015 19:02:24", "content": "I just submitted a 1.08 score with:
\n\nInput: 60x60
\nL1 weights: 1 x 32 x 5x5 convolution
\nmaxpool
\nL2 weights: 32 x 64 x 5x5 convolution
\nmaxpool
\nL3 weights: 64 x 128 x 5x5 convolution
\nmaxpool
\nflatten
\nL4: 128x4x4 x 2025
\nL5: 2025 x 121 (softmax)
\nDropout hidden: 0.5, dropout convolution: 0.2
\nBatch sizes of 512, learning rate of 0.001 with RMSProp; training source images rotated 45 degrees, then all 90 degrees, then all 180 degrees.
\nI tried higher dropout values, but it just slows learning down dramatically.
", "rawMarkdown": "", "votes": null }, { "id": "63163", "postDate": "01/25/2015 21:37:47", "content": "[quote=Jay Thoden van Velzen;63151]
\nI just submitted a 1.08 score with:
\nInput: 60x60
\nL1 weights: 1 x 32 x 5x5 convolution
\nmaxpool
\nL2 weights: 32 x 64 x 5x5 convolution
\nmaxpool
\nL3 weights: 64 x 128 x 5x5 convolution
\nmaxpool
\nflatten
\nL4: 128x4x4 x 2025
\nL5: 2025 x 121 (softmax)
\nDropout hidden: 0.5, dropout convolution: 0.2
\nBatch sizes of 512, learning rate of 0.001 with RMSProp; training source images rotated 45 degrees, then all 90 degrees, then all 180 degrees.
\nI tried higher dropout values, but it just slows learning down dramatically.
\n[/quote]
Maybe my implementation of RMSProp is incorrect, but I was unable to get RMSProp to work very well.
Miroslaw, can you provide a reference for RMSProp, I would like to try it out. Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "63394", "postDate": "01/31/2015 18:46:41", "content": "Here is my RMSProp function, as well as one with Nesterov momentum (though that didn't give me better results):
\n\ndef RMSprop(cost, params, lr=0.001, rho=0.9, epsilon=1e-6):
grads = T.grad(cost=cost, wrt=params)
updates = []
for p, g in zip(params, grads):
acc = theano.shared(p.get_value() * 0.)
acc_new = rho * acc + (1 - rho) * g ** 2
gradient_scaling = T.sqrt(acc_new + epsilon)
g = g / gradient_scaling
updates.append((acc, acc_new))
updates.append((p, p - lr * g))
return updates
def RMSprop_nesterov(cost, params, lr=0.001, rho=0.9, momentum=0.7, epsilon=1e-6):
grads = T.grad(cost=cost, wrt=params)
updates = []
for p, g in zip(params, grads):
mo_p = theano.shared(p.get_value() * 0.)
acc = theano.shared(p.get_value() * 0.)
acc_new = rho * acc + (1 - rho) * g**2step = lr * g / T.sqrt(acc_new + epsilon)
v = momentum * mo_p - step
w_ = p + momentum * v - step
updates.append((acc, acc_new))
updates.append((mo_p, v))
updates.append((p, w_))
return updates
You can share within a team or share on the forum, but it is against the rules to bounce ideas off of each other in email unless you are on the same team.
", "votes": null, "replies": [] }, { "id": 62802, "author_name": "msegala", "author_url": "", "post_date": "01/16/2015 20:31:48", "content": "Sorry, I did not know that. I guess I then withdraw this forum post.
", "votes": null, "replies": [] }, { "id": 62807, "author_name": "miroslaw", "author_url": "", "post_date": "01/17/2015 00:09:56", "content": "James,
Where does it specifically say we are unable to share ideas privately? I only see rules for code.
How are people supposed to team up without sharing any ideas prior? Seems like a poor strategy to blindly team up with another competitor with no insight into their methodology.
It's true the rules only refer to sharing code and data, which strikes me as strange since in most competitions you could easily describe your whole approach in an easy to replicate way without sharing any code or data. If someone describes their entire convnet architecture and data augmentation scheme, that's just as good as sharing the code.
\n\nAs far as blindly teaming up with no insight into each other's methodology, that's how I've always done it.
", "votes": null, "replies": [] }, { "id": 62810, "author_name": "msegala", "author_url": "", "post_date": "01/17/2015 02:51:52", "content": "To stay within the rules I don't mind having the conversation here. Basically I am interested in which architecture people are using to train their CNNs. I am currently using a pure theano approach (similar to this theano CNN) but cannot seem to break the 1.2 barrier in my error score.
\nAre people using (or have used in the past) any "black box" architectures such as Caffe or cuda-ConvNet2, or any others?
\nConsidering I am still getting a score about twice that of the leading scores right now I am keen tp try some new approaches.
", "votes": null, "replies": [] }, { "id": 62811, "author_name": "jfkingiii", "author_url": "", "post_date": "01/17/2015 03:53:38", "content": "I started with the cxxnet kaggle bowl example and built on that. It's not a very complex architecture, so I had to beef it up a bit. Data augmentation helps quite a bit.
", "votes": null, "replies": [] }, { "id": 62823, "author_name": "miroslaw", "author_url": "", "post_date": "01/17/2015 09:54:48", "content": "James, that's quite impressive that you blindly team up with people without sharing strategies prior. I doubt many other competitors do the same (especially in late game). It'd be interesting to see how effective that strategy is in practice.
\nHow would a rule on sharing ideas privately even be enforced?
From my experience with past kaggle competitions, there's always been sharing of ideas between competitors in private. Heck, I speak to competitors in private about ideas without the intention to compete myself.
In my opinion, there's a big difference between sharing strategies that may help a fellow competitor improve their skills as a data scientist, and giving away a winning strategy. (Hell, if you have a winning strategy, why would you share it in the first place?). If someone takes the time to show that they have a willingness to learn by asking questions to fellow competitors in private, then I have no qualms with helping as much as I can. I feel it's ridiculous to impose the constraint that those ideas must be made public.
Anyway, Mike, while I have not made a submission yet, I've had decent results with building an augmented dataset of the training data (CV results are ~0.9-1.09 log-loss) with a simple 2 layer CNN (100, 150 filters with max pooling) with a 500-121 hidden unit dense layer with a softmax classifier.
Augmentation of the dataset includes: random rotation (between 0-360), random cropping, and random mirroring.
Miroslaw,
\nThanks for the info. I think one of my big problems is I do no data augmentation, I just take the raw pixels as is and feed that to my net. Can you explain a bit on how to do this properly or point me to some references that go through it in detail.
\nAlso, I have seen winning solutions in other image classification competitions (kaggle-galaxies) blend multiple models together. Do you have a good idea of what they are doing here?
\nThanks!
", "votes": null, "replies": [] }, { "id": 62834, "author_name": "jfkingiii", "author_url": "", "post_date": "01/17/2015 16:26:26", "content": "See Nagadomi's post in this thread
\n\nhttp://www.kaggle.com/c/datasciencebowl/forums/t/11421/converting-images-to-standard-scale-code
\n\nfor his approach to data augmentation.
", "votes": null, "replies": [] }, { "id": 62839, "author_name": "miroslaw", "author_url": "", "post_date": "01/17/2015 18:01:42", "content": "Mike,
I recommend checking out this course: https://class.coursera.org/neuralnets-2012-001/lecture (Hopefully you can still audit the lectures)
Geoffrey Hinton covers combining models in lecture 10.
Here are the slides from that lecture in case you are unable to access the course material: https://d396qusza40orc.cloudfront.net/neuralnets/lecture_slides/lec10.pdf
Regarding image augmentation, I built some basic augmentation tools in python using the Pillow image processing library. When I have some spare time, I'll refactor my code and post it on the forums.
", "votes": null, "replies": [] }, { "id": 62852, "author_name": "nagadomi", "author_url": "", "post_date": "01/18/2015 06:02:46", "content": "deep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
", "votes": null, "replies": [] }, { "id": 62858, "author_name": "ilyakostrikov", "author_url": "", "post_date": "01/18/2015 13:54:51", "content": "[quote=nagadomi;62852]
\ndeep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
\n[/quote]
\nDo you preserve aspect ratio? I found that if you break aspect ratio then you can work with smaller resolution.
", "votes": null, "replies": [] }, { "id": 62867, "author_name": "nagadomi", "author_url": "", "post_date": "01/18/2015 17:09:58", "content": "[quote=Ilya Kostrikov;62858]
\n[quote=nagadomi;62852]
\ndeep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
\n[/quote]
\nDo you preserve aspect ratio? I found that if you break aspect ratio then you can work with smaller resolution.
\n[/quote]
\nwow, that is very useful.
I had thought that top-20 people preserve aspect ratio.
EDIT: In preprocessing, I preserve aspect ratio. but my data augmentation layer sometimes breaks aspect ratio.
", "votes": null, "replies": [] }, { "id": 62870, "author_name": "miroslaw", "author_url": "", "post_date": "01/18/2015 17:49:16", "content": "Likewise, I've always blindly gone and preserved aspect ratio in preprocessing.
Thanks for the tip Ilya!
I'm quite surprised too. I thought people preserve aspect ratio...or else it might hurt performance..
\n[quote=Ilya Kostrikov;62858]
\n[quote=nagadomi;62852]
\ndeep architecture with large input image(>= 80x80) helps quite a bit.
\nIn my CNN, the size of input image is 1x96x96.
\n[/quote]
\nDo you preserve aspect ratio? I found that if you break aspect ratio then you can work with smaller resolution.
\n[/quote]
", "votes": null, "replies": [] }, { "id": 62990, "author_name": "jaythodenvanvelzen", "author_url": "", "post_date": "01/22/2015 01:21:22", "content": "My best score (1.18 or so on the leaderboard) was with 48x48 sized images, just straight resized (and thus breaking aspect ratio). I have since run many, many more configurations with 94x94 and 96x96 aditional layers, different number of nodes, etc. while preserving aspect ratio (rotate 45 degrees, 90 degrees, flip vertical and flip horizontal, while trying also to "balance out" the data set across classes), and to my surprise have never surpassed my "dumb" resize score. The best I got was an improved validation score, but it came out substantially higher on the public leaderboard (1.3, instead of the 1.14 my validation score suggested).
\n\nVery counter-intuitive to me, as I had expected the more pixel resolution, the more refined details I could pick up on, but to be fair, I am learning as I go along with this...
", "votes": null, "replies": [] }, { "id": 62991, "author_name": "jfkingiii", "author_url": "", "post_date": "01/22/2015 01:27:26", "content": "Just resizing to 48x48 and then rotating 0/90/180/270 and mirroring should get you to a better lb score than that. I would check the generated images carefully and make sure there is not some mistake.
\nYour 1.18 is very close to my best lb score at 48x48 with only a random mirror.
", "votes": null, "replies": [] }, { "id": 62992, "author_name": "msegala", "author_url": "", "post_date": "01/22/2015 01:49:59", "content": "Can someone please confirm that my understanding of data augmentation is correct. Let assumes that we have 1 simple transformation of randomly rotating by 0/90/180/270 degrees and we have a total of 15 batches and 10 epochs.
\nDuring training, for each batch in each epoch, we randomly rotate that batch of data and proceed with calculating the training and validation loss based on that augmentation. Therefore, every batch in every epoch will have a random data augmentation step applied to it.
\nThanks!
", "votes": null, "replies": [] }, { "id": 62999, "author_name": "slobo777", "author_url": "", "post_date": "01/22/2015 09:51:17", "content": "[quote=mike1886;62992]
\nCan someone please confirm that my understanding of data augmentation is correct. Let assumes that we have 1 simple transformation of randomly rotating by 0/90/180/270 degrees and we have a total of 15 batches and 10 epochs.
\nDuring training, for each batch in each epoch, we randomly rotate that batch of data and proceed with calculating the training and validation loss based on that augmentation. Therefore, every batch in every epoch will have a random data augmentation step applied to it.
\n[/quote]
\nThat is one of many possible schemes, and I think it has a reasonable chance of improving the generalisation of any model capable of over-fitting.
\nThe general idea is to pick alterations to the input data that logically should not change the output class, and that remain representative of inputs you might expect to see in the test data.
\nYou have a few choices on how you then apply those alterations. With your scheme, as well as randomly altering items in each batch, you could pre-calculate all transformations and quadruple the size of your training set. If you had a scheme with more options, this is less feasible, but you could still just randomly generate a fixed training set size. Or you could effectively create an infinite online or semi-online data source, drawing from it on-demand.
", "votes": null, "replies": [] }, { "id": 63003, "author_name": "msegala", "author_url": "", "post_date": "01/22/2015 13:43:06", "content": "thanks Neil that was very helpful!
\nFor the scheme I described, do you tend to leave a larger portion of the batches untransformed?
", "votes": null, "replies": [] }, { "id": 63004, "author_name": "slobo777", "author_url": "", "post_date": "01/22/2015 13:55:00", "content": "[quote=mike1886;63003]
\nthanks Neil that was very helpful!
\nFor the scheme I described, do you tend to leave a larger portion of the batches untransformed?
\n[/quote]
\nI am not sure. My personal preference is to add the transforms (including a chance of no transform) to all inputs. However, I have no measure of whether that works in general. I haven't entered this competition, I'm just in the forums trying to learn more about CNNs.
", "votes": null, "replies": [] }, { "id": 63022, "author_name": "jaythodenvanvelzen", "author_url": "", "post_date": "01/22/2015 19:16:15", "content": "[quote=James King;62991]
\nJust resizing to 48x48 and then rotating 0/90/180/270 and mirroring should get you to a better lb score than that. I would check the generated images carefully and make sure there is not some mistake.
\nYour 1.18 is very close to my best lb score at 48x48 with only a random mirror.
\n[/quote]
\nThanks! I'll give that a try as well.
\nI am printing out test images (one for each class on data load), and they come out just fine. I am still running further tests on my net architecture (next up is trying 5x5 filters, rather than 3x3) and with the larger images sizes I have been "squaring" them by creating a blank image of width and height based on max(org_img.width, org_img.height), which preserves aspect ratio, as well as a 45 degree transform, expanding the size, which creates a square image, as well. I'll have to try to go back to the straight resize, aspect ratio be damned! :)
\nI am also suspecting now that selectively doing more data augmentation for those classes with fewer source images may actually be the wrong strategy...
\nNo test is wasted, though, as it is all an excellent exercise to see how this all works.
", "votes": null, "replies": [] }, { "id": 63028, "author_name": "adjgiulio", "author_url": "", "post_date": "01/22/2015 19:41:51", "content": "Is there a theoretical preferred way to combine real time augmentation (i.e. I feed 30k images into the network and flip them randomly at each epoch) and just increasing the size of the data by feeding more observation into the network (say I flip at 90, 180, 270 degrees, and feed 120k observations into the network)? Obviously the practical part is important, given the limited amount of GPU memory. Just curious to know what people is doing...
", "votes": null, "replies": [] }, { "id": 63035, "author_name": "songgc", "author_url": "", "post_date": "01/22/2015 22:16:28", "content": "[quote=Giulio;63028]
\nIs there a theoretical preferred way to combine real time augmentation (i.e. I feed 30k images into the network and flip them randomly at each epoch) and just increasing the size of the data by feeding more observation into the network (say I flip at 90, 180, 270 degrees, and feed 120k observations into the network)? Obviously the practical part is important, given the limited amount of GPU memory. Just curious to know what people is doing...
\n[/quote]
\nReal-time or online augmentation doesn't affect the GPU RAM requirement for SGD. It is because SGD is online learning. The RAM requirement is mainly determined by the model's total parameters and the mini-batch size.
", "votes": null, "replies": [] }, { "id": 63036, "author_name": "msegala", "author_url": "", "post_date": "01/23/2015 01:38:38", "content": "How many epochs are people using?
\nI am currently only using ~200-300 and using early stopping after 50 epochs of no improvement. However, I feel like I probably can let it run longer since it still doesn't seem to converge.
", "votes": null, "replies": [] }, { "id": 63037, "author_name": "miroslaw", "author_url": "", "post_date": "01/23/2015 02:43:21", "content": "Here's the code I'm currently using to create the rotations of images.
\nIt's nothing special, and could probably use some cleanup, but it gets the job done.
\nhttp://pastebin.com/SjTKUBGZ
\nIt's written in python. The only extra library needed is the Pillow library. You can install that if you have pip:
\n```
pip install Pillow
```
To use the code, all you need to do is pass a regular expression to the directory holding the images. The script will create image rotations at angles [45, 90, 135, 180, 225, 270, 315]. You can easily change this by modifying the rotation_angles variable.
\nThe output data will be written in the same directory with a _rot[angle].png suffix.
\nI have some more augmentation code to share, I just haven't had much time to clean up my code.
Usage example:
python ./rotate_images.py ~/kaggle/plankton/data/train/\\*\\*.png
This will recursively rotate all images that match the wildcard expression.
Do you guys mind sharing how deep and wide are your fully connected layers? The CXXNET example uses 2 layers of 512 units and I feel that's not enough for 121 classes?
\nAlso what are your techniques to optimize the structure of the CNN? I tried visualizing the convolutional layer but it's not really useful (as it is so small).
", "votes": null, "replies": [] }, { "id": 63142, "author_name": "msegala", "author_url": "", "post_date": "01/25/2015 15:23:16", "content": "My best results so far have come with
\n InputLayer (None, 1, 48, 48) produces 2304 outputs
Conv2DCCLayer (None, 192, 43, 43) produces 355008 outputs
MaxPool2DCCLayer (None, 192, 22, 22) produces 92928 outputs
DropoutLayer (None, 192, 22, 22) produces 92928 outputs
Conv2DCCLayer (None, 384, 18, 18) produces 124416 outputs
MaxPool2DCCLayer (None, 384, 9, 9) produces 31104 outputs
DropoutLayer (None, 384, 9, 9) produces 31104 outputs
DenseLayer (None, 1000) produces 1000 outputs
DropoutLayer (None, 1000) produces 1000 outputs
DenseLayer (None, 1000) produces 1000 outputs
DenseLayer (None, 121) produces 121 outputs
I have also tried adding in a third Conv2DCCLayer and also a DropoutLayer after my last DenseLayer but that seems to not help. I think I need to increase my layer size to something bigger than 192 and 384 but not sure.
\nWhat have you tried?
", "votes": null, "replies": [] }, { "id": 63145, "author_name": "felixlaumon", "author_url": "", "post_date": "01/25/2015 16:27:12", "content": "Ahh fellow nolearn user! I tried adding a 3rd convolutional layer too but I don't think it's helping. One thing that frustrates me is I don't know how to qualitatively compares 2 models. I tried visualizing the weights learned from the filter but it's hard to interpret.
\nNow I'm trying out maxout layers (layers.FeaturePoolLayer) (see http://arxiv.org/pdf/1302.4389v4.pdf) and debugging my batch iterator to make sure the data augmentation is working properly.
", "votes": null, "replies": [] }, { "id": 63149, "author_name": "msegala", "author_url": "", "post_date": "01/25/2015 18:10:45", "content": "Have you experimented with different dropout values? I found that 0.5 seems to work best but didn't try to many values or scenarios?
", "votes": null, "replies": [] }, { "id": 63151, "author_name": "jaythodenvanvelzen", "author_url": "", "post_date": "01/25/2015 19:02:24", "content": "I just submitted a 1.08 score with:
\n\nInput: 60x60
\nL1 weights: 1 x 32 x 5x5 convolution
\nmaxpool
\nL2 weights: 32 x 64 x 5x5 convolution
\nmaxpool
\nL3 weights: 64 x 128 x 5x5 convolution
\nmaxpool
\nflatten
\nL4: 128x4x4 x 2025
\nL5: 2025 x 121 (softmax)
\nDropout hidden: 0.5, dropout convolution: 0.2
\nBatch sizes of 512, learning rate of 0.001 with RMSProp; training source images rotated 45 degrees, then all 90 degrees, then all 180 degrees.
\nI tried higher dropout values, but it just slows learning down dramatically.
", "votes": null, "replies": [] }, { "id": 63163, "author_name": "miroslaw", "author_url": "", "post_date": "01/25/2015 21:37:47", "content": "[quote=Jay Thoden van Velzen;63151]
\nI just submitted a 1.08 score with:
\nInput: 60x60
\nL1 weights: 1 x 32 x 5x5 convolution
\nmaxpool
\nL2 weights: 32 x 64 x 5x5 convolution
\nmaxpool
\nL3 weights: 64 x 128 x 5x5 convolution
\nmaxpool
\nflatten
\nL4: 128x4x4 x 2025
\nL5: 2025 x 121 (softmax)
\nDropout hidden: 0.5, dropout convolution: 0.2
\nBatch sizes of 512, learning rate of 0.001 with RMSProp; training source images rotated 45 degrees, then all 90 degrees, then all 180 degrees.
\nI tried higher dropout values, but it just slows learning down dramatically.
\n[/quote]
Maybe my implementation of RMSProp is incorrect, but I was unable to get RMSProp to work very well.
Miroslaw, can you provide a reference for RMSProp, I would like to try it out. Thanks!
", "votes": null, "replies": [] }, { "id": 63394, "author_name": "jaythodenvanvelzen", "author_url": "", "post_date": "01/31/2015 18:46:41", "content": "Here is my RMSProp function, as well as one with Nesterov momentum (though that didn't give me better results):
\n\ndef RMSprop(cost, params, lr=0.001, rho=0.9, epsilon=1e-6):
grads = T.grad(cost=cost, wrt=params)
updates = []
for p, g in zip(params, grads):
acc = theano.shared(p.get_value() * 0.)
acc_new = rho * acc + (1 - rho) * g ** 2
gradient_scaling = T.sqrt(acc_new + epsilon)
g = g / gradient_scaling
updates.append((acc, acc_new))
updates.append((p, p - lr * g))
return updates
def RMSprop_nesterov(cost, params, lr=0.001, rho=0.9, momentum=0.7, epsilon=1e-6):
grads = T.grad(cost=cost, wrt=params)
updates = []
for p, g in zip(params, grads):
mo_p = theano.shared(p.get_value() * 0.)
acc = theano.shared(p.get_value() * 0.)
acc_new = rho * acc + (1 - rho) * g**2step = lr * g / T.sqrt(acc_new + epsilon)
v = momentum * mo_p - step
w_ = p + momentum * v - step
updates.append((acc, acc_new))
updates.append((mo_p, v))
updates.append((p, w_))
return updates