This is just a quick overview for those curious about our solution. It seems quite simple compared to the other top two teams. I would like to make a blog post of how I approached this competition since this was my first time using CNNs, and my experience may be valuable for others as I was rapidly improving my methods towards the end of the competition. Of course I have also extensively followed the progress of neural networks since 2012's ImageNet competition, which helped a lot.
\nWhy Spatially Sparse Networks?
\nI think this was probably a large advantage in this competition. Most of these images are simply white background. In fact I believe only 15% of the pixels had initial activations in my networks. This architecture allowed for a lot of flexibility in data augmentation since you can have really wide input fields and not greatly hurt performance.
\nNetwork Architecture
\nThe architectures are mostly based off of templates for Spatially Sparse Networks with varying the number of layers between max pools.
\nI tried 27 layer 2x2 convolutional filters with 5 2x2 Max Pooling layers. I believe that a 3x3 with 2 stride would have produced better results. I also had 18 layers with wider nets and 22 layers. All worked very well and produced single network scores of 0.618 - 0.625. Most networks had 4-5 max pooling layers.
\nI eventually settled on a network similar to VGG with 3x3 convolutional filters. This was a shorter network with only 14 layers. These scored from 0.610 to 0.620. They were also moderately small. My networks were 3 million to 11 million parameters in size with moderate drop out. (0.2)
\nDr. Ben Graham was able to achieve a single network score of 0.599 with a network consisting of 12 Fractional Max Pooling layers. (see FMP Paper) He also had a network with 0.608 which used 3x3 max pooling layers with a stride of 2.
\nData Augmentation
\nEverything was done in real time. Dr. Ben Graham used affine transformations and I was using perspective transformations. The real key is that pretty much anything that doesn't change classes is fair game. If you look at the test data, many classes are heavily biased to certain orientations. However, it is best not to try and match these biases.
\nToo much of a good thing can be bad. It was good to add small random variations in scaling, but if you increased it too much then the score was greatly hurt. I read somewhere that CNNs have problems dealing with data at different scales. It seems that this may be somewhat true.
\nSince this was a spatially sparse network I found that dividing the largest dimension of each input image by 2.5 and adding 30 produced much better results than simply scaling the largest dimension to 80 or 100 or something similar.
\nEncoding
\nI have no idea how much the methods I used here really helped. I thresholded out any pixels > 252. I then calculated the global mean and standard deviation of the training set, and each individual image over the remaining pixels. I also calculated a windowed (11x11) mean and standard deviation for each pixel. Those were combined to form 3 channels. I added a forth channel which used the global mean and image standard deviation.
\nI really should have explored how I encoded the images more as I believe this method caused several classes to be very difficult to predict properly. (see below)
\nBen used a different method which can be seen in his code at his page.
\nProblem Areas
\nI believe that the encoding method I used made several classes very difficult for my network to predict properly. This problem did not appear in Ben's networks.
\nArtifact_Edges was predicted as 2.4% of my total aggregated classes. It should have been closer to 0.56% (Based on training data). Protist_dark_center was predicted as 1.14% of my total aggregated classes. It should have been closer to 0.36%. These two classes were very large outliers for me.
\nOur final ensemble was a little better with these classes, but this definitely hurt our final score.
\nEnsemble
\nOur final ensemble had 15ish networks with Dr. Graham's best net being counted twice. However, an ensemble of 5 networks reached 99.8% of the best score.
\nI am really amazed out how you can add a network with a score of 0.63 and still see an improvement in score.
\nSome Things Different
\nSometimes training really deep nets with dropout can make convergence painfully slow at the start. So I implemented a gradual drop out where I start at 0 drop out and introduce more and more drop out as the score improves on the training set. Usually by the 10-20th epoch I have reached maximum dropout across all layers. This doesn't seem to hurt final performance of the network.
\nSome Things Bad
\nTo save time I tried to take partially trained networks and then change the input variables to create a new net. This was not a good idea.
\nLast Words
\nI haven't consulted with Dr. Graham before writing this so he may have something more to add.
\nIf you have any questions I will be glad to try and answer them.
", "messages": [ { "id": "67269", "postDate": "03/19/2015 19:02:58", "content": "This is just a quick overview for those curious about our solution. It seems quite simple compared to the other top two teams. I would like to make a blog post of how I approached this competition since this was my first time using CNNs, and my experience may be valuable for others as I was rapidly improving my methods towards the end of the competition. Of course I have also extensively followed the progress of neural networks since 2012's ImageNet competition, which helped a lot.
\nWhy Spatially Sparse Networks?
\nI think this was probably a large advantage in this competition. Most of these images are simply white background. In fact I believe only 15% of the pixels had initial activations in my networks. This architecture allowed for a lot of flexibility in data augmentation since you can have really wide input fields and not greatly hurt performance.
\nNetwork Architecture
\nThe architectures are mostly based off of templates for Spatially Sparse Networks with varying the number of layers between max pools.
\nI tried 27 layer 2x2 convolutional filters with 5 2x2 Max Pooling layers. I believe that a 3x3 with 2 stride would have produced better results. I also had 18 layers with wider nets and 22 layers. All worked very well and produced single network scores of 0.618 - 0.625. Most networks had 4-5 max pooling layers.
\nI eventually settled on a network similar to VGG with 3x3 convolutional filters. This was a shorter network with only 14 layers. These scored from 0.610 to 0.620. They were also moderately small. My networks were 3 million to 11 million parameters in size with moderate drop out. (0.2)
\nDr. Ben Graham was able to achieve a single network score of 0.599 with a network consisting of 12 Fractional Max Pooling layers. (see FMP Paper) He also had a network with 0.608 which used 3x3 max pooling layers with a stride of 2.
\nData Augmentation
\nEverything was done in real time. Dr. Ben Graham used affine transformations and I was using perspective transformations. The real key is that pretty much anything that doesn't change classes is fair game. If you look at the test data, many classes are heavily biased to certain orientations. However, it is best not to try and match these biases.
\nToo much of a good thing can be bad. It was good to add small random variations in scaling, but if you increased it too much then the score was greatly hurt. I read somewhere that CNNs have problems dealing with data at different scales. It seems that this may be somewhat true.
\nSince this was a spatially sparse network I found that dividing the largest dimension of each input image by 2.5 and adding 30 produced much better results than simply scaling the largest dimension to 80 or 100 or something similar.
\nEncoding
\nI have no idea how much the methods I used here really helped. I thresholded out any pixels > 252. I then calculated the global mean and standard deviation of the training set, and each individual image over the remaining pixels. I also calculated a windowed (11x11) mean and standard deviation for each pixel. Those were combined to form 3 channels. I added a forth channel which used the global mean and image standard deviation.
\nI really should have explored how I encoded the images more as I believe this method caused several classes to be very difficult to predict properly. (see below)
\nBen used a different method which can be seen in his code at his page.
\nProblem Areas
\nI believe that the encoding method I used made several classes very difficult for my network to predict properly. This problem did not appear in Ben's networks.
\nArtifact_Edges was predicted as 2.4% of my total aggregated classes. It should have been closer to 0.56% (Based on training data). Protist_dark_center was predicted as 1.14% of my total aggregated classes. It should have been closer to 0.36%. These two classes were very large outliers for me.
\nOur final ensemble was a little better with these classes, but this definitely hurt our final score.
\nEnsemble
\nOur final ensemble had 15ish networks with Dr. Graham's best net being counted twice. However, an ensemble of 5 networks reached 99.8% of the best score.
\nI am really amazed out how you can add a network with a score of 0.63 and still see an improvement in score.
\nSome Things Different
\nSometimes training really deep nets with dropout can make convergence painfully slow at the start. So I implemented a gradual drop out where I start at 0 drop out and introduce more and more drop out as the score improves on the training set. Usually by the 10-20th epoch I have reached maximum dropout across all layers. This doesn't seem to hurt final performance of the network.
\nSome Things Bad
\nTo save time I tried to take partially trained networks and then change the input variables to create a new net. This was not a good idea.
\nLast Words
\nI haven't consulted with Dr. Graham before writing this so he may have something more to add.
\nIf you have any questions I will be glad to try and answer them.
", "rawMarkdown": "", "votes": null }, { "id": "67280", "postDate": "03/19/2015 19:51:57", "content": "Nice and simple solution, thanks for sharing!
", "rawMarkdown": "", "votes": null }, { "id": "67323", "postDate": "03/19/2015 22:21:41", "content": "Great writeup!
\nI was wondering why none of the other solutions used sparse nets, it seems like such a perfect match for this data. With ~80% of the image space being more or less blank, a standard convnet will waste a LOT of cycles on doing nothing, basically.
", "rawMarkdown": "", "votes": null }, { "id": "67334", "postDate": "03/19/2015 23:22:40", "content": "This is really, really interesting! Thanks for the write-up! It's also quite different from what we came up with, so chances are that combining some of these ideas might improve things further.
\nAlso, massive respect to you if this is really your first time working with convnets. Amazing work!
\n[quote=Ryan Munion;67269]Dr. Ben Graham was able to achieve a single network score of 0.599 with a network consisting of 12 Fractional Max Pooling layers.[/quote]
\nIs this a leaderboard score? Was any averaging of predictions (i.e. test-time augmentation) done to arrive at this score? Because that's pretty impressive. I think the lowest we got for a single model (not counting models trained with pseudo-labeling) was about 0.607, and this was the score with test-time augmentation (i.e. each prediction was actually an average of 70 predictions).
\n[quote]Too much of a good thing can be bad. It was good to add small random variations in scaling, but if you increased it too much then the score was greatly hurt. I read somewhere that CNNs have problems dealing with data at different scales. It seems that this may be somewhat true.[/quote]
\nHow large were the scale variations in the end? We ended up jittering the scale of the images between 1/1.6 and 1.6 (after initial size-based rescaling), which I found pretty extreme, but it seemed to help.
\n[quote]Since this was a spatially sparse network I found that dividing the largest dimension of each input image by 2.5 and adding 30 produced much better results than simply scaling the largest dimension to 80 or 100 or something similar.[/quote]
\nSo if I understand this correctly, your rescaling approximately preserved the relative sizes of the organisms?
\n[quote]Ben used a different method which can be seen in his code at his page.[/quote]
\nIs this code available online anywhere yet? I can't seem to find it but I'm very curious :)
\n[quote]Our final ensemble had 15ish networks with Dr. Graham's best net being counted twice. However, an ensemble of 5 networks reached 99.8% of the best score.[/quote]
\nIf your single best network achieved 0.599 on the leaderboard I'm somewhat surprised that the ensemble score isn't better, to be honest! Did you just average the predictions of all the networks together?
\n[quote]I am really amazed out how you can add a network with a score of 0.63 and still see an improvement in score.[/quote]
\nWe had the same experience. We were sometimes very surprised by the models that would be selected by the ensembling procedure.
\n[quote]Sometimes training really deep nets with dropout can make convergence painfully slow at the start. So I implemented a gradual drop out where I start at 0 drop out and introduce more and more drop out as the score improves on the training set. Usually by the 10-20th epoch I have reached maximum dropout across all layers. This doesn't seem to hurt final performance of the network.[/quote]
\nThis is an excellent idea. We ended up not using dropout in any of the convolutional layers because it tended to slow down training too much. The few times we attempted to use it, the effect probably wasn't strong enough to make a difference.
", "rawMarkdown": "", "votes": null }, { "id": "67342", "postDate": "03/20/2015 00:32:33", "content": "Great questions! And yes, it is my first time though I do computer vision work for a living. I spent the first half of the competition trying to get a breakthrough on semisupervised learning on Cifar 100 data set. I learned that coming up with breakthroughs is hard!
\n[quote]
\nIs this a leaderboard score? Was any averaging of predictions (i.e. test-time augmentation) done to arrive at this score? Because that's pretty impressive. I think the lowest we got for a single model (not counting models trained with pseudo-labeling) was about 0.607, and this was the score with test-time augmentation (i.e. each prediction was actually an average of 70 predictions).
\n[/quote]
\nIt was a leaderboard score! He submitted it 2 hours before the end of the competition. It was a test time augmentation over 20-30 tests. All of our nets used test time augmentation. However, our method differed from yours in that I just did random transformations of the test data exactly like the training data.
\nHe used pseudo random overlapping fractional max pooling layers. Previously, the same net scored 0.608 I think. Unfortunately, substituting the net predictions didn't help our overall score very much.
\nNone of our nets used soft labeling techniques, though I did explore such methods early on.
\n[quote]How large were the scale variations in the end? We ended up jittering the scale of the images between 1/1.6 and 1.6 (after initial size-based rescaling), which I found pretty extreme, but it seemed to help.[/quote]
\nI believe they were 1/2.2 to 2.2. Some of this could be the size of the networks I was working with at the time. I ended up with only doing +- 30% on most of my networks.
\n[quote]So if I understand this correctly, your rescaling approximately preserved the relative sizes of the organisms?[/quote]
\nYes. It helped quite a bit. But it was horrible use of memory, because if I randomly selected several large images in a single batch my application would crash. So I had to reduce the batch size to 20.
\n[quote]
\nIs this code available online anywhere yet? I can't seem to find it but I'm very curious :)
\n[/quote]
\nHis FMP work is not yet available (It will be), but he has his spatially sparse code available which shows his augmentations.
\n[quote]
\nIf your single best network achieved 0.599 on the leaderboard I'm somewhat surprised that the ensemble score isn't better, to be honest! Did you just average the predictions of all the networks together?[/quote]
\nYes, we did just average the scores together. To count twice I simply dropped a copy of the prediction file into my combine directory. It was very crude, but I was running out of time. I think the biggest problem was that I was greatly over predicting artifact_edges and protist_dark_center. All of my networks were biased in this same manner. The local normalization I used was probably greatly over exaggerating the background noise in many of the images.
\nI did try replacing my predictions for those classes with Ben's predictions with five minutes left in the competition, but it was too little too late.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 67280, "author_name": "oldufo", "author_url": "", "post_date": "03/19/2015 19:51:57", "content": "Nice and simple solution, thanks for sharing!
", "votes": null, "replies": [] }, { "id": 67323, "author_name": "nicolaykorslund", "author_url": "", "post_date": "03/19/2015 22:21:41", "content": "Great writeup!
\nI was wondering why none of the other solutions used sparse nets, it seems like such a perfect match for this data. With ~80% of the image space being more or less blank, a standard convnet will waste a LOT of cycles on doing nothing, basically.
", "votes": null, "replies": [] }, { "id": 67334, "author_name": "sedielem", "author_url": "", "post_date": "03/19/2015 23:22:40", "content": "This is really, really interesting! Thanks for the write-up! It's also quite different from what we came up with, so chances are that combining some of these ideas might improve things further.
\nAlso, massive respect to you if this is really your first time working with convnets. Amazing work!
\n[quote=Ryan Munion;67269]Dr. Ben Graham was able to achieve a single network score of 0.599 with a network consisting of 12 Fractional Max Pooling layers.[/quote]
\nIs this a leaderboard score? Was any averaging of predictions (i.e. test-time augmentation) done to arrive at this score? Because that's pretty impressive. I think the lowest we got for a single model (not counting models trained with pseudo-labeling) was about 0.607, and this was the score with test-time augmentation (i.e. each prediction was actually an average of 70 predictions).
\n[quote]Too much of a good thing can be bad. It was good to add small random variations in scaling, but if you increased it too much then the score was greatly hurt. I read somewhere that CNNs have problems dealing with data at different scales. It seems that this may be somewhat true.[/quote]
\nHow large were the scale variations in the end? We ended up jittering the scale of the images between 1/1.6 and 1.6 (after initial size-based rescaling), which I found pretty extreme, but it seemed to help.
\n[quote]Since this was a spatially sparse network I found that dividing the largest dimension of each input image by 2.5 and adding 30 produced much better results than simply scaling the largest dimension to 80 or 100 or something similar.[/quote]
\nSo if I understand this correctly, your rescaling approximately preserved the relative sizes of the organisms?
\n[quote]Ben used a different method which can be seen in his code at his page.[/quote]
\nIs this code available online anywhere yet? I can't seem to find it but I'm very curious :)
\n[quote]Our final ensemble had 15ish networks with Dr. Graham's best net being counted twice. However, an ensemble of 5 networks reached 99.8% of the best score.[/quote]
\nIf your single best network achieved 0.599 on the leaderboard I'm somewhat surprised that the ensemble score isn't better, to be honest! Did you just average the predictions of all the networks together?
\n[quote]I am really amazed out how you can add a network with a score of 0.63 and still see an improvement in score.[/quote]
\nWe had the same experience. We were sometimes very surprised by the models that would be selected by the ensembling procedure.
\n[quote]Sometimes training really deep nets with dropout can make convergence painfully slow at the start. So I implemented a gradual drop out where I start at 0 drop out and introduce more and more drop out as the score improves on the training set. Usually by the 10-20th epoch I have reached maximum dropout across all layers. This doesn't seem to hurt final performance of the network.[/quote]
\nThis is an excellent idea. We ended up not using dropout in any of the convolutional layers because it tended to slow down training too much. The few times we attempted to use it, the effect probably wasn't strong enough to make a difference.
", "votes": null, "replies": [] }, { "id": 67342, "author_name": "iamthep", "author_url": "", "post_date": "03/20/2015 00:32:33", "content": "Great questions! And yes, it is my first time though I do computer vision work for a living. I spent the first half of the competition trying to get a breakthrough on semisupervised learning on Cifar 100 data set. I learned that coming up with breakthroughs is hard!
\n[quote]
\nIs this a leaderboard score? Was any averaging of predictions (i.e. test-time augmentation) done to arrive at this score? Because that's pretty impressive. I think the lowest we got for a single model (not counting models trained with pseudo-labeling) was about 0.607, and this was the score with test-time augmentation (i.e. each prediction was actually an average of 70 predictions).
\n[/quote]
\nIt was a leaderboard score! He submitted it 2 hours before the end of the competition. It was a test time augmentation over 20-30 tests. All of our nets used test time augmentation. However, our method differed from yours in that I just did random transformations of the test data exactly like the training data.
\nHe used pseudo random overlapping fractional max pooling layers. Previously, the same net scored 0.608 I think. Unfortunately, substituting the net predictions didn't help our overall score very much.
\nNone of our nets used soft labeling techniques, though I did explore such methods early on.
\n[quote]How large were the scale variations in the end? We ended up jittering the scale of the images between 1/1.6 and 1.6 (after initial size-based rescaling), which I found pretty extreme, but it seemed to help.[/quote]
\nI believe they were 1/2.2 to 2.2. Some of this could be the size of the networks I was working with at the time. I ended up with only doing +- 30% on most of my networks.
\n[quote]So if I understand this correctly, your rescaling approximately preserved the relative sizes of the organisms?[/quote]
\nYes. It helped quite a bit. But it was horrible use of memory, because if I randomly selected several large images in a single batch my application would crash. So I had to reduce the batch size to 20.
\n[quote]
\nIs this code available online anywhere yet? I can't seem to find it but I'm very curious :)
\n[/quote]
\nHis FMP work is not yet available (It will be), but he has his spatially sparse code available which shows his augmentations.
\n[quote]
\nIf your single best network achieved 0.599 on the leaderboard I'm somewhat surprised that the ensemble score isn't better, to be honest! Did you just average the predictions of all the networks together?[/quote]
\nYes, we did just average the scores together. To count twice I simply dropped a copy of the prediction file into my combine directory. It was very crude, but I was running out of time. I think the biggest problem was that I was greatly over predicting artifact_edges and protist_dark_center. All of my networks were biased in this same manner. The local normalization I used was probably greatly over exaggerating the background noise in many of the images.
\nI did try replacing my predictions for those classes with Ben's predictions with five minutes left in the competition, but it was too little too late.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "67269": "", "67280": "", "67323": "", "67334": "", "67342": "" }, "source": "existing_discussion", "complete": true, "competition_id": 3978, "competition_slug": "datasciencebowl", "team_id": 153620, "solution_url": "/competitions/datasciencebowl/writeups/poisson-process-poisson-process-solution" }