Sorry about my poor English.
\nBrief review of this competition.
I joined this competition at around two months before the deadline. One of my intention to joining this game is to validate an algorithm I recently developed. So basically I only use one bullet for this competition, it works pretty well. However I found a single bullet is not enough to win the game when Deep Sea surpassed me near the end of this competition. To win the first prize, I contacted Bing’s team for cooperation and finally we formed the team called Happy Lantern Festival (the name of a Chinese traditional festival) and won the second prize.
It is pleasant to work with Bing’s team, I have learned a lot during the cooperation. Due to the limited time (only 10 days left) and various time zones we are living in, we majorly cooperate in the simplest way – score combing. We moderately improved our scores after the team merge.
A practical theory for designing very deep convolutional neural network.
In Bing’s post, he has summarize the solutions from his side. Herein I majorly describe my bullet for this competition: a practical theory for designing very deep convolutional neural network. In short, I cast the design of deep architecture into an optimization problem. The objective is maximizing the depth of the target convolutional neural network, subjecting to two constraints (1) the c-value of each layer should not be too small, c-value is a metric I defined for measuring the capacity of a convolutional layer can learn new and more complex patterns; (2) the receptive field of the topmost convolutional layer in the feature-level should no larger than the size of input image. I proofed under certain conditions the close form optimal solution can be obtained. Please check the attached tech report for more detailed interpretations, mathematical formulations/solutions, as well as further discussions.
The networks designed according to the theory can achieve 0.606~0.610 score on the leader board, as a reference, the best of VGG-like networks (with some improvements) achieves around 0.63 score on the leader board.
Model combine
A set of good models does not imply a good ensemble. Performance stop growing when I add more than 4 good models into the ensemble. Interestingly, it helps to add some mediocre models (similar findings are also reported by Deep Sea). The final ensemble is consist of 4 good models, a VGG-like model and 2 variants of the good models. The 7-model ensemble achieved 0.582 leader board score. Combining with Bing’s score, we final achieved 0.574 score on the public leader board.
Training and testing data augmentation
I used affine transformation for both training augmentation and testing augmentation. The training augmentation is more aggressive comparing to the testing augmentation. For training, the scale range is 0.2~2.0, the shear range is -0.7~0.7, the ratio range is 0.6~1.4, the rotation range is –pi~pi; for testing the scale range 0.6~1.4, the shear range is -0.5~0.5, the ratio range is 0.8~1.2, the rotation range is –pi~pi. All parameters are randomly sampled from uniform distribution. It is worth noting the augmentation configurations are entangled with the CNN structures. The stronger the fitting power a CNN has, the more aggressive augmentation should be applied.
Some mistakes I made in this competition.
A bullet is not enough to win a competition. Deep sea use a very smart strategy in this competition i.e. accumulating many good bullets and fire them all at the very end of the competition.
I did not try Dr. Ben’s fractional max pooling. I used convnet2 for this competition, it is cool in terms of its speed, but its framework is not friendly to adding new features such as Fractional Max Pooling (Dr. Ben), Deeply Supervised Networks (Zhuowen Tu), Batch Normalization (Google) etc.
No task specific design/features. This is exactly where I appreciate the first prize winner deep sea, their cyclic pooling/rolling and additional specifically designed features are very insightful and substantially improve the score.
The attached files
1. I use convnet2 for this competition. I only changed the data provider of convnet2 for online data augmentation. The revised files are zipped in the revised_files.zip.
2. Algorithm report: A practical theory for designing very deep convolutional neural network
Sorry about my poor English.
\nBrief review of this competition.
I joined this competition at around two months before the deadline. One of my intention to joining this game is to validate an algorithm I recently developed. So basically I only use one bullet for this competition, it works pretty well. However I found a single bullet is not enough to win the game when Deep Sea surpassed me near the end of this competition. To win the first prize, I contacted Bing’s team for cooperation and finally we formed the team called Happy Lantern Festival (the name of a Chinese traditional festival) and won the second prize.
It is pleasant to work with Bing’s team, I have learned a lot during the cooperation. Due to the limited time (only 10 days left) and various time zones we are living in, we majorly cooperate in the simplest way – score combing. We moderately improved our scores after the team merge.
A practical theory for designing very deep convolutional neural network.
In Bing’s post, he has summarize the solutions from his side. Herein I majorly describe my bullet for this competition: a practical theory for designing very deep convolutional neural network. In short, I cast the design of deep architecture into an optimization problem. The objective is maximizing the depth of the target convolutional neural network, subjecting to two constraints (1) the c-value of each layer should not be too small, c-value is a metric I defined for measuring the capacity of a convolutional layer can learn new and more complex patterns; (2) the receptive field of the topmost convolutional layer in the feature-level should no larger than the size of input image. I proofed under certain conditions the close form optimal solution can be obtained. Please check the attached tech report for more detailed interpretations, mathematical formulations/solutions, as well as further discussions.
The networks designed according to the theory can achieve 0.606~0.610 score on the leader board, as a reference, the best of VGG-like networks (with some improvements) achieves around 0.63 score on the leader board.
Model combine
A set of good models does not imply a good ensemble. Performance stop growing when I add more than 4 good models into the ensemble. Interestingly, it helps to add some mediocre models (similar findings are also reported by Deep Sea). The final ensemble is consist of 4 good models, a VGG-like model and 2 variants of the good models. The 7-model ensemble achieved 0.582 leader board score. Combining with Bing’s score, we final achieved 0.574 score on the public leader board.
Training and testing data augmentation
I used affine transformation for both training augmentation and testing augmentation. The training augmentation is more aggressive comparing to the testing augmentation. For training, the scale range is 0.2~2.0, the shear range is -0.7~0.7, the ratio range is 0.6~1.4, the rotation range is –pi~pi; for testing the scale range 0.6~1.4, the shear range is -0.5~0.5, the ratio range is 0.8~1.2, the rotation range is –pi~pi. All parameters are randomly sampled from uniform distribution. It is worth noting the augmentation configurations are entangled with the CNN structures. The stronger the fitting power a CNN has, the more aggressive augmentation should be applied.
Some mistakes I made in this competition.
A bullet is not enough to win a competition. Deep sea use a very smart strategy in this competition i.e. accumulating many good bullets and fire them all at the very end of the competition.
I did not try Dr. Ben’s fractional max pooling. I used convnet2 for this competition, it is cool in terms of its speed, but its framework is not friendly to adding new features such as Fractional Max Pooling (Dr. Ben), Deeply Supervised Networks (Zhuowen Tu), Batch Normalization (Google) etc.
No task specific design/features. This is exactly where I appreciate the first prize winner deep sea, their cyclic pooling/rolling and additional specifically designed features are very insightful and substantially improve the score.
The attached files
1. I use convnet2 for this competition. I only changed the data provider of convnet2 for online data augmentation. The revised files are zipped in the revised_files.zip.
2. Algorithm report: A practical theory for designing very deep convolutional neural network
Thanks so much for this!
", "rawMarkdown": "", "votes": null }, { "id": "69209", "postDate": "03/31/2015 19:09:29", "content": "Great, thanks!
\nP.S. Do you have results for each of nets from table 3? Also it is interesting, why you jump from 48x48 to 112x112.
", "rawMarkdown": "", "votes": null }, { "id": "69227", "postDate": "03/31/2015 22:12:19", "content": "Congratulations to Xudong Cao, Bing Xu, and the rest of the Happy Lantern Festival team for achieving the 2nd place solution! It's very interesting that you talk about trying to use a single model for this competition compared to an ensemble approach. As noted in conversations with Sander from the Deep Sea team, we will likely only use a single best model in our actual implementation of this solution, in which case the 'single bullet' model might be a strong contender. We look forward to working through your code and model construction!
\n\n- Jessica, on behalf of the Cowen Lab
", "rawMarkdown": "", "votes": null }, { "id": "69256", "postDate": "04/01/2015 03:22:41", "content": "It would be interesting to see how good you could get by generating soft labels of training + test sets (dark knowledge) and then training a single net. You could just ensemble the top 10 predictions and get pretty good soft labels.
\nI know that by using soft labels using our ensembled predictions I could get much higher scoring single nets (<0.6). Though they didn't help the score at all when adding them back into the ensemble.
", "rawMarkdown": "", "votes": null }, { "id": "69274", "postDate": "04/01/2015 07:37:14", "content": "[quote=Jessica Luo;69227]
\nCongratulations to Xudong Cao, Bing Xu, and the rest of the Happy Lantern Festival team for achieving the 2nd place solution! It's very interesting that you talk about trying to use a single model for this competition compared to an ensemble approach. As noted in conversations with Sander from the Deep Sea team, we will likely only use a single best model in our actual implementation of this solution, in which case the 'single bullet' model might be a strong contender. We look forward to working through your code and model construction!
\n- Jessica, on behalf of the Cowen Lab
\n[/quote]
\nHi Jessica,
\nThank you very much!
\nI would like to clarify one misunderstanding. By one bullet, I mean I only use one algorithm for designing convolutional neural network, not only use one model/network. My best models achieve 0.606~0.610 on public leader board.
\nIf only one model is chosen, I recommend you try the model on 128x128 image using 3x3 filter size. You can find the network configuration in my algorithm report in Table 3. It is also worth trying Dr. Ben's fractional max pooling. The randomness in scale and translation is an effective approach to fight against the over-fitting problem.
\n\nThanks,
\nXudong
", "rawMarkdown": "", "votes": null }, { "id": "69276", "postDate": "04/01/2015 07:52:02", "content": "[quote=Ryan Munion;69256]
\nIt would be interesting to see how good you could get by generating soft labels of training + test sets (dark knowledge) and then training a single net. You could just ensemble the top 10 predictions and get pretty good soft labels.
\nI know that by using soft labels using our ensembled predictions I could get much higher scoring single nets (<0.6). Though they didn't help the score at all when adding them back into the ensemble.
\n[/quote]
\n\nHi Ryan,
\n\nThanks for your suggestions. I have tried pseudo labeling 130k testing data during the competition but I did not make it work. It did not improve the score on my local validation set. In the final submission, I did not use pseudo labeled 130k testing data for training.
\n\nIt is quite surprising to me that Dr. Ben's single model can achieve 0.599 score on the public leader board, that is why I regret that I did not try his fractional max pooling during the competition.
\n\nThanks,
\nXudong
", "rawMarkdown": "", "votes": null }, { "id": "69281", "postDate": "04/01/2015 08:24:08", "content": "[quote=old-ufo;69209]
\nGreat, thanks!
\nP.S. Do you have results for each of nets from table 3? Also it is interesting, why you jump from 48x48 to 112x112.
\n[/quote]
\n\nDear old-ufo,
\nYour name is very cool :)
\nBack to your question, I start training my model on 48x48 image with much less aggressive data augmentation. I quickly found that using larger image size and more aggressive data augmentation will greatly improve the performance. Therefore I trained more models on 96x96, 112x112, 128x128, 144x144 and 192x192 image size using various filter size i.e. 2x2, 3x3, 4x4, 5x5. In the end of this competition, I simply manually select several of them for model combination. In general, the models using 3x3, 4x4, 5x5 filter size performs better than the models using 2x2 filter size. But I found model with 2x2 filter size is complementary to the final ensemble.
\n\nThe model on 48x48 image using 5x5 filter size: 0.631 log loss score on the public leader board
\nThe model on 112x112 image using 2x2 filter size: 0.621 log loss score on the public leader board
", "rawMarkdown": "", "votes": null }, { "id": "69284", "postDate": "04/01/2015 08:34:59", "content": "Thanks for clarification. Looks like you are only person who experimented with 2x2 filters except Poisson Process team.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 69196, "author_name": "inversion", "author_url": "", "post_date": "03/31/2015 18:18:04", "content": "Thanks so much for this!
", "votes": null, "replies": [] }, { "id": 69209, "author_name": "oldufo", "author_url": "", "post_date": "03/31/2015 19:09:29", "content": "Great, thanks!
\nP.S. Do you have results for each of nets from table 3? Also it is interesting, why you jump from 48x48 to 112x112.
", "votes": null, "replies": [] }, { "id": 69227, "author_name": "jessicaluo", "author_url": "", "post_date": "03/31/2015 22:12:19", "content": "Congratulations to Xudong Cao, Bing Xu, and the rest of the Happy Lantern Festival team for achieving the 2nd place solution! It's very interesting that you talk about trying to use a single model for this competition compared to an ensemble approach. As noted in conversations with Sander from the Deep Sea team, we will likely only use a single best model in our actual implementation of this solution, in which case the 'single bullet' model might be a strong contender. We look forward to working through your code and model construction!
\n\n- Jessica, on behalf of the Cowen Lab
", "votes": null, "replies": [] }, { "id": 69256, "author_name": "iamthep", "author_url": "", "post_date": "04/01/2015 03:22:41", "content": "It would be interesting to see how good you could get by generating soft labels of training + test sets (dark knowledge) and then training a single net. You could just ensemble the top 10 predictions and get pretty good soft labels.
\nI know that by using soft labels using our ensembled predictions I could get much higher scoring single nets (<0.6). Though they didn't help the score at all when adding them back into the ensemble.
", "votes": null, "replies": [] }, { "id": 69274, "author_name": "xudongcao", "author_url": "", "post_date": "04/01/2015 07:37:14", "content": "[quote=Jessica Luo;69227]
\nCongratulations to Xudong Cao, Bing Xu, and the rest of the Happy Lantern Festival team for achieving the 2nd place solution! It's very interesting that you talk about trying to use a single model for this competition compared to an ensemble approach. As noted in conversations with Sander from the Deep Sea team, we will likely only use a single best model in our actual implementation of this solution, in which case the 'single bullet' model might be a strong contender. We look forward to working through your code and model construction!
\n- Jessica, on behalf of the Cowen Lab
\n[/quote]
\nHi Jessica,
\nThank you very much!
\nI would like to clarify one misunderstanding. By one bullet, I mean I only use one algorithm for designing convolutional neural network, not only use one model/network. My best models achieve 0.606~0.610 on public leader board.
\nIf only one model is chosen, I recommend you try the model on 128x128 image using 3x3 filter size. You can find the network configuration in my algorithm report in Table 3. It is also worth trying Dr. Ben's fractional max pooling. The randomness in scale and translation is an effective approach to fight against the over-fitting problem.
\n\nThanks,
\nXudong
", "votes": null, "replies": [] }, { "id": 69276, "author_name": "xudongcao", "author_url": "", "post_date": "04/01/2015 07:52:02", "content": "[quote=Ryan Munion;69256]
\nIt would be interesting to see how good you could get by generating soft labels of training + test sets (dark knowledge) and then training a single net. You could just ensemble the top 10 predictions and get pretty good soft labels.
\nI know that by using soft labels using our ensembled predictions I could get much higher scoring single nets (<0.6). Though they didn't help the score at all when adding them back into the ensemble.
\n[/quote]
\n\nHi Ryan,
\n\nThanks for your suggestions. I have tried pseudo labeling 130k testing data during the competition but I did not make it work. It did not improve the score on my local validation set. In the final submission, I did not use pseudo labeled 130k testing data for training.
\n\nIt is quite surprising to me that Dr. Ben's single model can achieve 0.599 score on the public leader board, that is why I regret that I did not try his fractional max pooling during the competition.
\n\nThanks,
\nXudong
", "votes": null, "replies": [] }, { "id": 69281, "author_name": "xudongcao", "author_url": "", "post_date": "04/01/2015 08:24:08", "content": "[quote=old-ufo;69209]
\nGreat, thanks!
\nP.S. Do you have results for each of nets from table 3? Also it is interesting, why you jump from 48x48 to 112x112.
\n[/quote]
\n\nDear old-ufo,
\nYour name is very cool :)
\nBack to your question, I start training my model on 48x48 image with much less aggressive data augmentation. I quickly found that using larger image size and more aggressive data augmentation will greatly improve the performance. Therefore I trained more models on 96x96, 112x112, 128x128, 144x144 and 192x192 image size using various filter size i.e. 2x2, 3x3, 4x4, 5x5. In the end of this competition, I simply manually select several of them for model combination. In general, the models using 3x3, 4x4, 5x5 filter size performs better than the models using 2x2 filter size. But I found model with 2x2 filter size is complementary to the final ensemble.
\n\nThe model on 48x48 image using 5x5 filter size: 0.631 log loss score on the public leader board
\nThe model on 112x112 image using 2x2 filter size: 0.621 log loss score on the public leader board
", "votes": null, "replies": [] }, { "id": 69284, "author_name": "oldufo", "author_url": "", "post_date": "04/01/2015 08:34:59", "content": "Thanks for clarification. Looks like you are only person who experimented with 2x2 filters except Poisson Process team.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "69182": "", "69196": "", "69209": "", "69227": "", "69256": "", "69274": "", "69276": "", "69281": "", "69284": "" }, "source": "meta" }