{
  "id": 12555,
  "title": "Hitting a wall with CNN",
  "url": "/competitions/datasciencebowl/discussion/12555",
  "author_name": "",
  "post_date": "2015-02-18T16:11:49.047Z",
  "votes": 1,
  "comment_count": 33,
  "views": 11661,
  "content": "<p>Like many others, my approach so far is&nbsp;based on convolutional neural networks.&nbsp;</p>\n<p>I've been experimenting with a lot of different things in terms of architecture, data augmentation, preprocessing, and training methods, but in the end most of my attempts end up right around 0.90 LL and 71-74% accuracy (on a 10% validation set).&nbsp;</p>\n<p>I'm wondering if there is much&nbsp;more&nbsp;to be gained from these kinds&nbsp;of optimizations, or if getting further requires different&nbsp;techniques.</p>\n<p>Would anyone be willing to comment on whether they've trained individual CNNs&nbsp;that perform substantially better than this?</p>",
  "messages": [
    {
      "id": "64507",
      "postDate": "02/18/2015 16:11:49",
      "content": "<p>Like many others, my approach so far is&nbsp;based on convolutional neural networks.&nbsp;</p>\n<p>I've been experimenting with a lot of different things in terms of architecture, data augmentation, preprocessing, and training methods, but in the end most of my attempts end up right around 0.90 LL and 71-74% accuracy (on a 10% validation set).&nbsp;</p>\n<p>I'm wondering if there is much&nbsp;more&nbsp;to be gained from these kinds&nbsp;of optimizations, or if getting further requires different&nbsp;techniques.</p>\n<p>Would anyone be willing to comment on whether they've trained individual CNNs&nbsp;that perform substantially better than this?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64511",
      "postDate": "02/18/2015 16:35:07",
      "content": "<p>I think, single CNN may even won this challenge. At least, it is possible to train single CNN which is&nbsp;substantially better than 0.90.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64518",
      "postDate": "02/18/2015 17:08:07",
      "content": "<p>Thanks, that is good to know.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64541",
      "postDate": "02/19/2015 03:13:54",
      "content": "<p>I've been able to get about 0.85 with a single CNN. By combining models (CNN and RF or multiple&nbsp;CNNs), I've been able to get that score down to about 0.79. It's certainly possible that a singe CNN will win, but it rarely hurts to do some averaging.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64563",
      "postDate": "02/19/2015 08:06:27",
      "content": "<p>Are you using GPU or CPU?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64565",
      "postDate": "02/19/2015 09:41:28",
      "content": "<p>GPU for sure :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64569",
      "postDate": "02/19/2015 11:21:43",
      "content": "<p>Is there any hope to reach good results without a GPU? :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64572",
      "postDate": "02/19/2015 11:43:08",
      "content": "<p>Theoretically, if you have guessed right architecture for classifier from first-second guess, then why not. Or with some non-CNN approach, may be.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64583",
      "postDate": "02/19/2015 17:26:15",
      "content": "<p>Like others I'm in a place where I can get to .8-ish with a CNN using 48x48 images and 90, 180 and 270 flips. I'm kind of a loss as far as where I'd see better return on time invested&nbsp;at this point. For this problem, should I spend time improving my data augmentation step, work with larger images, or keep tweaking the CNN architecture? I'm thinking data augmentation, mostly because CNN architectures are still dark magic for me... :-)<br>Any recommendation?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64587",
      "postDate": "02/19/2015 18:59:39",
      "content": "<p>[quote=Giulio;64583]</p>\n<p>Like others I'm in a place where I can get to .8-ish with a CNN using 48x48 images and 90, 180 and 270 flips. I'm kind of a loss as far as where I'd see better return on time invested&nbsp;at this point. For this problem, should I spend time improving my data augmentation step, work with larger images, or keep tweaking the CNN architecture? I'm thinking data augmentation, mostly because CNN architectures are still dark magic for me... :-)<br>Any recommendation?</p>\n<p>[/quote]</p>\n<p>That's interesting because I have 96x96 images, 20 degree rotations plus scaling and translation, and I only get to 0.81. I must be doing something wrong.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64602",
      "postDate": "02/19/2015 20:19:37",
      "content": "<p>@Giulio and @James. I basically have the exact same setup as @Giulio and cannot get past 0.8 for a single CNN. I have tried playing with translations, shearing, more rotations, and they always give much worse performance. Maybe my batch size is to small (or big) but I cannot manage to get any other data augmentation besides 90,180,270 degree flips to help.</p>\n\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64605",
      "postDate": "02/19/2015 20:26:44",
      "content": "<p>[quote=mike1886;64602]</p>\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>\n<p>[/quote]</p>\n<p>No, nothing in particular. The first thing I did was add the 90/180/270 degree rotations on 48x48 images using the cxxnet example. That got me, I think , to 1.06. Subsequent improvements came from adding translations and making the resized image bigger, and deepening the architecture so it would fit the additional examples.</p>\n<p>If you're saying I can get 0.80 with 90/180/270 rotations on 48x48, then maybe I need a new hobby.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64608",
      "postDate": "02/19/2015 20:28:17",
      "content": "<p>@James that is exactly what I have with a pretty deep architecture though.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64614",
      "postDate": "02/20/2015 01:05:39",
      "content": "<p>Same here &nbsp;I tried with much larger architecture with 128 channels in the first layer and 384 512 etc in other convolution layers and 8000 in hidden layer. That didn't help then I tried to increase the hidden layer to 12576 channels, that didn't help too. &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64625",
      "postDate": "02/20/2015 04:21:39",
      "content": "<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>upto 490 iterations. maybe instead of random flips, aligning images</p>\n<p>along long &amp; short axis nd centering can help however yet to have time to code this.</p>\n<p>[quote=James King;64605]</p>\n<p>[quote=mike1886;64602]</p>\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>\n<p>[/quote]</p>\n<p>No, nothing in particular. The first thing I did was add the 90/180/270 degree rotations on 48x48 images using the cxxnet example. That got me, I think , to 1.06. Subsequent improvements came from adding translations and making the resized image bigger, and deepening the architecture so it would fit the additional examples.</p>\n<p>If you're saying I can get 0.80 with 90/180/270 rotations on 48x48, then maybe I need a new hobby.</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64626",
      "postDate": "02/20/2015 04:29:41",
      "content": "<p>I incorporated something like that into my RF-based code. I aligned the long axis of the largest contiguous region along a specified direction. This combined with random horizontal and vertical mirroring was more effective than any rotation scheme I tried. The results were not as promising when I attempted a similar scheme with my CNN.</p>\n<p><br>[quote=Ertuka;64625]</p>\n<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>upto 490 iterations. maybe instead of random flips, aligning images</p>\n<p>along long &amp; short axis nd centering can help however yet to have time to code this.</p>\n<p>[quote=James King;64605]</p>\n<p>[quote=mike1886;64602]</p>\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>\n<p>[/quote]</p>\n<p>No, nothing in particular. The first thing I did was add the 90/180/270 degree rotations on 48x48 images using the cxxnet example. That got me, I think , to 1.06. Subsequent improvements came from adding translations and making the resized image bigger, and deepening the architecture so it would fit the additional examples.</p>\n<p>If you're saying I can get 0.80 with 90/180/270 rotations on 48x48, then maybe I need a new hobby.</p>\n<p>[/quote]</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64641",
      "postDate": "02/20/2015 12:21:39",
      "content": "<p>[quote=Ertuka;64625]</p>\n<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>[/quote]</p>\n\n<p>I tried again on 48x48 with three rotations last night. I added two additional convolution/relu pairs&nbsp;and let it train longer at it went to 0.95. So I'll try some additional parameter tweaks and see if I can get it to 0.8.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64647",
      "postDate": "02/20/2015 13:34:26",
      "content": "<p>I haven't been competing intensely in this competition yet, but my idea for improving over pure image processing is to try an approach to make better use of the metadata. &nbsp;</p>\n<p>This is a classification problem, but the classes are not truly independent. &nbsp;Each class is part of the taxonomic 'tree', and some classes are more closely related to each other than others. &nbsp;I wonder whether a tree-based approach to the classification is a good answer to this one - try to partition the test data progressively according to taxonomy, rather than imputing all the classes at the same time. &nbsp;</p>\n<p>Maybe a CNN can already capture these relationships between classes directly from the training data in any case, without knowledge of the tree, I don't know enough about CNNs to answer that.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64678",
      "postDate": "02/20/2015 23:24:57",
      "content": "<p>agreed</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64681",
      "postDate": "02/21/2015 00:39:43",
      "content": "<p>What library u use for Convolution neural network.</p>\n<p>I tried to run a simple test program using theano but didn't get any luck</p>\n<p>I will appreciate any help</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64691",
      "postDate": "02/21/2015 03:13:16",
      "content": "<p>I'm competing only to learn&nbsp;into the whole topic of machine learning and image recognition, no real chance for me to win anything else than knowledge here.</p>\n<p>I managed to get Theano working with GPU both on Windows and Linux (Amazon EC2), but no luck figuring out the right CNN architecture yet.</p>\n<p>Also tried classification&nbsp;at multiple levels along the&nbsp;taxonomic tree, but only with some well-chosen non-CNN image features. Zernike has some value, but only for certain classes. Using ORB descriptors does not seem to work, the images just vary too much (no real keypoints). I haven't tried HOG yet, but I see low chance that it would work.&nbsp;</p>\n<p>Complicated&nbsp;data augmentation techniques seems to improve only very little on the results, but they are essential with classes having only a few examples. (Really, they have only this few number of samples for those?)</p>\n<p>Still no luck with getting reasonable results. Everything works fine as long as I have only a few classes, but when I try to extend the approach to all 121 classes it always&nbsp;ends up with high LL. My&nbsp;submission today is just a little bit&nbsp;better than the&nbsp;Uniform Probability Benchmark, but at least it is. As a newcomer I'm happy to have this small result.</p>\n<p>Using Caffe for CNN might be less fragile than Theano and can also work with cuDNN (slightly faster while running on GPU).&nbsp;Current development Theano has problems with cuDNN which I could not fix.</p>\n<p>Nice to see the great community here. Keep up the good work and learn a lot!</p>\n<p>Any luck with non-CNN image features?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64698",
      "postDate": "02/21/2015 08:17:57",
      "content": "<p>@BeYourself,</p>\n<p>It is the first time also for me for images classification problem.</p>\n<p>Since it seems that the computing power is an issue here, I tried to do features extraction for the classification.</p>\n<p>I tried as you ORB, HOG and Haralick ones and still no data augmentation.</p>\n<p>The best results I obtained was with&nbsp; this pipeline:</p>\n<p>pre-filtering images in different way (edges filter or binary threshold)</p>\n<p>resize images to 64x64,</p>\n<p>Apply RandomizedPCA</p>\n<p>SVM with rbf kernel.</p>\n<p>I obtatined the same MCL~1.9 even using 13 Haralick features +SVM. Until now, that was my non CNN solution.</p>\n<p>I have'.t a GPU, so I'm trying to run Theano and CNN on my pc, with very poor results.</p>\n<p>Maybe I'll try GPU on amazon.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64699",
      "postDate": "02/21/2015 08:21:17",
      "content": "<p>@James Thanks for the tip, some reason I still cannot able to go under 1.06, must be missing something.</p>\n<p>[quote=James King;64641]</p>\n<p>[quote=Ertuka;64625]</p>\n<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>[/quote]</p>\n<p>I tried again on 48x48 with three rotations last night. I added two additional convolution/relu pairs&nbsp;and let it train longer at it went to 0.95. So I'll try some additional parameter tweaks and see if I can get it to 0.8.&nbsp;</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64702",
      "postDate": "02/21/2015 10:09:38",
      "content": "<p>[quote=Data-Lover;64681]</p>\n<p>What library u use for Convolution neural network.</p>\n<p>I tried to run a simple test program using theano but didn't get any luck</p>\n<p>I will appreciate any help</p>\n<p>[/quote]</p>\n<p>My first version was, basically, copy-paste of this code: http://deeplearning.net/tutorial/lenet.html#lenet with all activations replaced by relu's and it reached 1.56 without ata augmentation. Maybe you could try that too.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64750",
      "postDate": "02/23/2015 00:54:03",
      "content": "<p>Try <a href=\"http://caffe.berkeleyvision.org/\" target=\"_blank\">Caffe</a>, that's much less fragile than Theano and runs 2-5x faster, especially on a GPU. Try customizing&nbsp;the ImageNet example. I've just got it up and running in a single day. For image preprocessing you can still use Python, certainly. Features extracted can also be fed into Caffe (in theory), but I haven't tried that yet. Caffe is a general purpose DNN engine, not just for images, so I think it does worth to learn it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64758",
      "postDate": "02/23/2015 10:21:32",
      "content": "<p>[quote=BeYourself;64750]</p>\n<p>Try <a href=\"http://caffe.berkeleyvision.org/\" target=\"_blank\">Caffe</a>, that's much less fragile than Theano</p>\n<p>[/quote]</p>\n<p>I don't think I have anything against Caffe, but I'm wondering why you consider it &quot;less fragile&quot; than Theano? Theano is a much more general library, I feel that I'm learning more about what a CNN actually does by using Theano (and having to write a lot myself) than by using a library with off-the-shelf layers and routines such as Caffe. That said, if you have good arguments why the latter works better, I might be tempted to switch :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64798",
      "postDate": "02/24/2015 03:28:44",
      "content": "<p>You can also try cxxnet, here's their <a href=\"https://github.com/antinucleon/cxxnet/tree/master/example/kaggle_bowl\">Github page</a>&nbsp;to this competition. They also started a thread in these forums, that talk more about debugging familiar issues when installing.</p>\n<p>I have not yet tried caffe, but the CNN implementation is probably similar in both cxxnet and caffe (in which case it's much better to start with cxxnet, as they have given a step-by-step guide to getting started).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67088",
      "postDate": "03/18/2015 21:22:43",
      "content": "<p>I managed to try <a href=\"https://github.com/Theano/Theano\">Theano</a>&nbsp;(with <a href=\"https://github.com/benanne/Lasagne\">lasagne</a>),&nbsp;<a href=\"http://caffe.berkeleyvision.org/\">Caffe</a>&nbsp;and <a href=\"https://github.com/antinucleon/cxxnet\">cxxnet</a>&nbsp;before I finally had to give up due to lack of time. All&nbsp;of them worked well, but I finally failed&nbsp;to find&nbsp;the right mix of network architecture and image augmentation.</p>\n<p>The code base of cxxnet is indeed much smaller and simpler, but it comes at a price. For example it lacks scripting integration. Caffe has full support for high level scripting in Python, which makes tricks like <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/net_surgery.ipynb\">network surgery</a>&nbsp;possible. This is a great way of evolving your network architecture without completely losing your past&nbsp;training effort. It also allows for <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/filter_visualization.ipynb\">visualizing the internals of your neural&nbsp;network</a>, which makes so much easier to debug&nbsp;and understand it.</p>\n<p>The cxxnet revision&nbsp;I checked out had a non-obvious bug in mean image calculation, which pushed me back by at least one week by&nbsp;invalidating&nbsp;all of my experiments made before the discovery of the problem. All&nbsp;I got were completely&nbsp;misleading results.&nbsp;The bug was found by accident&nbsp;when I tried&nbsp;to generate the mean images myself in the&nbsp;image&nbsp;augmentation code to save some processing time.</p>\n<p>One&nbsp;positive side of cxxnet is its handling of GPU memory. It is much harder to run out of GPU memory with cxxnet, but it is always a problem with Caffe and Theano. It might have something to do with the memory management of the tensor library used, bit I haven't figured out the exact details. Another interesting detail is that it is often more efficient to run two cxxnet processes&nbsp;at the same time, in case you need to train multiple networks at the same time (for example specialists for each class). They will fit into the GPU memory nicely.</p>\n<p>Theano is a nice tool for experimentation with&nbsp;arbitrary&nbsp;algorithms, but Caffe and cxxnet are&nbsp;easier to use and more reliable tools in case you &quot;just&quot; want to train a CNN for a specific task. Theano can easily be the best tool for&nbsp;research, however. Setting up Theano on Windows to work with the GPU is very tricky, I suggest you to go straight to Linux instead. There are Amazon EC2 AMIs with Theano set up, those are good starting points. There is&nbsp;also at least one <a href=\"http://markus.com/install-theano-on-aws/\">HowTo</a>&nbsp;on this.</p>\n<p>Bottom line: Pick the right tool for the task, start simple, always&nbsp;double check all intermediate data/results and&nbsp;refine your&nbsp;solution&nbsp;gradually.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67130",
      "postDate": "03/19/2015 01:34:30",
      "content": "<p>Caffe&nbsp;</p>\n<p>color=255-color</p>\n<p>256x256 images &nbsp; 0/90/180/270 + all flip &nbsp; , &nbsp;one simple model gives public loss 0.88 &nbsp;&nbsp;</p>\n<p>256x256 images &nbsp; rotate 30 each time + all flip , average over 25 models final &nbsp;loss is 0.749</p>\n<p>Not sure how they manage to get 0.6 or 0.5 yet.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67180",
      "postDate": "03/19/2015 08:54:15",
      "content": "<p>@Steven Du, you have too little augmentation and too large images. One single vgg-spp-net can give at least 0.65 with augmentation described in winner`s solution.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67193",
      "postDate": "03/19/2015 09:51:54",
      "content": "<p>@old-ufo As I understand, you use vgg-spp-net. Could you tell me if you learn images using one size (ex. 96x96) or use multiple size as input (like in VGG paper)? The second option is rather hard to do in Caffe, need to read new database for ex. every epoch.</p>\n<p>And second question, do you use original image size when you get predictions?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67194",
      "postDate": "03/19/2015 09:59:46",
      "content": "<p>@Bartek, I use a real-time&nbsp;augmentation similar to&nbsp;this one&nbsp;https://github.com/senecaur/caffe-rta&nbsp;</p>\n<p>It is easy, when you use &quot;encoded=true&quot; flag when creating dataset and storing just original images in db.&nbsp;</p>\n<p>For prediction I used something very similar to TTA - little scale jittering, 90 degrees rotation and mirroring. 50-100 augmentations per net, which gave me ~0.01 compared to single prediction.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67241",
      "postDate": "03/19/2015 16:53:28",
      "content": "<p>Hi BeYourself. You might want to check the V2 version of cxxnet here&nbsp;https://github.com/dmlc/cxxnet &nbsp;. Which comes at more features, including python scripting:)</p>\n<p>I think&nbsp;the bug in mean image generation you mentioned was fixed. &nbsp;But it would be great if you could fire an issue if you have any comments on that</p>\n<p>Thanks for trying out!</p>\n<p>[quote=BeYourself;67088]</p>\n<p>I managed to try <a href=\"https://github.com/Theano/Theano\">Theano</a>&nbsp;(with <a href=\"https://github.com/benanne/Lasagne\">lasagne</a>),&nbsp;<a href=\"http://caffe.berkeleyvision.org/\">Caffe</a>&nbsp;and <a href=\"https://github.com/antinucleon/cxxnet\">cxxnet</a>&nbsp;before I finally had to give up due to lack of time. All&nbsp;of them worked well, but I finally failed&nbsp;to find&nbsp;the right mix of network architecture and image augmentation.</p>\n<p>The code base of cxxnet is indeed much smaller and simpler, but it comes at a price. For example it lacks scripting integration. Caffe has full support for high level scripting in Python, which makes tricks like <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/net_surgery.ipynb\">network surgery</a>&nbsp;possible. This is a great way of evolving your network architecture without completely losing your past&nbsp;training effort. It also allows for <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/filter_visualization.ipynb\">visualizing the internals of your neural&nbsp;network</a>, which makes so much easier to debug&nbsp;and understand it.</p>\n<p>The cxxnet revision&nbsp;I checked out had a non-obvious bug in mean image calculation, which pushed me back by at least one week by&nbsp;invalidating&nbsp;all of my experiments made before the discovery of the problem. All&nbsp;I got were completely&nbsp;misleading results.&nbsp;The bug was found by accident&nbsp;when I tried&nbsp;to generate the mean images myself in the&nbsp;image&nbsp;augmentation code to save some processing time.</p>\n<p>One&nbsp;positive side of cxxnet is its handling of GPU memory. It is much harder to run out of GPU memory with cxxnet, but it is always a problem with Caffe and Theano. It might have something to do with the memory management of the tensor library used, bit I haven't figured out the exact details. Another interesting detail is that it is often more efficient to run two cxxnet processes&nbsp;at the same time, in case you need to train multiple networks at the same time (for example specialists for each class). They will fit into the GPU memory nicely.</p>\n<p>Theano is a nice tool for experimentation with&nbsp;arbitrary&nbsp;algorithms, but Caffe and cxxnet are&nbsp;easier to use and more reliable tools in case you &quot;just&quot; want to train a CNN for a specific task. Theano can easily be the best tool for&nbsp;research, however. Setting up Theano on Windows to work with the GPU is very tricky, I suggest you to go straight to Linux instead. There are Amazon EC2 AMIs with Theano set up, those are good starting points. There is&nbsp;also at least one <a href=\"http://markus.com/install-theano-on-aws/\">HowTo</a>&nbsp;on this.</p>\n<p>Bottom line: Pick the right tool for the task, start simple, always&nbsp;double check all intermediate data/results and&nbsp;refine your&nbsp;solution&nbsp;gradually.</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67274",
      "postDate": "03/19/2015 19:33:03",
      "content": "<p>Thank you for the lots of very useful extra information on your approaches/solutions.</p>\n<p>I tried various image sizes from 48x48 up to 192x192. 256x256 could not really fit into my GPU with a reasonable batch size (yeah, I have an older one). I tried scaling with both breaking the aspect ratio and padding (keeping aspect ratio). Also tried image crops with no scaling, including center crop and side&nbsp;crops,&nbsp;random&nbsp;crops and various optimizations to select the crops with the most information in them. Also tried normalizing the images regarding to weight center and&nbsp;orientation. Then applied image augmentation with random scaling, rotation, noise, distortions, etc. They helped a bit, but still, no competitive score.</p>\n<p>None of these really worked, possibly because of the lack of a proper real-time image augmentation framework. (So I could not train with enough variations.) I also feel that applying all of these ideas was&nbsp;overkill and could easily lead to&nbsp;overcomplication. Also, any minor bugs can easily cause completely misleading results. More code, more bugs...&nbsp;People with much simpler solutions got much better results.</p>\n<p>It was nice to try out all the ideas I could find/have, but that's not the direct way to get a competitive score. Keep it simple s... :) All in all, it was a great learning experience.</p>\n<p>It is time for me to read/try&nbsp;all the publicly available solutions and learn from them.</p>\n<p>Please feel free to continue sharing the details of your solutions here, so we can all learn from them. Thank you in advance.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 64511,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "02/18/2015 16:35:07",
      "content": "<p>I think, single CNN may even won this challenge. At least, it is possible to train single CNN which is&nbsp;substantially better than 0.90.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64518,
      "author_name": "emolson",
      "author_url": "",
      "post_date": "02/18/2015 17:08:07",
      "content": "<p>Thanks, that is good to know.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64541,
      "author_name": "thekannman",
      "author_url": "",
      "post_date": "02/19/2015 03:13:54",
      "content": "<p>I've been able to get about 0.85 with a single CNN. By combining models (CNN and RF or multiple&nbsp;CNNs), I've been able to get that score down to about 0.79. It's certainly possible that a singe CNN will win, but it rarely hurts to do some averaging.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64563,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "02/19/2015 08:06:27",
      "content": "<p>Are you using GPU or CPU?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64565,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "02/19/2015 09:41:28",
      "content": "<p>GPU for sure :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64569,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "02/19/2015 11:21:43",
      "content": "<p>Is there any hope to reach good results without a GPU? :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64572,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "02/19/2015 11:43:08",
      "content": "<p>Theoretically, if you have guessed right architecture for classifier from first-second guess, then why not. Or with some non-CNN approach, may be.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64583,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "02/19/2015 17:26:15",
      "content": "<p>Like others I'm in a place where I can get to .8-ish with a CNN using 48x48 images and 90, 180 and 270 flips. I'm kind of a loss as far as where I'd see better return on time invested&nbsp;at this point. For this problem, should I spend time improving my data augmentation step, work with larger images, or keep tweaking the CNN architecture? I'm thinking data augmentation, mostly because CNN architectures are still dark magic for me... :-)<br>Any recommendation?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64587,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "02/19/2015 18:59:39",
      "content": "<p>[quote=Giulio;64583]</p>\n<p>Like others I'm in a place where I can get to .8-ish with a CNN using 48x48 images and 90, 180 and 270 flips. I'm kind of a loss as far as where I'd see better return on time invested&nbsp;at this point. For this problem, should I spend time improving my data augmentation step, work with larger images, or keep tweaking the CNN architecture? I'm thinking data augmentation, mostly because CNN architectures are still dark magic for me... :-)<br>Any recommendation?</p>\n<p>[/quote]</p>\n<p>That's interesting because I have 96x96 images, 20 degree rotations plus scaling and translation, and I only get to 0.81. I must be doing something wrong.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64602,
      "author_name": "msegala",
      "author_url": "",
      "post_date": "02/19/2015 20:19:37",
      "content": "<p>@Giulio and @James. I basically have the exact same setup as @Giulio and cannot get past 0.8 for a single CNN. I have tried playing with translations, shearing, more rotations, and they always give much worse performance. Maybe my batch size is to small (or big) but I cannot manage to get any other data augmentation besides 90,180,270 degree flips to help.</p>\n\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64605,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "02/19/2015 20:26:44",
      "content": "<p>[quote=mike1886;64602]</p>\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>\n<p>[/quote]</p>\n<p>No, nothing in particular. The first thing I did was add the 90/180/270 degree rotations on 48x48 images using the cxxnet example. That got me, I think , to 1.06. Subsequent improvements came from adding translations and making the resized image bigger, and deepening the architecture so it would fit the additional examples.</p>\n<p>If you're saying I can get 0.80 with 90/180/270 rotations on 48x48, then maybe I need a new hobby.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64608,
      "author_name": "msegala",
      "author_url": "",
      "post_date": "02/19/2015 20:28:17",
      "content": "<p>@James that is exactly what I have with a pretty deep architecture though.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64614,
      "author_name": "akilaw",
      "author_url": "",
      "post_date": "02/20/2015 01:05:39",
      "content": "<p>Same here &nbsp;I tried with much larger architecture with 128 channels in the first layer and 384 512 etc in other convolution layers and 8000 in hidden layer. That didn't help then I tried to increase the hidden layer to 12576 channels, that didn't help too. &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64625,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "02/20/2015 04:21:39",
      "content": "<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>upto 490 iterations. maybe instead of random flips, aligning images</p>\n<p>along long &amp; short axis nd centering can help however yet to have time to code this.</p>\n<p>[quote=James King;64605]</p>\n<p>[quote=mike1886;64602]</p>\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>\n<p>[/quote]</p>\n<p>No, nothing in particular. The first thing I did was add the 90/180/270 degree rotations on 48x48 images using the cxxnet example. That got me, I think , to 1.06. Subsequent improvements came from adding translations and making the resized image bigger, and deepening the architecture so it would fit the additional examples.</p>\n<p>If you're saying I can get 0.80 with 90/180/270 rotations on 48x48, then maybe I need a new hobby.</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64626,
      "author_name": "thekannman",
      "author_url": "",
      "post_date": "02/20/2015 04:29:41",
      "content": "<p>I incorporated something like that into my RF-based code. I aligned the long axis of the largest contiguous region along a specified direction. This combined with random horizontal and vertical mirroring was more effective than any rotation scheme I tried. The results were not as promising when I attempted a similar scheme with my CNN.</p>\n<p><br>[quote=Ertuka;64625]</p>\n<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>upto 490 iterations. maybe instead of random flips, aligning images</p>\n<p>along long &amp; short axis nd centering can help however yet to have time to code this.</p>\n<p>[quote=James King;64605]</p>\n<p>[quote=mike1886;64602]</p>\n<p>@James: Did you have to do anything in particular to add in more data augmentations?</p>\n<p>[/quote]</p>\n<p>No, nothing in particular. The first thing I did was add the 90/180/270 degree rotations on 48x48 images using the cxxnet example. That got me, I think , to 1.06. Subsequent improvements came from adding translations and making the resized image bigger, and deepening the architecture so it would fit the additional examples.</p>\n<p>If you're saying I can get 0.80 with 90/180/270 rotations on 48x48, then maybe I need a new hobby.</p>\n<p>[/quote]</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64641,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "02/20/2015 12:21:39",
      "content": "<p>[quote=Ertuka;64625]</p>\n<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>[/quote]</p>\n\n<p>I tried again on 48x48 with three rotations last night. I added two additional convolution/relu pairs&nbsp;and let it train longer at it went to 0.95. So I'll try some additional parameter tweaks and see if I can get it to 0.8.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64647,
      "author_name": "jaymoore",
      "author_url": "",
      "post_date": "02/20/2015 13:34:26",
      "content": "<p>I haven't been competing intensely in this competition yet, but my idea for improving over pure image processing is to try an approach to make better use of the metadata. &nbsp;</p>\n<p>This is a classification problem, but the classes are not truly independent. &nbsp;Each class is part of the taxonomic 'tree', and some classes are more closely related to each other than others. &nbsp;I wonder whether a tree-based approach to the classification is a good answer to this one - try to partition the test data progressively according to taxonomy, rather than imputing all the classes at the same time. &nbsp;</p>\n<p>Maybe a CNN can already capture these relationships between classes directly from the training data in any case, without knowledge of the tree, I don't know enough about CNNs to answer that.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64678,
      "author_name": "iosentinel",
      "author_url": "",
      "post_date": "02/20/2015 23:24:57",
      "content": "<p>agreed</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64681,
      "author_name": "",
      "author_url": "",
      "post_date": "02/21/2015 00:39:43",
      "content": "<p>What library u use for Convolution neural network.</p>\n<p>I tried to run a simple test program using theano but didn't get any luck</p>\n<p>I will appreciate any help</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64691,
      "author_name": "fviktor",
      "author_url": "",
      "post_date": "02/21/2015 03:13:16",
      "content": "<p>I'm competing only to learn&nbsp;into the whole topic of machine learning and image recognition, no real chance for me to win anything else than knowledge here.</p>\n<p>I managed to get Theano working with GPU both on Windows and Linux (Amazon EC2), but no luck figuring out the right CNN architecture yet.</p>\n<p>Also tried classification&nbsp;at multiple levels along the&nbsp;taxonomic tree, but only with some well-chosen non-CNN image features. Zernike has some value, but only for certain classes. Using ORB descriptors does not seem to work, the images just vary too much (no real keypoints). I haven't tried HOG yet, but I see low chance that it would work.&nbsp;</p>\n<p>Complicated&nbsp;data augmentation techniques seems to improve only very little on the results, but they are essential with classes having only a few examples. (Really, they have only this few number of samples for those?)</p>\n<p>Still no luck with getting reasonable results. Everything works fine as long as I have only a few classes, but when I try to extend the approach to all 121 classes it always&nbsp;ends up with high LL. My&nbsp;submission today is just a little bit&nbsp;better than the&nbsp;Uniform Probability Benchmark, but at least it is. As a newcomer I'm happy to have this small result.</p>\n<p>Using Caffe for CNN might be less fragile than Theano and can also work with cuDNN (slightly faster while running on GPU).&nbsp;Current development Theano has problems with cuDNN which I could not fix.</p>\n<p>Nice to see the great community here. Keep up the good work and learn a lot!</p>\n<p>Any luck with non-CNN image features?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64698,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "02/21/2015 08:17:57",
      "content": "<p>@BeYourself,</p>\n<p>It is the first time also for me for images classification problem.</p>\n<p>Since it seems that the computing power is an issue here, I tried to do features extraction for the classification.</p>\n<p>I tried as you ORB, HOG and Haralick ones and still no data augmentation.</p>\n<p>The best results I obtained was with&nbsp; this pipeline:</p>\n<p>pre-filtering images in different way (edges filter or binary threshold)</p>\n<p>resize images to 64x64,</p>\n<p>Apply RandomizedPCA</p>\n<p>SVM with rbf kernel.</p>\n<p>I obtatined the same MCL~1.9 even using 13 Haralick features +SVM. Until now, that was my non CNN solution.</p>\n<p>I have'.t a GPU, so I'm trying to run Theano and CNN on my pc, with very poor results.</p>\n<p>Maybe I'll try GPU on amazon.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64699,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "02/21/2015 08:21:17",
      "content": "<p>@James Thanks for the tip, some reason I still cannot able to go under 1.06, must be missing something.</p>\n<p>[quote=James King;64641]</p>\n<p>[quote=Ertuka;64625]</p>\n<p>@James,</p>\n<p>My results are close to yours: 48x48 images with below flips I also reached 1.09s</p>\n<p>[/quote]</p>\n<p>I tried again on 48x48 with three rotations last night. I added two additional convolution/relu pairs&nbsp;and let it train longer at it went to 0.95. So I'll try some additional parameter tweaks and see if I can get it to 0.8.&nbsp;</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64702,
      "author_name": "sovcharenko",
      "author_url": "",
      "post_date": "02/21/2015 10:09:38",
      "content": "<p>[quote=Data-Lover;64681]</p>\n<p>What library u use for Convolution neural network.</p>\n<p>I tried to run a simple test program using theano but didn't get any luck</p>\n<p>I will appreciate any help</p>\n<p>[/quote]</p>\n<p>My first version was, basically, copy-paste of this code: http://deeplearning.net/tutorial/lenet.html#lenet with all activations replaced by relu's and it reached 1.56 without ata augmentation. Maybe you could try that too.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64750,
      "author_name": "fviktor",
      "author_url": "",
      "post_date": "02/23/2015 00:54:03",
      "content": "<p>Try <a href=\"http://caffe.berkeleyvision.org/\" target=\"_blank\">Caffe</a>, that's much less fragile than Theano and runs 2-5x faster, especially on a GPU. Try customizing&nbsp;the ImageNet example. I've just got it up and running in a single day. For image preprocessing you can still use Python, certainly. Features extracted can also be fed into Caffe (in theory), but I haven't tried that yet. Caffe is a general purpose DNN engine, not just for images, so I think it does worth to learn it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64758,
      "author_name": "eelkespaak",
      "author_url": "",
      "post_date": "02/23/2015 10:21:32",
      "content": "<p>[quote=BeYourself;64750]</p>\n<p>Try <a href=\"http://caffe.berkeleyvision.org/\" target=\"_blank\">Caffe</a>, that's much less fragile than Theano</p>\n<p>[/quote]</p>\n<p>I don't think I have anything against Caffe, but I'm wondering why you consider it &quot;less fragile&quot; than Theano? Theano is a much more general library, I feel that I'm learning more about what a CNN actually does by using Theano (and having to write a lot myself) than by using a library with off-the-shelf layers and routines such as Caffe. That said, if you have good arguments why the latter works better, I might be tempted to switch :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64798,
      "author_name": "seyvu1",
      "author_url": "",
      "post_date": "02/24/2015 03:28:44",
      "content": "<p>You can also try cxxnet, here's their <a href=\"https://github.com/antinucleon/cxxnet/tree/master/example/kaggle_bowl\">Github page</a>&nbsp;to this competition. They also started a thread in these forums, that talk more about debugging familiar issues when installing.</p>\n<p>I have not yet tried caffe, but the CNN implementation is probably similar in both cxxnet and caffe (in which case it's much better to start with cxxnet, as they have given a step-by-step guide to getting started).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67088,
      "author_name": "fviktor",
      "author_url": "",
      "post_date": "03/18/2015 21:22:43",
      "content": "<p>I managed to try <a href=\"https://github.com/Theano/Theano\">Theano</a>&nbsp;(with <a href=\"https://github.com/benanne/Lasagne\">lasagne</a>),&nbsp;<a href=\"http://caffe.berkeleyvision.org/\">Caffe</a>&nbsp;and <a href=\"https://github.com/antinucleon/cxxnet\">cxxnet</a>&nbsp;before I finally had to give up due to lack of time. All&nbsp;of them worked well, but I finally failed&nbsp;to find&nbsp;the right mix of network architecture and image augmentation.</p>\n<p>The code base of cxxnet is indeed much smaller and simpler, but it comes at a price. For example it lacks scripting integration. Caffe has full support for high level scripting in Python, which makes tricks like <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/net_surgery.ipynb\">network surgery</a>&nbsp;possible. This is a great way of evolving your network architecture without completely losing your past&nbsp;training effort. It also allows for <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/filter_visualization.ipynb\">visualizing the internals of your neural&nbsp;network</a>, which makes so much easier to debug&nbsp;and understand it.</p>\n<p>The cxxnet revision&nbsp;I checked out had a non-obvious bug in mean image calculation, which pushed me back by at least one week by&nbsp;invalidating&nbsp;all of my experiments made before the discovery of the problem. All&nbsp;I got were completely&nbsp;misleading results.&nbsp;The bug was found by accident&nbsp;when I tried&nbsp;to generate the mean images myself in the&nbsp;image&nbsp;augmentation code to save some processing time.</p>\n<p>One&nbsp;positive side of cxxnet is its handling of GPU memory. It is much harder to run out of GPU memory with cxxnet, but it is always a problem with Caffe and Theano. It might have something to do with the memory management of the tensor library used, bit I haven't figured out the exact details. Another interesting detail is that it is often more efficient to run two cxxnet processes&nbsp;at the same time, in case you need to train multiple networks at the same time (for example specialists for each class). They will fit into the GPU memory nicely.</p>\n<p>Theano is a nice tool for experimentation with&nbsp;arbitrary&nbsp;algorithms, but Caffe and cxxnet are&nbsp;easier to use and more reliable tools in case you &quot;just&quot; want to train a CNN for a specific task. Theano can easily be the best tool for&nbsp;research, however. Setting up Theano on Windows to work with the GPU is very tricky, I suggest you to go straight to Linux instead. There are Amazon EC2 AMIs with Theano set up, those are good starting points. There is&nbsp;also at least one <a href=\"http://markus.com/install-theano-on-aws/\">HowTo</a>&nbsp;on this.</p>\n<p>Bottom line: Pick the right tool for the task, start simple, always&nbsp;double check all intermediate data/results and&nbsp;refine your&nbsp;solution&nbsp;gradually.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67130,
      "author_name": "stevendu",
      "author_url": "",
      "post_date": "03/19/2015 01:34:30",
      "content": "<p>Caffe&nbsp;</p>\n<p>color=255-color</p>\n<p>256x256 images &nbsp; 0/90/180/270 + all flip &nbsp; , &nbsp;one simple model gives public loss 0.88 &nbsp;&nbsp;</p>\n<p>256x256 images &nbsp; rotate 30 each time + all flip , average over 25 models final &nbsp;loss is 0.749</p>\n<p>Not sure how they manage to get 0.6 or 0.5 yet.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67180,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/19/2015 08:54:15",
      "content": "<p>@Steven Du, you have too little augmentation and too large images. One single vgg-spp-net can give at least 0.65 with augmentation described in winner`s solution.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67193,
      "author_name": "melgor",
      "author_url": "",
      "post_date": "03/19/2015 09:51:54",
      "content": "<p>@old-ufo As I understand, you use vgg-spp-net. Could you tell me if you learn images using one size (ex. 96x96) or use multiple size as input (like in VGG paper)? The second option is rather hard to do in Caffe, need to read new database for ex. every epoch.</p>\n<p>And second question, do you use original image size when you get predictions?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67194,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/19/2015 09:59:46",
      "content": "<p>@Bartek, I use a real-time&nbsp;augmentation similar to&nbsp;this one&nbsp;https://github.com/senecaur/caffe-rta&nbsp;</p>\n<p>It is easy, when you use &quot;encoded=true&quot; flag when creating dataset and storing just original images in db.&nbsp;</p>\n<p>For prediction I used something very similar to TTA - little scale jittering, 90 degrees rotation and mirroring. 50-100 augmentations per net, which gave me ~0.01 compared to single prediction.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67241,
      "author_name": "tqchen",
      "author_url": "",
      "post_date": "03/19/2015 16:53:28",
      "content": "<p>Hi BeYourself. You might want to check the V2 version of cxxnet here&nbsp;https://github.com/dmlc/cxxnet &nbsp;. Which comes at more features, including python scripting:)</p>\n<p>I think&nbsp;the bug in mean image generation you mentioned was fixed. &nbsp;But it would be great if you could fire an issue if you have any comments on that</p>\n<p>Thanks for trying out!</p>\n<p>[quote=BeYourself;67088]</p>\n<p>I managed to try <a href=\"https://github.com/Theano/Theano\">Theano</a>&nbsp;(with <a href=\"https://github.com/benanne/Lasagne\">lasagne</a>),&nbsp;<a href=\"http://caffe.berkeleyvision.org/\">Caffe</a>&nbsp;and <a href=\"https://github.com/antinucleon/cxxnet\">cxxnet</a>&nbsp;before I finally had to give up due to lack of time. All&nbsp;of them worked well, but I finally failed&nbsp;to find&nbsp;the right mix of network architecture and image augmentation.</p>\n<p>The code base of cxxnet is indeed much smaller and simpler, but it comes at a price. For example it lacks scripting integration. Caffe has full support for high level scripting in Python, which makes tricks like <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/net_surgery.ipynb\">network surgery</a>&nbsp;possible. This is a great way of evolving your network architecture without completely losing your past&nbsp;training effort. It also allows for <a href=\"http://nbviewer.ipython.org/github/udibr/caffe/blob/master/examples/filter_visualization.ipynb\">visualizing the internals of your neural&nbsp;network</a>, which makes so much easier to debug&nbsp;and understand it.</p>\n<p>The cxxnet revision&nbsp;I checked out had a non-obvious bug in mean image calculation, which pushed me back by at least one week by&nbsp;invalidating&nbsp;all of my experiments made before the discovery of the problem. All&nbsp;I got were completely&nbsp;misleading results.&nbsp;The bug was found by accident&nbsp;when I tried&nbsp;to generate the mean images myself in the&nbsp;image&nbsp;augmentation code to save some processing time.</p>\n<p>One&nbsp;positive side of cxxnet is its handling of GPU memory. It is much harder to run out of GPU memory with cxxnet, but it is always a problem with Caffe and Theano. It might have something to do with the memory management of the tensor library used, bit I haven't figured out the exact details. Another interesting detail is that it is often more efficient to run two cxxnet processes&nbsp;at the same time, in case you need to train multiple networks at the same time (for example specialists for each class). They will fit into the GPU memory nicely.</p>\n<p>Theano is a nice tool for experimentation with&nbsp;arbitrary&nbsp;algorithms, but Caffe and cxxnet are&nbsp;easier to use and more reliable tools in case you &quot;just&quot; want to train a CNN for a specific task. Theano can easily be the best tool for&nbsp;research, however. Setting up Theano on Windows to work with the GPU is very tricky, I suggest you to go straight to Linux instead. There are Amazon EC2 AMIs with Theano set up, those are good starting points. There is&nbsp;also at least one <a href=\"http://markus.com/install-theano-on-aws/\">HowTo</a>&nbsp;on this.</p>\n<p>Bottom line: Pick the right tool for the task, start simple, always&nbsp;double check all intermediate data/results and&nbsp;refine your&nbsp;solution&nbsp;gradually.</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67274,
      "author_name": "fviktor",
      "author_url": "",
      "post_date": "03/19/2015 19:33:03",
      "content": "<p>Thank you for the lots of very useful extra information on your approaches/solutions.</p>\n<p>I tried various image sizes from 48x48 up to 192x192. 256x256 could not really fit into my GPU with a reasonable batch size (yeah, I have an older one). I tried scaling with both breaking the aspect ratio and padding (keeping aspect ratio). Also tried image crops with no scaling, including center crop and side&nbsp;crops,&nbsp;random&nbsp;crops and various optimizations to select the crops with the most information in them. Also tried normalizing the images regarding to weight center and&nbsp;orientation. Then applied image augmentation with random scaling, rotation, noise, distortions, etc. They helped a bit, but still, no competitive score.</p>\n<p>None of these really worked, possibly because of the lack of a proper real-time image augmentation framework. (So I could not train with enough variations.) I also feel that applying all of these ideas was&nbsp;overkill and could easily lead to&nbsp;overcomplication. Also, any minor bugs can easily cause completely misleading results. More code, more bugs...&nbsp;People with much simpler solutions got much better results.</p>\n<p>It was nice to try out all the ideas I could find/have, but that's not the direct way to get a competitive score. Keep it simple s... :) All in all, it was a great learning experience.</p>\n<p>It is time for me to read/try&nbsp;all the publicly available solutions and learn from them.</p>\n<p>Please feel free to continue sharing the details of your solutions here, so we can all learn from them. Thank you in advance.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "64507": "",
    "64511": "",
    "64518": "",
    "64541": "",
    "64563": "",
    "64565": "",
    "64569": "",
    "64572": "",
    "64583": "",
    "64587": "",
    "64602": "",
    "64605": "",
    "64608": "",
    "64614": "",
    "64625": "",
    "64626": "",
    "64641": "",
    "64647": "",
    "64678": "",
    "64681": "",
    "64691": "",
    "64698": "",
    "64699": "",
    "64702": "",
    "64750": "",
    "64758": "",
    "64798": "",
    "67088": "",
    "67130": "",
    "67180": "",
    "67193": "",
    "67194": "",
    "67241": "",
    "67274": ""
  },
  "source": "meta"
}