{ "id": 7514, "title": "99.46% with Convolutional NN (Weka)", "url": "/competitions/digit-recognizer/discussion/7514", "author_name": "Johannes Amtén", "post_date": "2014-03-25T18:39:08.560000", "votes": 13, "comment_count": 13, "views": null, "content": "

I continued building on my Weka neural network implementation from here to implement convolutional neural networks.
Source code, downloads and installation instructions here:
https://github.com/amten/NeuralNetwork

\n

With regular neural networks, I got stuck at 98.83%. With convolutional neural networks, I was able to get 99.46% without problems.

\n

I used two convolutional layers, the first with 20 feature maps, the second with 100 feature maps, both layers with 5x5 patch-size and 2x2 max-pooling.
Also, I used batch-size=1, i.e. Stochastic Gradient Descent. That seemed to work a lot better than larger batch sizes for conv nets.
It took about an hour on a modern laptop for the training to converge.

", "messages": [ { "id": 41006, "postDate": "2014-03-25T18:39:08.560Z", "content": "

I continued building on my Weka neural network implementation from here to implement convolutional neural networks.
Source code, downloads and installation instructions here:
https://github.com/amten/NeuralNetwork

\n

With regular neural networks, I got stuck at 98.83%. With convolutional neural networks, I was able to get 99.46% without problems.

\n

I used two convolutional layers, the first with 20 feature maps, the second with 100 feature maps, both layers with 5x5 patch-size and 2x2 max-pooling.
Also, I used batch-size=1, i.e. Stochastic Gradient Descent. That seemed to work a lot better than larger batch sizes for conv nets.
It took about an hour on a modern laptop for the training to converge.

", "votes": 13 }, { "id": 51563, "postDate": "2014-08-03T19:45:33.957Z", "content": "

[quote=Mike Hall;51562]

\n

I have to learn more NN sometime, somewhere. I'm not remembering fully-connected or convolutional coming up in what I've seen so far, which has been pretty much limited to multi-layer perceptron.

\n

Thanks again.

\n

[/quote]

\n

I highly recommend the video lectures from Hugo Larochelles Neural Network class:

\n

https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH

\n

In chapter 9.6 he talks about convolutional neural networks for images.

", "votes": 3 }, { "id": 51171, "postDate": "2014-07-25T08:40:08.690Z", "content": "

[quote=Tgeek;51166]

\n

amazing work, well done!!!

\n

[/quote]

\n

Thank you very much :)

", "votes": 1 }, { "id": 51166, "postDate": "2014-07-25T05:34:49.787Z", "content": "

amazing work, well done!!!

", "votes": 1 }, { "id": 100685, "postDate": "2015-12-09T17:46:24.357Z", "content": "

Thanks a lot for sharing! I've installed this on WEKA. I'd like to do some image classification (RGB images and greyscale images). But I am not sure how to manage the input for CNN. Should I arrange the images as channel by channel or RGB value by RGB value?

", "rawMarkdown": "Thanks a lot for sharing! I've installed this on WEKA. I'd like to do some image classification (RGB images and greyscale images). But I am not sure how to manage the input for CNN. Should I arrange the images as channel by channel or RGB value by RGB value? " }, { "id": 100683, "postDate": "2015-12-09T17:19:34.713Z", "content": "

Thank you, Johannes Amtén! It's so good. But I have a question, when I implement your code to recognite characters on Android is very slow. I don't know why. Can you give me some advise how to use it on android.

", "rawMarkdown": "Thank you, Johannes Amtén! It's so good. But I have a question, when I implement your code to recognite characters on Android is very slow. I don't know why. Can you give me some advise how to use it on android." }, { "id": 90437, "postDate": "2015-08-26T15:28:10.677Z", "content": "

Excellent work ! just one question, how can visualize the learnt model with your weka implementation?

", "rawMarkdown": "Excellent work ! just one question, how can visualize the learnt model with your weka implementation?" }, { "id": 60272, "postDate": "2014-12-15T20:21:14.930Z", "content": "

[quote=Hang Zhang;60253]

\n

Hi Amten,

\n

Thank you for your amazing work! I am trying to use your implementation to another topics on kaggle, 'First step in Julia'. I used the same parameter shown above, and only got 66%. Can you give me some advise on choosing parameters.

\n

Thanks!

\n

Hang

\n

[/quote]

\n

Thank you. I haven't looked at that problem. Don't know if any other parameters would be better. It seems like 66% is quite a good score on that problem though. The benchmark scores for Random Forest/KNN is just 43% / 41% and a score of 66% would put you at spot 8 on the leaderboard(!)

" }, { "id": 60253, "postDate": "2014-12-15T19:10:33.937Z", "content": "

Hi Amten,

\n

Thank you for your amazing work! I am trying to use your implementation to another topics on kaggle, 'First step in Julia'. I used the same parameter shown above, and only got 66%. Can you give me some advise on choosing parameters.

\n

Thanks!

\n

Hang

" }, { "id": 59446, "postDate": "2014-12-03T07:58:17.010Z", "content": "

[quote=Yanyanyan;59437]

\n

Hi Amten,

\n

Thanks for your implementation.

\n

I have a question, how should we set the parameter in weka when we use this package.

\n

For example, "I used two convolutional layers, the first with 20 feature maps, the second with 100 feature maps, both layers with 5x5 patch-size and 2x2 max-pooling."

\n

how to set these parameters on weka hiddenlayer input would you please make a screenshot for me? 

\n

Thanks

\n

[/quote]

\n

Sure,

\n

Hidden layers parameter: 20-5-5-2-2,100-5-5-2-2

\n

\n\n

(Also, you can just press the "more" button for an explanation of the parameters in the UI.)

" }, { "id": 59437, "postDate": "2014-12-03T04:21:25.540Z", "content": "

Hi Amten,

\n

Thanks for your implementation.

\n

I have a question, how should we set the parameter in weka when we use this package.

\n

For example, "I used two convolutional layers, the first with 20 feature maps, the second with 100 feature maps, both layers with 5x5 patch-size and 2x2 max-pooling."

\n

how to set these parameters on weka hiddenlayer input would you please make a screenshot for me? 

\n

Thanks

" }, { "id": 51562, "postDate": "2014-08-03T18:19:42.710Z", "content": "

I have to learn more NN sometime, somewhere. I'm not remembering fully-connected or convolutional coming up in what I've seen so far, which has been pretty much limited to multi-layer perceptron.

\n

Thanks again.

" }, { "id": 51553, "postDate": "2014-08-03T15:16:24.967Z", "content": "

[quote=Mike Hall;51535]

\n

Works very nicely, even 'out of the box' with default parameters on this dataset. Is that still convolutional?

\n

I had to run it command line though due to memory problems with the Weka GUI. (This on OS X - I got rid of the blas related and it ran fine with just the jars. The linux doc link you provided says the OS X veclib [I think it was] implementation is fast enough you don't need to worry about it and it seems to be.

\n

Thanks.

\n

[/quote]

\n

Cool. No, the default setting is a single, fully-connected, hidden layer with 100 Units.

" }, { "id": 51535, "postDate": "2014-08-02T23:13:08.013Z", "content": "

Works very nicely, even 'out of the box' with default parameters on this dataset. Is that still convolutional?

\n

I had to run it command line though due to memory problems with the Weka GUI. (This on OS X - I got rid of the blas related and it ran fine with just the jars. The linux doc link you provided says the OS X veclib [I think it was] implementation is fast enough you don't need to worry about it and it seems to be.

\n

Thanks.

" } ], "comments": [ { "id": 51563, "author_name": "Johannes Amtén", "author_url": "", "post_date": "2014-08-03T19:45:33.957000", "content": "

[quote=Mike Hall;51562]

\n

I have to learn more NN sometime, somewhere. I'm not remembering fully-connected or convolutional coming up in what I've seen so far, which has been pretty much limited to multi-layer perceptron.

\n

Thanks again.

\n

[/quote]

\n

I highly recommend the video lectures from Hugo Larochelles Neural Network class:

\n

https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH

\n

In chapter 9.6 he talks about convolutional neural networks for images.

", "votes": 3, "replies": [] }, { "id": 51171, "author_name": "Johannes Amtén", "author_url": "", "post_date": "2014-07-25T08:40:08.690000", "content": "

[quote=Tgeek;51166]

\n

amazing work, well done!!!

\n

[/quote]

\n

Thank you very much :)

", "votes": 1, "replies": [] }, { "id": 51166, "author_name": "Tgeek", "author_url": "", "post_date": "2014-07-25T05:34:49.787000", "content": "

amazing work, well done!!!

", "votes": 1, "replies": [] }, { "id": 100685, "author_name": "Xuefei", "author_url": "", "post_date": "2015-12-09T17:46:24.357000", "content": "

Thanks a lot for sharing! I've installed this on WEKA. I'd like to do some image classification (RGB images and greyscale images). But I am not sure how to manage the input for CNN. Should I arrange the images as channel by channel or RGB value by RGB value?

", "votes": 0, "replies": [] }, { "id": 100683, "author_name": "TranHongNghi", "author_url": "", "post_date": "2015-12-09T17:19:34.713000", "content": "

Thank you, Johannes Amtén! It's so good. But I have a question, when I implement your code to recognite characters on Android is very slow. I don't know why. Can you give me some advise how to use it on android.

", "votes": 0, "replies": [] }, { "id": 90437, "author_name": "James10", "author_url": "", "post_date": "2015-08-26T15:28:10.677000", "content": "

Excellent work ! just one question, how can visualize the learnt model with your weka implementation?

", "votes": 0, "replies": [] }, { "id": 60272, "author_name": "Johannes Amtén", "author_url": "", "post_date": "2014-12-15T20:21:14.930000", "content": "

[quote=Hang Zhang;60253]

\n

Hi Amten,

\n

Thank you for your amazing work! I am trying to use your implementation to another topics on kaggle, 'First step in Julia'. I used the same parameter shown above, and only got 66%. Can you give me some advise on choosing parameters.

\n

Thanks!

\n

Hang

\n

[/quote]

\n

Thank you. I haven't looked at that problem. Don't know if any other parameters would be better. It seems like 66% is quite a good score on that problem though. The benchmark scores for Random Forest/KNN is just 43% / 41% and a score of 66% would put you at spot 8 on the leaderboard(!)

", "votes": 0, "replies": [] }, { "id": 60253, "author_name": "Hang Zhang", "author_url": "", "post_date": "2014-12-15T19:10:33.937000", "content": "

Hi Amten,

\n

Thank you for your amazing work! I am trying to use your implementation to another topics on kaggle, 'First step in Julia'. I used the same parameter shown above, and only got 66%. Can you give me some advise on choosing parameters.

\n

Thanks!

\n

Hang

", "votes": 0, "replies": [] }, { "id": 59446, "author_name": "Johannes Amtén", "author_url": "", "post_date": "2014-12-03T07:58:17.010000", "content": "

[quote=Yanyanyan;59437]

\n

Hi Amten,

\n

Thanks for your implementation.

\n

I have a question, how should we set the parameter in weka when we use this package.

\n

For example, "I used two convolutional layers, the first with 20 feature maps, the second with 100 feature maps, both layers with 5x5 patch-size and 2x2 max-pooling."

\n

how to set these parameters on weka hiddenlayer input would you please make a screenshot for me? 

\n

Thanks

\n

[/quote]

\n

Sure,

\n

Hidden layers parameter: 20-5-5-2-2,100-5-5-2-2

\n

\n\n

(Also, you can just press the "more" button for an explanation of the parameters in the UI.)

", "votes": 0, "replies": [] }, { "id": 59437, "author_name": "Yanyanyan", "author_url": "", "post_date": "2014-12-03T04:21:25.540000", "content": "

Hi Amten,

\n

Thanks for your implementation.

\n

I have a question, how should we set the parameter in weka when we use this package.

\n

For example, "I used two convolutional layers, the first with 20 feature maps, the second with 100 feature maps, both layers with 5x5 patch-size and 2x2 max-pooling."

\n

how to set these parameters on weka hiddenlayer input would you please make a screenshot for me? 

\n

Thanks

", "votes": 0, "replies": [] }, { "id": 51562, "author_name": "Mike Hall", "author_url": "", "post_date": "2014-08-03T18:19:42.710000", "content": "

I have to learn more NN sometime, somewhere. I'm not remembering fully-connected or convolutional coming up in what I've seen so far, which has been pretty much limited to multi-layer perceptron.

\n

Thanks again.

", "votes": 0, "replies": [] }, { "id": 51553, "author_name": "Johannes Amtén", "author_url": "", "post_date": "2014-08-03T15:16:24.967000", "content": "

[quote=Mike Hall;51535]

\n

Works very nicely, even 'out of the box' with default parameters on this dataset. Is that still convolutional?

\n

I had to run it command line though due to memory problems with the Weka GUI. (This on OS X - I got rid of the blas related and it ran fine with just the jars. The linux doc link you provided says the OS X veclib [I think it was] implementation is fast enough you don't need to worry about it and it seems to be.

\n

Thanks.

\n

[/quote]

\n

Cool. No, the default setting is a single, fully-connected, hidden layer with 100 Units.

", "votes": 0, "replies": [] }, { "id": 51535, "author_name": "Mike Hall", "author_url": "", "post_date": "2014-08-02T23:13:08.013000", "content": "

Works very nicely, even 'out of the box' with default parameters on this dataset. Is that still convolutional?

\n

I had to run it command line though due to memory problems with the Weka GUI. (This on OS X - I got rid of the blas related and it ran fine with just the jars. The linux doc link you provided says the OS X veclib [I think it was] implementation is fast enough you don't need to worry about it and it seems to be.

\n

Thanks.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "41006": "", "51563": "", "51171": "", "51166": "", "100685": "Thanks a lot for sharing! I've installed this on WEKA. I'd like to do some image classification (RGB images and greyscale images). But I am not sure how to manage the input for CNN. Should I arrange the images as channel by channel or RGB value by RGB value? ", "100683": "Thank you, Johannes Amtén! It's so good. But I have a question, when I implement your code to recognite characters on Android is very slow. I don't know why. Can you give me some advise how to use it on android.", "90437": "Excellent work ! just one question, how can visualize the learnt model with your weka implementation?", "60272": "", "60253": "", "59446": "", "59437": "", "51562": "", "51553": "", "51535": "" } }