{ "id": 10493, "title": "Train you very own deep convolutional network", "url": "/competitions/cifar-10/discussion/10493", "author_name": "", "post_date": "2014-09-30T20:02:36.070Z", "votes": 27, "comment_count": 7, "views": 20797, "content": "

Attached is a (new) implementation of a "spatially-sparse" convolutional network (http://arxiv.org/abs/1409.6070).

\n

Step 1) Look at the README file.
Step 2) Define a network architecture in terms of LeNet style layers. The input layer can be much larger than 32x32, and it will still run relatively efficiently (sparsity!). Be sure to include lots of Network-In-Network layers. [See runCifar10Kaggle.cu]
Step 3) Specify some kind of affine distortion to apply to the training data. [See OpenCVPicture_AffineTransform.h]
Step 4) Hope it all compiles.
Step 5) Run it for a long time; hope it does not crash.

\n

Requirements: CUDA (sm_20), OpenCV, Boost and a reasonably fast GPU

", "messages": [ { "id": "55439", "postDate": "09/30/2014 20:02:36", "content": "

Attached is a (new) implementation of a "spatially-sparse" convolutional network (http://arxiv.org/abs/1409.6070).

\n

Step 1) Look at the README file.
Step 2) Define a network architecture in terms of LeNet style layers. The input layer can be much larger than 32x32, and it will still run relatively efficiently (sparsity!). Be sure to include lots of Network-In-Network layers. [See runCifar10Kaggle.cu]
Step 3) Specify some kind of affine distortion to apply to the training data. [See OpenCVPicture_AffineTransform.h]
Step 4) Hope it all compiles.
Step 5) Run it for a long time; hope it does not crash.

\n

Requirements: CUDA (sm_20), OpenCV, Boost and a reasonably fast GPU

", "rawMarkdown": "", "votes": null }, { "id": "55713", "postDate": "10/07/2014 03:44:46", "content": "

I compile the code, then meet some error. The way I solve the errors is following:

\n

1.If you meet 

\n

/usr/lib/gcc/x86_64-linux-gnu/4.6/include/xmmintrin.h(1015): error: identifier "__builtin_ia32_vec_set_v4hi" is undefined/usr/lib/gcc/x86_64-linux-gnu/4.6/include/xmmintrin.h(1116): error: identifier "__builtin_ia32_pshufw" is undefined/usr/lib/gcc/x86_64-linux-gnu/4.6/include/emmintrin.h(1312): error: identifier "__builtin_ia32_vec_ext_v8hi" is undefined/usr/lib/gcc/x86_64-linux-gnu/4.6/include/emmintrin.h(1318): error: identifier "__builtin_ia32_vec_set_v8hi" is undefined4 errors detected in the compilation of "/tmp/tmpxft_00001d10_00000000-6_runCifar100.cpp1.ii".

\n

You can use cuda >=5.5, or do not use the -O2 in compiling

\n

2.If you meet  

\n

/usr/local/include/boost/assert.hpp:102:47: error: ‘noinline’ was not declared in this scope

\n

You can add -DBOOST_NOINLINE='__attribute__ ((noinline))'

\n

3.If you meet 

\n

/usr/bin/ld: /tmp/tmpxft_00004332_00000000-16_runCifar10Kaggle.o: undefined reference to symbol 'boost::system::system_category()'
/usr/bin/ld: note: 'boost::system::system_category()' is defined in DSO /usr/local/lib/libboost_system.so.1.55.0 so try adding it to the linker command line
/usr/local/lib/libboost_system.so.1.55.0: could not read symbols: Invalid operation

\n

You can add -lboost_system

\n

So the you can compile as:

\n

/usr/local/cuda-5.5/bin/nvcc -DBOOST_NOINLINE='__attribute__ ((noinline))' runCifar10Kaggle.cu readImageToMat.o -lopencv_core -lopencv_highgui -lopencv_imgproc -lrt -lcublas -lboost_thread -lboost_system -ljpeg -arch sm_20 -O2

", "rawMarkdown": "", "votes": null }, { "id": "55921", "postDate": "10/11/2014 10:41:13", "content": "

Thank you jiki for your suggestions and fixes.

\n

Revised program attached.

", "rawMarkdown": "", "votes": null }, { "id": "56267", "postDate": "10/20/2014 00:51:24", "content": "

Attached is the specification for a network architecture used get a test accuracy of 0.9553.

\n

Architecture: input=(3x126x126) -
 320C2 - 320C2 - MP2 -
 640C2 - 10% dropout - 640C2 - 10% dropout - MP2 -
 960C2 - 20% dropout - 960C2 - 20% dropout - MP2 -
 1280C2 - 30% dropout - 1280C2 - 30% dropout - MP2 -
 1600C2 - 40% dropout - 1600C2 - 40% dropout - MP2 -
 1920C2 - 50% dropout - 1920C1 - 50% dropout - 10C1 - Softmax output
 (inspired by
 Multi-column deep neural networks for image classification; Ciresan, Meier and Schmidhuber;
 Network In Network; Lin, Chen and Yan;
 Very Deep Convolutional Networks for Large-Scale Image Recognition; Simonyan and Zisserman)"""

\n

It is a deep convolutional network trained using SparseConvNet (see http://arxiv.org/abs/1409.6070) taking advantage of
- spatial-sparsity in the 126x126 input layer,
- batchwise dropout and a form of Nesterov's accelerated gradient descent,
- (very) leaky rectified linear units, and
- spatial and color training data augmentation.

\n

Train for 500+ iterations, with affine training data augmentation. Then run for 10 more iterations with the affine training data augmentation removed.

", "rawMarkdown": "", "votes": null }, { "id": "56310", "postDate": "10/21/2014 02:52:14", "content": "

Combined 5 different CNNs, get accuracy of 0.94740.

\n

The best single CNN get accuracy of 0.93640.

\n

Three CNNs use Benjamin Graham‘s spatially-sparse CNN, and two use convnet with dropconnect(see http://cs.nyu.edu/~wanli/dropc/ and http://code.google.com/p/cuda-convnet/).

", "rawMarkdown": "", "votes": null }, { "id": "64861", "postDate": "02/25/2015 14:05:58", "content": "

just wondering whether any one manage to compile the sparseConvNetKaggleCifar10Edition.zip  on Mac OS?

", "rawMarkdown": "", "votes": null }, { "id": "114461", "postDate": "04/11/2016 04:49:55", "content": "

Whenever I tweak the convolution parameters, even changing the number 160, the train set error rate will raise to 90% and become stable after a few hundred epochs. However the origin params won't cause such problem. Does anyone know what might be the cause?

", "rawMarkdown": "Whenever I tweak the convolution parameters, even changing the number 160, the train set error rate will raise to 90% and become stable after a few hundred epochs. However the origin params won't cause such problem. Does anyone know what might be the cause?", "votes": null }, { "id": "114497", "postDate": "04/11/2016 12:54:33", "content": "

A more recent version of the software can be found here:\nhttps://github.com/btgraham/SparseConvNet

", "rawMarkdown": "A more recent version of the software can be found here:\r\nhttps://github.com/btgraham/SparseConvNet", "votes": null } ], "comments": [ { "id": 55713, "author_name": "jiki103869", "author_url": "", "post_date": "10/07/2014 03:44:46", "content": "

I compile the code, then meet some error. The way I solve the errors is following:

\n

1.If you meet 

\n

/usr/lib/gcc/x86_64-linux-gnu/4.6/include/xmmintrin.h(1015): error: identifier "__builtin_ia32_vec_set_v4hi" is undefined/usr/lib/gcc/x86_64-linux-gnu/4.6/include/xmmintrin.h(1116): error: identifier "__builtin_ia32_pshufw" is undefined/usr/lib/gcc/x86_64-linux-gnu/4.6/include/emmintrin.h(1312): error: identifier "__builtin_ia32_vec_ext_v8hi" is undefined/usr/lib/gcc/x86_64-linux-gnu/4.6/include/emmintrin.h(1318): error: identifier "__builtin_ia32_vec_set_v8hi" is undefined4 errors detected in the compilation of "/tmp/tmpxft_00001d10_00000000-6_runCifar100.cpp1.ii".

\n

You can use cuda >=5.5, or do not use the -O2 in compiling

\n

2.If you meet  

\n

/usr/local/include/boost/assert.hpp:102:47: error: ‘noinline’ was not declared in this scope

\n

You can add -DBOOST_NOINLINE='__attribute__ ((noinline))'

\n

3.If you meet 

\n

/usr/bin/ld: /tmp/tmpxft_00004332_00000000-16_runCifar10Kaggle.o: undefined reference to symbol 'boost::system::system_category()'
/usr/bin/ld: note: 'boost::system::system_category()' is defined in DSO /usr/local/lib/libboost_system.so.1.55.0 so try adding it to the linker command line
/usr/local/lib/libboost_system.so.1.55.0: could not read symbols: Invalid operation

\n

You can add -lboost_system

\n

So the you can compile as:

\n

/usr/local/cuda-5.5/bin/nvcc -DBOOST_NOINLINE='__attribute__ ((noinline))' runCifar10Kaggle.cu readImageToMat.o -lopencv_core -lopencv_highgui -lopencv_imgproc -lrt -lcublas -lboost_thread -lboost_system -ljpeg -arch sm_20 -O2

", "votes": null, "replies": [] }, { "id": 55921, "author_name": "btgraham", "author_url": "", "post_date": "10/11/2014 10:41:13", "content": "

Thank you jiki for your suggestions and fixes.

\n

Revised program attached.

", "votes": null, "replies": [] }, { "id": 56267, "author_name": "btgraham", "author_url": "", "post_date": "10/20/2014 00:51:24", "content": "

Attached is the specification for a network architecture used get a test accuracy of 0.9553.

\n

Architecture: input=(3x126x126) -
 320C2 - 320C2 - MP2 -
 640C2 - 10% dropout - 640C2 - 10% dropout - MP2 -
 960C2 - 20% dropout - 960C2 - 20% dropout - MP2 -
 1280C2 - 30% dropout - 1280C2 - 30% dropout - MP2 -
 1600C2 - 40% dropout - 1600C2 - 40% dropout - MP2 -
 1920C2 - 50% dropout - 1920C1 - 50% dropout - 10C1 - Softmax output
 (inspired by
 Multi-column deep neural networks for image classification; Ciresan, Meier and Schmidhuber;
 Network In Network; Lin, Chen and Yan;
 Very Deep Convolutional Networks for Large-Scale Image Recognition; Simonyan and Zisserman)"""

\n

It is a deep convolutional network trained using SparseConvNet (see http://arxiv.org/abs/1409.6070) taking advantage of
- spatial-sparsity in the 126x126 input layer,
- batchwise dropout and a form of Nesterov's accelerated gradient descent,
- (very) leaky rectified linear units, and
- spatial and color training data augmentation.

\n

Train for 500+ iterations, with affine training data augmentation. Then run for 10 more iterations with the affine training data augmentation removed.

", "votes": null, "replies": [] }, { "id": 56310, "author_name": "jiki103869", "author_url": "", "post_date": "10/21/2014 02:52:14", "content": "

Combined 5 different CNNs, get accuracy of 0.94740.

\n

The best single CNN get accuracy of 0.93640.

\n

Three CNNs use Benjamin Graham‘s spatially-sparse CNN, and two use convnet with dropconnect(see http://cs.nyu.edu/~wanli/dropc/ and http://code.google.com/p/cuda-convnet/).

", "votes": null, "replies": [] }, { "id": 64861, "author_name": "vinhnguyen", "author_url": "", "post_date": "02/25/2015 14:05:58", "content": "

just wondering whether any one manage to compile the sparseConvNetKaggleCifar10Edition.zip  on Mac OS?

", "votes": null, "replies": [] }, { "id": 114461, "author_name": "julianyg", "author_url": "", "post_date": "04/11/2016 04:49:55", "content": "

Whenever I tweak the convolution parameters, even changing the number 160, the train set error rate will raise to 90% and become stable after a few hundred epochs. However the origin params won't cause such problem. Does anyone know what might be the cause?

", "votes": null, "replies": [] }, { "id": 114497, "author_name": "btgraham", "author_url": "", "post_date": "04/11/2016 12:54:33", "content": "

A more recent version of the software can be found here:\nhttps://github.com/btgraham/SparseConvNet

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "55439": "", "55713": "", "55921": "", "56267": "", "56310": "", "64861": "", "114461": "Whenever I tweak the convolution parameters, even changing the number 160, the train set error rate will raise to 90% and become stable after a few hundred epochs. However the origin params won't cause such problem. Does anyone know what might be the cause?", "114497": "A more recent version of the software can be found here:\r\nhttps://github.com/btgraham/SparseConvNet" }, "source": "meta" }