{
  "id": 12887,
  "title": "Brief Describe Method and CXXNET V2",
  "url": "/competitions/datasciencebowl/writeups/happy-lantern-festival-brief-describe-method-and-c",
  "author_name": "",
  "post_date": "2015-03-18T06:50:11.350Z",
  "votes": 18,
  "comment_count": 17,
  "views": 9082,
  "content": "<p>First I apologize&nbsp;that&nbsp;this post is not formal method sharing&nbsp;and CXXNET V2 comes a little late. Full document / configuration will post by Team lead Xudong Cao.</p>\n<p>In general, we are using extremely deep network (much deeper than VGG network) with Spatial Pyramid Pooling&nbsp;and Deeply-Supervised Nets. For activation function, we are using a strange/unpublished activation function. The activation function code is public in Github CXXNET repo for several months, I guess maybe there were&nbsp;some careful guys found it and used it. It will brings much faster converge speed than relu. It is not very like Leaky Relu/PRelu either. We still consider to publish it as an academic paper/workshop paper, but some theory problem need to be solved before that. We also used batch-normal layer, the code is public in repo after we read the paper.</p>\n<p>Thanks to CXXNET V2, the unique GPU-parameter server brings nearly linear speed up for multi-card training. So training such deep network is not so painful.&nbsp;</p>\n<p>Most augmentations are similar to&nbsp;Sander Dieleman's solution. Some&nbsp;minor tricks are shuffle image per epoch&nbsp;and real time random augmentation. CXXNET's new multi-thread binary image iterator is able to random shuffle overall doesn't hurt IO performance, so that we fully utilized computation power.</p>\n<p>New CXXNET V2 is moved to&nbsp;<a href=\"https://github.com/dmlc\">Distributed (Deep) Machine Learning Common</a>, The DMLC will gradually provide kinds of distributed solution for learning algorithm. I think this will be useful for kaggler since competition costs much much more computation resource than ever. Single machine winner is fading out.</p>\n<p>CXXNET V2 is located at&nbsp;<a href=\"https://github.com/dmlc/cxxnet\">https://github.com/dmlc/cxxnet </a>. New version will provide Python/Matlab training/prediction interface. I hope more people will take benefit from it in the <a href=\"https://www.kaggle.com/c/diabetic-retinopathy-detection\">Diabetic Retinopathy Detection competition</a>. &nbsp;</p>\n<p>Cheers,</p>\n<p>Bing</p>",
  "messages": [
    {
      "id": "66910",
      "postDate": "03/18/2015 06:31:02",
      "content": "<p>First I apologize&nbsp;that&nbsp;this post is not formal method sharing&nbsp;and CXXNET V2 comes a little late. Full document / configuration will post by Team lead Xudong Cao.</p>\n<p>In general, we are using extremely deep network (much deeper than VGG network) with Spatial Pyramid Pooling&nbsp;and Deeply-Supervised Nets. For activation function, we are using a strange/unpublished activation function. The activation function code is public in Github CXXNET repo for several months, I guess maybe there were&nbsp;some careful guys found it and used it. It will brings much faster converge speed than relu. It is not very like Leaky Relu/PRelu either. We still consider to publish it as an academic paper/workshop paper, but some theory problem need to be solved before that. We also used batch-normal layer, the code is public in repo after we read the paper.</p>\n<p>Thanks to CXXNET V2, the unique GPU-parameter server brings nearly linear speed up for multi-card training. So training such deep network is not so painful.&nbsp;</p>\n<p>Most augmentations are similar to&nbsp;Sander Dieleman's solution. Some&nbsp;minor tricks are shuffle image per epoch&nbsp;and real time random augmentation. CXXNET's new multi-thread binary image iterator is able to random shuffle overall doesn't hurt IO performance, so that we fully utilized computation power.</p>\n<p>New CXXNET V2 is moved to&nbsp;<a href=\"https://github.com/dmlc\">Distributed (Deep) Machine Learning Common</a>, The DMLC will gradually provide kinds of distributed solution for learning algorithm. I think this will be useful for kaggler since competition costs much much more computation resource than ever. Single machine winner is fading out.</p>\n<p>CXXNET V2 is located at&nbsp;<a href=\"https://github.com/dmlc/cxxnet\">https://github.com/dmlc/cxxnet </a>. New version will provide Python/Matlab training/prediction interface. I hope more people will take benefit from it in the <a href=\"https://www.kaggle.com/c/diabetic-retinopathy-detection\">Diabetic Retinopathy Detection competition</a>. &nbsp;</p>\n<p>Cheers,</p>\n<p>Bing</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66919",
      "postDate": "03/18/2015 08:56:26",
      "content": "<p>Congrats to your team. And I`m looking forward for your full method description, new activation function - is superb result of the competition!</p>\n<p>&gt;We still consider to publish it as an academic paper/workshop paper, but some theory problem need to be solved before that. <br>ICCV deadline is coming, good luck ;)</p>\n<p>P.S.&nbsp;It was really exciting (no offence, Bing, I just like lone wolfs), when single person hold 1st place for a long-long time</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66920",
      "postDate": "03/18/2015 09:01:36",
      "content": "<p>Thank you! Xudong Cao holds some black magic to design neural network. His structure&nbsp;works much better than ours. His single 48x48 network is able to achieve 0.631. I think his black magic should be uncovered by himself later and I am just talking about new and unique feature in CXXNET.</p>\n<p>[quote=old-ufo;66919]</p>\n<p>Congrats to your team. And I`m looking forward for your full method description, new activation function - is superb result of the competition!</p>\n<p>&gt;We still consider to publish it as an academic paper/workshop paper, but some theory problem need to be solved before that. <br>ICCV deadline is coming, good luck ;)</p>\n<p>P.S.&nbsp;It was really exciting (no offence, Bing, I just like lone wolfs), when single person hold 1st place for a long-long time</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66923",
      "postDate": "03/18/2015 09:15:28",
      "content": "<p>[quote=Bing Xu;66920]</p>\n<p>&nbsp;I am just talking about new and unique feature in CXXNET.</p>\n<p>[/quote]</p>\n<p>This means more new tricks and layers, that`s even more great!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66927",
      "postDate": "03/18/2015 09:50:20",
      "content": "<p>Hi Bing Xu,Was using CXXNet in this competition, unfortunately work caught up. &nbsp;Thanks for sharing and looking forward to hearing more. Any idea when V2 will be on the public sight? &nbsp;Keep up the good works!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66928",
      "postDate": "03/18/2015 09:55:46",
      "content": "<p>V2 used to be public in branch V2-refactor.</p>\n<p>Now CXXNET V2 is merged to master, and become part of distributed machine learning common.</p>\n\n<p>[quote=Patrick Chan;66927]</p>\n<p>Hi Bing Xu,Was using CXXNet in this competition, unfortunately work caught up. &nbsp;Thanks for sharing and looking forward to hearing more. Any idea when V2 will be on the public sight? &nbsp;Keep up the good works!</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66986",
      "postDate": "03/18/2015 15:25:30",
      "content": "<p>Bing,</p>\n<p>I tried to used similar config to imageNet with 5 convolution layers but never reached under logloss 1 using CXXNET. In all my attempts around100th iteration generally logloss started to fluctuate instead of decreasing. I am curious whether I needed to run more or used wrong configuration. Even I cloned your V2 branch and used new shear/rotation parameters and still not got much improvement (also never run more than 350-400 iterations).</p>\n<p>I looked briefly new repo and config file for bowl see a generic one (may not do better than 1.1 or 1.2). Without going to black magic, could you add one of the best config file to repo or here so that people like me still confused with CNN architecture can see what is difference. Also can you tell how many iterations you run generally to get good results?</p>\n<p>Due to my laptop warm up issue, I used continue feature all the time to cool down but I am not sure the randomization is working perfectly&nbsp; for stop and go since I observed my logloss for each start seem to get worse (it could be just a coincidence).</p>\n<p>Thanks</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67083",
      "postDate": "03/18/2015 20:20:29",
      "content": "<p>[quote=Bing Xu;66910]</p>\n<p>For activation function, we are using a strange/unpublished activation function. The activation function code is public in Github CXXNET repo for several months</p>\n<p>[/quote]</p>\n<p>This sounds very interesting! Is it in the repo still, or did you remove it? I can't seem to find any non-standard activation functions in the current cxxnet repo.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67138",
      "postDate": "03/19/2015 02:52:25",
      "content": "<p>I found it in the repo. Do a little digging in the history for op.h ;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67166",
      "postDate": "03/19/2015 07:16:34",
      "content": "<p>Bing, congrats to your team. CXXNET is a &nbsp;very useful tool, and I will try it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67177",
      "postDate": "03/19/2015 08:48:48",
      "content": "<p>[quote=D.H.;67138]</p>\n<p>I found it in the repo. Do a little digging in the history for op.h ;)</p>\n<p>[/quote]</p>\n\n<p>In fact I also was looking for it there but could not find it - perhaps some different branch???</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67183",
      "postDate": "03/19/2015 09:01:14",
      "content": "<p>[quote=D.H.;67138]</p>\n<p>I found it in the repo. Do a little digging in the history for op.h ;)</p>\n<p>[/quote]</p>\n<p>I don't know why everyone is being so mysterious about this but anyway, I'll bite. Are you talking about the 'bnll' one? Because that just looks like it was a&nbsp;different formulation of softplus.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67188",
      "postDate": "03/19/2015 09:32:15",
      "content": "<p>I think, about &quot;InsanityLayer&quot;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67190",
      "postDate": "03/19/2015 09:32:23",
      "content": "<p>[delet]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "69379",
      "postDate": "04/02/2015 01:46:48",
      "content": "<p>Bing,</p>\n<p>Since bowl.conf in github is not using this special layer, Can you post a good network config with Insanity layer/magical layer for cxxnet?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "69495",
      "postDate": "04/03/2015 04:39:56",
      "content": "<p>Xudong has posted the network structure. Just use insanity as activation.</p>\n<p>Warning: 48x48 one will cost more than 3GB ram and took about 2-3 days on a single Geforce 780</p>\n\n<p>[quote=Ertuka;69379]</p>\n<p>Bing,</p>\n<p>Since bowl.conf in github is not using this special layer, Can you post a good network config with Insanity layer/magical layer for cxxnet?</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "69543",
      "postDate": "04/03/2015 17:33:33",
      "content": "<p>Bing,</p>\n<p>Apologies that haven't went thru Xudong's post completely and I was using bowl.conf from cxxnet latest changeset, and each iteration seem to take 160 seconds on my laptop with GTX 860M with 2GB ram.</p>\n<p>Thanks for the warning. I am trying to understand this huge memory/speed change. Is that due to Insanity layer or Xudong's network structure.</p>\n<p>What is the complete iteration count in your good runs in this competition or could you post the logloss cost function wrt iteration count. The reason I am asking is that kind of blindly I tried various networks and logloss seem to stuck around 1 after 300-400 iterations. I am curious is it possible to go under 1 in less than 200 or 300 complete iteration of the training data?</p>\n<p>Thanks</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "69545",
      "postDate": "04/03/2015 17:49:55",
      "content": "<p>I suggest you do some study before random trying.&nbsp;</p>\n<p>Stanford cs231n is a good start guide, then try popular ImageNet papers.</p>\n<p>If you know how conv net works, then you will understand why vgg or xudong's net will cost so much computation resource.</p>\n<p>Personally, i think the magic is network structure but not optimization algorithm or activation function. Those two are just helpful for a good net.&nbsp;</p>\n<p>For network below 1.0, I remember someone has posted a net with 0.75 logloss by using graph lab, you can try it, it is same to cxxnet.</p>\n\n\n<p>[quote=Ertuka;69543]</p>\n<p>Bing,</p>\n<p>Apologies that haven't went thru Xudong's post completely and I was using bowl.conf from cxxnet latest changeset, and each iteration seem to take 160 seconds on my laptop with GTX 860M with 2GB ram.</p>\n<p>Thanks for the warning. I am trying to understand this huge memory/speed change. Is that due to Insanity layer or Xudong's network structure.</p>\n<p>What is the complete iteration count in your good runs in this competition or could you post the logloss cost function wrt iteration count. The reason I am asking is that kind of blindly I tried various networks and logloss seem to stuck around 1 after 300-400 iterations. I am curious is it possible to go under 1 in less than 200 or 300 complete iteration of the training data?</p>\n<p>Thanks</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 66919,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/18/2015 08:56:26",
      "content": "<p>Congrats to your team. And I`m looking forward for your full method description, new activation function - is superb result of the competition!</p>\n<p>&gt;We still consider to publish it as an academic paper/workshop paper, but some theory problem need to be solved before that. <br>ICCV deadline is coming, good luck ;)</p>\n<p>P.S.&nbsp;It was really exciting (no offence, Bing, I just like lone wolfs), when single person hold 1st place for a long-long time</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66920,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "03/18/2015 09:01:36",
      "content": "<p>Thank you! Xudong Cao holds some black magic to design neural network. His structure&nbsp;works much better than ours. His single 48x48 network is able to achieve 0.631. I think his black magic should be uncovered by himself later and I am just talking about new and unique feature in CXXNET.</p>\n<p>[quote=old-ufo;66919]</p>\n<p>Congrats to your team. And I`m looking forward for your full method description, new activation function - is superb result of the competition!</p>\n<p>&gt;We still consider to publish it as an academic paper/workshop paper, but some theory problem need to be solved before that. <br>ICCV deadline is coming, good luck ;)</p>\n<p>P.S.&nbsp;It was really exciting (no offence, Bing, I just like lone wolfs), when single person hold 1st place for a long-long time</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66923,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/18/2015 09:15:28",
      "content": "<p>[quote=Bing Xu;66920]</p>\n<p>&nbsp;I am just talking about new and unique feature in CXXNET.</p>\n<p>[/quote]</p>\n<p>This means more new tricks and layers, that`s even more great!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66927,
      "author_name": "drpatrickchan",
      "author_url": "",
      "post_date": "03/18/2015 09:50:20",
      "content": "<p>Hi Bing Xu,Was using CXXNet in this competition, unfortunately work caught up. &nbsp;Thanks for sharing and looking forward to hearing more. Any idea when V2 will be on the public sight? &nbsp;Keep up the good works!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66928,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "03/18/2015 09:55:46",
      "content": "<p>V2 used to be public in branch V2-refactor.</p>\n<p>Now CXXNET V2 is merged to master, and become part of distributed machine learning common.</p>\n\n<p>[quote=Patrick Chan;66927]</p>\n<p>Hi Bing Xu,Was using CXXNet in this competition, unfortunately work caught up. &nbsp;Thanks for sharing and looking forward to hearing more. Any idea when V2 will be on the public sight? &nbsp;Keep up the good works!</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66986,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "03/18/2015 15:25:30",
      "content": "<p>Bing,</p>\n<p>I tried to used similar config to imageNet with 5 convolution layers but never reached under logloss 1 using CXXNET. In all my attempts around100th iteration generally logloss started to fluctuate instead of decreasing. I am curious whether I needed to run more or used wrong configuration. Even I cloned your V2 branch and used new shear/rotation parameters and still not got much improvement (also never run more than 350-400 iterations).</p>\n<p>I looked briefly new repo and config file for bowl see a generic one (may not do better than 1.1 or 1.2). Without going to black magic, could you add one of the best config file to repo or here so that people like me still confused with CNN architecture can see what is difference. Also can you tell how many iterations you run generally to get good results?</p>\n<p>Due to my laptop warm up issue, I used continue feature all the time to cool down but I am not sure the randomization is working perfectly&nbsp; for stop and go since I observed my logloss for each start seem to get worse (it could be just a coincidence).</p>\n<p>Thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67083,
      "author_name": "nicolaykorslund",
      "author_url": "",
      "post_date": "03/18/2015 20:20:29",
      "content": "<p>[quote=Bing Xu;66910]</p>\n<p>For activation function, we are using a strange/unpublished activation function. The activation function code is public in Github CXXNET repo for several months</p>\n<p>[/quote]</p>\n<p>This sounds very interesting! Is it in the repo still, or did you remove it? I can't seem to find any non-standard activation functions in the current cxxnet repo.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67138,
      "author_name": "dhammack",
      "author_url": "",
      "post_date": "03/19/2015 02:52:25",
      "content": "<p>I found it in the repo. Do a little digging in the history for op.h ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67166,
      "author_name": "liangpang",
      "author_url": "",
      "post_date": "03/19/2015 07:16:34",
      "content": "<p>Bing, congrats to your team. CXXNET is a &nbsp;very useful tool, and I will try it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67177,
      "author_name": "romualdj",
      "author_url": "",
      "post_date": "03/19/2015 08:48:48",
      "content": "<p>[quote=D.H.;67138]</p>\n<p>I found it in the repo. Do a little digging in the history for op.h ;)</p>\n<p>[/quote]</p>\n\n<p>In fact I also was looking for it there but could not find it - perhaps some different branch???</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67183,
      "author_name": "sedielem",
      "author_url": "",
      "post_date": "03/19/2015 09:01:14",
      "content": "<p>[quote=D.H.;67138]</p>\n<p>I found it in the repo. Do a little digging in the history for op.h ;)</p>\n<p>[/quote]</p>\n<p>I don't know why everyone is being so mysterious about this but anyway, I'll bite. Are you talking about the 'bnll' one? Because that just looks like it was a&nbsp;different formulation of softplus.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67188,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/19/2015 09:32:15",
      "content": "<p>I think, about &quot;InsanityLayer&quot;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67190,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "03/19/2015 09:32:23",
      "content": "<p>[delet]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 69379,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "04/02/2015 01:46:48",
      "content": "<p>Bing,</p>\n<p>Since bowl.conf in github is not using this special layer, Can you post a good network config with Insanity layer/magical layer for cxxnet?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 69495,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "04/03/2015 04:39:56",
      "content": "<p>Xudong has posted the network structure. Just use insanity as activation.</p>\n<p>Warning: 48x48 one will cost more than 3GB ram and took about 2-3 days on a single Geforce 780</p>\n\n<p>[quote=Ertuka;69379]</p>\n<p>Bing,</p>\n<p>Since bowl.conf in github is not using this special layer, Can you post a good network config with Insanity layer/magical layer for cxxnet?</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 69543,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "04/03/2015 17:33:33",
      "content": "<p>Bing,</p>\n<p>Apologies that haven't went thru Xudong's post completely and I was using bowl.conf from cxxnet latest changeset, and each iteration seem to take 160 seconds on my laptop with GTX 860M with 2GB ram.</p>\n<p>Thanks for the warning. I am trying to understand this huge memory/speed change. Is that due to Insanity layer or Xudong's network structure.</p>\n<p>What is the complete iteration count in your good runs in this competition or could you post the logloss cost function wrt iteration count. The reason I am asking is that kind of blindly I tried various networks and logloss seem to stuck around 1 after 300-400 iterations. I am curious is it possible to go under 1 in less than 200 or 300 complete iteration of the training data?</p>\n<p>Thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 69545,
      "author_name": "binghsu",
      "author_url": "",
      "post_date": "04/03/2015 17:49:55",
      "content": "<p>I suggest you do some study before random trying.&nbsp;</p>\n<p>Stanford cs231n is a good start guide, then try popular ImageNet papers.</p>\n<p>If you know how conv net works, then you will understand why vgg or xudong's net will cost so much computation resource.</p>\n<p>Personally, i think the magic is network structure but not optimization algorithm or activation function. Those two are just helpful for a good net.&nbsp;</p>\n<p>For network below 1.0, I remember someone has posted a net with 0.75 logloss by using graph lab, you can try it, it is same to cxxnet.</p>\n\n\n<p>[quote=Ertuka;69543]</p>\n<p>Bing,</p>\n<p>Apologies that haven't went thru Xudong's post completely and I was using bowl.conf from cxxnet latest changeset, and each iteration seem to take 160 seconds on my laptop with GTX 860M with 2GB ram.</p>\n<p>Thanks for the warning. I am trying to understand this huge memory/speed change. Is that due to Insanity layer or Xudong's network structure.</p>\n<p>What is the complete iteration count in your good runs in this competition or could you post the logloss cost function wrt iteration count. The reason I am asking is that kind of blindly I tried various networks and logloss seem to stuck around 1 after 300-400 iterations. I am curious is it possible to go under 1 in less than 200 or 300 complete iteration of the training data?</p>\n<p>Thanks</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "66910": "",
    "66919": "",
    "66920": "",
    "66923": "",
    "66927": "",
    "66928": "",
    "66986": "",
    "67083": "",
    "67138": "",
    "67166": "",
    "67177": "",
    "67183": "",
    "67188": "",
    "67190": "",
    "69379": "",
    "69495": "",
    "69543": "",
    "69545": ""
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 3978,
  "competition_slug": "datasciencebowl",
  "team_id": 135256,
  "solution_url": "/competitions/datasciencebowl/writeups/happy-lantern-festival-brief-describe-method-and-c"
}