{
  "id": 12685,
  "title": "What kind of Hardware is necessary?",
  "url": "/competitions/datasciencebowl/discussion/12685",
  "author_name": "",
  "post_date": "2015-03-04T18:49:09.713Z",
  "votes": null,
  "comment_count": 10,
  "views": 2729,
  "content": "<p>I've been working with the Caffe framework to develop my solution to the competition, and am trying to figure out what is really necessary to make training and testing deep nets less time intensive.</p>\n\n<p>1. In general, what would be the specs on a state of the art desktop for this competition and other kaggle competitions in general?</p>\n\n<p>2. What kind of set up are most people here using?</p>\n\n<p>3. I have a computer with an Intel Q2600 3.4Ghz, GTX 580, and 8GB DDR3 RAM. Would this system be powerful enough to make training deep nets take only a few minutes?</p>\n<p>Thanks for any info!</p>\n\n<p>Best,</p>\n<p>Nitin</p>",
  "messages": [
    {
      "id": "65427",
      "postDate": "03/04/2015 18:49:09",
      "content": "<p>I've been working with the Caffe framework to develop my solution to the competition, and am trying to figure out what is really necessary to make training and testing deep nets less time intensive.</p>\n\n<p>1. In general, what would be the specs on a state of the art desktop for this competition and other kaggle competitions in general?</p>\n\n<p>2. What kind of set up are most people here using?</p>\n\n<p>3. I have a computer with an Intel Q2600 3.4Ghz, GTX 580, and 8GB DDR3 RAM. Would this system be powerful enough to make training deep nets take only a few minutes?</p>\n<p>Thanks for any info!</p>\n\n<p>Best,</p>\n<p>Nitin</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65430",
      "postDate": "03/04/2015 19:06:29",
      "content": "<p>A few minutes? &nbsp;Not any reasonably sized net. &nbsp;On that machine, I would expect any net that obtains &lt; 0.7 score to take closer to a day to train. &nbsp;I could be wrong as I am not familiar with Caffe performance, but I believe that most libraries are fairly close in speed.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65432",
      "postDate": "03/04/2015 19:32:49",
      "content": "<p>Ahh I see, but is one day of training considered a pretty fast speed? I guess the real issue I am having a hard time wrapping my head around then is, is there any way to iterate models faster without having to wait a day to see its performance? I want to play around with different net architectures more to familiarize myself with the process, but if training in the best case takes that long then its still a long process correct?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65435",
      "postDate": "03/04/2015 20:05:28",
      "content": "<p>The &quot;Public Start Guide of Deep Network&quot; thread on this forum shows how to get decent results in as little as five minutes.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65436",
      "postDate": "03/04/2015 20:31:29",
      "content": "<p>[quote=nitini;65432]</p>\n<p>Is there any way to iterate models faster without having to wait a day to see its performance?</p>\n<p>[/quote]</p>\n<p>Some things you can do to speed up iteration:</p>\n<ul>\n<li>Train on smaller image sizes</li>\n<li>Use a smaller net</li>\n<li>Stop after the net is, say, 90% converged</li>\n<li>Etc.</li>\n</ul>\n<p>That's a quick way to evaluate whether you're on the right track. (For example, you don't need to train for a day to know whether binarizing&nbsp;your images improves your score.) Then you can ramp up to make additional gains you feel you're in the right ball park.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65450",
      "postDate": "03/04/2015 21:51:13",
      "content": "<p>That 'five minutes' ad for cxxnet is very misleading! unless you have a GTX TITAN Black or&nbsp;GTX 780 Ti, its almost impossible to train even the example in the getting started in five minutes.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65469",
      "postDate": "03/05/2015 00:15:42",
      "content": "<p>Thats what I thought. Does anyone have any info on what to expect in terms of training times with a GTX 580 (1.5 gb ram).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65548",
      "postDate": "03/05/2015 21:30:45",
      "content": "<p>I have built a machine for this challenge:</p>\n<p>intel 5820k, 32Go DDR4, GTX 980.</p>\n<p>Our best model this far took 17 hours to be trained.</p>\n<p>I would say the biggest limitation to do deeper network right now is the graphic card memory.&nbsp;<br><br></p>\n<p>I had a GTX 560 Ti before.</p>\n<p>I followed the tutorial on this site &nbsp;<a href=\"http://fastml.com/running-things-on-a-gpu/\">http://fastml.com/running-things-on-a-gpu/</a></p>\n<p>When I ran&nbsp;<strong>run_gsn.py</strong> &nbsp;it took 43 sec to train 1 epoch.</p>\n<p>With the GTX 980 it takes 14 sec</p>\n<p>You don't necessary need a high end graphic card, but it will take longer to train that's all.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65614",
      "postDate": "03/06/2015 17:14:02",
      "content": "<p>I am training Caffe models on a GTX 660 with 2GB of GPU memory - so only a slight step up from your 580. My computer is pretty ancient (Core Duo 2.4GHz with 8GB RAM). I am using cuDNN for a slight speed-up over vanilla Caffe. My best model with LB score of 0.752 took less than a day to train (after the data was preprocessed into Caffe format). I think it was probably 95% of the way there after about 12 hours. So your setup is definitely worth sticking with for this competition. Hope that helps.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65649",
      "postDate": "03/06/2015 23:16:02",
      "content": "<p>Alexandre and Senecaur,</p>\n<p>Thanks for the info about running it&nbsp; more than10+ hours. Thanks to Bing, I use cxxnet and run</p>\n<p>less than 10 minutes each time and save models and keep on running after sometime not to</p>\n<p>burn my laptop with GTX 860M. I have not run any my model more than 1-2 hours yet</p>\n<p>always reaching the conclusion that My CNN architecture is stoped learning.</p>\n<p>I am curious how do you cool your PC, or do you partial runs like me?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65683",
      "postDate": "03/07/2015 15:13:05",
      "content": "<p>@Ertuka My PC is a desktop with an open case.&nbsp; I just use the built-in cooling fan on the GPU which never seems to get above 50% activation.&nbsp; Cooling doesn't seem to be an issue for me.&nbsp; I guess things are different with a laptop!</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 65430,
      "author_name": "iamthep",
      "author_url": "",
      "post_date": "03/04/2015 19:06:29",
      "content": "<p>A few minutes? &nbsp;Not any reasonably sized net. &nbsp;On that machine, I would expect any net that obtains &lt; 0.7 score to take closer to a day to train. &nbsp;I could be wrong as I am not familiar with Caffe performance, but I believe that most libraries are fairly close in speed.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65432,
      "author_name": "nitini",
      "author_url": "",
      "post_date": "03/04/2015 19:32:49",
      "content": "<p>Ahh I see, but is one day of training considered a pretty fast speed? I guess the real issue I am having a hard time wrapping my head around then is, is there any way to iterate models faster without having to wait a day to see its performance? I want to play around with different net architectures more to familiarize myself with the process, but if training in the best case takes that long then its still a long process correct?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65435,
      "author_name": "iamthep",
      "author_url": "",
      "post_date": "03/04/2015 20:05:28",
      "content": "<p>The &quot;Public Start Guide of Deep Network&quot; thread on this forum shows how to get decent results in as little as five minutes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65436,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "03/04/2015 20:31:29",
      "content": "<p>[quote=nitini;65432]</p>\n<p>Is there any way to iterate models faster without having to wait a day to see its performance?</p>\n<p>[/quote]</p>\n<p>Some things you can do to speed up iteration:</p>\n<ul>\n<li>Train on smaller image sizes</li>\n<li>Use a smaller net</li>\n<li>Stop after the net is, say, 90% converged</li>\n<li>Etc.</li>\n</ul>\n<p>That's a quick way to evaluate whether you're on the right track. (For example, you don't need to train for a day to know whether binarizing&nbsp;your images improves your score.) Then you can ramp up to make additional gains you feel you're in the right ball park.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65450,
      "author_name": "deepcnn",
      "author_url": "",
      "post_date": "03/04/2015 21:51:13",
      "content": "<p>That 'five minutes' ad for cxxnet is very misleading! unless you have a GTX TITAN Black or&nbsp;GTX 780 Ti, its almost impossible to train even the example in the getting started in five minutes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65469,
      "author_name": "nitini",
      "author_url": "",
      "post_date": "03/05/2015 00:15:42",
      "content": "<p>Thats what I thought. Does anyone have any info on what to expect in terms of training times with a GTX 580 (1.5 gb ram).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65548,
      "author_name": "alexandrekowalczyk",
      "author_url": "",
      "post_date": "03/05/2015 21:30:45",
      "content": "<p>I have built a machine for this challenge:</p>\n<p>intel 5820k, 32Go DDR4, GTX 980.</p>\n<p>Our best model this far took 17 hours to be trained.</p>\n<p>I would say the biggest limitation to do deeper network right now is the graphic card memory.&nbsp;<br><br></p>\n<p>I had a GTX 560 Ti before.</p>\n<p>I followed the tutorial on this site &nbsp;<a href=\"http://fastml.com/running-things-on-a-gpu/\">http://fastml.com/running-things-on-a-gpu/</a></p>\n<p>When I ran&nbsp;<strong>run_gsn.py</strong> &nbsp;it took 43 sec to train 1 epoch.</p>\n<p>With the GTX 980 it takes 14 sec</p>\n<p>You don't necessary need a high end graphic card, but it will take longer to train that's all.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65614,
      "author_name": "senecaur",
      "author_url": "",
      "post_date": "03/06/2015 17:14:02",
      "content": "<p>I am training Caffe models on a GTX 660 with 2GB of GPU memory - so only a slight step up from your 580. My computer is pretty ancient (Core Duo 2.4GHz with 8GB RAM). I am using cuDNN for a slight speed-up over vanilla Caffe. My best model with LB score of 0.752 took less than a day to train (after the data was preprocessed into Caffe format). I think it was probably 95% of the way there after about 12 hours. So your setup is definitely worth sticking with for this competition. Hope that helps.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65649,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "03/06/2015 23:16:02",
      "content": "<p>Alexandre and Senecaur,</p>\n<p>Thanks for the info about running it&nbsp; more than10+ hours. Thanks to Bing, I use cxxnet and run</p>\n<p>less than 10 minutes each time and save models and keep on running after sometime not to</p>\n<p>burn my laptop with GTX 860M. I have not run any my model more than 1-2 hours yet</p>\n<p>always reaching the conclusion that My CNN architecture is stoped learning.</p>\n<p>I am curious how do you cool your PC, or do you partial runs like me?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65683,
      "author_name": "senecaur",
      "author_url": "",
      "post_date": "03/07/2015 15:13:05",
      "content": "<p>@Ertuka My PC is a desktop with an open case.&nbsp; I just use the built-in cooling fan on the GPU which never seems to get above 50% activation.&nbsp; Cooling doesn't seem to be an issue for me.&nbsp; I guess things are different with a laptop!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "65427": "",
    "65430": "",
    "65432": "",
    "65435": "",
    "65436": "",
    "65450": "",
    "65469": "",
    "65548": "",
    "65614": "",
    "65649": "",
    "65683": ""
  },
  "source": "meta"
}