{
  "id": 12838,
  "title": "caffe training curves",
  "url": "/competitions/datasciencebowl/discussion/12838",
  "author_name": "",
  "post_date": "2015-03-16T21:39:44.503Z",
  "votes": 6,
  "comment_count": 10,
  "views": 12299,
  "content": "<p>Hi all,&nbsp; this comp is pretty much winding down and I want to give some kudos to all the strong competitors out there.&nbsp;&nbsp; Attached is my training curves for a Caffe model that I made.&nbsp; The validation is 25%, taken out prior to any data augmenting.&nbsp; I am quite happy with it as I muddled all on my own the last few days...not enough time to make anything competitive but a good learning experience.&nbsp; I thought I had the prediction aspect nailed but every submission has been clearly malformed.&nbsp; I hope to learn from anyone else after the comp if they had any good ways to predict in Caffe.&nbsp; I am using hdf5 data.</p>\n<p>I also will share my python script used to make my plot...it's very hacky but mostly works. I'm sure many of you use something similar but for those who don't, here it is.</p>\n<p>when training caffe:</p>\n<p>./build/tools/caffe train -solver /.../solver_hdf5.prototxt -gpu 0 &gt;&gt; /.../output.txt 2&gt;&amp;1</p>\n<p>to get the plot:</p>\n<p>python progress_plot.py /.../output.txt</p>\n<p>because the training line will append to output, if you ever encounter errors you may need to manually edit output.txt to take out the error text.</p>",
  "messages": [
    {
      "id": "66523",
      "postDate": "03/16/2015 21:39:44",
      "content": "<p>Hi all,&nbsp; this comp is pretty much winding down and I want to give some kudos to all the strong competitors out there.&nbsp;&nbsp; Attached is my training curves for a Caffe model that I made.&nbsp; The validation is 25%, taken out prior to any data augmenting.&nbsp; I am quite happy with it as I muddled all on my own the last few days...not enough time to make anything competitive but a good learning experience.&nbsp; I thought I had the prediction aspect nailed but every submission has been clearly malformed.&nbsp; I hope to learn from anyone else after the comp if they had any good ways to predict in Caffe.&nbsp; I am using hdf5 data.</p>\n<p>I also will share my python script used to make my plot...it's very hacky but mostly works. I'm sure many of you use something similar but for those who don't, here it is.</p>\n<p>when training caffe:</p>\n<p>./build/tools/caffe train -solver /.../solver_hdf5.prototxt -gpu 0 &gt;&gt; /.../output.txt 2&gt;&amp;1</p>\n<p>to get the plot:</p>\n<p>python progress_plot.py /.../output.txt</p>\n<p>because the training line will append to output, if you ever encounter errors you may need to manually edit output.txt to take out the error text.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66563",
      "postDate": "03/16/2015 23:18:58",
      "content": "<p>Here are mine, first with horizontal+vertical flip and AlexNet-style cropping augmentations, then with added zooming augmentation and then with added rotation augmentation.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67050",
      "postDate": "03/18/2015 19:00:20",
      "content": "<p>@zero zero, this is what I was looking for, Thanks for this plot.</p>\n<p>So you run training fora&nbsp; full 20000 iteration of all training data, right?</p>\n<p>How long did it take on what kind of HW?</p>\n<p>I was seeing fluctuations after 100th iteration and concluding that my 5 convolution layer CNN learning ability is pretty much stopping. Warm laptop wasn't very helpful to reach much higher iterations than 400 for my case.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67081",
      "postDate": "03/18/2015 20:12:24",
      "content": "<p>@lucas,</p>\n<p>If I understood your plots you reach logloss under 1 pretty quickly in 100th iteration of full training data. I tried upto 5 convolution layers and not able reach that level after 350 iterations. How many conv layers you used?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67207",
      "postDate": "03/19/2015 11:43:51",
      "content": "<p>@Ertuka: It probably was 6, but I can't access my logs right now.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67242",
      "postDate": "03/19/2015 16:55:09",
      "content": "<p>[quote=Ertuka;67050]</p>\n<p>@zero zero, this is what I was looking for, Thanks for this plot.</p>\n<p>So you run training fora&nbsp; full 20000 iteration of all training data, right?</p>\n<p>How long did it take on what kind of HW?</p>\n<p>I was seeing fluctuations after 100th iteration and concluding that my 5 convolution layer CNN learning ability is pretty much stopping. Warm laptop wasn't very helpful to reach much higher iterations than 400 for my case.</p>\n<p>[/quote]</p>\n<p>20000 iterations took about 8 hours on my gtx580(3G). 100-400 iterations...from what I saw with my data, learning is only just starting in that range so you can still see some fluctuations. Try running it for 1000 and see what happens. Too bad it's also a hardware problem! Although, the gtx580 is a pretty capable card and you can find them for sale for pretty cheap nowadays.</p>\n<p>I have since learned that Caffe may come with a utility that parses log files which could be better than my script.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "67246",
      "postDate": "03/19/2015 17:20:06",
      "content": "<p>@zero zero, I think our iteration counts are not the same thing: your 200000 iteration count is not a complete pass unless your GPU could do a complete pass in a second.</p>\n<p>Depending on the network architecture, one complete pass was around 30-60 seconds for me, I guess I run close to 1 hour compared to your 8 hours. Probably my network architecture wasn't right or good enough as much as I can drive from all this.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "78400",
      "postDate": "05/14/2015 14:37:21",
      "content": "<p>May be it will be useful for somebody. I've just discovered that caffe distribution itself contains a couple of scripts for parsing logs and plotting these types of charts:</p>\n<p>Supported chart types:<br> 0: Test accuracy vs. Iters<br> 1: Test accuracy vs. Seconds<br> 2: Test loss vs. Iters<br> 3: Test loss vs. Seconds<br> 4: Train learning rate vs. Iters<br> 5: Train learning rate vs. Seconds<br> 6: Train loss vs. Iters<br> 7: Train loss vs. Seconds</p>\n<p>You can find them here by default: $CAFFE_ROOT/tools/extra</p>\n<p>And also learning process logs by default put here in the following format:</p>\n<p>/tmp/&lt;program name&gt;.&lt;hostname&gt;.&lt;user name&gt;.log.&lt;severity <br>level&gt;.&lt;date&gt;.&lt;time&gt;.&lt;pid&gt;</p>\n<p>Usage for parsing and &nbsp;plotting:</p>\n<p>cd&nbsp;$CAFFE_ROOT/tools/extra</p>\n<p>python&nbsp;chart_type[0-7] /where/to/save.png /path/to/first.log</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "89277",
      "postDate": "08/13/2015 06:28:17",
      "content": "<p>[quote=lucas-b.eyer.be;66563]</p>\n<p>Here are mine, first with horizontal+vertical flip and AlexNet-style cropping augmentations, then with added zooming augmentation and then with added rotation augmentation.</p>\n<p>[/quote]</p>\n<p>Hi,</p>\n<p>How can you do zooming, rotating augmentation in Caffe?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "90402",
      "postDate": "08/26/2015 06:56:40",
      "content": "<p>There several pull requests and forks for this. </p>\n\n<p>My one <a href=\"https://github.com/ducha-aiki/caffe/tree/more-transforms\">https://github.com/ducha-aiki/caffe/tree/more-transforms</a></p>\n\n<p>More powerful ChenglongChen`s one <a href=\"https://github.com/ChenglongChen/caffe-windows\">https://github.com/ChenglongChen/caffe-windows</a></p>",
      "rawMarkdown": "There several pull requests and forks for this. \r\n\r\nMy one https://github.com/ducha-aiki/caffe/tree/more-transforms\r\n\r\nMore powerful ChenglongChen`s one https://github.com/ChenglongChen/caffe-windows",
      "votes": null
    },
    {
      "id": "132398",
      "postDate": "08/25/2016 06:04:20",
      "content": "<p>Any idea ,how to plot ROC for the same process</p>",
      "rawMarkdown": "Any idea ,how to plot ROC for the same process",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 66563,
      "author_name": "lucasbeyer",
      "author_url": "",
      "post_date": "03/16/2015 23:18:58",
      "content": "<p>Here are mine, first with horizontal+vertical flip and AlexNet-style cropping augmentations, then with added zooming augmentation and then with added rotation augmentation.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67050,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "03/18/2015 19:00:20",
      "content": "<p>@zero zero, this is what I was looking for, Thanks for this plot.</p>\n<p>So you run training fora&nbsp; full 20000 iteration of all training data, right?</p>\n<p>How long did it take on what kind of HW?</p>\n<p>I was seeing fluctuations after 100th iteration and concluding that my 5 convolution layer CNN learning ability is pretty much stopping. Warm laptop wasn't very helpful to reach much higher iterations than 400 for my case.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67081,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "03/18/2015 20:12:24",
      "content": "<p>@lucas,</p>\n<p>If I understood your plots you reach logloss under 1 pretty quickly in 100th iteration of full training data. I tried upto 5 convolution layers and not able reach that level after 350 iterations. How many conv layers you used?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67207,
      "author_name": "lucasbeyer",
      "author_url": "",
      "post_date": "03/19/2015 11:43:51",
      "content": "<p>@Ertuka: It probably was 6, but I can't access my logs right now.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67242,
      "author_name": "zerozero",
      "author_url": "",
      "post_date": "03/19/2015 16:55:09",
      "content": "<p>[quote=Ertuka;67050]</p>\n<p>@zero zero, this is what I was looking for, Thanks for this plot.</p>\n<p>So you run training fora&nbsp; full 20000 iteration of all training data, right?</p>\n<p>How long did it take on what kind of HW?</p>\n<p>I was seeing fluctuations after 100th iteration and concluding that my 5 convolution layer CNN learning ability is pretty much stopping. Warm laptop wasn't very helpful to reach much higher iterations than 400 for my case.</p>\n<p>[/quote]</p>\n<p>20000 iterations took about 8 hours on my gtx580(3G). 100-400 iterations...from what I saw with my data, learning is only just starting in that range so you can still see some fluctuations. Try running it for 1000 and see what happens. Too bad it's also a hardware problem! Although, the gtx580 is a pretty capable card and you can find them for sale for pretty cheap nowadays.</p>\n<p>I have since learned that Caffe may come with a utility that parses log files which could be better than my script.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 67246,
      "author_name": "esorar",
      "author_url": "",
      "post_date": "03/19/2015 17:20:06",
      "content": "<p>@zero zero, I think our iteration counts are not the same thing: your 200000 iteration count is not a complete pass unless your GPU could do a complete pass in a second.</p>\n<p>Depending on the network architecture, one complete pass was around 30-60 seconds for me, I guess I run close to 1 hour compared to your 8 hours. Probably my network architecture wasn't right or good enough as much as I can drive from all this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 78400,
      "author_name": "tsereda",
      "author_url": "",
      "post_date": "05/14/2015 14:37:21",
      "content": "<p>May be it will be useful for somebody. I've just discovered that caffe distribution itself contains a couple of scripts for parsing logs and plotting these types of charts:</p>\n<p>Supported chart types:<br> 0: Test accuracy vs. Iters<br> 1: Test accuracy vs. Seconds<br> 2: Test loss vs. Iters<br> 3: Test loss vs. Seconds<br> 4: Train learning rate vs. Iters<br> 5: Train learning rate vs. Seconds<br> 6: Train loss vs. Iters<br> 7: Train loss vs. Seconds</p>\n<p>You can find them here by default: $CAFFE_ROOT/tools/extra</p>\n<p>And also learning process logs by default put here in the following format:</p>\n<p>/tmp/&lt;program name&gt;.&lt;hostname&gt;.&lt;user name&gt;.log.&lt;severity <br>level&gt;.&lt;date&gt;.&lt;time&gt;.&lt;pid&gt;</p>\n<p>Usage for parsing and &nbsp;plotting:</p>\n<p>cd&nbsp;$CAFFE_ROOT/tools/extra</p>\n<p>python&nbsp;chart_type[0-7] /where/to/save.png /path/to/first.log</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 89277,
      "author_name": "duccuong",
      "author_url": "",
      "post_date": "08/13/2015 06:28:17",
      "content": "<p>[quote=lucas-b.eyer.be;66563]</p>\n<p>Here are mine, first with horizontal+vertical flip and AlexNet-style cropping augmentations, then with added zooming augmentation and then with added rotation augmentation.</p>\n<p>[/quote]</p>\n<p>Hi,</p>\n<p>How can you do zooming, rotating augmentation in Caffe?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 90402,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "08/26/2015 06:56:40",
      "content": "<p>There several pull requests and forks for this. </p>\n\n<p>My one <a href=\"https://github.com/ducha-aiki/caffe/tree/more-transforms\">https://github.com/ducha-aiki/caffe/tree/more-transforms</a></p>\n\n<p>More powerful ChenglongChen`s one <a href=\"https://github.com/ChenglongChen/caffe-windows\">https://github.com/ChenglongChen/caffe-windows</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132398,
      "author_name": "kruthika123",
      "author_url": "",
      "post_date": "08/25/2016 06:04:20",
      "content": "<p>Any idea ,how to plot ROC for the same process</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "66523": "",
    "66563": "",
    "67050": "",
    "67081": "",
    "67207": "",
    "67242": "",
    "67246": "",
    "78400": "",
    "89277": "",
    "90402": "There several pull requests and forks for this. \r\n\r\nMy one https://github.com/ducha-aiki/caffe/tree/more-transforms\r\n\r\nMore powerful ChenglongChen`s one https://github.com/ChenglongChen/caffe-windows",
    "132398": "Any idea ,how to plot ROC for the same process"
  },
  "source": "meta"
}