{ "id": 168199, "title": "Gradient accumulation", "url": "/competitions/global-wheat-detection/discussion/168199", "author_name": "", "post_date": "2020-07-19T15:35:33.349017Z", "votes": 7, "comment_count": 37, "views": 0, "content": "

Is there anyone who managed to train with a batch size higher than 1 with efficientdet-d5 ? (image size = 1024 )
\nhere is my implementation of gradient accumulation but I have the same problem (Cuda OOM .. )

\n

\"\"

", "messages": [ { "id": "935729", "postDate": "07/19/2020 15:35:33", "content": "

Is there anyone who managed to train with a batch size higher than 1 with efficientdet-d5 ? (image size = 1024 )
\nhere is my implementation of gradient accumulation but I have the same problem (Cuda OOM .. )

\n

\"\"

", "rawMarkdown": "Is there anyone who managed to train with a batch size higher than 1 with efficientdet-d5 ? (image size = 1024 ) \nhere is my implementation of gradient accumulation but I have the same problem (Cuda OOM .. ) \n\n![](https://www.googleapis.com/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4832379%2Fb0374574e9e266bc8ccae66b987f4a32%2Funnamed.png)", "votes": null }, { "id": "935739", "postDate": "07/19/2020 15:43:07", "content": "

Using Nvidia Apex you can reach batch size = 2\nIts super easy to setup, just have to import and add a few lines in the code.\nThe data I presented is for 16GB VRAM.

", "rawMarkdown": "Using Nvidia Apex you can reach batch size = 2\nIts super easy to setup, just have to import and add a few lines in the code.\nThe data I presented is for 16GB VRAM.", "votes": null }, { "id": "935740", "postDate": "07/19/2020 15:45:17", "content": "

Thanks for your comment.
\nI installed apex but what lines in code should I add ?

", "rawMarkdown": "Thanks for your comment. \nI installed apex but what lines in code should I add ?", "votes": null }, { "id": "935750", "postDate": "07/19/2020 15:53:45", "content": "

Take a look here https://github.com/NVIDIA/apex . An example snippet is there on how to use apex.

", "rawMarkdown": "Take a look here [https://github.com/NVIDIA/apex](https://github.com/NVIDIA/apex) . An example snippet is there on how to use apex.", "votes": null }, { "id": "935766", "postDate": "07/19/2020 16:04:06", "content": "

I'll tell you how I am using it currently,\n1. Use a kaggle dataset of apex repositery, you can use my public dataset\n2. sys.insert the dataset in kernel\n3. import apex.amp as amp\n4. in init function of Fitter class at the end write(as given in a1)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\n5. in the \"train_one_epoch\" function of Fitter class add \"with amp.scale_loss(loss, self.optimizer) as loss:\" just above \"loss.backward()\" (as given in a2)

\n\n

Verify the process once, I setup mine a long ago so maybe I could be missing some step here.

", "rawMarkdown": "I'll tell you how I am using it currently,\n1. Use a kaggle dataset of apex repositery, you can use my public dataset\n2. sys.insert the dataset in kernel\n3. import apex.amp as amp\n4. in init function of Fitter class at the end write(as given in a1)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\n5. in the \"train_one_epoch\" function of Fitter class add \"with amp.scale_loss(loss, self.optimizer) as loss:\" just above \"loss.backward()\" (as given in a2)\n\nVerify the process once, I setup mine a long ago so maybe I could be missing some step here.", "votes": null }, { "id": "935774", "postDate": "07/19/2020 16:12:26", "content": "

@ii5m0k3ii @mohammadzunaed thanks alot for your help 😄

", "rawMarkdown": "ii5m0k3ii @mohammadzunaed thanks alot for your help 😄", "votes": null }, { "id": "936137", "postDate": "07/20/2020 02:41:14", "content": "

Hi, i'm new to gradient accumulation.\nMay i ask is that necessary to normalize loss(loss /= accumulation_steps) in training?\nWhat's the use of it?

", "rawMarkdown": "Hi, i'm new to gradient accumulation.\nMay i ask is that necessary to normalize loss(loss /= accumulation_steps) in training?\nWhat's the use of it?", "votes": null }, { "id": "936932", "postDate": "07/20/2020 15:58:40", "content": "

Loss gradients are added(accumulated) by loss.backward() and loss / accumulation_steps divides the loss in advance to average the accumulated loss gradients.

", "rawMarkdown": "Loss gradients are added(accumulated) by loss.backward() and loss / accumulation_steps divides the loss in advance to average the accumulated loss gradients.", "votes": null }, { "id": "937565", "postDate": "07/21/2020 04:47:43", "content": "

Thank you for explanation!
\nI got a much smaller training loss(around 0.0x) when using gradient accumulation, while val loss seems normal(around 0.3-0.5). Is there something wrong?

", "rawMarkdown": "Thank you for explanation!\nI got a much smaller training loss(around 0.0x) when using gradient accumulation, while val loss seems normal(around 0.3-0.5). Is there something wrong?", "votes": null }, { "id": "939311", "postDate": "07/22/2020 06:53:17", "content": "

\"\"\nOne thing that I want to point out here is that there is schedule.step() on the metric of summary loss average that is running in fit function after every epoch. Also we need not do 'loss/accumulation_steps' here. PFA for better view

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2F31ac5010524ccd53e67ab944bd781ed3%2FScreenshot%20from%202020-07-22%2012-19-47.png?generation=1595400875664795&alt=media)\nOne thing that I want to point out here is that there is schedule.step() on the metric of summary loss average that is running in fit function after every epoch. Also we need not do 'loss/accumulation_steps' here. PFA for better view", "votes": null }, { "id": "939322", "postDate": "07/22/2020 07:04:05", "content": "

Also if you are running on a two-gpu system, gradient accumulation implementation should look like highlighted codes in the images. To explain model.model is actually the model object in this case which needs to passed to 'torch.nn.DataParallel'. Also I commented out the sccheduler.step in fit function and added it in train_one_epoch function.\n\"\"\n\"\"\n\"\"

", "rawMarkdown": "Also if you are running on a two-gpu system, gradient accumulation implementation should look like highlighted codes in the images. To explain model.model is actually the model object in this case which needs to passed to 'torch.nn.DataParallel'. Also I commented out the sccheduler.step in fit function and added it in train_one_epoch function.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2F7a3955f7f379416c679261928efa2fc4%2FScreenshot%20from%202020-07-22%2012-32-11.png?generation=1595401394622210&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2Fe90cf524fd2c910385414e5e2b26cc3d%2FScreenshot%20from%202020-07-22%2012-31-42.png?generation=1595401419998259&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2F801e194071f875cfd8f3a42d65c44cba%2FScreenshot%20from%202020-07-22%2012-31-16.png?generation=1595401441502720&alt=media)", "votes": null }, { "id": "939524", "postDate": "07/22/2020 09:37:57", "content": "

Hello!! If I run this in kaggle notebook,shoul I add them?

", "rawMarkdown": "Hello!! If I run this in kaggle notebook,shoul I add them?", "votes": null }, { "id": "939525", "postDate": "07/22/2020 09:37:58", "content": "

Hello!! If I run this in kaggle notebook,shoul I add them?

", "rawMarkdown": "Hello!! If I run this in kaggle notebook,shoul I add them?", "votes": null }, { "id": "940076", "postDate": "07/22/2020 17:01:33", "content": "

@ii5m0k3ii I did as you said and got this as output : \nIs it normal to have big train losses like the one in the picture ? \n\"\"

", "rawMarkdown": "ii5m0k3ii I did as you said and got this as output : \nIs it normal to have big train losses like the one in the picture ? \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4832379%2F266216d55aa4340ac9c5a7962d0c150c%2Floss.PNG?generation=1595437292084557&alt=media)", "votes": null }, { "id": "940084", "postDate": "07/22/2020 17:07:39", "content": "

Thanks for your help @kalyanpichuka 😄

", "rawMarkdown": "Thanks for your help @kalyanpichuka 😄", "votes": null }, { "id": "940165", "postDate": "07/22/2020 17:51:45", "content": "

Yeah, the train losses are huge compared to non-apex but Validation losses are small (<1 , ideally).\nAlso, those skips are quite common especially in the start.

", "rawMarkdown": "Yeah, the train losses are huge compared to non-apex but Validation losses are small (<1 , ideally).\nAlso, those skips are quite common especially in the start.", "votes": null }, { "id": "940249", "postDate": "07/22/2020 19:11:29", "content": "

It is really helpful for me.\nThanks!

", "rawMarkdown": "It is really helpful for me.\nThanks!", "votes": null }, { "id": "940353", "postDate": "07/22/2020 21:21:53", "content": "

I was using this kernel and got the same problem. Training loss is much lower (0.0x compared to 0.3-0.4), while validation loss higher than 0.4 even after 40+ epochs. I can't find the problem. Does anyone know what's wrong?

", "rawMarkdown": "I was using [this](https://www.kaggle.com/sramirez/training-efficientdet-gradient-accumulation-f1/comments) kernel and got the same problem. Training loss is much lower (0.0x compared to 0.3-0.4), while validation loss higher than 0.4 even after 40+ epochs. I can't find the problem. Does anyone know what's wrong?", "votes": null }, { "id": "940486", "postDate": "07/23/2020 01:25:27", "content": "

You can update loss in train_one_epoch func as what @kalyanpichuka did in his commets.\nMine seems right now.

", "rawMarkdown": "You can update loss in train_one_epoch func as what @kalyanpichuka did in his commets.\nMine seems right now.", "votes": null }, { "id": "941347", "postDate": "07/23/2020 07:18:29", "content": "

@kalyanpichuka is it taking care of batch momentum accordingly ?

", "rawMarkdown": "kalyanpichuka is it taking care of batch momentum accordingly ?", "votes": null }, { "id": "941802", "postDate": "07/23/2020 12:25:11", "content": "

Thank you! However, after modification, I got learning rate reduced to 1.2207e-8 just after the second epoch. Have you met the same problem?\n\"\"

", "rawMarkdown": "Thank you! However, after modification, I got learning rate reduced to 1.2207e-8 just after the second epoch. Have you met the same problem?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4080021%2Fe497fcd87ac60420d10f60297a77c057%2FScreen%20Shot%202020-07-23%20at%2020.24.51.png?generation=1595507109750196&alt=media)", "votes": null }, { "id": "942202", "postDate": "07/23/2020 16:27:18", "content": "

No, this is new to me.\nMaybe check lr_scheduler would help.

", "rawMarkdown": "No, this is new to me.\nMaybe check lr_scheduler would help.", "votes": null }, { "id": "943309", "postDate": "07/24/2020 09:25:25", "content": "

Check https://www.kaggle.com/c/global-wheat-detection/discussion/169117\nHe implemented gradient accumulation + apex\nYou can train 1024x1024 batch size 2 using his code

", "rawMarkdown": "Check https://www.kaggle.com/c/global-wheat-detection/discussion/169117\nHe implemented gradient accumulation + apex\nYou can train 1024x1024 batch size 2 using his code", "votes": null }, { "id": "943716", "postDate": "07/24/2020 14:32:16", "content": "

@kalyanpichuka Hello, may I ask if it's necessary to update the summary_loss together withe self.optimizer.step()?
\nI think we can move the summary_loss.update() out of the block and update loss each step by summary_loss.update(loss.detach().item(), batch_size). What do you think?

", "rawMarkdown": "kalyanpichuka Hello, may I ask if it's necessary to update the `summary_loss` together withe `self.optimizer.step()`? \nI think we can move the `summary_loss.update()` out of the block and update loss each step by `summary_loss.update(loss.detach().item(), batch_size)`. What do you think?", "votes": null }, { "id": "943823", "postDate": "07/24/2020 15:51:03", "content": "

Thanks @ahmmadrashid 😄

", "rawMarkdown": "Thanks @ahmmadrashid 😄", "votes": null }, { "id": "943856", "postDate": "07/24/2020 16:18:58", "content": "

@zacchaeus Maybe it's a problem with you scheduler implementation, show me where you do scheduler.step and show me what are you scheduler parameters !

", "rawMarkdown": "zacchaeus Maybe it's a problem with you scheduler implementation, show me where you do scheduler.step and show me what are you scheduler parameters !", "votes": null }, { "id": "943858", "postDate": "07/24/2020 16:19:17", "content": "

@ii5m0k3ii Thanks this helps !

", "rawMarkdown": "ii5m0k3ii Thanks this helps !", "votes": null }, { "id": "943873", "postDate": "07/24/2020 16:38:14", "content": "

I solved this issue. Thank you!

", "rawMarkdown": "I solved this issue. Thank you!", "votes": null }, { "id": "944114", "postDate": "07/24/2020 20:54:26", "content": "

@stevenyin you need to do update it like the one I shown in the image only because your loss should be shown only after the gradient accumulation steps are over..if you are doing 8 step gradient accumulation then loss should be shown after 8 steps..otherwise output would look like loss with out gradient accumulation..

", "rawMarkdown": "stevenyin you need to do update it like the one I shown in the image only because your loss should be shown only after the gradient accumulation steps are over..if you are doing 8 step gradient accumulation then loss should be shown after 8 steps..otherwise output would look like loss with out gradient accumulation..", "votes": null }, { "id": "944302", "postDate": "07/25/2020 02:14:32", "content": "

@kalyanpichuka, thank you for your explanation! I get it. I'll adapt my code into your way then. Thanks!

", "rawMarkdown": "kalyanpichuka, thank you for your explanation! I get it. I'll adapt my code into your way then. Thanks!", "votes": null }, { "id": "944340", "postDate": "07/25/2020 03:13:42", "content": "

@kalyanpichuka \ni perform lt like this. I thought step is what matters as that is when is accumulated grad used for update of weights, summary loss is for view only .Please correct me\n\nif step % self.accumulate == 0:\n self.optimizer.step()\n self.optimizer.zero_grad()\nif self.config.step_scheduler:\n self.scheduler.step()\nsummary_loss.update(loss.detach().item(), batch_size)\n

", "rawMarkdown": "kalyanpichuka \ni perform lt like this. I thought step is what matters as that is when is accumulated grad used for update of weights, summary loss is for view only .Please correct me\n```\nif step % self.accumulate == 0:\n self.optimizer.step()\n self.optimizer.zero_grad()\nif self.config.step_scheduler:\n self.scheduler.step()\nsummary_loss.update(loss.detach().item(), batch_size)\n```", "votes": null }, { "id": "944352", "postDate": "07/25/2020 03:37:49", "content": "

I did the same thing. The way just provides you a more straightforward visualization of \"accumulated loss\", nothing to affect the actual gradient accumulation.

", "rawMarkdown": "I did the same thing. The way just provides you a more straightforward visualization of \"accumulated loss\", nothing to affect the actual gradient accumulation.", "votes": null }, { "id": "944433", "postDate": "07/25/2020 05:45:54", "content": "

@jaideepvalani You shouldn't do that way..because you are averaging the loss and displaying it for every step..as you are doing gradient accumulation you need to sum the all the losses of all the steps included in a gradient accumulation and then do an average which will be given by adding

\n\n

summary_loss.update(loss.detach().item(), batch_size*self.accumulation_steps) inside the if loop of 'if (step+1) % self.accumulation_steps ==0 ' as shown in above image

", "rawMarkdown": "jaideepvalani You shouldn't do that way..because you are averaging the loss and displaying it for every step..as you are doing gradient accumulation you need to sum the all the losses of all the steps included in a gradient accumulation and then do an average which will be given by adding\n\nsummary_loss.update(loss.detach().item(), batch_size*self.accumulation_steps) inside the if loop of 'if (step+1) % self.accumulation_steps ==0 ' as shown in above image", "votes": null }, { "id": "944600", "postDate": "07/25/2020 07:51:01", "content": "

Thanku..\nif i understand it correctly ,if we keep it outside the if our calculated loss will be an obsolete or incorrect one,but actual one as result of batch accumulation will be visible only in iteration subsequent to when opt.step is executed ?\nfunctionally weights would still be getting updated post accumulation of gradient only ,just that we will incorrect picture of loss till gradient accumulation is completed?

", "rawMarkdown": "Thanku..\nif i understand it correctly ,if we keep it outside the if our calculated loss will be an obsolete or incorrect one,but actual one as result of batch accumulation will be visible only in iteration subsequent to when opt.step is executed ?\nfunctionally weights would still be getting updated post accumulation of gradient only ,just that we will incorrect picture of loss till gradient accumulation is completed?", "votes": null }, { "id": "944701", "postDate": "07/25/2020 09:13:07", "content": "

@jaideepvalani Exactly. Please do upvote the replies if you find them useful so that others also can use them. Thank you

", "rawMarkdown": "jaideepvalani Exactly. Please do upvote the replies if you find them useful so that others also can use them. Thank you", "votes": null }, { "id": "944802", "postDate": "07/25/2020 11:08:33", "content": "

@kalyanpichuka i did that..\n1) Here should we also average the loss =loss/accumulated steps ,i see in some forums its done that way\n2) I read several forums about it ,one thing common in all that unless you also adjust your Batch Norm stats as per accumulated batches you wont see the real benefit of batch acc, and it could work as good as without accumulation.

", "rawMarkdown": "kalyanpichuka i did that..\n1) Here should we also average the loss =loss/accumulated steps ,i see in some forums its done that way\n2) I read several forums about it ,one thing common in all that unless you also adjust your Batch Norm stats as per accumulated batches you wont see the real benefit of batch acc, and it could work as good as without accumulation.", "votes": null }, { "id": "945801", "postDate": "07/26/2020 06:28:28", "content": "

You should do (step+1) % self.accumulate == 0.

", "rawMarkdown": "You should do `(step+1) % self.accumulate == 0`.", "votes": null }, { "id": "946429", "postDate": "07/26/2020 15:13:57", "content": "

If I am doing img_size 1024 then loss increase 3 to 38\nWhy I am getting this type result?

", "rawMarkdown": "If I am doing img_size 1024 then loss increase 3 to 38\nWhy I am getting this type result?", "votes": null } ], "comments": [ { "id": 935739, "author_name": "ii5m0k3ii", "author_url": "", "post_date": "07/19/2020 15:43:07", "content": "

Using Nvidia Apex you can reach batch size = 2\nIts super easy to setup, just have to import and add a few lines in the code.\nThe data I presented is for 16GB VRAM.

", "votes": null, "replies": [ { "id": 935740, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/19/2020 15:45:17", "content": "

Thanks for your comment.
\nI installed apex but what lines in code should I add ?

", "votes": null, "replies": [] }, { "id": 935750, "author_name": "mohammadzunaed", "author_url": "", "post_date": "07/19/2020 15:53:45", "content": "

Take a look here https://github.com/NVIDIA/apex . An example snippet is there on how to use apex.

", "votes": null, "replies": [] }, { "id": 935766, "author_name": "ii5m0k3ii", "author_url": "", "post_date": "07/19/2020 16:04:06", "content": "

I'll tell you how I am using it currently,\n1. Use a kaggle dataset of apex repositery, you can use my public dataset\n2. sys.insert the dataset in kernel\n3. import apex.amp as amp\n4. in init function of Fitter class at the end write(as given in a1)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\n5. in the \"train_one_epoch\" function of Fitter class add \"with amp.scale_loss(loss, self.optimizer) as loss:\" just above \"loss.backward()\" (as given in a2)

\n\n

Verify the process once, I setup mine a long ago so maybe I could be missing some step here.

", "votes": null, "replies": [] }, { "id": 935774, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/19/2020 16:12:26", "content": "

@ii5m0k3ii @mohammadzunaed thanks alot for your help 😄

", "votes": null, "replies": [] }, { "id": 940076, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/22/2020 17:01:33", "content": "

@ii5m0k3ii I did as you said and got this as output : \nIs it normal to have big train losses like the one in the picture ? \n\"\"

", "votes": null, "replies": [] }, { "id": 940165, "author_name": "ii5m0k3ii", "author_url": "", "post_date": "07/22/2020 17:51:45", "content": "

Yeah, the train losses are huge compared to non-apex but Validation losses are small (<1 , ideally).\nAlso, those skips are quite common especially in the start.

", "votes": null, "replies": [] }, { "id": 943858, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/24/2020 16:19:17", "content": "

@ii5m0k3ii Thanks this helps !

", "votes": null, "replies": [] } ] }, { "id": 936137, "author_name": "hundredark", "author_url": "", "post_date": "07/20/2020 02:41:14", "content": "

Hi, i'm new to gradient accumulation.\nMay i ask is that necessary to normalize loss(loss /= accumulation_steps) in training?\nWhat's the use of it?

", "votes": null, "replies": [ { "id": 936932, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/20/2020 15:58:40", "content": "

Loss gradients are added(accumulated) by loss.backward() and loss / accumulation_steps divides the loss in advance to average the accumulated loss gradients.

", "votes": null, "replies": [] }, { "id": 937565, "author_name": "hundredark", "author_url": "", "post_date": "07/21/2020 04:47:43", "content": "

Thank you for explanation!
\nI got a much smaller training loss(around 0.0x) when using gradient accumulation, while val loss seems normal(around 0.3-0.5). Is there something wrong?

", "votes": null, "replies": [] }, { "id": 940353, "author_name": "zacchaeus", "author_url": "", "post_date": "07/22/2020 21:21:53", "content": "

I was using this kernel and got the same problem. Training loss is much lower (0.0x compared to 0.3-0.4), while validation loss higher than 0.4 even after 40+ epochs. I can't find the problem. Does anyone know what's wrong?

", "votes": null, "replies": [] }, { "id": 940486, "author_name": "hundredark", "author_url": "", "post_date": "07/23/2020 01:25:27", "content": "

You can update loss in train_one_epoch func as what @kalyanpichuka did in his commets.\nMine seems right now.

", "votes": null, "replies": [] }, { "id": 941802, "author_name": "zacchaeus", "author_url": "", "post_date": "07/23/2020 12:25:11", "content": "

Thank you! However, after modification, I got learning rate reduced to 1.2207e-8 just after the second epoch. Have you met the same problem?\n\"\"

", "votes": null, "replies": [] }, { "id": 942202, "author_name": "hundredark", "author_url": "", "post_date": "07/23/2020 16:27:18", "content": "

No, this is new to me.\nMaybe check lr_scheduler would help.

", "votes": null, "replies": [] }, { "id": 943856, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/24/2020 16:18:58", "content": "

@zacchaeus Maybe it's a problem with you scheduler implementation, show me where you do scheduler.step and show me what are you scheduler parameters !

", "votes": null, "replies": [] }, { "id": 943873, "author_name": "zacchaeus", "author_url": "", "post_date": "07/24/2020 16:38:14", "content": "

I solved this issue. Thank you!

", "votes": null, "replies": [] } ] }, { "id": 939311, "author_name": "kalyanpichuka", "author_url": "", "post_date": "07/22/2020 06:53:17", "content": "

\"\"\nOne thing that I want to point out here is that there is schedule.step() on the metric of summary loss average that is running in fit function after every epoch. Also we need not do 'loss/accumulation_steps' here. PFA for better view

", "votes": null, "replies": [ { "id": 946429, "author_name": "", "author_url": "", "post_date": "07/26/2020 15:13:57", "content": "

If I am doing img_size 1024 then loss increase 3 to 38\nWhy I am getting this type result?

", "votes": null, "replies": [] } ] }, { "id": 939322, "author_name": "kalyanpichuka", "author_url": "", "post_date": "07/22/2020 07:04:05", "content": "

Also if you are running on a two-gpu system, gradient accumulation implementation should look like highlighted codes in the images. To explain model.model is actually the model object in this case which needs to passed to 'torch.nn.DataParallel'. Also I commented out the sccheduler.step in fit function and added it in train_one_epoch function.\n\"\"\n\"\"\n\"\"

", "votes": null, "replies": [ { "id": 939524, "author_name": "qq1623620766", "author_url": "", "post_date": "07/22/2020 09:37:57", "content": "

Hello!! If I run this in kaggle notebook,shoul I add them?

", "votes": null, "replies": [] }, { "id": 939525, "author_name": "qq1623620766", "author_url": "", "post_date": "07/22/2020 09:37:58", "content": "

Hello!! If I run this in kaggle notebook,shoul I add them?

", "votes": null, "replies": [] }, { "id": 940084, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/22/2020 17:07:39", "content": "

Thanks for your help @kalyanpichuka 😄

", "votes": null, "replies": [] }, { "id": 940249, "author_name": "piantic", "author_url": "", "post_date": "07/22/2020 19:11:29", "content": "

It is really helpful for me.\nThanks!

", "votes": null, "replies": [] }, { "id": 943716, "author_name": "", "author_url": "", "post_date": "07/24/2020 14:32:16", "content": "

@kalyanpichuka Hello, may I ask if it's necessary to update the summary_loss together withe self.optimizer.step()?
\nI think we can move the summary_loss.update() out of the block and update loss each step by summary_loss.update(loss.detach().item(), batch_size). What do you think?

", "votes": null, "replies": [] }, { "id": 944114, "author_name": "kalyanpichuka", "author_url": "", "post_date": "07/24/2020 20:54:26", "content": "

@stevenyin you need to do update it like the one I shown in the image only because your loss should be shown only after the gradient accumulation steps are over..if you are doing 8 step gradient accumulation then loss should be shown after 8 steps..otherwise output would look like loss with out gradient accumulation..

", "votes": null, "replies": [ { "id": 944302, "author_name": "", "author_url": "", "post_date": "07/25/2020 02:14:32", "content": "

@kalyanpichuka, thank you for your explanation! I get it. I'll adapt my code into your way then. Thanks!

", "votes": null, "replies": [] } ] }, { "id": 944340, "author_name": "jaideepvalani", "author_url": "", "post_date": "07/25/2020 03:13:42", "content": "

@kalyanpichuka \ni perform lt like this. I thought step is what matters as that is when is accumulated grad used for update of weights, summary loss is for view only .Please correct me\n\nif step % self.accumulate == 0:\n self.optimizer.step()\n self.optimizer.zero_grad()\nif self.config.step_scheduler:\n self.scheduler.step()\nsummary_loss.update(loss.detach().item(), batch_size)\n

", "votes": null, "replies": [ { "id": 944352, "author_name": "", "author_url": "", "post_date": "07/25/2020 03:37:49", "content": "

I did the same thing. The way just provides you a more straightforward visualization of \"accumulated loss\", nothing to affect the actual gradient accumulation.

", "votes": null, "replies": [] }, { "id": 945801, "author_name": "mightyrains", "author_url": "", "post_date": "07/26/2020 06:28:28", "content": "

You should do (step+1) % self.accumulate == 0.

", "votes": null, "replies": [] } ] }, { "id": 944433, "author_name": "kalyanpichuka", "author_url": "", "post_date": "07/25/2020 05:45:54", "content": "

@jaideepvalani You shouldn't do that way..because you are averaging the loss and displaying it for every step..as you are doing gradient accumulation you need to sum the all the losses of all the steps included in a gradient accumulation and then do an average which will be given by adding

\n\n

summary_loss.update(loss.detach().item(), batch_size*self.accumulation_steps) inside the if loop of 'if (step+1) % self.accumulation_steps ==0 ' as shown in above image

", "votes": null, "replies": [] }, { "id": 944600, "author_name": "jaideepvalani", "author_url": "", "post_date": "07/25/2020 07:51:01", "content": "

Thanku..\nif i understand it correctly ,if we keep it outside the if our calculated loss will be an obsolete or incorrect one,but actual one as result of batch accumulation will be visible only in iteration subsequent to when opt.step is executed ?\nfunctionally weights would still be getting updated post accumulation of gradient only ,just that we will incorrect picture of loss till gradient accumulation is completed?

", "votes": null, "replies": [] }, { "id": 944701, "author_name": "kalyanpichuka", "author_url": "", "post_date": "07/25/2020 09:13:07", "content": "

@jaideepvalani Exactly. Please do upvote the replies if you find them useful so that others also can use them. Thank you

", "votes": null, "replies": [] }, { "id": 944802, "author_name": "jaideepvalani", "author_url": "", "post_date": "07/25/2020 11:08:33", "content": "

@kalyanpichuka i did that..\n1) Here should we also average the loss =loss/accumulated steps ,i see in some forums its done that way\n2) I read several forums about it ,one thing common in all that unless you also adjust your Batch Norm stats as per accumulated batches you wont see the real benefit of batch acc, and it could work as good as without accumulation.

", "votes": null, "replies": [] } ] }, { "id": 941347, "author_name": "jaideepvalani", "author_url": "", "post_date": "07/23/2020 07:18:29", "content": "

@kalyanpichuka is it taking care of batch momentum accordingly ?

", "votes": null, "replies": [] }, { "id": 943309, "author_name": "ahmmadrashid", "author_url": "", "post_date": "07/24/2020 09:25:25", "content": "

Check https://www.kaggle.com/c/global-wheat-detection/discussion/169117\nHe implemented gradient accumulation + apex\nYou can train 1024x1024 batch size 2 using his code

", "votes": null, "replies": [ { "id": 943823, "author_name": "nacirbouazizi", "author_url": "", "post_date": "07/24/2020 15:51:03", "content": "

Thanks @ahmmadrashid 😄

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "935729": "Is there anyone who managed to train with a batch size higher than 1 with efficientdet-d5 ? (image size = 1024 ) \nhere is my implementation of gradient accumulation but I have the same problem (Cuda OOM .. ) \n\n![](https://www.googleapis.com/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4832379%2Fb0374574e9e266bc8ccae66b987f4a32%2Funnamed.png)", "935739": "Using Nvidia Apex you can reach batch size = 2\nIts super easy to setup, just have to import and add a few lines in the code.\nThe data I presented is for 16GB VRAM.", "935740": "Thanks for your comment. \nI installed apex but what lines in code should I add ?", "935750": "Take a look here [https://github.com/NVIDIA/apex](https://github.com/NVIDIA/apex) . An example snippet is there on how to use apex.", "935766": "I'll tell you how I am using it currently,\n1. Use a kaggle dataset of apex repositery, you can use my public dataset\n2. sys.insert the dataset in kernel\n3. import apex.amp as amp\n4. in init function of Fitter class at the end write(as given in a1)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\n5. in the \"train_one_epoch\" function of Fitter class add \"with amp.scale_loss(loss, self.optimizer) as loss:\" just above \"loss.backward()\" (as given in a2)\n\nVerify the process once, I setup mine a long ago so maybe I could be missing some step here.", "935774": "ii5m0k3ii @mohammadzunaed thanks alot for your help 😄", "936137": "Hi, i'm new to gradient accumulation.\nMay i ask is that necessary to normalize loss(loss /= accumulation_steps) in training?\nWhat's the use of it?", "936932": "Loss gradients are added(accumulated) by loss.backward() and loss / accumulation_steps divides the loss in advance to average the accumulated loss gradients.", "937565": "Thank you for explanation!\nI got a much smaller training loss(around 0.0x) when using gradient accumulation, while val loss seems normal(around 0.3-0.5). Is there something wrong?", "939311": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2F31ac5010524ccd53e67ab944bd781ed3%2FScreenshot%20from%202020-07-22%2012-19-47.png?generation=1595400875664795&alt=media)\nOne thing that I want to point out here is that there is schedule.step() on the metric of summary loss average that is running in fit function after every epoch. Also we need not do 'loss/accumulation_steps' here. PFA for better view", "939322": "Also if you are running on a two-gpu system, gradient accumulation implementation should look like highlighted codes in the images. To explain model.model is actually the model object in this case which needs to passed to 'torch.nn.DataParallel'. Also I commented out the sccheduler.step in fit function and added it in train_one_epoch function.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2F7a3955f7f379416c679261928efa2fc4%2FScreenshot%20from%202020-07-22%2012-32-11.png?generation=1595401394622210&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2Fe90cf524fd2c910385414e5e2b26cc3d%2FScreenshot%20from%202020-07-22%2012-31-42.png?generation=1595401419998259&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1690514%2F801e194071f875cfd8f3a42d65c44cba%2FScreenshot%20from%202020-07-22%2012-31-16.png?generation=1595401441502720&alt=media)", "939524": "Hello!! If I run this in kaggle notebook,shoul I add them?", "939525": "Hello!! If I run this in kaggle notebook,shoul I add them?", "940076": "ii5m0k3ii I did as you said and got this as output : \nIs it normal to have big train losses like the one in the picture ? \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4832379%2F266216d55aa4340ac9c5a7962d0c150c%2Floss.PNG?generation=1595437292084557&alt=media)", "940084": "Thanks for your help @kalyanpichuka 😄", "940165": "Yeah, the train losses are huge compared to non-apex but Validation losses are small (<1 , ideally).\nAlso, those skips are quite common especially in the start.", "940249": "It is really helpful for me.\nThanks!", "940353": "I was using [this](https://www.kaggle.com/sramirez/training-efficientdet-gradient-accumulation-f1/comments) kernel and got the same problem. Training loss is much lower (0.0x compared to 0.3-0.4), while validation loss higher than 0.4 even after 40+ epochs. I can't find the problem. Does anyone know what's wrong?", "940486": "You can update loss in train_one_epoch func as what @kalyanpichuka did in his commets.\nMine seems right now.", "941347": "kalyanpichuka is it taking care of batch momentum accordingly ?", "941802": "Thank you! However, after modification, I got learning rate reduced to 1.2207e-8 just after the second epoch. Have you met the same problem?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4080021%2Fe497fcd87ac60420d10f60297a77c057%2FScreen%20Shot%202020-07-23%20at%2020.24.51.png?generation=1595507109750196&alt=media)", "942202": "No, this is new to me.\nMaybe check lr_scheduler would help.", "943309": "Check https://www.kaggle.com/c/global-wheat-detection/discussion/169117\nHe implemented gradient accumulation + apex\nYou can train 1024x1024 batch size 2 using his code", "943716": "kalyanpichuka Hello, may I ask if it's necessary to update the `summary_loss` together withe `self.optimizer.step()`? \nI think we can move the `summary_loss.update()` out of the block and update loss each step by `summary_loss.update(loss.detach().item(), batch_size)`. What do you think?", "943823": "Thanks @ahmmadrashid 😄", "943856": "zacchaeus Maybe it's a problem with you scheduler implementation, show me where you do scheduler.step and show me what are you scheduler parameters !", "943858": "ii5m0k3ii Thanks this helps !", "943873": "I solved this issue. Thank you!", "944114": "stevenyin you need to do update it like the one I shown in the image only because your loss should be shown only after the gradient accumulation steps are over..if you are doing 8 step gradient accumulation then loss should be shown after 8 steps..otherwise output would look like loss with out gradient accumulation..", "944302": "kalyanpichuka, thank you for your explanation! I get it. I'll adapt my code into your way then. Thanks!", "944340": "kalyanpichuka \ni perform lt like this. I thought step is what matters as that is when is accumulated grad used for update of weights, summary loss is for view only .Please correct me\n```\nif step % self.accumulate == 0:\n self.optimizer.step()\n self.optimizer.zero_grad()\nif self.config.step_scheduler:\n self.scheduler.step()\nsummary_loss.update(loss.detach().item(), batch_size)\n```", "944352": "I did the same thing. The way just provides you a more straightforward visualization of \"accumulated loss\", nothing to affect the actual gradient accumulation.", "944433": "jaideepvalani You shouldn't do that way..because you are averaging the loss and displaying it for every step..as you are doing gradient accumulation you need to sum the all the losses of all the steps included in a gradient accumulation and then do an average which will be given by adding\n\nsummary_loss.update(loss.detach().item(), batch_size*self.accumulation_steps) inside the if loop of 'if (step+1) % self.accumulation_steps ==0 ' as shown in above image", "944600": "Thanku..\nif i understand it correctly ,if we keep it outside the if our calculated loss will be an obsolete or incorrect one,but actual one as result of batch accumulation will be visible only in iteration subsequent to when opt.step is executed ?\nfunctionally weights would still be getting updated post accumulation of gradient only ,just that we will incorrect picture of loss till gradient accumulation is completed?", "944701": "jaideepvalani Exactly. Please do upvote the replies if you find them useful so that others also can use them. Thank you", "944802": "kalyanpichuka i did that..\n1) Here should we also average the loss =loss/accumulated steps ,i see in some forums its done that way\n2) I read several forums about it ,one thing common in all that unless you also adjust your Batch Norm stats as per accumulated batches you wont see the real benefit of batch acc, and it could work as good as without accumulation.", "945801": "You should do `(step+1) % self.accumulate == 0`.", "946429": "If I am doing img_size 1024 then loss increase 3 to 38\nWhy I am getting this type result?" }, "source": "meta" }