Hello friends,
\n\nWhile going through discussions, I stumbled upon \"simple trick\" by @phunghieu to speed up training. That provoked idea of this thread to accumulate all tips and tricks that can speed up training or inference in this competition. After all, it can save mother earth ( check out this and this research papers) and for some of us, our money.
\n\nSo, let me start:\n- Resize image: As mentioned here\n- Prefetch data to gpu: In tensorflow, this can be done easily using tf.data's prefetch() api. This is not available in pytorch as far as I know, but you can use memory pinning. For single gpu, you can modify your DataLoader as done here.\n- Preprocess on gpu: In tensorflow, there are image ops that can be executed on gpu. We can also use tools like DALI.\n- Mixed precision: You might already know this. If you have gpu with compute capability > 7.5 (rtx, for example), you can use mix precision to speed up training. Also you can fit few more images in a batch 😉. (Side note: Avoid converting your data to float16 for processing on cpu. Float16 operations are slower that float32 on cpu).
I can only remember these on top of my mind. If you know of any other techniques, please do share.
\n\nThanks and belated happy world environment day 😃.
", "messages": [ { "id": "903472", "postDate": "06/26/2020 21:11:12", "content": "Hello friends,
\n\nWhile going through discussions, I stumbled upon \"simple trick\" by @phunghieu to speed up training. That provoked idea of this thread to accumulate all tips and tricks that can speed up training or inference in this competition. After all, it can save mother earth ( check out this and this research papers) and for some of us, our money.
\n\nSo, let me start:\n- Resize image: As mentioned here\n- Prefetch data to gpu: In tensorflow, this can be done easily using tf.data's prefetch() api. This is not available in pytorch as far as I know, but you can use memory pinning. For single gpu, you can modify your DataLoader as done here.\n- Preprocess on gpu: In tensorflow, there are image ops that can be executed on gpu. We can also use tools like DALI.\n- Mixed precision: You might already know this. If you have gpu with compute capability > 7.5 (rtx, for example), you can use mix precision to speed up training. Also you can fit few more images in a batch 😉. (Side note: Avoid converting your data to float16 for processing on cpu. Float16 operations are slower that float32 on cpu).
I can only remember these on top of my mind. If you know of any other techniques, please do share.
\n\nThanks and belated happy world environment day 😃.
", "rawMarkdown": "Hello friends,\n\nWhile going through discussions, I stumbled upon [\"simple trick\"](https://www.kaggle.com/c/global-wheat-detection/discussion/160408) by @phunghieu to speed up training. That provoked idea of this thread to accumulate all tips and tricks that can speed up training or inference in this competition. After all, it can save mother earth ( check out [this](https://arxiv.org/abs/1910.09700) and [this](https://arxiv.org/abs/1906.02243) research papers) and for some of us, our money. \n\nSo, let me start:\n- **Resize image:** As mentioned [here](https://www.kaggle.com/c/global-wheat-detection/discussion/160408)\n- **Prefetch data to gpu:** In tensorflow, this can be done easily using tf.data's `prefetch()` api. This is not available in pytorch as far as I know, but you can use [memory pinning](https://pytorch.org/docs/stable/data.html#memory-pinning). For single gpu, you can modify your DataLoader as [done here](https://github.com/PyTorchLightning/pytorch-lightning/pull/1521/files).\n- **Preprocess on gpu:** In tensorflow, there are image ops that can be executed on gpu. We can also use tools like [DALI](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html).\n- **Mixed precision:** You might already know this. If you have gpu with compute capability > 7.5 (rtx, for example), you can use mix precision to speed up training. Also you can fit few more images in a batch 😉. (*Side note:* Avoid converting your data to float16 for processing on cpu. Float16 operations are slower that float32 on cpu).\n\nI can only remember these on top of my mind. If you know of any other techniques, please do share.\n\nThanks and belated happy world environment day 😃.", "votes": null }, { "id": "903905", "postDate": "06/27/2020 07:49:39", "content": "Great idea. Thanks for sharing!
\n\nstart with smaller batch_size and increase toward the end - early we want frequent updates, and at the end we do not want huge oscillations.
", "rawMarkdown": "start with smaller batch_size and increase toward the end - early we want frequent updates, and at the end we do not want huge oscillations.", "votes": null }, { "id": "904320", "postDate": "06/27/2020 14:29:46", "content": "Yes, and weight is not updated at every step, so some speed up can be due to that.
", "rawMarkdown": "Yes, and weight is not updated at every step, so some speed up can be due to that.", "votes": null }, { "id": "904324", "postDate": "06/27/2020 14:31:30", "content": "Paper on super convergence shows that properly adjusting learning rate can also speed up training.
", "rawMarkdown": "[Paper on super convergence](https://arxiv.org/abs/1708.07120) shows that properly adjusting learning rate can also speed up training.", "votes": null }, { "id": "904441", "postDate": "06/27/2020 16:17:35", "content": "thanks for this
", "rawMarkdown": "thanks for this", "votes": null }, { "id": "915368", "postDate": "07/04/2020 17:14:56", "content": "If you are using pytorch, the auto mixed precision built into the nightly builds is speeding up my training, more so than NVIDIA APEX (about 20-30% more). Pretty simple to implement https://pytorch.org/docs/master/notes/amp_examples.html
", "rawMarkdown": "If you are using pytorch, the auto mixed precision built into the nightly builds is speeding up my training, more so than NVIDIA APEX (about 20-30% more). Pretty simple to implement https://pytorch.org/docs/master/notes/amp_examples.html", "votes": null }, { "id": "945445", "postDate": "07/25/2020 20:06:54", "content": "Do you have an example of this implemented?
\n\nI'm trying to take the effecientdet kernel Alex pushed out and add mixed precision to it. Pytorch nightly doesn't even work with this kernel because of some missing ops, and adding NVidia's APEX to torch 1.5 is also failing due to anchors being on a different device id. Not sure how that's possible. What I've tried:
\n\n```
\n\nfrom apex import amp
\n\nself.optimizer = torch.optim.AdamW(self.model.parameters(), lr=config.lr)\nself.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\nself.model = self.model.to(self.device)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\nself.model = torch.nn.DataParallel(self.model, device_ids=self.config.device_ids)
\n\nbatch_size = images.shape[0]\nimages = torch.stack(images).to(self.device).float()\ntargets = {\n 'bbox': [target['boxes'].to(self.device).float() for target in targets],\n 'cls': [target['labels'].to(self.device).float() for target in targets],\n}\nloss = self.model(images, targets)['loss'] # using latest effdet, so new API here\nwith amp.scale_loss(loss, self.optimizer) as scaled_loss:\n scaled_loss.backward()\n```
\n\nResults in RuntimeError: expected device cuda:1 but got device cuda:0. Then PyTorch latest which has the built in apex is missing some op.
Thank you...
", "rawMarkdown": "Thank you...", "votes": null } ], "comments": [ { "id": 903905, "author_name": "reppic", "author_url": "", "post_date": "06/27/2020 07:49:39", "content": "Great idea. Thanks for sharing!
\n\nstart with smaller batch_size and increase toward the end - early we want frequent updates, and at the end we do not want huge oscillations.
", "votes": null, "replies": [] }, { "id": 904320, "author_name": "ashwan1", "author_url": "", "post_date": "06/27/2020 14:29:46", "content": "Yes, and weight is not updated at every step, so some speed up can be due to that.
", "votes": null, "replies": [] } ] }, { "id": 904324, "author_name": "ashwan1", "author_url": "", "post_date": "06/27/2020 14:31:30", "content": "Paper on super convergence shows that properly adjusting learning rate can also speed up training.
", "votes": null, "replies": [] }, { "id": 904441, "author_name": "yash612", "author_url": "", "post_date": "06/27/2020 16:17:35", "content": "thanks for this
", "votes": null, "replies": [] }, { "id": 915368, "author_name": "researn", "author_url": "", "post_date": "07/04/2020 17:14:56", "content": "If you are using pytorch, the auto mixed precision built into the nightly builds is speeding up my training, more so than NVIDIA APEX (about 20-30% more). Pretty simple to implement https://pytorch.org/docs/master/notes/amp_examples.html
", "votes": null, "replies": [ { "id": 945445, "author_name": "authman", "author_url": "", "post_date": "07/25/2020 20:06:54", "content": "Do you have an example of this implemented?
\n\nI'm trying to take the effecientdet kernel Alex pushed out and add mixed precision to it. Pytorch nightly doesn't even work with this kernel because of some missing ops, and adding NVidia's APEX to torch 1.5 is also failing due to anchors being on a different device id. Not sure how that's possible. What I've tried:
\n\n```
\n\nfrom apex import amp
\n\nself.optimizer = torch.optim.AdamW(self.model.parameters(), lr=config.lr)\nself.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\nself.model = self.model.to(self.device)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\nself.model = torch.nn.DataParallel(self.model, device_ids=self.config.device_ids)
\n\nbatch_size = images.shape[0]\nimages = torch.stack(images).to(self.device).float()\ntargets = {\n 'bbox': [target['boxes'].to(self.device).float() for target in targets],\n 'cls': [target['labels'].to(self.device).float() for target in targets],\n}\nloss = self.model(images, targets)['loss'] # using latest effdet, so new API here\nwith amp.scale_loss(loss, self.optimizer) as scaled_loss:\n scaled_loss.backward()\n```
\n\nResults in RuntimeError: expected device cuda:1 but got device cuda:0. Then PyTorch latest which has the built in apex is missing some op.
Thank you...
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "903472": "Hello friends,\n\nWhile going through discussions, I stumbled upon [\"simple trick\"](https://www.kaggle.com/c/global-wheat-detection/discussion/160408) by @phunghieu to speed up training. That provoked idea of this thread to accumulate all tips and tricks that can speed up training or inference in this competition. After all, it can save mother earth ( check out [this](https://arxiv.org/abs/1910.09700) and [this](https://arxiv.org/abs/1906.02243) research papers) and for some of us, our money. \n\nSo, let me start:\n- **Resize image:** As mentioned [here](https://www.kaggle.com/c/global-wheat-detection/discussion/160408)\n- **Prefetch data to gpu:** In tensorflow, this can be done easily using tf.data's `prefetch()` api. This is not available in pytorch as far as I know, but you can use [memory pinning](https://pytorch.org/docs/stable/data.html#memory-pinning). For single gpu, you can modify your DataLoader as [done here](https://github.com/PyTorchLightning/pytorch-lightning/pull/1521/files).\n- **Preprocess on gpu:** In tensorflow, there are image ops that can be executed on gpu. We can also use tools like [DALI](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html).\n- **Mixed precision:** You might already know this. If you have gpu with compute capability > 7.5 (rtx, for example), you can use mix precision to speed up training. Also you can fit few more images in a batch 😉. (*Side note:* Avoid converting your data to float16 for processing on cpu. Float16 operations are slower that float32 on cpu).\n\nI can only remember these on top of my mind. If you know of any other techniques, please do share.\n\nThanks and belated happy world environment day 😃.", "903905": "Great idea. Thanks for sharing!\n\n* **Gradient Accumulation:** lets you increase your effective batch size which can reduce the amount of noise in your weight updates. It has the potential to speed up convergence and seems to be most effective near the end of a training run.", "904040": "start with smaller batch_size and increase toward the end - early we want frequent updates, and at the end we do not want huge oscillations.", "904320": "Yes, and weight is not updated at every step, so some speed up can be due to that.", "904324": "[Paper on super convergence](https://arxiv.org/abs/1708.07120) shows that properly adjusting learning rate can also speed up training.", "904441": "thanks for this", "915368": "If you are using pytorch, the auto mixed precision built into the nightly builds is speeding up my training, more so than NVIDIA APEX (about 20-30% more). Pretty simple to implement https://pytorch.org/docs/master/notes/amp_examples.html", "945445": "Do you have an example of this implemented?\n\nI'm trying to take the effecientdet kernel Alex pushed out and add mixed precision to it. Pytorch nightly doesn't even work with this kernel because of some missing ops, and adding NVidia's APEX to torch 1.5 is also failing due to anchors being on a different device id. Not sure how that's possible. What I've tried:\n\n```\n# Up top:\nfrom apex import amp\n\n# Inside Fitter()'s init\nself.optimizer = torch.optim.AdamW(self.model.parameters(), lr=config.lr)\nself.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\nself.model = self.model.to(self.device)\nself.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level='O1')\nself.model = torch.nn.DataParallel(self.model, device_ids=self.config.device_ids)\n\n# Inside fit one epoch\nbatch_size = images.shape[0]\nimages = torch.stack(images).to(self.device).float()\ntargets = {\n 'bbox': [target['boxes'].to(self.device).float() for target in targets],\n 'cls': [target['labels'].to(self.device).float() for target in targets],\n}\nloss = self.model(images, targets)['loss'] # using latest effdet, so new API here\nwith amp.scale_loss(loss, self.optimizer) as scaled_loss:\n scaled_loss.backward()\n```\n\nResults in `RuntimeError: expected device cuda:1 but got device cuda:0`. Then PyTorch latest which has the built in apex is missing some op.", "945724": "Thank you..." }, "source": "meta" }