Since apex is used in some notebooks in this competitions I started read a little more about it.\nI came across some good articles on the subject. Below are some of them in case any of you are also interested.\nhttps://medium.com/the-artificial-impostor/use-nvidia-apex-for-easy-mixed-precision-training-in-pytorch-46841c6eed8c\nhttps://towardsdatascience.com/faster-and-memory-efficient-pytorch-models-using-amp-50fd3c8dd7fe
", "messages": [ { "id": "951016", "postDate": "07/29/2020 20:00:18", "content": "Since apex is used in some notebooks in this competitions I started read a little more about it.\nI came across some good articles on the subject. Below are some of them in case any of you are also interested.\nhttps://medium.com/the-artificial-impostor/use-nvidia-apex-for-easy-mixed-precision-training-in-pytorch-46841c6eed8c\nhttps://towardsdatascience.com/faster-and-memory-efficient-pytorch-models-using-amp-50fd3c8dd7fe
", "rawMarkdown": "Since apex is used in some notebooks in this competitions I started read a little more about it.\nI came across some good articles on the subject. Below are some of them in case any of you are also interested.\nhttps://medium.com/the-artificial-impostor/use-nvidia-apex-for-easy-mixed-precision-training-in-pytorch-46841c6eed8c\nhttps://towardsdatascience.com/faster-and-memory-efficient-pytorch-models-using-amp-50fd3c8dd7fe", "votes": null }, { "id": "951040", "postDate": "07/29/2020 20:44:32", "content": "Just to add, pytorch v1.6 officially has the native AMP support, so we might not have to use NVIDIA's apex anymore.\nhttps://pytorch.org/blog/pytorch-1.6-released/
", "rawMarkdown": "Just to add, pytorch v1.6 officially has the native AMP support, so we might not have to use NVIDIA's apex anymore.\n[https://pytorch.org/blog/pytorch-1.6-released/](https://pytorch.org/blog/pytorch-1.6-released/)", "votes": null }, { "id": "951095", "postDate": "07/29/2020 21:51:50", "content": "Ok thanks, released yesterday, thats was a new one indeed. Have you tried with the Pytorch version? Same results and performance?
", "rawMarkdown": "Ok thanks, released yesterday, thats was a new one indeed. Have you tried with the Pytorch version? Same results and performance?", "votes": null }, { "id": "951100", "postDate": "07/29/2020 21:58:10", "content": "I did try it once but couldn't get it to work. As far as I know the results and performance should be the same.
", "rawMarkdown": "I did try it once but couldn't get it to work. As far as I know the results and performance should be the same.", "votes": null }, { "id": "951105", "postDate": "07/29/2020 22:05:00", "content": "Both speed and model metric performance are a tad bit behind. For speed, time per epoch was 158s averaged over 10 runs w/ apex, now its ~162 with native. Please note, I was using Pytorch 1.7dev, not 1.6. And then for model metric, there was a very slight degradation as well. I tested it with a dataset outside of this competition so reporting the actual cv scores is meaningless here, especially w/o understanding the loss function. Native also requires more lines of code than NVidia's (you have to use gradscaler) and additionally wrap your forward calls and in @autocast() if you're training on multiple gpus.
", "rawMarkdown": "Both speed and model metric performance are a tad bit behind. For speed, time per epoch was 158s averaged over 10 runs w/ apex, now its ~162 with native. Please note, I was using Pytorch 1.7dev, not 1.6. And then for model metric, there was a very slight degradation as well. I tested it with a dataset outside of this competition so reporting the actual cv scores is meaningless here, especially w/o understanding the loss function. Native also requires more lines of code than NVidia's (you have to use gradscaler) and additionally wrap your forward calls and in @autocast() if you're training on multiple gpus.", "votes": null }, { "id": "951122", "postDate": "07/29/2020 22:31:56", "content": "Good to know, thanks for the status update from your testing. Surely more will be heard about this feature and how to use it, good initiative to have it built-in.
", "rawMarkdown": "Good to know, thanks for the status update from your testing. Surely more will be heard about this feature and how to use it, good initiative to have it built-in.", "votes": null } ], "comments": [ { "id": 951040, "author_name": "jonykarki", "author_url": "", "post_date": "07/29/2020 20:44:32", "content": "Just to add, pytorch v1.6 officially has the native AMP support, so we might not have to use NVIDIA's apex anymore.\nhttps://pytorch.org/blog/pytorch-1.6-released/
", "votes": null, "replies": [ { "id": 951095, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 21:51:50", "content": "Ok thanks, released yesterday, thats was a new one indeed. Have you tried with the Pytorch version? Same results and performance?
", "votes": null, "replies": [] }, { "id": 951100, "author_name": "jonykarki", "author_url": "", "post_date": "07/29/2020 21:58:10", "content": "I did try it once but couldn't get it to work. As far as I know the results and performance should be the same.
", "votes": null, "replies": [] }, { "id": 951105, "author_name": "authman", "author_url": "", "post_date": "07/29/2020 22:05:00", "content": "Both speed and model metric performance are a tad bit behind. For speed, time per epoch was 158s averaged over 10 runs w/ apex, now its ~162 with native. Please note, I was using Pytorch 1.7dev, not 1.6. And then for model metric, there was a very slight degradation as well. I tested it with a dataset outside of this competition so reporting the actual cv scores is meaningless here, especially w/o understanding the loss function. Native also requires more lines of code than NVidia's (you have to use gradscaler) and additionally wrap your forward calls and in @autocast() if you're training on multiple gpus.
", "votes": null, "replies": [] }, { "id": 951122, "author_name": "kirderf", "author_url": "", "post_date": "07/29/2020 22:31:56", "content": "Good to know, thanks for the status update from your testing. Surely more will be heard about this feature and how to use it, good initiative to have it built-in.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "951016": "Since apex is used in some notebooks in this competitions I started read a little more about it.\nI came across some good articles on the subject. Below are some of them in case any of you are also interested.\nhttps://medium.com/the-artificial-impostor/use-nvidia-apex-for-easy-mixed-precision-training-in-pytorch-46841c6eed8c\nhttps://towardsdatascience.com/faster-and-memory-efficient-pytorch-models-using-amp-50fd3c8dd7fe", "951040": "Just to add, pytorch v1.6 officially has the native AMP support, so we might not have to use NVIDIA's apex anymore.\n[https://pytorch.org/blog/pytorch-1.6-released/](https://pytorch.org/blog/pytorch-1.6-released/)", "951095": "Ok thanks, released yesterday, thats was a new one indeed. Have you tried with the Pytorch version? Same results and performance?", "951100": "I did try it once but couldn't get it to work. As far as I know the results and performance should be the same.", "951105": "Both speed and model metric performance are a tad bit behind. For speed, time per epoch was 158s averaged over 10 runs w/ apex, now its ~162 with native. Please note, I was using Pytorch 1.7dev, not 1.6. And then for model metric, there was a very slight degradation as well. I tested it with a dataset outside of this competition so reporting the actual cv scores is meaningless here, especially w/o understanding the loss function. Native also requires more lines of code than NVidia's (you have to use gradscaler) and additionally wrap your forward calls and in @autocast() if you're training on multiple gpus.", "951122": "Good to know, thanks for the status update from your testing. Surely more will be heard about this feature and how to use it, good initiative to have it built-in." }, "source": "meta" }