I have attempted to use the TPU to train a PyTorch model adapted from the Faster RCNN model in torchvision. To do this I followed this notebook in another competition which splits the training into 8-folds and splits the training across all 8 TPU devices. To do that with this competition I decided to do a \"leave-one-out\" fold for each of the 7 sources and then one fold with all the data.
\n\nUnfortunately, I've run into this problem. The following error message comes up during the first epoch of training the models simultaneously.\nRuntimeError: Cannot access data pointer of Tensor that doesn't have storage\nI believe this means that I have run out of memory. I tried again only training one fold on one device and the same error arose.
Has anybody successfully been able to use TPU to train PyTorch models in this competition? I would appreciate any help here
", "messages": [ { "id": "852656", "postDate": "05/18/2020 15:32:06", "content": "I have attempted to use the TPU to train a PyTorch model adapted from the Faster RCNN model in torchvision. To do this I followed this notebook in another competition which splits the training into 8-folds and splits the training across all 8 TPU devices. To do that with this competition I decided to do a \"leave-one-out\" fold for each of the 7 sources and then one fold with all the data.
\n\nUnfortunately, I've run into this problem. The following error message comes up during the first epoch of training the models simultaneously.\nRuntimeError: Cannot access data pointer of Tensor that doesn't have storage\nI believe this means that I have run out of memory. I tried again only training one fold on one device and the same error arose.
Has anybody successfully been able to use TPU to train PyTorch models in this competition? I would appreciate any help here
", "rawMarkdown": "I have attempted to use the TPU to train a PyTorch model adapted from the Faster RCNN model in torchvision. To do this I followed [this notebook](https://www.kaggle.com/tarunpaparaju/panda-challenge-resnet-multitask-8-fold-on-tpu?scriptVersionId=32652649) in another competition which splits the training into 8-folds and splits the training across all 8 TPU devices. To do that with this competition I decided to do a \"leave-one-out\" fold for each of the 7 sources and then one fold with all the data.\n\nUnfortunately, I've run into this problem. The following error message comes up during the first epoch of training the models simultaneously.\n`RuntimeError: Cannot access data pointer of Tensor that doesn't have storage`\nI believe this means that I have run out of memory. I tried again only training one fold on one device and the same error arose.\n\nHas anybody successfully been able to use TPU to train PyTorch models in this competition? I would appreciate any help here", "votes": null }, { "id": "852812", "postDate": "05/18/2020 17:39:26", "content": "I'm not testing TPU for Pytorch in this competition.\nBut I know some useful kernels in other competions.\n- https://www.kaggle.com/piantic/pytorch-tpu\n- https://www.kaggle.com/welkinfeng/plant-pathology-pytorch-tpu-efficientnet-b4
\n\nAnd you are also take some kernel in other competition.\nSee this for Pytorch + TPU using Pytorch XLA and you can get 'Using all 8 TPU cores in parallel can dramatically speed up KFold training.'\n- https://www.kaggle.com/tarunpaparaju/alaska2-steganalysis-efficientnet-b3-pytorch
\n\nI hope this helpful for you.\n:)
", "rawMarkdown": "I'm not testing TPU for Pytorch in this competition.\nBut I know some useful kernels in other competions.\n- https://www.kaggle.com/piantic/pytorch-tpu\n- https://www.kaggle.com/welkinfeng/plant-pathology-pytorch-tpu-efficientnet-b4\n\nAnd you are also take some kernel in other competition.\nSee this for Pytorch + TPU using Pytorch XLA and you can get 'Using all 8 TPU cores in parallel can dramatically speed up KFold training.'\n- https://www.kaggle.com/tarunpaparaju/alaska2-steganalysis-efficientnet-b3-pytorch\n\nI hope this helpful for you.\n:)", "votes": null }, { "id": "852860", "postDate": "05/18/2020 18:24:08", "content": "I tried and faced the same problem, also created a issue with pytorch-xla here. \nThe problem is created by \nreturn torch.ops.torchvision.nms(boxes, scores, iou_threshold)\nwhich is a C++ implemented op, which requires storage which on TPU is not possible somehow.
Update: I figured out the problem from this informative github issue. It's caused because nms is only currently supported on cpu/cuda and it doesn't appear they plan on supporting it anytime soon. There may be a work around where you send the tensors to cpu and perform nms there but the hit to performance will be big.
\n\nUpdate 2: I made a kernel with the workaround however it is way too slow. Looks like TPU's won't be viable for the majority of this competition.
", "rawMarkdown": "Update: I figured out the problem from [this](https://github.com/pytorch/xla/issues/1667) informative github issue. It's caused because nms is only currently supported on cpu/cuda and it doesn't appear they plan on supporting it anytime soon. There may be a work around where you send the tensors to cpu and perform nms there but the hit to performance will be big.\n\nUpdate 2: I made a kernel with the workaround however it is way too slow. Looks like TPU's won't be viable for the majority of this competition.", "votes": null }, { "id": "918706", "postDate": "07/07/2020 12:17:14", "content": "Could you maybe give me a hint of the workaround? A piece of code would be really helpful.
", "rawMarkdown": "Could you maybe give me a hint of the workaround? A piece of code would be really helpful.", "votes": null }, { "id": "919157", "postDate": "07/07/2020 18:29:01", "content": "As I mentioned in my comment, it was too slow to be practical. Furthermore, with the new clarification on solution licensing my solution is not allowed reguardless.
", "rawMarkdown": "As I mentioned in my comment, it was too slow to be practical. Furthermore, with [the new clarification on solution licensing](https://www.kaggle.com/c/global-wheat-detection/discussion/164845) my solution is not allowed reguardless.", "votes": null } ], "comments": [ { "id": 852812, "author_name": "piantic", "author_url": "", "post_date": "05/18/2020 17:39:26", "content": "I'm not testing TPU for Pytorch in this competition.\nBut I know some useful kernels in other competions.\n- https://www.kaggle.com/piantic/pytorch-tpu\n- https://www.kaggle.com/welkinfeng/plant-pathology-pytorch-tpu-efficientnet-b4
\n\nAnd you are also take some kernel in other competition.\nSee this for Pytorch + TPU using Pytorch XLA and you can get 'Using all 8 TPU cores in parallel can dramatically speed up KFold training.'\n- https://www.kaggle.com/tarunpaparaju/alaska2-steganalysis-efficientnet-b3-pytorch
\n\nI hope this helpful for you.\n:)
", "votes": null, "replies": [] }, { "id": 852860, "author_name": "salilm23", "author_url": "", "post_date": "05/18/2020 18:24:08", "content": "I tried and faced the same problem, also created a issue with pytorch-xla here. \nThe problem is created by \nreturn torch.ops.torchvision.nms(boxes, scores, iou_threshold)\nwhich is a C++ implemented op, which requires storage which on TPU is not possible somehow.
Update: I figured out the problem from this informative github issue. It's caused because nms is only currently supported on cpu/cuda and it doesn't appear they plan on supporting it anytime soon. There may be a work around where you send the tensors to cpu and perform nms there but the hit to performance will be big.
\n\nUpdate 2: I made a kernel with the workaround however it is way too slow. Looks like TPU's won't be viable for the majority of this competition.
", "votes": null, "replies": [ { "id": 918706, "author_name": "sunny478", "author_url": "", "post_date": "07/07/2020 12:17:14", "content": "Could you maybe give me a hint of the workaround? A piece of code would be really helpful.
", "votes": null, "replies": [] }, { "id": 919157, "author_name": "matthewmasters", "author_url": "", "post_date": "07/07/2020 18:29:01", "content": "As I mentioned in my comment, it was too slow to be practical. Furthermore, with the new clarification on solution licensing my solution is not allowed reguardless.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "852656": "I have attempted to use the TPU to train a PyTorch model adapted from the Faster RCNN model in torchvision. To do this I followed [this notebook](https://www.kaggle.com/tarunpaparaju/panda-challenge-resnet-multitask-8-fold-on-tpu?scriptVersionId=32652649) in another competition which splits the training into 8-folds and splits the training across all 8 TPU devices. To do that with this competition I decided to do a \"leave-one-out\" fold for each of the 7 sources and then one fold with all the data.\n\nUnfortunately, I've run into this problem. The following error message comes up during the first epoch of training the models simultaneously.\n`RuntimeError: Cannot access data pointer of Tensor that doesn't have storage`\nI believe this means that I have run out of memory. I tried again only training one fold on one device and the same error arose.\n\nHas anybody successfully been able to use TPU to train PyTorch models in this competition? I would appreciate any help here", "852812": "I'm not testing TPU for Pytorch in this competition.\nBut I know some useful kernels in other competions.\n- https://www.kaggle.com/piantic/pytorch-tpu\n- https://www.kaggle.com/welkinfeng/plant-pathology-pytorch-tpu-efficientnet-b4\n\nAnd you are also take some kernel in other competition.\nSee this for Pytorch + TPU using Pytorch XLA and you can get 'Using all 8 TPU cores in parallel can dramatically speed up KFold training.'\n- https://www.kaggle.com/tarunpaparaju/alaska2-steganalysis-efficientnet-b3-pytorch\n\nI hope this helpful for you.\n:)", "852860": "I tried and faced the same problem, also created a issue with pytorch-xla [here](https://github.com/pytorch/xla/issues/2064). \nThe problem is created by \n`return torch.ops.torchvision.nms(boxes, scores, iou_threshold)`\nwhich is a C++ implemented op, which requires storage which on TPU is not possible somehow.", "857083": "Update: I figured out the problem from [this](https://github.com/pytorch/xla/issues/1667) informative github issue. It's caused because nms is only currently supported on cpu/cuda and it doesn't appear they plan on supporting it anytime soon. There may be a work around where you send the tensors to cpu and perform nms there but the hit to performance will be big.\n\nUpdate 2: I made a kernel with the workaround however it is way too slow. Looks like TPU's won't be viable for the majority of this competition.", "918706": "Could you maybe give me a hint of the workaround? A piece of code would be really helpful.", "919157": "As I mentioned in my comment, it was too slow to be practical. Furthermore, with [the new clarification on solution licensing](https://www.kaggle.com/c/global-wheat-detection/discussion/164845) my solution is not allowed reguardless." }, "source": "meta" }