Hi,
\n\nI'm using a batch-size of 12 and an image-size of 512x512, everytime I call my already in PyTorch implemented fasterrcnn_resnet50 model like this:
\n\nloss_dict = model(images_list,targets_list)
it takes 13 GB of space, calling it again will lead to CUDA Out of Memory Error. I have been trying to fix this issues for a couple of days now but I just cant get behind it.
\n\nI tried to remove it after one iteration with \ndel(loss_dict)\nand afterwards calling \ntorch.cuda.empty_cache()\nbut it does not help.
I have seen other people using the same model with a batch_size of 16 and 1024x1024 image_size without any problems.
\n\nEven when I put my batch_size to 1 it will still crash due to OOM after a while. \nDoes anyone have any ideas? I dont mind sharing my notebook with you, if you want to look over
\n\nThank you in advance.
\n\nEDIT: I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function.
", "messages": [ { "id": 889055, "postDate": "2020-06-16T18:34:01.147Z", "content": "Reduce your batch-size to 4 and check again. If this error persists, there is a memory leak somewhere in your code. Refer to https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory\nor look for a starter notebook in the Notebooks section and compare it with your training loop.
", "rawMarkdown": "Reduce your batch-size to 4 and check again. If this error persists, there is a memory leak somewhere in your code. Refer to https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory\nor look for a starter notebook in the Notebooks section and compare it with your training loop.", "votes": 1, "replies": [ { "id": 889104, "postDate": "2020-06-16T19:00:51.647Z", "content": "Thanks for your reply.
\n\nEven when I put the batch-size to 1 it will still crash due to OOM Error. I checked my code and compared it to others I just cant find the right spot. Do you have any ideas where it could be? Im deleting a lot of tensors but it doesnt do anything.
", "rawMarkdown": "Thanks for your reply.\n\nEven when I put the batch-size to 1 it will still crash due to OOM Error. I checked my code and compared it to others I just cant find the right spot. Do you have any ideas where it could be? Im deleting a lot of tensors but it doesnt do anything." }, { "id": 889108, "postDate": "2020-06-16T19:04:13.660Z", "content": "It is hard to say for me. Does your dataloader send the tensors to cuda or do you do it when training?\nIf you could provide a bit more code of your dataloader and training parts, maybe I could see the issue
", "rawMarkdown": "It is hard to say for me. Does your dataloader send the tensors to cuda or do you do it when training?\nIf you could provide a bit more code of your dataloader and training parts, maybe I could see the issue", "votes": 1 }, { "id": 889146, "postDate": "2020-06-16T19:48:32.447Z", "content": "Thanks for your reply.
\n\nI send the tensors during the training step to cuda. Sure I can share some of my code:
\n\ngetitem Function of DataSet:
\n\n```\ndef getitem(self,index):\n img_file = self.image_csv.loc[[index]].values[0][1]\n img = cv2.imread(self.image_path + img_file + self.extension)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
\n\n for coord in self.image_csv[self.image_csv['image_id']==img_file].bbox:\n coord = ast.literal_eval(coord)\n bbox_coord = [coord[0], coord[1], coord[0]+coord[2], coord[1]+coord[3]]\n self.coord_list.append(bbox_coord)\n\n coord_tens = torch.as_tensor(self.coord_list, dtype=torch.float32)\n label_tens = torch.ones(len(self.coord_list), dtype=torch.int64)\n image_id_tens = torch.tensor([index])\n area_tens = (coord_tens[:, 3] - coord_tens[:, 1]) * (coord_tens[:, 2] - coord_tens[:, 0])\n iscrowd_tens = torch.zeros(len(self.coord_list), dtype=torch.int64)\n\n target = {}\n target[\"boxes\"] = coord_tens\n target[\"labels\"] = label_tens\n target[\"image_id\"] = image_id_tens\n target[\"area\"] = area_tens\n target[\"iscrowd\"] = iscrowd_tens\n\n if self.transforms is not None:\n img_tens = self.transforms(img)\n #TODO target_transform\n else:\n img_tens = torchvision.transforms.ToTensor()(img)\n\n return img_tens, target\n```
\n\nTraining Code:
\n\n```\nmodel.train()\n for images, targets in train_dl:\n images_list = list(image.to(device) for image in images)\n targets_list = [{k: v.to(device) for k, v in t.items()} for t in targets]\n loss_dict = model.forward(images_list,targets_list)\n losses = sum(loss_val for loss_val in loss_dict.values())\n loss_value = losses.item()
\n\n batch_size = len(train_dl)\n train_size += batch_size\n train_loss += loss_value * batch_size\n\n optimizer.zero_grad()\n losses.backward()\n optimizer.step()\n\n if lr_scheduler is not None:\n lr_scheduler.step()\n\n avg_train_loss = train_loss / train_size\n```
", "rawMarkdown": "Thanks for your reply.\n\nI send the tensors during the training step to cuda. Sure I can share some of my code:\n\ngetitem Function of DataSet:\n\n```\ndef __getitem__(self,index):\n img_file = self.image_csv.loc[[index]].values[0][1]\n img = cv2.imread(self.image_path + img_file + self.extension)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n \n for coord in self.image_csv[self.image_csv['image_id']==img_file].bbox:\n coord = ast.literal_eval(coord)\n bbox_coord = [coord[0], coord[1], coord[0]+coord[2], coord[1]+coord[3]]\n self.coord_list.append(bbox_coord)\n \n coord_tens = torch.as_tensor(self.coord_list, dtype=torch.float32)\n label_tens = torch.ones(len(self.coord_list), dtype=torch.int64)\n image_id_tens = torch.tensor([index])\n area_tens = (coord_tens[:, 3] - coord_tens[:, 1]) * (coord_tens[:, 2] - coord_tens[:, 0])\n iscrowd_tens = torch.zeros(len(self.coord_list), dtype=torch.int64)\n \n target = {}\n target[\"boxes\"] = coord_tens\n target[\"labels\"] = label_tens\n target[\"image_id\"] = image_id_tens\n target[\"area\"] = area_tens\n target[\"iscrowd\"] = iscrowd_tens\n \n if self.transforms is not None:\n img_tens = self.transforms(img)\n #TODO target_transform\n else:\n img_tens = torchvision.transforms.ToTensor()(img)\n \n return img_tens, target\n```\n\nTraining Code:\n\n```\nmodel.train()\n for images, targets in train_dl:\n images_list = list(image.to(device) for image in images)\n targets_list = [{k: v.to(device) for k, v in t.items()} for t in targets]\n loss_dict = model.forward(images_list,targets_list)\n losses = sum(loss_val for loss_val in loss_dict.values())\n loss_value = losses.item()\n \n batch_size = len(train_dl)\n train_size += batch_size\n train_loss += loss_value * batch_size\n\n optimizer.zero_grad()\n losses.backward()\n optimizer.step()\n \n if lr_scheduler is not None:\n lr_scheduler.step()\n\n avg_train_loss = train_loss / train_size\n```" }, { "id": 889156, "postDate": "2020-06-16T19:59:40.240Z", "content": "I don't actually see any error here. Are you training with a python script or a notebook? What kind of hardware are you using?
", "rawMarkdown": "I don't actually see any error here. Are you training with a python script or a notebook? What kind of hardware are you using?", "votes": 2 }, { "id": 889160, "postDate": "2020-06-16T20:03:43.357Z", "content": "I'm training with a PyThon Notebook and I'm using the Kaggle GPU
\n\nI did check out some Starter Codes and they go up to 15GB GPU Usage and never higher even though they have an image-size of 1024x1024 and a batch-size of 16
\n\nHow is it about the model definition:
\n\n```\nmodel = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)\nfasterrcnn_weights = torch.load(\"../input/torchvisionfasterrcnn/fasterrcnn_resnet50_fpn.pth\")
\n\nmodel.load_state_dict(fasterrcnn_weights)
\n\nnum_classes = 2
\n\nmodel.roi_heads.box_predictor = FastRCNNPredictor(model.roi_heads.box_predictor.cls_score.in_features, num_classes)\n```
", "rawMarkdown": "I'm training with a PyThon Notebook and I'm using the Kaggle GPU\n\nI did check out some Starter Codes and they go up to 15GB GPU Usage and never higher even though they have an image-size of 1024x1024 and a batch-size of 16\n\nHow is it about the model definition:\n\n```\nmodel = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)\nfasterrcnn_weights = torch.load(\"../input/torchvisionfasterrcnn/fasterrcnn_resnet50_fpn.pth\")\n\nmodel.load_state_dict(fasterrcnn_weights)\n\n# background + wheat head\nnum_classes = 2\n\n#change pretrained cls_score + bbox_pred\nmodel.roi_heads.box_predictor = FastRCNNPredictor(model.roi_heads.box_predictor.cls_score.in_features, num_classes)\n```" }, { "id": 889167, "postDate": "2020-06-16T20:07:59.370Z", "content": "The model definition looks fine. I'm assuming you have .to(device) or .cuda() soon after that? Does the problem still happen if you start from a fresh session and run through everything once?
Yes afterwards in my train function I use:
\n\n\ndevice = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nmodel.to(device)\nlr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=3,gamma=0.1)\n
Even when I refresh and start a completly clean session it will crash. It does run one iteration with all num_workers and afterwards, for a second call it will crash depending on the batch-size and image-size but like i mentioned even both on low will make it crash after a while.
", "rawMarkdown": "Yes afterwards in my train function I use:\n\n```\ndevice = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nmodel.to(device)\nlr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=3,gamma=0.1)\n```\n\nEven when I refresh and start a completly clean session it will crash. It does run one iteration with all num_workers and afterwards, for a second call it will crash depending on the batch-size and image-size but like i mentioned even both on low will make it crash after a while." }, { "id": 918378, "postDate": "2020-07-07T08:08:14.953Z", "content": "Can I please share my notebook with you? Maybe you can see the problem instantly?
\n\nEDIT: I added you, you can now view my notebook if you want. Thank you in advance.
", "rawMarkdown": "Can I please share my notebook with you? Maybe you can see the problem instantly?\n\nEDIT: I added you, you can now view my notebook if you want. Thank you in advance." } ] }, { "id": 918367, "postDate": "2020-07-07T08:00:55.627Z", "content": "Still having the same problem, I tried reducing the image-size of the inital images by using a different Dataset, it did not help. If anyone is interested and willing to help, I would share my notebook so he can look over.
", "rawMarkdown": "Still having the same problem, I tried reducing the image-size of the inital images by using a different Dataset, it did not help. If anyone is interested and willing to help, I would share my notebook so he can look over." }, { "id": 905617, "postDate": "2020-06-28T17:19:57.680Z", "content": "Does anyone else has a solution? Im still having the same problem, I would also not mind sharing my complete notebook with you.
\n\nThank you in advance.
", "rawMarkdown": "Does anyone else has a solution? Im still having the same problem, I would also not mind sharing my complete notebook with you.\n\nThank you in advance." }, { "id": 887619, "postDate": "2020-06-15T19:14:51.607Z", "content": "Hi,
\n\nI'm using a batch-size of 12 and an image-size of 512x512, everytime I call my already in PyTorch implemented fasterrcnn_resnet50 model like this:
\n\nloss_dict = model(images_list,targets_list)
it takes 13 GB of space, calling it again will lead to CUDA Out of Memory Error. I have been trying to fix this issues for a couple of days now but I just cant get behind it.
\n\nI tried to remove it after one iteration with \ndel(loss_dict)\nand afterwards calling \ntorch.cuda.empty_cache()\nbut it does not help.
I have seen other people using the same model with a batch_size of 16 and 1024x1024 image_size without any problems.
\n\nEven when I put my batch_size to 1 it will still crash due to OOM after a while. \nDoes anyone have any ideas? I dont mind sharing my notebook with you, if you want to look over
\n\nThank you in advance.
\n\nEDIT: I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function.
", "rawMarkdown": "Hi, \n\nI'm using a batch-size of 12 and an image-size of 512x512, everytime I call my already in PyTorch implemented fasterrcnn_resnet50 model like this: \n\n`loss_dict = model(images_list,targets_list)`\n\nit takes 13 GB of space, calling it again will lead to CUDA Out of Memory Error. I have been trying to fix this issues for a couple of days now but I just cant get behind it.\n\nI tried to remove it after one iteration with \n`del(loss_dict)`\nand afterwards calling \n`torch.cuda.empty_cache() `\nbut it does not help.\n\nI have seen other people using the same model with a batch_size of 16 and 1024x1024 image_size without any problems.\n\nEven when I put my batch_size to 1 it will still crash due to OOM after a while. \nDoes anyone have any ideas? I dont mind sharing my notebook with you, if you want to look over \n\nThank you in advance.\n\n**EDIT: I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function.**" }, { "id": 918371, "postDate": "2020-07-07T08:04:17.350Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 918400, "postDate": "2020-07-07T08:27:26.057Z", "content": "Have you tried using a debugging tool to see why the loss_dict is actually so large?
I personally did not, only with the debug_memory function but it doesnt really tell anything.
\n\nHow much space does it take for you when you call once:
\n\nloss_dict = model(images, targets)
My model is slightly different but I can tell you it shouldn't be large at all. The loss dict is supposed to return 4 tensors each containing one float32. So it should be like a single kB, not big at all.
", "rawMarkdown": "My model is slightly different but I can tell you it shouldn't be large at all. The loss dict is supposed to return 4 tensors each containing one float32. So it should be like a single kB, not big at all." }, { "id": 919399, "postDate": "2020-07-07T20:36:53.003Z", "content": "Okay, hmm thats weird. Did you check my notebook out? I shared it with you.
", "rawMarkdown": "Okay, hmm thats weird. Did you check my notebook out? I shared it with you.\n" }, { "id": 924110, "postDate": "2020-07-11T09:04:18.270Z", "content": "I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function.
", "rawMarkdown": "I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function." }, { "id": 924363, "postDate": "2020-07-11T11:53:51.303Z", "content": "I'm glad you were able to work it out! I did check out the notebook but was unable to find that problem. Best of luck to you
", "rawMarkdown": "I'm glad you were able to work it out! I did check out the notebook but was unable to find that problem. Best of luck to you", "votes": 1 }, { "id": 924423, "postDate": "2020-07-11T12:18:45.570Z", "content": "Thank you for helping.
\n\nI wish you all the best too :)
", "rawMarkdown": "Thank you for helping.\n\nI wish you all the best too :)" } ] } ], "comments": [ { "id": 889055, "author_name": "Utsav Nandi", "author_url": "", "post_date": "2020-06-16T18:34:01.147000", "content": "Reduce your batch-size to 4 and check again. If this error persists, there is a memory leak somewhere in your code. Refer to https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory\nor look for a starter notebook in the Notebooks section and compare it with your training loop.
", "votes": 1, "replies": [ { "id": 889104, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-06-16T19:00:51.647000", "content": "Thanks for your reply.
\n\nEven when I put the batch-size to 1 it will still crash due to OOM Error. I checked my code and compared it to others I just cant find the right spot. Do you have any ideas where it could be? Im deleting a lot of tensors but it doesnt do anything.
", "votes": 0, "replies": [] }, { "id": 889108, "author_name": "Matt", "author_url": "", "post_date": "2020-06-16T19:04:13.660000", "content": "It is hard to say for me. Does your dataloader send the tensors to cuda or do you do it when training?\nIf you could provide a bit more code of your dataloader and training parts, maybe I could see the issue
", "votes": 1, "replies": [] }, { "id": 889146, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-06-16T19:48:32.447000", "content": "Thanks for your reply.
\n\nI send the tensors during the training step to cuda. Sure I can share some of my code:
\n\ngetitem Function of DataSet:
\n\n```\ndef getitem(self,index):\n img_file = self.image_csv.loc[[index]].values[0][1]\n img = cv2.imread(self.image_path + img_file + self.extension)\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
\n\n for coord in self.image_csv[self.image_csv['image_id']==img_file].bbox:\n coord = ast.literal_eval(coord)\n bbox_coord = [coord[0], coord[1], coord[0]+coord[2], coord[1]+coord[3]]\n self.coord_list.append(bbox_coord)\n\n coord_tens = torch.as_tensor(self.coord_list, dtype=torch.float32)\n label_tens = torch.ones(len(self.coord_list), dtype=torch.int64)\n image_id_tens = torch.tensor([index])\n area_tens = (coord_tens[:, 3] - coord_tens[:, 1]) * (coord_tens[:, 2] - coord_tens[:, 0])\n iscrowd_tens = torch.zeros(len(self.coord_list), dtype=torch.int64)\n\n target = {}\n target[\"boxes\"] = coord_tens\n target[\"labels\"] = label_tens\n target[\"image_id\"] = image_id_tens\n target[\"area\"] = area_tens\n target[\"iscrowd\"] = iscrowd_tens\n\n if self.transforms is not None:\n img_tens = self.transforms(img)\n #TODO target_transform\n else:\n img_tens = torchvision.transforms.ToTensor()(img)\n\n return img_tens, target\n```
\n\nTraining Code:
\n\n```\nmodel.train()\n for images, targets in train_dl:\n images_list = list(image.to(device) for image in images)\n targets_list = [{k: v.to(device) for k, v in t.items()} for t in targets]\n loss_dict = model.forward(images_list,targets_list)\n losses = sum(loss_val for loss_val in loss_dict.values())\n loss_value = losses.item()
\n\n batch_size = len(train_dl)\n train_size += batch_size\n train_loss += loss_value * batch_size\n\n optimizer.zero_grad()\n losses.backward()\n optimizer.step()\n\n if lr_scheduler is not None:\n lr_scheduler.step()\n\n avg_train_loss = train_loss / train_size\n```
", "votes": 0, "replies": [] }, { "id": 889156, "author_name": "Matt", "author_url": "", "post_date": "2020-06-16T19:59:40.240000", "content": "I don't actually see any error here. Are you training with a python script or a notebook? What kind of hardware are you using?
", "votes": 2, "replies": [] }, { "id": 889160, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-06-16T20:03:43.357000", "content": "I'm training with a PyThon Notebook and I'm using the Kaggle GPU
\n\nI did check out some Starter Codes and they go up to 15GB GPU Usage and never higher even though they have an image-size of 1024x1024 and a batch-size of 16
\n\nHow is it about the model definition:
\n\n```\nmodel = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=False)\nfasterrcnn_weights = torch.load(\"../input/torchvisionfasterrcnn/fasterrcnn_resnet50_fpn.pth\")
\n\nmodel.load_state_dict(fasterrcnn_weights)
\n\nnum_classes = 2
\n\nmodel.roi_heads.box_predictor = FastRCNNPredictor(model.roi_heads.box_predictor.cls_score.in_features, num_classes)\n```
", "votes": 0, "replies": [] }, { "id": 889167, "author_name": "Matt", "author_url": "", "post_date": "2020-06-16T20:07:59.370000", "content": "The model definition looks fine. I'm assuming you have .to(device) or .cuda() soon after that? Does the problem still happen if you start from a fresh session and run through everything once?
Yes afterwards in my train function I use:
\n\n\ndevice = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nmodel.to(device)\nlr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=3,gamma=0.1)\n
Even when I refresh and start a completly clean session it will crash. It does run one iteration with all num_workers and afterwards, for a second call it will crash depending on the batch-size and image-size but like i mentioned even both on low will make it crash after a while.
", "votes": 0, "replies": [] }, { "id": 918378, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-07-07T08:08:14.953000", "content": "Can I please share my notebook with you? Maybe you can see the problem instantly?
\n\nEDIT: I added you, you can now view my notebook if you want. Thank you in advance.
", "votes": 0, "replies": [] } ] }, { "id": 918367, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-07-07T08:00:55.627000", "content": "Still having the same problem, I tried reducing the image-size of the inital images by using a different Dataset, it did not help. If anyone is interested and willing to help, I would share my notebook so he can look over.
", "votes": 0, "replies": [] }, { "id": 905617, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-06-28T17:19:57.680000", "content": "Does anyone else has a solution? Im still having the same problem, I would also not mind sharing my complete notebook with you.
\n\nThank you in advance.
", "votes": 0, "replies": [] }, { "id": 918371, "author_name": "", "author_url": "", "post_date": "2020-07-07T08:04:17.350000", "content": "", "votes": 1, "replies": [ { "id": 918400, "author_name": "Matt", "author_url": "", "post_date": "2020-07-07T08:27:26.057000", "content": "Have you tried using a debugging tool to see why the loss_dict is actually so large?
I personally did not, only with the debug_memory function but it doesnt really tell anything.
\n\nHow much space does it take for you when you call once:
\n\nloss_dict = model(images, targets)
My model is slightly different but I can tell you it shouldn't be large at all. The loss dict is supposed to return 4 tensors each containing one float32. So it should be like a single kB, not big at all.
", "votes": 0, "replies": [] }, { "id": 919399, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-07-07T20:36:53.003000", "content": "Okay, hmm thats weird. Did you check my notebook out? I shared it with you.
", "votes": 0, "replies": [] }, { "id": 924110, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-07-11T09:04:18.270000", "content": "I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function.
", "votes": 0, "replies": [] }, { "id": 924363, "author_name": "Matt", "author_url": "", "post_date": "2020-07-11T11:53:51.303000", "content": "I'm glad you were able to work it out! I did check out the notebook but was unable to find that problem. Best of luck to you
", "votes": 1, "replies": [] }, { "id": 924423, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-07-11T12:18:45.570000", "content": "Thank you for helping.
\n\nI wish you all the best too :)
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "889055": "Reduce your batch-size to 4 and check again. If this error persists, there is a memory leak somewhere in your code. Refer to https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory\nor look for a starter notebook in the Notebooks section and compare it with your training loop.", "918367": "Still having the same problem, I tried reducing the image-size of the inital images by using a different Dataset, it did not help. If anyone is interested and willing to help, I would share my notebook so he can look over.", "905617": "Does anyone else has a solution? Im still having the same problem, I would also not mind sharing my complete notebook with you.\n\nThank you in advance.", "887619": "Hi, \n\nI'm using a batch-size of 12 and an image-size of 512x512, everytime I call my already in PyTorch implemented fasterrcnn_resnet50 model like this: \n\n`loss_dict = model(images_list,targets_list)`\n\nit takes 13 GB of space, calling it again will lead to CUDA Out of Memory Error. I have been trying to fix this issues for a couple of days now but I just cant get behind it.\n\nI tried to remove it after one iteration with \n`del(loss_dict)`\nand afterwards calling \n`torch.cuda.empty_cache() `\nbut it does not help.\n\nI have seen other people using the same model with a batch_size of 16 and 1024x1024 image_size without any problems.\n\nEven when I put my batch_size to 1 it will still crash due to OOM after a while. \nDoes anyone have any ideas? I dont mind sharing my notebook with you, if you want to look over \n\nThank you in advance.\n\n**EDIT: I found my error, I had a always appending list of bboxes and each iteration the bboxes of the previous images got also included in the present image, it was very hard to find because it was hidden in my dataset and I always tought the error is in the train-function.**", "918371": "" } }