Hi,\nAfter some iters/epochs the training loss becomes NaN.\nI use basic augmentations like rotate + flip. I've checked the box coordinates during the data loader:\n- no negative values\n- no values outside of image\n- if no box then fill coordinates with zero
\n\nI've printed the boxes just before the first NaN loss and coordinates look OK. Min width and height are arround 12/16 pixels. I've also tried various learning rates.
\n\nWhat could cause the issue ? That's frustrating because the same code works well for efficientDet !
", "messages": [ { "id": "878365", "postDate": "06/08/2020 13:26:47", "content": "Hi,\nAfter some iters/epochs the training loss becomes NaN.\nI use basic augmentations like rotate + flip. I've checked the box coordinates during the data loader:\n- no negative values\n- no values outside of image\n- if no box then fill coordinates with zero
\n\nI've printed the boxes just before the first NaN loss and coordinates look OK. Min width and height are arround 12/16 pixels. I've also tried various learning rates.
\n\nWhat could cause the issue ? That's frustrating because the same code works well for efficientDet !
", "rawMarkdown": "Hi,\nAfter some iters/epochs the training loss becomes NaN.\nI use basic augmentations like rotate + flip. I've checked the box coordinates during the data loader:\n- no negative values\n- no values outside of image\n- if no box then fill coordinates with zero\n\nI've printed the boxes just before the first NaN loss and coordinates look OK. Min width and height are arround 12/16 pixels. I've also tried various learning rates.\n\nWhat could cause the issue ? That's frustrating because the same code works well for efficientDet !", "votes": null }, { "id": "878534", "postDate": "06/08/2020 16:09:39", "content": "I am facing the same problem\nIn my case nan loss comes when I set no. of epochs high( almost 10)\nbut while small epochs like 5 doesn't show nan loss\nI am not getting why
", "rawMarkdown": "I am facing the same problem\nIn my case nan loss comes when I set no. of epochs high( almost 10)\nbut while small epochs like 5 doesn't show nan loss\nI am not getting why", "votes": null }, { "id": "878571", "postDate": "06/08/2020 16:32:49", "content": "Only other thing I can think of is a gradient explosion, though I don't think that is the cause
", "rawMarkdown": "Only other thing I can think of is a gradient explosion, though I don't think that is the cause", "votes": null }, { "id": "878771", "postDate": "06/08/2020 20:23:47", "content": "OK I've finally managed to make it work. There was a .long() (that I forgot to remove) conversion in my training loop so that, for example, a bbox in the data loader looks like:\n[ 22.4000, 0.0000, 73.6000, 0.6400]\nAnd the model got as input:\n[ 22, 0, 73, 0]\nwhich are weird coordinates \n@matthewmasters Thanks for your help\n@mrinath I hope it'll give you some clues
Ah, that makes sense. I'm glad you figured it out
", "rawMarkdown": "Ah, that makes sense. I'm glad you figured it out", "votes": null }, { "id": "879170", "postDate": "06/09/2020 09:05:47", "content": "I am facing the same problem in a different contest(PANDAS) tho. I am getting a NaN validation loss for every epoch. I found that the model was outputting NaN for some images. When I saw the tensor outputs after each layer, some of the channels were NaN. What could have caused this? Any idea?
", "rawMarkdown": "I am facing the same problem in a different contest(PANDAS) tho. I am getting a NaN validation loss for every epoch. I found that the model was outputting NaN for some images. When I saw the tensor outputs after each layer, some of the channels were NaN. What could have caused this? Any idea?", "votes": null }, { "id": "879192", "postDate": "06/09/2020 09:24:32", "content": "Did you normalize the data ?
", "rawMarkdown": "Did you normalize the data ?", "votes": null }, { "id": "879422", "postDate": "06/09/2020 13:09:29", "content": "I tried to remove the (.long)\nbut an error comes up\n--> 162 lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]\n 163 rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]\n 164
\n\nRuntimeError: Expected object of scalar type double but got scalar type float for argument 'other'
", "rawMarkdown": "I tried to remove the (.long)\nbut an error comes up\n--> 162 lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]\n 163 rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]\n 164 \n\nRuntimeError: Expected object of scalar type double but got scalar type float for argument 'other'", "votes": null }, { "id": "879471", "postDate": "06/09/2020 13:55:10", "content": "I faced similar issue. You need to modify your data loader like this:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4462580%2Fdf7734b1559de8f6bc299513ec37bc07%2Fdata.PNG?generation=1591710574837938&alt=media\")
\nIt solves three potential issues:\n- You need to use float32 format for the bbox coordinates\n- Advanced augmentations may output an image with no ground truth. In this case I fill the ground truth coordinates with [0, 1, 2, 3]\n- Do not forget to reshape to add extra dimension for the batch size
yes
", "rawMarkdown": "yes", "votes": null }, { "id": "943164", "postDate": "07/24/2020 07:15:05", "content": "My losses are NaN right from the beginning. I tried everything in this thread, but nothing has worked yet. are there can more suggestions?\nI keep getting a \"upsample_bilinear2d_out_frame\" not implemented for 'Byte' RunTime Error until i add\nmodel = model.float()\nand\nimages = [image.float() for image in images]\nAnd after adding these two lines, the training runs and i get NaN losses\nI am a newbie to PyTorch, so i'm having a very hard time figuring out where everything is going wrong.
I figured it out, faster rcnn with resnet 50 backbone in pytorch expects coordinates in format x1,y1,x2,y2 while i was providing the coordinates in x,y,w,h format
", "rawMarkdown": "I figured it out, faster rcnn with resnet 50 backbone in pytorch expects coordinates in format x1,y1,x2,y2 while i was providing the coordinates in x,y,w,h format", "votes": null } ], "comments": [ { "id": 878534, "author_name": "mrinath", "author_url": "", "post_date": "06/08/2020 16:09:39", "content": "I am facing the same problem\nIn my case nan loss comes when I set no. of epochs high( almost 10)\nbut while small epochs like 5 doesn't show nan loss\nI am not getting why
", "votes": null, "replies": [] }, { "id": 878571, "author_name": "matthewmasters", "author_url": "", "post_date": "06/08/2020 16:32:49", "content": "Only other thing I can think of is a gradient explosion, though I don't think that is the cause
", "votes": null, "replies": [] }, { "id": 878771, "author_name": "alexj21", "author_url": "", "post_date": "06/08/2020 20:23:47", "content": "OK I've finally managed to make it work. There was a .long() (that I forgot to remove) conversion in my training loop so that, for example, a bbox in the data loader looks like:\n[ 22.4000, 0.0000, 73.6000, 0.6400]\nAnd the model got as input:\n[ 22, 0, 73, 0]\nwhich are weird coordinates \n@matthewmasters Thanks for your help\n@mrinath I hope it'll give you some clues
Ah, that makes sense. I'm glad you figured it out
", "votes": null, "replies": [] }, { "id": 879422, "author_name": "mrinath", "author_url": "", "post_date": "06/09/2020 13:09:29", "content": "I tried to remove the (.long)\nbut an error comes up\n--> 162 lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]\n 163 rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]\n 164
\n\nRuntimeError: Expected object of scalar type double but got scalar type float for argument 'other'
", "votes": null, "replies": [] }, { "id": 879471, "author_name": "alexj21", "author_url": "", "post_date": "06/09/2020 13:55:10", "content": "I faced similar issue. You need to modify your data loader like this:\n\nIt solves three potential issues:\n- You need to use float32 format for the bbox coordinates\n- Advanced augmentations may output an image with no ground truth. In this case I fill the ground truth coordinates with [0, 1, 2, 3]\n- Do not forget to reshape to add extra dimension for the batch size
I am facing the same problem in a different contest(PANDAS) tho. I am getting a NaN validation loss for every epoch. I found that the model was outputting NaN for some images. When I saw the tensor outputs after each layer, some of the channels were NaN. What could have caused this? Any idea?
", "votes": null, "replies": [ { "id": 879192, "author_name": "alexj21", "author_url": "", "post_date": "06/09/2020 09:24:32", "content": "Did you normalize the data ?
", "votes": null, "replies": [] }, { "id": 880248, "author_name": "venky2506", "author_url": "", "post_date": "06/10/2020 06:16:11", "content": "yes
", "votes": null, "replies": [] } ] }, { "id": 943164, "author_name": "daenys2000", "author_url": "", "post_date": "07/24/2020 07:15:05", "content": "My losses are NaN right from the beginning. I tried everything in this thread, but nothing has worked yet. are there can more suggestions?\nI keep getting a \"upsample_bilinear2d_out_frame\" not implemented for 'Byte' RunTime Error until i add\nmodel = model.float()\nand\nimages = [image.float() for image in images]\nAnd after adding these two lines, the training runs and i get NaN losses\nI am a newbie to PyTorch, so i'm having a very hard time figuring out where everything is going wrong.
I figured it out, faster rcnn with resnet 50 backbone in pytorch expects coordinates in format x1,y1,x2,y2 while i was providing the coordinates in x,y,w,h format
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "878365": "Hi,\nAfter some iters/epochs the training loss becomes NaN.\nI use basic augmentations like rotate + flip. I've checked the box coordinates during the data loader:\n- no negative values\n- no values outside of image\n- if no box then fill coordinates with zero\n\nI've printed the boxes just before the first NaN loss and coordinates look OK. Min width and height are arround 12/16 pixels. I've also tried various learning rates.\n\nWhat could cause the issue ? That's frustrating because the same code works well for efficientDet !", "878534": "I am facing the same problem\nIn my case nan loss comes when I set no. of epochs high( almost 10)\nbut while small epochs like 5 doesn't show nan loss\nI am not getting why", "878571": "Only other thing I can think of is a gradient explosion, though I don't think that is the cause", "878771": "OK I've finally managed to make it work. There was a .long() (that I forgot to remove) conversion in my training loop so that, for example, a bbox in the data loader looks like:\n`[ 22.4000, 0.0000, 73.6000, 0.6400]`\nAnd the model got as input:\n`[ 22, 0, 73, 0]`\nwhich are weird coordinates \n@matthewmasters Thanks for your help\n@mrinath I hope it'll give you some clues", "878775": "Ah, that makes sense. I'm glad you figured it out", "879170": "I am facing the same problem in a different contest(PANDAS) tho. I am getting a NaN validation loss for every epoch. I found that the model was outputting NaN for some images. When I saw the tensor outputs after each layer, some of the channels were NaN. What could have caused this? Any idea?", "879192": "Did you normalize the data ?", "879422": "I tried to remove the (.long)\nbut an error comes up\n--> 162 lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]\n 163 rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]\n 164 \n\nRuntimeError: Expected object of scalar type double but got scalar type float for argument 'other'", "879471": "I faced similar issue. You need to modify your data loader like this:\n\nIt solves three potential issues:\n- You need to use float32 format for the bbox coordinates\n- Advanced augmentations may output an image with no ground truth. In this case I fill the ground truth coordinates with [0, 1, 2, 3]\n- Do not forget to reshape to add extra dimension for the batch size", "880248": "yes", "943164": "My losses are NaN right from the beginning. I tried everything in this thread, but nothing has worked yet. are there can more suggestions?\nI keep getting a \"upsample_bilinear2d_out_frame\" not implemented for 'Byte' RunTime Error until i add\n`model = model.float()`\nand\n`images = [image.float() for image in images]`\nAnd after adding these two lines, the training runs and i get NaN losses\nI am a newbie to PyTorch, so i'm having a very hard time figuring out where everything is going wrong.", "946296": "I figured it out, faster rcnn with resnet 50 backbone in pytorch expects coordinates in format x1,y1,x2,y2 while i was providing the coordinates in x,y,w,h format" }, "source": "meta" }