Shout out to all the great and helpful kernels,\nI have tried to run several of them but when I set epoch no. a little high(7,8)\nloss turns to nan, \nAnyone solved??
", "messages": [ { "id": "870610", "postDate": "06/01/2020 19:06:26", "content": "Shout out to all the great and helpful kernels,\nI have tried to run several of them but when I set epoch no. a little high(7,8)\nloss turns to nan, \nAnyone solved??
", "rawMarkdown": "Shout out to all the great and helpful kernels,\nI have tried to run several of them but when I set epoch no. a little high(7,8)\nloss turns to nan, \nAnyone solved??", "votes": null }, { "id": "870635", "postDate": "06/01/2020 19:14:50", "content": "Could be that during one of these epochs your dataloader produces an example without boxes
", "rawMarkdown": "Could be that during one of these epochs your dataloader produces an example without boxes", "votes": null }, { "id": "870646", "postDate": "06/01/2020 19:24:22", "content": "What part of the dataloader does that?
", "rawMarkdown": "What part of the dataloader does that?", "votes": null }, { "id": "870650", "postDate": "06/01/2020 19:28:34", "content": "It would likely be some augmentation you have, like a random crop or shift. You can check to see if theres any boxes after the augmentation has been applied. If not, you can try another random augmentation.
", "rawMarkdown": "It would likely be some augmentation you have, like a random crop or shift. You can check to see if theres any boxes after the augmentation has been applied. If not, you can try another random augmentation.", "votes": null }, { "id": "877665", "postDate": "06/07/2020 20:32:26", "content": "I have tried it,But still the same, any more suggestions??
", "rawMarkdown": "I have tried it,But still the same, any more suggestions??", "votes": null }, { "id": "877728", "postDate": "06/07/2020 22:21:42", "content": "I think the likely issue is that you provide a NaN in the input at some point. The best way to determine what is causing it is to debug and put a breakpoint when a NaN is encountered. \nIt could also be an \"exploding gradient\" problem but I think that's less likely.
", "rawMarkdown": "I think the likely issue is that you provide a NaN in the input at some point. The best way to determine what is causing it is to debug and put a breakpoint when a NaN is encountered. \nIt could also be an \"exploding gradient\" problem but I think that's less likely.", "votes": null }, { "id": "888626", "postDate": "06/16/2020 13:24:32", "content": "I had an equal problem, check your bounding boxes. My coordinates were wrong, adjusting them fixed the problem for me and i got right losses calculated.
", "rawMarkdown": "I had an equal problem, check your bounding boxes. My coordinates were wrong, adjusting them fixed the problem for me and i got right losses calculated.", "votes": null } ], "comments": [ { "id": 870635, "author_name": "matthewmasters", "author_url": "", "post_date": "06/01/2020 19:14:50", "content": "Could be that during one of these epochs your dataloader produces an example without boxes
", "votes": null, "replies": [ { "id": 870646, "author_name": "mrinath", "author_url": "", "post_date": "06/01/2020 19:24:22", "content": "What part of the dataloader does that?
", "votes": null, "replies": [] }, { "id": 870650, "author_name": "matthewmasters", "author_url": "", "post_date": "06/01/2020 19:28:34", "content": "It would likely be some augmentation you have, like a random crop or shift. You can check to see if theres any boxes after the augmentation has been applied. If not, you can try another random augmentation.
", "votes": null, "replies": [] }, { "id": 877665, "author_name": "mrinath", "author_url": "", "post_date": "06/07/2020 20:32:26", "content": "I have tried it,But still the same, any more suggestions??
", "votes": null, "replies": [] }, { "id": 877728, "author_name": "matthewmasters", "author_url": "", "post_date": "06/07/2020 22:21:42", "content": "I think the likely issue is that you provide a NaN in the input at some point. The best way to determine what is causing it is to debug and put a breakpoint when a NaN is encountered. \nIt could also be an \"exploding gradient\" problem but I think that's less likely.
", "votes": null, "replies": [] } ] }, { "id": 888626, "author_name": "aliabdin1", "author_url": "", "post_date": "06/16/2020 13:24:32", "content": "I had an equal problem, check your bounding boxes. My coordinates were wrong, adjusting them fixed the problem for me and i got right losses calculated.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "870610": "Shout out to all the great and helpful kernels,\nI have tried to run several of them but when I set epoch no. a little high(7,8)\nloss turns to nan, \nAnyone solved??", "870635": "Could be that during one of these epochs your dataloader produces an example without boxes", "870646": "What part of the dataloader does that?", "870650": "It would likely be some augmentation you have, like a random crop or shift. You can check to see if theres any boxes after the augmentation has been applied. If not, you can try another random augmentation.", "877665": "I have tried it,But still the same, any more suggestions??", "877728": "I think the likely issue is that you provide a NaN in the input at some point. The best way to determine what is causing it is to debug and put a breakpoint when a NaN is encountered. \nIt could also be an \"exploding gradient\" problem but I think that's less likely.", "888626": "I had an equal problem, check your bounding boxes. My coordinates were wrong, adjusting them fixed the problem for me and i got right losses calculated." }, "source": "meta" }