I'm having some problems with the 'regression head' in Pytorch with MAE not converging below about 2-3 even when I feed the scalar coupling itself as an input feature.
\n\nI can get convergence if I do things like feed the coupling constant x 5 as X and predict scalar coupling constant and decay the learning rate to 1e-5.
\n\nNot got a huge amount of experience with Pytorch so any tips are good tips. Have been using a couple of nn.Linear layers, L1_loss. Have tried a few different optimizers.
", "messages": [ { "id": 589152, "postDate": "2019-07-31T14:31:34.753Z", "content": "mae is not a very good loss function. See an old discussion in this forum where @titericz disclosed that using mse was way better for lgb than mae. The same is most probably true for NNs. Other loss that may be interesting is huber loss.
", "rawMarkdown": "mae is not a very good loss function. See an old discussion in this forum where @titericz disclosed that using mse was way better for lgb than mae. The same is most probably true for NNs. Other loss that may be interesting is huber loss. ", "votes": 4 }, { "id": 589226, "postDate": "2019-07-31T16:36:36.253Z", "content": "Simple example: try minimizing |x|, step size 1, initial guess 0.5. Following the negative gradient's direction (which is -1 in this simple case), you get to -0.5, applying gradient descent again, you are back to 0.5...now it is bouncing back and forth near the minimum.
\n\nMath behind: either your gradient or your step size should decay well enough for convergence. The best choice for a convex smooth loss would be the inverse of the maximum eigenvalue of the Hessian matrix at the current iteration. However, MAE loss does not have a smooth 2nd derivative...
\n\nConclusion: use a smooth loss function, set the CV metrics to be MAE.
", "rawMarkdown": "Simple example: try minimizing |x|, step size 1, initial guess 0.5. Following the negative gradient's direction (which is -1 in this simple case), you get to -0.5, applying gradient descent again, you are back to 0.5...now it is bouncing back and forth near the minimum.\n\nMath behind: either your gradient or your step size should decay well enough for convergence. The best choice for a convex smooth loss would be the inverse of the maximum eigenvalue of the Hessian matrix at the current iteration. However, MAE loss does not have a smooth 2nd derivative...\n\nConclusion: use a smooth loss function, set the CV metrics to be MAE.", "votes": 1 }, { "id": 589147, "postDate": "2019-07-31T14:24:17.450Z", "content": "I'm having some problems with the 'regression head' in Pytorch with MAE not converging below about 2-3 even when I feed the scalar coupling itself as an input feature.
\n\nI can get convergence if I do things like feed the coupling constant x 5 as X and predict scalar coupling constant and decay the learning rate to 1e-5.
\n\nNot got a huge amount of experience with Pytorch so any tips are good tips. Have been using a couple of nn.Linear layers, L1_loss. Have tried a few different optimizers.
", "rawMarkdown": "I'm having some problems with the 'regression head' in Pytorch with MAE not converging below about 2-3 even when I feed the scalar coupling itself as an input feature.\n\nI can get convergence if I do things like feed the coupling constant x 5 as X and predict scalar coupling constant and decay the learning rate to 1e-5.\n\nNot got a huge amount of experience with Pytorch so any tips are good tips. Have been using a couple of nn.Linear layers, L1_loss. Have tried a few different optimizers. " }, { "id": 589186, "postDate": "2019-07-31T15:34:41.200Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 589152, "author_name": "CPMP", "author_url": "", "post_date": "2019-07-31T14:31:34.753000", "content": "mae is not a very good loss function. See an old discussion in this forum where @titericz disclosed that using mse was way better for lgb than mae. The same is most probably true for NNs. Other loss that may be interesting is huber loss.
", "votes": 4, "replies": [] }, { "id": 589226, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2019-07-31T16:36:36.253000", "content": "Simple example: try minimizing |x|, step size 1, initial guess 0.5. Following the negative gradient's direction (which is -1 in this simple case), you get to -0.5, applying gradient descent again, you are back to 0.5...now it is bouncing back and forth near the minimum.
\n\nMath behind: either your gradient or your step size should decay well enough for convergence. The best choice for a convex smooth loss would be the inverse of the maximum eigenvalue of the Hessian matrix at the current iteration. However, MAE loss does not have a smooth 2nd derivative...
\n\nConclusion: use a smooth loss function, set the CV metrics to be MAE.
", "votes": 1, "replies": [] }, { "id": 589186, "author_name": "", "author_url": "", "post_date": "2019-07-31T15:34:41.200000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "589152": "mae is not a very good loss function. See an old discussion in this forum where @titericz disclosed that using mse was way better for lgb than mae. The same is most probably true for NNs. Other loss that may be interesting is huber loss. ", "589226": "Simple example: try minimizing |x|, step size 1, initial guess 0.5. Following the negative gradient's direction (which is -1 in this simple case), you get to -0.5, applying gradient descent again, you are back to 0.5...now it is bouncing back and forth near the minimum.\n\nMath behind: either your gradient or your step size should decay well enough for convergence. The best choice for a convex smooth loss would be the inverse of the maximum eigenvalue of the Hessian matrix at the current iteration. However, MAE loss does not have a smooth 2nd derivative...\n\nConclusion: use a smooth loss function, set the CV metrics to be MAE.", "589147": "I'm having some problems with the 'regression head' in Pytorch with MAE not converging below about 2-3 even when I feed the scalar coupling itself as an input feature.\n\nI can get convergence if I do things like feed the coupling constant x 5 as X and predict scalar coupling constant and decay the learning rate to 1e-5.\n\nNot got a huge amount of experience with Pytorch so any tips are good tips. Have been using a couple of nn.Linear layers, L1_loss. Have tried a few different optimizers. ", "589186": "" } }