I'm new to machine learning and I'd like to confirm that I'm debugging in the right direction.
\nWhen training my model using SGD I get fluctuations in the prediction error, which is to be expected, but my fluctuations seem very large. For example, here are my miss-class percentages for each epoch:
\n1.0
0.34
0.71
0.10
0.10
0.75
0.58
0.87
0.96
0.41
(training is still completing...)
The fact that the error goes down to 0.34 after just one epoch and then jumps from 0.10 to 0.75, then up to 0.96 seems like a lot of useless scattering about and I would expect smaller jumps with an overall downward trend.
Is this normal behavior when using SGD? I'm wondering if this is due to using too high a learning rate, or an incorrect momentum?
", "messages": [ { "id": "62199", "postDate": "01/02/2015 02:54:40", "content": "I'm new to machine learning and I'd like to confirm that I'm debugging in the right direction.
\nWhen training my model using SGD I get fluctuations in the prediction error, which is to be expected, but my fluctuations seem very large. For example, here are my miss-class percentages for each epoch:
\n1.0
0.34
0.71
0.10
0.10
0.75
0.58
0.87
0.96
0.41
(training is still completing...)
The fact that the error goes down to 0.34 after just one epoch and then jumps from 0.10 to 0.75, then up to 0.96 seems like a lot of useless scattering about and I would expect smaller jumps with an overall downward trend.
Is this normal behavior when using SGD? I'm wondering if this is due to using too high a learning rate, or an incorrect momentum?
", "rawMarkdown": "", "votes": null }, { "id": "62218", "postDate": "01/02/2015 15:53:14", "content": "By nature SGD will have a tendency to oscillate. If the oscillation is very pronounced it might be because you use a too big learning rate.
\nDecreasing the learning rate when the algorithm runs allow to diminish the oscillation.
\n\nSource: http://cs229.stanford.edu/notes/cs229-notes1.pdf
\nNote however that it may never “converge” to the minimum, and the parameters θ will keep oscillating around the minimum of J(θ); but in practice most of the values near the minimum will be reasonably good approximations to the true minimum.2
\nWhile it is more common to run stochastic gradient descent as we have described it
and with a fixed learning rate α, by slowly letting the learning rate α decrease to zero as
the algorithm runs, it is also possible to ensure that the parameters will converge to the
global minimum rather then merely oscillate around the minimum.
[quote=jstaker7;62199]
\nFor example, here are my miss-class percentages for each epoch:
\n1.0
0.34
0.71
0.10
[/quote]
\nThese look very chaotic and also a bit too good to be true - I don't think you could really get to 0.10 misclassification by a random overshoot caused by too high a learning rate. Are you certain these are per epoch numbers? They look more like per batch numbers with a relatively small batch size. When you happen to hit a batch that is pretty easy to classify, the accuracy goes way up. Then the network sees a harder batch and it drops back down.
", "rawMarkdown": "", "votes": null }, { "id": "62327", "postDate": "01/06/2015 04:27:54", "content": "Hi James and Alexandre,
\nSorry for the delay in commenting, it took me quite a bit of time to root out the problem. Your comments were both extremely helpful in pointing me in the right direction and I ultimately solved the problem.
\nI used to consider batch size as a not very important hyper parameter, and only important for managing computational resources, but I was wrong! I now know that it can significantly impact learning.
\nThank you!
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 62218, "author_name": "alexandrekowalczyk", "author_url": "", "post_date": "01/02/2015 15:53:14", "content": "By nature SGD will have a tendency to oscillate. If the oscillation is very pronounced it might be because you use a too big learning rate.
\nDecreasing the learning rate when the algorithm runs allow to diminish the oscillation.
\n\nSource: http://cs229.stanford.edu/notes/cs229-notes1.pdf
\nNote however that it may never “converge” to the minimum, and the parameters θ will keep oscillating around the minimum of J(θ); but in practice most of the values near the minimum will be reasonably good approximations to the true minimum.2
\nWhile it is more common to run stochastic gradient descent as we have described it
and with a fixed learning rate α, by slowly letting the learning rate α decrease to zero as
the algorithm runs, it is also possible to ensure that the parameters will converge to the
global minimum rather then merely oscillate around the minimum.
[quote=jstaker7;62199]
\nFor example, here are my miss-class percentages for each epoch:
\n1.0
0.34
0.71
0.10
[/quote]
\nThese look very chaotic and also a bit too good to be true - I don't think you could really get to 0.10 misclassification by a random overshoot caused by too high a learning rate. Are you certain these are per epoch numbers? They look more like per batch numbers with a relatively small batch size. When you happen to hit a batch that is pretty easy to classify, the accuracy goes way up. Then the network sees a harder batch and it drops back down.
", "votes": null, "replies": [] }, { "id": 62327, "author_name": "jstaker7", "author_url": "", "post_date": "01/06/2015 04:27:54", "content": "Hi James and Alexandre,
\nSorry for the delay in commenting, it took me quite a bit of time to root out the problem. Your comments were both extremely helpful in pointing me in the right direction and I ultimately solved the problem.
\nI used to consider batch size as a not very important hyper parameter, and only important for managing computational resources, but I was wrong! I now know that it can significantly impact learning.
\nThank you!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "62199": "", "62218": "", "62219": "", "62327": "" }, "source": "meta" }