New to Keras/TensorFlow(and NN in general) and ask a basic question here
\n\nAre all the algorithms(SGD, Adam, Adagrad, etc.) to solve the parameter values used by Keras/TensorFlow just variants of Stochastic Gradient Descent? https://keras.io/optimizers/
\n\nLook like scklearn uses algorithms(such as Newton) other than SGD to solve Logistic Regression. http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
\n\nSo wonder if NN in real practice/implementation also can use other algo to solve the parameter values.
", "messages": [ { "id": "161807", "postDate": "02/15/2017 22:53:11", "content": "New to Keras/TensorFlow(and NN in general) and ask a basic question here
\n\nAre all the algorithms(SGD, Adam, Adagrad, etc.) to solve the parameter values used by Keras/TensorFlow just variants of Stochastic Gradient Descent? https://keras.io/optimizers/
\n\nLook like scklearn uses algorithms(such as Newton) other than SGD to solve Logistic Regression. http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
\n\nSo wonder if NN in real practice/implementation also can use other algo to solve the parameter values.
", "rawMarkdown": "New to Keras/TensorFlow(and NN in general) and ask a basic question here\n\nAre all the algorithms(SGD, Adam, Adagrad, etc.) to solve the parameter values used by Keras/TensorFlow just variants of Stochastic Gradient Descent? https://keras.io/optimizers/\n\nLook like scklearn uses algorithms(such as Newton) other than SGD to solve Logistic Regression. http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n\nSo wonder if NN in real practice/implementation also can use other algo to solve the parameter values.", "votes": null }, { "id": "161812", "postDate": "02/15/2017 23:31:58", "content": "Yes, they are variations of gradient descent. People have experimented with other techniques (eg gradient free and second order methods) but they aren't used nearly as much.
", "rawMarkdown": "Yes, they are variations of gradient descent. People have experimented with other techniques (eg gradient free and second order methods) but they aren't used nearly as much.", "votes": null }, { "id": "162243", "postDate": "02/17/2017 20:24:48", "content": "I often refer to the following site, it shows how these different algorithms work:\nhttp://sebastianruder.com/optimizing-gradient-descent/
", "rawMarkdown": "I often refer to the following site, it shows how these different algorithms work:\nhttp://sebastianruder.com/optimizing-gradient-descent/", "votes": null }, { "id": "162699", "postDate": "02/20/2017 19:03:58", "content": "It is possible to use higher order methods, see http://www.deeplearningbook.org/contents/optimization.html specifically section 8.6 (for the actual implementation of the methods) and section 8.2 also has some discussion on the theory behind these methods.
", "rawMarkdown": "It is possible to use higher order methods, see [http://www.deeplearningbook.org/contents/optimization.html][1] specifically section 8.6 (for the actual implementation of the methods) and section 8.2 also has some discussion on the theory behind these methods.\n\n\n [1]: http://www.deeplearningbook.org/contents/optimization.html", "votes": null } ], "comments": [ { "id": 161812, "author_name": "tylerwilson1123", "author_url": "", "post_date": "02/15/2017 23:31:58", "content": "Yes, they are variations of gradient descent. People have experimented with other techniques (eg gradient free and second order methods) but they aren't used nearly as much.
", "votes": null, "replies": [] }, { "id": 162243, "author_name": "stmaxx", "author_url": "", "post_date": "02/17/2017 20:24:48", "content": "I often refer to the following site, it shows how these different algorithms work:\nhttp://sebastianruder.com/optimizing-gradient-descent/
", "votes": null, "replies": [] }, { "id": 162699, "author_name": "meissnereffect", "author_url": "", "post_date": "02/20/2017 19:03:58", "content": "It is possible to use higher order methods, see http://www.deeplearningbook.org/contents/optimization.html specifically section 8.6 (for the actual implementation of the methods) and section 8.2 also has some discussion on the theory behind these methods.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "161807": "New to Keras/TensorFlow(and NN in general) and ask a basic question here\n\nAre all the algorithms(SGD, Adam, Adagrad, etc.) to solve the parameter values used by Keras/TensorFlow just variants of Stochastic Gradient Descent? https://keras.io/optimizers/\n\nLook like scklearn uses algorithms(such as Newton) other than SGD to solve Logistic Regression. http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n\nSo wonder if NN in real practice/implementation also can use other algo to solve the parameter values.", "161812": "Yes, they are variations of gradient descent. People have experimented with other techniques (eg gradient free and second order methods) but they aren't used nearly as much.", "162243": "I often refer to the following site, it shows how these different algorithms work:\nhttp://sebastianruder.com/optimizing-gradient-descent/", "162699": "It is possible to use higher order methods, see [http://www.deeplearningbook.org/contents/optimization.html][1] specifically section 8.6 (for the actual implementation of the methods) and section 8.2 also has some discussion on the theory behind these methods.\n\n\n [1]: http://www.deeplearningbook.org/contents/optimization.html" }, "source": "meta" }