I tried to use sklearn for my baseline model. \nMore specifically I used Stochastic Gradient Descent (SGDRegressor) because of the size of the dataset but the results are far worse from my naive predictor.
\n\nHas anyone tried the specific algorithm or any other from sklearn which scales good with big datasets?
", "messages": [ { "id": "395286", "postDate": "09/28/2018 08:38:07", "content": "I tried to use sklearn for my baseline model. \nMore specifically I used Stochastic Gradient Descent (SGDRegressor) because of the size of the dataset but the results are far worse from my naive predictor.
\n\nHas anyone tried the specific algorithm or any other from sklearn which scales good with big datasets?
", "rawMarkdown": "I tried to use sklearn for my baseline model. \nMore specifically I used Stochastic Gradient Descent (SGDRegressor) because of the size of the dataset but the results are far worse from my naive predictor.\n\nHas anyone tried the specific algorithm or any other from sklearn which scales good with big datasets?", "votes": null }, { "id": "395355", "postDate": "09/28/2018 12:23:10", "content": "Hello Yannis !
\n\nThere are tons of algorithms which you can use on Sklearn (you have a list right there : http://scikit-learn.org/stable/supervised_learning.html).
\n\nSome algorithms such as LightGBM and Catboost scales very good on big datasets.. You can use lightGBM API (example here : https://github.com/Microsoft/LightGBM/blob/master/examples/python-guide/sklearn_example.py) to use it with the sklearn pipeline, which is not the case with Catboost but its speed is impressive.
", "rawMarkdown": "Hello Yannis ! \n\nThere are tons of algorithms which you can use on Sklearn (you have a list right there : http://scikit-learn.org/stable/supervised_learning.html).\n\nSome algorithms such as LightGBM and Catboost scales very good on big datasets.. You can use lightGBM API (example here : https://github.com/Microsoft/LightGBM/blob/master/examples/python-guide/sklearn_example.py) to use it with the sklearn pipeline, which is not the case with Catboost but its speed is impressive.", "votes": null }, { "id": "395360", "postDate": "09/28/2018 12:31:12", "content": "Thanks for your reply.
\n\nI know the available algorithms in sklearn. The problem is that because they process the dataset in one batch (except SGD) it is practically impossible to run anything bigger than 100,000 rows. \nAlso, since I just want a baseline score to track my progress, I was asking if someone has a hint about any sklearn implementation (not lightGBM etc.) butsure I'll give lightGBM a try later.
", "rawMarkdown": "Thanks for your reply.\n\nI know the available algorithms in sklearn. The problem is that because they process the dataset in one batch (except SGD) it is practically impossible to run anything bigger than 100,000 rows. \nAlso, since I just want a baseline score to track my progress, I was asking if someone has a hint about any sklearn implementation (not lightGBM etc.) butsure I'll give lightGBM a try later.", "votes": null }, { "id": "395364", "postDate": "09/28/2018 12:47:32", "content": "Hi Yannis,
\n\nFor benchmark purposes, you may use Linear regression and Random Forest in sklearn, they scale quite well, and you can control used memory or execution time with hyper parameters. If you use Ridge or Lasso for Linear regression you can set number of iterations (max_iter : int, optional) for learning and directly control execution time. Additionally, you should set normalize = True, since normalization speeds up convergence time.
\n\nIn case of Random forest try with small number of trees (n_estimators), small number of features for building trees (max_features) and some stopping criteria that controls tree depth (max_depth or min_samples_split)
\n\nAdditionally, if you are using cross validation make sure that your n_jobs parameter is set to number of your CPU cores (and you may set number of folds to the number of cores, so you wan't have unnecessary waiting). Even though algorithms are not learned in parallel, parallelization of folds will significantly save your execution time.
", "rawMarkdown": "Hi Yannis,\n\nFor benchmark purposes, you may use Linear regression and Random Forest in sklearn, they scale quite well, and you can control used memory or execution time with hyper parameters. If you use Ridge or Lasso for Linear regression you can set number of iterations (max_iter : int, optional) for learning and directly control execution time. Additionally, you should set normalize = True, since normalization speeds up convergence time.\n\nIn case of Random forest try with small number of trees (n_estimators), small number of features for building trees (max_features) and some stopping criteria that controls tree depth (max_depth or min_samples_split)\n\nAdditionally, if you are using cross validation make sure that your n_jobs parameter is set to number of your CPU cores (and you may set number of folds to the number of cores, so you wan't have unnecessary waiting). Even though algorithms are not learned in parallel, parallelization of folds will significantly save your execution time.", "votes": null }, { "id": "395368", "postDate": "09/28/2018 12:51:40", "content": "Thanks, @vucko83. I'll try your suggestions.
", "rawMarkdown": "Thanks, @vucko83. I'll try your suggestions.", "votes": null }, { "id": "395411", "postDate": "09/28/2018 14:27:25", "content": "Yannis, can you please explain why lightgbm is not appropriate for you? Is it slow? Or you cannot load all data into the memory?
", "rawMarkdown": "Yannis, can you please explain why lightgbm is not appropriate for you? Is it slow? Or you cannot load all data into the memory?", "votes": null }, { "id": "395442", "postDate": "09/28/2018 15:29:46", "content": "Lightgbm actually has a RF implementation as well and I assume it is faster than the implementation in sklearn
", "rawMarkdown": "Lightgbm actually has a RF implementation as well and I assume it is faster than the implementation in sklearn", "votes": null }, { "id": "395464", "postDate": "09/28/2018 16:20:12", "content": "ExtraTrees from sklearn fit in kernel memory without problems, but lightgbm RF or GBM is faster
", "rawMarkdown": "ExtraTrees from sklearn fit in kernel memory without problems, but lightgbm RF or GBM is faster", "votes": null }, { "id": "395900", "postDate": "09/29/2018 14:56:00", "content": "Hey, Alexander.\nIt is not that lightGBM is not appropriate for me. It is just that I have not used it again.\nI know that it is very popular in Kaggle, but I was trying to start with an algorithm I'm more familiar to understand better the challenges of the dataset. I'll study about the algorithm tomorrow, and I'll try it.\nIf anyone has a great resource about it, please feel free to send.
", "rawMarkdown": "Hey, Alexander.\nIt is not that lightGBM is not appropriate for me. It is just that I have not used it again.\nI know that it is very popular in Kaggle, but I was trying to start with an algorithm I'm more familiar to understand better the challenges of the dataset. I'll study about the algorithm tomorrow, and I'll try it.\nIf anyone has a great resource about it, please feel free to send.", "votes": null }, { "id": "395944", "postDate": "09/29/2018 16:27:48", "content": "Never mind, I got it. 😉
", "rawMarkdown": "Never mind, I got it. 😉", "votes": null }, { "id": "395945", "postDate": "09/29/2018 16:28:43", "content": "Thanks, Misha!
", "rawMarkdown": "Thanks, Misha!", "votes": null } ], "comments": [ { "id": 395355, "author_name": "xtelima", "author_url": "", "post_date": "09/28/2018 12:23:10", "content": "Hello Yannis !
\n\nThere are tons of algorithms which you can use on Sklearn (you have a list right there : http://scikit-learn.org/stable/supervised_learning.html).
\n\nSome algorithms such as LightGBM and Catboost scales very good on big datasets.. You can use lightGBM API (example here : https://github.com/Microsoft/LightGBM/blob/master/examples/python-guide/sklearn_example.py) to use it with the sklearn pipeline, which is not the case with Catboost but its speed is impressive.
", "votes": null, "replies": [ { "id": 395360, "author_name": "yannisp", "author_url": "", "post_date": "09/28/2018 12:31:12", "content": "Thanks for your reply.
\n\nI know the available algorithms in sklearn. The problem is that because they process the dataset in one batch (except SGD) it is practically impossible to run anything bigger than 100,000 rows. \nAlso, since I just want a baseline score to track my progress, I was asking if someone has a hint about any sklearn implementation (not lightGBM etc.) butsure I'll give lightGBM a try later.
", "votes": null, "replies": [] } ] }, { "id": 395364, "author_name": "vucko83", "author_url": "", "post_date": "09/28/2018 12:47:32", "content": "Hi Yannis,
\n\nFor benchmark purposes, you may use Linear regression and Random Forest in sklearn, they scale quite well, and you can control used memory or execution time with hyper parameters. If you use Ridge or Lasso for Linear regression you can set number of iterations (max_iter : int, optional) for learning and directly control execution time. Additionally, you should set normalize = True, since normalization speeds up convergence time.
\n\nIn case of Random forest try with small number of trees (n_estimators), small number of features for building trees (max_features) and some stopping criteria that controls tree depth (max_depth or min_samples_split)
\n\nAdditionally, if you are using cross validation make sure that your n_jobs parameter is set to number of your CPU cores (and you may set number of folds to the number of cores, so you wan't have unnecessary waiting). Even though algorithms are not learned in parallel, parallelization of folds will significantly save your execution time.
", "votes": null, "replies": [ { "id": 395368, "author_name": "yannisp", "author_url": "", "post_date": "09/28/2018 12:51:40", "content": "Thanks, @vucko83. I'll try your suggestions.
", "votes": null, "replies": [] }, { "id": 395442, "author_name": "mlisovyi", "author_url": "", "post_date": "09/28/2018 15:29:46", "content": "Lightgbm actually has a RF implementation as well and I assume it is faster than the implementation in sklearn
", "votes": null, "replies": [] } ] }, { "id": 395411, "author_name": "alexfir", "author_url": "", "post_date": "09/28/2018 14:27:25", "content": "Yannis, can you please explain why lightgbm is not appropriate for you? Is it slow? Or you cannot load all data into the memory?
", "votes": null, "replies": [ { "id": 395900, "author_name": "yannisp", "author_url": "", "post_date": "09/29/2018 14:56:00", "content": "Hey, Alexander.\nIt is not that lightGBM is not appropriate for me. It is just that I have not used it again.\nI know that it is very popular in Kaggle, but I was trying to start with an algorithm I'm more familiar to understand better the challenges of the dataset. I'll study about the algorithm tomorrow, and I'll try it.\nIf anyone has a great resource about it, please feel free to send.
", "votes": null, "replies": [] }, { "id": 395944, "author_name": "yannisp", "author_url": "", "post_date": "09/29/2018 16:27:48", "content": "Never mind, I got it. 😉
", "votes": null, "replies": [] } ] }, { "id": 395464, "author_name": "mlisovyi", "author_url": "", "post_date": "09/28/2018 16:20:12", "content": "ExtraTrees from sklearn fit in kernel memory without problems, but lightgbm RF or GBM is faster
", "votes": null, "replies": [ { "id": 395945, "author_name": "yannisp", "author_url": "", "post_date": "09/29/2018 16:28:43", "content": "Thanks, Misha!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "395286": "I tried to use sklearn for my baseline model. \nMore specifically I used Stochastic Gradient Descent (SGDRegressor) because of the size of the dataset but the results are far worse from my naive predictor.\n\nHas anyone tried the specific algorithm or any other from sklearn which scales good with big datasets?", "395355": "Hello Yannis ! \n\nThere are tons of algorithms which you can use on Sklearn (you have a list right there : http://scikit-learn.org/stable/supervised_learning.html).\n\nSome algorithms such as LightGBM and Catboost scales very good on big datasets.. You can use lightGBM API (example here : https://github.com/Microsoft/LightGBM/blob/master/examples/python-guide/sklearn_example.py) to use it with the sklearn pipeline, which is not the case with Catboost but its speed is impressive.", "395360": "Thanks for your reply.\n\nI know the available algorithms in sklearn. The problem is that because they process the dataset in one batch (except SGD) it is practically impossible to run anything bigger than 100,000 rows. \nAlso, since I just want a baseline score to track my progress, I was asking if someone has a hint about any sklearn implementation (not lightGBM etc.) butsure I'll give lightGBM a try later.", "395364": "Hi Yannis,\n\nFor benchmark purposes, you may use Linear regression and Random Forest in sklearn, they scale quite well, and you can control used memory or execution time with hyper parameters. If you use Ridge or Lasso for Linear regression you can set number of iterations (max_iter : int, optional) for learning and directly control execution time. Additionally, you should set normalize = True, since normalization speeds up convergence time.\n\nIn case of Random forest try with small number of trees (n_estimators), small number of features for building trees (max_features) and some stopping criteria that controls tree depth (max_depth or min_samples_split)\n\nAdditionally, if you are using cross validation make sure that your n_jobs parameter is set to number of your CPU cores (and you may set number of folds to the number of cores, so you wan't have unnecessary waiting). Even though algorithms are not learned in parallel, parallelization of folds will significantly save your execution time.", "395368": "Thanks, @vucko83. I'll try your suggestions.", "395411": "Yannis, can you please explain why lightgbm is not appropriate for you? Is it slow? Or you cannot load all data into the memory?", "395442": "Lightgbm actually has a RF implementation as well and I assume it is faster than the implementation in sklearn", "395464": "ExtraTrees from sklearn fit in kernel memory without problems, but lightgbm RF or GBM is faster", "395900": "Hey, Alexander.\nIt is not that lightGBM is not appropriate for me. It is just that I have not used it again.\nI know that it is very popular in Kaggle, but I was trying to start with an algorithm I'm more familiar to understand better the challenges of the dataset. I'll study about the algorithm tomorrow, and I'll try it.\nIf anyone has a great resource about it, please feel free to send.", "395944": "Never mind, I got it. 😉", "395945": "Thanks, Misha!" }, "source": "meta" }