{ "id": 5806, "title": "Practical significance of results", "url": "/competitions/ams-2014-solar-energy-prediction-contest/discussion/5806", "author_name": "", "post_date": "2013-09-19T08:33:05.807Z", "votes": null, "comment_count": 16, "views": 4430, "content": "

Hi, 

\n

I'm curious about the practical significance of the results in this competition so far. In particular, does an error reduction of 100,000 MSE matter in practice? 

\n

thanks, 

\n

 Peter

", "messages": [ { "id": "31161", "postDate": "09/19/2013 08:33:05", "content": "

Hi, 

\n

I'm curious about the practical significance of the results in this competition so far. In particular, does an error reduction of 100,000 MSE matter in practice? 

\n

thanks, 

\n

 Peter

", "rawMarkdown": "", "votes": null }, { "id": "31258", "postDate": "09/20/2013 15:37:57", "content": "

Hi Peter.  I have another question.

\n

I see in scikit-learn manual that you are the author of sklearn.ensemble package.  The computation time for gradient boosting regression is very long (>30 hours) on core-2 Intel uP with 4 GB RAM,  for say  1000 regression trees of depth 10.

\n

Do you get less computation time ?

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31260", "postDate": "09/20/2013 16:44:05", "content": "

Hey, can you create a stackoverflow question or github issue and send me the link - i dont think its the right place here and i dont want to hijack the thread

", "rawMarkdown": "", "votes": null }, { "id": "31285", "postDate": "09/20/2013 23:59:19", "content": "

@Peter

\n

If you dont mind i would like to know if you are using your GradientBoostedRegressor for this task, and if takes much ram on your computer. I don't think its hijacking the forum if its is competition related. Except that this is not the subject (title of this thread) :P

\n

I'm curious to compare python perfomance with R. 

", "rawMarkdown": "", "votes": null }, { "id": "31293", "postDate": "09/21/2013 04:31:20", "content": "

@Peter Since the amount of energy produced from solar and wind farms depends on the weather conditions, forecasts of the expected solar and wind energy are needed to know how much power needs to be purchased from more stable sources (e.g., coal, oil, natural gas) in order to meet demand. Decreasing the forecast error means that more power can be purchased in advance when it is cheaper and that excess production is not wasted. Even small decreases in forecast error (1 to 10% MAE) can have fairly large economic impacts. So far there has been a ~25% improvement in MAE of the course of the contest. If this contest was predicting energy for an actual solar farm, the predictive improvements could have produced a multimillion dollar economic impact.

", "rawMarkdown": "", "votes": null }, { "id": "31295", "postDate": "09/21/2013 07:21:17", "content": "

@StormMiner: Thanks for the detailed information - much appreciated!

\n

 

\n

disclaimer: I'm not the author of sklearn.ensemble - I merely created the gradient boosting implementation - much of the work in ensemble has been contributed by another fellow Kaggler: Gilles Louppe [2] .

\n

@others: I've created a github issue to track performance related issues in GradientBoostingRegressor - lets move the discussion there, its more convenient and other sklearn devs can comment. Here is the link: https://github.com/scikit-learn/scikit-learn/issues/2466  . Please get involved in the discussion and add your findings / concerns.

\n

 

\n

Regarding GradientBoostingRegressor: there is a performance regression (for certain hyper-parameter settings) in the latest release (0.14) - we switched our tree implementation which is now significantly faster for Random Forest but slower when building shallow trees (and no random split point selection). You might want to switch to the 0.13.1 release until it is fixed.

\n

 

\n

Regarding the difference between R's gbm and sklearn GradientBoosting implementation: the last throughout comparison I did was based on the old tree engine (0.13.1 release) - the results are summarized in another kaggle post [1] . The key point here is that gbm and sklearn learn different trees - sklearn grows complete trees of max_depth whereas gbm only grows a single branch of max_depth. The later is much faster but the former gives slightly better performance in my experience. At one point I'd like to add both strategies to sklearn's implementation.

\n

 

\n

Another performance tipp: optimize the ``max_features`` hyper-parameter - it will make training much faster and is also great for variance control.

\n

 

\n

@lamb: I hardly grow trees deeper than max_depth 7 - you should look into deviance plots whether you are overfitting - if so, try to tune ``max_features`` (learning rate tuning is not so good because it will further increase the runtime of the algorithm).

\n

 

\n

[1] http://www.kaggle.com/c/online-sales/forums/t/2135/congrats-to-the-winners/12130#post12130

\n

[2] http://www.kaggle.com/users/1955/gilles-louppe

", "rawMarkdown": "", "votes": null }, { "id": "31305", "postDate": "09/21/2013 14:06:26", "content": "

@Peter, thanks!

\n

Are you using GradientBoostedRegression for this task?

\n

Now i do have some GBRT questions to ask, unrelated to this competition, whats is the proper channel for me to ask them? I looked at the help, and didnt found what i seek, maybe it isn't implemented, but i need to ask anyway ...

", "rawMarkdown": "", "votes": null }, { "id": "31306", "postDate": "09/21/2013 14:23:41", "content": "

@leustagos: yes, I do - I tried RandomForest too and it basically gives me the same result.

\n

you can write me an email to firstname.lastname@gmail.com - or for more sklearn related question just write an email to the sklearn mailing list or open a github issue (for problems/enhancement proposals)

", "rawMarkdown": "", "votes": null }, { "id": "31865", "postDate": "10/01/2013 15:13:28", "content": "

@Leustagos,

\n

 

\n

Hello.  Did you perhaps try:

\n

 

\n

model=RandomForestRegressor(n_estimators=500,max_depth=5,n_jobs=2)

\n

for:   training data shape (5113,2160) (5113,98).

\n

It takes very long time on 2-core uP.

", "rawMarkdown": "", "votes": null }, { "id": "31866", "postDate": "10/01/2013 15:21:35", "content": "

If you don't set max_features then RandomForestRegressor  will search over all of them to find the best split [1]. Setting ``max_features`` to some small number eg 0.1 (10 % of the features) will make it run considerably faster.

\n

 

\n

[1] http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor

", "rawMarkdown": "", "votes": null }, { "id": "31867", "postDate": "10/01/2013 15:28:45", "content": "

@Peter,

\n

Thanks for the tip.   That's it.

", "rawMarkdown": "", "votes": null }, { "id": "31868", "postDate": "10/01/2013 15:29:29", "content": "

Since you asked me, i wouldnt use rf for this task. RF implementations minimizes MSE, while this competition error metric is MAE.

\n

And Petter advice also applies to GradientBoostedRegressor, that does minimize MAD with the lad loss.

", "rawMarkdown": "", "votes": null }, { "id": "31869", "postDate": "10/01/2013 15:37:03", "content": "

@Leustagos,

\n

Thanks.  An error comes up when GradientBoostedRegressor is applied on training data shape (5113,2160) (5113,98).

", "rawMarkdown": "", "votes": null }, { "id": "31871", "postDate": "10/01/2013 15:45:29", "content": "

GradientBoostingRegressor does not support multiple outputs (y.shape[1] > 1)

", "rawMarkdown": "", "votes": null }, { "id": "31876", "postDate": "10/01/2013 17:24:24", "content": "

Isnt this the same as doing one model per output?

", "rawMarkdown": "", "votes": null }, { "id": "31882", "postDate": "10/01/2013 18:48:17", "content": "

A multi-output tree computes split points by measuring impurity for each output - for each split it chooses the point that has min mean impurity - so its different from training one tree per output. The model can capture correlation between outputs.

", "rawMarkdown": "", "votes": null }, { "id": "31883", "postDate": "10/01/2013 18:54:41", "content": "

That was too quick. I agree with Lucas, if you want to use GradientBoostingRegressor for multi-output problems you need to train n_output models. The outputs will be predicted in isolation - no correlation can be exploited. This is different from RandomForestRegressor which uses multi-output trees.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 31258, "author_name": "ijaganjac", "author_url": "", "post_date": "09/20/2013 15:37:57", "content": "

Hi Peter.  I have another question.

\n

I see in scikit-learn manual that you are the author of sklearn.ensemble package.  The computation time for gradient boosting regression is very long (>30 hours) on core-2 Intel uP with 4 GB RAM,  for say  1000 regression trees of depth 10.

\n

Do you get less computation time ?

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 31260, "author_name": "pprett", "author_url": "", "post_date": "09/20/2013 16:44:05", "content": "

Hey, can you create a stackoverflow question or github issue and send me the link - i dont think its the right place here and i dont want to hijack the thread

", "votes": null, "replies": [] }, { "id": 31285, "author_name": "leustagos", "author_url": "", "post_date": "09/20/2013 23:59:19", "content": "

@Peter

\n

If you dont mind i would like to know if you are using your GradientBoostedRegressor for this task, and if takes much ram on your computer. I don't think its hijacking the forum if its is competition related. Except that this is not the subject (title of this thread) :P

\n

I'm curious to compare python perfomance with R. 

", "votes": null, "replies": [] }, { "id": 31293, "author_name": "djgagne", "author_url": "", "post_date": "09/21/2013 04:31:20", "content": "

@Peter Since the amount of energy produced from solar and wind farms depends on the weather conditions, forecasts of the expected solar and wind energy are needed to know how much power needs to be purchased from more stable sources (e.g., coal, oil, natural gas) in order to meet demand. Decreasing the forecast error means that more power can be purchased in advance when it is cheaper and that excess production is not wasted. Even small decreases in forecast error (1 to 10% MAE) can have fairly large economic impacts. So far there has been a ~25% improvement in MAE of the course of the contest. If this contest was predicting energy for an actual solar farm, the predictive improvements could have produced a multimillion dollar economic impact.

", "votes": null, "replies": [] }, { "id": 31295, "author_name": "pprett", "author_url": "", "post_date": "09/21/2013 07:21:17", "content": "

@StormMiner: Thanks for the detailed information - much appreciated!

\n

 

\n

disclaimer: I'm not the author of sklearn.ensemble - I merely created the gradient boosting implementation - much of the work in ensemble has been contributed by another fellow Kaggler: Gilles Louppe [2] .

\n

@others: I've created a github issue to track performance related issues in GradientBoostingRegressor - lets move the discussion there, its more convenient and other sklearn devs can comment. Here is the link: https://github.com/scikit-learn/scikit-learn/issues/2466  . Please get involved in the discussion and add your findings / concerns.

\n

 

\n

Regarding GradientBoostingRegressor: there is a performance regression (for certain hyper-parameter settings) in the latest release (0.14) - we switched our tree implementation which is now significantly faster for Random Forest but slower when building shallow trees (and no random split point selection). You might want to switch to the 0.13.1 release until it is fixed.

\n

 

\n

Regarding the difference between R's gbm and sklearn GradientBoosting implementation: the last throughout comparison I did was based on the old tree engine (0.13.1 release) - the results are summarized in another kaggle post [1] . The key point here is that gbm and sklearn learn different trees - sklearn grows complete trees of max_depth whereas gbm only grows a single branch of max_depth. The later is much faster but the former gives slightly better performance in my experience. At one point I'd like to add both strategies to sklearn's implementation.

\n

 

\n

Another performance tipp: optimize the ``max_features`` hyper-parameter - it will make training much faster and is also great for variance control.

\n

 

\n

@lamb: I hardly grow trees deeper than max_depth 7 - you should look into deviance plots whether you are overfitting - if so, try to tune ``max_features`` (learning rate tuning is not so good because it will further increase the runtime of the algorithm).

\n

 

\n

[1] http://www.kaggle.com/c/online-sales/forums/t/2135/congrats-to-the-winners/12130#post12130

\n

[2] http://www.kaggle.com/users/1955/gilles-louppe

", "votes": null, "replies": [] }, { "id": 31305, "author_name": "leustagos", "author_url": "", "post_date": "09/21/2013 14:06:26", "content": "

@Peter, thanks!

\n

Are you using GradientBoostedRegression for this task?

\n

Now i do have some GBRT questions to ask, unrelated to this competition, whats is the proper channel for me to ask them? I looked at the help, and didnt found what i seek, maybe it isn't implemented, but i need to ask anyway ...

", "votes": null, "replies": [] }, { "id": 31306, "author_name": "pprett", "author_url": "", "post_date": "09/21/2013 14:23:41", "content": "

@leustagos: yes, I do - I tried RandomForest too and it basically gives me the same result.

\n

you can write me an email to firstname.lastname@gmail.com - or for more sklearn related question just write an email to the sklearn mailing list or open a github issue (for problems/enhancement proposals)

", "votes": null, "replies": [] }, { "id": 31865, "author_name": "ijaganjac", "author_url": "", "post_date": "10/01/2013 15:13:28", "content": "

@Leustagos,

\n

 

\n

Hello.  Did you perhaps try:

\n

 

\n

model=RandomForestRegressor(n_estimators=500,max_depth=5,n_jobs=2)

\n

for:   training data shape (5113,2160) (5113,98).

\n

It takes very long time on 2-core uP.

", "votes": null, "replies": [] }, { "id": 31866, "author_name": "pprett", "author_url": "", "post_date": "10/01/2013 15:21:35", "content": "

If you don't set max_features then RandomForestRegressor  will search over all of them to find the best split [1]. Setting ``max_features`` to some small number eg 0.1 (10 % of the features) will make it run considerably faster.

\n

 

\n

[1] http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor

", "votes": null, "replies": [] }, { "id": 31867, "author_name": "ijaganjac", "author_url": "", "post_date": "10/01/2013 15:28:45", "content": "

@Peter,

\n

Thanks for the tip.   That's it.

", "votes": null, "replies": [] }, { "id": 31868, "author_name": "leustagos", "author_url": "", "post_date": "10/01/2013 15:29:29", "content": "

Since you asked me, i wouldnt use rf for this task. RF implementations minimizes MSE, while this competition error metric is MAE.

\n

And Petter advice also applies to GradientBoostedRegressor, that does minimize MAD with the lad loss.

", "votes": null, "replies": [] }, { "id": 31869, "author_name": "ijaganjac", "author_url": "", "post_date": "10/01/2013 15:37:03", "content": "

@Leustagos,

\n

Thanks.  An error comes up when GradientBoostedRegressor is applied on training data shape (5113,2160) (5113,98).

", "votes": null, "replies": [] }, { "id": 31871, "author_name": "pprett", "author_url": "", "post_date": "10/01/2013 15:45:29", "content": "

GradientBoostingRegressor does not support multiple outputs (y.shape[1] > 1)

", "votes": null, "replies": [] }, { "id": 31876, "author_name": "leustagos", "author_url": "", "post_date": "10/01/2013 17:24:24", "content": "

Isnt this the same as doing one model per output?

", "votes": null, "replies": [] }, { "id": 31882, "author_name": "pprett", "author_url": "", "post_date": "10/01/2013 18:48:17", "content": "

A multi-output tree computes split points by measuring impurity for each output - for each split it chooses the point that has min mean impurity - so its different from training one tree per output. The model can capture correlation between outputs.

", "votes": null, "replies": [] }, { "id": 31883, "author_name": "pprett", "author_url": "", "post_date": "10/01/2013 18:54:41", "content": "

That was too quick. I agree with Lucas, if you want to use GradientBoostingRegressor for multi-output problems you need to train n_output models. The outputs will be predicted in isolation - no correlation can be exploited. This is different from RandomForestRegressor which uses multi-output trees.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "31161": "", "31258": "", "31260": "", "31285": "", "31293": "", "31295": "", "31305": "", "31306": "", "31865": "", "31866": "", "31867": "", "31868": "", "31869": "", "31871": "", "31876": "", "31882": "", "31883": "" }, "source": "meta" }