I tried Chau Ngoc Huynh's fabulous kernel https://www.kaggle.com/chauhuynh/my-first-kernel-3-699, and with a 10-fold cross-validation LightGBM, I get a CV of 3.6652 for 'dart' boosting, as opposed to a CV of 3.6518 for 'gbdt' boosting.
\n\nDart takes a lot longer to run, so it just seems like the worse option all-around, but even in the LightGBM documentation (https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html), dart is recommended for better accuracy, which is clearly not the case here.
\n\nI was wondering if anyone had any intuition as to why this difference occurs.
", "messages": [ { "id": "435279", "postDate": "12/07/2018 19:32:36", "content": "I tried Chau Ngoc Huynh's fabulous kernel https://www.kaggle.com/chauhuynh/my-first-kernel-3-699, and with a 10-fold cross-validation LightGBM, I get a CV of 3.6652 for 'dart' boosting, as opposed to a CV of 3.6518 for 'gbdt' boosting.
\n\nDart takes a lot longer to run, so it just seems like the worse option all-around, but even in the LightGBM documentation (https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html), dart is recommended for better accuracy, which is clearly not the case here.
\n\nI was wondering if anyone had any intuition as to why this difference occurs.
", "rawMarkdown": "I tried *Chau Ngoc Huynh's* fabulous kernel https://www.kaggle.com/chauhuynh/my-first-kernel-3-699, and with a 10-fold cross-validation LightGBM, I get a CV of 3.6652 for '**dart**' boosting, as opposed to a CV of 3.6518 for **'gbdt**' boosting. \n\n**Dart** takes a lot longer to run, so it just seems like the worse option all-around, but even in the LightGBM documentation (https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html), **dart** is recommended for better accuracy, which is clearly not the case here. \n\n\nI was wondering if anyone had any intuition as to why this difference occurs.", "votes": null }, { "id": "435393", "postDate": "12/08/2018 00:34:37", "content": "This varies from dataset to dataset. In this particular competition I'm getting consistently better CV results with DART actually (contrary to your own observation), but on Home Credit, for example, GBDT was king for me.
\n\nHowever, I wouldn't expect DART to work better as a drop-in replacement for GBDT without changing any other hyperparameters. In my experience predictions from DART and GBDT models tend to be quite correlated when validation results are similar, but the models can still look a bit different. My DART models all use different hyperparameters than the current best scoring GBDT kernel.
", "rawMarkdown": "This varies from dataset to dataset. In this particular competition I'm getting consistently better CV results with DART actually (contrary to your own observation), but on Home Credit, for example, GBDT was king for me. \n\nHowever, I wouldn't expect DART to work better as a drop-in replacement for GBDT without changing any other hyperparameters. In my experience predictions from DART and GBDT models tend to be quite correlated when validation results are similar, but the models can still look a bit different. My DART models all use different hyperparameters than the current best scoring GBDT kernel.", "votes": null }, { "id": "435396", "postDate": "12/08/2018 00:43:47", "content": "You may need to retune your dart. In particular, a higher learning rate is likely going to be helpful. I'm a bit skeptical of the claim though that dart leads to better accuracy... I've only seen it be more accurate when I have a lot of variables (and a low column sampling rate).
", "rawMarkdown": "You may need to retune your dart. In particular, a higher learning rate is likely going to be helpful. I'm a bit skeptical of the claim though that dart leads to better accuracy... I've only seen it be more accurate when I have *a lot* of variables (and a low column sampling rate).", "votes": null }, { "id": "435584", "postDate": "12/08/2018 10:20:32", "content": "I've made the same observations, and I find the learning rate thing to be particularly interesting. With GBDT, you can almost always keep decreasing the learning rate all the way down to something like 0.001 (after tuning the other hyperparameters at a higher learning rate, of course) and you will keep getting incremental performance gains at the cost of training time – which at some point becomes intractable, but on Zillow I remember training with super-low learning rates up to almost 20000 iterations.
\n\nDART does not seem to work this way. If I do a hyperparameter search with DART on a higher learning rate, and then try to lower it significantly for training the final model(s), what usually happens for me is that some folds get better and some get worse (I've never seen this happen with GBDT). Sometimes the CV score ends up being higher, sometimes it's lower, sometimes it's more or less the same.
\n\nAlthough most boosting aficionados would consider this bad practice, I'm increasinly tempted to start treating the learning rate a hyperparameter to be tuned when working with DART models.
", "rawMarkdown": "I've made the same observations, and I find the learning rate thing to be particularly interesting. With GBDT, you can almost always keep decreasing the learning rate all the way down to something like 0.001 (after tuning the other hyperparameters at a higher learning rate, of course) and you will keep getting incremental performance gains at the cost of training time – which at some point becomes intractable, but on Zillow I remember training with super-low learning rates up to almost 20000 iterations.\n\nDART does not seem to work this way. If I do a hyperparameter search with DART on a higher learning rate, and then try to lower it significantly for training the final model(s), what usually happens for me is that some folds get better and some get worse (I've never seen this happen with GBDT). Sometimes the CV score ends up being higher, sometimes it's lower, sometimes it's more or less the same. \n\nAlthough most boosting aficionados would consider this bad practice, I'm increasinly tempted to start treating the learning rate a hyperparameter to be tuned when working with DART models.", "votes": null }, { "id": "435717", "postDate": "12/08/2018 16:19:40", "content": "Ah I see, this is very helpful, thank you!
", "rawMarkdown": "Ah I see, this is very helpful, thank you!", "votes": null }, { "id": "435718", "postDate": "12/08/2018 16:20:09", "content": "Thanks for the intuition, I'll try a different set hyperparams w DART and see if I can get a better CV
", "rawMarkdown": "Thanks for the intuition, I'll try a different set hyperparams w DART and see if I can get a better CV", "votes": null }, { "id": "436154", "postDate": "12/09/2018 17:42:13", "content": "regarding to Dart/GBDT trade-off, my experience so far is that Dart is more accurate and faster on LARGER datasets - Talking Data 2018 & Instarcart for instance when you have more than 10+ rows. On smaller dataset such as this, seems GBDT is better.
\n\nadditionally, Dart is always useful for adding diversity to your stacking :)
", "rawMarkdown": "regarding to Dart/GBDT trade-off, my experience so far is that Dart is more accurate and faster on LARGER datasets - Talking Data 2018 & Instarcart for instance when you have more than 10+ rows. On smaller dataset such as this, seems GBDT is better. \n\nadditionally, Dart is always useful for adding diversity to your stacking :)", "votes": null } ], "comments": [ { "id": 435393, "author_name": "keegil", "author_url": "", "post_date": "12/08/2018 00:34:37", "content": "This varies from dataset to dataset. In this particular competition I'm getting consistently better CV results with DART actually (contrary to your own observation), but on Home Credit, for example, GBDT was king for me.
\n\nHowever, I wouldn't expect DART to work better as a drop-in replacement for GBDT without changing any other hyperparameters. In my experience predictions from DART and GBDT models tend to be quite correlated when validation results are similar, but the models can still look a bit different. My DART models all use different hyperparameters than the current best scoring GBDT kernel.
", "votes": null, "replies": [ { "id": 435718, "author_name": "delayedkarma", "author_url": "", "post_date": "12/08/2018 16:20:09", "content": "Thanks for the intuition, I'll try a different set hyperparams w DART and see if I can get a better CV
", "votes": null, "replies": [] } ] }, { "id": 435396, "author_name": "peterhurford", "author_url": "", "post_date": "12/08/2018 00:43:47", "content": "You may need to retune your dart. In particular, a higher learning rate is likely going to be helpful. I'm a bit skeptical of the claim though that dart leads to better accuracy... I've only seen it be more accurate when I have a lot of variables (and a low column sampling rate).
", "votes": null, "replies": [ { "id": 435584, "author_name": "keegil", "author_url": "", "post_date": "12/08/2018 10:20:32", "content": "I've made the same observations, and I find the learning rate thing to be particularly interesting. With GBDT, you can almost always keep decreasing the learning rate all the way down to something like 0.001 (after tuning the other hyperparameters at a higher learning rate, of course) and you will keep getting incremental performance gains at the cost of training time – which at some point becomes intractable, but on Zillow I remember training with super-low learning rates up to almost 20000 iterations.
\n\nDART does not seem to work this way. If I do a hyperparameter search with DART on a higher learning rate, and then try to lower it significantly for training the final model(s), what usually happens for me is that some folds get better and some get worse (I've never seen this happen with GBDT). Sometimes the CV score ends up being higher, sometimes it's lower, sometimes it's more or less the same.
\n\nAlthough most boosting aficionados would consider this bad practice, I'm increasinly tempted to start treating the learning rate a hyperparameter to be tuned when working with DART models.
", "votes": null, "replies": [] }, { "id": 435717, "author_name": "delayedkarma", "author_url": "", "post_date": "12/08/2018 16:19:40", "content": "Ah I see, this is very helpful, thank you!
", "votes": null, "replies": [] } ] }, { "id": 436154, "author_name": "yifanxie", "author_url": "", "post_date": "12/09/2018 17:42:13", "content": "regarding to Dart/GBDT trade-off, my experience so far is that Dart is more accurate and faster on LARGER datasets - Talking Data 2018 & Instarcart for instance when you have more than 10+ rows. On smaller dataset such as this, seems GBDT is better.
\n\nadditionally, Dart is always useful for adding diversity to your stacking :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "435279": "I tried *Chau Ngoc Huynh's* fabulous kernel https://www.kaggle.com/chauhuynh/my-first-kernel-3-699, and with a 10-fold cross-validation LightGBM, I get a CV of 3.6652 for '**dart**' boosting, as opposed to a CV of 3.6518 for **'gbdt**' boosting. \n\n**Dart** takes a lot longer to run, so it just seems like the worse option all-around, but even in the LightGBM documentation (https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html), **dart** is recommended for better accuracy, which is clearly not the case here. \n\n\nI was wondering if anyone had any intuition as to why this difference occurs.", "435393": "This varies from dataset to dataset. In this particular competition I'm getting consistently better CV results with DART actually (contrary to your own observation), but on Home Credit, for example, GBDT was king for me. \n\nHowever, I wouldn't expect DART to work better as a drop-in replacement for GBDT without changing any other hyperparameters. In my experience predictions from DART and GBDT models tend to be quite correlated when validation results are similar, but the models can still look a bit different. My DART models all use different hyperparameters than the current best scoring GBDT kernel.", "435396": "You may need to retune your dart. In particular, a higher learning rate is likely going to be helpful. I'm a bit skeptical of the claim though that dart leads to better accuracy... I've only seen it be more accurate when I have *a lot* of variables (and a low column sampling rate).", "435584": "I've made the same observations, and I find the learning rate thing to be particularly interesting. With GBDT, you can almost always keep decreasing the learning rate all the way down to something like 0.001 (after tuning the other hyperparameters at a higher learning rate, of course) and you will keep getting incremental performance gains at the cost of training time – which at some point becomes intractable, but on Zillow I remember training with super-low learning rates up to almost 20000 iterations.\n\nDART does not seem to work this way. If I do a hyperparameter search with DART on a higher learning rate, and then try to lower it significantly for training the final model(s), what usually happens for me is that some folds get better and some get worse (I've never seen this happen with GBDT). Sometimes the CV score ends up being higher, sometimes it's lower, sometimes it's more or less the same. \n\nAlthough most boosting aficionados would consider this bad practice, I'm increasinly tempted to start treating the learning rate a hyperparameter to be tuned when working with DART models.", "435717": "Ah I see, this is very helpful, thank you!", "435718": "Thanks for the intuition, I'll try a different set hyperparams w DART and see if I can get a better CV", "436154": "regarding to Dart/GBDT trade-off, my experience so far is that Dart is more accurate and faster on LARGER datasets - Talking Data 2018 & Instarcart for instance when you have more than 10+ rows. On smaller dataset such as this, seems GBDT is better. \n\nadditionally, Dart is always useful for adding diversity to your stacking :)" }, "source": "meta" }