{ "id": 47775, "title": "Lesson not learned?", "url": "/competitions/favorita-grocery-sales-forecasting/discussion/47775", "author_name": "Andy Harless", "post_date": "2018-01-18T18:34:42.662000", "votes": 6, "comment_count": 0, "views": 0, "content": "

In Porto Seguro my team came in just short of gold, and I noted that my teammate's simple rank average would've taken gold easily, while my cleverly weighted ensemble ended up being overfit. It appears I have failed to take my own advice. I did a late submission for Favorita that was a simple arithmetic average of each teammate's best model (according to public LB score*), and it got .515 private, which, by current standings, would be gold. It's hard to imagine we would ever have been prescient enough to do something so simple, since all our best models used LightGBM and they were all derived in one way or another from the Ceshine/Lingzhi/Clive model, so it would inevitably have seemed obvious that there wasn't enough diversity. But I guess the lesson still applies. Or maybe 3 lessons:

\n\n
    \n
  1. Keep your ensemble weights, and your ensembling process, simple (unless you have a good way to validate the ensemble, which you could if you had enough unbiased data and could use an extra holdout set, untainted by previous overfitting, to choose ensemble weights. But that's basically just stacking a constrained OLS on top of your models, which is fine, but if you're going to stack, why just use a constrained OLS?)
    2. \n
  2. If your individual models are good, correlations of .97 to .98 (which we had) are quite respectable, and you don't necessarily need more diversity. In particular, you don't necessarily need a diversity of fitting methods.
    3. \n
  3. For a team, a simple arithmetic average of everyone's best models might in general be a good choice as a second final submission.
    4. \n
\n\n

*I'm cheating a little on the definition of \"best,\" because none of my best models were ever submitted except as part of the ensemble, so I chose my best for this purpose based on public LB scores from a late submission, while I chose my teammates' best on scores we had actually seen.

", "messages": [ { "id": 270693, "postDate": "2018-01-18T18:34:42.663Z", "content": "

In Porto Seguro my team came in just short of gold, and I noted that my teammate's simple rank average would've taken gold easily, while my cleverly weighted ensemble ended up being overfit. It appears I have failed to take my own advice. I did a late submission for Favorita that was a simple arithmetic average of each teammate's best model (according to public LB score*), and it got .515 private, which, by current standings, would be gold. It's hard to imagine we would ever have been prescient enough to do something so simple, since all our best models used LightGBM and they were all derived in one way or another from the Ceshine/Lingzhi/Clive model, so it would inevitably have seemed obvious that there wasn't enough diversity. But I guess the lesson still applies. Or maybe 3 lessons:

\n\n
    \n
  1. Keep your ensemble weights, and your ensembling process, simple (unless you have a good way to validate the ensemble, which you could if you had enough unbiased data and could use an extra holdout set, untainted by previous overfitting, to choose ensemble weights. But that's basically just stacking a constrained OLS on top of your models, which is fine, but if you're going to stack, why just use a constrained OLS?)
    2. \n
  2. If your individual models are good, correlations of .97 to .98 (which we had) are quite respectable, and you don't necessarily need more diversity. In particular, you don't necessarily need a diversity of fitting methods.
    3. \n
  3. For a team, a simple arithmetic average of everyone's best models might in general be a good choice as a second final submission.
    4. \n
\n\n

*I'm cheating a little on the definition of \"best,\" because none of my best models were ever submitted except as part of the ensemble, so I chose my best for this purpose based on public LB scores from a late submission, while I chose my teammates' best on scores we had actually seen.

", "rawMarkdown": "In Porto Seguro my team came in just short of gold, and I noted that [my teammate's simple rank average][1] would've taken gold easily, while my cleverly weighted ensemble ended up being overfit. It appears I have failed to take my own advice. I did a late submission for Favorita that was a simple arithmetic average of each teammate's best model (according to public LB score*), and it got .515 private, which, by current standings, would be gold. It's hard to imagine we would ever have been prescient enough to do something so simple, since all our best models used LightGBM and they were all derived in one way or another from the Ceshine/Lingzhi/Clive model, so it would inevitably have seemed obvious that there wasn't enough diversity. But I guess the lesson still applies. Or maybe 3 lessons:\n\n 1. *Keep your ensemble weights, and your ensembling process, simple* \n 2. If your individual models are good, *correlations of .97 to .98* (which we had) *are quite respectable*, and you don't necessarily need more diversity. In particular, you don't necessarily need a diversity of fitting methods.\n 3. For a team, *a simple arithmetic average of everyone's best models* might in general be a good choice as a second final submission.\n\n*\n\n [1]: https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44759#latest-253318", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "270693": "In Porto Seguro my team came in just short of gold, and I noted that [my teammate's simple rank average][1] would've taken gold easily, while my cleverly weighted ensemble ended up being overfit. It appears I have failed to take my own advice. I did a late submission for Favorita that was a simple arithmetic average of each teammate's best model (according to public LB score*), and it got .515 private, which, by current standings, would be gold. It's hard to imagine we would ever have been prescient enough to do something so simple, since all our best models used LightGBM and they were all derived in one way or another from the Ceshine/Lingzhi/Clive model, so it would inevitably have seemed obvious that there wasn't enough diversity. But I guess the lesson still applies. Or maybe 3 lessons:\n\n 1. *Keep your ensemble weights, and your ensembling process, simple* \n 2. If your individual models are good, *correlations of .97 to .98* (which we had) *are quite respectable*, and you don't necessarily need more diversity. In particular, you don't necessarily need a diversity of fitting methods.\n 3. For a team, *a simple arithmetic average of everyone's best models* might in general be a good choice as a second final submission.\n\n*\n\n [1]: https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44759#latest-253318" } }