Given the fact that this NCAA competition only provides historical data for validation, it's hard for us to hypertune our models. Since luck often plays a significant role in the sports competition result prediction, could anyone offer some valuable advice on how to hypertune models and perform cross validation?
", "messages": [ { "id": "757699", "postDate": "02/27/2020 02:55:51", "content": "Given the fact that this NCAA competition only provides historical data for validation, it's hard for us to hypertune our models. Since luck often plays a significant role in the sports competition result prediction, could anyone offer some valuable advice on how to hypertune models and perform cross validation?
", "rawMarkdown": "Given the fact that this NCAA competition only provides historical data for validation, it's hard for us to hypertune our models. Since luck often plays a significant role in the sports competition result prediction, could anyone offer some valuable advice on how to hypertune models and perform cross validation?", "votes": null }, { "id": "758186", "postDate": "02/27/2020 14:29:01", "content": "Since no one has replied, I'll give my thoughts. \nIn sports, you can't remove luck from the equation; if your model is 75% accurate, the other 25% is that which cannot be modeled - aka luck. Luck could be injuries/illness, ref calls, weird ball bounces, bad food on a flight to the game, players getting poor rest, players distracted by off the court issues, players thinking more about playing pro than the tournament, players folding in pressure situations(although I think this one is somewhat indirectly quantifiable), team infighting...on and on.
\nAll we have is historical data to go off of. You should approach this competition with two goals in mind. One - we all have mostly the same data, so you need to ask yourself what you can do differently that maximizes the likelihood of doing well. Two - try to model every game as closely as you can to the expected probability of the actual outcome. That is the exact way to minimize for luck but it can't be eliminated from your model.
Luck plays less of a role in the score differential than it does in the final game outcome. As others have suggested previously, building and evaluating your model as a regression problem on the score differential and then (somehow) converting your predicted differential into a probability is a decent place to start. But as @btesch mentioned below, luck is a major part of each of these tournaments, and this year will certainly be no different.
", "rawMarkdown": "Luck plays less of a role in the score differential than it does in the final game outcome. As others have suggested previously, building and evaluating your model as a regression problem on the score differential and then (somehow) converting your predicted differential into a probability is a decent place to start. But as @btesch mentioned below, luck is a major part of each of these tournaments, and this year will certainly be no different.", "votes": null }, { "id": "759888", "postDate": "02/29/2020 15:28:50", "content": "I agree with the previously posted comments. In addition, my method for hypertuning parameters involved holding out tournament data for a given year, and toggling the parameters to optimize scoring using late submissions in the previous contests.
\n\nLong story short - Use CV and iterate through key hyperparameters until you minimize whatever error metric you plan on using.
", "rawMarkdown": "I agree with the previously posted comments. In addition, my method for hypertuning parameters involved holding out tournament data for a given year, and toggling the parameters to optimize scoring using late submissions in the previous contests.\n\nLong story short - Use CV and iterate through key hyperparameters until you minimize whatever error metric you plan on using.", "votes": null }, { "id": "759939", "postDate": "02/29/2020 16:28:54", "content": "Modeling point differences instead of win/loss removes a lot of luck from the equation. (I personally wish this competition had switched over to using point differences and RMSE years ago, as that would make it much harder to gamble on the tournament results. Here's an excellent blog post from 4 years ago describing the issue: http://netprophetblog.blogspot.com/2016/03/scoring-kaggle-contest.html).
\n\nIn your local CV, if your RMSE is <10, you're overfitting: check for target leakage.
\n\nYou can convert predicted point differences to win probabilities using logistic regression, or simply an equation like logistic(0.1655*pred_point_diff).
\n\nIn R that's plogis(0.1655*pred_point_diff) and in python that's scipy.special.expit(0.1655*pred_point_diff).
\n\nAnother approach that works well in R is pnorm(0.09656*pred_point_diff).
", "rawMarkdown": "Modeling point differences instead of win/loss removes a lot of luck from the equation. (I personally wish this competition had switched over to using point differences and RMSE years ago, as that would make it much harder to gamble on the tournament results. Here's an excellent blog post from 4 years ago describing the issue: http://netprophetblog.blogspot.com/2016/03/scoring-kaggle-contest.html). \n\nIn your local CV, if your RMSE is <10, you're overfitting: check for target leakage.\n\nYou can convert predicted point differences to win probabilities using logistic regression, or simply an equation like logistic(0.1655*pred_point_diff).\n\nIn R that's plogis(0.1655*pred_point_diff) and in python that's scipy.special.expit(0.1655*pred_point_diff).\n\nAnother approach that works well in R is pnorm(0.09656*pred_point_diff).", "votes": null }, { "id": "759999", "postDate": "02/29/2020 17:52:16", "content": "I've been using a scale of 0.15 for the difference -> probability conversion, based on eyeballing the probabilities it produces. Is 0.1655 optimal for some reason?
", "rawMarkdown": "I've been using a scale of 0.15 for the difference -> probability conversion, based on eyeballing the probabilities it produces. Is 0.1655 optimal for some reason?", "votes": null }, { "id": "760230", "postDate": "03/01/2020 02:22:44", "content": "@zachmayer I'm not sure that I agree with your statement. Converting between expected margin of victory and win probability is linear, so I don't see how luck is removed. How would the 'gambling' aspect be reduced?
", "rawMarkdown": "zachmayer I'm not sure that I agree with your statement. Converting between expected margin of victory and win probability is linear, so I don't see how luck is removed. How would the 'gambling' aspect be reduced?", "votes": null }, { "id": "760593", "postDate": "03/01/2020 13:59:03", "content": "That’s because with probabilities you can be really aggressive with your submission to minimize the log loss. With spreads it mostly depends on how good your model is.
", "rawMarkdown": "That’s because with probabilities you can be really aggressive with your submission to minimize the log loss. With spreads it mostly depends on how good your model is.", "votes": null }, { "id": "760606", "postDate": "03/01/2020 14:18:55", "content": "Some people suggested different metrics, for example CRPS as used in NFL sounds very promising for comps like this. Not sure why they again settled for the same metric knowing all the issues.
", "rawMarkdown": "Some people suggested different metrics, for example CRPS as used in NFL sounds very promising for comps like this. Not sure why they again settled for the same metric knowing all the issues.", "votes": null } ], "comments": [ { "id": 758186, "author_name": "btesch", "author_url": "", "post_date": "02/27/2020 14:29:01", "content": "Since no one has replied, I'll give my thoughts. \nIn sports, you can't remove luck from the equation; if your model is 75% accurate, the other 25% is that which cannot be modeled - aka luck. Luck could be injuries/illness, ref calls, weird ball bounces, bad food on a flight to the game, players getting poor rest, players distracted by off the court issues, players thinking more about playing pro than the tournament, players folding in pressure situations(although I think this one is somewhat indirectly quantifiable), team infighting...on and on.
\nAll we have is historical data to go off of. You should approach this competition with two goals in mind. One - we all have mostly the same data, so you need to ask yourself what you can do differently that maximizes the likelihood of doing well. Two - try to model every game as closely as you can to the expected probability of the actual outcome. That is the exact way to minimize for luck but it can't be eliminated from your model.
Luck plays less of a role in the score differential than it does in the final game outcome. As others have suggested previously, building and evaluating your model as a regression problem on the score differential and then (somehow) converting your predicted differential into a probability is a decent place to start. But as @btesch mentioned below, luck is a major part of each of these tournaments, and this year will certainly be no different.
", "votes": null, "replies": [] }, { "id": 759888, "author_name": "jleecook", "author_url": "", "post_date": "02/29/2020 15:28:50", "content": "I agree with the previously posted comments. In addition, my method for hypertuning parameters involved holding out tournament data for a given year, and toggling the parameters to optimize scoring using late submissions in the previous contests.
\n\nLong story short - Use CV and iterate through key hyperparameters until you minimize whatever error metric you plan on using.
", "votes": null, "replies": [] }, { "id": 759939, "author_name": "zachmayer", "author_url": "", "post_date": "02/29/2020 16:28:54", "content": "Modeling point differences instead of win/loss removes a lot of luck from the equation. (I personally wish this competition had switched over to using point differences and RMSE years ago, as that would make it much harder to gamble on the tournament results. Here's an excellent blog post from 4 years ago describing the issue: http://netprophetblog.blogspot.com/2016/03/scoring-kaggle-contest.html).
\n\nIn your local CV, if your RMSE is <10, you're overfitting: check for target leakage.
\n\nYou can convert predicted point differences to win probabilities using logistic regression, or simply an equation like logistic(0.1655*pred_point_diff).
\n\nIn R that's plogis(0.1655*pred_point_diff) and in python that's scipy.special.expit(0.1655*pred_point_diff).
\n\nAnother approach that works well in R is pnorm(0.09656*pred_point_diff).
", "votes": null, "replies": [ { "id": 759999, "author_name": "joshstone", "author_url": "", "post_date": "02/29/2020 17:52:16", "content": "I've been using a scale of 0.15 for the difference -> probability conversion, based on eyeballing the probabilities it produces. Is 0.1655 optimal for some reason?
", "votes": null, "replies": [] }, { "id": 760230, "author_name": "btesch", "author_url": "", "post_date": "03/01/2020 02:22:44", "content": "@zachmayer I'm not sure that I agree with your statement. Converting between expected margin of victory and win probability is linear, so I don't see how luck is removed. How would the 'gambling' aspect be reduced?
", "votes": null, "replies": [] }, { "id": 760593, "author_name": "sreepriya", "author_url": "", "post_date": "03/01/2020 13:59:03", "content": "That’s because with probabilities you can be really aggressive with your submission to minimize the log loss. With spreads it mostly depends on how good your model is.
", "votes": null, "replies": [] } ] }, { "id": 760606, "author_name": "philippsinger", "author_url": "", "post_date": "03/01/2020 14:18:55", "content": "Some people suggested different metrics, for example CRPS as used in NFL sounds very promising for comps like this. Not sure why they again settled for the same metric knowing all the issues.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "757699": "Given the fact that this NCAA competition only provides historical data for validation, it's hard for us to hypertune our models. Since luck often plays a significant role in the sports competition result prediction, could anyone offer some valuable advice on how to hypertune models and perform cross validation?", "758186": "Since no one has replied, I'll give my thoughts. \nIn sports, you can't remove luck from the equation; if your model is 75% accurate, the other 25% is that which cannot be modeled - aka luck. Luck could be injuries/illness, ref calls, weird ball bounces, bad food on a flight to the game, players getting poor rest, players distracted by off the court issues, players thinking more about playing pro than the tournament, players folding in pressure situations(although I think this one is somewhat indirectly quantifiable), team infighting...on and on. \nAll we have is historical data to go off of. You should approach this competition with two goals in mind. One - we all have mostly the same data, so you need to ask yourself what you can do differently that maximizes the likelihood of doing well. Two - try to model every game as closely as you can to the expected probability of the actual outcome. That is the exact way to minimize for luck but it can't be eliminated from your model.", "758336": "Luck plays less of a role in the score differential than it does in the final game outcome. As others have suggested previously, building and evaluating your model as a regression problem on the score differential and then (somehow) converting your predicted differential into a probability is a decent place to start. But as @btesch mentioned below, luck is a major part of each of these tournaments, and this year will certainly be no different.", "759888": "I agree with the previously posted comments. In addition, my method for hypertuning parameters involved holding out tournament data for a given year, and toggling the parameters to optimize scoring using late submissions in the previous contests.\n\nLong story short - Use CV and iterate through key hyperparameters until you minimize whatever error metric you plan on using.", "759939": "Modeling point differences instead of win/loss removes a lot of luck from the equation. (I personally wish this competition had switched over to using point differences and RMSE years ago, as that would make it much harder to gamble on the tournament results. Here's an excellent blog post from 4 years ago describing the issue: http://netprophetblog.blogspot.com/2016/03/scoring-kaggle-contest.html). \n\nIn your local CV, if your RMSE is <10, you're overfitting: check for target leakage.\n\nYou can convert predicted point differences to win probabilities using logistic regression, or simply an equation like logistic(0.1655*pred_point_diff).\n\nIn R that's plogis(0.1655*pred_point_diff) and in python that's scipy.special.expit(0.1655*pred_point_diff).\n\nAnother approach that works well in R is pnorm(0.09656*pred_point_diff).", "759999": "I've been using a scale of 0.15 for the difference -> probability conversion, based on eyeballing the probabilities it produces. Is 0.1655 optimal for some reason?", "760230": "zachmayer I'm not sure that I agree with your statement. Converting between expected margin of victory and win probability is linear, so I don't see how luck is removed. How would the 'gambling' aspect be reduced?", "760593": "That’s because with probabilities you can be really aggressive with your submission to minimize the log loss. With spreads it mostly depends on how good your model is.", "760606": "Some people suggested different metrics, for example CRPS as used in NFL sounds very promising for comps like this. Not sure why they again settled for the same metric knowing all the issues." }, "source": "meta" }