I have a question regarding XGBoost and whether or not I could be approaching model fitting incorrectly. When I predicting a holdout set using this model, I am seeing some significant differences in predictions depending who is considered \"Team A\".
\n\nFor example, Gonzaga vs. BYU might give me a predicted game margin of Gonzaga winning by 2 if Gonzaga if considered \"Team A\", but if BYU is considered to be \"Team A\" then the model predicts Gonzaga to win by 7. Since their information should mirror each other when they matchup, I would think the predictions would be very close (maybe not exact), but a 5 point difference in margin of victory is significant.
\n\nDoes anyone have any thoughts on what I could be doing? I have double checked my holdout data set to ensure that the information does in fact mirror each other and I didn't make an error wrangling the data. I hope this makes sense.
", "messages": [ { "id": "767600", "postDate": "03/09/2020 21:48:43", "content": "I have a question regarding XGBoost and whether or not I could be approaching model fitting incorrectly. When I predicting a holdout set using this model, I am seeing some significant differences in predictions depending who is considered \"Team A\".
\n\nFor example, Gonzaga vs. BYU might give me a predicted game margin of Gonzaga winning by 2 if Gonzaga if considered \"Team A\", but if BYU is considered to be \"Team A\" then the model predicts Gonzaga to win by 7. Since their information should mirror each other when they matchup, I would think the predictions would be very close (maybe not exact), but a 5 point difference in margin of victory is significant.
\n\nDoes anyone have any thoughts on what I could be doing? I have double checked my holdout data set to ensure that the information does in fact mirror each other and I didn't make an error wrangling the data. I hope this makes sense.
", "rawMarkdown": "I have a question regarding XGBoost and whether or not I could be approaching model fitting incorrectly. When I predicting a holdout set using this model, I am seeing some significant differences in predictions depending who is considered \"Team A\". \n\nFor example, Gonzaga vs. BYU might give me a predicted game margin of Gonzaga winning by 2 if Gonzaga if considered \"Team A\", but if BYU is considered to be \"Team A\" then the model predicts Gonzaga to win by 7. Since their information should mirror each other when they matchup, I would think the predictions would be very close (maybe not exact), but a 5 point difference in margin of victory is significant. \n\nDoes anyone have any thoughts on what I could be doing? I have double checked my holdout data set to ensure that the information does in fact mirror each other and I didn't make an error wrangling the data. I hope this makes sense.", "votes": null }, { "id": "767660", "postDate": "03/10/2020 00:58:44", "content": "I had a similar issue earlier and never really figured out what was going on. I have been taking the mean of the predictions as my final result, but if you do figure out what is happening I would love to hear!
\n\nHere is a thread with a few comments about the problem for reference:\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/130855
", "rawMarkdown": "I had a similar issue earlier and never really figured out what was going on. I have been taking the mean of the predictions as my final result, but if you do figure out what is happening I would love to hear!\n\nHere is a thread with a few comments about the problem for reference:\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/130855", "votes": null }, { "id": "767680", "postDate": "03/10/2020 01:53:23", "content": "@rsa013 referenced our earlier conversation on the topic. If you predict a game as A vs B, and B vs A, you'll get different results. There are all types of ways to rectify the differences. I don't love my approach because I feel like it's not aggressive enough but I don't have a better way as of yet. \nI'll be curious to know who decides to rectify the differences and who doesn't. I will be. I believe it's an issue that has be resolved to remove 'luck' from the equation. If not, your model is dependent on which team is listed first. Maybe someone has found a better way to control for this but I haven't.
", "rawMarkdown": "rsa013 referenced our earlier conversation on the topic. If you predict a game as A vs B, and B vs A, you'll get different results. There are all types of ways to rectify the differences. I don't love my approach because I feel like it's not aggressive enough but I don't have a better way as of yet. \nI'll be curious to know who decides to rectify the differences and who doesn't. I will be. I believe it's an issue that has be resolved to remove 'luck' from the equation. If not, your model is dependent on which team is listed first. Maybe someone has found a better way to control for this but I haven't.", "votes": null }, { "id": "767742", "postDate": "03/10/2020 03:41:10", "content": "Ah sorry for duplicating your question. I was also considering the mean of the predictions. If I have some additional time to explore this, I may see if the prediction errors are greater in a certain direction (i.e. favorites vs. underdogs). I’ll report back any findings that might be helpful.
", "rawMarkdown": "Ah sorry for duplicating your question. I was also considering the mean of the predictions. If I have some additional time to explore this, I may see if the prediction errors are greater in a certain direction (i.e. favorites vs. underdogs). I’ll report back any findings that might be helpful.", "votes": null }, { "id": "767778", "postDate": "03/10/2020 04:49:00", "content": "If I understand XGBoost correctly (and it's entirely possible that I do not), the underlying CARTs that serve as the building blocks are not forced to have any kind of symmetry. Even if you balance your data (i.e., by doubling it and switching the features), you're still likely to end up with some kind of asymmetry due to the randomness in subsampling training data for each iteration or from randomly selecting features for splits during tree learning. But you would hope that the asymmetries average out for the most part. I agree that a 5 point difference seems significant and might be an indication that your model is overtuned to the particular random noise that you got during training.
\n\nAs for how to fix it, I'm not sure. You might be able to force some symmetry through a customized objective/evaluation function, but I'm not exactly sure how to do it or if that would work. Otherwise, I'd suggest what other posters have tried: run your model both ways and then combine the results in a symmetric fashion (e.g., arithmetic mean)
\n\nHave you tried making a histogram of the discrepancies? I'd expect it to be roughly normally distributed. It's possible that the BYU/Gonzaga example is an outlier and that most of the differences are near zero.
\n\nDo the same examples give the same discrepancies even when training with a different random seed?
", "rawMarkdown": "If I understand XGBoost correctly (and it's entirely possible that I do not), the underlying CARTs that serve as the building blocks are not forced to have any kind of symmetry. Even if you balance your data (i.e., by doubling it and switching the features), you're still likely to end up with some kind of asymmetry due to the randomness in subsampling training data for each iteration or from randomly selecting features for splits during tree learning. But you would hope that the asymmetries average out for the most part. I agree that a 5 point difference seems significant and might be an indication that your model is overtuned to the particular random noise that you got during training.\n\nAs for how to fix it, I'm not sure. You might be able to force some symmetry through a customized objective/evaluation function, but I'm not exactly sure how to do it or if that would work. Otherwise, I'd suggest what other posters have tried: run your model both ways and then combine the results in a symmetric fashion (e.g., arithmetic mean)\n\nHave you tried making a histogram of the discrepancies? I'd expect it to be roughly normally distributed. It's possible that the BYU/Gonzaga example is an outlier and that most of the differences are near zero.\n\nDo the same examples give the same discrepancies even when training with a different random seed?", "votes": null } ], "comments": [ { "id": 767660, "author_name": "rsa013", "author_url": "", "post_date": "03/10/2020 00:58:44", "content": "I had a similar issue earlier and never really figured out what was going on. I have been taking the mean of the predictions as my final result, but if you do figure out what is happening I would love to hear!
\n\nHere is a thread with a few comments about the problem for reference:\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/130855
", "votes": null, "replies": [ { "id": 767742, "author_name": "seanyman84", "author_url": "", "post_date": "03/10/2020 03:41:10", "content": "Ah sorry for duplicating your question. I was also considering the mean of the predictions. If I have some additional time to explore this, I may see if the prediction errors are greater in a certain direction (i.e. favorites vs. underdogs). I’ll report back any findings that might be helpful.
", "votes": null, "replies": [] } ] }, { "id": 767680, "author_name": "btesch", "author_url": "", "post_date": "03/10/2020 01:53:23", "content": "@rsa013 referenced our earlier conversation on the topic. If you predict a game as A vs B, and B vs A, you'll get different results. There are all types of ways to rectify the differences. I don't love my approach because I feel like it's not aggressive enough but I don't have a better way as of yet. \nI'll be curious to know who decides to rectify the differences and who doesn't. I will be. I believe it's an issue that has be resolved to remove 'luck' from the equation. If not, your model is dependent on which team is listed first. Maybe someone has found a better way to control for this but I haven't.
", "votes": null, "replies": [] }, { "id": 767778, "author_name": "goodspellr", "author_url": "", "post_date": "03/10/2020 04:49:00", "content": "If I understand XGBoost correctly (and it's entirely possible that I do not), the underlying CARTs that serve as the building blocks are not forced to have any kind of symmetry. Even if you balance your data (i.e., by doubling it and switching the features), you're still likely to end up with some kind of asymmetry due to the randomness in subsampling training data for each iteration or from randomly selecting features for splits during tree learning. But you would hope that the asymmetries average out for the most part. I agree that a 5 point difference seems significant and might be an indication that your model is overtuned to the particular random noise that you got during training.
\n\nAs for how to fix it, I'm not sure. You might be able to force some symmetry through a customized objective/evaluation function, but I'm not exactly sure how to do it or if that would work. Otherwise, I'd suggest what other posters have tried: run your model both ways and then combine the results in a symmetric fashion (e.g., arithmetic mean)
\n\nHave you tried making a histogram of the discrepancies? I'd expect it to be roughly normally distributed. It's possible that the BYU/Gonzaga example is an outlier and that most of the differences are near zero.
\n\nDo the same examples give the same discrepancies even when training with a different random seed?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "767600": "I have a question regarding XGBoost and whether or not I could be approaching model fitting incorrectly. When I predicting a holdout set using this model, I am seeing some significant differences in predictions depending who is considered \"Team A\". \n\nFor example, Gonzaga vs. BYU might give me a predicted game margin of Gonzaga winning by 2 if Gonzaga if considered \"Team A\", but if BYU is considered to be \"Team A\" then the model predicts Gonzaga to win by 7. Since their information should mirror each other when they matchup, I would think the predictions would be very close (maybe not exact), but a 5 point difference in margin of victory is significant. \n\nDoes anyone have any thoughts on what I could be doing? I have double checked my holdout data set to ensure that the information does in fact mirror each other and I didn't make an error wrangling the data. I hope this makes sense.", "767660": "I had a similar issue earlier and never really figured out what was going on. I have been taking the mean of the predictions as my final result, but if you do figure out what is happening I would love to hear!\n\nHere is a thread with a few comments about the problem for reference:\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/130855", "767680": "rsa013 referenced our earlier conversation on the topic. If you predict a game as A vs B, and B vs A, you'll get different results. There are all types of ways to rectify the differences. I don't love my approach because I feel like it's not aggressive enough but I don't have a better way as of yet. \nI'll be curious to know who decides to rectify the differences and who doesn't. I will be. I believe it's an issue that has be resolved to remove 'luck' from the equation. If not, your model is dependent on which team is listed first. Maybe someone has found a better way to control for this but I haven't.", "767742": "Ah sorry for duplicating your question. I was also considering the mean of the predictions. If I have some additional time to explore this, I may see if the prediction errors are greater in a certain direction (i.e. favorites vs. underdogs). I’ll report back any findings that might be helpful.", "767778": "If I understand XGBoost correctly (and it's entirely possible that I do not), the underlying CARTs that serve as the building blocks are not forced to have any kind of symmetry. Even if you balance your data (i.e., by doubling it and switching the features), you're still likely to end up with some kind of asymmetry due to the randomness in subsampling training data for each iteration or from randomly selecting features for splits during tree learning. But you would hope that the asymmetries average out for the most part. I agree that a 5 point difference seems significant and might be an indication that your model is overtuned to the particular random noise that you got during training.\n\nAs for how to fix it, I'm not sure. You might be able to force some symmetry through a customized objective/evaluation function, but I'm not exactly sure how to do it or if that would work. Otherwise, I'd suggest what other posters have tried: run your model both ways and then combine the results in a symmetric fashion (e.g., arithmetic mean)\n\nHave you tried making a histogram of the discrepancies? I'd expect it to be roughly normally distributed. It's possible that the BYU/Gonzaga example is an outlier and that most of the differences are near zero.\n\nDo the same examples give the same discrepancies even when training with a different random seed?" }, "source": "meta" }