I'm working on a model for the first time and am noticing that two of the models I've built (a neural net and XGBoost) both give me slightly different probabilities depending on the order I feed the teams into the model. The numbers aren't drastically different, but different enough that I was wondering if I am approaching this incorrectly.
\n\nI am training the models on each matchup twice, which I thought would balance the model out. It seems to be working somewhat, but not completely. Is this just within the noise of these models or do you take a drastically different approach? Thanks!
", "messages": [ { "id": "747767", "postDate": "02/16/2020 20:20:44", "content": "I'm working on a model for the first time and am noticing that two of the models I've built (a neural net and XGBoost) both give me slightly different probabilities depending on the order I feed the teams into the model. The numbers aren't drastically different, but different enough that I was wondering if I am approaching this incorrectly.
\n\nI am training the models on each matchup twice, which I thought would balance the model out. It seems to be working somewhat, but not completely. Is this just within the noise of these models or do you take a drastically different approach? Thanks!
", "rawMarkdown": "I'm working on a model for the first time and am noticing that two of the models I've built (a neural net and XGBoost) both give me slightly different probabilities depending on the order I feed the teams into the model. The numbers aren't drastically different, but different enough that I was wondering if I am approaching this incorrectly. \n\nI am training the models on each matchup twice, which I thought would balance the model out. It seems to be working somewhat, but not completely. Is this just within the noise of these models or do you take a drastically different approach? Thanks!", "votes": null }, { "id": "747842", "postDate": "02/16/2020 23:31:48", "content": "Changing the order of the teams shouldn't change the results. I believe that both of those algorithms will return slightly different results every iteration. There are ways to control for this -usually by setting a seed. But without knowing what neural net you're using, I can't say. You can see a similar question in the link. \nhttps://stackoverflow.com/questions/42819987/pythons-xgbregressor-vs-rs-xgboost
", "rawMarkdown": "Changing the order of the teams shouldn't change the results. I believe that both of those algorithms will return slightly different results every iteration. There are ways to control for this -usually by setting a seed. But without knowing what neural net you're using, I can't say. You can see a similar question in the link. \nhttps://stackoverflow.com/questions/42819987/pythons-xgbregressor-vs-rs-xgboost", "votes": null }, { "id": "748001", "postDate": "02/17/2020 05:14:35", "content": "Hi Ben, thanks. Yes, I am actually able to repeat the results each iteration by setting a seed. The part that isn't consistent is that if I might get a 62% chance of winning if I model Virginia vs. Texas Tech, but if I model Texas Tech vs. Virginia I get 40%.
\n\nThe model behaves correctly and sums to 1 for the win/loss odds of a given matchup, but if I flip the inputs to the model I get a different win/loss breakdown. Maybe, I'm too far in the weeds here, but it seemed like a reasonable sense check and I was a little surprised by the discrepancy.
", "rawMarkdown": "Hi Ben, thanks. Yes, I am actually able to repeat the results each iteration by setting a seed. The part that isn't consistent is that if I might get a 62% chance of winning if I model Virginia vs. Texas Tech, but if I model Texas Tech vs. Virginia I get 40%. \n\nThe model behaves correctly and sums to 1 for the win/loss odds of a given matchup, but if I flip the inputs to the model I get a different win/loss breakdown. Maybe, I'm too far in the weeds here, but it seemed like a reasonable sense check and I was a little surprised by the discrepancy.", "votes": null }, { "id": "748014", "postDate": "02/17/2020 05:32:27", "content": "I got the same thing. I actually tested this earlier today for a season and my results summed to between .83 and 1.11. \nYou would think that if A vs B = n that B vs A = 1-n but it's not the case. \nI don't think you're too far into the weeds. I adjusted my code so that the probabilities always summed to 1. For example, I had an instance where both teams had a 55% chance to win. This obviously isn't what I want my model to return. Depending on how you write your code, you might get an unfavorable probability in your submission file.
", "rawMarkdown": "I got the same thing. I actually tested this earlier today for a season and my results summed to between .83 and 1.11. \nYou would think that if A vs B = n that B vs A = 1-n but it's not the case. \nI don't think you're too far into the weeds. I adjusted my code so that the probabilities always summed to 1. For example, I had an instance where both teams had a 55% chance to win. This obviously isn't what I want my model to return. Depending on how you write your code, you might get an unfavorable probability in your submission file.", "votes": null }, { "id": "748491", "postDate": "02/17/2020 15:19:58", "content": "Okay. Thanks for the feedback! Will have to think about the best way to approach it.
", "rawMarkdown": "Okay. Thanks for the feedback! Will have to think about the best way to approach it.", "votes": null }, { "id": "748585", "postDate": "02/17/2020 17:37:53", "content": "I'm taking the probability of A / (probability of A + probability of B) to get my results. It will always sum to 1. But I haven't quantified this approach. There might be a better way - possibly a weighting. I'll think about it and post here if I change it up.
\n\nA random thought would be that you could predict the games each way and use one submission for each attempt. I might look into that.
", "rawMarkdown": "I'm taking the probability of A / (probability of A + probability of B) to get my results. It will always sum to 1. But I haven't quantified this approach. There might be a better way - possibly a weighting. I'll think about it and post here if I change it up. \n\nA random thought would be that you could predict the games each way and use one submission for each attempt. I *might* look into that.", "votes": null }, { "id": "748709", "postDate": "02/17/2020 21:57:24", "content": "Ahh interesting idea. I'm currently predicting every game each way and then just taking the average. I was thinking about trying to use the max(A,B) to make a more aggressive model. I'll report back if I give it a shot.
", "rawMarkdown": "Ahh interesting idea. I'm currently predicting every game each way and then just taking the average. I was thinking about trying to use the max(A,B) to make a more aggressive model. I'll report back if I give it a shot.", "votes": null }, { "id": "748731", "postDate": "02/17/2020 22:51:24", "content": "I don't know anything about your methodology but you have to be careful about the asymmetry of the results data. Because WTeam is always the winning team and LTeam is the losing team, that means that always WScore > LScore. You may want to consider flipping the data and using each game record twice, once with WTeam against LTeam, and once with LTeam against WTeam, so then the two columns are more like Team and OppTeam. The same if you split it out into Home Team and Away Team; the Home Team will do better on average against the Away Team. I think sometimes people use 2x the data, or sometimes they randomly decide, for each row, whether to use the data or the flipped data.
", "rawMarkdown": "I don't know anything about your methodology but you have to be careful about the asymmetry of the results data. Because WTeam is always the winning team and LTeam is the losing team, that means that always WScore > LScore. You may want to consider flipping the data and using each game record twice, once with WTeam against LTeam, and once with LTeam against WTeam, so then the two columns are more like Team and OppTeam. The same if you split it out into Home Team and Away Team; the Home Team will do better on average against the Away Team. I think sometimes people use 2x the data, or sometimes they randomly decide, for each row, whether to use the data or the flipped data.", "votes": null }, { "id": "767658", "postDate": "03/10/2020 00:57:12", "content": "Hi @jeffsonas, sorry just saw this! I was actually flipping my input data and using 2x the data to train and validate, which is part of the reason I was surprised to still see a difference. I thought that would make it a non-issue, but it seems like it's still happening.
", "rawMarkdown": "Hi @jeffsonas, sorry just saw this! I was actually flipping my input data and using 2x the data to train and validate, which is part of the reason I was surprised to still see a difference. I thought that would make it a non-issue, but it seems like it's still happening.", "votes": null } ], "comments": [ { "id": 747842, "author_name": "btesch", "author_url": "", "post_date": "02/16/2020 23:31:48", "content": "Changing the order of the teams shouldn't change the results. I believe that both of those algorithms will return slightly different results every iteration. There are ways to control for this -usually by setting a seed. But without knowing what neural net you're using, I can't say. You can see a similar question in the link. \nhttps://stackoverflow.com/questions/42819987/pythons-xgbregressor-vs-rs-xgboost
", "votes": null, "replies": [ { "id": 748001, "author_name": "rsa013", "author_url": "", "post_date": "02/17/2020 05:14:35", "content": "Hi Ben, thanks. Yes, I am actually able to repeat the results each iteration by setting a seed. The part that isn't consistent is that if I might get a 62% chance of winning if I model Virginia vs. Texas Tech, but if I model Texas Tech vs. Virginia I get 40%.
\n\nThe model behaves correctly and sums to 1 for the win/loss odds of a given matchup, but if I flip the inputs to the model I get a different win/loss breakdown. Maybe, I'm too far in the weeds here, but it seemed like a reasonable sense check and I was a little surprised by the discrepancy.
", "votes": null, "replies": [] }, { "id": 748014, "author_name": "btesch", "author_url": "", "post_date": "02/17/2020 05:32:27", "content": "I got the same thing. I actually tested this earlier today for a season and my results summed to between .83 and 1.11. \nYou would think that if A vs B = n that B vs A = 1-n but it's not the case. \nI don't think you're too far into the weeds. I adjusted my code so that the probabilities always summed to 1. For example, I had an instance where both teams had a 55% chance to win. This obviously isn't what I want my model to return. Depending on how you write your code, you might get an unfavorable probability in your submission file.
", "votes": null, "replies": [] }, { "id": 748491, "author_name": "rsa013", "author_url": "", "post_date": "02/17/2020 15:19:58", "content": "Okay. Thanks for the feedback! Will have to think about the best way to approach it.
", "votes": null, "replies": [] }, { "id": 748585, "author_name": "btesch", "author_url": "", "post_date": "02/17/2020 17:37:53", "content": "I'm taking the probability of A / (probability of A + probability of B) to get my results. It will always sum to 1. But I haven't quantified this approach. There might be a better way - possibly a weighting. I'll think about it and post here if I change it up.
\n\nA random thought would be that you could predict the games each way and use one submission for each attempt. I might look into that.
", "votes": null, "replies": [] }, { "id": 748709, "author_name": "rsa013", "author_url": "", "post_date": "02/17/2020 21:57:24", "content": "Ahh interesting idea. I'm currently predicting every game each way and then just taking the average. I was thinking about trying to use the max(A,B) to make a more aggressive model. I'll report back if I give it a shot.
", "votes": null, "replies": [] } ] }, { "id": 748731, "author_name": "jeffsonas", "author_url": "", "post_date": "02/17/2020 22:51:24", "content": "I don't know anything about your methodology but you have to be careful about the asymmetry of the results data. Because WTeam is always the winning team and LTeam is the losing team, that means that always WScore > LScore. You may want to consider flipping the data and using each game record twice, once with WTeam against LTeam, and once with LTeam against WTeam, so then the two columns are more like Team and OppTeam. The same if you split it out into Home Team and Away Team; the Home Team will do better on average against the Away Team. I think sometimes people use 2x the data, or sometimes they randomly decide, for each row, whether to use the data or the flipped data.
", "votes": null, "replies": [ { "id": 767658, "author_name": "rsa013", "author_url": "", "post_date": "03/10/2020 00:57:12", "content": "Hi @jeffsonas, sorry just saw this! I was actually flipping my input data and using 2x the data to train and validate, which is part of the reason I was surprised to still see a difference. I thought that would make it a non-issue, but it seems like it's still happening.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "747767": "I'm working on a model for the first time and am noticing that two of the models I've built (a neural net and XGBoost) both give me slightly different probabilities depending on the order I feed the teams into the model. The numbers aren't drastically different, but different enough that I was wondering if I am approaching this incorrectly. \n\nI am training the models on each matchup twice, which I thought would balance the model out. It seems to be working somewhat, but not completely. Is this just within the noise of these models or do you take a drastically different approach? Thanks!", "747842": "Changing the order of the teams shouldn't change the results. I believe that both of those algorithms will return slightly different results every iteration. There are ways to control for this -usually by setting a seed. But without knowing what neural net you're using, I can't say. You can see a similar question in the link. \nhttps://stackoverflow.com/questions/42819987/pythons-xgbregressor-vs-rs-xgboost", "748001": "Hi Ben, thanks. Yes, I am actually able to repeat the results each iteration by setting a seed. The part that isn't consistent is that if I might get a 62% chance of winning if I model Virginia vs. Texas Tech, but if I model Texas Tech vs. Virginia I get 40%. \n\nThe model behaves correctly and sums to 1 for the win/loss odds of a given matchup, but if I flip the inputs to the model I get a different win/loss breakdown. Maybe, I'm too far in the weeds here, but it seemed like a reasonable sense check and I was a little surprised by the discrepancy.", "748014": "I got the same thing. I actually tested this earlier today for a season and my results summed to between .83 and 1.11. \nYou would think that if A vs B = n that B vs A = 1-n but it's not the case. \nI don't think you're too far into the weeds. I adjusted my code so that the probabilities always summed to 1. For example, I had an instance where both teams had a 55% chance to win. This obviously isn't what I want my model to return. Depending on how you write your code, you might get an unfavorable probability in your submission file.", "748491": "Okay. Thanks for the feedback! Will have to think about the best way to approach it.", "748585": "I'm taking the probability of A / (probability of A + probability of B) to get my results. It will always sum to 1. But I haven't quantified this approach. There might be a better way - possibly a weighting. I'll think about it and post here if I change it up. \n\nA random thought would be that you could predict the games each way and use one submission for each attempt. I *might* look into that.", "748709": "Ahh interesting idea. I'm currently predicting every game each way and then just taking the average. I was thinking about trying to use the max(A,B) to make a more aggressive model. I'll report back if I give it a shot.", "748731": "I don't know anything about your methodology but you have to be careful about the asymmetry of the results data. Because WTeam is always the winning team and LTeam is the losing team, that means that always WScore > LScore. You may want to consider flipping the data and using each game record twice, once with WTeam against LTeam, and once with LTeam against WTeam, so then the two columns are more like Team and OppTeam. The same if you split it out into Home Team and Away Team; the Home Team will do better on average against the Away Team. I think sometimes people use 2x the data, or sometimes they randomly decide, for each row, whether to use the data or the flipped data.", "767658": "Hi @jeffsonas, sorry just saw this! I was actually flipping my input data and using 2x the data to train and validate, which is part of the reason I was surprised to still see a difference. I thought that would make it a non-issue, but it seems like it's still happening." }, "source": "meta" }