I was going through the previous years' competition discussions , and saw a post by the competition host, which goes -
\n\n\n\n\nJust to remind people. The log-loss scoring function means you are penalized strongly for a prediction that is both confident and wrong. There isn't that much benefit you get from a super-confident correct prediction (such as 99%), versus a confident correct prediction (such as 95%). Because of this, many people choose to \"clip\" their predictions at some cutoff threshold like 10%/90%, or 5%/95%, or 1%/99%. You will have to decide on your own what the best risk level to take is. But anyone who predicted 99% or more for Virginia over UMBC last year (in the men's contest) was basically eliminated from the money as soon as the game completed, whereas the people who successfully predicted 90% for the other three #1 seeds, rather than 95% or 99%, lost a relatively small amount of ground. There are many schools of thought on this - but remember to take it into consideration while you still can! I am sure there was a huge population of participants last year who wished, in retrospect, that they had clipped their men's predictions more severely. I know I certainly did…
\n
I tried clipping my predictions , as
\n\nnp.clip(preds , 0.05 , 0.95)
and my score was slightly worse after clipping. Maybe we can get a better score with different thresholds? What do you guys think? Share your experiments / results here!
", "messages": [ { "id": "747717", "postDate": "02/16/2020 18:59:28", "content": "I was going through the previous years' competition discussions , and saw a post by the competition host, which goes -
\n\n\n\n\nJust to remind people. The log-loss scoring function means you are penalized strongly for a prediction that is both confident and wrong. There isn't that much benefit you get from a super-confident correct prediction (such as 99%), versus a confident correct prediction (such as 95%). Because of this, many people choose to \"clip\" their predictions at some cutoff threshold like 10%/90%, or 5%/95%, or 1%/99%. You will have to decide on your own what the best risk level to take is. But anyone who predicted 99% or more for Virginia over UMBC last year (in the men's contest) was basically eliminated from the money as soon as the game completed, whereas the people who successfully predicted 90% for the other three #1 seeds, rather than 95% or 99%, lost a relatively small amount of ground. There are many schools of thought on this - but remember to take it into consideration while you still can! I am sure there was a huge population of participants last year who wished, in retrospect, that they had clipped their men's predictions more severely. I know I certainly did…
\n
I tried clipping my predictions , as
\n\nnp.clip(preds , 0.05 , 0.95)
and my score was slightly worse after clipping. Maybe we can get a better score with different thresholds? What do you guys think? Share your experiments / results here!
", "rawMarkdown": "I was going through the previous years' competition discussions , and saw a post by the competition host, which goes -\n\n> Just to remind people. The log-loss scoring function means you are penalized strongly for a prediction that is both confident and wrong. There isn't that much benefit you get from a super-confident correct prediction (such as 99%), versus a confident correct prediction (such as 95%). Because of this, many people choose to \"clip\" their predictions at some cutoff threshold like 10%/90%, or 5%/95%, or 1%/99%. You will have to decide on your own what the best risk level to take is. But anyone who predicted 99% or more for Virginia over UMBC last year (in the men's contest) was basically eliminated from the money as soon as the game completed, whereas the people who successfully predicted 90% for the other three #1 seeds, rather than 95% or 99%, lost a relatively small amount of ground. There are many schools of thought on this - but remember to take it into consideration while you still can! I am sure there was a huge population of participants last year who wished, in retrospect, that they had clipped their men's predictions more severely. I know I certainly did…\n\nI tried clipping my predictions , as \n\n`np.clip(preds , 0.05 , 0.95)`\n\nand my score was slightly worse after clipping. Maybe we can get a better score with different thresholds? What do you guys think? Share your experiments / results here!", "votes": null }, { "id": "747756", "postDate": "02/16/2020 20:01:23", "content": "It's really a risk mitigation strategy. Clipping may not improve your average score across the five years, but it probably gives you a better chance of avoiding catastrophe (from a huge upset like #16 UMBC over #1 Virgina) at a relatively small cost. If your model already had very few 95%+ predictions, then clipping won't do much for you. And a lot depends on what others are doing. There is certainly a counter-argument, that if you want to stand out from the crowd, you really need to do something extreme, and staying conservative via clipping is just going to lump you in with the others. You might even dilate your predictions away from 50%, to try and squeeze out a few more logloss increments. Again, everyone has to decide for themselves how to handle this situation, but it is best to be aware of it.
", "rawMarkdown": "It's really a risk mitigation strategy. Clipping may not improve your average score across the five years, but it probably gives you a better chance of avoiding catastrophe (from a huge upset like #16 UMBC over #1 Virgina) at a relatively small cost. If your model already had very few 95%+ predictions, then clipping won't do much for you. And a lot depends on what others are doing. There is certainly a counter-argument, that if you want to stand out from the crowd, you really need to do something extreme, and staying conservative via clipping is just going to lump you in with the others. You might even dilate your predictions away from 50%, to try and squeeze out a few more logloss increments. Again, everyone has to decide for themselves how to handle this situation, but it is best to be aware of it.", "votes": null }, { "id": "747764", "postDate": "02/16/2020 20:17:11", "content": "This is a risk tolerance issue. One interesting feature of this comp is that everyone gets two submissions. It might make sense to clip one submission and leave the other one although I think there are better options with a second submission.\nThe reason to clip is prevent against catastrophe in the event of a major upset. This year, more than most seems like some top team will get upset unexpectedly.\nIf you look at https://www.masseyratings.com/ranks?s=cb you'll see a live composite of the Massey ordinals. \nGonzaga is the #2 team but one rating system has them at 18th.\nSan Diego St (the only undefeated team in the nation) is as high as 1st and as low as 26th depending on the system. \nA lot of sports talk radio is expressing that they don't know who's good this year because of all of the top ranked teams losing.
\n\nGiven that this event happens annually, there are only two final submissions and the difference between 99% and right to 98% and right is so minimal vs 99% and wrong and 98% and wrong, I can't see doing anything but clipping at least minimally.
", "rawMarkdown": "This is a risk tolerance issue. One interesting feature of this comp is that everyone gets two submissions. It might make sense to clip one submission and leave the other one although I think there are better options with a second submission.\nThe reason to clip is prevent against catastrophe in the event of a major upset. This year, more than most seems like some top team will get upset unexpectedly.\nIf you look at https://www.masseyratings.com/ranks?s=cb you'll see a live composite of the Massey ordinals. \nGonzaga is the #2 team but one rating system has them at 18th.\nSan Diego St (the only undefeated team in the nation) is as high as 1st and as low as 26th depending on the system. \nA lot of sports talk radio is expressing that they don't know who's good this year because of all of the top ranked teams losing. \n\nGiven that this event happens annually, there are only two final submissions and the difference between 99% and right to 98% and right is so minimal vs 99% and wrong and 98% and wrong, I can't see doing anything but clipping at least minimally.", "votes": null }, { "id": "752533", "postDate": "02/21/2020 05:53:25", "content": "In some simulations that I have done, I've found that clipping tends to keep you from sinking all the way to the bottom, but it (along with any other conservative strategy) also prevents you from having a chance at winning.
\n\nIn fact, I've found that someone with the absolutely correct probabilities can usually get into the top 10%, but will rarely win. There are too many other people out there who have roughly the right probabilities and the noise in their estimates and the small sample size helps some number of them outperform the true probabilities in this contest. To win, you have to intentionally be more aggressive and have luck on your side.
", "rawMarkdown": "In some simulations that I have done, I've found that clipping tends to keep you from sinking all the way to the bottom, but it (along with any other conservative strategy) also prevents you from having a chance at winning.\n\nIn fact, I've found that someone with the absolutely correct probabilities can usually get into the top 10%, but will rarely win. There are too many other people out there who have roughly the right probabilities and the noise in their estimates and the small sample size helps some number of them outperform the true probabilities in this contest. To win, you have to intentionally be more aggressive and have luck on your side.", "votes": null } ], "comments": [ { "id": 747756, "author_name": "jeffsonas", "author_url": "", "post_date": "02/16/2020 20:01:23", "content": "It's really a risk mitigation strategy. Clipping may not improve your average score across the five years, but it probably gives you a better chance of avoiding catastrophe (from a huge upset like #16 UMBC over #1 Virgina) at a relatively small cost. If your model already had very few 95%+ predictions, then clipping won't do much for you. And a lot depends on what others are doing. There is certainly a counter-argument, that if you want to stand out from the crowd, you really need to do something extreme, and staying conservative via clipping is just going to lump you in with the others. You might even dilate your predictions away from 50%, to try and squeeze out a few more logloss increments. Again, everyone has to decide for themselves how to handle this situation, but it is best to be aware of it.
", "votes": null, "replies": [ { "id": 752533, "author_name": "goodspellr", "author_url": "", "post_date": "02/21/2020 05:53:25", "content": "In some simulations that I have done, I've found that clipping tends to keep you from sinking all the way to the bottom, but it (along with any other conservative strategy) also prevents you from having a chance at winning.
\n\nIn fact, I've found that someone with the absolutely correct probabilities can usually get into the top 10%, but will rarely win. There are too many other people out there who have roughly the right probabilities and the noise in their estimates and the small sample size helps some number of them outperform the true probabilities in this contest. To win, you have to intentionally be more aggressive and have luck on your side.
", "votes": null, "replies": [] } ] }, { "id": 747764, "author_name": "btesch", "author_url": "", "post_date": "02/16/2020 20:17:11", "content": "This is a risk tolerance issue. One interesting feature of this comp is that everyone gets two submissions. It might make sense to clip one submission and leave the other one although I think there are better options with a second submission.\nThe reason to clip is prevent against catastrophe in the event of a major upset. This year, more than most seems like some top team will get upset unexpectedly.\nIf you look at https://www.masseyratings.com/ranks?s=cb you'll see a live composite of the Massey ordinals. \nGonzaga is the #2 team but one rating system has them at 18th.\nSan Diego St (the only undefeated team in the nation) is as high as 1st and as low as 26th depending on the system. \nA lot of sports talk radio is expressing that they don't know who's good this year because of all of the top ranked teams losing.
\n\nGiven that this event happens annually, there are only two final submissions and the difference between 99% and right to 98% and right is so minimal vs 99% and wrong and 98% and wrong, I can't see doing anything but clipping at least minimally.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "747717": "I was going through the previous years' competition discussions , and saw a post by the competition host, which goes -\n\n> Just to remind people. The log-loss scoring function means you are penalized strongly for a prediction that is both confident and wrong. There isn't that much benefit you get from a super-confident correct prediction (such as 99%), versus a confident correct prediction (such as 95%). Because of this, many people choose to \"clip\" their predictions at some cutoff threshold like 10%/90%, or 5%/95%, or 1%/99%. You will have to decide on your own what the best risk level to take is. But anyone who predicted 99% or more for Virginia over UMBC last year (in the men's contest) was basically eliminated from the money as soon as the game completed, whereas the people who successfully predicted 90% for the other three #1 seeds, rather than 95% or 99%, lost a relatively small amount of ground. There are many schools of thought on this - but remember to take it into consideration while you still can! I am sure there was a huge population of participants last year who wished, in retrospect, that they had clipped their men's predictions more severely. I know I certainly did…\n\nI tried clipping my predictions , as \n\n`np.clip(preds , 0.05 , 0.95)`\n\nand my score was slightly worse after clipping. Maybe we can get a better score with different thresholds? What do you guys think? Share your experiments / results here!", "747756": "It's really a risk mitigation strategy. Clipping may not improve your average score across the five years, but it probably gives you a better chance of avoiding catastrophe (from a huge upset like #16 UMBC over #1 Virgina) at a relatively small cost. If your model already had very few 95%+ predictions, then clipping won't do much for you. And a lot depends on what others are doing. There is certainly a counter-argument, that if you want to stand out from the crowd, you really need to do something extreme, and staying conservative via clipping is just going to lump you in with the others. You might even dilate your predictions away from 50%, to try and squeeze out a few more logloss increments. Again, everyone has to decide for themselves how to handle this situation, but it is best to be aware of it.", "747764": "This is a risk tolerance issue. One interesting feature of this comp is that everyone gets two submissions. It might make sense to clip one submission and leave the other one although I think there are better options with a second submission.\nThe reason to clip is prevent against catastrophe in the event of a major upset. This year, more than most seems like some top team will get upset unexpectedly.\nIf you look at https://www.masseyratings.com/ranks?s=cb you'll see a live composite of the Massey ordinals. \nGonzaga is the #2 team but one rating system has them at 18th.\nSan Diego St (the only undefeated team in the nation) is as high as 1st and as low as 26th depending on the system. \nA lot of sports talk radio is expressing that they don't know who's good this year because of all of the top ranked teams losing. \n\nGiven that this event happens annually, there are only two final submissions and the difference between 99% and right to 98% and right is so minimal vs 99% and wrong and 98% and wrong, I can't see doing anything but clipping at least minimally.", "752533": "In some simulations that I have done, I've found that clipping tends to keep you from sinking all the way to the bottom, but it (along with any other conservative strategy) also prevents you from having a chance at winning.\n\nIn fact, I've found that someone with the absolutely correct probabilities can usually get into the top 10%, but will rarely win. There are too many other people out there who have roughly the right probabilities and the noise in their estimates and the small sample size helps some number of them outperform the true probabilities in this contest. To win, you have to intentionally be more aggressive and have luck on your side." }, "source": "meta" }