I've looked at a lot of different approaches on how to model this problem. Because the first comp was in 2014, I trained a model(s) that used only the data up to 2013 and did not retrain it. After a lot of parameter tuning, I was able to get results that would have yielded places:\n2014: 1st\n2015: 10th\n2016: 1st\n2017: 4th\n2018: 3rd\n2019: 331st
\n\nTwo thoughts, first being that I didn't think this approach would do this well. Second, what is going on with last year? It had the lowest winning score ever, and the lowest score at the 10th, 25th, and 50th percentile. It also had 40 submissions finish with the exact same score - 0.49527 - which I can only assume was a public notebook submission. \nIs anyone else seeing this in their approach? Is last year difficult to model in relation to other years? Even when I retrain my model with new data, 2019 seems to be an enigma. I can get into the top 15% but that seems to be limits.
", "messages": [ { "id": "752976", "postDate": "02/21/2020 15:34:01", "content": "I've looked at a lot of different approaches on how to model this problem. Because the first comp was in 2014, I trained a model(s) that used only the data up to 2013 and did not retrain it. After a lot of parameter tuning, I was able to get results that would have yielded places:\n2014: 1st\n2015: 10th\n2016: 1st\n2017: 4th\n2018: 3rd\n2019: 331st
\n\nTwo thoughts, first being that I didn't think this approach would do this well. Second, what is going on with last year? It had the lowest winning score ever, and the lowest score at the 10th, 25th, and 50th percentile. It also had 40 submissions finish with the exact same score - 0.49527 - which I can only assume was a public notebook submission. \nIs anyone else seeing this in their approach? Is last year difficult to model in relation to other years? Even when I retrain my model with new data, 2019 seems to be an enigma. I can get into the top 15% but that seems to be limits.
", "rawMarkdown": "I've looked at a lot of different approaches on how to model this problem. Because the first comp was in 2014, I trained a model(s) that used only the data up to 2013 and did not retrain it. After a lot of parameter tuning, I was able to get results that would have yielded places:\n2014: 1st\n2015: 10th\n2016: 1st\n2017: 4th\n2018: 3rd\n2019: 331st\n\nTwo thoughts, first being that I didn't think this approach would do this well. Second, what is going on with last year? It had the lowest winning score ever, and the lowest score at the 10th, 25th, and 50th percentile. It also had 40 submissions finish with the exact same score - 0.49527 - which I can only assume was a public notebook submission. \nIs anyone else seeing this in their approach? Is last year difficult to model in relation to other years? Even when I retrain my model with new data, 2019 seems to be an enigma. I can get into the top 15% but that seems to be limits.", "votes": null }, { "id": "753025", "postDate": "02/21/2020 16:11:48", "content": "I'm not too sure why this would happen but I have some ideas/suggestions. You may want to think about the players and the game for a situation like this. I'm not sure of all the rule changes or years but I know recently college shortened their shot clock from 35 to 30 seconds and they moved to the NBA 3 distance. Maybe some of the rules had an effect on this.
\n\nAlso as far as the 2019 tournamnet goes, if I remember correctly that was a year when many of the veteran teams did well and outperformed the one and done freshman teams, maybe that helps.
", "rawMarkdown": "I'm not too sure why this would happen but I have some ideas/suggestions. You may want to think about the players and the game for a situation like this. I'm not sure of all the rule changes or years but I know recently college shortened their shot clock from 35 to 30 seconds and they moved to the NBA 3 distance. Maybe some of the rules had an effect on this. \n\nAlso as far as the 2019 tournamnet goes, if I remember correctly that was a year when many of the veteran teams did well and outperformed the one and done freshman teams, maybe that helps.", "votes": null }, { "id": "753735", "postDate": "02/22/2020 15:54:52", "content": "Was 2019 full of upsets? I think upsets are the most risky factor here , and can swing your score by a huge margin. Try clipping your predictions for 2019 , and then check the results maybe? That would confirm it.
", "rawMarkdown": "Was 2019 full of upsets? I think upsets are the most risky factor here , and can swing your score by a huge margin. Try clipping your predictions for 2019 , and then check the results maybe? That would confirm it.", "votes": null }, { "id": "753755", "postDate": "02/22/2020 16:16:26", "content": "Last year was the lowest logloss score in the history of the tourn so I'm 99% certain it wasn't full of major upsets. A 13 beat a 4 - which would be it unless you count 12 beating 5's but that happens every year. I ran a correlation analysis for 2014-18 winners scores and the avg rank difference of a win and they are highly correlated. More upsets equal higher leaderboard scores. I think that might be my problem - I don't use seeding in my model.
", "rawMarkdown": "Last year was the lowest logloss score in the history of the tourn so I'm 99% certain it wasn't full of major upsets. A 13 beat a 4 - which would be it unless you count 12 beating 5's but that happens every year. I ran a correlation analysis for 2014-18 winners scores and the avg rank difference of a win and they are highly correlated. More upsets equal higher leaderboard scores. I think that might be my problem - I don't use seeding in my model.", "votes": null }, { "id": "753979", "postDate": "02/22/2020 22:15:17", "content": "I see something similar in my model, although your retrofitted third place in 2018 surprises me. I use the same model that gave me place 220 in 2018 (top 24%), a year that was full of major upsets. If I had used this model in previous competitions, it could have led to places
\n\n2014: 5th\n2015: 4th\n2016: 2nd\n2017: 10th
\n\nI didn't participate last year where it would have achieved a very meager 546th place. Last year had no major upsets, the fraction of games the model got wrong and the probabilities of those games are also unexciting. Overall, the model's performance is on par with 2014-2017. My conclusion is that the competitive environment changed and that a model that would have performed reasonably well in the past is not competitive anymore.
", "rawMarkdown": "I see something similar in my model, although your retrofitted third place in 2018 surprises me. I use the same model that gave me place 220 in 2018 (top 24%), a year that was full of major upsets. If I had used this model in previous competitions, it could have led to places\n\n2014: 5th\n2015: 4th\n2016: 2nd\n2017: 10th\n\nI didn't participate last year where it would have achieved a very meager 546th place. Last year had no major upsets, the fraction of games the model got wrong and the probabilities of those games are also unexciting. Overall, the model's performance is on par with 2014-2017. My conclusion is that the competitive environment changed and that a model that would have performed reasonably well in the past is not competitive anymore.", "votes": null }, { "id": "754763", "postDate": "02/24/2020 03:01:44", "content": "After a deep dive, I've been able to resolve some of the weirdness of my model from last year. If you read one of my other replies, I had it figured out. In updating my model, I brought 2019 down to competitive levels but at the cost of other year's results. It models in the top 10%-20% consistently no matter how I tune the nobs. I still haven't been able to crack .4147 though. Every year is a different animal so a worthwhile goal is to create a model that will respond well if there are a lot of upsets or not.
", "rawMarkdown": "After a deep dive, I've been able to resolve some of the weirdness of my model from last year. If you read one of my other replies, I had it figured out. In updating my model, I brought 2019 down to competitive levels but at the cost of other year's results. It models in the top 10%-20% consistently no matter how I tune the nobs. I still haven't been able to crack .4147 though. Every year is a different animal so a worthwhile goal is to create a model that will respond well if there are a lot of upsets or not.", "votes": null }, { "id": "757663", "postDate": "02/27/2020 02:15:29", "content": "Last year's winner 'bet' on some first round games, overriding the model. That can help a lot but really hard to generalize and big potential luck factors.
", "rawMarkdown": "Last year's winner 'bet' on some first round games, overriding the model. That can help a lot but really hard to generalize and big potential luck factors.", "votes": null }, { "id": "757677", "postDate": "02/27/2020 02:33:22", "content": "Funny. I was actually thinking about that exact idea yesterday and the legitimacy of doing it. I guess you don't have to declare it as data. It's still a dubious practice in my opinion. I think I'd rather live or die with my own abilities over those of someone else's. But it seems to have paid off - at least for the winner.
", "rawMarkdown": "Funny. I was actually thinking about that exact idea yesterday and the legitimacy of doing it. I guess you don't have to declare it as data. It's still a dubious practice in my opinion. I think I'd rather live or die with my own abilities over those of someone else's. But it seems to have paid off - at least for the winner.", "votes": null }, { "id": "760773", "postDate": "03/01/2020 17:53:34", "content": "Yeah, it's admirable to stand by the predictions of your model. But thinking of it as a competition and tricks for prevailing seems to not entirely lie outside the spirit of the competition (I think.)
", "rawMarkdown": "Yeah, it's admirable to stand by the predictions of your model. But thinking of it as a competition and tricks for prevailing seems to not entirely lie outside the spirit of the competition (I think.)", "votes": null } ], "comments": [ { "id": 753025, "author_name": "proselotis", "author_url": "", "post_date": "02/21/2020 16:11:48", "content": "I'm not too sure why this would happen but I have some ideas/suggestions. You may want to think about the players and the game for a situation like this. I'm not sure of all the rule changes or years but I know recently college shortened their shot clock from 35 to 30 seconds and they moved to the NBA 3 distance. Maybe some of the rules had an effect on this.
\n\nAlso as far as the 2019 tournamnet goes, if I remember correctly that was a year when many of the veteran teams did well and outperformed the one and done freshman teams, maybe that helps.
", "votes": null, "replies": [] }, { "id": 753735, "author_name": "p4rallax", "author_url": "", "post_date": "02/22/2020 15:54:52", "content": "Was 2019 full of upsets? I think upsets are the most risky factor here , and can swing your score by a huge margin. Try clipping your predictions for 2019 , and then check the results maybe? That would confirm it.
", "votes": null, "replies": [] }, { "id": 753755, "author_name": "btesch", "author_url": "", "post_date": "02/22/2020 16:16:26", "content": "Last year was the lowest logloss score in the history of the tourn so I'm 99% certain it wasn't full of major upsets. A 13 beat a 4 - which would be it unless you count 12 beating 5's but that happens every year. I ran a correlation analysis for 2014-18 winners scores and the avg rank difference of a win and they are highly correlated. More upsets equal higher leaderboard scores. I think that might be my problem - I don't use seeding in my model.
", "votes": null, "replies": [ { "id": 757663, "author_name": "scharlesworth", "author_url": "", "post_date": "02/27/2020 02:15:29", "content": "Last year's winner 'bet' on some first round games, overriding the model. That can help a lot but really hard to generalize and big potential luck factors.
", "votes": null, "replies": [] }, { "id": 757677, "author_name": "btesch", "author_url": "", "post_date": "02/27/2020 02:33:22", "content": "Funny. I was actually thinking about that exact idea yesterday and the legitimacy of doing it. I guess you don't have to declare it as data. It's still a dubious practice in my opinion. I think I'd rather live or die with my own abilities over those of someone else's. But it seems to have paid off - at least for the winner.
", "votes": null, "replies": [] }, { "id": 760773, "author_name": "scharlesworth", "author_url": "", "post_date": "03/01/2020 17:53:34", "content": "Yeah, it's admirable to stand by the predictions of your model. But thinking of it as a competition and tricks for prevailing seems to not entirely lie outside the spirit of the competition (I think.)
", "votes": null, "replies": [] } ] }, { "id": 753979, "author_name": "joergdietrich", "author_url": "", "post_date": "02/22/2020 22:15:17", "content": "I see something similar in my model, although your retrofitted third place in 2018 surprises me. I use the same model that gave me place 220 in 2018 (top 24%), a year that was full of major upsets. If I had used this model in previous competitions, it could have led to places
\n\n2014: 5th\n2015: 4th\n2016: 2nd\n2017: 10th
\n\nI didn't participate last year where it would have achieved a very meager 546th place. Last year had no major upsets, the fraction of games the model got wrong and the probabilities of those games are also unexciting. Overall, the model's performance is on par with 2014-2017. My conclusion is that the competitive environment changed and that a model that would have performed reasonably well in the past is not competitive anymore.
", "votes": null, "replies": [] }, { "id": 754763, "author_name": "btesch", "author_url": "", "post_date": "02/24/2020 03:01:44", "content": "After a deep dive, I've been able to resolve some of the weirdness of my model from last year. If you read one of my other replies, I had it figured out. In updating my model, I brought 2019 down to competitive levels but at the cost of other year's results. It models in the top 10%-20% consistently no matter how I tune the nobs. I still haven't been able to crack .4147 though. Every year is a different animal so a worthwhile goal is to create a model that will respond well if there are a lot of upsets or not.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "752976": "I've looked at a lot of different approaches on how to model this problem. Because the first comp was in 2014, I trained a model(s) that used only the data up to 2013 and did not retrain it. After a lot of parameter tuning, I was able to get results that would have yielded places:\n2014: 1st\n2015: 10th\n2016: 1st\n2017: 4th\n2018: 3rd\n2019: 331st\n\nTwo thoughts, first being that I didn't think this approach would do this well. Second, what is going on with last year? It had the lowest winning score ever, and the lowest score at the 10th, 25th, and 50th percentile. It also had 40 submissions finish with the exact same score - 0.49527 - which I can only assume was a public notebook submission. \nIs anyone else seeing this in their approach? Is last year difficult to model in relation to other years? Even when I retrain my model with new data, 2019 seems to be an enigma. I can get into the top 15% but that seems to be limits.", "753025": "I'm not too sure why this would happen but I have some ideas/suggestions. You may want to think about the players and the game for a situation like this. I'm not sure of all the rule changes or years but I know recently college shortened their shot clock from 35 to 30 seconds and they moved to the NBA 3 distance. Maybe some of the rules had an effect on this. \n\nAlso as far as the 2019 tournamnet goes, if I remember correctly that was a year when many of the veteran teams did well and outperformed the one and done freshman teams, maybe that helps.", "753735": "Was 2019 full of upsets? I think upsets are the most risky factor here , and can swing your score by a huge margin. Try clipping your predictions for 2019 , and then check the results maybe? That would confirm it.", "753755": "Last year was the lowest logloss score in the history of the tourn so I'm 99% certain it wasn't full of major upsets. A 13 beat a 4 - which would be it unless you count 12 beating 5's but that happens every year. I ran a correlation analysis for 2014-18 winners scores and the avg rank difference of a win and they are highly correlated. More upsets equal higher leaderboard scores. I think that might be my problem - I don't use seeding in my model.", "753979": "I see something similar in my model, although your retrofitted third place in 2018 surprises me. I use the same model that gave me place 220 in 2018 (top 24%), a year that was full of major upsets. If I had used this model in previous competitions, it could have led to places\n\n2014: 5th\n2015: 4th\n2016: 2nd\n2017: 10th\n\nI didn't participate last year where it would have achieved a very meager 546th place. Last year had no major upsets, the fraction of games the model got wrong and the probabilities of those games are also unexciting. Overall, the model's performance is on par with 2014-2017. My conclusion is that the competitive environment changed and that a model that would have performed reasonably well in the past is not competitive anymore.", "754763": "After a deep dive, I've been able to resolve some of the weirdness of my model from last year. If you read one of my other replies, I had it figured out. In updating my model, I brought 2019 down to competitive levels but at the cost of other year's results. It models in the top 10%-20% consistently no matter how I tune the nobs. I still haven't been able to crack .4147 though. Every year is a different animal so a worthwhile goal is to create a model that will respond well if there are a lot of upsets or not.", "757663": "Last year's winner 'bet' on some first round games, overriding the model. That can help a lot but really hard to generalize and big potential luck factors.", "757677": "Funny. I was actually thinking about that exact idea yesterday and the legitimacy of doing it. I guess you don't have to declare it as data. It's still a dubious practice in my opinion. I think I'd rather live or die with my own abilities over those of someone else's. But it seems to have paid off - at least for the winner.", "760773": "Yeah, it's admirable to stand by the predictions of your model. But thinking of it as a competition and tricks for prevailing seems to not entirely lie outside the spirit of the competition (I think.)" }, "source": "meta" }