First off, I know the people who are scoring 0.00000 are simply submitting the ground truth, I'm sure they are all aware of this and are just trolling.
\n\nThis is for the people scoring below 0.46, please note that your model almost certainly contains an information leak or at the very least is severely overfit. The winning submissions 2015-2019 have scored:\n- 2015: 0.43893\n- 2016: 0.48130\n- 2017: 0.43857\n- 2018: 0.53194\n- 2019: 0.41477
\n\nIf your model was as good as the winning submission in each of the last 5 years, you would score 0.46110. If you are scoring less than this on the current leaderboard please do check your model for leakage
", "messages": [ { "id": "748675", "postDate": "02/17/2020 20:25:08", "content": "First off, I know the people who are scoring 0.00000 are simply submitting the ground truth, I'm sure they are all aware of this and are just trolling.
\n\nThis is for the people scoring below 0.46, please note that your model almost certainly contains an information leak or at the very least is severely overfit. The winning submissions 2015-2019 have scored:\n- 2015: 0.43893\n- 2016: 0.48130\n- 2017: 0.43857\n- 2018: 0.53194\n- 2019: 0.41477
\n\nIf your model was as good as the winning submission in each of the last 5 years, you would score 0.46110. If you are scoring less than this on the current leaderboard please do check your model for leakage
", "rawMarkdown": "First off, I know the people who are scoring 0.00000 are simply submitting the ground truth, I'm sure they are all aware of this and are just trolling.\n\nThis is for the people scoring below 0.46, please note that your model almost certainly contains an **information leak** or at the very least is severely overfit. The **winning** submissions 2015-2019 have scored:\n- 2015: 0.43893\n- 2016: 0.48130\n- 2017: 0.43857\n- 2018: 0.53194\n- 2019: 0.41477\n\nIf your model was as good as the winning submission in each of the last 5 years, you would score 0.46110. If you are scoring less than this on the current leaderboard please do check your model for leakage", "votes": null }, { "id": "748701", "postDate": "02/17/2020 21:43:24", "content": "Another way to look at this is to identify the specific contest participants who have done best over the years in the real Stage 2 contest, and to look at their scores across the same five-year period. The gold standard in our contests has certainly been the team \"One Shining MGF\", made up of Michael J. Lopez and Gregory Matthews, although last year Michael Lopez did not participate and the team was named \"Mike Lopez Abandoned Me\". They won the first contest in 2014, and had three top-15 finishes in the five years since then, which matches the five-year timespan of our Stage 1 submission file.
\n\nTheir Stage 2 contest scores across those five years were:\n2015: 0.45202\n2016: 0.51318\n2017: 0.47748\n2018: 0.59325\n2019: 0.43873\nAverage across five years: 0.494932
\n\nI couldn't find any other competitors that had broken the 0.5 average score across that time range. It's kind of an informal sample, because team names and emails change over the years, as have team compositions themselves. But I did find five other competitors who averaged better than 0.52 across their Stage 2 scores in the five contests between 2015 and 2019:\n PEC: 0.5095\n Yale Bulldogs: 0.5115\n Monte McNair: 0.5129\n Zach: 0.5138\n Jeff Sonas: 0.5189
\n\nAnd if you take the average Stage 2 score each year from those six competitors, you get these:\n2015: 0.475455\n2016: 0.546592\n2017: 0.477630\n2018: 0.587862\n2019: 0.463723\nAverage across five years: 0.510252
\n\nA couple of other things to acknowledge here. In Stage 2, you get two submissions to play with, which certainly improves your expected score. For instance, you can submit the exact same set of predictions, but take one 50-50 game and predict a different winner with 100% confidence in each of your two. And also, the 2018 tournament was crazy because of the #16 over #1 upset, so it became very important to have not made too confident a prediction there.
\n\nI do think that any Stage 1 average score below 0.48 is almost certainly benefiting from leakage, even if just overfitting. As I said last year on this topic at this point, you should not be able to beat \"One Shining MGF\" by such a large margin.
", "rawMarkdown": "Another way to look at this is to identify the specific contest participants who have done best over the years in the real Stage 2 contest, and to look at their scores across the same five-year period. The gold standard in our contests has certainly been the team \"One Shining MGF\", made up of Michael J. Lopez and Gregory Matthews, although last year Michael Lopez did not participate and the team was named \"Mike Lopez Abandoned Me\". They won the first contest in 2014, and had three top-15 finishes in the five years since then, which matches the five-year timespan of our Stage 1 submission file.\n\nTheir Stage 2 contest scores across those five years were:\n2015: 0.45202\n2016: 0.51318\n2017: 0.47748\n2018: 0.59325\n2019: 0.43873\nAverage across five years: 0.494932\n\nI couldn't find any other competitors that had broken the 0.5 average score across that time range. It's kind of an informal sample, because team names and emails change over the years, as have team compositions themselves. But I did find five other competitors who averaged better than 0.52 across their Stage 2 scores in the five contests between 2015 and 2019:\n PEC: 0.5095\n Yale Bulldogs: 0.5115\n Monte McNair: 0.5129\n Zach: 0.5138\n Jeff Sonas: 0.5189\n\nAnd if you take the average Stage 2 score each year from those six competitors, you get these:\n2015: 0.475455\n2016: 0.546592\n2017: 0.477630\n2018: 0.587862\n2019: 0.463723\nAverage across five years: 0.510252\n\nA couple of other things to acknowledge here. In Stage 2, you get two submissions to play with, which certainly improves your expected score. For instance, you can submit the exact same set of predictions, but take one 50-50 game and predict a different winner with 100% confidence in each of your two. And also, the 2018 tournament was crazy because of the #16 over #1 upset, so it became very important to have not made too confident a prediction there.\n\t\t\t\t\nI do think that any Stage 1 average score below 0.48 is almost certainly benefiting from leakage, even if just overfitting. As I said last year on this topic at this point, you should not be able to beat \"One Shining MGF\" by such a large margin.", "votes": null }, { "id": "748705", "postDate": "02/17/2020 21:52:34", "content": "2014: .5295
\n\nIf anyone wants to know what a reasonable winning score would have been prior to 2014, look at the correlation between the avg diff of the winner's seed and the loser's seed and the bracket challenge winner's score. There's a very high correlation and it's easy to model.
", "rawMarkdown": "2014: .5295\n\nIf anyone wants to know what a reasonable winning score would have been prior to 2014, look at the correlation between the avg diff of the winner's seed and the loser's seed and the bracket challenge winner's score. There's a very high correlation and it's easy to model.", "votes": null }, { "id": "751562", "postDate": "02/20/2020 09:28:42", "content": "This is very informative, thank you so much.
\n\nCould you please clarify a little more on this part?
\n\n\n\n\nIn Stage 2, you get two submissions to play with
\n
What do you mean by two submissions?
", "rawMarkdown": "This is very informative, thank you so much.\n\nCould you please clarify a little more on this part?\n\n> In Stage 2, you get two submissions to play with\n\nWhat do you mean by two submissions?", "votes": null }, { "id": "751703", "postDate": "02/20/2020 12:27:26", "content": "@nroman It's just the same as other competitions, you can submit 2. However in this competition luck plays a big role too, so you can choose what results to override in those 2 subs.
", "rawMarkdown": "nroman It's just the same as other competitions, you can submit 2. However in this competition luck plays a big role too, so you can choose what results to override in those 2 subs.", "votes": null }, { "id": "751739", "postDate": "02/20/2020 13:14:52", "content": "I realized that while we were just training different models and waiting for the new data of the second stage.\nHow can we build specially a bad model without new data?
\n\nAlthough I found some valuable advice on this (thanks to @nroman):\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539
", "rawMarkdown": "I realized that while we were just training different models and waiting for the new data of the second stage.\nHow can we build specially a bad model without new data?\n\nAlthough I found some valuable advice on this (thanks to @nroman):\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539", "votes": null }, { "id": "753143", "postDate": "02/21/2020 18:58:03", "content": "You will only have a few days during Stage 2 in which to submit prediction files. Pairings are announced on Sunday afternoon, and we will release the Stage 2 data (including a Stage 2 sample submission file) either Sunday night or Monday. And the round of 64 starts Thursday morning. You can actually submit as many prediction files as you want during those few days, but before the submission deadline, you must pick at most two of your submitted files to be your official \"selected\" submissions. This means everyone has two shots at winning. Even if one of your other submissions does great, it doesn't matter because you didn't select it beforehand. You can either have your two entries be highly correlated with minor tweaks between them, or you can have them be staggeringly different. Or anywhere in between; it's up to you.
", "rawMarkdown": "You will only have a few days during Stage 2 in which to submit prediction files. Pairings are announced on Sunday afternoon, and we will release the Stage 2 data (including a Stage 2 sample submission file) either Sunday night or Monday. And the round of 64 starts Thursday morning. You can actually submit as many prediction files as you want during those few days, but before the submission deadline, you must pick at most two of your submitted files to be your official \"selected\" submissions. This means everyone has two shots at winning. Even if one of your other submissions does great, it doesn't matter because you didn't select it beforehand. You can either have your two entries be highly correlated with minor tweaks between them, or you can have them be staggeringly different. Or anywhere in between; it's up to you.", "votes": null }, { "id": "754531", "postDate": "02/23/2020 17:48:48", "content": "Thank you for this!\nVery informative as 0.46 is a nice benchmark to have.
\n\nAs for me, this is my first NCAA Challenge and I clearly overfitted the heck out of this stage.\nTime to think about some nice validation strategy!
", "rawMarkdown": "Thank you for this!\nVery informative as **0.46** is a nice **benchmark** to have.\n\nAs for me, this is my first NCAA Challenge and I clearly overfitted the heck out of this stage.\nTime to think about some nice validation strategy!", "votes": null }, { "id": "754751", "postDate": "02/24/2020 02:24:46", "content": "", "rawMarkdown": "", "votes": null }, { "id": "756006", "postDate": "02/25/2020 10:44:30", "content": "Thank you
", "rawMarkdown": "Thank you", "votes": null }, { "id": "756198", "postDate": "02/25/2020 14:02:28", "content": "Thanks a lot, @jeffsonas, this is very informative.
", "rawMarkdown": "Thanks a lot, @jeffsonas, this is very informative.", "votes": null } ], "comments": [ { "id": 748701, "author_name": "jeffsonas", "author_url": "", "post_date": "02/17/2020 21:43:24", "content": "Another way to look at this is to identify the specific contest participants who have done best over the years in the real Stage 2 contest, and to look at their scores across the same five-year period. The gold standard in our contests has certainly been the team \"One Shining MGF\", made up of Michael J. Lopez and Gregory Matthews, although last year Michael Lopez did not participate and the team was named \"Mike Lopez Abandoned Me\". They won the first contest in 2014, and had three top-15 finishes in the five years since then, which matches the five-year timespan of our Stage 1 submission file.
\n\nTheir Stage 2 contest scores across those five years were:\n2015: 0.45202\n2016: 0.51318\n2017: 0.47748\n2018: 0.59325\n2019: 0.43873\nAverage across five years: 0.494932
\n\nI couldn't find any other competitors that had broken the 0.5 average score across that time range. It's kind of an informal sample, because team names and emails change over the years, as have team compositions themselves. But I did find five other competitors who averaged better than 0.52 across their Stage 2 scores in the five contests between 2015 and 2019:\n PEC: 0.5095\n Yale Bulldogs: 0.5115\n Monte McNair: 0.5129\n Zach: 0.5138\n Jeff Sonas: 0.5189
\n\nAnd if you take the average Stage 2 score each year from those six competitors, you get these:\n2015: 0.475455\n2016: 0.546592\n2017: 0.477630\n2018: 0.587862\n2019: 0.463723\nAverage across five years: 0.510252
\n\nA couple of other things to acknowledge here. In Stage 2, you get two submissions to play with, which certainly improves your expected score. For instance, you can submit the exact same set of predictions, but take one 50-50 game and predict a different winner with 100% confidence in each of your two. And also, the 2018 tournament was crazy because of the #16 over #1 upset, so it became very important to have not made too confident a prediction there.
\n\nI do think that any Stage 1 average score below 0.48 is almost certainly benefiting from leakage, even if just overfitting. As I said last year on this topic at this point, you should not be able to beat \"One Shining MGF\" by such a large margin.
", "votes": null, "replies": [ { "id": 751562, "author_name": "nroman", "author_url": "", "post_date": "02/20/2020 09:28:42", "content": "This is very informative, thank you so much.
\n\nCould you please clarify a little more on this part?
\n\n\n\n\nIn Stage 2, you get two submissions to play with
\n
What do you mean by two submissions?
", "votes": null, "replies": [] }, { "id": 751703, "author_name": "ajinomoto132", "author_url": "", "post_date": "02/20/2020 12:27:26", "content": "@nroman It's just the same as other competitions, you can submit 2. However in this competition luck plays a big role too, so you can choose what results to override in those 2 subs.
", "votes": null, "replies": [] }, { "id": 753143, "author_name": "jeffsonas", "author_url": "", "post_date": "02/21/2020 18:58:03", "content": "You will only have a few days during Stage 2 in which to submit prediction files. Pairings are announced on Sunday afternoon, and we will release the Stage 2 data (including a Stage 2 sample submission file) either Sunday night or Monday. And the round of 64 starts Thursday morning. You can actually submit as many prediction files as you want during those few days, but before the submission deadline, you must pick at most two of your submitted files to be your official \"selected\" submissions. This means everyone has two shots at winning. Even if one of your other submissions does great, it doesn't matter because you didn't select it beforehand. You can either have your two entries be highly correlated with minor tweaks between them, or you can have them be staggeringly different. Or anywhere in between; it's up to you.
", "votes": null, "replies": [] }, { "id": 756198, "author_name": "lepusarcticus", "author_url": "", "post_date": "02/25/2020 14:02:28", "content": "Thanks a lot, @jeffsonas, this is very informative.
", "votes": null, "replies": [] } ] }, { "id": 748705, "author_name": "btesch", "author_url": "", "post_date": "02/17/2020 21:52:34", "content": "2014: .5295
\n\nIf anyone wants to know what a reasonable winning score would have been prior to 2014, look at the correlation between the avg diff of the winner's seed and the loser's seed and the bracket challenge winner's score. There's a very high correlation and it's easy to model.
", "votes": null, "replies": [] }, { "id": 751739, "author_name": "vbmokin", "author_url": "", "post_date": "02/20/2020 13:14:52", "content": "I realized that while we were just training different models and waiting for the new data of the second stage.\nHow can we build specially a bad model without new data?
\n\nAlthough I found some valuable advice on this (thanks to @nroman):\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539
", "votes": null, "replies": [] }, { "id": 754531, "author_name": "roburadu", "author_url": "", "post_date": "02/23/2020 17:48:48", "content": "Thank you for this!\nVery informative as 0.46 is a nice benchmark to have.
\n\nAs for me, this is my first NCAA Challenge and I clearly overfitted the heck out of this stage.\nTime to think about some nice validation strategy!
", "votes": null, "replies": [] }, { "id": 754751, "author_name": "", "author_url": "", "post_date": "02/24/2020 02:24:46", "content": "", "votes": null, "replies": [] }, { "id": 756006, "author_name": "mkadeer2013", "author_url": "", "post_date": "02/25/2020 10:44:30", "content": "Thank you
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "748675": "First off, I know the people who are scoring 0.00000 are simply submitting the ground truth, I'm sure they are all aware of this and are just trolling.\n\nThis is for the people scoring below 0.46, please note that your model almost certainly contains an **information leak** or at the very least is severely overfit. The **winning** submissions 2015-2019 have scored:\n- 2015: 0.43893\n- 2016: 0.48130\n- 2017: 0.43857\n- 2018: 0.53194\n- 2019: 0.41477\n\nIf your model was as good as the winning submission in each of the last 5 years, you would score 0.46110. If you are scoring less than this on the current leaderboard please do check your model for leakage", "748701": "Another way to look at this is to identify the specific contest participants who have done best over the years in the real Stage 2 contest, and to look at their scores across the same five-year period. The gold standard in our contests has certainly been the team \"One Shining MGF\", made up of Michael J. Lopez and Gregory Matthews, although last year Michael Lopez did not participate and the team was named \"Mike Lopez Abandoned Me\". They won the first contest in 2014, and had three top-15 finishes in the five years since then, which matches the five-year timespan of our Stage 1 submission file.\n\nTheir Stage 2 contest scores across those five years were:\n2015: 0.45202\n2016: 0.51318\n2017: 0.47748\n2018: 0.59325\n2019: 0.43873\nAverage across five years: 0.494932\n\nI couldn't find any other competitors that had broken the 0.5 average score across that time range. It's kind of an informal sample, because team names and emails change over the years, as have team compositions themselves. But I did find five other competitors who averaged better than 0.52 across their Stage 2 scores in the five contests between 2015 and 2019:\n PEC: 0.5095\n Yale Bulldogs: 0.5115\n Monte McNair: 0.5129\n Zach: 0.5138\n Jeff Sonas: 0.5189\n\nAnd if you take the average Stage 2 score each year from those six competitors, you get these:\n2015: 0.475455\n2016: 0.546592\n2017: 0.477630\n2018: 0.587862\n2019: 0.463723\nAverage across five years: 0.510252\n\nA couple of other things to acknowledge here. In Stage 2, you get two submissions to play with, which certainly improves your expected score. For instance, you can submit the exact same set of predictions, but take one 50-50 game and predict a different winner with 100% confidence in each of your two. And also, the 2018 tournament was crazy because of the #16 over #1 upset, so it became very important to have not made too confident a prediction there.\n\t\t\t\t\nI do think that any Stage 1 average score below 0.48 is almost certainly benefiting from leakage, even if just overfitting. As I said last year on this topic at this point, you should not be able to beat \"One Shining MGF\" by such a large margin.", "748705": "2014: .5295\n\nIf anyone wants to know what a reasonable winning score would have been prior to 2014, look at the correlation between the avg diff of the winner's seed and the loser's seed and the bracket challenge winner's score. There's a very high correlation and it's easy to model.", "751562": "This is very informative, thank you so much.\n\nCould you please clarify a little more on this part?\n\n> In Stage 2, you get two submissions to play with\n\nWhat do you mean by two submissions?", "751703": "nroman It's just the same as other competitions, you can submit 2. However in this competition luck plays a big role too, so you can choose what results to override in those 2 subs.", "751739": "I realized that while we were just training different models and waiting for the new data of the second stage.\nHow can we build specially a bad model without new data?\n\nAlthough I found some valuable advice on this (thanks to @nroman):\nhttps://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539", "753143": "You will only have a few days during Stage 2 in which to submit prediction files. Pairings are announced on Sunday afternoon, and we will release the Stage 2 data (including a Stage 2 sample submission file) either Sunday night or Monday. And the round of 64 starts Thursday morning. You can actually submit as many prediction files as you want during those few days, but before the submission deadline, you must pick at most two of your submitted files to be your official \"selected\" submissions. This means everyone has two shots at winning. Even if one of your other submissions does great, it doesn't matter because you didn't select it beforehand. You can either have your two entries be highly correlated with minor tweaks between them, or you can have them be staggeringly different. Or anywhere in between; it's up to you.", "754531": "Thank you for this!\nVery informative as **0.46** is a nice **benchmark** to have.\n\nAs for me, this is my first NCAA Challenge and I clearly overfitted the heck out of this stage.\nTime to think about some nice validation strategy!", "754751": "", "756006": "Thank you", "756198": "Thanks a lot, @jeffsonas, this is very informative." }, "source": "meta" }