My understanding is that in the current competition, the final private leaderboard will be based on the publicly available test.csv (please correct me if I'm wrong). So now teams seem to be focusing on each example in test.csv and try to solve them exactly using solvers etc.
\nIt's cool, but it would be also interesting to have one more leaderboard, with a private/hidden test set. In that leaderboard, the focus would be more on generalization and algorithms with low-latency inference.
\nIt would be very interesting to compare and discuss the best solutions from these two types of competitions.
\nWould the organizers consider adding such additional leaderboard (maybe in a form of a new competition)?
", "messages": [ { "id": "1079283", "postDate": "11/15/2020 20:58:54", "content": "My understanding is that in the current competition, the final private leaderboard will be based on the publicly available test.csv (please correct me if I'm wrong). So now teams seem to be focusing on each example in test.csv and try to solve them exactly using solvers etc.
\nIt's cool, but it would be also interesting to have one more leaderboard, with a private/hidden test set. In that leaderboard, the focus would be more on generalization and algorithms with low-latency inference.
\nIt would be very interesting to compare and discuss the best solutions from these two types of competitions.
\nWould the organizers consider adding such additional leaderboard (maybe in a form of a new competition)?
", "rawMarkdown": "My understanding is that in the current competition, the final private leaderboard will be based on the publicly available test.csv (please correct me if I'm wrong). So now teams seem to be focusing on each example in test.csv and try to solve them exactly using solvers etc.\n\nIt's cool, but it would be also interesting to have one more leaderboard, with a private/hidden test set. In that leaderboard, the focus would be more on generalization and algorithms with low-latency inference.\n\nIt would be very interesting to compare and discuss the best solutions from these two types of competitions.\n\nWould the organizers consider adding such additional leaderboard (maybe in a form of a new competition)?", "votes": null }, { "id": "1079627", "postDate": "11/16/2020 10:24:16", "content": "So far as I see in \"Prizes\":
\n\n\nKaggle Swag: Highest Scoring Kaggle Notebook (as determined by Leaderboard submission score). To be eligible for this prize, the Notebook must do the entirety of its computation within the Notebook.
\n
I think it will be very nice if organizer will have one more test that and run all leaderboard's submission on new test to understand whose notebook is the best.
\nIMHO this approach is better rather than manually checked \"do this notebook do all computation within the Notebook or pre-use some data\"?
\nBut I'd like to hear Organizer's response too.
", "rawMarkdown": "So far as I see in \"Prizes\":\n> Kaggle Swag: Highest Scoring Kaggle Notebook (as determined by Leaderboard submission score). To be eligible for this prize, the Notebook must do the entirety of its computation within the Notebook.\n\nI think it will be very nice if organizer will have one more test that and run all leaderboard's submission on new test to understand whose notebook is the best.\n\nIMHO this approach is better rather than manually checked \"do this notebook do all computation within the Notebook or pre-use some data\"? \n\nBut I'd like to hear Organizer's response too.", "votes": null }, { "id": "1095773", "postDate": "11/29/2020 23:56:33", "content": "Even for the after competition private data, I am not sure how it is constructed. From the \"Leadboard\": The private leaderboard is calculated with approximately 50% of the test data.
\nDoes it mean the private data will be from the test data? If that is the case, then one can run 100 hours to achieve high accuracy and save the results into the code or input data. So I hope the private data should be independently constructed.
", "rawMarkdown": "Even for the after competition private data, I am not sure how it is constructed. From the \"Leadboard\": The private leaderboard is calculated with approximately 50% of the test data.\n\nDoes it mean the private data will be from the test data? If that is the case, then one can run 100 hours to achieve high accuracy and save the results into the code or input data. So I hope the private data should be independently constructed.", "votes": null }, { "id": "1095859", "postDate": "11/30/2020 02:36:44", "content": "@mathtester exactly, my understanding is that the private leaderboard's data will not really be private. Hence my suggestion to create a parallel competition, focused more on generalization to unseen examples. It would be great if the organizers could clarify though - maybe my understanding is not correct.
", "rawMarkdown": "mathtester exactly, my understanding is that the private leaderboard's data will not really be private. Hence my suggestion to create a parallel competition, focused more on generalization to unseen examples. It would be great if the organizers could clarify though - maybe my understanding is not correct.", "votes": null }, { "id": "1095988", "postDate": "11/30/2020 05:59:09", "content": "The label under header of leaderboard tells \"private and public boards are calculated on 100% data\".
", "rawMarkdown": "The label under header of leaderboard tells \"private and public boards are calculated on 100% data\".", "votes": null } ], "comments": [ { "id": 1079627, "author_name": "knstqq", "author_url": "", "post_date": "11/16/2020 10:24:16", "content": "So far as I see in \"Prizes\":
\n\n\nKaggle Swag: Highest Scoring Kaggle Notebook (as determined by Leaderboard submission score). To be eligible for this prize, the Notebook must do the entirety of its computation within the Notebook.
\n
I think it will be very nice if organizer will have one more test that and run all leaderboard's submission on new test to understand whose notebook is the best.
\nIMHO this approach is better rather than manually checked \"do this notebook do all computation within the Notebook or pre-use some data\"?
\nBut I'd like to hear Organizer's response too.
", "votes": null, "replies": [] }, { "id": 1095773, "author_name": "mathtester", "author_url": "", "post_date": "11/29/2020 23:56:33", "content": "Even for the after competition private data, I am not sure how it is constructed. From the \"Leadboard\": The private leaderboard is calculated with approximately 50% of the test data.
\nDoes it mean the private data will be from the test data? If that is the case, then one can run 100 hours to achieve high accuracy and save the results into the code or input data. So I hope the private data should be independently constructed.
", "votes": null, "replies": [ { "id": 1095859, "author_name": "sebastko", "author_url": "", "post_date": "11/30/2020 02:36:44", "content": "@mathtester exactly, my understanding is that the private leaderboard's data will not really be private. Hence my suggestion to create a parallel competition, focused more on generalization to unseen examples. It would be great if the organizers could clarify though - maybe my understanding is not correct.
", "votes": null, "replies": [] } ] }, { "id": 1095988, "author_name": "knstqq", "author_url": "", "post_date": "11/30/2020 05:59:09", "content": "The label under header of leaderboard tells \"private and public boards are calculated on 100% data\".
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1079283": "My understanding is that in the current competition, the final private leaderboard will be based on the publicly available test.csv (please correct me if I'm wrong). So now teams seem to be focusing on each example in test.csv and try to solve them exactly using solvers etc.\n\nIt's cool, but it would be also interesting to have one more leaderboard, with a private/hidden test set. In that leaderboard, the focus would be more on generalization and algorithms with low-latency inference.\n\nIt would be very interesting to compare and discuss the best solutions from these two types of competitions.\n\nWould the organizers consider adding such additional leaderboard (maybe in a form of a new competition)?", "1079627": "So far as I see in \"Prizes\":\n> Kaggle Swag: Highest Scoring Kaggle Notebook (as determined by Leaderboard submission score). To be eligible for this prize, the Notebook must do the entirety of its computation within the Notebook.\n\nI think it will be very nice if organizer will have one more test that and run all leaderboard's submission on new test to understand whose notebook is the best.\n\nIMHO this approach is better rather than manually checked \"do this notebook do all computation within the Notebook or pre-use some data\"? \n\nBut I'd like to hear Organizer's response too.", "1095773": "Even for the after competition private data, I am not sure how it is constructed. From the \"Leadboard\": The private leaderboard is calculated with approximately 50% of the test data.\n\nDoes it mean the private data will be from the test data? If that is the case, then one can run 100 hours to achieve high accuracy and save the results into the code or input data. So I hope the private data should be independently constructed.", "1095859": "mathtester exactly, my understanding is that the private leaderboard's data will not really be private. Hence my suggestion to create a parallel competition, focused more on generalization to unseen examples. It would be great if the organizers could clarify though - maybe my understanding is not correct.", "1095988": "The label under header of leaderboard tells \"private and public boards are calculated on 100% data\"." }, "source": "meta" }