I'm not that familiar with Kaggle competitions, so maybe this is a known thing in those circles. However, thinking about the secret data used in the public LB test set, would it not be possible to deduce at least which proteins are being used for the ranking? Theoretically, someone could randomly spike 50% of their predictions, submit to the leaderboard and then continue to submit different randomly spiked sets of predictions. Eventually by a stochastic binary search process, surely you could fairly accurately deduce which target sequences are forming part of the public LB scoring? It wouldn't take that many cycles to narrow down the subset. I'm not sure it would be practical to deduce the labels this way, but at least the hidden targets could be revealed. Obviously this wouldn't help in the blind final assessment, but it could make interpreting the public LB tricky.
", "messages": [ { "id": "2234800", "postDate": "04/25/2023 13:55:04", "content": "I'm not that familiar with Kaggle competitions, so maybe this is a known thing in those circles. However, thinking about the secret data used in the public LB test set, would it not be possible to deduce at least which proteins are being used for the ranking? Theoretically, someone could randomly spike 50% of their predictions, submit to the leaderboard and then continue to submit different randomly spiked sets of predictions. Eventually by a stochastic binary search process, surely you could fairly accurately deduce which target sequences are forming part of the public LB scoring? It wouldn't take that many cycles to narrow down the subset. I'm not sure it would be practical to deduce the labels this way, but at least the hidden targets could be revealed. Obviously this wouldn't help in the blind final assessment, but it could make interpreting the public LB tricky.
", "rawMarkdown": "I'm not that familiar with Kaggle competitions, so maybe this is a known thing in those circles. However, thinking about the secret data used in the public LB test set, would it not be possible to deduce at least which proteins are being used for the ranking? Theoretically, someone could randomly spike 50% of their predictions, submit to the leaderboard and then continue to submit different randomly spiked sets of predictions. Eventually by a stochastic binary search process, surely you could fairly accurately deduce which target sequences are forming part of the public LB scoring? It wouldn't take that many cycles to narrow down the subset. I'm not sure it would be practical to deduce the labels this way, but at least the hidden targets could be revealed. Obviously this wouldn't help in the blind final assessment, but it could make interpreting the public LB tricky.", "votes": null }, { "id": "2234822", "postDate": "04/25/2023 14:10:09", "content": "Yes indeed, LB probing (as it is called on Kaggle) is definitely possible in this competition. Depending on the details of the specific competition, there are often ways to get the info without costing too many submissions, for instance by making submissions designed to fail unless (or if) a certain test item is part of the public LB.
\nI'm not endorsing the practice, nor condemning it - merely pointing out that it is common on Kaggle.
", "rawMarkdown": "Yes indeed, LB probing (as it is called on Kaggle) is definitely possible in this competition. Depending on the details of the specific competition, there are often ways to get the info without costing too many submissions, for instance by making submissions designed to fail unless (or if) a certain test item is part of the public LB.\n\nI'm not endorsing the practice, nor condemning it - merely pointing out that it is common on Kaggle.", "votes": null }, { "id": "2234841", "postDate": "04/25/2023 14:19:52", "content": "The Public leaderboard is a very small subset of unreleased data. It is there to help people validate that their models are working as intended and their submissions are in the correct format (e.g., they score correctly). It would be very unwise to use the Public LB to validate model quality (rather than local validation).
", "rawMarkdown": "The Public leaderboard is a very small subset of unreleased data. It is there to help people validate that their models are working as intended and their submissions are in the correct format (e.g., they score correctly). It would be very unwise to use the Public LB to validate model quality (rather than local validation).", "votes": null }, { "id": "2234882", "postDate": "04/25/2023 14:46:12", "content": "Thanks - nice to know it has a name at least! The only advantages I can see is if you just like seeing your name at the top of a leaderboard or reducing the number of predictions you need to generate and upload during development. Actually that last one is actually a real advantage as I'm guessing less than 0.1% of the predictions in a submission are being referenced for the public LB, and a lot of bandwidth and CPU power is being wasted processing the files. Still, the warning about not using the data to validate models is certainly good advice.
", "rawMarkdown": "Thanks - nice to know it has a name at least! The only advantages I can see is if you just like seeing your name at the top of a leaderboard or reducing the number of predictions you need to generate and upload during development. Actually that last one is actually a real advantage as I'm guessing less than 0.1% of the predictions in a submission are being referenced for the public LB, and a lot of bandwidth and CPU power is being wasted processing the files. Still, the warning about not using the data to validate models is certainly good advice.", "votes": null }, { "id": "2235007", "postDate": "04/25/2023 16:18:10", "content": "\n\nThe Public leaderboard is a very small subset of unreleased data.
\n
But then what does this phrase on the leaderboard page mean?
\n\n", "rawMarkdown": ">The Public leaderboard is a very small subset of unreleased data.\n\nBut then what does this phrase on the leaderboard page mean?\n\n>This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.", "votes": null }, { "id": "2235127", "postDate": "04/25/2023 18:50:07", "content": "This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.
\n
This is a (confusing) artifact of the re-run metric. It can be safely ignored.
", "rawMarkdown": "This is a (confusing) artifact of the re-run metric. It can be safely ignored.", "votes": null }, { "id": "2235741", "postDate": "04/26/2023 10:13:48", "content": "Would it be possible to get a rough estimate of what \"a very small subset\" is? Are we talking about 10s, 100s, or 1000s of proteins? Just to get a feeling of how representative the public LB might be.
\nOf course, I understand if you'd rather keep this a secret.
Since the assessment is not on the public leaderboard as it is now, but on a set that will be generated on December, you would want to make your models as general as possible. This has been noted in other discussions.
\nAlso, being \"representative\" has very little meaning in the protein sequence data world, which has highly imbalanced datasets to start off with. https://pubmed.ncbi.nlm.nih.gov/23737737/
", "rawMarkdown": "Since the assessment is not on the public leaderboard as it is now, but on a set that will be generated on December, you would want to make your models as general as possible. This has been noted in other discussions. \n\nAlso, being \"representative\" has very little meaning in the protein sequence data world, which has highly imbalanced datasets to start off with. [https://pubmed.ncbi.nlm.nih.gov/23737737/](https://pubmed.ncbi.nlm.nih.gov/23737737/)", "votes": null }, { "id": "2236137", "postDate": "04/26/2023 15:04:43", "content": "Thank you. Maybe I should have phrased it better. With \"representative\" I didn't mean the data set being representative for all proteins; I know that redundancy can be a pain in bioinformatics. What I meant was more along the way of the ranking in the public LB being somewhat \"representative\" of the ranking in the final one. While the model's ability to generalize will play a major part in that, I'd guess that a public ranking being based on 1000 proteins should be closer to the final ranking than if the public ranking was only based on 10 proteins.
", "rawMarkdown": "Thank you. Maybe I should have phrased it better. With \"representative\" I didn't mean the data set being representative for all proteins; I know that redundancy can be a pain in bioinformatics. What I meant was more along the way of the ranking in the public LB being somewhat \"representative\" of the ranking in the final one. While the model's ability to generalize will play a major part in that, I'd guess that a public ranking being based on 1000 proteins should be closer to the final ranking than if the public ranking was only based on 10 proteins.", "votes": null } ], "comments": [ { "id": 2234822, "author_name": "jbomitchell", "author_url": "", "post_date": "04/25/2023 14:10:09", "content": "Yes indeed, LB probing (as it is called on Kaggle) is definitely possible in this competition. Depending on the details of the specific competition, there are often ways to get the info without costing too many submissions, for instance by making submissions designed to fail unless (or if) a certain test item is part of the public LB.
\nI'm not endorsing the practice, nor condemning it - merely pointing out that it is common on Kaggle.
", "votes": null, "replies": [ { "id": 2234882, "author_name": "biodav", "author_url": "", "post_date": "04/25/2023 14:46:12", "content": "Thanks - nice to know it has a name at least! The only advantages I can see is if you just like seeing your name at the top of a leaderboard or reducing the number of predictions you need to generate and upload during development. Actually that last one is actually a real advantage as I'm guessing less than 0.1% of the predictions in a submission are being referenced for the public LB, and a lot of bandwidth and CPU power is being wasted processing the files. Still, the warning about not using the data to validate models is certainly good advice.
", "votes": null, "replies": [] } ] }, { "id": 2234841, "author_name": "inversion", "author_url": "", "post_date": "04/25/2023 14:19:52", "content": "The Public leaderboard is a very small subset of unreleased data. It is there to help people validate that their models are working as intended and their submissions are in the correct format (e.g., they score correctly). It would be very unwise to use the Public LB to validate model quality (rather than local validation).
", "votes": null, "replies": [ { "id": 2235007, "author_name": "sergeifironov", "author_url": "", "post_date": "04/25/2023 16:18:10", "content": "\n\nThe Public leaderboard is a very small subset of unreleased data.
\n
But then what does this phrase on the leaderboard page mean?
\n\n", "votes": null, "replies": [ { "id": 2235127, "author_name": "inversion", "author_url": "", "post_date": "04/25/2023 18:50:07", "content": "This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.
\n
This is a (confusing) artifact of the re-run metric. It can be safely ignored.
", "votes": null, "replies": [ { "id": 2235741, "author_name": "norixmb", "author_url": "", "post_date": "04/26/2023 10:13:48", "content": "Would it be possible to get a rough estimate of what \"a very small subset\" is? Are we talking about 10s, 100s, or 1000s of proteins? Just to get a feeling of how representative the public LB might be.
\nOf course, I understand if you'd rather keep this a secret.
Since the assessment is not on the public leaderboard as it is now, but on a set that will be generated on December, you would want to make your models as general as possible. This has been noted in other discussions.
\nAlso, being \"representative\" has very little meaning in the protein sequence data world, which has highly imbalanced datasets to start off with. https://pubmed.ncbi.nlm.nih.gov/23737737/
", "votes": null, "replies": [ { "id": 2236137, "author_name": "norixmb", "author_url": "", "post_date": "04/26/2023 15:04:43", "content": "Thank you. Maybe I should have phrased it better. With \"representative\" I didn't mean the data set being representative for all proteins; I know that redundancy can be a pain in bioinformatics. What I meant was more along the way of the ranking in the public LB being somewhat \"representative\" of the ranking in the final one. While the model's ability to generalize will play a major part in that, I'd guess that a public ranking being based on 1000 proteins should be closer to the final ranking than if the public ranking was only based on 10 proteins.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2234800": "I'm not that familiar with Kaggle competitions, so maybe this is a known thing in those circles. However, thinking about the secret data used in the public LB test set, would it not be possible to deduce at least which proteins are being used for the ranking? Theoretically, someone could randomly spike 50% of their predictions, submit to the leaderboard and then continue to submit different randomly spiked sets of predictions. Eventually by a stochastic binary search process, surely you could fairly accurately deduce which target sequences are forming part of the public LB scoring? It wouldn't take that many cycles to narrow down the subset. I'm not sure it would be practical to deduce the labels this way, but at least the hidden targets could be revealed. Obviously this wouldn't help in the blind final assessment, but it could make interpreting the public LB tricky.", "2234822": "Yes indeed, LB probing (as it is called on Kaggle) is definitely possible in this competition. Depending on the details of the specific competition, there are often ways to get the info without costing too many submissions, for instance by making submissions designed to fail unless (or if) a certain test item is part of the public LB.\n\nI'm not endorsing the practice, nor condemning it - merely pointing out that it is common on Kaggle.", "2234841": "The Public leaderboard is a very small subset of unreleased data. It is there to help people validate that their models are working as intended and their submissions are in the correct format (e.g., they score correctly). It would be very unwise to use the Public LB to validate model quality (rather than local validation).", "2234882": "Thanks - nice to know it has a name at least! The only advantages I can see is if you just like seeing your name at the top of a leaderboard or reducing the number of predictions you need to generate and upload during development. Actually that last one is actually a real advantage as I'm guessing less than 0.1% of the predictions in a submission are being referenced for the public LB, and a lot of bandwidth and CPU power is being wasted processing the files. Still, the warning about not using the data to validate models is certainly good advice.", "2235007": ">The Public leaderboard is a very small subset of unreleased data.\n\nBut then what does this phrase on the leaderboard page mean?\n\n>This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.", "2235127": "This is a (confusing) artifact of the re-run metric. It can be safely ignored.", "2235741": "Would it be possible to get a rough estimate of what \"a very small subset\" is? Are we talking about 10s, 100s, or 1000s of proteins? Just to get a feeling of how representative the public LB might be.\nOf course, I understand if you'd rather keep this a secret.", "2236118": "Since the assessment is not on the public leaderboard as it is now, but on a set that will be generated on December, you would want to make your models as general as possible. This has been noted in other discussions. \n\nAlso, being \"representative\" has very little meaning in the protein sequence data world, which has highly imbalanced datasets to start off with. [https://pubmed.ncbi.nlm.nih.gov/23737737/](https://pubmed.ncbi.nlm.nih.gov/23737737/)", "2236137": "Thank you. Maybe I should have phrased it better. With \"representative\" I didn't mean the data set being representative for all proteins; I know that redundancy can be a pain in bioinformatics. What I meant was more along the way of the ranking in the public LB being somewhat \"representative\" of the ranking in the final one. While the model's ability to generalize will play a major part in that, I'd guess that a public ranking being based on 1000 proteins should be closer to the final ranking than if the public ranking was only based on 10 proteins." }, "source": "meta" }