Thanks the organizers and Kaggle again for the great challenge !
\nIt seems the number of proteins on public LB is about 100. It is not much, and so causes various instabilities for the models scoring. The organizers themselves several time suggested here on forum that it would be unwise to rely only on public LB. Simple experiments taking same neural network public notebook e.g. \"Simple MLP\" and just reruning it and resubmiting will show big variation of the public scores.
\nQuestion: I wonder would it be possible to add more proteins to public LB ?
\nSince now any way scores of participants changing due to metric correction, it might be a good timing to do it now.
\nIt seems to me even adding some randomly selected proteins from the test would be better rather than having around 100 proteins on LB. It opens the way to \"cheat\" finding that data in public sources, but such way will fail on private LB and do not have big sense, so do not think top participants will do it. While it will give more stable LB.
\nThe other option can be add more proteins which accumulated annotations since 3 months from the start of the challenge but not yet publicaly available, although it seems more difficult.
", "messages": [ { "id": "2342332", "postDate": "07/12/2023 17:35:09", "content": "Thanks the organizers and Kaggle again for the great challenge !
\nIt seems the number of proteins on public LB is about 100. It is not much, and so causes various instabilities for the models scoring. The organizers themselves several time suggested here on forum that it would be unwise to rely only on public LB. Simple experiments taking same neural network public notebook e.g. \"Simple MLP\" and just reruning it and resubmiting will show big variation of the public scores.
\nQuestion: I wonder would it be possible to add more proteins to public LB ?
\nSince now any way scores of participants changing due to metric correction, it might be a good timing to do it now.
\nIt seems to me even adding some randomly selected proteins from the test would be better rather than having around 100 proteins on LB. It opens the way to \"cheat\" finding that data in public sources, but such way will fail on private LB and do not have big sense, so do not think top participants will do it. While it will give more stable LB.
\nThe other option can be add more proteins which accumulated annotations since 3 months from the start of the challenge but not yet publicaly available, although it seems more difficult.
", "rawMarkdown": "Thanks the organizers and Kaggle again for the great challenge !\n\nIt seems the number of proteins on public LB is about 100. It is not much, and so causes various instabilities for the models scoring. The organizers themselves several time suggested here on forum that it would be unwise to rely only on public LB. Simple experiments taking same neural network public notebook e.g. \"Simple MLP\" and just reruning it and resubmiting will show big variation of the public scores. \n\n**Question:** I wonder would it be possible to add more proteins to public LB ? \n\nSince now any way scores of participants changing due to metric correction, it might be a good timing to do it now. \n\nIt seems to me even adding some randomly selected proteins from the test would be better rather than having around 100 proteins on LB. It opens the way to \"cheat\" finding that data in public sources, but such way will fail on private LB and do not have big sense, so do not think top participants will do it. While it will give more stable LB. \n\nThe other option can be add more proteins which accumulated annotations since 3 months from the start of the challenge but not yet publicaly available, although it seems more difficult.", "votes": null }, { "id": "2343581", "postDate": "07/13/2023 19:15:00", "content": "Hi Alexander, I am quite surprised to know that we have only 100 proteins in the public LB. Can you please guide me how you found this out? Thank you.
", "rawMarkdown": "Hi Alexander, I am quite surprised to know that we have only 100 proteins in the public LB. Can you please guide me how you found this out? Thank you.", "votes": null }, { "id": "2343660", "postDate": "07/13/2023 21:41:34", "content": "Hi, Nabil, it is easy to make such an estimate in principle - submiting some N randomly selected proteins we get zero scores for N say 1000, thus you estimate if the number of proteins on LB would be say 600, then 60000(train size wo duplicates) 60000/600=100 you expect every 100 randomly selected proteins contain 1 LB protein, but here you see 1000 without LB proteins. Something like that.
\nOne of the participants on the first webinar suggested the number 60 proteins, I do not want to be so pessimistic and have written above \"about 100\", but it can be 60 indeed.
\nAs for me I suffered a lot from trying to match CV LB scores and seeing lots of instabilities of LB score for which I see that explanation.
\nAlso somewhat similar explanation can probably be the only reason to explain post by MT - where he described jump 0.51 - 0.54 just adding 50 proteins.
", "rawMarkdown": "Hi, Nabil, it is easy to make such an estimate in principle - submiting some N randomly selected proteins we get zero scores for N say 1000, thus you estimate if the number of proteins on LB would be say 600, then 60000(train size wo duplicates) 60000/600=100 you expect every 100 randomly selected proteins contain 1 LB protein, but here you see 1000 without LB proteins. Something like that. \n\nOne of the participants on the first webinar suggested the number 60 proteins, I do not want to be so pessimistic and have written above \"about 100\", but it can be 60 indeed. \n\nAs for me I suffered a lot from trying to match CV LB scores and seeing lots of instabilities of LB score for which I see that explanation. \n\nAlso somewhat similar explanation can probably be the only reason to explain post by MT - where he described jump 0.51 - 0.54 just adding 50 proteins.", "votes": null }, { "id": "2343701", "postDate": "07/13/2023 22:32:32", "content": "We know from experience, if we had any public data, it would quickly be gamed, making the leaderboard fairly meaningless. That, and updating the number of proteins would require re-running every submission to date (over 12,000!) again, so it's impractical and would't provide benefit compared to creating larger holdout sets from the training set to do local validation.
", "rawMarkdown": "We know from experience, if we had any public data, it would quickly be gamed, making the leaderboard fairly meaningless. That, and updating the number of proteins would require re-running every submission to date (over 12,000!) again, so it's impractical and would't provide benefit compared to creating larger holdout sets from the training set to do local validation.", "votes": null }, { "id": "2343737", "postDate": "07/14/2023 00:19:17", "content": "Thank you for your detailed response. I have also been facing a similar issue, even if my local testing improves significantly, my leaderboard score only gets lower.
", "rawMarkdown": "Thank you for your detailed response. I have also been facing a similar issue, even if my local testing improves significantly, my leaderboard score only gets lower.", "votes": null }, { "id": "2344212", "postDate": "07/14/2023 10:56:31", "content": "I see your points.
\nBut it is not clear how to create a reasonable local validation for the task.
\nJust the random split with any size of holdout does not seem to capture all the issues.
There are several issues:
\n1) grouping proteins by similarity - to create groupwise fold - that can be done e.g. with Diamond - thus we can hope model would work better on \"unseen\" proteins.
\nBut there are more issues:
\n2) labels we have are of low quality and moreover that quality is not uniformly distributed across proteins - what kind of quality will we have on private part - not clear - hopefully it would be better than average train quality, it is not clear how much better and how to take that into account
\n3) validation which some people used on previous CAFA - is time-series like - trying to predict future data from the paster one, there are many thing unclear for me with these way
", "rawMarkdown": "I see your points.\n\nBut it is not clear how to create a reasonable local validation for the task. \nJust the random split with any size of holdout does not seem to capture all the issues.\n\nThere are several issues: \n\n1) grouping proteins by similarity - to create groupwise fold - that can be done e.g. with Diamond - thus we can hope model would work better on \"unseen\" proteins. \n\nBut there are more issues: \n\n2) labels we have are of low quality and moreover that quality is not uniformly distributed across proteins - what kind of quality will we have on private part - not clear - hopefully it would be better than average train quality, it is not clear how much better and how to take that into account\n\n3) validation which some people used on previous CAFA - is time-series like - trying to predict future data from the paster one, there are many thing unclear for me with these way", "votes": null } ], "comments": [ { "id": 2343581, "author_name": "robin0", "author_url": "", "post_date": "07/13/2023 19:15:00", "content": "Hi Alexander, I am quite surprised to know that we have only 100 proteins in the public LB. Can you please guide me how you found this out? Thank you.
", "votes": null, "replies": [ { "id": 2343660, "author_name": "alexandervc", "author_url": "", "post_date": "07/13/2023 21:41:34", "content": "Hi, Nabil, it is easy to make such an estimate in principle - submiting some N randomly selected proteins we get zero scores for N say 1000, thus you estimate if the number of proteins on LB would be say 600, then 60000(train size wo duplicates) 60000/600=100 you expect every 100 randomly selected proteins contain 1 LB protein, but here you see 1000 without LB proteins. Something like that.
\nOne of the participants on the first webinar suggested the number 60 proteins, I do not want to be so pessimistic and have written above \"about 100\", but it can be 60 indeed.
\nAs for me I suffered a lot from trying to match CV LB scores and seeing lots of instabilities of LB score for which I see that explanation.
\nAlso somewhat similar explanation can probably be the only reason to explain post by MT - where he described jump 0.51 - 0.54 just adding 50 proteins.
", "votes": null, "replies": [ { "id": 2343737, "author_name": "robin0", "author_url": "", "post_date": "07/14/2023 00:19:17", "content": "Thank you for your detailed response. I have also been facing a similar issue, even if my local testing improves significantly, my leaderboard score only gets lower.
", "votes": null, "replies": [] } ] } ] }, { "id": 2343701, "author_name": "inversion", "author_url": "", "post_date": "07/13/2023 22:32:32", "content": "We know from experience, if we had any public data, it would quickly be gamed, making the leaderboard fairly meaningless. That, and updating the number of proteins would require re-running every submission to date (over 12,000!) again, so it's impractical and would't provide benefit compared to creating larger holdout sets from the training set to do local validation.
", "votes": null, "replies": [ { "id": 2344212, "author_name": "alexandervc", "author_url": "", "post_date": "07/14/2023 10:56:31", "content": "I see your points.
\nBut it is not clear how to create a reasonable local validation for the task.
\nJust the random split with any size of holdout does not seem to capture all the issues.
There are several issues:
\n1) grouping proteins by similarity - to create groupwise fold - that can be done e.g. with Diamond - thus we can hope model would work better on \"unseen\" proteins.
\nBut there are more issues:
\n2) labels we have are of low quality and moreover that quality is not uniformly distributed across proteins - what kind of quality will we have on private part - not clear - hopefully it would be better than average train quality, it is not clear how much better and how to take that into account
\n3) validation which some people used on previous CAFA - is time-series like - trying to predict future data from the paster one, there are many thing unclear for me with these way
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2342332": "Thanks the organizers and Kaggle again for the great challenge !\n\nIt seems the number of proteins on public LB is about 100. It is not much, and so causes various instabilities for the models scoring. The organizers themselves several time suggested here on forum that it would be unwise to rely only on public LB. Simple experiments taking same neural network public notebook e.g. \"Simple MLP\" and just reruning it and resubmiting will show big variation of the public scores. \n\n**Question:** I wonder would it be possible to add more proteins to public LB ? \n\nSince now any way scores of participants changing due to metric correction, it might be a good timing to do it now. \n\nIt seems to me even adding some randomly selected proteins from the test would be better rather than having around 100 proteins on LB. It opens the way to \"cheat\" finding that data in public sources, but such way will fail on private LB and do not have big sense, so do not think top participants will do it. While it will give more stable LB. \n\nThe other option can be add more proteins which accumulated annotations since 3 months from the start of the challenge but not yet publicaly available, although it seems more difficult.", "2343581": "Hi Alexander, I am quite surprised to know that we have only 100 proteins in the public LB. Can you please guide me how you found this out? Thank you.", "2343660": "Hi, Nabil, it is easy to make such an estimate in principle - submiting some N randomly selected proteins we get zero scores for N say 1000, thus you estimate if the number of proteins on LB would be say 600, then 60000(train size wo duplicates) 60000/600=100 you expect every 100 randomly selected proteins contain 1 LB protein, but here you see 1000 without LB proteins. Something like that. \n\nOne of the participants on the first webinar suggested the number 60 proteins, I do not want to be so pessimistic and have written above \"about 100\", but it can be 60 indeed. \n\nAs for me I suffered a lot from trying to match CV LB scores and seeing lots of instabilities of LB score for which I see that explanation. \n\nAlso somewhat similar explanation can probably be the only reason to explain post by MT - where he described jump 0.51 - 0.54 just adding 50 proteins.", "2343701": "We know from experience, if we had any public data, it would quickly be gamed, making the leaderboard fairly meaningless. That, and updating the number of proteins would require re-running every submission to date (over 12,000!) again, so it's impractical and would't provide benefit compared to creating larger holdout sets from the training set to do local validation.", "2343737": "Thank you for your detailed response. I have also been facing a similar issue, even if my local testing improves significantly, my leaderboard score only gets lower.", "2344212": "I see your points.\n\nBut it is not clear how to create a reasonable local validation for the task. \nJust the random split with any size of holdout does not seem to capture all the issues.\n\nThere are several issues: \n\n1) grouping proteins by similarity - to create groupwise fold - that can be done e.g. with Diamond - thus we can hope model would work better on \"unseen\" proteins. \n\nBut there are more issues: \n\n2) labels we have are of low quality and moreover that quality is not uniformly distributed across proteins - what kind of quality will we have on private part - not clear - hopefully it would be better than average train quality, it is not clear how much better and how to take that into account\n\n3) validation which some people used on previous CAFA - is time-series like - trying to predict future data from the paster one, there are many thing unclear for me with these way" }, "source": "meta" }