RT.
\njust like what happened in https://www.kaggle.com/competitions/icr-identify-age-related-conditions/leaderboard.
RT.
\njust like what happened in https://www.kaggle.com/competitions/icr-identify-age-related-conditions/leaderboard.
\"The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation. The final test set will consist of proteins that will have accumulated functional terms after the submission deadline and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected. Overall, the participants are encouraged to maximize the generalization performance and use the leaderboard only as a rough indicator of their model's performance.\"the host said in Evaluation.
", "rawMarkdown": "\"The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation. The final test set will consist of proteins that will have accumulated functional terms after the submission deadline and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected. Overall, the participants are encouraged to maximize the generalization performance and use the leaderboard only as a rough indicator of their model's performance.\"the host said in Evaluation.", "votes": null }, { "id": "2389160", "postDate": "08/13/2023 20:21:51", "content": "Sorry , it seems my understanding was wrong, and I understood it too late.
\nThe truth is - the actual notebook by MT \"merge_datasets\" contains 190+k proteins - which cover supertest quite well - about 90-95% - and so can perform quite well on private. ( I had wrong understanding that it contains only 30-50 proteins - because early days that was discussed on the forum. If that would be the case all pulics based on \"merge_datasets\" will shake-up very very strongly on private - that was my message in the original post. ).
\nSo that quick-go predictions might cover and work well on private test set /
\nAnd so public solutions using that idea might perform well.
\nAnd so they will not shake-up.
\nAnd there is no easy way to estimate whether they work or not - at least not what I know at the moment.
Original post:
\nLet me share some concerns about possible shake-up which can be caused by not careful use of some public notebooks , and how to avoid it.
\nAs it was discussed before the number of proteins on LB is very small - about 100.
\nIt seems some public notebooks use internet available annotation-candidates for a (part of) THESE PARTICULAR public LB-proteins.
\nIt seems these prediction-candidates are not bad - and thus the scores uplifts,
\nbut since we expect different proteins to be on private LB and to have much more proteins of them, the scores obtained with the help of the small number of prediction-candidates does not correspond to the actual scores which we will see on private LB.
Here MT kindly shared his findings about the situation: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/404853
\nAs he explained one can download quick-go annotation-candidates (which might be different from actual LB labels - but seems the difference is not that much big) for small number of proteins. And adding these predictions to your model - you will uplift.
It seems all the high-scored public notebooks use these predictions without carefully explaining the possible caveats.
\nThus it might cause shake-up for those who are using these publics not carefully.
PS
\nLet me repeat the same in other words - the correct annotations for the public LB proteins seems NOT yet available in inet (at least that is what is assumed). But what is available - some hypothesis about these annotations - and it was even by the organizers - try to use the so-called electronic annotations (not experimental) as a kind of possible component for the models.
\nThe general wisdom of Kaggle - not to use public notebooks without careful check - for many competitions in the past it causes lot of shake-ups. So everyone just need to follow that standard Kaggle wisdom.
\nPSPS
\nBesides of that we may have shake-up due to \"biological reasons\" - because our predictions will not fit real biology - but that is part of the challenge. However I do not see that it can be that much big as on \"ICR\" challenge - which was just lottery from the beginning.
", "rawMarkdown": "Sorry , it seems my understanding was wrong, and I understood it too late.\n\nThe truth is - the actual notebook by MT \"merge_datasets\" contains 190+k proteins - which cover supertest quite well - about 90-95% - and so can perform quite well on private. ( I had wrong understanding that it contains only 30-50 proteins - because early days that was discussed on the forum. If that would be the case all pulics based on \"merge_datasets\" will shake-up very very strongly on private - that was my message in the original post. ). \n\nSo that quick-go predictions might cover and work well on private test set /\nAnd so public solutions using that idea might perform well. \nAnd so they will not shake-up. \nAnd there is no easy way to estimate whether they work or not - at least not what I know at the moment.\n\n\n----------\n\nOriginal post: \n\nLet me share some concerns about possible shake-up which can be caused by not careful use of some public notebooks , and how to avoid it. \n\nAs it was discussed before the number of proteins on LB is very small - about 100. \nIt seems some public notebooks use internet available annotation-candidates for a (part of) THESE PARTICULAR public LB-proteins.\nIt seems these prediction-candidates are not bad - and thus the scores uplifts,\nbut since we expect different proteins to be on private LB and to have much more proteins of them, the scores obtained with the help of the small number of prediction-candidates does not correspond to the actual scores which we will see on private LB. \n\nHere MT kindly shared his findings about the situation: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/404853 \nAs he explained one can download quick-go annotation-candidates (which might be different from actual LB labels - but seems the difference is not that much big) for small number of proteins. And adding these predictions to your model - you will uplift.\n\nIt seems all the high-scored public notebooks use these predictions without carefully explaining the possible caveats. \nThus it might cause shake-up for those who are using these publics not carefully.\n\n---------\n\nPS\n\nLet me repeat the same in other words - the correct annotations for the public LB proteins seems NOT yet available in inet (at least that is what is assumed). But what is available - some hypothesis about these annotations - and it was even by the organizers - try to use the so-called electronic annotations (not experimental) as a kind of possible component for the models.\n\nThe general wisdom of Kaggle - not to use public notebooks without careful check - for many competitions in the past it causes lot of shake-ups. So everyone just need to follow that standard Kaggle wisdom.\n\n---------\n\nPSPS\n\nBesides of that we may have shake-up due to \"biological reasons\" - because our predictions will not fit real biology - but that is part of the challenge. However I do not see that it can be that much big as on \"ICR\" challenge - which was just lottery from the beginning.", "votes": null }, { "id": "2396668", "postDate": "08/18/2023 11:53:00", "content": "Broadly, teams near the top of the LB are there for one of two reasons:
\nThey are bioinformatics professionals (academic or biotech) who know what they're doing and have excellent scores because they have strong models. These will remain near the top.
They are have overfitted the public LB and are heading for an almighty fall when the results come out.
I would be surprised if many teams come 'from nowhere' (outside the medals, ~top 150) into the gold zone, unless they're deliberately sandbagging their scores on the public LB.
\nBased on previous CAFAs, I'd expect 50-100 'professional' teams to end up in the medals. Most of those will have decent public scores too come submission time, even if they post their models on the last day.
", "rawMarkdown": "Broadly, teams near the top of the LB are there for one of two reasons:\n\n1. They are bioinformatics professionals (academic or biotech) who know what they're doing and have excellent scores because they have strong models. These will remain near the top.\n\n2. They are have overfitted the public LB and are heading for an almighty fall when the results come out.\n\nI would be surprised if many teams come 'from nowhere' (outside the medals, ~top 150) into the gold zone, unless they're deliberately sandbagging their scores on the public LB.\n\nBased on previous CAFAs, I'd expect 50-100 'professional' teams to end up in the medals. Most of those will have decent public scores too come submission time, even if they post their models on the last day.", "votes": null } ], "comments": [ { "id": 2388901, "author_name": "siboxiao", "author_url": "", "post_date": "08/13/2023 17:04:56", "content": "\"The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation. The final test set will consist of proteins that will have accumulated functional terms after the submission deadline and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected. Overall, the participants are encouraged to maximize the generalization performance and use the leaderboard only as a rough indicator of their model's performance.\"the host said in Evaluation.
", "votes": null, "replies": [] }, { "id": 2389160, "author_name": "alexandervc", "author_url": "", "post_date": "08/13/2023 20:21:51", "content": "Sorry , it seems my understanding was wrong, and I understood it too late.
\nThe truth is - the actual notebook by MT \"merge_datasets\" contains 190+k proteins - which cover supertest quite well - about 90-95% - and so can perform quite well on private. ( I had wrong understanding that it contains only 30-50 proteins - because early days that was discussed on the forum. If that would be the case all pulics based on \"merge_datasets\" will shake-up very very strongly on private - that was my message in the original post. ).
\nSo that quick-go predictions might cover and work well on private test set /
\nAnd so public solutions using that idea might perform well.
\nAnd so they will not shake-up.
\nAnd there is no easy way to estimate whether they work or not - at least not what I know at the moment.
Original post:
\nLet me share some concerns about possible shake-up which can be caused by not careful use of some public notebooks , and how to avoid it.
\nAs it was discussed before the number of proteins on LB is very small - about 100.
\nIt seems some public notebooks use internet available annotation-candidates for a (part of) THESE PARTICULAR public LB-proteins.
\nIt seems these prediction-candidates are not bad - and thus the scores uplifts,
\nbut since we expect different proteins to be on private LB and to have much more proteins of them, the scores obtained with the help of the small number of prediction-candidates does not correspond to the actual scores which we will see on private LB.
Here MT kindly shared his findings about the situation: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/404853
\nAs he explained one can download quick-go annotation-candidates (which might be different from actual LB labels - but seems the difference is not that much big) for small number of proteins. And adding these predictions to your model - you will uplift.
It seems all the high-scored public notebooks use these predictions without carefully explaining the possible caveats.
\nThus it might cause shake-up for those who are using these publics not carefully.
PS
\nLet me repeat the same in other words - the correct annotations for the public LB proteins seems NOT yet available in inet (at least that is what is assumed). But what is available - some hypothesis about these annotations - and it was even by the organizers - try to use the so-called electronic annotations (not experimental) as a kind of possible component for the models.
\nThe general wisdom of Kaggle - not to use public notebooks without careful check - for many competitions in the past it causes lot of shake-ups. So everyone just need to follow that standard Kaggle wisdom.
\nPSPS
\nBesides of that we may have shake-up due to \"biological reasons\" - because our predictions will not fit real biology - but that is part of the challenge. However I do not see that it can be that much big as on \"ICR\" challenge - which was just lottery from the beginning.
", "votes": null, "replies": [] }, { "id": 2396668, "author_name": "jbomitchell", "author_url": "", "post_date": "08/18/2023 11:53:00", "content": "Broadly, teams near the top of the LB are there for one of two reasons:
\nThey are bioinformatics professionals (academic or biotech) who know what they're doing and have excellent scores because they have strong models. These will remain near the top.
They are have overfitted the public LB and are heading for an almighty fall when the results come out.
I would be surprised if many teams come 'from nowhere' (outside the medals, ~top 150) into the gold zone, unless they're deliberately sandbagging their scores on the public LB.
\nBased on previous CAFAs, I'd expect 50-100 'professional' teams to end up in the medals. Most of those will have decent public scores too come submission time, even if they post their models on the last day.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2387069": "RT.\njust like what happened in https://www.kaggle.com/competitions/icr-identify-age-related-conditions/leaderboard.", "2388901": "\"The participants are cautioned that the leaderboard was designed to display method performance on a relatively small selection of proteins from the test superset (see Data), provided to us by the UniProtKB team but not available in UniProtKB or other public databases. These proteins will not be included in the test set for the subontologies used for the leaderboard evaluation. The final test set will consist of proteins that will have accumulated functional terms after the submission deadline and therefore, some distribution shift between the sample of proteins used for the leaderboard and the final evaluation sample is to be expected. Overall, the participants are encouraged to maximize the generalization performance and use the leaderboard only as a rough indicator of their model's performance.\"the host said in Evaluation.", "2389160": "Sorry , it seems my understanding was wrong, and I understood it too late.\n\nThe truth is - the actual notebook by MT \"merge_datasets\" contains 190+k proteins - which cover supertest quite well - about 90-95% - and so can perform quite well on private. ( I had wrong understanding that it contains only 30-50 proteins - because early days that was discussed on the forum. If that would be the case all pulics based on \"merge_datasets\" will shake-up very very strongly on private - that was my message in the original post. ). \n\nSo that quick-go predictions might cover and work well on private test set /\nAnd so public solutions using that idea might perform well. \nAnd so they will not shake-up. \nAnd there is no easy way to estimate whether they work or not - at least not what I know at the moment.\n\n\n----------\n\nOriginal post: \n\nLet me share some concerns about possible shake-up which can be caused by not careful use of some public notebooks , and how to avoid it. \n\nAs it was discussed before the number of proteins on LB is very small - about 100. \nIt seems some public notebooks use internet available annotation-candidates for a (part of) THESE PARTICULAR public LB-proteins.\nIt seems these prediction-candidates are not bad - and thus the scores uplifts,\nbut since we expect different proteins to be on private LB and to have much more proteins of them, the scores obtained with the help of the small number of prediction-candidates does not correspond to the actual scores which we will see on private LB. \n\nHere MT kindly shared his findings about the situation: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/404853 \nAs he explained one can download quick-go annotation-candidates (which might be different from actual LB labels - but seems the difference is not that much big) for small number of proteins. And adding these predictions to your model - you will uplift.\n\nIt seems all the high-scored public notebooks use these predictions without carefully explaining the possible caveats. \nThus it might cause shake-up for those who are using these publics not carefully.\n\n---------\n\nPS\n\nLet me repeat the same in other words - the correct annotations for the public LB proteins seems NOT yet available in inet (at least that is what is assumed). But what is available - some hypothesis about these annotations - and it was even by the organizers - try to use the so-called electronic annotations (not experimental) as a kind of possible component for the models.\n\nThe general wisdom of Kaggle - not to use public notebooks without careful check - for many competitions in the past it causes lot of shake-ups. So everyone just need to follow that standard Kaggle wisdom.\n\n---------\n\nPSPS\n\nBesides of that we may have shake-up due to \"biological reasons\" - because our predictions will not fit real biology - but that is part of the challenge. However I do not see that it can be that much big as on \"ICR\" challenge - which was just lottery from the beginning.", "2396668": "Broadly, teams near the top of the LB are there for one of two reasons:\n\n1. They are bioinformatics professionals (academic or biotech) who know what they're doing and have excellent scores because they have strong models. These will remain near the top.\n\n2. They are have overfitted the public LB and are heading for an almighty fall when the results come out.\n\nI would be surprised if many teams come 'from nowhere' (outside the medals, ~top 150) into the gold zone, unless they're deliberately sandbagging their scores on the public LB.\n\nBased on previous CAFAs, I'd expect 50-100 'professional' teams to end up in the medals. Most of those will have decent public scores too come submission time, even if they post their models on the last day." }, "source": "meta" }