Hi everyone
\n@ahphan (sorry for directly tagging)
\nI have two questions about the private leaderboard evaluation that I'd like to clarify with the organizers:
\n1. Predictions Outside Test Superset
\nAccording to the data description:
\nTest Superset: The test superset is a set of protein sequences on which the participants are asked to predict GO terms...\n\nTest Set: The test set is unknown at the beginning of the competition. It will contain protein sequences (and their functions) from the test superset that gained experimental annotations between the submission deadline and the time of evaluation.\nThe testsuperset.fasta file contains 224,309 proteins. However, I've noticed several public notebooks generating predictions for ~270k proteins.
\nMy question: If a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?
\nThis affects whether we should invest effort in predicting GO terms for proteins beyond the official test superset.
\n2. Zero-IA Terms Affecting Score
\nAfter the recent IA (Information Accretion) TSV file update, there are 11,828 GO terms with IA = 0.
\nMy understanding was that zero-IA terms wouldn't contribute to the weighted F-max calculation (since their weight would be zero). However, when I removed all IA=0 term predictions from my best submission, my public LB score dropped significantly (0.9% - 1.6%).
\nThis seems counterintuitive. Could the organizers clarify:
\nAre zero-IA terms actually contributing to the score calculation?\nIs there something about the metric computation I'm misunderstanding?\nAny clarification would be greatly appreciated. Thanks!
", "messages": [ { "id": "3398662", "postDate": "01/29/2026 13:56:54", "content": "Hi everyone
\n@ahphan (sorry for directly tagging)
\nI have two questions about the private leaderboard evaluation that I'd like to clarify with the organizers:
\n1. Predictions Outside Test Superset
\nAccording to the data description:
\nTest Superset: The test superset is a set of protein sequences on which the participants are asked to predict GO terms...\n\nTest Set: The test set is unknown at the beginning of the competition. It will contain protein sequences (and their functions) from the test superset that gained experimental annotations between the submission deadline and the time of evaluation.\nThe testsuperset.fasta file contains 224,309 proteins. However, I've noticed several public notebooks generating predictions for ~270k proteins.
\nMy question: If a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?
\nThis affects whether we should invest effort in predicting GO terms for proteins beyond the official test superset.
\n2. Zero-IA Terms Affecting Score
\nAfter the recent IA (Information Accretion) TSV file update, there are 11,828 GO terms with IA = 0.
\nMy understanding was that zero-IA terms wouldn't contribute to the weighted F-max calculation (since their weight would be zero). However, when I removed all IA=0 term predictions from my best submission, my public LB score dropped significantly (0.9% - 1.6%).
\nThis seems counterintuitive. Could the organizers clarify:
\nAre zero-IA terms actually contributing to the score calculation?\nIs there something about the metric computation I'm misunderstanding?\nAny clarification would be greatly appreciated. Thanks!
", "rawMarkdown": "Hi everyone\n\n@ahphan (sorry for directly tagging)\n\nI have two questions about the private leaderboard evaluation that I'd like to clarify with the organizers:\n\n**1. Predictions Outside Test Superset**\n\nAccording to the data description:\n```\nTest Superset: The test superset is a set of protein sequences on which the participants are asked to predict GO terms...\n\nTest Set: The test set is unknown at the beginning of the competition. It will contain protein sequences (and their functions) from the test superset that gained experimental annotations between the submission deadline and the time of evaluation.\n```\n\nThe testsuperset.fasta file contains 224,309 proteins. However, I've noticed several public notebooks generating predictions for ~270k proteins.\n\nMy question: If a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?\n\nThis affects whether we should invest effort in predicting GO terms for proteins beyond the official test superset.\n\n**2. Zero-IA Terms Affecting Score**\n\nAfter the recent IA (Information Accretion) TSV file update, there are 11,828 GO terms with IA = 0.\n\nMy understanding was that zero-IA terms wouldn't contribute to the weighted F-max calculation (since their weight would be zero). However, when I removed all IA=0 term predictions from my best submission, my public LB score dropped significantly (0.9% - 1.6%).\n\nThis seems counterintuitive. Could the organizers clarify:\n\nAre zero-IA terms actually contributing to the score calculation?\nIs there something about the metric computation I'm misunderstanding?\nAny clarification would be greatly appreciated. Thanks!", "votes": null }, { "id": "3398944", "postDate": "01/30/2026 00:13:54", "content": "The target is clearly defined: the 224,309 proteins in testSUPERset.fasta. But only some of them will be NK-PK-LK. \nsubmitting 270K proteins have no sense for me
", "rawMarkdown": "The target is clearly defined: the 224,309 proteins in testSUPERset.fasta. But only some of them will be NK-PK-LK. \nsubmitting 270K proteins have no sense for me", "votes": null }, { "id": "3399029", "postDate": "01/30/2026 04:40:06", "content": "yeah, me too. Those public notebooks made me really confused ðŸ«
", "rawMarkdown": "yeah, me too. Those public notebooks made me really confused 🫠", "votes": null }, { "id": "3399034", "postDate": "01/30/2026 04:55:50", "content": "\n\nIf a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?
\n
Proteins outside of the test superset will not be considered in private scoring. Evaluation is strictly limited to the 224,309 proteins in testsuperset.fasta
\n\n\nAre zero-IA terms actually contributing to the score calculation?
\n
Zero-IA terms themselves will not contribute to the score calculation. However, I can think of a scenario when there is a child term with IA=0, and this child term propagates to IA>0 parents. If you removed this child term from your predictions, and its parents were not already included in your prediction file, it means that the IA>0 parents don't have a chance to show up through propagation, and might lead to decrease in LB score. That's just my guess of how it could happen.
", "rawMarkdown": ">If a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?\n\nProteins outside of the test superset will not be considered in private scoring. Evaluation is strictly limited to the 224,309 proteins in testsuperset.fasta\n\n>Are zero-IA terms actually contributing to the score calculation?\n\nZero-IA terms themselves will not contribute to the score calculation. However, I can think of a scenario when there is a child term with IA=0, and this child term propagates to IA>0 parents. If you removed this child term from your predictions, and its parents were not already included in your prediction file, it means that the IA>0 parents don't have a chance to show up through propagation, and might lead to decrease in LB score. That's just my guess of how it could happen.", "votes": null }, { "id": "3399142", "postDate": "01/30/2026 09:22:54", "content": "Thanks a lot for your response.
", "rawMarkdown": "Thanks a lot for your response.", "votes": null }, { "id": "3399986", "postDate": "01/31/2026 16:39:46", "content": "There is another point that puzzles me. Shouldn't the confidence score submitted be between 0 and 1? I observed the output of the public code and found that some items in the submission.tsv file even have probabilities higher than 1. How is this possible?
", "rawMarkdown": "There is another point that puzzles me. Shouldn't the confidence score submitted be between 0 and 1? I observed the output of the public code and found that some items in the submission.tsv file even have probabilities higher than 1. How is this possible?", "votes": null }, { "id": "3400013", "postDate": "01/31/2026 17:26:13", "content": "What really matters is the ranking.
\nA score of 50 would be accepted with a 0.99 tau on evaluation, just like a score of 1.
\nI suppose that when people apply transformations to try to calibrate their prediction methods—or ensembles of multiple models—they sometimes go a bit overboard.
", "rawMarkdown": "What really matters is the ranking.\n\nA score of 50 would be accepted with a 0.99 tau on evaluation, just like a score of 1.\n\nI suppose that when people apply transformations to try to calibrate their prediction methods—or ensembles of multiple models—they sometimes go a bit overboard.", "votes": null }, { "id": "3400127", "postDate": "02/01/2026 01:43:13", "content": "Ahhhhhhh, I see, THX. 😀
", "rawMarkdown": "Ahhhhhhh, I see, THX. 😀", "votes": null } ], "comments": [ { "id": 3398944, "author_name": "xaxipirulazo", "author_url": "", "post_date": "01/30/2026 00:13:54", "content": "The target is clearly defined: the 224,309 proteins in testSUPERset.fasta. But only some of them will be NK-PK-LK. \nsubmitting 270K proteins have no sense for me
", "votes": null, "replies": [ { "id": 3399029, "author_name": "andy2709", "author_url": "", "post_date": "01/30/2026 04:40:06", "content": "yeah, me too. Those public notebooks made me really confused ðŸ«
", "votes": null, "replies": [] } ] }, { "id": 3399034, "author_name": "ahphan", "author_url": "", "post_date": "01/30/2026 04:55:50", "content": "\n\nIf a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?
\n
Proteins outside of the test superset will not be considered in private scoring. Evaluation is strictly limited to the 224,309 proteins in testsuperset.fasta
\n\n\nAre zero-IA terms actually contributing to the score calculation?
\n
Zero-IA terms themselves will not contribute to the score calculation. However, I can think of a scenario when there is a child term with IA=0, and this child term propagates to IA>0 parents. If you removed this child term from your predictions, and its parents were not already included in your prediction file, it means that the IA>0 parents don't have a chance to show up through propagation, and might lead to decrease in LB score. That's just my guess of how it could happen.
", "votes": null, "replies": [ { "id": 3399142, "author_name": "andy2709", "author_url": "", "post_date": "01/30/2026 09:22:54", "content": "Thanks a lot for your response.
", "votes": null, "replies": [] } ] }, { "id": 3399986, "author_name": "zejunfool", "author_url": "", "post_date": "01/31/2026 16:39:46", "content": "There is another point that puzzles me. Shouldn't the confidence score submitted be between 0 and 1? I observed the output of the public code and found that some items in the submission.tsv file even have probabilities higher than 1. How is this possible?
", "votes": null, "replies": [ { "id": 3400013, "author_name": "xaxipirulazo", "author_url": "", "post_date": "01/31/2026 17:26:13", "content": "What really matters is the ranking.
\nA score of 50 would be accepted with a 0.99 tau on evaluation, just like a score of 1.
\nI suppose that when people apply transformations to try to calibrate their prediction methods—or ensembles of multiple models—they sometimes go a bit overboard.
", "votes": null, "replies": [ { "id": 3400127, "author_name": "zejunfool", "author_url": "", "post_date": "02/01/2026 01:43:13", "content": "Ahhhhhhh, I see, THX. 😀
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3398662": "Hi everyone\n\n@ahphan (sorry for directly tagging)\n\nI have two questions about the private leaderboard evaluation that I'd like to clarify with the organizers:\n\n**1. Predictions Outside Test Superset**\n\nAccording to the data description:\n```\nTest Superset: The test superset is a set of protein sequences on which the participants are asked to predict GO terms...\n\nTest Set: The test set is unknown at the beginning of the competition. It will contain protein sequences (and their functions) from the test superset that gained experimental annotations between the submission deadline and the time of evaluation.\n```\n\nThe testsuperset.fasta file contains 224,309 proteins. However, I've noticed several public notebooks generating predictions for ~270k proteins.\n\nMy question: If a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?\n\nThis affects whether we should invest effort in predicting GO terms for proteins beyond the official test superset.\n\n**2. Zero-IA Terms Affecting Score**\n\nAfter the recent IA (Information Accretion) TSV file update, there are 11,828 GO terms with IA = 0.\n\nMy understanding was that zero-IA terms wouldn't contribute to the weighted F-max calculation (since their weight would be zero). However, when I removed all IA=0 term predictions from my best submission, my public LB score dropped significantly (0.9% - 1.6%).\n\nThis seems counterintuitive. Could the organizers clarify:\n\nAre zero-IA terms actually contributing to the score calculation?\nIs there something about the metric computation I'm misunderstanding?\nAny clarification would be greatly appreciated. Thanks!", "3398944": "The target is clearly defined: the 224,309 proteins in testSUPERset.fasta. But only some of them will be NK-PK-LK. \nsubmitting 270K proteins have no sense for me", "3399029": "yeah, me too. Those public notebooks made me really confused 🫠", "3399034": ">If a protein outside the test superset gains experimental annotations during the evaluation period, will it be included in private scoring? Or is the evaluation strictly limited to the 224,309 proteins in testsuperset.fasta?\n\nProteins outside of the test superset will not be considered in private scoring. Evaluation is strictly limited to the 224,309 proteins in testsuperset.fasta\n\n>Are zero-IA terms actually contributing to the score calculation?\n\nZero-IA terms themselves will not contribute to the score calculation. However, I can think of a scenario when there is a child term with IA=0, and this child term propagates to IA>0 parents. If you removed this child term from your predictions, and its parents were not already included in your prediction file, it means that the IA>0 parents don't have a chance to show up through propagation, and might lead to decrease in LB score. That's just my guess of how it could happen.", "3399142": "Thanks a lot for your response.", "3399986": "There is another point that puzzles me. Shouldn't the confidence score submitted be between 0 and 1? I observed the output of the public code and found that some items in the submission.tsv file even have probabilities higher than 1. How is this possible?", "3400013": "What really matters is the ranking.\n\nA score of 50 would be accepted with a 0.99 tau on evaluation, just like a score of 1.\n\nI suppose that when people apply transformations to try to calibrate their prediction methods—or ensembles of multiple models—they sometimes go a bit overboard.", "3400127": "Ahhhhhhh, I see, THX. 😀" }, "source": "meta" }