In the Overview > Evaluation > Submission File, it says:
\nThe score must be in the interval (0, 1.000] and contain up to 3 (three) significant figures.
\nApparently, this not being enforced. I just wanted to clarify whether this will or won't be enforced during the curation phase of the competition. Thanks.
In the Overview > Evaluation > Submission File, it says:
\nThe score must be in the interval (0, 1.000] and contain up to 3 (three) significant figures.
\nApparently, this not being enforced. I just wanted to clarify whether this will or won't be enforced during the curation phase of the competition. Thanks.
Hi. As far I know the metric scans the optimum threshold at binarization. So I assume that scan is each 1e-3. So I think you can put as many digits you want, you will obtain the same score. Only submission.tsv will be bigger.
", "rawMarkdown": "Hi. As far I know the metric scans the optimum threshold at binarization. So I assume that scan is each 1e-3. So I think you can put as many digits you want, you will obtain the same score. Only submission.tsv will be bigger.", "votes": null }, { "id": "3331040", "postDate": "11/16/2025 11:11:40", "content": "I think you are right, although I think the scan is done at each 1e-2.
\nMy question is prompted by the fact that I've seen predicted values such as 0.010000128100661499. This is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border, so I'm not sure if it will be included in the same binarization interval or not.
\nAnother issue is that, should the organizers start enforcing the 3 digit rule, then suddenly a submission that was working on the public LB might throw an error during the curation phase…
\n\nThis is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border
\n
I'm not sure if I understand it correctly. But that would be only noticeable if the predictions optimum threshold would be close to 1e-2. I think predictions are a way far from that situation. I think we should just submit rounded to 3 significant digits as beign asked.
", "rawMarkdown": ">This is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border\n\nI'm not sure if I understand it correctly. But that would be only noticeable if the predictions optimum threshold would be close to 1e-2. I think predictions are a way far from that situation. I think we should just submit rounded to 3 significant digits as beign asked.", "votes": null }, { "id": "3331099", "postDate": "11/16/2025 11:28:29", "content": "I just meant that if the scan is made at each 1e-2 (at 0.01, 0.02, 0.03, …, 0.97, 0.98, 0.99) then anything that falls exactly on a border (e.g. 0.45) might be evaluated differently from something slightly above (e.g. 0.4500001) or slightly below (e.g. 0.4499999). If we round to 3 digits, then they are all the same (0.450).
", "rawMarkdown": "I just meant that if the scan is made at each 1e-2 (at 0.01, 0.02, 0.03, ..., 0.97, 0.98, 0.99) then anything that falls exactly on a border (e.g. 0.45) might be evaluated differently from something slightly above (e.g. 0.4500001) or slightly below (e.g. 0.4499999). If we round to 3 digits, then they are all the same (0.450).", "votes": null }, { "id": "3331245", "postDate": "11/16/2025 12:08:53", "content": "Now I get it. Totally true.
", "rawMarkdown": "Now I get it. Totally true.", "votes": null }, { "id": "3332310", "postDate": "11/16/2025 17:01:43", "content": "The 3 significant figure rule is not being enforced for the public leaderboard and won't be enforced in the final evaluation. It is mainly there to discourage submissions with too many significant figures that could lead to larger file size. The current threshold step is 0.01 for the public leaderboard. For now, the organizers have not discussed if this threshold step will change to 0.001 for the final evaluation or not.
", "rawMarkdown": "The 3 significant figure rule is not being enforced for the public leaderboard and won't be enforced in the final evaluation. It is mainly there to discourage submissions with too many significant figures that could lead to larger file size. The current threshold step is 0.01 for the public leaderboard. For now, the organizers have not discussed if this threshold step will change to 0.001 for the final evaluation or not.", "votes": null }, { "id": "3333134", "postDate": "11/17/2025 05:00:32", "content": "Is the threshold optimized per sub-ontology or globally?
", "rawMarkdown": "Is the threshold optimized per sub-ontology or globally?", "votes": null }, { "id": "3333375", "postDate": "11/17/2025 06:49:26", "content": "If it means the prediction with the probability lower than 0.01 well have nothing to the score
", "rawMarkdown": "If it means the prediction with the probability lower than 0.01 well have nothing to the score", "votes": null }, { "id": "3333600", "postDate": "11/17/2025 08:19:48", "content": "I would say globally? Per subontolgy sounds very expensive.
", "rawMarkdown": "I would say globally? Per subontolgy sounds very expensive.", "votes": null }, { "id": "3334438", "postDate": "11/17/2025 14:37:30", "content": "It is per subontology, each subontology is always scored separately, then the best F1 score (at best threshold) for each subontology is recorded, and then averaged across 3 subontologies. Note that each type of knowledge subset (no-knowledge, limited-knowledge, and partial-knowledge) is also evaluated separately, then averaged across 3 subsets. Overall, it is a 9-way average.
", "rawMarkdown": "It is per subontology, each subontology is always scored separately, then the best F1 score (at best threshold) for each subontology is recorded, and then averaged across 3 subontologies. Note that each type of knowledge subset (no-knowledge, limited-knowledge, and partial-knowledge) is also evaluated separately, then averaged across 3 subsets. Overall, it is a 9-way average.", "votes": null }, { "id": "3334442", "postDate": "11/17/2025 14:38:01", "content": "yes, that is correct
", "rawMarkdown": "yes, that is correct", "votes": null }, { "id": "3334537", "postDate": "11/17/2025 15:09:03", "content": "Thanks for clarification. But subontology is each of the 3 branches right? I though @sroger was talking about each go term. My fault.
\nEDIT: Oh. And just to get it clear. The 1500 terms per protein is 500 each subontology right? Not 1500 toguether. I mean like possible 400/600/500, that would effectively be 400/500/500, 100 predictions lost.
", "rawMarkdown": "Thanks for clarification. But subontology is each of the 3 branches right? I though @sroger was talking about each go term. My fault.\n\nEDIT: Oh. And just to get it clear. The 1500 terms per protein is 500 each subontology right? Not 1500 toguether. I mean like possible 400/600/500, that would effectively be 400/500/500, 100 predictions lost.", "votes": null }, { "id": "3334577", "postDate": "11/17/2025 15:19:38", "content": "Yes, 1500 terms per protein is enforced during file parsing on Kaggle, and 500 per subontology is enforced later in cafaevaluator package. So yes, if you submit 400/600/500 it will become 400/500/500.
", "rawMarkdown": "Yes, 1500 terms per protein is enforced during file parsing on Kaggle, and 500 per subontology is enforced later in cafaevaluator package. So yes, if you submit 400/600/500 it will become 400/500/500.", "votes": null }, { "id": "3341171", "postDate": "11/20/2025 02:31:33", "content": "Sir, I have another question: is the private lb now is based on 227 snapshot of uniprot of all protein? I am wondering it
", "rawMarkdown": "Sir, I have another question: is the private lb now is based on 227 snapshot of uniprot of all protein? I am wondering it", "votes": null }, { "id": "3341173", "postDate": "11/20/2025 02:34:47", "content": "There is currently no private leaderboard at the moment. Private leaderboard is just displaying the same thing as public leaderboard.
", "rawMarkdown": "There is currently no private leaderboard at the moment. Private leaderboard is just displaying the same thing as public leaderboard.", "votes": null }, { "id": "3342541", "postDate": "11/21/2025 03:26:16", "content": "Oh, I am wrong, I want to double check the evaluation of the public lb and the label of the training set, for avoiding the data leakage. For the training set, is the annotation from this version 226 of uniprot and 227 for the public lb, right?I saw the discription of the data is not aligned with the release date of the version 226
", "rawMarkdown": "Oh, I am wrong, I want to double check the evaluation of the public lb and the label of the training set, for avoiding the data leakage. For the training set, is the annotation from this version 226 of uniprot and 227 for the public lb, right?I saw the discription of the data is not aligned with the release date of the version 226", "votes": null }, { "id": "3342561", "postDate": "11/21/2025 03:56:34", "content": "The release date inside the goa_uniprot_all.gaf.gz does not matter since UniProt and GOA groups just published files on different dates. The correct version to reproduce training data (GOA file) would be version 226, and the protein sequences were from UniProt release 2025_03 (released 18 June 2025).
\nIn the public leaderboard, known GO terms from version 226 were excluded during evaluation, but version 227 GO terms were not excluded. However, since GOA file version 227 is already public, the known GO terms would be excluded from the private leaderboard during final evaluation.
", "rawMarkdown": "The release date inside the goa_uniprot_all.gaf.gz does not matter since UniProt and GOA groups just published files on different dates. The correct version to reproduce training data (GOA file) would be version 226, and the protein sequences were from UniProt release 2025_03 (released 18 June 2025). \n\nIn the **public** leaderboard, known GO terms from version 226 were excluded during evaluation, but version 227 GO terms were not excluded. However, since GOA file version 227 is already public, the known GO terms would be excluded from the **private** leaderboard during final evaluation.", "votes": null }, { "id": "3343715", "postDate": "11/22/2025 00:01:23", "content": "What type of model do u use? I am trying multilabel classification logistic regression with deep neural networks and different methods to avoid overfitting and cope with class imbalance. I am also trying attention and residual and the ensemble by stacking the 3 NN, my weighted F scores do never jump over 0.2 🤣, I searched for literature in bioinformatics journal and apparently the best method is to combine for training data the sequences with GO term labels, 3 D structures and tertiary structure data, literature text mining and sequence homology. I can never train this with my laptop nor design these models 😅.
", "rawMarkdown": "What type of model do u use? I am trying multilabel classification logistic regression with deep neural networks and different methods to avoid overfitting and cope with class imbalance. I am also trying attention and residual and the ensemble by stacking the 3 NN, my weighted F scores do never jump over 0.2 🤣, I searched for literature in bioinformatics journal and apparently the best method is to combine for training data the sequences with GO term labels, 3 D structures and tertiary structure data, literature text mining and sequence homology. I can never train this with my laptop nor design these models 😅.", "votes": null }, { "id": "3343717", "postDate": "11/22/2025 00:04:46", "content": "I also calculated ProtT5 embeddings
", "rawMarkdown": "I also calculated ProtT5 embeddings", "votes": null }, { "id": "3348889", "postDate": "11/26/2025 09:35:48", "content": "Hello, sir, I have another question. Is the CAFA6 with the largest test set than CAFA1-5. Then if the whole test set will be included in the score in the final private lb or only a part of them.
", "rawMarkdown": "Hello, sir, I have another question. Is the CAFA6 with the largest test set than CAFA1-5. Then if the whole test set will be included in the score in the final private lb or only a part of them.", "votes": null }, { "id": "3348895", "postDate": "11/26/2025 09:38:59", "content": "If all of them have new experimental annotations
", "rawMarkdown": "If all of them have new experimental annotations", "votes": null }, { "id": "3349215", "postDate": "11/26/2025 14:36:22", "content": "Yes, all proteins from the test superset that gain new experimental annotations between submission deadline and final evaluation date June 1, 2026 (including experimental evidence codes and TAS and IC) will be included in the final private lb
", "rawMarkdown": "Yes, all proteins from the test superset that gain new experimental annotations between submission deadline and final evaluation date June 1, 2026 (including experimental evidence codes and TAS and IC) will be included in the final private lb", "votes": null } ], "comments": [ { "id": 3330978, "author_name": "sacuscreed", "author_url": "", "post_date": "11/16/2025 10:57:16", "content": "Hi. As far I know the metric scans the optimum threshold at binarization. So I assume that scan is each 1e-3. So I think you can put as many digits you want, you will obtain the same score. Only submission.tsv will be bigger.
", "votes": null, "replies": [ { "id": 3331040, "author_name": "diogoff", "author_url": "", "post_date": "11/16/2025 11:11:40", "content": "I think you are right, although I think the scan is done at each 1e-2.
\nMy question is prompted by the fact that I've seen predicted values such as 0.010000128100661499. This is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border, so I'm not sure if it will be included in the same binarization interval or not.
\nAnother issue is that, should the organizers start enforcing the 3 digit rule, then suddenly a submission that was working on the public LB might throw an error during the curation phase…
\n\nThis is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border
\n
I'm not sure if I understand it correctly. But that would be only noticeable if the predictions optimum threshold would be close to 1e-2. I think predictions are a way far from that situation. I think we should just submit rounded to 3 significant digits as beign asked.
", "votes": null, "replies": [ { "id": 3331099, "author_name": "diogoff", "author_url": "", "post_date": "11/16/2025 11:28:29", "content": "I just meant that if the scan is made at each 1e-2 (at 0.01, 0.02, 0.03, …, 0.97, 0.98, 0.99) then anything that falls exactly on a border (e.g. 0.45) might be evaluated differently from something slightly above (e.g. 0.4500001) or slightly below (e.g. 0.4499999). If we round to 3 digits, then they are all the same (0.450).
", "votes": null, "replies": [ { "id": 3331245, "author_name": "sacuscreed", "author_url": "", "post_date": "11/16/2025 12:08:53", "content": "Now I get it. Totally true.
", "votes": null, "replies": [] }, { "id": 3343715, "author_name": "joostertryckx", "author_url": "", "post_date": "11/22/2025 00:01:23", "content": "What type of model do u use? I am trying multilabel classification logistic regression with deep neural networks and different methods to avoid overfitting and cope with class imbalance. I am also trying attention and residual and the ensemble by stacking the 3 NN, my weighted F scores do never jump over 0.2 🤣, I searched for literature in bioinformatics journal and apparently the best method is to combine for training data the sequences with GO term labels, 3 D structures and tertiary structure data, literature text mining and sequence homology. I can never train this with my laptop nor design these models 😅.
", "votes": null, "replies": [ { "id": 3343717, "author_name": "joostertryckx", "author_url": "", "post_date": "11/22/2025 00:04:46", "content": "I also calculated ProtT5 embeddings
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3332310, "author_name": "ahphan", "author_url": "", "post_date": "11/16/2025 17:01:43", "content": "The 3 significant figure rule is not being enforced for the public leaderboard and won't be enforced in the final evaluation. It is mainly there to discourage submissions with too many significant figures that could lead to larger file size. The current threshold step is 0.01 for the public leaderboard. For now, the organizers have not discussed if this threshold step will change to 0.001 for the final evaluation or not.
", "votes": null, "replies": [ { "id": 3333134, "author_name": "sroger", "author_url": "", "post_date": "11/17/2025 05:00:32", "content": "Is the threshold optimized per sub-ontology or globally?
", "votes": null, "replies": [ { "id": 3333600, "author_name": "sacuscreed", "author_url": "", "post_date": "11/17/2025 08:19:48", "content": "I would say globally? Per subontolgy sounds very expensive.
", "votes": null, "replies": [ { "id": 3334438, "author_name": "ahphan", "author_url": "", "post_date": "11/17/2025 14:37:30", "content": "It is per subontology, each subontology is always scored separately, then the best F1 score (at best threshold) for each subontology is recorded, and then averaged across 3 subontologies. Note that each type of knowledge subset (no-knowledge, limited-knowledge, and partial-knowledge) is also evaluated separately, then averaged across 3 subsets. Overall, it is a 9-way average.
", "votes": null, "replies": [ { "id": 3334537, "author_name": "sacuscreed", "author_url": "", "post_date": "11/17/2025 15:09:03", "content": "Thanks for clarification. But subontology is each of the 3 branches right? I though @sroger was talking about each go term. My fault.
\nEDIT: Oh. And just to get it clear. The 1500 terms per protein is 500 each subontology right? Not 1500 toguether. I mean like possible 400/600/500, that would effectively be 400/500/500, 100 predictions lost.
", "votes": null, "replies": [ { "id": 3334577, "author_name": "ahphan", "author_url": "", "post_date": "11/17/2025 15:19:38", "content": "Yes, 1500 terms per protein is enforced during file parsing on Kaggle, and 500 per subontology is enforced later in cafaevaluator package. So yes, if you submit 400/600/500 it will become 400/500/500.
", "votes": null, "replies": [ { "id": 3341171, "author_name": "aizealotwu", "author_url": "", "post_date": "11/20/2025 02:31:33", "content": "Sir, I have another question: is the private lb now is based on 227 snapshot of uniprot of all protein? I am wondering it
", "votes": null, "replies": [ { "id": 3341173, "author_name": "ahphan", "author_url": "", "post_date": "11/20/2025 02:34:47", "content": "There is currently no private leaderboard at the moment. Private leaderboard is just displaying the same thing as public leaderboard.
", "votes": null, "replies": [ { "id": 3342541, "author_name": "aizealotwu", "author_url": "", "post_date": "11/21/2025 03:26:16", "content": "Oh, I am wrong, I want to double check the evaluation of the public lb and the label of the training set, for avoiding the data leakage. For the training set, is the annotation from this version 226 of uniprot and 227 for the public lb, right?I saw the discription of the data is not aligned with the release date of the version 226
", "votes": null, "replies": [ { "id": 3342561, "author_name": "ahphan", "author_url": "", "post_date": "11/21/2025 03:56:34", "content": "The release date inside the goa_uniprot_all.gaf.gz does not matter since UniProt and GOA groups just published files on different dates. The correct version to reproduce training data (GOA file) would be version 226, and the protein sequences were from UniProt release 2025_03 (released 18 June 2025).
\nIn the public leaderboard, known GO terms from version 226 were excluded during evaluation, but version 227 GO terms were not excluded. However, since GOA file version 227 is already public, the known GO terms would be excluded from the private leaderboard during final evaluation.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 3333375, "author_name": "aizealotwu", "author_url": "", "post_date": "11/17/2025 06:49:26", "content": "If it means the prediction with the probability lower than 0.01 well have nothing to the score
", "votes": null, "replies": [ { "id": 3334442, "author_name": "ahphan", "author_url": "", "post_date": "11/17/2025 14:38:01", "content": "yes, that is correct
", "votes": null, "replies": [ { "id": 3348889, "author_name": "aizealotwu", "author_url": "", "post_date": "11/26/2025 09:35:48", "content": "Hello, sir, I have another question. Is the CAFA6 with the largest test set than CAFA1-5. Then if the whole test set will be included in the score in the final private lb or only a part of them.
", "votes": null, "replies": [] }, { "id": 3348895, "author_name": "aizealotwu", "author_url": "", "post_date": "11/26/2025 09:38:59", "content": "If all of them have new experimental annotations
", "votes": null, "replies": [ { "id": 3349215, "author_name": "ahphan", "author_url": "", "post_date": "11/26/2025 14:36:22", "content": "Yes, all proteins from the test superset that gain new experimental annotations between submission deadline and final evaluation date June 1, 2026 (including experimental evidence codes and TAS and IC) will be included in the final private lb
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "3330628": "In the Overview > Evaluation > Submission File, it says: \n*The score must be in the interval (0, 1.000] and **contain up to 3 (three) significant figures**.* \nApparently, this not being enforced. I just wanted to clarify whether this will or won't be enforced during the curation phase of the competition. Thanks.", "3330978": "Hi. As far I know the metric scans the optimum threshold at binarization. So I assume that scan is each 1e-3. So I think you can put as many digits you want, you will obtain the same score. Only submission.tsv will be bigger.", "3331040": "I think you are right, although I think the scan is done at each 1e-2. \nMy question is prompted by the fact that I've seen predicted values such as 0.010000128100661499. This is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border, so I'm not sure if it will be included in the same binarization interval or not. \nAnother issue is that, should the organizers start enforcing the 3 digit rule, then suddenly a submission that was working on the public LB might throw an error during the curation phase...", "3331066": ">This is slightly above the binarization border of 0.01. In case we convert to 3 digits, it will be on top of the binarization border\n\nI'm not sure if I understand it correctly. But that would be only noticeable if the predictions optimum threshold would be close to 1e-2. I think predictions are a way far from that situation. I think we should just submit rounded to 3 significant digits as beign asked.", "3331099": "I just meant that if the scan is made at each 1e-2 (at 0.01, 0.02, 0.03, ..., 0.97, 0.98, 0.99) then anything that falls exactly on a border (e.g. 0.45) might be evaluated differently from something slightly above (e.g. 0.4500001) or slightly below (e.g. 0.4499999). If we round to 3 digits, then they are all the same (0.450).", "3331245": "Now I get it. Totally true.", "3332310": "The 3 significant figure rule is not being enforced for the public leaderboard and won't be enforced in the final evaluation. It is mainly there to discourage submissions with too many significant figures that could lead to larger file size. The current threshold step is 0.01 for the public leaderboard. For now, the organizers have not discussed if this threshold step will change to 0.001 for the final evaluation or not.", "3333134": "Is the threshold optimized per sub-ontology or globally?", "3333375": "If it means the prediction with the probability lower than 0.01 well have nothing to the score", "3333600": "I would say globally? Per subontolgy sounds very expensive.", "3334438": "It is per subontology, each subontology is always scored separately, then the best F1 score (at best threshold) for each subontology is recorded, and then averaged across 3 subontologies. Note that each type of knowledge subset (no-knowledge, limited-knowledge, and partial-knowledge) is also evaluated separately, then averaged across 3 subsets. Overall, it is a 9-way average.", "3334442": "yes, that is correct", "3334537": "Thanks for clarification. But subontology is each of the 3 branches right? I though @sroger was talking about each go term. My fault.\n\nEDIT: Oh. And just to get it clear. The 1500 terms per protein is 500 each subontology right? Not 1500 toguether. I mean like possible 400/600/500, that would effectively be 400/500/500, 100 predictions lost.", "3334577": "Yes, 1500 terms per protein is enforced during file parsing on Kaggle, and 500 per subontology is enforced later in cafaevaluator package. So yes, if you submit 400/600/500 it will become 400/500/500.", "3341171": "Sir, I have another question: is the private lb now is based on 227 snapshot of uniprot of all protein? I am wondering it", "3341173": "There is currently no private leaderboard at the moment. Private leaderboard is just displaying the same thing as public leaderboard.", "3342541": "Oh, I am wrong, I want to double check the evaluation of the public lb and the label of the training set, for avoiding the data leakage. For the training set, is the annotation from this version 226 of uniprot and 227 for the public lb, right?I saw the discription of the data is not aligned with the release date of the version 226", "3342561": "The release date inside the goa_uniprot_all.gaf.gz does not matter since UniProt and GOA groups just published files on different dates. The correct version to reproduce training data (GOA file) would be version 226, and the protein sequences were from UniProt release 2025_03 (released 18 June 2025). \n\nIn the **public** leaderboard, known GO terms from version 226 were excluded during evaluation, but version 227 GO terms were not excluded. However, since GOA file version 227 is already public, the known GO terms would be excluded from the **private** leaderboard during final evaluation.", "3343715": "What type of model do u use? I am trying multilabel classification logistic regression with deep neural networks and different methods to avoid overfitting and cope with class imbalance. I am also trying attention and residual and the ensemble by stacking the 3 NN, my weighted F scores do never jump over 0.2 🤣, I searched for literature in bioinformatics journal and apparently the best method is to combine for training data the sequences with GO term labels, 3 D structures and tertiary structure data, literature text mining and sequence homology. I can never train this with my laptop nor design these models 😅.", "3343717": "I also calculated ProtT5 embeddings", "3348889": "Hello, sir, I have another question. Is the CAFA6 with the largest test set than CAFA1-5. Then if the whole test set will be included in the score in the final private lb or only a part of them.", "3348895": "If all of them have new experimental annotations", "3349215": "Yes, all proteins from the test superset that gain new experimental annotations between submission deadline and final evaluation date June 1, 2026 (including experimental evidence codes and TAS and IC) will be included in the final private lb" }, "source": "meta" }