We used the following data versions to create the ground truth for the March 2026 update of the leaderboard (leaderboard rescore is still pending at the moment):
\ntestsuperset.fasta file (224,309 proteins originally).New annotations (of proteins in the 224,121 unchanged sequences) are collected between t0 and t1 (and categorized into No Knowledge, Limited Knowledge, Partial Knowledge). Only GO terms that exist in both the original frozen graph (go-basic.obo in Data tab) and t0 graph are included in the ground truth creation (called terms-of-interest).
The full configuration that was used to run cafaevaluator (example for PK evaluation):
cafaeval \"go-basic.obo\" \"predictions/\" \"Mar2026_groundtruth_PK.tsv\" -ia \"IA.tsv\" -known \"Mar2026_groundtruth_PK_known.tsv\" -no_orphans -norm cafa -prop fill -toi \"Mar2026_groundtruth_terms_of_interest.txt\" -max_terms 500 -th_step 0.001\"go-basic.obo\": original frozen graph in Data tab. \"IA.tsv\": original Information Accretion file in Data tab.\"Mar2026_groundtruth_PK.tsv\": new annotations collected between t0 and t1. -known \"Mar2026_groundtruth_PK_known.tsv\": public annotations that existed up to t0 (see public knowledge cutoff above). Only PK evaluation uses -known argument-toi \"Mar2026_groundtruth_terms_of_interest.txt\": contains GO terms that exist in both the original frozen graph (go-basic.obo in Data tab) and t0 graphth_step is now 0.001 to provide finer resolution (previously 0.01)We used the following data versions to create the ground truth for the March 2026 update of the leaderboard (leaderboard rescore is still pending at the moment):
\ntestsuperset.fasta file (224,309 proteins originally).New annotations (of proteins in the 224,121 unchanged sequences) are collected between t0 and t1 (and categorized into No Knowledge, Limited Knowledge, Partial Knowledge). Only GO terms that exist in both the original frozen graph (go-basic.obo in Data tab) and t0 graph are included in the ground truth creation (called terms-of-interest).
The full configuration that was used to run cafaevaluator (example for PK evaluation):
cafaeval \"go-basic.obo\" \"predictions/\" \"Mar2026_groundtruth_PK.tsv\" -ia \"IA.tsv\" -known \"Mar2026_groundtruth_PK_known.tsv\" -no_orphans -norm cafa -prop fill -toi \"Mar2026_groundtruth_terms_of_interest.txt\" -max_terms 500 -th_step 0.001\"go-basic.obo\": original frozen graph in Data tab. \"IA.tsv\": original Information Accretion file in Data tab.\"Mar2026_groundtruth_PK.tsv\": new annotations collected between t0 and t1. -known \"Mar2026_groundtruth_PK_known.tsv\": public annotations that existed up to t0 (see public knowledge cutoff above). Only PK evaluation uses -known argument-toi \"Mar2026_groundtruth_terms_of_interest.txt\": contains GO terms that exist in both the original frozen graph (go-basic.obo in Data tab) and t0 graphth_step is now 0.001 to provide finer resolution (previously 0.01)Public knowledge and Most updated GOA dates have no sense for me.
\nT0 2026-12-04 > T1 2026-03-04
\nTypo?
", "rawMarkdown": "Public knowledge and Most updated GOA dates have no sense for me.\n\nT0 2026-12-04 > T1 2026-03-04 \n\nTypo?", "votes": null }, { "id": "3433980", "postDate": "04/02/2026 12:06:58", "content": "Thanks for catching the typo, I have updated the post\nT0 2025-12-04, and T1 2026-03-04
", "rawMarkdown": "Thanks for catching the typo, I have updated the post\nT0 2025-12-04, and T1 2026-03-04", "votes": null }, { "id": "3434463", "postDate": "04/03/2026 00:39:57", "content": "Dear organizer, after the March 2026 leaderboard update, all our submissions show errors. Could you please help us check this? @ahphan
", "rawMarkdown": "Dear organizer, after the March 2026 leaderboard update, all our submissions show errors. Could you please help us check this? @ahphan", "votes": null }, { "id": "3434971", "postDate": "04/03/2026 14:28:14", "content": "Kaggle is reporting delay in rescoring due to compute capacity. For now, the leaderboard is still not fully rescored yet.
", "rawMarkdown": "Kaggle is reporting delay in rescoring due to compute capacity. For now, the leaderboard is still not fully rescored yet.", "votes": null }, { "id": "3436196", "postDate": "04/05/2026 13:36:19", "content": "Got it, thank you for your reply!
", "rawMarkdown": "Got it, thank you for your reply!", "votes": null }, { "id": "3438589", "postDate": "04/09/2026 12:01:27", "content": "hi, Has it been scored?
", "rawMarkdown": "hi, Has it been scored?", "votes": null } ], "comments": [ { "id": 3433917, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/02/2026 10:08:36", "content": "Public knowledge and Most updated GOA dates have no sense for me.
\nT0 2026-12-04 > T1 2026-03-04
\nTypo?
", "votes": null, "replies": [ { "id": 3433980, "author_name": "ahphan", "author_url": "", "post_date": "04/02/2026 12:06:58", "content": "Thanks for catching the typo, I have updated the post\nT0 2025-12-04, and T1 2026-03-04
", "votes": null, "replies": [] } ] }, { "id": 3434463, "author_name": "discovering", "author_url": "", "post_date": "04/03/2026 00:39:57", "content": "Dear organizer, after the March 2026 leaderboard update, all our submissions show errors. Could you please help us check this? @ahphan
", "votes": null, "replies": [ { "id": 3434971, "author_name": "ahphan", "author_url": "", "post_date": "04/03/2026 14:28:14", "content": "Kaggle is reporting delay in rescoring due to compute capacity. For now, the leaderboard is still not fully rescored yet.
", "votes": null, "replies": [ { "id": 3436196, "author_name": "discovering", "author_url": "", "post_date": "04/05/2026 13:36:19", "content": "Got it, thank you for your reply!
", "votes": null, "replies": [] }, { "id": 3438589, "author_name": "aizealotwu", "author_url": "", "post_date": "04/09/2026 12:01:27", "content": "hi, Has it been scored?
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3431847": "# Data versions:\nWe used the following data versions to create the ground truth for the March 2026 update of the leaderboard (**leaderboard rescore is still pending** at the moment):\n- Most updated GOA (**t1**): GOA UniProt file released on 2026-03-04, [here](https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz)\n- Public knowledge cutoff (**t0**): GOA UniProt file released on 2025-12-04, [here](https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.229.gz). Any GO terms included in this public knowledge **are not included** in the ground truth.\n- GO graph for public knowledge cutoff (**t0 graph**): OBO file released on 2025-10-10, [here](https://release.geneontology.org/2025-10-10/ontology/go-basic.obo). This graph is used to propagate all annotations at t0. \n- UniProt SwissProt sequences: UniProt FASTA file released on 2026-01-28, [here](https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz). This file is only used to compare current SwissProt protein sequences with the original `testsuperset.fasta` file (224,309 proteins originally).\n - Proteins with unchanged sequences: 224,121. Only these proteins will move on to ground truth creation.\n - Proteins removed from SwissProt: 19\n - Proteins with sequences changed: 169\n\n#Ground truth creation:\nNew annotations (of proteins in the 224,121 unchanged sequences) are collected between t0 and t1 (and categorized into No Knowledge, Limited Knowledge, Partial Knowledge). Only GO terms that exist in both the original frozen graph (`go-basic.obo` in Data tab) and t0 graph are included in the ground truth creation (called terms-of-interest). \n\n#Evaluation: \nThe full configuration that was used to run `cafaevaluator` (example for PK evaluation): \n\n- `cafaeval \"go-basic.obo\" \"predictions/\" \"Mar2026_groundtruth_PK.tsv\" -ia \"IA.tsv\" -known \"Mar2026_groundtruth_PK_known.tsv\" -no_orphans -norm cafa -prop fill -toi \"Mar2026_groundtruth_terms_of_interest.txt\" -max_terms 500 -th_step 0.001`\n- `\"go-basic.obo\"`: original frozen graph in Data tab. `\"IA.tsv\"`: original Information Accretion file in Data tab.\n- `\"Mar2026_groundtruth_PK.tsv\"`: new annotations collected between t0 and t1. \n- ********`-known \"Mar2026_groundtruth_PK_known.tsv\"`: public annotations that existed up to t0 (see public knowledge cutoff above). Only PK evaluation uses `-known` argument\n- `-toi \"Mar2026_groundtruth_terms_of_interest.txt\"`: contains GO terms that exist in both the original frozen graph (`go-basic.obo` in Data tab) and t0 graph\n- **`th_step` is now 0.001** to provide finer resolution (previously 0.01)\n\n#Statistics:\n- Number of NK proteins: roughly 123 (plus a number of proteins in the hidden test set)\n- Number of LK proteins: roughly 231 (plus a number of proteins in the hidden test set)\n- Number of PK proteins: roughly 2308 (plus a number of proteins in the hidden test set)", "3433917": "Public knowledge and Most updated GOA dates have no sense for me.\n\nT0 2026-12-04 > T1 2026-03-04 \n\nTypo?", "3433980": "Thanks for catching the typo, I have updated the post\nT0 2025-12-04, and T1 2026-03-04", "3434463": "Dear organizer, after the March 2026 leaderboard update, all our submissions show errors. Could you please help us check this? @ahphan", "3434971": "Kaggle is reporting delay in rescoring due to compute capacity. For now, the leaderboard is still not fully rescored yet.", "3436196": "Got it, thank you for your reply!", "3438589": "hi, Has it been scored?" }, "source": "meta" }