Is the private leaderboard evaluation completed? Why is GOA baseline achieving such a high score, beating many state-of-the-art models? 🥱 In particular, the performance of the GOA baseline significantly improved in the most recent dataset, as shown in this benchmark: https://functionbench.ne
", "messages": [ { "id": 3466046, "postDate": "2026-06-03T11:28:53.187Z", "content": "Non-experimental annotations carry far more predictive signal than people usually assume. If a method does not exploit that prior knowledge, that is a limitation of the method, not an artifact of the baseline.
\nAnd the experimental annotations are not what is inflating the score here: in the evaluation they are handled through the knowledge filter (the no-knowledge / limited-knowledge split), so the GOA baseline is not getting credit it should not have. I checked this and it is done correctly. That is precisely why it is called a baseline, it is the bar any model is expected to clear by using what is already known.
\nThere is actually an even stronger baseline available, QuickGO, which also incorporates third-party / external annotations. Most of those, after curation, will likely transition to IEA/EXP over time.
\nOne thing that is not explicit in the competition description: this task implicitly contains a subproblem. It is less \"what's those proteins function\" and more \"which of the existing annotations will transition to experimental status during this period, and can we enrich those annotations in some other way?\". Framed that way, the strength of the GOA baseline makes a lot of sense.
\nHonestly, this field still has a long way to go, but I think the ceiling depends less on the method you pick (PLM embeddings, sequence similarity, and so on) and more on the data you feed it. How many proteins do you let count as reference to extrapolate from? Do you bring in extra signal such as structure, or published literature, or not? And then, on top of all that, there is the whole curation and calibration apparatus.
", "rawMarkdown": "Non-experimental annotations carry far more predictive signal than people usually assume. If a method does not exploit that prior knowledge, that is a limitation of the method, not an artifact of the baseline.\n\nAnd the experimental annotations are not what is inflating the score here: in the evaluation they are handled through the knowledge filter (the no-knowledge / limited-knowledge split), so the GOA baseline is not getting credit it should not have. I checked this and it is done correctly. That is precisely why it is called a baseline, it is the bar any model is expected to clear by using what is already known.\n\nThere is actually an even stronger baseline available, QuickGO, which also incorporates third-party / external annotations. Most of those, after curation, will likely transition to IEA/EXP over time.\n\nOne thing that is not explicit in the competition description: this task implicitly contains a subproblem. It is less \"what's those proteins function\" and more \"which of the existing annotations will transition to experimental status during this period, and can we enrich those annotations in some other way?\". Framed that way, the strength of the GOA baseline makes a lot of sense.\n\nHonestly, this field still has a long way to go, but I think the ceiling depends less on the method you pick (PLM embeddings, sequence similarity, and so on) and more on the data you feed it. How many proteins do you let count as reference to extrapolate from? Do you bring in extra signal such as structure, or published literature, or not? And then, on top of all that, there is the whole curation and calibration apparatus.", "votes": 3, "replies": [ { "id": 3466197, "postDate": "2026-06-03T18:49:31.423Z", "content": "Small clarification: QuickGO should be similar to the GOA nonexperimental baseline, since they both come from the FTP site of EBI, this directory. From the original goa_uniprot_all.gaf.gz (in UNIPROT directory), we kept only the proteins in the testsuperset.fasta to create the GOA Non-exp baseline. There is another source of annotations that we incorporated (not in the baseline that is currently on the leaderboard) in the public knowledge cutoff at t0 for this competition, that is GO annotations from SwissProt .dat files, here. These two sources of GO annotations (UniProt GOA file on FTP site, and GO annotations in .dat file) do not overlap entirely.
Other than that, I agree with your interpretation.
", "rawMarkdown": "Small clarification: QuickGO should be similar to the GOA nonexperimental baseline, since they both come from the FTP site of EBI, [this directory](https://ftp.ebi.ac.uk/pub/databases/GO/goa/). From the original `goa_uniprot_all.gaf.gz` (in UNIPROT directory), we kept only the proteins in the `testsuperset.fasta` to create the `GOA Non-exp` baseline. There is another source of annotations that we incorporated (not in the baseline that is currently on the leaderboard) in the public knowledge cutoff at t0 for this competition, that is GO annotations from SwissProt .dat files, [here](https://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.dat.gz). These two sources of GO annotations (UniProt GOA file on FTP site, and GO annotations in .dat file) do not overlap entirely. \n\nOther than that, I agree with your interpretation.", "votes": 1, "replies": [ { "id": 3466293, "postDate": "2026-06-03T23:32:08.547Z", "content": "Thanks Ann, that clears it up.
\nI had mostly been going through QuickGO out of habit, its viewer is just easy to read when I want to sanity-check what the method is doing. Any difference I thought I saw was probably down to that, or to how I was extracting and handling the data on my side, rather than the source itself. I take your point that at the source level it and the FTP file should largely coincide. Thanks as well for the pointer on the SwissProt .dat annotations.
", "rawMarkdown": "Thanks Ann, that clears it up.\n\nI had mostly been going through QuickGO out of habit, its viewer is just easy to read when I want to sanity-check what the method is doing. Any difference I thought I saw was probably down to that, or to how I was extracting and handling the data on my side, rather than the source itself. I take your point that at the source level it and the FTP file should largely coincide. Thanks as well for the pointer on the SwissProt .dat annotations.", "votes": 1 }, { "id": 3466482, "postDate": "2026-06-04T07:24:32.430Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 3468226, "postDate": "2026-06-08T17:22:20.820Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3479763, "postDate": "2026-06-24T05:26:47.457Z", "content": "lots of nonsense 😴
", "rawMarkdown": "lots of nonsense 😴", "votes": -1 } ] }, { "id": 3465752, "postDate": "2026-06-02T19:16:33.770Z", "content": "Is the private leaderboard evaluation completed? Why is GOA baseline achieving such a high score, beating many state-of-the-art models? 🥱 In particular, the performance of the GOA baseline significantly improved in the most recent dataset, as shown in this benchmark: https://functionbench.ne
", "rawMarkdown": "Is the private leaderboard evaluation completed? Why is GOA baseline achieving such a high score, beating many state-of-the-art models? 🥱 In particular, the performance of the GOA baseline significantly improved in the most recent dataset, as shown in this benchmark: https://functionbench.ne" } ], "comments": [ { "id": 3466046, "author_name": "xaxipiruli", "author_url": "", "post_date": "2026-06-03T11:28:53.187000", "content": "Non-experimental annotations carry far more predictive signal than people usually assume. If a method does not exploit that prior knowledge, that is a limitation of the method, not an artifact of the baseline.
\nAnd the experimental annotations are not what is inflating the score here: in the evaluation they are handled through the knowledge filter (the no-knowledge / limited-knowledge split), so the GOA baseline is not getting credit it should not have. I checked this and it is done correctly. That is precisely why it is called a baseline, it is the bar any model is expected to clear by using what is already known.
\nThere is actually an even stronger baseline available, QuickGO, which also incorporates third-party / external annotations. Most of those, after curation, will likely transition to IEA/EXP over time.
\nOne thing that is not explicit in the competition description: this task implicitly contains a subproblem. It is less \"what's those proteins function\" and more \"which of the existing annotations will transition to experimental status during this period, and can we enrich those annotations in some other way?\". Framed that way, the strength of the GOA baseline makes a lot of sense.
\nHonestly, this field still has a long way to go, but I think the ceiling depends less on the method you pick (PLM embeddings, sequence similarity, and so on) and more on the data you feed it. How many proteins do you let count as reference to extrapolate from? Do you bring in extra signal such as structure, or published literature, or not? And then, on top of all that, there is the whole curation and calibration apparatus.
", "votes": 3, "replies": [ { "id": 3466197, "author_name": "An Phan", "author_url": "", "post_date": "2026-06-03T18:49:31.423000", "content": "Small clarification: QuickGO should be similar to the GOA nonexperimental baseline, since they both come from the FTP site of EBI, this directory. From the original goa_uniprot_all.gaf.gz (in UNIPROT directory), we kept only the proteins in the testsuperset.fasta to create the GOA Non-exp baseline. There is another source of annotations that we incorporated (not in the baseline that is currently on the leaderboard) in the public knowledge cutoff at t0 for this competition, that is GO annotations from SwissProt .dat files, here. These two sources of GO annotations (UniProt GOA file on FTP site, and GO annotations in .dat file) do not overlap entirely.
Other than that, I agree with your interpretation.
", "votes": 1, "replies": [ { "id": 3466293, "author_name": "xaxipiruli", "author_url": "", "post_date": "2026-06-03T23:32:08.547000", "content": "Thanks Ann, that clears it up.
\nI had mostly been going through QuickGO out of habit, its viewer is just easy to read when I want to sanity-check what the method is doing. Any difference I thought I saw was probably down to that, or to how I was extracting and handling the data on my side, rather than the source itself. I take your point that at the source level it and the FTP file should largely coincide. Thanks as well for the pointer on the SwissProt .dat annotations.
", "votes": 1, "replies": [] }, { "id": 3466482, "author_name": "", "author_url": "", "post_date": "2026-06-04T07:24:32.430000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 3468226, "author_name": "", "author_url": "", "post_date": "2026-06-08T17:22:20.820000", "content": "", "votes": 0, "replies": [] }, { "id": 3479763, "author_name": "Guojun1", "author_url": "", "post_date": "2026-06-24T05:26:47.457000", "content": "lots of nonsense 😴
", "votes": -1, "replies": [] } ] } ], "raw_markdown_by_id": { "3466046": "Non-experimental annotations carry far more predictive signal than people usually assume. If a method does not exploit that prior knowledge, that is a limitation of the method, not an artifact of the baseline.\n\nAnd the experimental annotations are not what is inflating the score here: in the evaluation they are handled through the knowledge filter (the no-knowledge / limited-knowledge split), so the GOA baseline is not getting credit it should not have. I checked this and it is done correctly. That is precisely why it is called a baseline, it is the bar any model is expected to clear by using what is already known.\n\nThere is actually an even stronger baseline available, QuickGO, which also incorporates third-party / external annotations. Most of those, after curation, will likely transition to IEA/EXP over time.\n\nOne thing that is not explicit in the competition description: this task implicitly contains a subproblem. It is less \"what's those proteins function\" and more \"which of the existing annotations will transition to experimental status during this period, and can we enrich those annotations in some other way?\". Framed that way, the strength of the GOA baseline makes a lot of sense.\n\nHonestly, this field still has a long way to go, but I think the ceiling depends less on the method you pick (PLM embeddings, sequence similarity, and so on) and more on the data you feed it. How many proteins do you let count as reference to extrapolate from? Do you bring in extra signal such as structure, or published literature, or not? And then, on top of all that, there is the whole curation and calibration apparatus.", "3465752": "Is the private leaderboard evaluation completed? Why is GOA baseline achieving such a high score, beating many state-of-the-art models? 🥱 In particular, the performance of the GOA baseline significantly improved in the most recent dataset, as shown in this benchmark: https://functionbench.ne" } }