{ "id": 681811, "title": "PROTEA — Interactive GO term prediction & CAFA evaluation sandbox [live demo]", "url": "/competitions/cafa-6-protein-function-prediction/discussion/681811", "author_name": "", "post_date": "2026-03-17T11:44:21.097706800Z", "votes": 6, "comment_count": 3, "views": 0, "content": "

Hey everyone,

\n

After spending months trying to truly understand how CAFA evaluation works (the PK/NK/LK split, the temporal holdout, why Fmax matters), I ended up building a tool to make it tangible.

\n

What it does:

\n

You upload a set of proteins → it computes embeddings (ESM2/ESMC) → does KNN transfer of GO annotations from a reference annotation set → evaluates predictions against a newer annotation set using the CAFA protocol (NK/LK × BPO/MFO/CCO).

\n

The key thing is that you can watch the holdout mechanism live: pick GOA release 200 as training, GOA 229 as ground truth, run predictions, see Fmax per aspect and category. Change the scoring formula, re-evaluate instantly and no re-running inference. It makes the relationship between annotation quality, embedding distance and Fmax very concrete.

\n

Evaluation uses the official cafaeval package (the same evaluator used in the competition) so the numbers should be directly comparable to CAFA 6 submissions if my intuition and implementation are correct, which is part of what I'm trying to validate.

\n

On the data side: we work with the full reviewed UniProtKB/Swiss-Prot historical GOA releases, which gives us a clean temporal split for validation (train on release N, evaluate on release M). We also have QuickGO integrated for production use, pulling live annotations via their API though validating against that set requires waiting for the database to update and re-extracting, so it's better suited for inference than for benchmarking.

\n

Why it might interest you:

\n\n

Live demo: https://protea.ngrok.app

\n

Code: https://github.com/frapercan/PROTEA

\n

It's a thesis project so rough around some edges, but the core pipeline (embedding → KNN → CAFA eval) is solid. Happy to answer questions or hear what you'd do differently.

", "messages": [ { "id": "3422286", "postDate": "03/17/2026 11:44:21", "content": "

Hey everyone,

\n

After spending months trying to truly understand how CAFA evaluation works (the PK/NK/LK split, the temporal holdout, why Fmax matters), I ended up building a tool to make it tangible.

\n

What it does:

\n

You upload a set of proteins → it computes embeddings (ESM2/ESMC) → does KNN transfer of GO annotations from a reference annotation set → evaluates predictions against a newer annotation set using the CAFA protocol (NK/LK × BPO/MFO/CCO).

\n

The key thing is that you can watch the holdout mechanism live: pick GOA release 200 as training, GOA 229 as ground truth, run predictions, see Fmax per aspect and category. Change the scoring formula, re-evaluate instantly and no re-running inference. It makes the relationship between annotation quality, embedding distance and Fmax very concrete.

\n

Evaluation uses the official cafaeval package (the same evaluator used in the competition) so the numbers should be directly comparable to CAFA 6 submissions if my intuition and implementation are correct, which is part of what I'm trying to validate.

\n

On the data side: we work with the full reviewed UniProtKB/Swiss-Prot historical GOA releases, which gives us a clean temporal split for validation (train on release N, evaluate on release M). We also have QuickGO integrated for production use, pulling live annotations via their API though validating against that set requires waiting for the database to update and re-extracting, so it's better suited for inference than for benchmarking.

\n

Why it might interest you:

\n\n

Live demo: https://protea.ngrok.app

\n

Code: https://github.com/frapercan/PROTEA

\n

It's a thesis project so rough around some edges, but the core pipeline (embedding → KNN → CAFA eval) is solid. Happy to answer questions or hear what you'd do differently.

", "rawMarkdown": "Hey everyone,\n\nAfter spending months trying to truly understand how CAFA evaluation works (the PK/NK/LK split, the temporal holdout, why Fmax matters), I ended up building a tool to make it tangible.\n\nWhat it does:\n\nYou upload a set of proteins → it computes embeddings (ESM2/ESMC) → does KNN transfer of GO annotations from a reference annotation set → evaluates predictions against a newer annotation set using the CAFA protocol (NK/LK × BPO/MFO/CCO).\n\nThe key thing is that you can watch the holdout mechanism live: pick GOA release 200 as training, GOA 229 as ground truth, run predictions, see Fmax per aspect and category. Change the scoring formula, re-evaluate instantly and no re-running inference. It makes the relationship between annotation quality, embedding distance and Fmax very concrete.\n\nEvaluation uses the official cafaeval package (the same evaluator used in the competition) so the numbers should be directly comparable to CAFA 6 submissions if my intuition and implementation are correct, which is part of what I'm trying to validate. \n\nOn the data side: we work with the full reviewed UniProtKB/Swiss-Prot historical GOA releases, which gives us a clean temporal split for validation (train on release N, evaluate on release M). We also have QuickGO integrated for production use, pulling live annotations via their API though validating against that set requires waiting for the database to update and re-extracting, so it's better suited for inference than for benchmarking.\n\nWhy it might interest you:\n\n- If you're preparing for CAFA 7 and want to understand the evaluation protocol hands-on before submitting\n- If you're into the engineering side: the system is fully distributed and scalable (FastAPI + RabbitMQ workers + pgvector) \n- If you're building protein function predictors and want a reproducible benchmark harness\n- If you just want to see what PLM embeddings actually buy you for GO annotation transfer\n\nLive demo: https://protea.ngrok.app\n\nCode: https://github.com/frapercan/PROTEA\n\nIt's a thesis project so rough around some edges, but the core pipeline (embedding → KNN → CAFA eval) is solid. Happy to answer questions or hear what you'd do differently.", "votes": null }, { "id": "3438600", "postDate": "04/09/2026 12:13:05", "content": "

good job, man! hope we could get the good result this time

", "rawMarkdown": "good job, man! hope we could get the good result this time", "votes": null }, { "id": "3440098", "postDate": "04/11/2026 22:54:17", "content": "

thanks mate

", "rawMarkdown": "thanks mate", "votes": null }, { "id": "3440102", "postDate": "04/11/2026 23:18:53", "content": "

Update — 12 Apr 2026

\n

Quick heads up on the current state of the demo:

\n

Right now the system is computing embeddings for 8 protein language models (ESM2-650M, ESM2-3B, ESMC-300M, ESMC-600M, Ankh-base, Ankh-large, ProtT5-XL, ProstT5-XL) over the full Swiss-Prot reference set. Until those finish, any sequence you upload won't start computing until it reaches the front of the queue, so annotations may take a while to come back — probably until the end of the week.

\n

I also can't promise the service will be completely bug-free during this window. I'm making constant changes to the code, and this is effectively my research production machine while I try to get results together for a paper. If you hit an error or something looks off, it's very likely me pushing something that broke half an hour earlier.

\n

Thanks for the attention.

", "rawMarkdown": "**Update — 12 Apr 2026**\n\nQuick heads up on the current state of the demo:\n\nRight now the system is computing embeddings for **8 protein language models** (ESM2-650M, ESM2-3B, ESMC-300M, ESMC-600M, Ankh-base, Ankh-large, ProtT5-XL, ProstT5-XL) over the full Swiss-Prot reference set. Until those finish, any sequence you upload won't start computing until it reaches the front of the queue, so annotations may take a while to come back — probably until the end of the week.\n\nI also can't promise the service will be completely bug-free during this window. I'm making constant changes to the code, and this is effectively my research production machine while I try to get results together for a paper. If you hit an error or something looks off, it's very likely me pushing something that broke half an hour earlier.\n\nThanks for the attention.", "votes": null } ], "comments": [ { "id": 3438600, "author_name": "aizealotwu", "author_url": "", "post_date": "04/09/2026 12:13:05", "content": "

good job, man! hope we could get the good result this time

", "votes": null, "replies": [ { "id": 3440098, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/11/2026 22:54:17", "content": "

thanks mate

", "votes": null, "replies": [] } ] }, { "id": 3440102, "author_name": "xaxipirulazo", "author_url": "", "post_date": "04/11/2026 23:18:53", "content": "

Update — 12 Apr 2026

\n

Quick heads up on the current state of the demo:

\n

Right now the system is computing embeddings for 8 protein language models (ESM2-650M, ESM2-3B, ESMC-300M, ESMC-600M, Ankh-base, Ankh-large, ProtT5-XL, ProstT5-XL) over the full Swiss-Prot reference set. Until those finish, any sequence you upload won't start computing until it reaches the front of the queue, so annotations may take a while to come back — probably until the end of the week.

\n

I also can't promise the service will be completely bug-free during this window. I'm making constant changes to the code, and this is effectively my research production machine while I try to get results together for a paper. If you hit an error or something looks off, it's very likely me pushing something that broke half an hour earlier.

\n

Thanks for the attention.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3422286": "Hey everyone,\n\nAfter spending months trying to truly understand how CAFA evaluation works (the PK/NK/LK split, the temporal holdout, why Fmax matters), I ended up building a tool to make it tangible.\n\nWhat it does:\n\nYou upload a set of proteins → it computes embeddings (ESM2/ESMC) → does KNN transfer of GO annotations from a reference annotation set → evaluates predictions against a newer annotation set using the CAFA protocol (NK/LK × BPO/MFO/CCO).\n\nThe key thing is that you can watch the holdout mechanism live: pick GOA release 200 as training, GOA 229 as ground truth, run predictions, see Fmax per aspect and category. Change the scoring formula, re-evaluate instantly and no re-running inference. It makes the relationship between annotation quality, embedding distance and Fmax very concrete.\n\nEvaluation uses the official cafaeval package (the same evaluator used in the competition) so the numbers should be directly comparable to CAFA 6 submissions if my intuition and implementation are correct, which is part of what I'm trying to validate. \n\nOn the data side: we work with the full reviewed UniProtKB/Swiss-Prot historical GOA releases, which gives us a clean temporal split for validation (train on release N, evaluate on release M). We also have QuickGO integrated for production use, pulling live annotations via their API though validating against that set requires waiting for the database to update and re-extracting, so it's better suited for inference than for benchmarking.\n\nWhy it might interest you:\n\n- If you're preparing for CAFA 7 and want to understand the evaluation protocol hands-on before submitting\n- If you're into the engineering side: the system is fully distributed and scalable (FastAPI + RabbitMQ workers + pgvector) \n- If you're building protein function predictors and want a reproducible benchmark harness\n- If you just want to see what PLM embeddings actually buy you for GO annotation transfer\n\nLive demo: https://protea.ngrok.app\n\nCode: https://github.com/frapercan/PROTEA\n\nIt's a thesis project so rough around some edges, but the core pipeline (embedding → KNN → CAFA eval) is solid. Happy to answer questions or hear what you'd do differently.", "3438600": "good job, man! hope we could get the good result this time", "3440098": "thanks mate", "3440102": "**Update — 12 Apr 2026**\n\nQuick heads up on the current state of the demo:\n\nRight now the system is computing embeddings for **8 protein language models** (ESM2-650M, ESM2-3B, ESMC-300M, ESMC-600M, Ankh-base, Ankh-large, ProtT5-XL, ProstT5-XL) over the full Swiss-Prot reference set. Until those finish, any sequence you upload won't start computing until it reaches the front of the queue, so annotations may take a while to come back — probably until the end of the week.\n\nI also can't promise the service will be completely bug-free during this window. I'm making constant changes to the code, and this is effectively my research production machine while I try to get results together for a paper. If you hit an error or something looks off, it's very likely me pushing something that broke half an hour earlier.\n\nThanks for the attention." }, "source": "meta" }