As the CAFA 6 challenge is winding down, I would like to thank you for all your hard work. I would also like to inform you of an exciting opportunity to showcase your work further.
\nLAFA is a web server we recently developed that constantly assesses protein function prediction models. The models are containerized and run on periodically updated benchmarks of protein function annotations compiled from several sources.
\nIf you wish to add your model to those already on the LAFA server, please fill out this form and someone from our team will get back to you. This is an excellent opportunity to showcase your work.
\nThere is no obligation on your part to provide your model, and you can always request that it be removed. All we ask is that you license your method under an open source license and work with us to containerize it for LAFA.
\nFeel free to contact us with any questions you may have.
\n\n\n", "messages": [ { "id": "3462008", "postDate": "05/21/2026 22:56:31", "content": "As the CAFA 6 challenge is winding down, I would like to thank you for all your hard work. I would also like to inform you of an exciting opportunity to showcase your work further.
\nLAFA is a web server we recently developed that constantly assesses protein function prediction models. The models are containerized and run on periodically updated benchmarks of protein function annotations compiled from several sources.
\nIf you wish to add your model to those already on the LAFA server, please fill out this form and someone from our team will get back to you. This is an excellent opportunity to showcase your work.
\nThere is no obligation on your part to provide your model, and you can always request that it be removed. All we ask is that you license your method under an open source license and work with us to containerize it for LAFA.
\nFeel free to contact us with any questions you may have.
\n\n\n", "rawMarkdown": "As the CAFA 6 challenge is winding down, I would like to thank you for all your hard work. I would also like to inform you of an exciting opportunity to showcase your work further. \n\nLAFA is a [web server](https://functionbench.net/) we recently developed that constantly assesses protein function prediction models. The models are containerized and run on periodically updated benchmarks of protein function annotations compiled from several sources.\n\nIf you wish to add your model to those already on the LAFA server, please [fill out this form](https://forms.gle/x3TZioBXUE5EkYMq6) and someone from our team will get back to you. This is an excellent opportunity to showcase your work.\n\nThere is no obligation on your part to provide your model, and you can always request that it be removed. All we ask is that you license your method under an open source license and work with us to containerize it for LAFA. \n\n\nFeel free to contact us with any questions you may have. \n\n[The LAFA server](https://functionbench.net/)\n\n[A preprint describing LAFA](http://arxiv.org/abs/2604.20782)\n\n[How to containerize](https://github.com/anphan0828/LAFA_container_guide)", "votes": null }, { "id": "3462548", "postDate": "05/23/2026 19:17:59", "content": "Great news !\nI wonder would it possible to create snapshots of LAFA as community Kaggle competitions ?\nOn Kaggle it is easy to create such competitions and everyone can create and participate it for free. \nThere will be not many participants - community competitions do not offer prizes. \nBut for those who are interested - it would be great to have such a benchmark.
\nYou can even make leaderboard targets publicly available - but since there will no prizes - there will no reason to cheat. (If someone would submit perfect prediction or prediction with score greater 0.7 - just ban him).
\nSo it might complement LAFA, leaving aside containerization issue, which might be simple for some models/modelers.
", "rawMarkdown": "Great news !\nI wonder would it possible to create snapshots of LAFA as community Kaggle competitions ?\nOn Kaggle it is easy to create such competitions and everyone can create and participate it for free. \nThere will be not many participants - community competitions do not offer prizes. \nBut for those who are interested - it would be great to have such a benchmark.\n\nYou can even make leaderboard targets publicly available - but since there will no prizes - there will no reason to cheat. (If someone would submit perfect prediction or prediction with score greater 0.7 - just ban him). \n\nSo it might complement LAFA, leaving aside containerization issue, which might be simple for some models/modelers.", "votes": null }, { "id": "3463248", "postDate": "05/26/2026 11:44:24", "content": "Thanks for the heads up. I'm currently recomputing my pipeline to align it with the LAFA evaluation window. The mechanism to publish LAFA-compatible containers is already in place on my side, so I'll be able to showcase my best datasets and models through it. Once I wrap up experimentation and stabilize the software versioning, I'll send my method over.
\nThanks!
", "rawMarkdown": "Thanks for the heads up. I'm currently recomputing my pipeline to align it with the LAFA evaluation window. The mechanism to publish LAFA-compatible containers is already in place on my side, so I'll be able to showcase my best datasets and models through it. Once I wrap up experimentation and stabilize the software versioning, I'll send my method over. \n\nThanks!", "votes": null }, { "id": "3463267", "postDate": "05/26/2026 13:00:25", "content": "Are the train and validation dataset splits available locally (i.e from the github)? That's always been a pain point in evaluating CAFA historically (getting train+test)
", "rawMarkdown": "Are the train and validation dataset splits available locally (i.e from the github)? That's always been a pain point in evaluating CAFA historically (getting train+test)", "votes": null }, { "id": "3463335", "postDate": "05/26/2026 16:02:09", "content": "A few reasons why Kaggle is not a good fit for running LAFA:
\nKaggle uses only one metric to assess predictions, which flattens the different GO terms, and the no-knowledge / limited knowledge / partial knowledge annotations. It also uses only precision recall, without incorporating GO semantics. LAFA uses all the classic CAFA paper metrics.
I'm not sure how continuous modeling will work on Kaggle. People would need to commit to putting the exact same models each time for continous assessments.
Yes, at each data release (rougly 2 months), we will prepare a train dataset (proteins with experimentally validated terms) and upload it on HuggingFace. See past datasets here. It is still a prospective evaluation, i.e., the evaluation will be performed as annotation data continues to accumulate. The key difference is that the evaluation will be updated with new data and head-to-head evaluation will be available every 2 months as well.
", "rawMarkdown": "Yes, at each data release (rougly 2 months), we will prepare a train dataset (proteins with experimentally validated terms) and upload it on HuggingFace. See past datasets [here](https://huggingface.co/datasets/anphan0828/lafa/tree/main). It is still a prospective evaluation, i.e., the evaluation will be performed as annotation data continues to accumulate. The key difference is that the evaluation will be updated with new data and head-to-head evaluation will be available every 2 months as well.", "votes": null }, { "id": "3464491", "postDate": "05/30/2026 03:29:47", "content": "Hi, I just submitted the LAFA intake form. My CAFA 6 score was modest, but I think the continuous evaluation setup of LAFA may suit my approach better. I've prepared a containerized, MIT-licensed pipeline and already filled out the form.
", "rawMarkdown": "Hi, I just submitted the LAFA intake form. My CAFA 6 score was modest, but I think the continuous evaluation setup of LAFA may suit my approach better. I've prepared a containerized, MIT-licensed pipeline and already filled out the form.", "votes": null }, { "id": "3466628", "postDate": "06/04/2026 11:37:01", "content": "Following up again, this time with a couple of logistics questions for the hosts.\nOn my entry: PROTEA-KNN shares the protein language model with the LAFA ProtT5 baseline (both build on ProtT5 embeddings), but the method itself is different. It is a KNN that uses the full reviewed UniProtKB/Swiss-Prot set as reference, with K selected per GO aspect rather than per protein, plus a different scoring scheme on top. So it is not a competitive, heavily engineered entry yet, more of an honest, baseline-adjacent starting point. I still need to document it properly, align the packaging to your conventions, and finish a round of experimentation before sending it over.\nWhat I have so far:
\nContainer: ghcr.io/frapercan/protea/knn-v1
\nDataset: XaxiPiruli/protea-lafa-knn-v227
\nLive instance: https://protea-lafa.ngrok.app/ (temporary demo for debugging purpouses, may be offline)
\nI've also submitted the intake form.\nA few questions on how you plan to run this:
\nContribution mechanism. Is the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?\nEntries per participant. Do you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?\nVersioning over time. Since LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side? This ties back to the earlier point about committing the same model for each assessment.
\nOne thing I want to flag honestly: what I have now is deliberately naive. I can already see clear ways to push the score up, for example concatenating with the baselines, adding InterProScan-derived features, or increasing the number of neighbors for under-represented proteins. Before I go down that path, I'd rather know whether richer, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility. Any recommendation on what is most useful to LAFA would help me prioritize.\nThanks again.
", "rawMarkdown": "Following up again, this time with a couple of logistics questions for the hosts.\nOn my entry: PROTEA-KNN shares the protein language model with the LAFA ProtT5 baseline (both build on ProtT5 embeddings), but the method itself is different. It is a KNN that uses the full reviewed UniProtKB/Swiss-Prot set as reference, with K selected per GO aspect rather than per protein, plus a different scoring scheme on top. So it is not a competitive, heavily engineered entry yet, more of an honest, baseline-adjacent starting point. I still need to document it properly, align the packaging to your conventions, and finish a round of experimentation before sending it over.\nWhat I have so far:\n\nContainer: ghcr.io/frapercan/protea/knn-v1\n\nDataset: XaxiPiruli/protea-lafa-knn-v227\n\nLive instance: https://protea-lafa.ngrok.app/ (temporary demo for debugging purpouses, may be offline)\n\nI've also submitted the intake form.\nA few questions on how you plan to run this:\n\nContribution mechanism. Is the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?\nEntries per participant. Do you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?\nVersioning over time. Since LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side? This ties back to the earlier point about committing the same model for each assessment.\n\nOne thing I want to flag honestly: what I have now is deliberately naive. I can already see clear ways to push the score up, for example concatenating with the baselines, adding InterProScan-derived features, or increasing the number of neighbors for under-represented proteins. Before I go down that path, I'd rather know whether richer, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility. Any recommendation on what is most useful to LAFA would help me prioritize.\nThanks again.", "votes": null }, { "id": "3466681", "postDate": "06/04/2026 13:53:43", "content": "\n\nIs the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?
\n
For now, purely the intake form plus working with us to containerize, and have your containerized method publicly accessible (I see that you have uploaded it on GitHub container registry). There are some hard requirements about the template (i.e., input and output format, container must run offline) which can be found in the container guide. Other details such as GPU requirements, size of input batch, can be communicated with us and we'll make it work on our LAFA backend.
\n\n\nDo you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?
\n
We don't limit number of methods per participant, since LAFA is designed to be a benchmarking system and not a competition. However, we do want to choose the most competitive methods to be displayed in order to capture the state-of-the-art of the protein function prediction field.
\n\n\nSince LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side?
\n
Let's say you submit \"PROTEA-KNN\" right now (under a format of a container image). \"PROTEA-KNN\" will enter LAFA starting Jun_2026 timepoint. We store those predictions generated using Jun_2026 test set, and evaluate these on multiple future time windows (e.g., Jun_2026 to Aug_2026, Jun_2026 to Oct_2026, Jun_2026 to Dec_2026, and also Aug_2026 to Oct_2026 and so on). This constitutes the head-to-head comparison across releases for PROTEA-KNN, as longer accumulation windows have a larger benchmark set and more accurate evaluation.
\nAdditionally, you can also choose to update PROTEA-KNN (e.g., re-training with newer data in Aug_2026) and submit the container image to LAFA again. The retrained model will be named \"PROTEA-KNN v1\" and will also be evaluated on future time windows similar to \"PROTEA-KNN\", giving us a new insight on how training data recency affect model performance. For example, does PROTEA-KNN (with outdated training data) perform worse than PROTEA-KNN v1 on the same time window Aug_2026 to Oct_2026?. LAFA allows us to look into it. More details can be found in our preprint
\n\n\nRicher, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility.
\n
We welcome richer, ensembled methods, as long as they are still containerized, usable and reproducible for anyone who wishes to use them (we will test them first before displaying them on the LAFA server), as this is the goal of LAFA. Larger methods might require external data storage for method artifacts, weights, etc, so method developers are required to host the external data storage in a persistent data volume, ensuring usability.
", "rawMarkdown": ">Is the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?\n\nFor now, purely the intake form plus working with us to containerize, and have your containerized method publicly accessible (I see that you have uploaded it on GitHub container registry). There are some hard requirements about the template (i.e., input and output format, container must run offline) which can be found in the [container guide](https://github.com/anphan0828/LAFA_container_guide). Other details such as GPU requirements, size of input batch, can be communicated with us and we'll make it work on our LAFA backend.\n\n>Do you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?\n\nWe don't limit number of methods per participant, since LAFA is designed to be a benchmarking system and not a competition. However, we do want to choose the most competitive methods to be displayed in order to capture the state-of-the-art of the protein function prediction field.\n\n>Since LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side?\n\nLet's say you submit \"PROTEA-KNN\" right now (under a format of a container image). \"PROTEA-KNN\" will enter LAFA starting Jun_2026 timepoint. We store those predictions generated using Jun_2026 test set, and evaluate these on multiple future time windows (e.g., Jun_2026 to Aug_2026, Jun_2026 to Oct_2026, Jun_2026 to Dec_2026, and also Aug_2026 to Oct_2026 and so on). This constitutes the head-to-head comparison across releases for PROTEA-KNN, as longer accumulation windows have a larger benchmark set and more accurate evaluation. \n\nAdditionally, you can also choose to update PROTEA-KNN (e.g., re-training with newer data in Aug_2026) and submit the container image to LAFA again. The retrained model will be named \"**PROTEA-KNN v1**\" and will also be evaluated on future time windows similar to \"PROTEA-KNN\", giving us a new insight on how training data recency affect model performance. For example, does PROTEA-KNN (with outdated training data) perform worse than PROTEA-KNN v1 on the same time window Aug_2026 to Oct_2026?. LAFA allows us to look into it. More details can be found in our [preprint](https://arxiv.org/pdf/2604.20782)\n\n>Richer, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility. \n\nWe welcome richer, ensembled methods, as long as they are still containerized, usable and reproducible for anyone who wishes to use them (we will test them first before displaying them on the LAFA server), as this is the goal of LAFA. Larger methods might require external data storage for method artifacts, weights, etc, so method developers are required to host the external data storage in a persistent data volume, ensuring usability.", "votes": null }, { "id": "3529923", "postDate": "09/29/2026 07:35:04", "content": "Hi An and Iddo,
\nI submitted the intake form about a week ago and have the container ready and publicly available on Docker Hub, following the LAFA container guide. I mainly wanted to make sure my form was received and that I haven't missed any additional step on my side.
\nI also noticed that the public LAFA evaluation windows do not seem to have been updated recently. Is LAFA currently accepting and integrating new methods, and are new evaluation windows still planned?
\nThanks!\nAntonina
", "rawMarkdown": "Hi An and Iddo,\n\nI submitted the intake form about a week ago and have the container ready and publicly available on Docker Hub, following the LAFA container guide. I mainly wanted to make sure my form was received and that I haven't missed any additional step on my side.\n\nI also noticed that the public LAFA evaluation windows do not seem to have been updated recently. Is LAFA currently accepting and integrating new methods, and are new evaluation windows still planned?\n\nThanks!\nAntonina", "votes": null }, { "id": "3530088", "postDate": "09/29/2026 14:08:48", "content": "Hi Antonina,\nThank you for your interest in LAFA. Please email me or Iddo for further instructions on submitting the container to LAFA or any questions in general.
\nRegarding public LAFA evaluation, the backend is undergoing substantial improvements following the reviewers' comments on LAFA paper to incorporate organism-specific GO databases for releases after March 2026. We expect to have the new releases within 1-2 weeks (Jun 2026 release and Sep 2026 release). LAFA is actively accepting methods; however, note that depending on the training cutoff date of your method you might need to wait till the next data release to see your method published.
", "rawMarkdown": "Hi Antonina,\nThank you for your interest in LAFA. Please email me or Iddo for further instructions on submitting the container to LAFA or any questions in general. \n\nRegarding public LAFA evaluation, the backend is undergoing substantial improvements following the reviewers' comments on LAFA paper to incorporate organism-specific GO databases for releases after March 2026. We expect to have the new releases within 1-2 weeks (Jun 2026 release and Sep 2026 release). LAFA is actively accepting methods; however, note that depending on the training cutoff date of your method you might need to wait till the next data release to see your method published.", "votes": null } ], "comments": [ { "id": 3462548, "author_name": "alexandervc", "author_url": "", "post_date": "05/23/2026 19:17:59", "content": "Great news !\nI wonder would it possible to create snapshots of LAFA as community Kaggle competitions ?\nOn Kaggle it is easy to create such competitions and everyone can create and participate it for free. \nThere will be not many participants - community competitions do not offer prizes. \nBut for those who are interested - it would be great to have such a benchmark.
\nYou can even make leaderboard targets publicly available - but since there will no prizes - there will no reason to cheat. (If someone would submit perfect prediction or prediction with score greater 0.7 - just ban him).
\nSo it might complement LAFA, leaving aside containerization issue, which might be simple for some models/modelers.
", "votes": null, "replies": [ { "id": 3463335, "author_name": "iddux1", "author_url": "", "post_date": "05/26/2026 16:02:09", "content": "A few reasons why Kaggle is not a good fit for running LAFA:
\nKaggle uses only one metric to assess predictions, which flattens the different GO terms, and the no-knowledge / limited knowledge / partial knowledge annotations. It also uses only precision recall, without incorporating GO semantics. LAFA uses all the classic CAFA paper metrics.
I'm not sure how continuous modeling will work on Kaggle. People would need to commit to putting the exact same models each time for continous assessments.
Thanks for the heads up. I'm currently recomputing my pipeline to align it with the LAFA evaluation window. The mechanism to publish LAFA-compatible containers is already in place on my side, so I'll be able to showcase my best datasets and models through it. Once I wrap up experimentation and stabilize the software versioning, I'll send my method over.
\nThanks!
", "votes": null, "replies": [] }, { "id": 3463267, "author_name": "danofer", "author_url": "", "post_date": "05/26/2026 13:00:25", "content": "Are the train and validation dataset splits available locally (i.e from the github)? That's always been a pain point in evaluating CAFA historically (getting train+test)
", "votes": null, "replies": [ { "id": 3463340, "author_name": "ahphan", "author_url": "", "post_date": "05/26/2026 16:15:28", "content": "Yes, at each data release (rougly 2 months), we will prepare a train dataset (proteins with experimentally validated terms) and upload it on HuggingFace. See past datasets here. It is still a prospective evaluation, i.e., the evaluation will be performed as annotation data continues to accumulate. The key difference is that the evaluation will be updated with new data and head-to-head evaluation will be available every 2 months as well.
", "votes": null, "replies": [] } ] }, { "id": 3464491, "author_name": "fernandosr85", "author_url": "", "post_date": "05/30/2026 03:29:47", "content": "Hi, I just submitted the LAFA intake form. My CAFA 6 score was modest, but I think the continuous evaluation setup of LAFA may suit my approach better. I've prepared a containerized, MIT-licensed pipeline and already filled out the form.
", "votes": null, "replies": [] }, { "id": 3466628, "author_name": "xaxipirulazo", "author_url": "", "post_date": "06/04/2026 11:37:01", "content": "Following up again, this time with a couple of logistics questions for the hosts.\nOn my entry: PROTEA-KNN shares the protein language model with the LAFA ProtT5 baseline (both build on ProtT5 embeddings), but the method itself is different. It is a KNN that uses the full reviewed UniProtKB/Swiss-Prot set as reference, with K selected per GO aspect rather than per protein, plus a different scoring scheme on top. So it is not a competitive, heavily engineered entry yet, more of an honest, baseline-adjacent starting point. I still need to document it properly, align the packaging to your conventions, and finish a round of experimentation before sending it over.\nWhat I have so far:
\nContainer: ghcr.io/frapercan/protea/knn-v1
\nDataset: XaxiPiruli/protea-lafa-knn-v227
\nLive instance: https://protea-lafa.ngrok.app/ (temporary demo for debugging purpouses, may be offline)
\nI've also submitted the intake form.\nA few questions on how you plan to run this:
\nContribution mechanism. Is the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?\nEntries per participant. Do you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?\nVersioning over time. Since LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side? This ties back to the earlier point about committing the same model for each assessment.
\nOne thing I want to flag honestly: what I have now is deliberately naive. I can already see clear ways to push the score up, for example concatenating with the baselines, adding InterProScan-derived features, or increasing the number of neighbors for under-represented proteins. Before I go down that path, I'd rather know whether richer, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility. Any recommendation on what is most useful to LAFA would help me prioritize.\nThanks again.
", "votes": null, "replies": [ { "id": 3466681, "author_name": "ahphan", "author_url": "", "post_date": "06/04/2026 13:53:43", "content": "\n\nIs the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?
\n
For now, purely the intake form plus working with us to containerize, and have your containerized method publicly accessible (I see that you have uploaded it on GitHub container registry). There are some hard requirements about the template (i.e., input and output format, container must run offline) which can be found in the container guide. Other details such as GPU requirements, size of input batch, can be communicated with us and we'll make it work on our LAFA backend.
\n\n\nDo you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?
\n
We don't limit number of methods per participant, since LAFA is designed to be a benchmarking system and not a competition. However, we do want to choose the most competitive methods to be displayed in order to capture the state-of-the-art of the protein function prediction field.
\n\n\nSince LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side?
\n
Let's say you submit \"PROTEA-KNN\" right now (under a format of a container image). \"PROTEA-KNN\" will enter LAFA starting Jun_2026 timepoint. We store those predictions generated using Jun_2026 test set, and evaluate these on multiple future time windows (e.g., Jun_2026 to Aug_2026, Jun_2026 to Oct_2026, Jun_2026 to Dec_2026, and also Aug_2026 to Oct_2026 and so on). This constitutes the head-to-head comparison across releases for PROTEA-KNN, as longer accumulation windows have a larger benchmark set and more accurate evaluation.
\nAdditionally, you can also choose to update PROTEA-KNN (e.g., re-training with newer data in Aug_2026) and submit the container image to LAFA again. The retrained model will be named \"PROTEA-KNN v1\" and will also be evaluated on future time windows similar to \"PROTEA-KNN\", giving us a new insight on how training data recency affect model performance. For example, does PROTEA-KNN (with outdated training data) perform worse than PROTEA-KNN v1 on the same time window Aug_2026 to Oct_2026?. LAFA allows us to look into it. More details can be found in our preprint
\n\n\nRicher, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility.
\n
We welcome richer, ensembled methods, as long as they are still containerized, usable and reproducible for anyone who wishes to use them (we will test them first before displaying them on the LAFA server), as this is the goal of LAFA. Larger methods might require external data storage for method artifacts, weights, etc, so method developers are required to host the external data storage in a persistent data volume, ensuring usability.
", "votes": null, "replies": [] } ] }, { "id": 3529923, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "09/29/2026 07:35:04", "content": "Hi An and Iddo,
\nI submitted the intake form about a week ago and have the container ready and publicly available on Docker Hub, following the LAFA container guide. I mainly wanted to make sure my form was received and that I haven't missed any additional step on my side.
\nI also noticed that the public LAFA evaluation windows do not seem to have been updated recently. Is LAFA currently accepting and integrating new methods, and are new evaluation windows still planned?
\nThanks!\nAntonina
", "votes": null, "replies": [ { "id": 3530088, "author_name": "ahphan", "author_url": "", "post_date": "09/29/2026 14:08:48", "content": "Hi Antonina,\nThank you for your interest in LAFA. Please email me or Iddo for further instructions on submitting the container to LAFA or any questions in general.
\nRegarding public LAFA evaluation, the backend is undergoing substantial improvements following the reviewers' comments on LAFA paper to incorporate organism-specific GO databases for releases after March 2026. We expect to have the new releases within 1-2 weeks (Jun 2026 release and Sep 2026 release). LAFA is actively accepting methods; however, note that depending on the training cutoff date of your method you might need to wait till the next data release to see your method published.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3462008": "As the CAFA 6 challenge is winding down, I would like to thank you for all your hard work. I would also like to inform you of an exciting opportunity to showcase your work further. \n\nLAFA is a [web server](https://functionbench.net/) we recently developed that constantly assesses protein function prediction models. The models are containerized and run on periodically updated benchmarks of protein function annotations compiled from several sources.\n\nIf you wish to add your model to those already on the LAFA server, please [fill out this form](https://forms.gle/x3TZioBXUE5EkYMq6) and someone from our team will get back to you. This is an excellent opportunity to showcase your work.\n\nThere is no obligation on your part to provide your model, and you can always request that it be removed. All we ask is that you license your method under an open source license and work with us to containerize it for LAFA. \n\n\nFeel free to contact us with any questions you may have. \n\n[The LAFA server](https://functionbench.net/)\n\n[A preprint describing LAFA](http://arxiv.org/abs/2604.20782)\n\n[How to containerize](https://github.com/anphan0828/LAFA_container_guide)", "3462548": "Great news !\nI wonder would it possible to create snapshots of LAFA as community Kaggle competitions ?\nOn Kaggle it is easy to create such competitions and everyone can create and participate it for free. \nThere will be not many participants - community competitions do not offer prizes. \nBut for those who are interested - it would be great to have such a benchmark.\n\nYou can even make leaderboard targets publicly available - but since there will no prizes - there will no reason to cheat. (If someone would submit perfect prediction or prediction with score greater 0.7 - just ban him). \n\nSo it might complement LAFA, leaving aside containerization issue, which might be simple for some models/modelers.", "3463248": "Thanks for the heads up. I'm currently recomputing my pipeline to align it with the LAFA evaluation window. The mechanism to publish LAFA-compatible containers is already in place on my side, so I'll be able to showcase my best datasets and models through it. Once I wrap up experimentation and stabilize the software versioning, I'll send my method over. \n\nThanks!", "3463267": "Are the train and validation dataset splits available locally (i.e from the github)? That's always been a pain point in evaluating CAFA historically (getting train+test)", "3463335": "A few reasons why Kaggle is not a good fit for running LAFA:\n\n1. Kaggle uses only one metric to assess predictions, which flattens the different GO terms, and the no-knowledge / limited knowledge / partial knowledge annotations. It also uses only precision recall, without incorporating GO semantics. LAFA uses all the classic CAFA paper metrics.\n\n2. I'm not sure how continuous modeling will work on Kaggle. People would need to commit to putting the exact same models each time for continous assessments.", "3463340": "Yes, at each data release (rougly 2 months), we will prepare a train dataset (proteins with experimentally validated terms) and upload it on HuggingFace. See past datasets [here](https://huggingface.co/datasets/anphan0828/lafa/tree/main). It is still a prospective evaluation, i.e., the evaluation will be performed as annotation data continues to accumulate. The key difference is that the evaluation will be updated with new data and head-to-head evaluation will be available every 2 months as well.", "3464491": "Hi, I just submitted the LAFA intake form. My CAFA 6 score was modest, but I think the continuous evaluation setup of LAFA may suit my approach better. I've prepared a containerized, MIT-licensed pipeline and already filled out the form.", "3466628": "Following up again, this time with a couple of logistics questions for the hosts.\nOn my entry: PROTEA-KNN shares the protein language model with the LAFA ProtT5 baseline (both build on ProtT5 embeddings), but the method itself is different. It is a KNN that uses the full reviewed UniProtKB/Swiss-Prot set as reference, with K selected per GO aspect rather than per protein, plus a different scoring scheme on top. So it is not a competitive, heavily engineered entry yet, more of an honest, baseline-adjacent starting point. I still need to document it properly, align the packaging to your conventions, and finish a round of experimentation before sending it over.\nWhat I have so far:\n\nContainer: ghcr.io/frapercan/protea/knn-v1\n\nDataset: XaxiPiruli/protea-lafa-knn-v227\n\nLive instance: https://protea-lafa.ngrok.app/ (temporary demo for debugging purpouses, may be offline)\n\nI've also submitted the intake form.\nA few questions on how you plan to run this:\n\nContribution mechanism. Is the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?\nEntries per participant. Do you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?\nVersioning over time. Since LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side? This ties back to the earlier point about committing the same model for each assessment.\n\nOne thing I want to flag honestly: what I have now is deliberately naive. I can already see clear ways to push the score up, for example concatenating with the baselines, adding InterProScan-derived features, or increasing the number of neighbors for under-represented proteins. Before I go down that path, I'd rather know whether richer, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility. Any recommendation on what is most useful to LAFA would help me prioritize.\nThanks again.", "3466681": ">Is the intended flow a Pull Request to a LAFA repo, or purely the intake form plus working with you to containerize? Is there a target repo structure or template you'd like methods to follow?\n\nFor now, purely the intake form plus working with us to containerize, and have your containerized method publicly accessible (I see that you have uploaded it on GitHub container registry). There are some hard requirements about the template (i.e., input and output format, container must run offline) which can be found in the [container guide](https://github.com/anphan0828/LAFA_container_guide). Other details such as GPU requirements, size of input batch, can be communicated with us and we'll make it work on our LAFA backend.\n\n>Do you want a single method per participant, or are multiple variants acceptable (for example, a clean baseline-style method and a richer one)?\n\nWe don't limit number of methods per participant, since LAFA is designed to be a benchmarking system and not a competition. However, we do want to choose the most competitive methods to be displayed in order to capture the state-of-the-art of the protein function prediction field.\n\n>Since LAFA is a continuous, prospective assessment, how do you want method evolution handled? Do you expect a frozen container per entry for head-to-head comparability across releases, or can a method be updated between windows with versioning tracked on your side?\n\nLet's say you submit \"PROTEA-KNN\" right now (under a format of a container image). \"PROTEA-KNN\" will enter LAFA starting Jun_2026 timepoint. We store those predictions generated using Jun_2026 test set, and evaluate these on multiple future time windows (e.g., Jun_2026 to Aug_2026, Jun_2026 to Oct_2026, Jun_2026 to Dec_2026, and also Aug_2026 to Oct_2026 and so on). This constitutes the head-to-head comparison across releases for PROTEA-KNN, as longer accumulation windows have a larger benchmark set and more accurate evaluation. \n\nAdditionally, you can also choose to update PROTEA-KNN (e.g., re-training with newer data in Aug_2026) and submit the container image to LAFA again. The retrained model will be named \"**PROTEA-KNN v1**\" and will also be evaluated on future time windows similar to \"PROTEA-KNN\", giving us a new insight on how training data recency affect model performance. For example, does PROTEA-KNN (with outdated training data) perform worse than PROTEA-KNN v1 on the same time window Aug_2026 to Oct_2026?. LAFA allows us to look into it. More details can be found in our [preprint](https://arxiv.org/pdf/2604.20782)\n\n>Richer, ensembled methods are in scope and welcome, or whether you'd prefer cleaner single-signal methods for interpretability and reproducibility. \n\nWe welcome richer, ensembled methods, as long as they are still containerized, usable and reproducible for anyone who wishes to use them (we will test them first before displaying them on the LAFA server), as this is the goal of LAFA. Larger methods might require external data storage for method artifacts, weights, etc, so method developers are required to host the external data storage in a persistent data volume, ensuring usability.", "3529923": "Hi An and Iddo,\n\nI submitted the intake form about a week ago and have the container ready and publicly available on Docker Hub, following the LAFA container guide. I mainly wanted to make sure my form was received and that I haven't missed any additional step on my side.\n\nI also noticed that the public LAFA evaluation windows do not seem to have been updated recently. Is LAFA currently accepting and integrating new methods, and are new evaluation windows still planned?\n\nThanks!\nAntonina", "3530088": "Hi Antonina,\nThank you for your interest in LAFA. Please email me or Iddo for further instructions on submitting the container to LAFA or any questions in general. \n\nRegarding public LAFA evaluation, the backend is undergoing substantial improvements following the reviewers' comments on LAFA paper to incorporate organism-specific GO databases for releases after March 2026. We expect to have the new releases within 1-2 weeks (Jun 2026 release and Sep 2026 release). LAFA is actively accepting methods; however, note that depending on the training cutoff date of your method you might need to wait till the next data release to see your method published." }, "source": "meta" }