{
  "id": 612097,
  "title": "CAFA-evaluator Python tool in Gene Ontology (the official CAFA evaluation software)",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/612097",
  "author_name": "",
  "post_date": "2025-10-16T18:45:53.888261900Z",
  "votes": 21,
  "comment_count": 2,
  "views": 0,
  "content": "<h2>CAFA-evaluator: a Python tool for benchmarking ontological classification methods</h2>\n<p><strong>CITATION:</strong><br>\nCAFA-evaluator: A Python Tool for Benchmarking Ontological Classification Methods<br>\nD Piovesan, D Zago, P Joshi, MC De Paolis Kaluza, M Mehdiabadi, R Ramola, AM Monzon, W Reade, I Friedberg, P Radivojac, SCE Tosatto<br>\nBioinformatics Advances (2024) - DOI: 10.1093/bioadv/vbae043</p>\n<blockquote>\n  <p>Availability and implementation: <a href=\"https://pypi.org/project/cafaeval\" target=\"_blank\">https://pypi.org/project/cafaeval</a></p>\n</blockquote>\n<p>\"The authors presented CAFA-evaluator, a powerful Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"</p>\n<p>\"The program requirements include a small number of standard Python libraries, making CAFA-evaluator easy to maintain. The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. Owing to its reliability and accuracy, the organizers have selected CAFA-evaluator as <strong>the official CAFA evaluation software</strong>.\"</p>\n<p><a href=\"https://pubmed.ncbi.nlm.nih.gov/38545087/\" target=\"_blank\">https://pubmed.ncbi.nlm.nih.gov/38545087/</a></p>\n<h2>CAFA-evaluator - Python Program</h2>\n<p>\"CAFA-evaluator is a Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"</p>\n<p>\"The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. The package also contains a Jupyter Notebook to generate precision-recall and remaining uncertainty–misinformation curves. CAFA-evaluator implementation was inspired by the original Matlab code used in CAFA2 assessment and available at https://github.com/yuxjiang/CAFA2.\"</p>\n<p>Visit the CAFA-evaluator wiki for more information about the algorithm: <a href=\"https://github.com/BioComputingUP/CAFA-evaluator/wiki\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator/wiki</a></p>\n<p><a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a></p>\n<h2>The most updated CAFA-evaluator that is used in CAFA6</h2>\n<p><a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK\" target=\"_blank\">https://github.com/claradepaolis/CAFA-evaluator-PK</a></p>",
  "messages": [
    {
      "id": "3302908",
      "postDate": "10/16/2025 18:45:53",
      "content": "<h2>CAFA-evaluator: a Python tool for benchmarking ontological classification methods</h2>\n<p><strong>CITATION:</strong><br>\nCAFA-evaluator: A Python Tool for Benchmarking Ontological Classification Methods<br>\nD Piovesan, D Zago, P Joshi, MC De Paolis Kaluza, M Mehdiabadi, R Ramola, AM Monzon, W Reade, I Friedberg, P Radivojac, SCE Tosatto<br>\nBioinformatics Advances (2024) - DOI: 10.1093/bioadv/vbae043</p>\n<blockquote>\n  <p>Availability and implementation: <a href=\"https://pypi.org/project/cafaeval\" target=\"_blank\">https://pypi.org/project/cafaeval</a></p>\n</blockquote>\n<p>\"The authors presented CAFA-evaluator, a powerful Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"</p>\n<p>\"The program requirements include a small number of standard Python libraries, making CAFA-evaluator easy to maintain. The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. Owing to its reliability and accuracy, the organizers have selected CAFA-evaluator as <strong>the official CAFA evaluation software</strong>.\"</p>\n<p><a href=\"https://pubmed.ncbi.nlm.nih.gov/38545087/\" target=\"_blank\">https://pubmed.ncbi.nlm.nih.gov/38545087/</a></p>\n<h2>CAFA-evaluator - Python Program</h2>\n<p>\"CAFA-evaluator is a Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"</p>\n<p>\"The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. The package also contains a Jupyter Notebook to generate precision-recall and remaining uncertainty–misinformation curves. CAFA-evaluator implementation was inspired by the original Matlab code used in CAFA2 assessment and available at https://github.com/yuxjiang/CAFA2.\"</p>\n<p>Visit the CAFA-evaluator wiki for more information about the algorithm: <a href=\"https://github.com/BioComputingUP/CAFA-evaluator/wiki\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator/wiki</a></p>\n<p><a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a></p>\n<h2>The most updated CAFA-evaluator that is used in CAFA6</h2>\n<p><a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK\" target=\"_blank\">https://github.com/claradepaolis/CAFA-evaluator-PK</a></p>",
      "rawMarkdown": "## CAFA-evaluator: a Python tool for benchmarking ontological classification methods\n\n**CITATION:**\nCAFA-evaluator: A Python Tool for Benchmarking Ontological Classification Methods\nD Piovesan, D Zago, P Joshi, MC De Paolis Kaluza, M Mehdiabadi, R Ramola, AM Monzon, W Reade, I Friedberg, P Radivojac, SCE Tosatto\nBioinformatics Advances (2024) - DOI: 10.1093/bioadv/vbae043\n\n>Availability and implementation: https://pypi.org/project/cafaeval\n\n\"The authors presented CAFA-evaluator, a powerful Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"\n\n\"The program requirements include a small number of standard Python libraries, making CAFA-evaluator easy to maintain. The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. Owing to its reliability and accuracy, the organizers have selected CAFA-evaluator as **the official CAFA evaluation software**.\"\n\nhttps://pubmed.ncbi.nlm.nih.gov/38545087/\n\n## CAFA-evaluator - Python Program\n\n\"CAFA-evaluator is a Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"\n\n\"The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. The package also contains a Jupyter Notebook to generate precision-recall and remaining uncertainty–misinformation curves. CAFA-evaluator implementation was inspired by the original Matlab code used in CAFA2 assessment and available at https://github.com/yuxjiang/CAFA2.\"\n\nVisit the CAFA-evaluator wiki for more information about the algorithm: https://github.com/BioComputingUP/CAFA-evaluator/wiki\n\nhttps://github.com/BioComputingUP/CAFA-evaluator\n\n## The most updated CAFA-evaluator that is used in CAFA6\n\nhttps://github.com/claradepaolis/CAFA-evaluator-PK",
      "votes": null
    },
    {
      "id": "3302948",
      "postDate": "10/16/2025 20:31:01",
      "content": "<p>Please note that this GitHub repository is the most updated CAFA-evaluator that is used in CAFA6: <br>\n<a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK\" target=\"_blank\">https://github.com/claradepaolis/CAFA-evaluator-PK</a>. This repository includes evaluation for partial-knowledge proteins.</p>",
      "rawMarkdown": "Please note that this GitHub repository is the most updated CAFA-evaluator that is used in CAFA6: \n[https://github.com/claradepaolis/CAFA-evaluator-PK](https://github.com/claradepaolis/CAFA-evaluator-PK). This repository includes evaluation for partial-knowledge proteins.",
      "votes": null
    },
    {
      "id": "3302972",
      "postDate": "10/16/2025 22:19:39",
      "content": "<p>Thank you Ahphan. Noted.  I included that link on the discussion topic.</p>",
      "rawMarkdown": "Thank you Ahphan. Noted.  I included that link on the discussion topic.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3302948,
      "author_name": "ahphan",
      "author_url": "",
      "post_date": "10/16/2025 20:31:01",
      "content": "<p>Please note that this GitHub repository is the most updated CAFA-evaluator that is used in CAFA6: <br>\n<a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK\" target=\"_blank\">https://github.com/claradepaolis/CAFA-evaluator-PK</a>. This repository includes evaluation for partial-knowledge proteins.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3302972,
          "author_name": "mpwolke",
          "author_url": "",
          "post_date": "10/16/2025 22:19:39",
          "content": "<p>Thank you Ahphan. Noted.  I included that link on the discussion topic.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3302908": "## CAFA-evaluator: a Python tool for benchmarking ontological classification methods\n\n**CITATION:**\nCAFA-evaluator: A Python Tool for Benchmarking Ontological Classification Methods\nD Piovesan, D Zago, P Joshi, MC De Paolis Kaluza, M Mehdiabadi, R Ramola, AM Monzon, W Reade, I Friedberg, P Radivojac, SCE Tosatto\nBioinformatics Advances (2024) - DOI: 10.1093/bioadv/vbae043\n\n>Availability and implementation: https://pypi.org/project/cafaeval\n\n\"The authors presented CAFA-evaluator, a powerful Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"\n\n\"The program requirements include a small number of standard Python libraries, making CAFA-evaluator easy to maintain. The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. Owing to its reliability and accuracy, the organizers have selected CAFA-evaluator as **the official CAFA evaluation software**.\"\n\nhttps://pubmed.ncbi.nlm.nih.gov/38545087/\n\n## CAFA-evaluator - Python Program\n\n\"CAFA-evaluator is a Python program designed to evaluate the performance of prediction methods on targets with hierarchical concept dependencies. It generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn from a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\"\n\n\"The code replicates the Critical Assessment of protein Function Annotation (CAFA) benchmarking, which evaluates predictions of the consistent subgraphs in Gene Ontology. The package also contains a Jupyter Notebook to generate precision-recall and remaining uncertainty–misinformation curves. CAFA-evaluator implementation was inspired by the original Matlab code used in CAFA2 assessment and available at https://github.com/yuxjiang/CAFA2.\"\n\nVisit the CAFA-evaluator wiki for more information about the algorithm: https://github.com/BioComputingUP/CAFA-evaluator/wiki\n\nhttps://github.com/BioComputingUP/CAFA-evaluator\n\n## The most updated CAFA-evaluator that is used in CAFA6\n\nhttps://github.com/claradepaolis/CAFA-evaluator-PK",
    "3302948": "Please note that this GitHub repository is the most updated CAFA-evaluator that is used in CAFA6: \n[https://github.com/claradepaolis/CAFA-evaluator-PK](https://github.com/claradepaolis/CAFA-evaluator-PK). This repository includes evaluation for partial-knowledge proteins.",
    "3302972": "Thank you Ahphan. Noted.  I included that link on the discussion topic."
  },
  "source": "meta"
}