{
  "id": 614145,
  "title": "How to evaluate according to the officer way ",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/614145",
  "author_name": "common-people",
  "post_date": "2025-11-01T13:05:45.024000",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Did the CAFA5 have the same problem. I tried to use the ia to weighted the f1max score but the offline score is much higher than the online. Are there some one who participated to the CAFA5?\nAnd I saw officer has provided a code named CAFA4 last edition. I am wonder if this is the right way to set the correct evaluation method? Help!</p>",
  "messages": [
    {
      "id": 3309877,
      "postDate": "2025-11-01T16:31:59.750Z",
      "content": "<p>Here are some steps that might be helpful.</p>\n<ol>\n<li><p>Clone the GitHub repository:<br>\n<code>$ git clone https://github.com/claradepaolis/CAFA-evaluator-PK.git</code></p></li>\n<li><p>Then install via:<br>\n<code>$ pip install CAFA-evaluator-PK/</code></p></li>\n<li><p>Now you should be able to run:<br>\n<code>$ cafaeval</code></p></li>\n<li><p>For this competition, I would run it like:<br>\n<code>$ cafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file</code><br>\nwhere:</p>\n<ul>\n<li><code>accretion_file</code> is <code>IA.tsv</code> in this competition;</li>\n<li><code>ontology_file</code> is <code>go-basic.obo</code> in this competition;</li>\n<li><code>prediction_folder</code> is a folder that you must create, and place your submission file(s) inside (in the same format as <code>sample_submission.tsv</code>);</li>\n<li><code>ground_truth_file</code> is a file that contains the true GO annotations (similar to <code>train_terms.csv</code> but I guess without the header and without the third column).</li></ul></li>\n<li><p>Be prepared for this to take a lot of memory, and take a while to run. Then look for the results in a <code>results</code> folder.</p></li>\n</ol>",
      "rawMarkdown": "Here are some steps that might be helpful.\n\n1. Clone the GitHub repository:  \n`$ git clone https://github.com/claradepaolis/CAFA-evaluator-PK.git`\n\n2. Then install via:  \n`$ pip install CAFA-evaluator-PK/`\n\n3. Now you should be able to run:  \n`$ cafaeval`\n\n4. For this competition, I would run it like:  \n`$ cafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file`  \nwhere:\n  - `accretion_file` is `IA.tsv` in this competition;\n  - `ontology_file` is `go-basic.obo` in this competition;\n  - `prediction_folder` is a folder that you must create, and place your submission file(s) inside (in the same format as `sample_submission.tsv`);\n  - `ground_truth_file` is a file that contains the true GO annotations (similar to `train_terms.csv` but I guess without the header and without the third column).\n\n5. Be prepared for this to take a lot of memory, and take a while to run. Then look for the results in a `results` folder.",
      "votes": 7,
      "replies": [
        {
          "id": 3309880,
          "postDate": "2025-11-01T16:38:55.070Z",
          "content": "<p>Thank you， this is really helpful</p>",
          "rawMarkdown": "Thank you， this is really helpful"
        },
        {
          "id": 3310680,
          "postDate": "2025-11-03T13:26:11.040Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 3309806,
      "postDate": "2025-11-01T13:05:45.023Z",
      "content": "<p>Did the CAFA5 have the same problem. I tried to use the ia to weighted the f1max score but the offline score is much higher than the online. Are there some one who participated to the CAFA5?\nAnd I saw officer has provided a code named CAFA4 last edition. I am wonder if this is the right way to set the correct evaluation method? Help!</p>",
      "rawMarkdown": "Did the CAFA5 have the same problem. I tried to use the ia to weighted the f1max score but the offline score is much higher than the online. Are there some one who participated to the CAFA5?\nAnd I saw officer has provided a code named CAFA4 last edition. I am wonder if this is the right way to set the correct evaluation method? Help!",
      "votes": 3
    }
  ],
  "comments": [
    {
      "id": 3309877,
      "author_name": "Diogo R. Ferreira",
      "author_url": "",
      "post_date": "2025-11-01T16:31:59.750000",
      "content": "<p>Here are some steps that might be helpful.</p>\n<ol>\n<li><p>Clone the GitHub repository:<br>\n<code>$ git clone https://github.com/claradepaolis/CAFA-evaluator-PK.git</code></p></li>\n<li><p>Then install via:<br>\n<code>$ pip install CAFA-evaluator-PK/</code></p></li>\n<li><p>Now you should be able to run:<br>\n<code>$ cafaeval</code></p></li>\n<li><p>For this competition, I would run it like:<br>\n<code>$ cafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file</code><br>\nwhere:</p>\n<ul>\n<li><code>accretion_file</code> is <code>IA.tsv</code> in this competition;</li>\n<li><code>ontology_file</code> is <code>go-basic.obo</code> in this competition;</li>\n<li><code>prediction_folder</code> is a folder that you must create, and place your submission file(s) inside (in the same format as <code>sample_submission.tsv</code>);</li>\n<li><code>ground_truth_file</code> is a file that contains the true GO annotations (similar to <code>train_terms.csv</code> but I guess without the header and without the third column).</li></ul></li>\n<li><p>Be prepared for this to take a lot of memory, and take a while to run. Then look for the results in a <code>results</code> folder.</p></li>\n</ol>",
      "votes": 7,
      "replies": [
        {
          "id": 3309880,
          "author_name": "common-people",
          "author_url": "",
          "post_date": "2025-11-01T16:38:55.070000",
          "content": "<p>Thank you， this is really helpful</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 3310680,
          "author_name": "",
          "author_url": "",
          "post_date": "2025-11-03T13:26:11.040000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3309877": "Here are some steps that might be helpful.\n\n1. Clone the GitHub repository:  \n`$ git clone https://github.com/claradepaolis/CAFA-evaluator-PK.git`\n\n2. Then install via:  \n`$ pip install CAFA-evaluator-PK/`\n\n3. Now you should be able to run:  \n`$ cafaeval`\n\n4. For this competition, I would run it like:  \n`$ cafaeval -ia accretion_file ontology_file prediction_folder ground_truth_file`  \nwhere:\n  - `accretion_file` is `IA.tsv` in this competition;\n  - `ontology_file` is `go-basic.obo` in this competition;\n  - `prediction_folder` is a folder that you must create, and place your submission file(s) inside (in the same format as `sample_submission.tsv`);\n  - `ground_truth_file` is a file that contains the true GO annotations (similar to `train_terms.csv` but I guess without the header and without the third column).\n\n5. Be prepared for this to take a lot of memory, and take a while to run. Then look for the results in a `results` folder.",
    "3309806": "Did the CAFA5 have the same problem. I tried to use the ia to weighted the f1max score but the offline score is much higher than the online. Are there some one who participated to the CAFA5?\nAnd I saw officer has provided a code named CAFA4 last edition. I am wonder if this is the right way to set the correct evaluation method? Help!"
  }
}