{
  "id": 411201,
  "title": "Running the evaluator",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/411201",
  "author_name": "",
  "post_date": "2023-05-18T05:47:19.453264400Z",
  "votes": 5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I wanted to check if there is anyone else trying to run the evaluator from the github (<a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a>) to estimate model accuracy based simply on a train/test split of the given training data. </p>\n<p>What are the settings you use to obtain accurate results that reflect the evaluation criteria of the leaderboard? Specifically -norm, -prop, and whether IA should be included. </p>",
  "messages": [
    {
      "id": "2264060",
      "postDate": "05/18/2023 05:47:19",
      "content": "<p>I wanted to check if there is anyone else trying to run the evaluator from the github (<a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a>) to estimate model accuracy based simply on a train/test split of the given training data. </p>\n<p>What are the settings you use to obtain accurate results that reflect the evaluation criteria of the leaderboard? Specifically -norm, -prop, and whether IA should be included. </p>",
      "rawMarkdown": "I wanted to check if there is anyone else trying to run the evaluator from the github (https://github.com/BioComputingUP/CAFA-evaluator) to estimate model accuracy based simply on a train/test split of the given training data. \n\nWhat are the settings you use to obtain accurate results that reflect the evaluation criteria of the leaderboard? Specifically -norm, -prop, and whether IA should be included.",
      "votes": null
    },
    {
      "id": "2270603",
      "postDate": "05/23/2023 09:11:44",
      "content": "<p>Hi there,</p>\n<p>To run the program, please refer to the \"CAFA5 Challenge\" section at the top of the <a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">README</a> and use the described settings. After running the program, you should examine the \"evaluation_best_wf.tsv\" result file and calculate the average (arithmetic mean) of the weighted F-score (\"wf\" column\") over the three ontologies. For your convenience, you can use the IA and ontology files that are provided on Kaggle.</p>\n<p>Best regards,<br>\nDamiano</p>",
      "rawMarkdown": "Hi there,\n\nTo run the program, please refer to the \"CAFA5 Challenge\" section at the top of the [README](https://github.com/BioComputingUP/CAFA-evaluator) and use the described settings. After running the program, you should examine the \"evaluation_best_wf.tsv\" result file and calculate the average (arithmetic mean) of the weighted F-score (\"wf\" column\") over the three ontologies. For your convenience, you can use the IA and ontology files that are provided on Kaggle.\n\nBest regards,\nDamiano",
      "votes": null
    },
    {
      "id": "2270786",
      "postDate": "05/23/2023 12:24:08",
      "content": "<p><a href=\"https://www.kaggle.com/code/sergeifironov/validate-ridge\" target=\"_blank\">https://www.kaggle.com/code/sergeifironov/validate-ridge</a></p>",
      "rawMarkdown": "https://www.kaggle.com/code/sergeifironov/validate-ridge",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2270603,
      "author_name": "damianopiovesan",
      "author_url": "",
      "post_date": "05/23/2023 09:11:44",
      "content": "<p>Hi there,</p>\n<p>To run the program, please refer to the \"CAFA5 Challenge\" section at the top of the <a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">README</a> and use the described settings. After running the program, you should examine the \"evaluation_best_wf.tsv\" result file and calculate the average (arithmetic mean) of the weighted F-score (\"wf\" column\") over the three ontologies. For your convenience, you can use the IA and ontology files that are provided on Kaggle.</p>\n<p>Best regards,<br>\nDamiano</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2270786,
      "author_name": "alexandervc",
      "author_url": "",
      "post_date": "05/23/2023 12:24:08",
      "content": "<p><a href=\"https://www.kaggle.com/code/sergeifironov/validate-ridge\" target=\"_blank\">https://www.kaggle.com/code/sergeifironov/validate-ridge</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2264060": "I wanted to check if there is anyone else trying to run the evaluator from the github (https://github.com/BioComputingUP/CAFA-evaluator) to estimate model accuracy based simply on a train/test split of the given training data. \n\nWhat are the settings you use to obtain accurate results that reflect the evaluation criteria of the leaderboard? Specifically -norm, -prop, and whether IA should be included.",
    "2270603": "Hi there,\n\nTo run the program, please refer to the \"CAFA5 Challenge\" section at the top of the [README](https://github.com/BioComputingUP/CAFA-evaluator) and use the described settings. After running the program, you should examine the \"evaluation_best_wf.tsv\" result file and calculate the average (arithmetic mean) of the weighted F-score (\"wf\" column\") over the three ontologies. For your convenience, you can use the IA and ontology files that are provided on Kaggle.\n\nBest regards,\nDamiano",
    "2270786": "https://www.kaggle.com/code/sergeifironov/validate-ridge"
  },
  "source": "meta"
}