{
  "id": 419127,
  "title": "The metric is calculated with: prop_mode='fill' or prop_mode='max' ?",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/419127",
  "author_name": "",
  "post_date": "2023-06-24T09:28:41.263082300Z",
  "votes": 12,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Evaluation for CAFA: <br>\n<a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a> </p>\n<p>See also Kaggle notebook by Sergei Fironov: <a href=\"https://www.kaggle.com/code/sergeifironov/validate-ridge#kln-462\" target=\"_blank\">https://www.kaggle.com/code/sergeifironov/validate-ridge#kln-462</a></p>\n<p>There are two modes for the \"propagation\": one is \"fill\", another is \"max\", see the following line of code: </p>\n<pre><code>preds = pred_parser(preds, ontologies, gt, prop_mode=, max_terms=)\n</code></pre>\n<p><strong>Question:</strong>  In the current competition the default mode \"fill\" is used , is it true ? </p>\n<p>And max_terms=500 , is it correct ? </p>",
  "messages": [
    {
      "id": "2315636",
      "postDate": "06/24/2023 09:28:41",
      "content": "<p>Evaluation for CAFA: <br>\n<a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a> </p>\n<p>See also Kaggle notebook by Sergei Fironov: <a href=\"https://www.kaggle.com/code/sergeifironov/validate-ridge#kln-462\" target=\"_blank\">https://www.kaggle.com/code/sergeifironov/validate-ridge#kln-462</a></p>\n<p>There are two modes for the \"propagation\": one is \"fill\", another is \"max\", see the following line of code: </p>\n<pre><code>preds = pred_parser(preds, ontologies, gt, prop_mode=, max_terms=)\n</code></pre>\n<p><strong>Question:</strong>  In the current competition the default mode \"fill\" is used , is it true ? </p>\n<p>And max_terms=500 , is it correct ? </p>",
      "rawMarkdown": "Evaluation for CAFA: \nhttps://github.com/BioComputingUP/CAFA-evaluator \n\nSee also Kaggle notebook by Sergei Fironov: https://www.kaggle.com/code/sergeifironov/validate-ridge#kln-462\n\nThere are two modes for the \"propagation\": one is \"fill\", another is \"max\", see the following line of code: \n\n```python\npreds = pred_parser(preds, ontologies, gt, prop_mode='fill', max_terms=500)\n\n```\n\n**Question:**  In the current competition the default mode \"fill\" is used , is it true ? \n\nAnd max_terms=500 , is it correct ?",
      "votes": null
    },
    {
      "id": "2318126",
      "postDate": "06/26/2023 07:48:08",
      "content": "<p>Hi Alexander,</p>\n<p>I confirm the \"fill\" and \"500\" parameters match those used in Kaggle. I have updated recently the README file in the <a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">CAFA-evaluator</a> repository with this information.</p>\n<p>Best regards</p>",
      "rawMarkdown": "Hi Alexander,\n\nI confirm the \"fill\" and \"500\" parameters match those used in Kaggle. I have updated recently the README file in the [CAFA-evaluator](https://github.com/BioComputingUP/CAFA-evaluator) repository with this information.\n\nBest regards",
      "votes": null
    },
    {
      "id": "2318426",
      "postDate": "06/26/2023 11:09:06",
      "content": "<p>Thanks a lot !</p>",
      "rawMarkdown": "Thanks a lot !",
      "votes": null
    },
    {
      "id": "2320458",
      "postDate": "06/27/2023 19:43:10",
      "content": "<p>This thread is great. Thanks a lot for sharing this insight. I had not noticed this and it has brought my offline eval closer to my public test set scores.</p>\n<p>Kudos!</p>",
      "rawMarkdown": "This thread is great. Thanks a lot for sharing this insight. I had not noticed this and it has brought my offline eval closer to my public test set scores.\n\nKudos!",
      "votes": null
    },
    {
      "id": "2354903",
      "postDate": "07/22/2023 23:06:04",
      "content": "<p>Hi Damiano,</p>\n<p>If <code>max_terms=500</code>, does it mean that the <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation\" target=\"_blank\">Evaluation</a> section should read</p>\n<blockquote>\n  <p>…to limit prediction file sizes, one target cannot be associated with more than 1,500 terms (maximum of 500 terms in each of MF, BP, and CC subontologies).</p>\n</blockquote>\n<p>From <a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a></p>\n<blockquote>\n  <p>The -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached. </p>\n</blockquote>",
      "rawMarkdown": "Hi Damiano,\n\nIf `max_terms=500`, does it mean that the [Evaluation](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation) section should read\n\n>...to limit prediction file sizes, one target cannot be associated with more than 1,500 terms (maximum of 500 terms in each of MF, BP, and CC subontologies).\n\nFrom https://github.com/BioComputingUP/CAFA-evaluator\n\n>The -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2318126,
      "author_name": "damianopiovesan",
      "author_url": "",
      "post_date": "06/26/2023 07:48:08",
      "content": "<p>Hi Alexander,</p>\n<p>I confirm the \"fill\" and \"500\" parameters match those used in Kaggle. I have updated recently the README file in the <a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">CAFA-evaluator</a> repository with this information.</p>\n<p>Best regards</p>",
      "votes": null,
      "replies": [
        {
          "id": 2318426,
          "author_name": "alexandervc",
          "author_url": "",
          "post_date": "06/26/2023 11:09:06",
          "content": "<p>Thanks a lot !</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2354903,
          "author_name": "dustinthewind",
          "author_url": "",
          "post_date": "07/22/2023 23:06:04",
          "content": "<p>Hi Damiano,</p>\n<p>If <code>max_terms=500</code>, does it mean that the <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation\" target=\"_blank\">Evaluation</a> section should read</p>\n<blockquote>\n  <p>…to limit prediction file sizes, one target cannot be associated with more than 1,500 terms (maximum of 500 terms in each of MF, BP, and CC subontologies).</p>\n</blockquote>\n<p>From <a href=\"https://github.com/BioComputingUP/CAFA-evaluator\" target=\"_blank\">https://github.com/BioComputingUP/CAFA-evaluator</a></p>\n<blockquote>\n  <p>The -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached. </p>\n</blockquote>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2320458,
      "author_name": "maxwbuckley",
      "author_url": "",
      "post_date": "06/27/2023 19:43:10",
      "content": "<p>This thread is great. Thanks a lot for sharing this insight. I had not noticed this and it has brought my offline eval closer to my public test set scores.</p>\n<p>Kudos!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2315636": "Evaluation for CAFA: \nhttps://github.com/BioComputingUP/CAFA-evaluator \n\nSee also Kaggle notebook by Sergei Fironov: https://www.kaggle.com/code/sergeifironov/validate-ridge#kln-462\n\nThere are two modes for the \"propagation\": one is \"fill\", another is \"max\", see the following line of code: \n\n```python\npreds = pred_parser(preds, ontologies, gt, prop_mode='fill', max_terms=500)\n\n```\n\n**Question:**  In the current competition the default mode \"fill\" is used , is it true ? \n\nAnd max_terms=500 , is it correct ?",
    "2318126": "Hi Alexander,\n\nI confirm the \"fill\" and \"500\" parameters match those used in Kaggle. I have updated recently the README file in the [CAFA-evaluator](https://github.com/BioComputingUP/CAFA-evaluator) repository with this information.\n\nBest regards",
    "2318426": "Thanks a lot !",
    "2320458": "This thread is great. Thanks a lot for sharing this insight. I had not noticed this and it has brought my offline eval closer to my public test set scores.\n\nKudos!",
    "2354903": "Hi Damiano,\n\nIf `max_terms=500`, does it mean that the [Evaluation](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation) section should read\n\n>...to limit prediction file sizes, one target cannot be associated with more than 1,500 terms (maximum of 500 terms in each of MF, BP, and CC subontologies).\n\nFrom https://github.com/BioComputingUP/CAFA-evaluator\n\n>The -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached."
  },
  "source": "meta"
}