{
  "id": 412853,
  "title": "Leaderboard public score calculation",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/412853",
  "author_name": "",
  "post_date": "2023-05-25T14:07:56.752133200Z",
  "votes": 4,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Dear CAFA organizer,<br>\nI have two questions on the CAFA5 kaggle leaderboard.</p>\n<ol>\n<li><p>Approximately how many test proteins are used to calculate the public score? Although the organizers urgent caution against it, many participants are probably trying to optimize their method for better public score in the leaderboard. If there are ~1000 proteins used for public score calculation, this is certainly reasonable. If there is &lt;10 proteins for public score calculation, we should not even try to use public score to judge our model performance.</p></li>\n<li><p>In previous CAFA, the organizer will backpropagate all parent terms given the predicted child terms for the participants. Is this still true for CAFA5? I submitted two models with identical prediction, except that the former does not perform parent term propagation while the latter does. The public score is quite different as calculated by Kaggle leaderboard.</p></li>\n</ol>",
  "messages": [
    {
      "id": "2273934",
      "postDate": "05/25/2023 14:07:56",
      "content": "<p>Dear CAFA organizer,<br>\nI have two questions on the CAFA5 kaggle leaderboard.</p>\n<ol>\n<li><p>Approximately how many test proteins are used to calculate the public score? Although the organizers urgent caution against it, many participants are probably trying to optimize their method for better public score in the leaderboard. If there are ~1000 proteins used for public score calculation, this is certainly reasonable. If there is &lt;10 proteins for public score calculation, we should not even try to use public score to judge our model performance.</p></li>\n<li><p>In previous CAFA, the organizer will backpropagate all parent terms given the predicted child terms for the participants. Is this still true for CAFA5? I submitted two models with identical prediction, except that the former does not perform parent term propagation while the latter does. The public score is quite different as calculated by Kaggle leaderboard.</p></li>\n</ol>",
      "rawMarkdown": "Dear CAFA organizer,\nI have two questions on the CAFA5 kaggle leaderboard.\n\n1. Approximately how many test proteins are used to calculate the public score? Although the organizers urgent caution against it, many participants are probably trying to optimize their method for better public score in the leaderboard. If there are ~1000 proteins used for public score calculation, this is certainly reasonable. If there is <10 proteins for public score calculation, we should not even try to use public score to judge our model performance.\n\n2. In previous CAFA, the organizer will backpropagate all parent terms given the predicted child terms for the participants. Is this still true for CAFA5? I submitted two models with identical prediction, except that the former does not perform parent term propagation while the latter does. The public score is quite different as calculated by Kaggle leaderboard.",
      "votes": null
    },
    {
      "id": "2274114",
      "postDate": "05/25/2023 16:20:23",
      "content": "<p>1) The number is large enough to be meaningful. Also, as in most future-data competitions, it is best to not get too hung up on the leaderboard scoring. </p>\n<p>2) Depends on how you propagate. We propagate via both <code>is_a</code> and <code>part_of</code> relationships. If in your code you used only one or the other, or included some other relationships, your results will be different. </p>",
      "rawMarkdown": "1) The number is large enough to be meaningful. Also, as in most future-data competitions, it is best to not get too hung up on the leaderboard scoring. \n\n2) Depends on how you propagate. We propagate via both `is_a` and `part_of` relationships. If in your code you used only one or the other, or included some other relationships, your results will be different.",
      "votes": null
    },
    {
      "id": "2278579",
      "postDate": "05/28/2023 20:09:56",
      "content": "<p>For 2), I only propagate is_a. However, cafa5 kaggle propagate both is_a and part_of, it should not matter if I propagate via is_a or not. I will resubmit another model based on both is_a and part_of, but I still suspect something is wrong with the propagation part of the assessment code.</p>",
      "rawMarkdown": "For 2), I only propagate is_a. However, cafa5 kaggle propagate both is_a and part_of, it should not matter if I propagate via is_a or not. I will resubmit another model based on both is_a and part_of, but I still suspect something is wrong with the propagation part of the assessment code.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2274114,
      "author_name": "iddux1",
      "author_url": "",
      "post_date": "05/25/2023 16:20:23",
      "content": "<p>1) The number is large enough to be meaningful. Also, as in most future-data competitions, it is best to not get too hung up on the leaderboard scoring. </p>\n<p>2) Depends on how you propagate. We propagate via both <code>is_a</code> and <code>part_of</code> relationships. If in your code you used only one or the other, or included some other relationships, your results will be different. </p>",
      "votes": null,
      "replies": [
        {
          "id": 2278579,
          "author_name": "gdsyzxzcx",
          "author_url": "",
          "post_date": "05/28/2023 20:09:56",
          "content": "<p>For 2), I only propagate is_a. However, cafa5 kaggle propagate both is_a and part_of, it should not matter if I propagate via is_a or not. I will resubmit another model based on both is_a and part_of, but I still suspect something is wrong with the propagation part of the assessment code.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2273934": "Dear CAFA organizer,\nI have two questions on the CAFA5 kaggle leaderboard.\n\n1. Approximately how many test proteins are used to calculate the public score? Although the organizers urgent caution against it, many participants are probably trying to optimize their method for better public score in the leaderboard. If there are ~1000 proteins used for public score calculation, this is certainly reasonable. If there is <10 proteins for public score calculation, we should not even try to use public score to judge our model performance.\n\n2. In previous CAFA, the organizer will backpropagate all parent terms given the predicted child terms for the participants. Is this still true for CAFA5? I submitted two models with identical prediction, except that the former does not perform parent term propagation while the latter does. The public score is quite different as calculated by Kaggle leaderboard.",
    "2274114": "1) The number is large enough to be meaningful. Also, as in most future-data competitions, it is best to not get too hung up on the leaderboard scoring. \n\n2) Depends on how you propagate. We propagate via both `is_a` and `part_of` relationships. If in your code you used only one or the other, or included some other relationships, your results will be different.",
    "2278579": "For 2), I only propagate is_a. However, cafa5 kaggle propagate both is_a and part_of, it should not matter if I propagate via is_a or not. I will resubmit another model based on both is_a and part_of, but I still suspect something is wrong with the propagation part of the assessment code."
  },
  "source": "meta"
}