{
  "id": 425389,
  "title": "If all proteins in the private test set are novel?",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/425389",
  "author_name": "",
  "post_date": "2023-07-18T13:38:46.105629700Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I want to know if all proteins in the private test set are novel?<br>\nIf I predict the known function of a non-novel protein, will it affect the scoring? </p>",
  "messages": [
    {
      "id": "2349531",
      "postDate": "07/18/2023 13:38:46",
      "content": "<p>I want to know if all proteins in the private test set are novel?<br>\nIf I predict the known function of a non-novel protein, will it affect the scoring? </p>",
      "rawMarkdown": "I want to know if all proteins in the private test set are novel?\nIf I predict the known function of a non-novel protein, will it affect the scoring?",
      "votes": null
    },
    {
      "id": "2351143",
      "postDate": "07/19/2023 21:14:16",
      "content": "<blockquote>\n  <p>I want to know if all proteins in the private test set are novel?</p>\n</blockquote>\n<p>None of the proteins are novel, as you cannot predict the function of a proteins you can't see. What we do have are proteins whose functions, in on GO aspect  to more, that are yet unknown: i.e. not yet verified experimentally.</p>\n<blockquote>\n  <p>If I predict the known function of a non-novel protein, will it affect the scoring? </p>\n</blockquote>\n<p>We will simply not score that prediction. </p>",
      "rawMarkdown": "> I want to know if all proteins in the private test set are novel?\n\nNone of the proteins are novel, as you cannot predict the function of a proteins you can't see. What we do have are proteins whose functions, in on GO aspect  to more, that are yet unknown: i.e. not yet verified experimentally.\n\n> If I predict the known function of a non-novel protein, will it affect the scoring? \n\nWe will simply not score that prediction.",
      "votes": null
    },
    {
      "id": "2351278",
      "postDate": "07/20/2023 02:35:08",
      "content": "<p>Thanks for your reply.</p>\n<p><code>None of the proteins are novel</code></p>\n<p>Can I understand that all proteins evaluated in the final assessment are included in the current train and test sets?</p>\n<p><code>We will simply not score that prediction</code></p>\n<p>I didn't see this feature in the CAFA-evaluator. So, is there an additional process in evaluation to exclude the known functions of proteins?</p>",
      "rawMarkdown": "Thanks for your reply.\n\n`None of the proteins are novel`\n\nCan I understand that all proteins evaluated in the final assessment are included in the current train and test sets?\n\n`We will simply not score that prediction`\n\nI didn't see this feature in the CAFA-evaluator. So, is there an additional process in evaluation to exclude the known functions of proteins?",
      "votes": null
    },
    {
      "id": "2351689",
      "postDate": "07/20/2023 10:31:40",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/daishu\" target=\"_blank\">@daishu</a>,</p>\n<p>For the final evaluation, there will be three test sets (<em>i.e.,</em> <code>BPO</code>, <code>MFO</code> and <code>CCO</code>) constructed, one per namespace. AFAIK, if a protein \\(p_i \\) already has a experimentally-verified annotation in one subontology (<em>e.g.,</em> <code>MFO</code>), then this protein won't be included in the final test set of <code>MFO</code> (my thought about the final test set is posted <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/421887\" target=\"_blank\">here</a>).</p>\n<p>Hence, we can expect that we won't see \\(p_i \\) in <code>GroundTruth</code> object of <code>MFO</code>. If you predict a known function of a non-novel protein, the condition at the line 126 of the <code>pred_parser</code> function of  <code>parser.py</code> is <code>False</code> (because \\(p_i \\) doesn't exist in <code>gts[ns].ids</code>, where <code>ns</code> is <code>\"molecular_function\"</code> following the example above),</p>\n<pre><code> ns  gts and p_id  gts:\n</code></pre>\n<p>This is my understanding about the second question, hope this helps.</p>",
      "rawMarkdown": "Hi @daishu,\n\nFor the final evaluation, there will be three test sets (*i.e.,* `BPO`, `MFO` and `CCO`) constructed, one per namespace. AFAIK, if a protein \\\\(p_i \\\\) already has a experimentally-verified annotation in one subontology (*e.g.,* `MFO`), then this protein won't be included in the final test set of `MFO` (my thought about the final test set is posted [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/421887)).\n\nHence, we can expect that we won't see \\\\(p_i \\\\) in `GroundTruth` object of `MFO`. If you predict a known function of a non-novel protein, the condition at the line 126 of the `pred_parser` function of  `parser.py` is `False` (because \\\\(p_i \\\\) doesn't exist in `gts[ns].ids`, where `ns` is `\"molecular_function\"` following the example above),\n\n```\nif ns in gts and p_id in gts[ns].ids:\n```\n\nThis is my understanding about the second question, hope this helps.",
      "votes": null
    },
    {
      "id": "2351937",
      "postDate": "07/20/2023 13:45:20",
      "content": "<p>Thank you very much. I think you are right. This will help me construct the validation set.</p>",
      "rawMarkdown": "Thank you very much. I think you are right. This will help me construct the validation set.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2351143,
      "author_name": "iddux1",
      "author_url": "",
      "post_date": "07/19/2023 21:14:16",
      "content": "<blockquote>\n  <p>I want to know if all proteins in the private test set are novel?</p>\n</blockquote>\n<p>None of the proteins are novel, as you cannot predict the function of a proteins you can't see. What we do have are proteins whose functions, in on GO aspect  to more, that are yet unknown: i.e. not yet verified experimentally.</p>\n<blockquote>\n  <p>If I predict the known function of a non-novel protein, will it affect the scoring? </p>\n</blockquote>\n<p>We will simply not score that prediction. </p>",
      "votes": null,
      "replies": [
        {
          "id": 2351278,
          "author_name": "daishu",
          "author_url": "",
          "post_date": "07/20/2023 02:35:08",
          "content": "<p>Thanks for your reply.</p>\n<p><code>None of the proteins are novel</code></p>\n<p>Can I understand that all proteins evaluated in the final assessment are included in the current train and test sets?</p>\n<p><code>We will simply not score that prediction</code></p>\n<p>I didn't see this feature in the CAFA-evaluator. So, is there an additional process in evaluation to exclude the known functions of proteins?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2351689,
              "author_name": "abaojiang",
              "author_url": "",
              "post_date": "07/20/2023 10:31:40",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/daishu\" target=\"_blank\">@daishu</a>,</p>\n<p>For the final evaluation, there will be three test sets (<em>i.e.,</em> <code>BPO</code>, <code>MFO</code> and <code>CCO</code>) constructed, one per namespace. AFAIK, if a protein \\(p_i \\) already has a experimentally-verified annotation in one subontology (<em>e.g.,</em> <code>MFO</code>), then this protein won't be included in the final test set of <code>MFO</code> (my thought about the final test set is posted <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/421887\" target=\"_blank\">here</a>).</p>\n<p>Hence, we can expect that we won't see \\(p_i \\) in <code>GroundTruth</code> object of <code>MFO</code>. If you predict a known function of a non-novel protein, the condition at the line 126 of the <code>pred_parser</code> function of  <code>parser.py</code> is <code>False</code> (because \\(p_i \\) doesn't exist in <code>gts[ns].ids</code>, where <code>ns</code> is <code>\"molecular_function\"</code> following the example above),</p>\n<pre><code> ns  gts and p_id  gts:\n</code></pre>\n<p>This is my understanding about the second question, hope this helps.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2351937,
                  "author_name": "daishu",
                  "author_url": "",
                  "post_date": "07/20/2023 13:45:20",
                  "content": "<p>Thank you very much. I think you are right. This will help me construct the validation set.</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2349531": "I want to know if all proteins in the private test set are novel?\nIf I predict the known function of a non-novel protein, will it affect the scoring?",
    "2351143": "> I want to know if all proteins in the private test set are novel?\n\nNone of the proteins are novel, as you cannot predict the function of a proteins you can't see. What we do have are proteins whose functions, in on GO aspect  to more, that are yet unknown: i.e. not yet verified experimentally.\n\n> If I predict the known function of a non-novel protein, will it affect the scoring? \n\nWe will simply not score that prediction.",
    "2351278": "Thanks for your reply.\n\n`None of the proteins are novel`\n\nCan I understand that all proteins evaluated in the final assessment are included in the current train and test sets?\n\n`We will simply not score that prediction`\n\nI didn't see this feature in the CAFA-evaluator. So, is there an additional process in evaluation to exclude the known functions of proteins?",
    "2351689": "Hi @daishu,\n\nFor the final evaluation, there will be three test sets (*i.e.,* `BPO`, `MFO` and `CCO`) constructed, one per namespace. AFAIK, if a protein \\\\(p_i \\\\) already has a experimentally-verified annotation in one subontology (*e.g.,* `MFO`), then this protein won't be included in the final test set of `MFO` (my thought about the final test set is posted [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/421887)).\n\nHence, we can expect that we won't see \\\\(p_i \\\\) in `GroundTruth` object of `MFO`. If you predict a known function of a non-novel protein, the condition at the line 126 of the `pred_parser` function of  `parser.py` is `False` (because \\\\(p_i \\\\) doesn't exist in `gts[ns].ids`, where `ns` is `\"molecular_function\"` following the example above),\n\n```\nif ns in gts and p_id in gts[ns].ids:\n```\n\nThis is my understanding about the second question, hope this helps.",
    "2351937": "Thank you very much. I think you are right. This will help me construct the validation set."
  },
  "source": "meta"
}