{
  "id": 433293,
  "title": "There is no private test data?",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/433293",
  "author_name": "",
  "post_date": "2023-08-21T06:51:24.756972800Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>After coming across a few discussions, it's being said that private test data would be made from public test dataset from which proteins will be selected. Is it right? And if that's the case, can top scoring public notebooks outperform even on private LB?  </p>",
  "messages": [
    {
      "id": "2400611",
      "postDate": "08/21/2023 06:51:24",
      "content": "<p>After coming across a few discussions, it's being said that private test data would be made from public test dataset from which proteins will be selected. Is it right? And if that's the case, can top scoring public notebooks outperform even on private LB?  </p>",
      "rawMarkdown": "After coming across a few discussions, it's being said that private test data would be made from public test dataset from which proteins will be selected. Is it right? And if that's the case, can top scoring public notebooks outperform even on private LB?",
      "votes": null
    },
    {
      "id": "2400669",
      "postDate": "08/21/2023 07:25:02",
      "content": "<p>I will tell you my understanding from about 3 months ago when I followed the discussion, but <em>caveat emptor</em>: I joined the competition only a week ago.</p>\n<p>All the proteins that will be evaluated as a private test set are already in test data. At this point I don't think the organizers know their identity either. It will depend on which proteins get reliable annotations by the end of the year. If that subset of freshly annotated proteins ends up being similar to what was used for the public LB, then those merging notebooks may end up doing well. It boils down to luck as to what proteins will be newly annotated. In such a case, I prefer to build a general model than to hope that public and future private subsets will be aligned.</p>\n<p>In case it is not clear: I don't think we will have a private LB tomorrow, and for quite some time after that.</p>",
      "rawMarkdown": "I will tell you my understanding from about 3 months ago when I followed the discussion, but *caveat emptor*: I joined the competition only a week ago.\n\nAll the proteins that will be evaluated as a private test set are already in test data. At this point I don't think the organizers know their identity either. It will depend on which proteins get reliable annotations by the end of the year. If that subset of freshly annotated proteins ends up being similar to what was used for the public LB, then those merging notebooks may end up doing well. It boils down to luck as to what proteins will be newly annotated. In such a case, I prefer to build a general model than to hope that public and future private subsets will be aligned.\n\nIn case it is not clear: I don't think we will have a private LB tomorrow, and for quite some time after that.",
      "votes": null
    },
    {
      "id": "2401182",
      "postDate": "08/21/2023 13:21:20",
      "content": "<p>Fair enough. I completely agree with you. Thank you for taking your time to answer. Much appreciated! :)</p>",
      "rawMarkdown": "Fair enough. I completely agree with you. Thank you for taking your time to answer. Much appreciated! :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2400669,
      "author_name": "tilii7",
      "author_url": "",
      "post_date": "08/21/2023 07:25:02",
      "content": "<p>I will tell you my understanding from about 3 months ago when I followed the discussion, but <em>caveat emptor</em>: I joined the competition only a week ago.</p>\n<p>All the proteins that will be evaluated as a private test set are already in test data. At this point I don't think the organizers know their identity either. It will depend on which proteins get reliable annotations by the end of the year. If that subset of freshly annotated proteins ends up being similar to what was used for the public LB, then those merging notebooks may end up doing well. It boils down to luck as to what proteins will be newly annotated. In such a case, I prefer to build a general model than to hope that public and future private subsets will be aligned.</p>\n<p>In case it is not clear: I don't think we will have a private LB tomorrow, and for quite some time after that.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2401182,
          "author_name": "ahsuna123",
          "author_url": "",
          "post_date": "08/21/2023 13:21:20",
          "content": "<p>Fair enough. I completely agree with you. Thank you for taking your time to answer. Much appreciated! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2400611": "After coming across a few discussions, it's being said that private test data would be made from public test dataset from which proteins will be selected. Is it right? And if that's the case, can top scoring public notebooks outperform even on private LB?",
    "2400669": "I will tell you my understanding from about 3 months ago when I followed the discussion, but *caveat emptor*: I joined the competition only a week ago.\n\nAll the proteins that will be evaluated as a private test set are already in test data. At this point I don't think the organizers know their identity either. It will depend on which proteins get reliable annotations by the end of the year. If that subset of freshly annotated proteins ends up being similar to what was used for the public LB, then those merging notebooks may end up doing well. It boils down to luck as to what proteins will be newly annotated. In such a case, I prefer to build a general model than to hope that public and future private subsets will be aligned.\n\nIn case it is not clear: I don't think we will have a private LB tomorrow, and for quite some time after that.",
    "2401182": "Fair enough. I completely agree with you. Thank you for taking your time to answer. Much appreciated! :)"
  },
  "source": "meta"
}