{
  "id": 433761,
  "title": "Please share your methods to construct validation sets 🤗",
  "url": "/competitions/cafa-5-protein-function-prediction/discussion/433761",
  "author_name": "",
  "post_date": "2023-08-22T19:34:00.548480500Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>In the same spirit as this call to <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433543\" target=\"_blank\">action</a>,  I think it would be very insightful for everybody to see how the experts approached the problem of creating a trustworthy validation scheme.</p>\n<p>Not being an expert myself, it took me a while to understand the evaluation regime and I tried to replicate it as best as I can, but my <a href=\"https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up\" target=\"_blank\">solution</a> might still be lacking due to some bug or misunderstanding.</p>\n<p>Thank you!</p>\n<p>Earlier <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/409329\" target=\"_blank\">discussion</a></p>",
  "messages": [
    {
      "id": "2403669",
      "postDate": "08/22/2023 19:34:00",
      "content": "<p>In the same spirit as this call to <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433543\" target=\"_blank\">action</a>,  I think it would be very insightful for everybody to see how the experts approached the problem of creating a trustworthy validation scheme.</p>\n<p>Not being an expert myself, it took me a while to understand the evaluation regime and I tried to replicate it as best as I can, but my <a href=\"https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up\" target=\"_blank\">solution</a> might still be lacking due to some bug or misunderstanding.</p>\n<p>Thank you!</p>\n<p>Earlier <a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/409329\" target=\"_blank\">discussion</a></p>",
      "rawMarkdown": "In the same spirit as this call to [action](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433543),  I think it would be very insightful for everybody to see how the experts approached the problem of creating a trustworthy validation scheme.\n\nNot being an expert myself, it took me a while to understand the evaluation regime and I tried to replicate it as best as I can, but my [solution](https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up) might still be lacking due to some bug or misunderstanding.\n\nThank you!\n\nEarlier [discussion](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/409329)",
      "votes": null
    },
    {
      "id": "2403820",
      "postDate": "08/22/2023 21:45:51",
      "content": "<p>I did a repeated multilabel stratified k-fold where both number of folds and repeats were 5, so a total of 25 folds with replacement. Target labels were used for the \"multilabel\" part, as the idea was to equally distribute similar proteins across folds.</p>",
      "rawMarkdown": "I did a repeated multilabel stratified k-fold where both number of folds and repeats were 5, so a total of 25 folds with replacement. Target labels were used for the \"multilabel\" part, as the idea was to equally distribute similar proteins across folds.",
      "votes": null
    },
    {
      "id": "2403824",
      "postDate": "08/22/2023 21:52:35",
      "content": "<p>GOTHresher can be used to remove imbalance from data sets: <a href=\"https://github.com/FriedbergLab/GOThresher\" target=\"_blank\">https://github.com/FriedbergLab/GOThresher</a></p>\n<p><a href=\"https://academic.oup.com/bioinformatics/article/39/1/btad048/6998200\" target=\"_blank\">https://academic.oup.com/bioinformatics/article/39/1/btad048/6998200</a></p>",
      "rawMarkdown": "GOTHresher can be used to remove imbalance from data sets: https://github.com/FriedbergLab/GOThresher\n\nhttps://academic.oup.com/bioinformatics/article/39/1/btad048/6998200",
      "votes": null
    },
    {
      "id": "2403957",
      "postDate": "08/23/2023 01:13:52",
      "content": "<p>Hi! I shared MultilabelStratifiedKFold for DEMO<br>\n<a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805\" target=\"_blank\">https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805</a></p>",
      "rawMarkdown": "Hi! I shared MultilabelStratifiedKFold for DEMO\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2403820,
      "author_name": "tilii7",
      "author_url": "",
      "post_date": "08/22/2023 21:45:51",
      "content": "<p>I did a repeated multilabel stratified k-fold where both number of folds and repeats were 5, so a total of 25 folds with replacement. Target labels were used for the \"multilabel\" part, as the idea was to equally distribute similar proteins across folds.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2403824,
      "author_name": "iddux1",
      "author_url": "",
      "post_date": "08/22/2023 21:52:35",
      "content": "<p>GOTHresher can be used to remove imbalance from data sets: <a href=\"https://github.com/FriedbergLab/GOThresher\" target=\"_blank\">https://github.com/FriedbergLab/GOThresher</a></p>\n<p><a href=\"https://academic.oup.com/bioinformatics/article/39/1/btad048/6998200\" target=\"_blank\">https://academic.oup.com/bioinformatics/article/39/1/btad048/6998200</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2403957,
      "author_name": "isakatsuyoshi",
      "author_url": "",
      "post_date": "08/23/2023 01:13:52",
      "content": "<p>Hi! I shared MultilabelStratifiedKFold for DEMO<br>\n<a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805\" target=\"_blank\">https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2403669": "In the same spirit as this call to [action](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433543),  I think it would be very insightful for everybody to see how the experts approached the problem of creating a trustworthy validation scheme.\n\nNot being an expert myself, it took me a while to understand the evaluation regime and I tried to replicate it as best as I can, but my [solution](https://www.kaggle.com/code/dustinthewind/validation-set-that-mimicks-the-evaluation-set-up) might still be lacking due to some bug or misunderstanding.\n\nThank you!\n\nEarlier [discussion](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/409329)",
    "2403820": "I did a repeated multilabel stratified k-fold where both number of folds and repeats were 5, so a total of 25 folds with replacement. Target labels were used for the \"multilabel\" part, as the idea was to equally distribute similar proteins across folds.",
    "2403824": "GOTHresher can be used to remove imbalance from data sets: https://github.com/FriedbergLab/GOThresher\n\nhttps://academic.oup.com/bioinformatics/article/39/1/btad048/6998200",
    "2403957": "Hi! I shared MultilabelStratifiedKFold for DEMO\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805"
  },
  "source": "meta"
}