{
  "id": 433603,
  "title": "Public 9th/Private 331st Place Solution - A Non-Bioinformatician's Journey in CAFA",
  "url": "/competitions/cafa-5-protein-function-prediction/writeups/graphninjas-public-9th-private-331st-place-solutio",
  "author_name": "",
  "post_date": "2024-08-23T08:31:52.047Z",
  "votes": 29,
  "comment_count": 18,
  "views": 0,
  "content": "<h1>1. Public 9th/Private 331st Place Solution -  A Non-Bioinformatician's Journey in CAFA</h1>\n<p>First and foremost, I would like to express my gratitude to my teammates, the organizers, Kaggle's management team, and everyone who participated with us. Through this contest, I have fallen in love with CAFA. I'm looking forward to CAFA6.<br>\nI will introduce our best CV solution. Unlike many top Kagglers, we are not bioinformaticians, so our focus is on machine learning.</p>\n<h1>2. Competition Overview and Impressions</h1>\n<ul>\n<li>This competition is a multi-target task to predict biological functions (multiple) for each protein.</li>\n<li>An extremely imbalanced dataset where most labels exist less than 1%.</li>\n<li>A complex competition metric, considering Information Accumulation IA for the maximum F-value.</li>\n<li>As many participants felt, this competition was challenging. Conversely, hard work can make a difference.</li>\n</ul>\n<h1>3. Data</h1>\n<h2>3.1 X(N, M): Explanatory Variables (Embedding Representation &amp; Features)</h2>\n<ul>\n<li>Created data<ul>\n<li>Embedding representations: ESM2, T5, ProtBERT, SGT, etc.</li>\n<li>Features: Physicochemical properties calculated from amino acid sequences using BioPython, Levenshtein distance, Species, time information features obtained from the UniProt database, is_common, annotation scores, etc.</li>\n<li>Reduction of embedding representations and features to 3 dimensions using t-SNE/UMAP.</li></ul></li>\n<li>Data type: Converted to float32 (to save memory).</li>\n<li>Preprocessing<ul>\n<li>Combination of standardization, MinMax transformation, logarithmic transformation, interquartile range transformation, clipping, etc., based on the distribution of each feature.</li>\n<li>Concatenation of numerous embedding representations and features for training.</li></ul></li>\n<li>Feature selection<ul>\n<li>Removal of features with std less than 0.1 (excluding originally small values such as amino acid sequence ratios).</li>\n<li>Removal of one of the features if corr is 0.95 or higher.</li>\n<li>Removal of features with too different distributions between train/test.</li>\n<li>Feature removal relying on TabNet importance and experience.</li></ul></li>\n<li>Examples of used datasets<ul>\n<li>X0: tSVD2048(ESM2, T5, Species) + time features from UniProt + annotation score + is_common + UMAP(Species) + UMAP(ProtBERT) + UMAP(Levenshtein distance)</li>\n<li>X1: ESM</li>\n<li>X2: T5+Species</li></ul></li>\n</ul>\n<h2>3.2 y(N, K): Target Variables &amp; Labels</h2>\n<ul>\n<li>Updated some y labels using the UniProt database.</li>\n<li>For groups with identical amino acid sequences, if one is 1, others are also assigned 1.</li>\n<li>y is int8, labels are converted to \"&lt;U10\" data type (to save memory).</li>\n<li>GO terms are extracted in the order of IA × frequency, with 1100, 300, 850 items for BP, CC, MF, respectively.</li>\n</ul>\n<h2>3.3 ids: Index</h2>\n<ul>\n<li>Removed one duplicate in test_ids.</li>\n<li>Sorted train_ids, test_ids alphabetically (to handle all data in the same format!).</li>\n<li>Indices for multi-label stratified cross-validation are saved in pickle (to save computation time).</li>\n</ul>\n<h1>4. Validation Method</h1>\n<ul>\n<li>In this contest, the validation method was one of the essential elements. In conclusion, we used multi-target stratified cross-validation.</li>\n<li>Averaged across k=4,5,6 splits in cross-validation. This not only increases robustness but also improves CV by +0.03-0.04!</li>\n<li>In the competition's IA-weighted maximum F-value, we used 1100, 300, 850 GO terms for BP, CC, MF, respectively, for training, and 5000, 2079, 4609 GO terms for evaluation. This brings us closer to the leaderboard scoring calculated for all GOs.</li>\n<li>Trials and errors until the exploration of other validation methods<ul>\n<li>First, we attempted to split by time series. We used the UniProt API to obtain the creation date for each ID. Ultimately, we did not proceed with time-series splitting.</li>\n<li>Next, we evaluated the difference between train and test using adversarial validation. LightGBM and XGBoost had strange AUC values below 0.2, but MLP had an AUC of 0.5. This reaffirmed that embedding representations are suitable for MLP. Since the AUC was 0.5, we hypothesized that the distributions of train and test were not significantly different.</li>\n<li>Based on the above adversarial validation results, we switched to making cross-validation more robust. Initially, we used random cross-validation.</li>\n<li>However, since it's imbalanced data, we switched to multi-label stratified cross-validation. In conclusion, this was the best.</li>\n<li>For tuning the model validation system, checks for ① consistency of local tests (no significant variations in errors between folds), ② correlation between local validation scores and leaderboard were necessary. ① is fine. For ②, since the number of IDs used in the public leaderboard was estimated to be around 60, we lowered the priority.</li>\n<li>In multi-label stratified cross-validation, we calculated the IA-weighted MSE for the ratio of labels considered, but eventually, we targeted all GOs for stratification.</li></ul></li>\n</ul>\n<h1>5. Models</h1>\n<p>The models were inspired by the top solutions of <a href=\"https://www.kaggle.com/competitions/lish-moa/overview\" target=\"_blank\">this competition</a>. Additionally, we used basic models for diversity.</p>\n<ul>\n<li>MLP: Multiple dataset patterns were input to create diverse MLPs.</li>\n<li>1D-CNN: Ensemble member.</li>\n<li>TabNet: Used only for feature selection.</li>\n<li>knn: Ensemble member.</li>\n<li>Ridge: Was an ensemble member but the weight became 0.</li>\n</ul>\n<p>Example of weighted CV highest: Aspect, [MLP(X0), MLP(X1), MLP(X2), knn, 1D-CNN]=CV score</p>\n<ul>\n<li>BP, [0.8, 0.1, 0.1, 0.3, 0.3]=0.5589730655644398</li>\n<li>CC, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6453054780984085</li>\n<li>MF, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6771821551</li>\n</ul>\n<p>Correlation matrix calculated from CC's test predictions (other aspects, train predictions were also able to create diverse models)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fc864fe0de0aeb2a8f15e206b1e01fbb2%2F2023-08-22%20195144.png?generation=1692701663574782&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "2402773",
      "postDate": "08/22/2023 10:32:33",
      "content": "<h1>1. Public 9th/Private 331st Place Solution -  A Non-Bioinformatician's Journey in CAFA</h1>\n<p>First and foremost, I would like to express my gratitude to my teammates, the organizers, Kaggle's management team, and everyone who participated with us. Through this contest, I have fallen in love with CAFA. I'm looking forward to CAFA6.<br>\nI will introduce our best CV solution. Unlike many top Kagglers, we are not bioinformaticians, so our focus is on machine learning.</p>\n<h1>2. Competition Overview and Impressions</h1>\n<ul>\n<li>This competition is a multi-target task to predict biological functions (multiple) for each protein.</li>\n<li>An extremely imbalanced dataset where most labels exist less than 1%.</li>\n<li>A complex competition metric, considering Information Accumulation IA for the maximum F-value.</li>\n<li>As many participants felt, this competition was challenging. Conversely, hard work can make a difference.</li>\n</ul>\n<h1>3. Data</h1>\n<h2>3.1 X(N, M): Explanatory Variables (Embedding Representation &amp; Features)</h2>\n<ul>\n<li>Created data<ul>\n<li>Embedding representations: ESM2, T5, ProtBERT, SGT, etc.</li>\n<li>Features: Physicochemical properties calculated from amino acid sequences using BioPython, Levenshtein distance, Species, time information features obtained from the UniProt database, is_common, annotation scores, etc.</li>\n<li>Reduction of embedding representations and features to 3 dimensions using t-SNE/UMAP.</li></ul></li>\n<li>Data type: Converted to float32 (to save memory).</li>\n<li>Preprocessing<ul>\n<li>Combination of standardization, MinMax transformation, logarithmic transformation, interquartile range transformation, clipping, etc., based on the distribution of each feature.</li>\n<li>Concatenation of numerous embedding representations and features for training.</li></ul></li>\n<li>Feature selection<ul>\n<li>Removal of features with std less than 0.1 (excluding originally small values such as amino acid sequence ratios).</li>\n<li>Removal of one of the features if corr is 0.95 or higher.</li>\n<li>Removal of features with too different distributions between train/test.</li>\n<li>Feature removal relying on TabNet importance and experience.</li></ul></li>\n<li>Examples of used datasets<ul>\n<li>X0: tSVD2048(ESM2, T5, Species) + time features from UniProt + annotation score + is_common + UMAP(Species) + UMAP(ProtBERT) + UMAP(Levenshtein distance)</li>\n<li>X1: ESM</li>\n<li>X2: T5+Species</li></ul></li>\n</ul>\n<h2>3.2 y(N, K): Target Variables &amp; Labels</h2>\n<ul>\n<li>Updated some y labels using the UniProt database.</li>\n<li>For groups with identical amino acid sequences, if one is 1, others are also assigned 1.</li>\n<li>y is int8, labels are converted to \"&lt;U10\" data type (to save memory).</li>\n<li>GO terms are extracted in the order of IA × frequency, with 1100, 300, 850 items for BP, CC, MF, respectively.</li>\n</ul>\n<h2>3.3 ids: Index</h2>\n<ul>\n<li>Removed one duplicate in test_ids.</li>\n<li>Sorted train_ids, test_ids alphabetically (to handle all data in the same format!).</li>\n<li>Indices for multi-label stratified cross-validation are saved in pickle (to save computation time).</li>\n</ul>\n<h1>4. Validation Method</h1>\n<ul>\n<li>In this contest, the validation method was one of the essential elements. In conclusion, we used multi-target stratified cross-validation.</li>\n<li>Averaged across k=4,5,6 splits in cross-validation. This not only increases robustness but also improves CV by +0.03-0.04!</li>\n<li>In the competition's IA-weighted maximum F-value, we used 1100, 300, 850 GO terms for BP, CC, MF, respectively, for training, and 5000, 2079, 4609 GO terms for evaluation. This brings us closer to the leaderboard scoring calculated for all GOs.</li>\n<li>Trials and errors until the exploration of other validation methods<ul>\n<li>First, we attempted to split by time series. We used the UniProt API to obtain the creation date for each ID. Ultimately, we did not proceed with time-series splitting.</li>\n<li>Next, we evaluated the difference between train and test using adversarial validation. LightGBM and XGBoost had strange AUC values below 0.2, but MLP had an AUC of 0.5. This reaffirmed that embedding representations are suitable for MLP. Since the AUC was 0.5, we hypothesized that the distributions of train and test were not significantly different.</li>\n<li>Based on the above adversarial validation results, we switched to making cross-validation more robust. Initially, we used random cross-validation.</li>\n<li>However, since it's imbalanced data, we switched to multi-label stratified cross-validation. In conclusion, this was the best.</li>\n<li>For tuning the model validation system, checks for ① consistency of local tests (no significant variations in errors between folds), ② correlation between local validation scores and leaderboard were necessary. ① is fine. For ②, since the number of IDs used in the public leaderboard was estimated to be around 60, we lowered the priority.</li>\n<li>In multi-label stratified cross-validation, we calculated the IA-weighted MSE for the ratio of labels considered, but eventually, we targeted all GOs for stratification.</li></ul></li>\n</ul>\n<h1>5. Models</h1>\n<p>The models were inspired by the top solutions of <a href=\"https://www.kaggle.com/competitions/lish-moa/overview\" target=\"_blank\">this competition</a>. Additionally, we used basic models for diversity.</p>\n<ul>\n<li>MLP: Multiple dataset patterns were input to create diverse MLPs.</li>\n<li>1D-CNN: Ensemble member.</li>\n<li>TabNet: Used only for feature selection.</li>\n<li>knn: Ensemble member.</li>\n<li>Ridge: Was an ensemble member but the weight became 0.</li>\n</ul>\n<p>Example of weighted CV highest: Aspect, [MLP(X0), MLP(X1), MLP(X2), knn, 1D-CNN]=CV score</p>\n<ul>\n<li>BP, [0.8, 0.1, 0.1, 0.3, 0.3]=0.5589730655644398</li>\n<li>CC, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6453054780984085</li>\n<li>MF, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6771821551</li>\n</ul>\n<p>Correlation matrix calculated from CC's test predictions (other aspects, train predictions were also able to create diverse models)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fc864fe0de0aeb2a8f15e206b1e01fbb2%2F2023-08-22%20195144.png?generation=1692701663574782&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "# 1. Public 9th/Private 331st Place Solution -  A Non-Bioinformatician's Journey in CAFA\n\nFirst and foremost, I would like to express my gratitude to my teammates, the organizers, Kaggle's management team, and everyone who participated with us. Through this contest, I have fallen in love with CAFA. I'm looking forward to CAFA6.\nI will introduce our best CV solution. Unlike many top Kagglers, we are not bioinformaticians, so our focus is on machine learning.\n\n# 2. Competition Overview and Impressions\n- This competition is a multi-target task to predict biological functions (multiple) for each protein.\n- An extremely imbalanced dataset where most labels exist less than 1%.\n- A complex competition metric, considering Information Accumulation IA for the maximum F-value.\n- As many participants felt, this competition was challenging. Conversely, hard work can make a difference.\n\n# 3. Data\n\n## 3.1 X(N, M): Explanatory Variables (Embedding Representation & Features)\n- Created data\n  - Embedding representations: ESM2, T5, ProtBERT, SGT, etc.\n  - Features: Physicochemical properties calculated from amino acid sequences using BioPython, Levenshtein distance, Species, time information features obtained from the UniProt database, is_common, annotation scores, etc.\n  - Reduction of embedding representations and features to 3 dimensions using t-SNE/UMAP.\n- Data type: Converted to float32 (to save memory).\n- Preprocessing\n  - Combination of standardization, MinMax transformation, logarithmic transformation, interquartile range transformation, clipping, etc., based on the distribution of each feature.\n  - Concatenation of numerous embedding representations and features for training.\n- Feature selection\n    - Removal of features with std less than 0.1 (excluding originally small values such as amino acid sequence ratios).\n    - Removal of one of the features if corr is 0.95 or higher.\n    - Removal of features with too different distributions between train/test.\n    - Feature removal relying on TabNet importance and experience.\n- Examples of used datasets\n  - X0: tSVD2048(ESM2, T5, Species) + time features from UniProt + annotation score + is_common + UMAP(Species) + UMAP(ProtBERT) + UMAP(Levenshtein distance)\n  - X1: ESM\n  - X2: T5+Species\n\n## 3.2 y(N, K): Target Variables & Labels\n- Updated some y labels using the UniProt database.\n- For groups with identical amino acid sequences, if one is 1, others are also assigned 1.\n- y is int8, labels are converted to \"<U10\" data type (to save memory).\n- GO terms are extracted in the order of IA × frequency, with 1100, 300, 850 items for BP, CC, MF, respectively.\n\n## 3.3 ids: Index\n- Removed one duplicate in test_ids.\n- Sorted train_ids, test_ids alphabetically (to handle all data in the same format!).\n- Indices for multi-label stratified cross-validation are saved in pickle (to save computation time).\n\n# 4. Validation Method\n- In this contest, the validation method was one of the essential elements. In conclusion, we used multi-target stratified cross-validation.\n- Averaged across k=4,5,6 splits in cross-validation. This not only increases robustness but also improves CV by +0.03-0.04!\n- In the competition's IA-weighted maximum F-value, we used 1100, 300, 850 GO terms for BP, CC, MF, respectively, for training, and 5000, 2079, 4609 GO terms for evaluation. This brings us closer to the leaderboard scoring calculated for all GOs.\n- Trials and errors until the exploration of other validation methods\n  - First, we attempted to split by time series. We used the UniProt API to obtain the creation date for each ID. Ultimately, we did not proceed with time-series splitting.\n  - Next, we evaluated the difference between train and test using adversarial validation. LightGBM and XGBoost had strange AUC values below 0.2, but MLP had an AUC of 0.5. This reaffirmed that embedding representations are suitable for MLP. Since the AUC was 0.5, we hypothesized that the distributions of train and test were not significantly different.\n  - Based on the above adversarial validation results, we switched to making cross-validation more robust. Initially, we used random cross-validation.\n  - However, since it's imbalanced data, we switched to multi-label stratified cross-validation. In conclusion, this was the best.\n  - For tuning the model validation system, checks for ① consistency of local tests (no significant variations in errors between folds), ② correlation between local validation scores and leaderboard were necessary. ① is fine. For ②, since the number of IDs used in the public leaderboard was estimated to be around 60, we lowered the priority.\n  - In multi-label stratified cross-validation, we calculated the IA-weighted MSE for the ratio of labels considered, but eventually, we targeted all GOs for stratification.\n\n# 5. Models\nThe models were inspired by the top solutions of [this competition](https://www.kaggle.com/competitions/lish-moa/overview). Additionally, we used basic models for diversity.\n- MLP: Multiple dataset patterns were input to create diverse MLPs.\n- 1D-CNN: Ensemble member.\n- TabNet: Used only for feature selection.\n- knn: Ensemble member.\n- Ridge: Was an ensemble member but the weight became 0.\n\n\nExample of weighted CV highest: Aspect, [MLP(X0), MLP(X1), MLP(X2), knn, 1D-CNN]=CV score\n- BP, [0.8, 0.1, 0.1, 0.3, 0.3]=0.5589730655644398\n- CC, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6453054780984085\n- MF, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6771821551\n\nCorrelation matrix calculated from CC's test predictions (other aspects, train predictions were also able to create diverse models)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fc864fe0de0aeb2a8f15e206b1e01fbb2%2F2023-08-22%20195144.png?generation=1692701663574782&alt=media)",
      "votes": null
    },
    {
      "id": "2402896",
      "postDate": "08/22/2023 11:46:47",
      "content": "<p>Great solution </p>",
      "rawMarkdown": "Great solution",
      "votes": null
    },
    {
      "id": "2402910",
      "postDate": "08/22/2023 11:58:10",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!",
      "votes": null
    },
    {
      "id": "2402968",
      "postDate": "08/22/2023 12:29:14",
      "content": "<p>Congratulations, it seems to be a very strong solution</p>",
      "rawMarkdown": "Congratulations, it seems to be a very strong solution",
      "votes": null
    },
    {
      "id": "2402972",
      "postDate": "08/22/2023 12:32:00",
      "content": "<p>Thank you for your kind words!</p>",
      "rawMarkdown": "Thank you for your kind words!",
      "votes": null
    },
    {
      "id": "2403341",
      "postDate": "08/22/2023 16:16:06",
      "content": "<p>Hi!<br>\nThank you for sharing how your solution works, I'm a newbie in the data science field that graduated in biology and I found this competition to be very hard and (obviously) outside of my current capabilities, and I was hoping that participants form the top teams would share how they did to learn more. Your post was super valuable in that regard, thank you for that!</p>",
      "rawMarkdown": "Hi!\nThank you for sharing how your solution works, I'm a newbie in the data science field that graduated in biology and I found this competition to be very hard and (obviously) outside of my current capabilities, and I was hoping that participants form the top teams would share how they did to learn more. Your post was super valuable in that regard, thank you for that!",
      "votes": null
    },
    {
      "id": "2403353",
      "postDate": "08/22/2023 16:33:18",
      "content": "<p>Congratulations, It was great !</p>",
      "rawMarkdown": "Congratulations, It was great !",
      "votes": null
    },
    {
      "id": "2403607",
      "postDate": "08/22/2023 18:36:48",
      "content": "<p>I think it is really good and strong solution from ML side with no bioonfirmatician background.<br>\nI hope you will get your kaggle competition master </p>",
      "rawMarkdown": "I think it is really good and strong solution from ML side with no bioonfirmatician background.\nI hope you will get your kaggle competition master",
      "votes": null
    },
    {
      "id": "2403789",
      "postDate": "08/22/2023 21:13:35",
      "content": "<p>Great solution, congratulations! It seems that approaching this as a tabular competition with all the tricks and best practices worked out really well :) </p>",
      "rawMarkdown": "Great solution, congratulations! It seems that approaching this as a tabular competition with all the tricks and best practices worked out really well :)",
      "votes": null
    },
    {
      "id": "2403868",
      "postDate": "08/22/2023 22:45:32",
      "content": "<p>Thanks for sharing. That mean k-fold strategy is very interesting, I've never heard of it before</p>",
      "rawMarkdown": "Thanks for sharing. That mean k-fold strategy is very interesting, I've never heard of it before",
      "votes": null
    },
    {
      "id": "2403916",
      "postDate": "08/23/2023 00:03:24",
      "content": "<p>Thank you! I was considering seed averaging at first, but I thought the randomness would be reduced since I am using multi-label stratified cross-validation. That's when I came up with the idea of averaging with this mean k-fold strategy.</p>",
      "rawMarkdown": "Thank you! I was considering seed averaging at first, but I thought the randomness would be reduced since I am using multi-label stratified cross-validation. That's when I came up with the idea of averaging with this mean k-fold strategy.",
      "votes": null
    },
    {
      "id": "2403918",
      "postDate": "08/23/2023 00:04:03",
      "content": "<p>Thank you! Actually, I have read all the Kaggle-related books released in Japan Hahaha.</p>",
      "rawMarkdown": "Thank you! Actually, I have read all the Kaggle-related books released in Japan Hahaha.",
      "votes": null
    },
    {
      "id": "2403919",
      "postDate": "08/23/2023 00:04:46",
      "content": "<p>Thank you for the warm words! I'm looking forward to seeing the solutions shared by bioinformaticians.</p>",
      "rawMarkdown": "Thank you for the warm words! I'm looking forward to seeing the solutions shared by bioinformaticians.",
      "votes": null
    },
    {
      "id": "2403922",
      "postDate": "08/23/2023 00:05:37",
      "content": "<p>Thanks for the encouraging words!</p>",
      "rawMarkdown": "Thanks for the encouraging words!",
      "votes": null
    },
    {
      "id": "2403924",
      "postDate": "08/23/2023 00:06:29",
      "content": "<p>Thank you! It was a very difficult competition. Breaking down the difficulties little by little worked well.</p>",
      "rawMarkdown": "Thank you! It was a very difficult competition. Breaking down the difficulties little by little worked well.",
      "votes": null
    },
    {
      "id": "2403964",
      "postDate": "08/23/2023 01:19:10",
      "content": "<p>I shared MultilabelStratifiedKFold code for DEMO.<br>\n<a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805\" target=\"_blank\">https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805</a></p>",
      "rawMarkdown": "I shared MultilabelStratifiedKFold code for DEMO.\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805",
      "votes": null
    },
    {
      "id": "2405067",
      "postDate": "08/23/2023 17:05:18",
      "content": "<p>Thanks! A delightful reading.  I wondered if you tried combining your predictions with quickgo or other external predictions/database.</p>",
      "rawMarkdown": "Thanks! A delightful reading.  I wondered if you tried combining your predictions with quickgo or other external predictions/database.",
      "votes": null
    },
    {
      "id": "2405194",
      "postDate": "08/23/2023 18:14:44",
      "content": "<p>Thank you! I was really torn between the potential for high accuracy and increasing complexity. In the end, I only combined it with quickgo (which you shared with us!).I'm also curious how other participants have combined external predictions and database.</p>",
      "rawMarkdown": "Thank you! I was really torn between the potential for high accuracy and increasing complexity. In the end, I only combined it with quickgo (which you shared with us!).I'm also curious how other participants have combined external predictions and database.",
      "votes": null
    },
    {
      "id": "2408983",
      "postDate": "08/26/2023 01:41:22",
      "content": "<p>I am happy to share the notebook for generating the dataset.<br>\n\"Sophisticated Data Preprocessing for the CAFA5 Dataset\"<br>\n<a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/434656\" target=\"_blank\">https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/434656</a></p>",
      "rawMarkdown": "I am happy to share the notebook for generating the dataset.\n\"Sophisticated Data Preprocessing for the CAFA5 Dataset\"\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/434656",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2402896,
      "author_name": "julianmukaj",
      "author_url": "",
      "post_date": "08/22/2023 11:46:47",
      "content": "<p>Great solution </p>",
      "votes": null,
      "replies": [
        {
          "id": 2402910,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/22/2023 11:58:10",
          "content": "<p>Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2402968,
      "author_name": "gabrielbianchin",
      "author_url": "",
      "post_date": "08/22/2023 12:29:14",
      "content": "<p>Congratulations, it seems to be a very strong solution</p>",
      "votes": null,
      "replies": [
        {
          "id": 2402972,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/22/2023 12:32:00",
          "content": "<p>Thank you for your kind words!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2403341,
      "author_name": "vembe06",
      "author_url": "",
      "post_date": "08/22/2023 16:16:06",
      "content": "<p>Hi!<br>\nThank you for sharing how your solution works, I'm a newbie in the data science field that graduated in biology and I found this competition to be very hard and (obviously) outside of my current capabilities, and I was hoping that participants form the top teams would share how they did to learn more. Your post was super valuable in that regard, thank you for that!</p>",
      "votes": null,
      "replies": [
        {
          "id": 2403924,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/23/2023 00:06:29",
          "content": "<p>Thank you! It was a very difficult competition. Breaking down the difficulties little by little worked well.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2403353,
      "author_name": "rbarahmand",
      "author_url": "",
      "post_date": "08/22/2023 16:33:18",
      "content": "<p>Congratulations, It was great !</p>",
      "votes": null,
      "replies": [
        {
          "id": 2403922,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/23/2023 00:05:37",
          "content": "<p>Thanks for the encouraging words!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2403607,
      "author_name": "bejeweled",
      "author_url": "",
      "post_date": "08/22/2023 18:36:48",
      "content": "<p>I think it is really good and strong solution from ML side with no bioonfirmatician background.<br>\nI hope you will get your kaggle competition master </p>",
      "votes": null,
      "replies": [
        {
          "id": 2403919,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/23/2023 00:04:46",
          "content": "<p>Thank you for the warm words! I'm looking forward to seeing the solutions shared by bioinformaticians.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2403789,
      "author_name": "thedrcat",
      "author_url": "",
      "post_date": "08/22/2023 21:13:35",
      "content": "<p>Great solution, congratulations! It seems that approaching this as a tabular competition with all the tricks and best practices worked out really well :) </p>",
      "votes": null,
      "replies": [
        {
          "id": 2403918,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/23/2023 00:04:03",
          "content": "<p>Thank you! Actually, I have read all the Kaggle-related books released in Japan Hahaha.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2403868,
      "author_name": "samuelcortinhas",
      "author_url": "",
      "post_date": "08/22/2023 22:45:32",
      "content": "<p>Thanks for sharing. That mean k-fold strategy is very interesting, I've never heard of it before</p>",
      "votes": null,
      "replies": [
        {
          "id": 2403916,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/23/2023 00:03:24",
          "content": "<p>Thank you! I was considering seed averaging at first, but I thought the randomness would be reduced since I am using multi-label stratified cross-validation. That's when I came up with the idea of averaging with this mean k-fold strategy.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2403964,
      "author_name": "isakatsuyoshi",
      "author_url": "",
      "post_date": "08/23/2023 01:19:10",
      "content": "<p>I shared MultilabelStratifiedKFold code for DEMO.<br>\n<a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805\" target=\"_blank\">https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 2408983,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/26/2023 01:41:22",
          "content": "<p>I am happy to share the notebook for generating the dataset.<br>\n\"Sophisticated Data Preprocessing for the CAFA5 Dataset\"<br>\n<a href=\"https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/434656\" target=\"_blank\">https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/434656</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2405067,
      "author_name": "mtinti",
      "author_url": "",
      "post_date": "08/23/2023 17:05:18",
      "content": "<p>Thanks! A delightful reading.  I wondered if you tried combining your predictions with quickgo or other external predictions/database.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2405194,
          "author_name": "isakatsuyoshi",
          "author_url": "",
          "post_date": "08/23/2023 18:14:44",
          "content": "<p>Thank you! I was really torn between the potential for high accuracy and increasing complexity. In the end, I only combined it with quickgo (which you shared with us!).I'm also curious how other participants have combined external predictions and database.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2402773": "# 1. Public 9th/Private 331st Place Solution -  A Non-Bioinformatician's Journey in CAFA\n\nFirst and foremost, I would like to express my gratitude to my teammates, the organizers, Kaggle's management team, and everyone who participated with us. Through this contest, I have fallen in love with CAFA. I'm looking forward to CAFA6.\nI will introduce our best CV solution. Unlike many top Kagglers, we are not bioinformaticians, so our focus is on machine learning.\n\n# 2. Competition Overview and Impressions\n- This competition is a multi-target task to predict biological functions (multiple) for each protein.\n- An extremely imbalanced dataset where most labels exist less than 1%.\n- A complex competition metric, considering Information Accumulation IA for the maximum F-value.\n- As many participants felt, this competition was challenging. Conversely, hard work can make a difference.\n\n# 3. Data\n\n## 3.1 X(N, M): Explanatory Variables (Embedding Representation & Features)\n- Created data\n  - Embedding representations: ESM2, T5, ProtBERT, SGT, etc.\n  - Features: Physicochemical properties calculated from amino acid sequences using BioPython, Levenshtein distance, Species, time information features obtained from the UniProt database, is_common, annotation scores, etc.\n  - Reduction of embedding representations and features to 3 dimensions using t-SNE/UMAP.\n- Data type: Converted to float32 (to save memory).\n- Preprocessing\n  - Combination of standardization, MinMax transformation, logarithmic transformation, interquartile range transformation, clipping, etc., based on the distribution of each feature.\n  - Concatenation of numerous embedding representations and features for training.\n- Feature selection\n    - Removal of features with std less than 0.1 (excluding originally small values such as amino acid sequence ratios).\n    - Removal of one of the features if corr is 0.95 or higher.\n    - Removal of features with too different distributions between train/test.\n    - Feature removal relying on TabNet importance and experience.\n- Examples of used datasets\n  - X0: tSVD2048(ESM2, T5, Species) + time features from UniProt + annotation score + is_common + UMAP(Species) + UMAP(ProtBERT) + UMAP(Levenshtein distance)\n  - X1: ESM\n  - X2: T5+Species\n\n## 3.2 y(N, K): Target Variables & Labels\n- Updated some y labels using the UniProt database.\n- For groups with identical amino acid sequences, if one is 1, others are also assigned 1.\n- y is int8, labels are converted to \"<U10\" data type (to save memory).\n- GO terms are extracted in the order of IA × frequency, with 1100, 300, 850 items for BP, CC, MF, respectively.\n\n## 3.3 ids: Index\n- Removed one duplicate in test_ids.\n- Sorted train_ids, test_ids alphabetically (to handle all data in the same format!).\n- Indices for multi-label stratified cross-validation are saved in pickle (to save computation time).\n\n# 4. Validation Method\n- In this contest, the validation method was one of the essential elements. In conclusion, we used multi-target stratified cross-validation.\n- Averaged across k=4,5,6 splits in cross-validation. This not only increases robustness but also improves CV by +0.03-0.04!\n- In the competition's IA-weighted maximum F-value, we used 1100, 300, 850 GO terms for BP, CC, MF, respectively, for training, and 5000, 2079, 4609 GO terms for evaluation. This brings us closer to the leaderboard scoring calculated for all GOs.\n- Trials and errors until the exploration of other validation methods\n  - First, we attempted to split by time series. We used the UniProt API to obtain the creation date for each ID. Ultimately, we did not proceed with time-series splitting.\n  - Next, we evaluated the difference between train and test using adversarial validation. LightGBM and XGBoost had strange AUC values below 0.2, but MLP had an AUC of 0.5. This reaffirmed that embedding representations are suitable for MLP. Since the AUC was 0.5, we hypothesized that the distributions of train and test were not significantly different.\n  - Based on the above adversarial validation results, we switched to making cross-validation more robust. Initially, we used random cross-validation.\n  - However, since it's imbalanced data, we switched to multi-label stratified cross-validation. In conclusion, this was the best.\n  - For tuning the model validation system, checks for ① consistency of local tests (no significant variations in errors between folds), ② correlation between local validation scores and leaderboard were necessary. ① is fine. For ②, since the number of IDs used in the public leaderboard was estimated to be around 60, we lowered the priority.\n  - In multi-label stratified cross-validation, we calculated the IA-weighted MSE for the ratio of labels considered, but eventually, we targeted all GOs for stratification.\n\n# 5. Models\nThe models were inspired by the top solutions of [this competition](https://www.kaggle.com/competitions/lish-moa/overview). Additionally, we used basic models for diversity.\n- MLP: Multiple dataset patterns were input to create diverse MLPs.\n- 1D-CNN: Ensemble member.\n- TabNet: Used only for feature selection.\n- knn: Ensemble member.\n- Ridge: Was an ensemble member but the weight became 0.\n\n\nExample of weighted CV highest: Aspect, [MLP(X0), MLP(X1), MLP(X2), knn, 1D-CNN]=CV score\n- BP, [0.8, 0.1, 0.1, 0.3, 0.3]=0.5589730655644398\n- CC, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6453054780984085\n- MF, [0.6, 0.12, 0.16, 0.12, 0.05]=0.6771821551\n\nCorrelation matrix calculated from CC's test predictions (other aspects, train predictions were also able to create diverse models)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5927283%2Fc864fe0de0aeb2a8f15e206b1e01fbb2%2F2023-08-22%20195144.png?generation=1692701663574782&alt=media)",
    "2402896": "Great solution",
    "2402910": "Thank you!",
    "2402968": "Congratulations, it seems to be a very strong solution",
    "2402972": "Thank you for your kind words!",
    "2403341": "Hi!\nThank you for sharing how your solution works, I'm a newbie in the data science field that graduated in biology and I found this competition to be very hard and (obviously) outside of my current capabilities, and I was hoping that participants form the top teams would share how they did to learn more. Your post was super valuable in that regard, thank you for that!",
    "2403353": "Congratulations, It was great !",
    "2403607": "I think it is really good and strong solution from ML side with no bioonfirmatician background.\nI hope you will get your kaggle competition master",
    "2403789": "Great solution, congratulations! It seems that approaching this as a tabular competition with all the tricks and best practices worked out really well :)",
    "2403868": "Thanks for sharing. That mean k-fold strategy is very interesting, I've never heard of it before",
    "2403916": "Thank you! I was considering seed averaging at first, but I thought the randomness would be reduced since I am using multi-label stratified cross-validation. That's when I came up with the idea of averaging with this mean k-fold strategy.",
    "2403918": "Thank you! Actually, I have read all the Kaggle-related books released in Japan Hahaha.",
    "2403919": "Thank you for the warm words! I'm looking forward to seeing the solutions shared by bioinformaticians.",
    "2403922": "Thanks for the encouraging words!",
    "2403924": "Thank you! It was a very difficult competition. Breaking down the difficulties little by little worked well.",
    "2403964": "I shared MultilabelStratifiedKFold code for DEMO.\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433805",
    "2405067": "Thanks! A delightful reading.  I wondered if you tried combining your predictions with quickgo or other external predictions/database.",
    "2405194": "Thank you! I was really torn between the potential for high accuracy and increasing complexity. In the end, I only combined it with quickgo (which you shared with us!).I'm also curious how other participants have combined external predictions and database.",
    "2408983": "I am happy to share the notebook for generating the dataset.\n\"Sophisticated Data Preprocessing for the CAFA5 Dataset\"\nhttps://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/434656"
  },
  "source": "meta"
}