It was an incredible adventure and the first competition I ever took part in! I was very lucky to find an experienced teammate - @ogurtsov , who was the leader of our team. This journey was not just about algorithms and data; it was a journey of exploration, curiosity, and learning, so thanks to the organizers, fellow participants, and the Kaggle community!
\nHere is a brief overview of our 13th-place LB solution:
\n1) Data:
\nWe used the initial train set and also annotations from the UniProt and QuickGO databases (filtered terms with NOT qualifier, evidence code \"ND\", obsolete terms, and those that are not present in the competition OBO file) in two ways: to update the train set (proteins with experimentally confirmed terms only) and to merge with predictions (test proteins with terms never experimentally confirmed).
2) Features
\nESM2 3B protein sequence embeddings kindly shared by @andreylalaley with Z-score normalization (means and SDs of train data applied to train, validation, and test).
\n3) Targets:
\nOur target selection involved selecting the top 500 GO terms based on their frequency, or terms with a frequency above a certain threshold (BP > 250, MF and CC > 50).
4) Neural Networks:
\nThe modeling strategy involved multiple Multi-Layer Perceptrons (MLPs) with different hyperparameters and layers, written with Torch or Tensorflow, R language. Each was trained on the initial or updated training set and different sets of targets. To ensure robustness, we employed cross-validation (CV) with 10 folds (one for validation and 9 for training) followed by averaging the predictions. For some models, we found it more effective to validate on train proteins only, in these cases, we used randomized 10-fold CV to avoid reducing the validation set. Importantly, we trained a separate MLP for each ontology, each time only on these proteins that have at least one term of the selected top.
\nUPD: For all MLPs we used AdamW optimizer, once cycle learning rate and early stopping callback (max 100 epoch, monitored metric - valid loss, with patience 1, 3 or 5) or callback reducing the learning rate by a factor of 0.5 (patience = 1)
\n5) Post-processing:
\nFirst, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. Also, after each step, we made corrections to ensure that the predictions of parent terms were not smaller than the predictions of child terms (in such cases we propagated max prediction of child terms). Finally, before the final merge with GO annotations, the predictions were filtered again to include only terms in the top 450 by frequency based on the original training set and also rare terms if they had predictions > 0.5.
6) Ensemble:
\nPredictions from 6 MLPs were simply averaged. The resulting predictions were averaged again with exported GO annotations (the set of test proteins with terms never experimentally confirmed).
For the secondary solution, we did weighted averaging with GO annotations and selected the top 35 predictions for each ontology.
\nI hope that this will be beneficial for others, serving as an example of what can be done (if we shake up) or what should be avoided (if we shake down at the end) 😆.
", "messages": [ { "id": "2407129", "postDate": "08/24/2023 20:55:16", "content": "It was an incredible adventure and the first competition I ever took part in! I was very lucky to find an experienced teammate - @ogurtsov , who was the leader of our team. This journey was not just about algorithms and data; it was a journey of exploration, curiosity, and learning, so thanks to the organizers, fellow participants, and the Kaggle community!
\nHere is a brief overview of our 13th-place LB solution:
\n1) Data:
\nWe used the initial train set and also annotations from the UniProt and QuickGO databases (filtered terms with NOT qualifier, evidence code \"ND\", obsolete terms, and those that are not present in the competition OBO file) in two ways: to update the train set (proteins with experimentally confirmed terms only) and to merge with predictions (test proteins with terms never experimentally confirmed).
2) Features
\nESM2 3B protein sequence embeddings kindly shared by @andreylalaley with Z-score normalization (means and SDs of train data applied to train, validation, and test).
\n3) Targets:
\nOur target selection involved selecting the top 500 GO terms based on their frequency, or terms with a frequency above a certain threshold (BP > 250, MF and CC > 50).
4) Neural Networks:
\nThe modeling strategy involved multiple Multi-Layer Perceptrons (MLPs) with different hyperparameters and layers, written with Torch or Tensorflow, R language. Each was trained on the initial or updated training set and different sets of targets. To ensure robustness, we employed cross-validation (CV) with 10 folds (one for validation and 9 for training) followed by averaging the predictions. For some models, we found it more effective to validate on train proteins only, in these cases, we used randomized 10-fold CV to avoid reducing the validation set. Importantly, we trained a separate MLP for each ontology, each time only on these proteins that have at least one term of the selected top.
\nUPD: For all MLPs we used AdamW optimizer, once cycle learning rate and early stopping callback (max 100 epoch, monitored metric - valid loss, with patience 1, 3 or 5) or callback reducing the learning rate by a factor of 0.5 (patience = 1)
\n5) Post-processing:
\nFirst, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. Also, after each step, we made corrections to ensure that the predictions of parent terms were not smaller than the predictions of child terms (in such cases we propagated max prediction of child terms). Finally, before the final merge with GO annotations, the predictions were filtered again to include only terms in the top 450 by frequency based on the original training set and also rare terms if they had predictions > 0.5.
6) Ensemble:
\nPredictions from 6 MLPs were simply averaged. The resulting predictions were averaged again with exported GO annotations (the set of test proteins with terms never experimentally confirmed).
For the secondary solution, we did weighted averaging with GO annotations and selected the top 35 predictions for each ontology.
\nI hope that this will be beneficial for others, serving as an example of what can be done (if we shake up) or what should be avoided (if we shake down at the end) 😆.
", "rawMarkdown": "It was an incredible adventure and the first competition I ever took part in! I was very lucky to find an experienced teammate - @ogurtsov , who was the leader of our team. This journey was not just about algorithms and data; it was a journey of exploration, curiosity, and learning, so thanks to the organizers, fellow participants, and the Kaggle community!\n\nHere is a brief overview of our 13th-place LB solution:\n\n**1) Data:**\nWe used the initial train set and also annotations from the UniProt and QuickGO databases (filtered terms with NOT qualifier, evidence code \"ND\", obsolete terms, and those that are not present in the competition OBO file) in two ways: to update the train set (proteins with experimentally confirmed terms only) and to merge with predictions (test proteins with terms never experimentally confirmed).\n\n**2) Features**\n\nESM2 3B protein sequence embeddings kindly shared by @andreylalaley with Z-score normalization (means and SDs of train data applied to train, validation, and test).\n\n**3) Targets:**\nOur target selection involved selecting the top 500 GO terms based on their frequency, or terms with a frequency above a certain threshold (BP > 250, MF and CC > 50).\n\n**4) Neural Networks:**\n\nThe modeling strategy involved multiple Multi-Layer Perceptrons (MLPs) with different hyperparameters and layers, written with Torch or Tensorflow, R language. Each was trained on the initial or updated training set and different sets of targets. To ensure robustness, we employed cross-validation (CV) with 10 folds (one for validation and 9 for training) followed by averaging the predictions. For some models, we found it more effective to validate on train proteins only, in these cases, we used randomized 10-fold CV to avoid reducing the validation set. Importantly, we trained a separate MLP for each ontology, each time only on these proteins that have at least one term of the selected top.\n\nUPD: For all MLPs we used AdamW optimizer, [once cycle learning rate](https://torch.mlverse.org/docs/reference/lr_one_cycle.html) and early stopping callback (max 100 epoch, monitored metric - valid loss, with patience 1, 3 or 5) or [callback reducing the learning rate](https://rdrr.io/cran/keras/man/callback_reduce_lr_on_plateau.html) by a factor of 0.5 (patience = 1)\n\n**5) Post-processing:**\nFirst, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. Also, after each step, we made corrections to ensure that the predictions of parent terms were not smaller than the predictions of child terms (in such cases we propagated max prediction of child terms). Finally, before the final merge with GO annotations, the predictions were filtered again to include only terms in the top 450 by frequency based on the original training set and also rare terms if they had predictions > 0.5.\n\n**6) Ensemble:**\nPredictions from 6 MLPs were simply averaged. The resulting predictions were averaged again with exported GO annotations (the set of test proteins with terms never experimentally confirmed).\n\nFor the secondary solution, we did weighted averaging with GO annotations and selected the top 35 predictions for each ontology.\n\nI hope that this will be beneficial for others, serving as an example of what can be done (if we shake up) or what should be avoided (if we shake down at the end) 😆.", "votes": null }, { "id": "2407155", "postDate": "08/24/2023 21:35:39", "content": "Wow! That's a proper machine learning approach that is independent of ensembling with best public submission's \".tsv's\" I have seen so far.
", "rawMarkdown": "Wow! That's a proper machine learning approach that is independent of ensembling with best public submission's \".tsv's\" I have seen so far.", "votes": null }, { "id": "2410533", "postDate": "08/27/2023 04:06:15", "content": "Thanks for your sharing antoninadolgorukova! Your post-processing is so impressive!
", "rawMarkdown": "Thanks for your sharing antoninadolgorukova! Your post-processing is so impressive!", "votes": null }, { "id": "3336415", "postDate": "11/18/2025 13:32:35", "content": "Hello, could I ask a question:First, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. How you get the taxon constraints
", "rawMarkdown": "Hello, could I ask a question:First, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. How you get the taxon constraints", "votes": null } ], "comments": [ { "id": 2407155, "author_name": "victorogobi", "author_url": "", "post_date": "08/24/2023 21:35:39", "content": "Wow! That's a proper machine learning approach that is independent of ensembling with best public submission's \".tsv's\" I have seen so far.
", "votes": null, "replies": [] }, { "id": 2410533, "author_name": "songqizhou", "author_url": "", "post_date": "08/27/2023 04:06:15", "content": "Thanks for your sharing antoninadolgorukova! Your post-processing is so impressive!
", "votes": null, "replies": [] }, { "id": 3336415, "author_name": "aizealotwu", "author_url": "", "post_date": "11/18/2025 13:32:35", "content": "Hello, could I ask a question:First, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. How you get the taxon constraints
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2407129": "It was an incredible adventure and the first competition I ever took part in! I was very lucky to find an experienced teammate - @ogurtsov , who was the leader of our team. This journey was not just about algorithms and data; it was a journey of exploration, curiosity, and learning, so thanks to the organizers, fellow participants, and the Kaggle community!\n\nHere is a brief overview of our 13th-place LB solution:\n\n**1) Data:**\nWe used the initial train set and also annotations from the UniProt and QuickGO databases (filtered terms with NOT qualifier, evidence code \"ND\", obsolete terms, and those that are not present in the competition OBO file) in two ways: to update the train set (proteins with experimentally confirmed terms only) and to merge with predictions (test proteins with terms never experimentally confirmed).\n\n**2) Features**\n\nESM2 3B protein sequence embeddings kindly shared by @andreylalaley with Z-score normalization (means and SDs of train data applied to train, validation, and test).\n\n**3) Targets:**\nOur target selection involved selecting the top 500 GO terms based on their frequency, or terms with a frequency above a certain threshold (BP > 250, MF and CC > 50).\n\n**4) Neural Networks:**\n\nThe modeling strategy involved multiple Multi-Layer Perceptrons (MLPs) with different hyperparameters and layers, written with Torch or Tensorflow, R language. Each was trained on the initial or updated training set and different sets of targets. To ensure robustness, we employed cross-validation (CV) with 10 folds (one for validation and 9 for training) followed by averaging the predictions. For some models, we found it more effective to validate on train proteins only, in these cases, we used randomized 10-fold CV to avoid reducing the validation set. Importantly, we trained a separate MLP for each ontology, each time only on these proteins that have at least one term of the selected top.\n\nUPD: For all MLPs we used AdamW optimizer, [once cycle learning rate](https://torch.mlverse.org/docs/reference/lr_one_cycle.html) and early stopping callback (max 100 epoch, monitored metric - valid loss, with patience 1, 3 or 5) or [callback reducing the learning rate](https://rdrr.io/cran/keras/man/callback_reduce_lr_on_plateau.html) by a factor of 0.5 (patience = 1)\n\n**5) Post-processing:**\nFirst, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. Also, after each step, we made corrections to ensure that the predictions of parent terms were not smaller than the predictions of child terms (in such cases we propagated max prediction of child terms). Finally, before the final merge with GO annotations, the predictions were filtered again to include only terms in the top 450 by frequency based on the original training set and also rare terms if they had predictions > 0.5.\n\n**6) Ensemble:**\nPredictions from 6 MLPs were simply averaged. The resulting predictions were averaged again with exported GO annotations (the set of test proteins with terms never experimentally confirmed).\n\nFor the secondary solution, we did weighted averaging with GO annotations and selected the top 35 predictions for each ontology.\n\nI hope that this will be beneficial for others, serving as an example of what can be done (if we shake up) or what should be avoided (if we shake down at the end) 😆.", "2407155": "Wow! That's a proper machine learning approach that is independent of ensembling with best public submission's \".tsv's\" I have seen so far.", "2410533": "Thanks for your sharing antoninadolgorukova! Your post-processing is so impressive!", "3336415": "Hello, could I ask a question:First, we filtered the predictions using taxon constraints to eliminate terms incompatible with certain species. How you get the taxon constraints" }, "source": "meta" }