A great adventure is coming to an end for me, and I imagine I'm not the only one. This competition has been an incredible source of personal improvement. I'd like to warmly thank all the organizers for this great research and development project.
\nI would like to thank you all for the great interest you showed for my notebook about ProteiNet (https://www.kaggle.com/code/henriupton/proteinet-pytorch-ems2-t5-protbert-embeddings). Thanks to this, a large number of bugs and defects have been corrected in this new version. Following this, we implemented (me and my teammates Masakazu Sato, Franck Lin) various additions to build its big brother: ProteiNet v2!
\nTraining Notebook : https://www.kaggle.com/code/henriupton/proteinet-v2-training-notebook/notebook
\nInference Notebook : https://www.kaggle.com/code/henriupton/proteinet-v2-inference-notebook/notebook
\nGitHub : https://github.com/henriupton99/proteinet-cafa5
This new version actually aims to train not one, not two, but 3 models, all three specialized in predicting a group of GOs of a particular aspect among the three sets presented for CAFA5: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC). As I'd like to continue exchanging ideas on how to improve my post-competition model in the future, here is an exhaustive list of the most important innovations:
\nRather than training a single model to predict the scores of all GOs for each protein, train three separate models capable of predicting the scores of each protein for GOs of a specific aspect (BPO, MFO, CCO). This is why we call these three models \"experts\". This practice has several theoretical virtues, such as the fact that each model aims to perform multilabel classification on a smaller number of classes. In addition, they are trained to work on GO embeddings that are highly likely to be parent/child, as they come from the same aspect. Once the models have been trained, the predictions of each model will be concatenated to form the final submission.
The GO classes to be predicted are no longer naively the top K of the most frequent GO classes in the database. It has been discussed time and again that various other methods can be used to select GO classes more strategically. In addition, as we train experts based on aspect groups, we define as classes for each model the top K most frequent GOs filtered on aspects. The number of classes per aspect is defined on the basis of these observations: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491
We restrict the GO classes to be predicted to those whose evidence code is Inferred from Experiment (EXP) (and its subgroups).(https://wiki.geneontology.org/index.php/Inferred_from_Experiment_(EXP)) This choice stems from the desire to be as close as possible to the explanations given in the Background section of the Competition Evaluation page: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation
We incorporate the weights given by IA.txt into our Cross Entropy loss function when training the model. Again in an effort to keep up with the competition, the weights enable us to place greater emphasis on infrequent GOs (at the root of the graph), which are consequently the most important ones. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237)(https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)
We have implemented a cross-validation (CV) process to prevent from overfitting. There has long been a consensus that incorporating this method into the pipeline of one's work reduces the risk of overfitting on the public LB and by consequence get a good result on the private LB. The actual CV is composed of 5 folds (so 5 different classic 80-20 train-test splits). (https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-use-k-fold-cross-validation-with-pytorch.md) Once the models perform well for the 5 folds, we train the final models on the full train set thanks to the config hyperparameter VALIDATION_MODE = False.
Once the predictions were formed for the three expert models, we defined a minimal score threshold in order to filter the predictions. For each prediction row, if it does not exceed the threshold, it is deleted from the final submission. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431652) This hyperparameter can be tuned thanks to the constant PROB_THRESHOLD in the config class. It allows to benefit at maximum from the propagation process of GOs predictions. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)
A great adventure is coming to an end for me, and I imagine I'm not the only one. This competition has been an incredible source of personal improvement. I'd like to warmly thank all the organizers for this great research and development project.
\nI would like to thank you all for the great interest you showed for my notebook about ProteiNet (https://www.kaggle.com/code/henriupton/proteinet-pytorch-ems2-t5-protbert-embeddings). Thanks to this, a large number of bugs and defects have been corrected in this new version. Following this, we implemented (me and my teammates Masakazu Sato, Franck Lin) various additions to build its big brother: ProteiNet v2!
\nTraining Notebook : https://www.kaggle.com/code/henriupton/proteinet-v2-training-notebook/notebook
\nInference Notebook : https://www.kaggle.com/code/henriupton/proteinet-v2-inference-notebook/notebook
\nGitHub : https://github.com/henriupton99/proteinet-cafa5
This new version actually aims to train not one, not two, but 3 models, all three specialized in predicting a group of GOs of a particular aspect among the three sets presented for CAFA5: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC). As I'd like to continue exchanging ideas on how to improve my post-competition model in the future, here is an exhaustive list of the most important innovations:
\nRather than training a single model to predict the scores of all GOs for each protein, train three separate models capable of predicting the scores of each protein for GOs of a specific aspect (BPO, MFO, CCO). This is why we call these three models \"experts\". This practice has several theoretical virtues, such as the fact that each model aims to perform multilabel classification on a smaller number of classes. In addition, they are trained to work on GO embeddings that are highly likely to be parent/child, as they come from the same aspect. Once the models have been trained, the predictions of each model will be concatenated to form the final submission.
The GO classes to be predicted are no longer naively the top K of the most frequent GO classes in the database. It has been discussed time and again that various other methods can be used to select GO classes more strategically. In addition, as we train experts based on aspect groups, we define as classes for each model the top K most frequent GOs filtered on aspects. The number of classes per aspect is defined on the basis of these observations: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431491
We restrict the GO classes to be predicted to those whose evidence code is Inferred from Experiment (EXP) (and its subgroups).(https://wiki.geneontology.org/index.php/Inferred_from_Experiment_(EXP)) This choice stems from the desire to be as close as possible to the explanations given in the Background section of the Competition Evaluation page: https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation
We incorporate the weights given by IA.txt into our Cross Entropy loss function when training the model. Again in an effort to keep up with the competition, the weights enable us to place greater emphasis on infrequent GOs (at the root of the graph), which are consequently the most important ones. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/405237)(https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)
We have implemented a cross-validation (CV) process to prevent from overfitting. There has long been a consensus that incorporating this method into the pipeline of one's work reduces the risk of overfitting on the public LB and by consequence get a good result on the private LB. The actual CV is composed of 5 folds (so 5 different classic 80-20 train-test splits). (https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-use-k-fold-cross-validation-with-pytorch.md) Once the models perform well for the 5 folds, we train the final models on the full train set thanks to the config hyperparameter VALIDATION_MODE = False.
Once the predictions were formed for the three expert models, we defined a minimal score threshold in order to filter the predictions. For each prediction row, if it does not exceed the threshold, it is deleted from the final submission. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/431652) This hyperparameter can be tuned thanks to the constant PROB_THRESHOLD in the config class. It allows to benefit at maximum from the propagation process of GOs predictions. (https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/overview/evaluation)