Firstly thanks to the organizers for this amazing competition and the huge work done to make it possible! I am quite new to the protein analysis field, and it was exciting to work on such a challenging problem and learn more about it. Also many thanks to the community for the valuable discussions and great solution ideas they shared.
\nI was really inspired by the HoendorfLab work on the DeepGOZero model (https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515) and spent quite a lot of time tuning it for this competition.
\nAs I got from the paper, the original approach was to embed GO terms into an n-d space as multidimensional balls by learning for each GO term an n-d center vector and a (1d) radius. The positions and sizes of the balls are learned in the way that they reflect real relationships in the GO ontology (for example, if A is a subclass of B, the sphere of A will lie inside the sphere of B). Then, the protein is mapped to the same embedding space and classified based on its position wrt GO term balls.
\nIn the original architecture (published on GitHub https://github.com/bio-ontology-research-group/deepgozero/blob/main/deepgozero.py), the probability of a protein belonging to the class C is computed as sigmoid(p @ (f + c).T + r_c) where p is learned protein embedding, c and r_c are center vector and radius of class C ball, and f is a relation in the ontology, also a learned vector.
\nTo be honest, I didn’t manage to understand why this function on the C center and radius best reflects the probability of protein embedding belonging to class C (I would greatly appreciate it if the HoendorfLab team could elaborate on this).
\nThis original approach already performed very well being even naively applied to the competition data (with T5 embeddings as protein input features) - 0.499 LB.
After experiments with several different approaches to obtain classification probabilities, I omitted the direct mapping of the protein into the same space as GO term embeddings. Instead, I learned GO term embeddings with the same set of losses proposed by authors and applied attention-like mapping to transform each GO term embedding into a weight vector with weights summing to 1. The probability that protein p belongs to class C is then calculated as sigmoid(p @ attn(f + c) + r_c).
\nI used T5 embedding as input features of the protein. I also added taxonomy_id for 48 most common species in the training set (+ taxonomy_id -1 for all the rest) and learned embeddings for their representation inside the model. Generally in my experiments, adding taxonomy_id information improved LB score by 0.01-0.02.
\nThis model gave 0.537 LB.
I extended the training set provided by the organizers by parsing the GOA UniProt annotations. It contains annotations for >120k proteins out of 142k in the training set, and these annotations can be wider because they are not filtered by evidence codes.
\nI also filtered them by a larger set of evidence codes, including the annotations that were inferred based on computational analysis and some others. Since annotations with different evidences may have different reliability, I assigned them reliability scores from 0 to 1. For example, annotation inferred from lab experiment (highly reliable) results in the target class probability 1, while annotation inferred from computational analysis (less reliable) results in the target class probability of only 0.4. This improved performance to 0.56 LB.
I also parsed GOA UniProt annotations for the test set following the same logic and ensembled them with the model predictions using the following procedure:
\n1) for each GO term, take an average of the model-predicted probability and GOA reliability score
\n2) if the resulted s score is less than predicted by the model, keep the predicted score.
\nIt gave 0.601 LB. (Unfortunately, I think this ensembling is a kind of overfitting since I directly use existing labels for the test set)
Also, I didn’t have time to train the model on different CV splits, and my final submission only uses a single model. Probably ensembling models from different splits could improve overall model robustness and LB score.
\nI have read through several great solutions already posted and regret a bit that I haven't spent more time experimenting with different protein embeddings like ESM or fine-tuning T5 embeddings for this competition. However, the idea of learning ontology-informed embeddings proposed in DeepGoZero was so beautiful that it captured all my attention 😅
", "messages": [ { "id": "2404560", "postDate": "08/23/2023 10:10:02", "content": "Firstly thanks to the organizers for this amazing competition and the huge work done to make it possible! I am quite new to the protein analysis field, and it was exciting to work on such a challenging problem and learn more about it. Also many thanks to the community for the valuable discussions and great solution ideas they shared.
\nI was really inspired by the HoendorfLab work on the DeepGOZero model (https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515) and spent quite a lot of time tuning it for this competition.
\nAs I got from the paper, the original approach was to embed GO terms into an n-d space as multidimensional balls by learning for each GO term an n-d center vector and a (1d) radius. The positions and sizes of the balls are learned in the way that they reflect real relationships in the GO ontology (for example, if A is a subclass of B, the sphere of A will lie inside the sphere of B). Then, the protein is mapped to the same embedding space and classified based on its position wrt GO term balls.
\nIn the original architecture (published on GitHub https://github.com/bio-ontology-research-group/deepgozero/blob/main/deepgozero.py), the probability of a protein belonging to the class C is computed as sigmoid(p @ (f + c).T + r_c) where p is learned protein embedding, c and r_c are center vector and radius of class C ball, and f is a relation in the ontology, also a learned vector.
\nTo be honest, I didn’t manage to understand why this function on the C center and radius best reflects the probability of protein embedding belonging to class C (I would greatly appreciate it if the HoendorfLab team could elaborate on this).
\nThis original approach already performed very well being even naively applied to the competition data (with T5 embeddings as protein input features) - 0.499 LB.
After experiments with several different approaches to obtain classification probabilities, I omitted the direct mapping of the protein into the same space as GO term embeddings. Instead, I learned GO term embeddings with the same set of losses proposed by authors and applied attention-like mapping to transform each GO term embedding into a weight vector with weights summing to 1. The probability that protein p belongs to class C is then calculated as sigmoid(p @ attn(f + c) + r_c).
\nI used T5 embedding as input features of the protein. I also added taxonomy_id for 48 most common species in the training set (+ taxonomy_id -1 for all the rest) and learned embeddings for their representation inside the model. Generally in my experiments, adding taxonomy_id information improved LB score by 0.01-0.02.
\nThis model gave 0.537 LB.
I extended the training set provided by the organizers by parsing the GOA UniProt annotations. It contains annotations for >120k proteins out of 142k in the training set, and these annotations can be wider because they are not filtered by evidence codes.
\nI also filtered them by a larger set of evidence codes, including the annotations that were inferred based on computational analysis and some others. Since annotations with different evidences may have different reliability, I assigned them reliability scores from 0 to 1. For example, annotation inferred from lab experiment (highly reliable) results in the target class probability 1, while annotation inferred from computational analysis (less reliable) results in the target class probability of only 0.4. This improved performance to 0.56 LB.
I also parsed GOA UniProt annotations for the test set following the same logic and ensembled them with the model predictions using the following procedure:
\n1) for each GO term, take an average of the model-predicted probability and GOA reliability score
\n2) if the resulted s score is less than predicted by the model, keep the predicted score.
\nIt gave 0.601 LB. (Unfortunately, I think this ensembling is a kind of overfitting since I directly use existing labels for the test set)
Also, I didn’t have time to train the model on different CV splits, and my final submission only uses a single model. Probably ensembling models from different splits could improve overall model robustness and LB score.
\nI have read through several great solutions already posted and regret a bit that I haven't spent more time experimenting with different protein embeddings like ESM or fine-tuning T5 embeddings for this competition. However, the idea of learning ontology-informed embeddings proposed in DeepGoZero was so beautiful that it captured all my attention 😅
", "rawMarkdown": "Firstly thanks to the organizers for this amazing competition and the huge work done to make it possible! I am quite new to the protein analysis field, and it was exciting to work on such a challenging problem and learn more about it. Also many thanks to the community for the valuable discussions and great solution ideas they shared.\n\nI was really inspired by the HoendorfLab work on the DeepGOZero model (https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515) and spent quite a lot of time tuning it for this competition.\n\n### Original approach\nAs I got from the paper, the original approach was to embed GO terms into an n-d space as multidimensional balls by learning for each GO term an n-d center vector and a (1d) radius. The positions and sizes of the balls are learned in the way that they reflect real relationships in the GO ontology (for example, if A is a subclass of B, the sphere of A will lie inside the sphere of B). Then, the protein is mapped to the same embedding space and classified based on its position wrt GO term balls. \nIn the original architecture (published on GitHub https://github.com/bio-ontology-research-group/deepgozero/blob/main/deepgozero.py), the probability of a protein belonging to the class C is computed as `sigmoid(p @ (f + c).T + r_c)` where `p` is learned protein embedding, `c` and `r_c` are center vector and radius of class C ball, and `f` is a relation in the ontology, also a learned vector. \nTo be honest, I didn’t manage to understand why this function on the C center and radius best reflects the probability of protein embedding belonging to class C (I would greatly appreciate it if the HoendorfLab team could elaborate on this). \nThis original approach already performed very well being even naively applied to the competition data (with T5 embeddings as protein input features) - **0.499 LB**. \n\n### Model\nAfter experiments with several different approaches to obtain classification probabilities, I omitted the direct mapping of the protein into the same space as GO term embeddings. Instead, I learned GO term embeddings with the same set of losses proposed by authors and applied attention-like mapping to transform each GO term embedding into a weight vector with weights summing to 1. The probability that protein p belongs to class C is then calculated as `sigmoid(p @ attn(f + c) + r_c)`. \nI used T5 embedding as input features of the protein. I also added taxonomy_id for 48 most common species in the training set (+ taxonomy_id `-1` for all the rest) and learned embeddings for their representation inside the model. Generally in my experiments, adding taxonomy_id information improved LB score by 0.01-0.02. \nThis model gave **0.537 LB**.\n\n### Data augmentation\nI extended the training set provided by the organizers by parsing the GOA UniProt annotations. It contains annotations for >120k proteins out of 142k in the training set, and these annotations can be wider because they are not filtered by evidence codes. \nI also filtered them by a larger set of evidence codes, including the annotations that were inferred based on computational analysis and some others. Since annotations with different evidences may have different reliability, I assigned them reliability scores from 0 to 1. For example, annotation inferred from lab experiment (highly reliable) results in the target class probability `1`, while annotation inferred from computational analysis (less reliable) results in the target class probability of only `0.4`. This improved performance to **0.56 LB**.\n\n### Ensembling with GOA UniProt annotations \nI also parsed GOA UniProt annotations for the test set following the same logic and ensembled them with the model predictions using the following procedure: \n1) for each GO term, take an average of the model-predicted probability and GOA reliability score \n2) if the resulted s score is less than predicted by the model, keep the predicted score. \nIt gave **0.601 LB**. (Unfortunately, I think this ensembling is a kind of overfitting since I directly use existing labels for the test set)\n\nAlso, I didn’t have time to train the model on different CV splits, and my final submission only uses a single model. Probably ensembling models from different splits could improve overall model robustness and LB score.\n\nI have read through several great solutions already posted and regret a bit that I haven't spent more time experimenting with different protein embeddings like ESM or fine-tuning T5 embeddings for this competition. However, the idea of learning ontology-informed embeddings proposed in DeepGoZero was so beautiful that it captured all my attention 😅", "votes": null }, { "id": "2405591", "postDate": "08/24/2023 02:18:20", "content": "The \"Data augmentation\" part is also very important and valuable, maybe you can create a dataset or notebook about it?
\nI think your solution is robust enough to survive on the private LB. Congratulations in advance!
", "rawMarkdown": "The \"Data augmentation\" part is also very important and valuable, maybe you can create a dataset or notebook about it?\n\nI think your solution is robust enough to survive on the private LB. Congratulations in advance!", "votes": null }, { "id": "2406376", "postDate": "08/24/2023 11:49:11", "content": "Thank you for the kind words. Yes, data augmentation actually gave unexpected (for me) boost both for this model and other DNN models I tried
\nI definitely will share both dataset and code I used to process it (in couple days, I need to rest a bit😄) and add here the links
Thank you very much
", "rawMarkdown": "Thank you very much", "votes": null } ], "comments": [ { "id": 2405591, "author_name": "awater1223", "author_url": "", "post_date": "08/24/2023 02:18:20", "content": "The \"Data augmentation\" part is also very important and valuable, maybe you can create a dataset or notebook about it?
\nI think your solution is robust enough to survive on the private LB. Congratulations in advance!
", "votes": null, "replies": [ { "id": 2406376, "author_name": "aucrock", "author_url": "", "post_date": "08/24/2023 11:49:11", "content": "Thank you for the kind words. Yes, data augmentation actually gave unexpected (for me) boost both for this model and other DNN models I tried
\nI definitely will share both dataset and code I used to process it (in couple days, I need to rest a bit😄) and add here the links
Thank you very much
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2404560": "Firstly thanks to the organizers for this amazing competition and the huge work done to make it possible! I am quite new to the protein analysis field, and it was exciting to work on such a challenging problem and learn more about it. Also many thanks to the community for the valuable discussions and great solution ideas they shared.\n\nI was really inspired by the HoendorfLab work on the DeepGOZero model (https://academic.oup.com/bioinformatics/article/38/Supplement_1/i238/6617515) and spent quite a lot of time tuning it for this competition.\n\n### Original approach\nAs I got from the paper, the original approach was to embed GO terms into an n-d space as multidimensional balls by learning for each GO term an n-d center vector and a (1d) radius. The positions and sizes of the balls are learned in the way that they reflect real relationships in the GO ontology (for example, if A is a subclass of B, the sphere of A will lie inside the sphere of B). Then, the protein is mapped to the same embedding space and classified based on its position wrt GO term balls. \nIn the original architecture (published on GitHub https://github.com/bio-ontology-research-group/deepgozero/blob/main/deepgozero.py), the probability of a protein belonging to the class C is computed as `sigmoid(p @ (f + c).T + r_c)` where `p` is learned protein embedding, `c` and `r_c` are center vector and radius of class C ball, and `f` is a relation in the ontology, also a learned vector. \nTo be honest, I didn’t manage to understand why this function on the C center and radius best reflects the probability of protein embedding belonging to class C (I would greatly appreciate it if the HoendorfLab team could elaborate on this). \nThis original approach already performed very well being even naively applied to the competition data (with T5 embeddings as protein input features) - **0.499 LB**. \n\n### Model\nAfter experiments with several different approaches to obtain classification probabilities, I omitted the direct mapping of the protein into the same space as GO term embeddings. Instead, I learned GO term embeddings with the same set of losses proposed by authors and applied attention-like mapping to transform each GO term embedding into a weight vector with weights summing to 1. The probability that protein p belongs to class C is then calculated as `sigmoid(p @ attn(f + c) + r_c)`. \nI used T5 embedding as input features of the protein. I also added taxonomy_id for 48 most common species in the training set (+ taxonomy_id `-1` for all the rest) and learned embeddings for their representation inside the model. Generally in my experiments, adding taxonomy_id information improved LB score by 0.01-0.02. \nThis model gave **0.537 LB**.\n\n### Data augmentation\nI extended the training set provided by the organizers by parsing the GOA UniProt annotations. It contains annotations for >120k proteins out of 142k in the training set, and these annotations can be wider because they are not filtered by evidence codes. \nI also filtered them by a larger set of evidence codes, including the annotations that were inferred based on computational analysis and some others. Since annotations with different evidences may have different reliability, I assigned them reliability scores from 0 to 1. For example, annotation inferred from lab experiment (highly reliable) results in the target class probability `1`, while annotation inferred from computational analysis (less reliable) results in the target class probability of only `0.4`. This improved performance to **0.56 LB**.\n\n### Ensembling with GOA UniProt annotations \nI also parsed GOA UniProt annotations for the test set following the same logic and ensembled them with the model predictions using the following procedure: \n1) for each GO term, take an average of the model-predicted probability and GOA reliability score \n2) if the resulted s score is less than predicted by the model, keep the predicted score. \nIt gave **0.601 LB**. (Unfortunately, I think this ensembling is a kind of overfitting since I directly use existing labels for the test set)\n\nAlso, I didn’t have time to train the model on different CV splits, and my final submission only uses a single model. Probably ensembling models from different splits could improve overall model robustness and LB score.\n\nI have read through several great solutions already posted and regret a bit that I haven't spent more time experimenting with different protein embeddings like ESM or fine-tuning T5 embeddings for this competition. However, the idea of learning ontology-informed embeddings proposed in DeepGoZero was so beautiful that it captured all my attention 😅", "2405591": "The \"Data augmentation\" part is also very important and valuable, maybe you can create a dataset or notebook about it?\n\nI think your solution is robust enough to survive on the private LB. Congratulations in advance!", "2406376": "Thank you for the kind words. Yes, data augmentation actually gave unexpected (for me) boost both for this model and other DNN models I tried\nI definitely will share both dataset and code I used to process it (in couple days, I need to rest a bit😄) and add here the links", "2415145": "Thank you very much" }, "source": "meta" }