I wonder what score can be achieved using only sequence information - embeddings, k-mers, whatever ?
\nBut NOT using the things like \"quick-go\", \"taxons\" all the other things.
\nI think the question is of some importance for research since for some completely new proteins we have nothing else except the structural information.
\nPS
\nFrom what I know:
\nLB Public Score 0.53605 :
\nFrom here:
\n\nThe model - blend of Ridge, NN with skip connections proposed by Andrey Shevtsov (3 random seeds),
\nand sklearn MLP (3 random seeds). Concatenation of t5 and esm2-size1280 embeddings is used. Targets: 1100, 450,300,
\ndownsampling of train - 43k proteins which have all 3 annotations - ideas proposed by Zoltan.
PS
\nThe notebook originates from my own notebook with suitable choice of models.
", "messages": [ { "id": "2403115", "postDate": "08/22/2023 14:04:54", "content": "I wonder what score can be achieved using only sequence information - embeddings, k-mers, whatever ?
\nBut NOT using the things like \"quick-go\", \"taxons\" all the other things.
\nI think the question is of some importance for research since for some completely new proteins we have nothing else except the structural information.
\nPS
\nFrom what I know:
\nLB Public Score 0.53605 :
\nFrom here:
\n\nThe model - blend of Ridge, NN with skip connections proposed by Andrey Shevtsov (3 random seeds),
\nand sklearn MLP (3 random seeds). Concatenation of t5 and esm2-size1280 embeddings is used. Targets: 1100, 450,300,
\ndownsampling of train - 43k proteins which have all 3 annotations - ideas proposed by Zoltan.
PS
\nThe notebook originates from my own notebook with suitable choice of models.
", "rawMarkdown": "I wonder what score can be achieved using only sequence information - embeddings, k-mers, whatever ? \n\nBut NOT using the things like \"quick-go\", \"taxons\" all the other things.\n\nI think the question is of some importance for research since for some completely new proteins we have nothing else except the structural information.\n\n\nPS\n\nFrom what I know: \n\nLB Public Score 0.53605 :\n\nFrom here:\n\n V5: https://www.kaggle.com/code/adaluodao/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139934127\n\nThe model - blend of Ridge, NN with skip connections proposed by Andrey Shevtsov (3 random seeds),\nand sklearn MLP (3 random seeds). Concatenation of t5 and esm2-size1280 embeddings is used. Targets: 1100, 450,300, \ndownsampling of train - 43k proteins which have all 3 annotations - ideas proposed by Zoltan. \n\nPS\n\nThe notebook originates from my own notebook with suitable choice of models.", "votes": null }, { "id": "2403178", "postDate": "08/22/2023 14:42:03", "content": "I got LB = 0.539, CV =0.55. with a T5 and ESM ensemble. When adding SPROF-GO it improves to 0.55 in LB.
\nMy best solution in CV (CV=0.59), which includes knn and models trained with different number of labels, gets only 0.53 in LB.
\nI've done random CV deleting repeatet sequences and also sequences with big PIDE. When deleting sequences with big PIDE knn solution is the best in CV.
This models are boosted in LB from 0.55 to ~0.62 when combining with quickgo this way: if a pair (root, term_id) is in quickgo, use only quickgo labels, in another case use top 90 labels for each pair(root, term_id) from my ensemble. But this tricks wont do nothing in private LB since I guess this pairs (root, term_id) that are in quickGo won´t be in private test set.
", "rawMarkdown": "I got LB = 0.539, CV =0.55. with a T5 and ESM ensemble. When adding SPROF-GO it improves to 0.55 in LB. \nMy best solution in CV (CV=0.59), which includes knn and models trained with different number of labels, gets only 0.53 in LB. \nI've done random CV deleting repeatet sequences and also sequences with big PIDE. When deleting sequences with big PIDE knn solution is the best in CV.\n\nThis models are boosted in LB from 0.55 to ~0.62 when combining with quickgo this way: if a pair (root, term_id) is in quickgo, use only quickgo labels, in another case use top 90 labels for each pair(root, term_id) from my ensemble. But this tricks wont do nothing in private LB since I guess this pairs (root, term_id) that are in quickGo won´t be in private test set.", "votes": null }, { "id": "2403242", "postDate": "08/22/2023 15:17:11", "content": "I was able to get a good score (0.553 LB) with EMS2_3B + PCA of 1500 + MLP with dropouts + small modifications with the optimizer (AdamW) and adding more labels to the training (my best solution had 3000).
\nAlso I've tried the quick-go annotations that gave me a 0.58 LB, but I didn't select that submission.
", "rawMarkdown": "I was able to get a good score (0.553 LB) with EMS2_3B + PCA of 1500 + MLP with dropouts + small modifications with the optimizer (AdamW) and adding more labels to the training (my best solution had 3000). \n\nAlso I've tried the quick-go annotations that gave me a 0.58 LB, but I didn't select that submission.", "votes": null }, { "id": "2403275", "postDate": "08/22/2023 15:31:43", "content": "Thanks a lot!
\nDo not you know sprof-go uses only structural information or not?
if it's the public sprof-go I created, it uses only protein sequences.
", "rawMarkdown": "if it's the public sprof-go I created, it uses only protein sequences.", "votes": null }, { "id": "2403305", "postDate": "08/22/2023 15:50:36", "content": "Yes, it is. Thanks @mtinti
", "rawMarkdown": "Yes, it is. Thanks @mtinti", "votes": null }, { "id": "2403317", "postDate": "08/22/2023 15:56:03", "content": "Interesting question! Our best ensemble of multiple MLPs with R (torch, tensorflow) and10-fold CV gave 0.56425 LB.
\nWe used ESM 3B, predicted top 500 targets or top by frequency with more than 250 observations for BPO and 50 for MFO and CCO, separately for each ontology. I think we will publish the model variations we used soon.
", "rawMarkdown": "Interesting question! Our best ensemble of multiple MLPs with R (torch, tensorflow) and10-fold CV gave 0.56425 LB.\n\nWe used ESM 3B, predicted top 500 targets or top by frequency with more than 250 observations for BPO and 50 for MFO and CCO, separately for each ontology. I think we will publish the model variations we used soon.", "votes": null }, { "id": "2403355", "postDate": "08/22/2023 16:37:05", "content": "Sorry what \"root\" means ?
\nIn the phrase \"if a pair (root, term_id)….
Sorry with root i mean subontology.
\nFor example, if a protein has quickgo annotations only in BPO, in that protein I only use my ensemble predictions in MFO and CCO labels, and in BPO i use quickgo labels with prob=1.
\nI also checked whats happens not adding my predictions in any subontology if a protein has annotations in one of them or more in quickgo. That way in the example I wouldnt add my ensemble predictions in MFO and CCO either, that protein would only have BPO quickgo labels with prob=1. Doing this my score drops from 0.62 to 0.58, which means that for that proteins that appear in quickgo it is better to add my predictions in subontologies not present in quickgo, at least in LB samples.
I got LB 0.537 with T5 embeddings using a DeepGOZero model developed by Robert Hoehndorf research group (https://pubmed.ncbi.nlm.nih.gov/35758802/) with slight custom modifications. It is a single model without any ensembling trained on T5 embeddings.
\n(UPD: sorry, I checked, and in this model I have also incorporated species information. It tended to only slightly improve LB performance for other my models, though I haven’t checked how large is the improvement for this particular one).
In my experiments, adding more GO terms only improved the performance of DeepGOZero model. This one was trained on ~4500, 6000 and 3200 terms from MFO, BPO and CCO subontologies respectively.
\nI also tried to augment the training set (not the test set) with additional annotations from GOA UniProt. In this case, the same model yielded 0.56 LB
", "rawMarkdown": "I got LB 0.537 with T5 embeddings using a DeepGOZero model developed by Robert Hoehndorf research group (https://pubmed.ncbi.nlm.nih.gov/35758802/) with slight custom modifications. It is a single model without any ensembling trained on T5 embeddings.\n(UPD: sorry, I checked, and in this model I have also incorporated species information. It tended to only slightly improve LB performance for other my models, though I haven’t checked how large is the improvement for this particular one). \n\nIn my experiments, adding more GO terms only improved the performance of DeepGOZero model. This one was trained on ~4500, 6000 and 3200 terms from MFO, BPO and CCO subontologies respectively. \n\nI also tried to augment the training set (not the test set) with additional annotations from GOA UniProt. In this case, the same model yielded 0.56 LB", "votes": null }, { "id": "2403609", "postDate": "08/22/2023 18:37:11", "content": "I got LB 0.57867 using the 3 last layers from T5 as embeddings with a simple DNN model with tunning of the loss function and training data. That is my best LB single-component method.
", "rawMarkdown": "I got LB 0.57867 using the 3 last layers from T5 as embeddings with a simple DNN model with tunning of the loss function and training data. That is my best LB single-component method.", "votes": null }, { "id": "2403647", "postDate": "08/22/2023 19:07:21", "content": "Wow ! Can you share your code please ?
", "rawMarkdown": "Wow ! Can you share your code please ?", "votes": null }, { "id": "2403663", "postDate": "08/22/2023 19:23:32", "content": "I think my solution qualifies as sequence-only. My best individual model (0.547 on public LB) is T5 embeddings + RPS-BLAST search results. For the latter part, I used custom profile databases (COG, KOG, SMART, TIGRFams v15, Pfam v35). All folded together into a single Keras network.
", "rawMarkdown": "I think my solution qualifies as sequence-only. My best individual model (0.547 on public LB) is T5 embeddings + RPS-BLAST search results. For the latter part, I used custom profile databases (COG, KOG, SMART, TIGRFams v15, Pfam v35). All folded together into a single Keras network.", "votes": null }, { "id": "2403769", "postDate": "08/22/2023 20:55:10", "content": "I second that please, its a really interesting model!
", "rawMarkdown": "I second that please, its a really interesting model!", "votes": null }, { "id": "2404058", "postDate": "08/23/2023 03:22:39", "content": "Sure, let me ask my PI for permission.
", "rawMarkdown": "Sure, let me ask my PI for permission.", "votes": null }, { "id": "2404267", "postDate": "08/23/2023 06:34:33", "content": "I only used the embedding information of the sequence, My best individual model (0.540 on public LB) is a 4 layers DNN. To prevent overfitting, I did not increase my participation in the public list and only submitted it once. Afterwards, by replacing the dataset and features, several three sets of models were created using DNN and CNN randomly, and fused with the results of KNN, diamond, and sprof. The fusion score was only around 0.536. Add the result of quickgo, with a score of 0.573. The reason for not adjusting the fusion plan with higher scores is that I am worried that it will overfit the public list. I don't know if my concerns are correct, but I want to give it a try.
", "rawMarkdown": "I only used the embedding information of the sequence, My best individual model (0.540 on public LB) is a 4 layers DNN. To prevent overfitting, I did not increase my participation in the public list and only submitted it once. Afterwards, by replacing the dataset and features, several three sets of models were created using DNN and CNN randomly, and fused with the results of KNN, diamond, and sprof. The fusion score was only around 0.536. Add the result of quickgo, with a score of 0.573. The reason for not adjusting the fusion plan with higher scores is that I am worried that it will overfit the public list. I don't know if my concerns are correct, but I want to give it a try.", "votes": null }, { "id": "2404293", "postDate": "08/23/2023 06:58:12", "content": "By the way, my DNN uses over 4000 labels because I hope to predict as many labels as possible. After integrating with solutions such as knn and diamond, a total of more than 3w of labels were obtained. So far, I regret not integrating the final results with public notebooks.
", "rawMarkdown": "By the way, my DNN uses over 4000 labels because I hope to predict as many labels as possible. After integrating with solutions such as knn and diamond, a total of more than 3w of labels were obtained. So far, I regret not integrating the final results with public notebooks.", "votes": null }, { "id": "2404321", "postDate": "08/23/2023 07:16:36", "content": "I suddenly remembered forgetting to select the highest 1500 GO terms for each protein. Perhaps this is the reason for the decrease in model fusion scores. It's very funny.
", "rawMarkdown": "I suddenly remembered forgetting to select the highest 1500 GO terms for each protein. Perhaps this is the reason for the decrease in model fusion scores. It's very funny.", "votes": null }, { "id": "2404338", "postDate": "08/23/2023 07:32:03", "content": "Can you please describe how you tune the loss function? I did try to use the IA values or term frequencies, but I could not improve my score.
", "rawMarkdown": "Can you please describe how you tune the loss function? I did try to use the IA values or term frequencies, but I could not improve my score.", "votes": null }, { "id": "2404440", "postDate": "08/23/2023 09:00:53", "content": "That is an excellent result with DNN.
\nWould be great to share it.
\nAs for me MLP with 1000,2000 droupouts and batchnorms was the best. I tried to increased depth many time - but always got worse results.
Would be very if you can share you approach !
\nQuite interesting to look
Thank you Alexander Chervov! You are always so active and helpful! Every time I compete with you, I can learn a lot of knowledge, and congratulations on achieving excellent results 😀! I used the network structure of DenseNet, with Dense layer 's parameter is 2200 and Dropout's parameter is 0.2. During the model training process, I designed some tricks to effectively process imbalanced data.
", "rawMarkdown": "Thank you Alexander Chervov! You are always so active and helpful! Every time I compete with you, I can learn a lot of knowledge, and congratulations on achieving excellent results 😀! I used the network structure of DenseNet, with Dense layer 's parameter is 2200 and Dropout's parameter is 0.2. During the model training process, I designed some tricks to effectively process imbalanced data.", "votes": null }, { "id": "2405045", "postDate": "08/23/2023 16:45:23", "content": "Hello, mtinti. I used IA values * term frequencies and the results improved from 0.47 to 0.49 in a public notebook. So I used it.
", "rawMarkdown": "Hello, mtinti. I used IA values * term frequencies and the results improved from 0.47 to 0.49 in a public notebook. So I used it.", "votes": null }, { "id": "2405797", "postDate": "08/24/2023 05:47:33", "content": "Hi MT, I spoke with my PI. Unfortunately, She doesn't want me to disclose the specifics until my paper is submitted. I can let you know once I finished my paper.
\nUsing IA to weigh each term improves the wFmax in my local CV. It is weird that it didn't improve your score. This might be because our output dimension is different. For my case, 5311 BPO, 1626 CCO, and 2871 MFO terms were used for each ontology.
I got LB = 0.539, CV =0.55. with a T5 and ESM ensemble. When adding SPROF-GO it improves to 0.55 in LB.
\nMy best solution in CV (CV=0.59), which includes knn and models trained with different number of labels, gets only 0.53 in LB.
\nI've done random CV deleting repeatet sequences and also sequences with big PIDE. When deleting sequences with big PIDE knn solution is the best in CV.
This models are boosted in LB from 0.55 to ~0.62 when combining with quickgo this way: if a pair (root, term_id) is in quickgo, use only quickgo labels, in another case use top 90 labels for each pair(root, term_id) from my ensemble. But this tricks wont do nothing in private LB since I guess this pairs (root, term_id) that are in quickGo won´t be in private test set.
", "votes": null, "replies": [ { "id": 2403275, "author_name": "alexandervc", "author_url": "", "post_date": "08/22/2023 15:31:43", "content": "Thanks a lot!
\nDo not you know sprof-go uses only structural information or not?
if it's the public sprof-go I created, it uses only protein sequences.
", "votes": null, "replies": [ { "id": 2403305, "author_name": "mpp1998", "author_url": "", "post_date": "08/22/2023 15:50:36", "content": "Yes, it is. Thanks @mtinti
", "votes": null, "replies": [] } ] } ] }, { "id": 2403355, "author_name": "alexandervc", "author_url": "", "post_date": "08/22/2023 16:37:05", "content": "Sorry what \"root\" means ?
\nIn the phrase \"if a pair (root, term_id)….
Sorry with root i mean subontology.
\nFor example, if a protein has quickgo annotations only in BPO, in that protein I only use my ensemble predictions in MFO and CCO labels, and in BPO i use quickgo labels with prob=1.
\nI also checked whats happens not adding my predictions in any subontology if a protein has annotations in one of them or more in quickgo. That way in the example I wouldnt add my ensemble predictions in MFO and CCO either, that protein would only have BPO quickgo labels with prob=1. Doing this my score drops from 0.62 to 0.58, which means that for that proteins that appear in quickgo it is better to add my predictions in subontologies not present in quickgo, at least in LB samples.
I was able to get a good score (0.553 LB) with EMS2_3B + PCA of 1500 + MLP with dropouts + small modifications with the optimizer (AdamW) and adding more labels to the training (my best solution had 3000).
\nAlso I've tried the quick-go annotations that gave me a 0.58 LB, but I didn't select that submission.
", "votes": null, "replies": [] }, { "id": 2403317, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "08/22/2023 15:56:03", "content": "Interesting question! Our best ensemble of multiple MLPs with R (torch, tensorflow) and10-fold CV gave 0.56425 LB.
\nWe used ESM 3B, predicted top 500 targets or top by frequency with more than 250 observations for BPO and 50 for MFO and CCO, separately for each ontology. I think we will publish the model variations we used soon.
", "votes": null, "replies": [] }, { "id": 2403598, "author_name": "aucrock", "author_url": "", "post_date": "08/22/2023 18:25:21", "content": "I got LB 0.537 with T5 embeddings using a DeepGOZero model developed by Robert Hoehndorf research group (https://pubmed.ncbi.nlm.nih.gov/35758802/) with slight custom modifications. It is a single model without any ensembling trained on T5 embeddings.
\n(UPD: sorry, I checked, and in this model I have also incorporated species information. It tended to only slightly improve LB performance for other my models, though I haven’t checked how large is the improvement for this particular one).
In my experiments, adding more GO terms only improved the performance of DeepGOZero model. This one was trained on ~4500, 6000 and 3200 terms from MFO, BPO and CCO subontologies respectively.
\nI also tried to augment the training set (not the test set) with additional annotations from GOA UniProt. In this case, the same model yielded 0.56 LB
", "votes": null, "replies": [] }, { "id": 2403609, "author_name": "evansliu", "author_url": "", "post_date": "08/22/2023 18:37:11", "content": "I got LB 0.57867 using the 3 last layers from T5 as embeddings with a simple DNN model with tunning of the loss function and training data. That is my best LB single-component method.
", "votes": null, "replies": [ { "id": 2403647, "author_name": "alexandervc", "author_url": "", "post_date": "08/22/2023 19:07:21", "content": "Wow ! Can you share your code please ?
", "votes": null, "replies": [ { "id": 2403769, "author_name": "mtinti", "author_url": "", "post_date": "08/22/2023 20:55:10", "content": "I second that please, its a really interesting model!
", "votes": null, "replies": [ { "id": 2404058, "author_name": "evansliu", "author_url": "", "post_date": "08/23/2023 03:22:39", "content": "Sure, let me ask my PI for permission.
", "votes": null, "replies": [ { "id": 2404338, "author_name": "mtinti", "author_url": "", "post_date": "08/23/2023 07:32:03", "content": "Can you please describe how you tune the loss function? I did try to use the IA values or term frequencies, but I could not improve my score.
", "votes": null, "replies": [ { "id": 2405045, "author_name": "songqizhou", "author_url": "", "post_date": "08/23/2023 16:45:23", "content": "Hello, mtinti. I used IA values * term frequencies and the results improved from 0.47 to 0.49 in a public notebook. So I used it.
", "votes": null, "replies": [] }, { "id": 2405797, "author_name": "evansliu", "author_url": "", "post_date": "08/24/2023 05:47:33", "content": "Hi MT, I spoke with my PI. Unfortunately, She doesn't want me to disclose the specifics until my paper is submitted. I can let you know once I finished my paper.
\nUsing IA to weigh each term improves the wFmax in my local CV. It is weird that it didn't improve your score. This might be because our output dimension is different. For my case, 5311 BPO, 1626 CCO, and 2871 MFO terms were used for each ontology.
I think my solution qualifies as sequence-only. My best individual model (0.547 on public LB) is T5 embeddings + RPS-BLAST search results. For the latter part, I used custom profile databases (COG, KOG, SMART, TIGRFams v15, Pfam v35). All folded together into a single Keras network.
", "votes": null, "replies": [ { "id": 2404623, "author_name": "alexandervc", "author_url": "", "post_date": "08/23/2023 11:35:02", "content": "Would be very if you can share you approach !
\nQuite interesting to look
I only used the embedding information of the sequence, My best individual model (0.540 on public LB) is a 4 layers DNN. To prevent overfitting, I did not increase my participation in the public list and only submitted it once. Afterwards, by replacing the dataset and features, several three sets of models were created using DNN and CNN randomly, and fused with the results of KNN, diamond, and sprof. The fusion score was only around 0.536. Add the result of quickgo, with a score of 0.573. The reason for not adjusting the fusion plan with higher scores is that I am worried that it will overfit the public list. I don't know if my concerns are correct, but I want to give it a try.
", "votes": null, "replies": [ { "id": 2404293, "author_name": "songqizhou", "author_url": "", "post_date": "08/23/2023 06:58:12", "content": "By the way, my DNN uses over 4000 labels because I hope to predict as many labels as possible. After integrating with solutions such as knn and diamond, a total of more than 3w of labels were obtained. So far, I regret not integrating the final results with public notebooks.
", "votes": null, "replies": [ { "id": 2404321, "author_name": "songqizhou", "author_url": "", "post_date": "08/23/2023 07:16:36", "content": "I suddenly remembered forgetting to select the highest 1500 GO terms for each protein. Perhaps this is the reason for the decrease in model fusion scores. It's very funny.
", "votes": null, "replies": [] } ] }, { "id": 2404440, "author_name": "alexandervc", "author_url": "", "post_date": "08/23/2023 09:00:53", "content": "That is an excellent result with DNN.
\nWould be great to share it.
\nAs for me MLP with 1000,2000 droupouts and batchnorms was the best. I tried to increased depth many time - but always got worse results.
Thank you Alexander Chervov! You are always so active and helpful! Every time I compete with you, I can learn a lot of knowledge, and congratulations on achieving excellent results 😀! I used the network structure of DenseNet, with Dense layer 's parameter is 2200 and Dropout's parameter is 0.2. During the model training process, I designed some tricks to effectively process imbalanced data.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2403115": "I wonder what score can be achieved using only sequence information - embeddings, k-mers, whatever ? \n\nBut NOT using the things like \"quick-go\", \"taxons\" all the other things.\n\nI think the question is of some importance for research since for some completely new proteins we have nothing else except the structural information.\n\n\nPS\n\nFrom what I know: \n\nLB Public Score 0.53605 :\n\nFrom here:\n\n V5: https://www.kaggle.com/code/adaluodao/pytorch-keras-etc-3-blend-cafa-metric-etc?scriptVersionId=139934127\n\nThe model - blend of Ridge, NN with skip connections proposed by Andrey Shevtsov (3 random seeds),\nand sklearn MLP (3 random seeds). Concatenation of t5 and esm2-size1280 embeddings is used. Targets: 1100, 450,300, \ndownsampling of train - 43k proteins which have all 3 annotations - ideas proposed by Zoltan. \n\nPS\n\nThe notebook originates from my own notebook with suitable choice of models.", "2403178": "I got LB = 0.539, CV =0.55. with a T5 and ESM ensemble. When adding SPROF-GO it improves to 0.55 in LB. \nMy best solution in CV (CV=0.59), which includes knn and models trained with different number of labels, gets only 0.53 in LB. \nI've done random CV deleting repeatet sequences and also sequences with big PIDE. When deleting sequences with big PIDE knn solution is the best in CV.\n\nThis models are boosted in LB from 0.55 to ~0.62 when combining with quickgo this way: if a pair (root, term_id) is in quickgo, use only quickgo labels, in another case use top 90 labels for each pair(root, term_id) from my ensemble. But this tricks wont do nothing in private LB since I guess this pairs (root, term_id) that are in quickGo won´t be in private test set.", "2403242": "I was able to get a good score (0.553 LB) with EMS2_3B + PCA of 1500 + MLP with dropouts + small modifications with the optimizer (AdamW) and adding more labels to the training (my best solution had 3000). \n\nAlso I've tried the quick-go annotations that gave me a 0.58 LB, but I didn't select that submission.", "2403275": "Thanks a lot!\nDo not you know sprof-go uses only structural information or not?", "2403289": "if it's the public sprof-go I created, it uses only protein sequences.", "2403305": "Yes, it is. Thanks @mtinti", "2403317": "Interesting question! Our best ensemble of multiple MLPs with R (torch, tensorflow) and10-fold CV gave 0.56425 LB.\n\nWe used ESM 3B, predicted top 500 targets or top by frequency with more than 250 observations for BPO and 50 for MFO and CCO, separately for each ontology. I think we will publish the model variations we used soon.", "2403355": "Sorry what \"root\" means ?\nIn the phrase \"if a pair (root, term_id)....", "2403418": "Sorry with root i mean subontology.\nFor example, if a protein has quickgo annotations only in BPO, in that protein I only use my ensemble predictions in MFO and CCO labels, and in BPO i use quickgo labels with prob=1.\nI also checked whats happens not adding my predictions in any subontology if a protein has annotations in one of them or more in quickgo. That way in the example I wouldnt add my ensemble predictions in MFO and CCO either, that protein would only have BPO quickgo labels with prob=1. Doing this my score drops from 0.62 to 0.58, which means that for that proteins that appear in quickgo it is better to add my predictions in subontologies not present in quickgo, at least in LB samples.", "2403598": "I got LB 0.537 with T5 embeddings using a DeepGOZero model developed by Robert Hoehndorf research group (https://pubmed.ncbi.nlm.nih.gov/35758802/) with slight custom modifications. It is a single model without any ensembling trained on T5 embeddings.\n(UPD: sorry, I checked, and in this model I have also incorporated species information. It tended to only slightly improve LB performance for other my models, though I haven’t checked how large is the improvement for this particular one). \n\nIn my experiments, adding more GO terms only improved the performance of DeepGOZero model. This one was trained on ~4500, 6000 and 3200 terms from MFO, BPO and CCO subontologies respectively. \n\nI also tried to augment the training set (not the test set) with additional annotations from GOA UniProt. In this case, the same model yielded 0.56 LB", "2403609": "I got LB 0.57867 using the 3 last layers from T5 as embeddings with a simple DNN model with tunning of the loss function and training data. That is my best LB single-component method.", "2403647": "Wow ! Can you share your code please ?", "2403663": "I think my solution qualifies as sequence-only. My best individual model (0.547 on public LB) is T5 embeddings + RPS-BLAST search results. For the latter part, I used custom profile databases (COG, KOG, SMART, TIGRFams v15, Pfam v35). All folded together into a single Keras network.", "2403769": "I second that please, its a really interesting model!", "2404058": "Sure, let me ask my PI for permission.", "2404267": "I only used the embedding information of the sequence, My best individual model (0.540 on public LB) is a 4 layers DNN. To prevent overfitting, I did not increase my participation in the public list and only submitted it once. Afterwards, by replacing the dataset and features, several three sets of models were created using DNN and CNN randomly, and fused with the results of KNN, diamond, and sprof. The fusion score was only around 0.536. Add the result of quickgo, with a score of 0.573. The reason for not adjusting the fusion plan with higher scores is that I am worried that it will overfit the public list. I don't know if my concerns are correct, but I want to give it a try.", "2404293": "By the way, my DNN uses over 4000 labels because I hope to predict as many labels as possible. After integrating with solutions such as knn and diamond, a total of more than 3w of labels were obtained. So far, I regret not integrating the final results with public notebooks.", "2404321": "I suddenly remembered forgetting to select the highest 1500 GO terms for each protein. Perhaps this is the reason for the decrease in model fusion scores. It's very funny.", "2404338": "Can you please describe how you tune the loss function? I did try to use the IA values or term frequencies, but I could not improve my score.", "2404440": "That is an excellent result with DNN. \nWould be great to share it. \nAs for me MLP with 1000,2000 droupouts and batchnorms was the best. I tried to increased depth many time - but always got worse results.", "2404623": "Would be very if you can share you approach ! \nQuite interesting to look", "2405041": "Thank you Alexander Chervov! You are always so active and helpful! Every time I compete with you, I can learn a lot of knowledge, and congratulations on achieving excellent results 😀! I used the network structure of DenseNet, with Dense layer 's parameter is 2200 and Dropout's parameter is 0.2. During the model training process, I designed some tricks to effectively process imbalanced data.", "2405045": "Hello, mtinti. I used IA values * term frequencies and the results improved from 0.47 to 0.49 in a public notebook. So I used it.", "2405797": "Hi MT, I spoke with my PI. Unfortunately, She doesn't want me to disclose the specifics until my paper is submitted. I can let you know once I finished my paper.\nUsing IA to weigh each term improves the wFmax in my local CV. It is weird that it didn't improve your score. This might be because our output dimension is different. For my case, 5311 BPO, 1626 CCO, and 2871 MFO terms were used for each ontology." }, "source": "meta" }