\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F493138%2F811187e34a3c754644173b4bfe870c3c%2Fgo_pic.png?generation=1692752062442778&alt=media\")
\n\nFinally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n
I think this line may have thrown many of us when formulating a strategy for final predictions. A couple of thoughts on that subject.
\nFirst, I know that submitting more than 1500 terms per target doesn't invalidate the submission, as I have done it. I assume that they score only the first 1500 entries per target and ignore the rest. That still may be undesirable, as some confident predictions may not be included in the first 1500 entries.
\nI have done 3 independent models of 2000 top entries for each category. Yes, a total of 6000 categories. I knew intuitively that for most proteins the number of predicted categories would not be anywhere near 1500, and that turned out to be true. If you are not confident about that, this thing works: simply clip out all the prediction p<0.03 and that will remove most of spurious subonthology predictions.
\nStill, for about 50-100 proteins more than 1500 subontologies were predicted. As I said above, it makes not difference in scoring except that we have no control as to which 1500 are included. To get around that, I reverse-sorted all probabilities (highest on top), and did a groupby() like this:
df = df()()\nThis retains only 1500 predictions per target protein according to highest probabilities.
\nIt is a separate issue whether predicting such a large number of subontologies is a good idea, as that could create specificity problems for our predictions. It didn't seem like a huge problem in my hands, and I'd rather do something like that than guess how many top groups to assign each category (800, 500 and 200, or whatever other partitioning schemes were used).
", "messages": [ { "id": "2403803", "postDate": "08/22/2023 21:27:33", "content": "\n\nFinally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.
\n
I think this line may have thrown many of us when formulating a strategy for final predictions. A couple of thoughts on that subject.
\nFirst, I know that submitting more than 1500 terms per target doesn't invalidate the submission, as I have done it. I assume that they score only the first 1500 entries per target and ignore the rest. That still may be undesirable, as some confident predictions may not be included in the first 1500 entries.
\nI have done 3 independent models of 2000 top entries for each category. Yes, a total of 6000 categories. I knew intuitively that for most proteins the number of predicted categories would not be anywhere near 1500, and that turned out to be true. If you are not confident about that, this thing works: simply clip out all the prediction p<0.03 and that will remove most of spurious subonthology predictions.
\nStill, for about 50-100 proteins more than 1500 subontologies were predicted. As I said above, it makes not difference in scoring except that we have no control as to which 1500 are included. To get around that, I reverse-sorted all probabilities (highest on top), and did a groupby() like this:
df = df()()\nThis retains only 1500 predictions per target protein according to highest probabilities.
\nIt is a separate issue whether predicting such a large number of subontologies is a good idea, as that could create specificity problems for our predictions. It didn't seem like a huge problem in my hands, and I'd rather do something like that than guess how many top groups to assign each category (800, 500 and 200, or whatever other partitioning schemes were used).
", "rawMarkdown": "> Finally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.\n\nI think this line may have thrown many of us when formulating a strategy for final predictions. A couple of thoughts on that subject.\n\nFirst, I know that submitting more than 1500 terms per target doesn't invalidate the submission, as I have done it. I assume that they score only the first 1500 entries per target and ignore the rest. That still may be undesirable, as some confident predictions may not be included in the first 1500 entries.\n\nI have done 3 independent models of 2000 top entries for each category. Yes, a total of 6000 categories. I knew intuitively that for most proteins the number of predicted categories would not be anywhere near 1500, and that turned out to be true. If you are not confident about that, this thing works: simply clip out all the prediction p<0.03 and that will remove most of spurious subonthology predictions.\n\nStill, for about 50-100 proteins more than 1500 subontologies were predicted. As I said above, it makes not difference in scoring except that we have no control as to which 1500 are included. To get around that, I reverse-sorted all probabilities (highest on top), and did a `groupby()` like this:\n\n df = df.groupby(['ID_col']).head(1500)\n\nThis retains only 1500 predictions per target protein according to highest probabilities.\n\nIt is a separate issue whether predicting such a large number of subontologies is a good idea, as that could create specificity problems for our predictions. It didn't seem like a huge problem in my hands, and I'd rather do something like that than guess how many top groups to assign each category (800, 500 and 200, or whatever other partitioning schemes were used).", "votes": null }, { "id": "2403944", "postDate": "08/23/2023 00:51:32", "content": "Dear Tilii,
\nThank you for all your posts; I'm learning a lot from you! Finding your insightful comments feels to me like unpacking a Christmas gift.
\n\n\nI assume that they score only the first 1500 entries per target and ignore the rest.
\n
Your intuition is spot-on, as always, Tili! The hosts describe it in their evaluation github:
\n\n\nThe -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached. The score is not checked, meaning that terms are not sorted before the check, and the check is performed before propagation.
\n
And -max_terms is set to 500. So, to be precise, it's not
\n\n\n1500 terms per target
\n
but rather 500 terms per target x ontology pair. That's why we, with Darek @thedrcat , did
\nnew_sub_df = (sub_df\n (by=, ascending=False)\n ()\n ())\nRegarding the number of terms for modeling, we didn't limit ourselves when possible. E.g., for PSI-BLAST or FoldSeek-based matching, we allowed all GO terms. Moreover, during the competition's final hours, we discovered that introducing additional predictions based on the ontological graph helps. Let's assume we predicted GO:0043232 (intracellular non-membrane-bounded organelle) for a target. From the ontological DAG, we have
\n
So we'd propagate the predicted confidence of GO:0043232 to all its parents in the picture, introducing any parent terms not present in original predictions. In the ontological graph, we used \"is_a\" and \"part_of\" relationships, as per the hosts' evaluation github info. I now checked that, on average, we introduced 14.5 additional terms per target (with a median of 11 additional terms). To each parent, we would assign maximum confidence of his children's predictions. And only after the prediction propagation to parents, we sorted the predictions. Based on public LB, the prediction propagation to parents before sorting gave us a 0.02 boost. Regarding the overall number of unique GO terms, before the prediction propagation, we had ~32k. With the prediction propagation, we got ~30k, because the introduced parent terms might push out of the top 500 some low-confidence children predictions.
", "rawMarkdown": "Dear Tilii,\n\nThank you for all your posts; I'm learning a lot from you! Finding your insightful comments feels to me like unpacking a Christmas gift.\n\n> I assume that they score only the first 1500 entries per target and ignore the rest.\n\nYour intuition is spot-on, as always, Tili! The hosts describe it in [their evaluation github](https://github.com/BioComputingUP/CAFA-evaluator/tree/master): \n> The -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached. The score is not checked, meaning that terms are not sorted before the check, and the check is performed before propagation.\n\n\nAnd -max_terms is set to 500. So, to be precise, it's not \n>1500 terms per target\n\nbut rather 500 terms per target x ontology pair. That's why we, with Darek @thedrcat , did\n```\nnew_sub_df = (sub_df\n .sort_values(by=['EntryID', 'aspect', 'proba'], ascending=False)\n .groupby(['EntryID', 'aspect'])\n .head(500))\n```\n\nRegarding the number of terms for modeling, we didn't limit ourselves when possible. E.g., for PSI-BLAST or FoldSeek-based matching, we allowed all GO terms. Moreover, during the competition's final hours, we discovered that introducing additional predictions based on the ontological graph helps. Let's assume we predicted GO:0043232 (intracellular non-membrane-bounded organelle) for a target. From the ontological DAG, we have\n\n\nSo we'd propagate the predicted confidence of GO:0043232 to all its parents in the picture, introducing any parent terms not present in original predictions. In the ontological graph, we used \"is_a\" and \"part_of\" relationships, as per the [hosts' evaluation github](https://github.com/BioComputingUP/CAFA-evaluator/tree/master) info. I now checked that, on average, we introduced 14.5 additional terms per target (with a median of 11 additional terms). To each parent, we would assign maximum confidence of his children's predictions. And only after the prediction propagation to parents, we sorted the predictions. Based on public LB, the prediction propagation to parents before sorting gave us a 0.02 boost. Regarding the overall number of unique GO terms, before the prediction propagation, we had ~32k. With the prediction propagation, we got ~30k, because the introduced parent terms might push out of the top 500 some low-confidence children predictions.", "votes": null }, { "id": "2403960", "postDate": "08/23/2023 01:16:43", "content": "\n\nThank you for all your posts
\n
I'll take the compliment, even though I honestly don't see that you have much to learn from me. Your ProFun package is excellent, and it was only mu aversion to using anything as a black-box tool that stopped me from using it. One of these days I will get to look under the hood and understand exactly what is happening, and I expect to be a happy user afterwards.
\n\n\nbut rather 500 terms per target x ontology pair
\n
If this is the case, your approach is correct and mine isn't. My understanding was 1500 terms per target total, regardless of category distribution. @iddux1 @predragradivojac @claradepaolis Could you provide a definitive answer on this issue?
\nI'll need a fresh brain to digest the last part of your post, and may have more to say tomorrow.
", "rawMarkdown": "> Thank you for all your posts\n\nI'll take the compliment, even though I honestly don't see that you have much to learn from me. Your ProFun package is excellent, and it was only mu aversion to using anything as a black-box tool that stopped me from using it. One of these days I will get to look under the hood and understand exactly what is happening, and I expect to be a happy user afterwards.\n\n> but rather 500 terms per target x ontology pair\n\nIf this is the case, your approach is correct and mine isn't. My understanding was 1500 terms per target total, regardless of category distribution. @iddux1 @predragradivojac @claradepaolis Could you provide a definitive answer on this issue?\n\nI'll need a fresh brain to digest the last part of your post, and may have more to say tomorrow.", "votes": null }, { "id": "2403971", "postDate": "08/23/2023 01:28:19", "content": "Dear Tilii,
\nThank you so much for your kind words! It means a lot coming from you. And your kind feedback motivates me to work on the ProFun package more, once I have time.
", "rawMarkdown": "Dear Tilii,\n\nThank you so much for your kind words! It means a lot coming from you. And your kind feedback motivates me to work on the ProFun package more, once I have time.", "votes": null }, { "id": "2404347", "postDate": "08/23/2023 07:38:03", "content": "Hi Tilii,
\nIt may not be necessary to submit so many predictions.
\nI tried to submit different number of predictions on MFO including top 50, top 100, top 200. But the LB scores for the last two are the same as the top 50.
\nSo to save time and memory, I am using top 100 prediction results on all the three types of GO.
", "rawMarkdown": "Hi Tilii,\n\nIt may not be necessary to submit so many predictions.\n\nI tried to submit different number of predictions on MFO including top 50, top 100, top 200. But the LB scores for the last two are the same as the top 50.\n\nSo to save time and memory, I am using top 100 prediction results on all the three types of GO.", "votes": null }, { "id": "2404379", "postDate": "08/23/2023 08:00:27", "content": "Hi @tilii7,
\nIf the official repo of CAFA-evaluator is used in the end, then I think the following statement is correct.
\n\nbut rather 500 terms per target x ontology pair
\n
I copy-paste an explanation of how the prediction file is parsed in the official evaluator here. Hope it helps.
", "rawMarkdown": "Hi @tilii7,\n\nIf the official repo of `CAFA-evaluator` is used in the end, then I think the following statement is correct.\n\n> but rather 500 terms per target x ontology pair\n\nI copy-paste an explanation of how the prediction file is parsed in the official evaluator [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433868#2404364). Hope it helps.", "votes": null }, { "id": "2404393", "postDate": "08/23/2023 08:11:08", "content": "Hi @impimp,
\nI think the reason why the score is the same is that tau_arr is used to derive the score. In all of the situations you mention (i.e., submit with top50, 100 and 200), the best tau might be the same one, and the function solidify_prediction will zero out the proba below this tau. Hence, it's possible that the solidified prediction matrix is exactly the same one in these 3 situations. Then, the final score is the same.
\nI just figured this out two days before the competition end, and used a simple method to optimize (to save time) the number of predicting terms for submission.
That is an interesting question !
\nDo not have definitive answer and we tried different ways and for different models the best choice was different.
From the just theoretical point of view - many terms are very similar in Biological Process and Molecular function,
\nas well as some biologically inspired correlations should be with Cellar Component.
\nThat is why it is reasonable to think that training for the model on ALL subontologies at once,
\nshould benefit from correlations of tartgets in different subontologies.
\nThat is for for neural networks or \"pyboost\" for example which try to learn many targets at once.
\nThat is why that was main route pursued to us.
\n\nIt may not be necessary to submit so many predictions.
\n
You are absolutely right, as for most proteins 50 categories is enough. Still, there are at least 100-200 proteins for which a larger number is required.
", "rawMarkdown": "> It may not be necessary to submit so many predictions.\n\nYou are absolutely right, as for most proteins 50 categories is enough. Still, there are at least 100-200 proteins for which a larger number is required.", "votes": null }, { "id": "2404899", "postDate": "08/23/2023 15:09:02", "content": "Yup, my approach was the same.
", "rawMarkdown": "Yup, my approach was the same.", "votes": null }, { "id": "2404944", "postDate": "08/23/2023 15:41:56", "content": "I have been developing a model for each subontology since the start. I was afraid that a model trained on incomplete data might mistakenly assign a zero probability to a protein for a specific GO term, not because the protein doesn't have that function, process, or location, but because the annotation is absent from the training data. In retrospect, I'm thinking now that I could have used pseudo-labeling. For example, taking the highest confidence predictions from one model/subontology and add these predictions to the train data.
", "rawMarkdown": "I have been developing a model for each subontology since the start. I was afraid that a model trained on incomplete data might mistakenly assign a zero probability to a protein for a specific GO term, not because the protein doesn't have that function, process, or location, but because the annotation is absent from the training data. In retrospect, I'm thinking now that I could have used pseudo-labeling. For example, taking the highest confidence predictions from one model/subontology and add these predictions to the train data.", "votes": null }, { "id": "2405008", "postDate": "08/23/2023 16:15:57", "content": "Pseudo-labeling sounds like a good idea, though one would have to experiment with labeling thresholds.
", "rawMarkdown": "Pseudo-labeling sounds like a good idea, though one would have to experiment with labeling thresholds.", "votes": null }, { "id": "2405541", "postDate": "08/24/2023 01:04:21", "content": "Hi, @abaojiang
\nIs this tau you speak of utilized in the calculation of max-F1 values?
Hi @impimp,
\nExactly. In evaluation.py, you can see there's a tau_arr used to derive the max-F1. And the logic behind the scene is as follows,
tau to solidify the prediction matrix (i.e., each entry is a probability).tau. Entries below tau are zeroed out.Note that for each tau, there exists a corresponding F metric. Then, the final max-F1 is the maximum among all these F1s at different taus. If you're interested in looking into the detailed implementation, the code snippet is at line 60 - 91 in evaluation.py.
Dear Tilii,
\nThank you for all your posts; I'm learning a lot from you! Finding your insightful comments feels to me like unpacking a Christmas gift.
\n\n\nI assume that they score only the first 1500 entries per target and ignore the rest.
\n
Your intuition is spot-on, as always, Tili! The hosts describe it in their evaluation github:
\n\n\nThe -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached. The score is not checked, meaning that terms are not sorted before the check, and the check is performed before propagation.
\n
And -max_terms is set to 500. So, to be precise, it's not
\n\n\n1500 terms per target
\n
but rather 500 terms per target x ontology pair. That's why we, with Darek @thedrcat , did
\nnew_sub_df = (sub_df\n (by=, ascending=False)\n ()\n ())\nRegarding the number of terms for modeling, we didn't limit ourselves when possible. E.g., for PSI-BLAST or FoldSeek-based matching, we allowed all GO terms. Moreover, during the competition's final hours, we discovered that introducing additional predictions based on the ontological graph helps. Let's assume we predicted GO:0043232 (intracellular non-membrane-bounded organelle) for a target. From the ontological DAG, we have
\n
So we'd propagate the predicted confidence of GO:0043232 to all its parents in the picture, introducing any parent terms not present in original predictions. In the ontological graph, we used \"is_a\" and \"part_of\" relationships, as per the hosts' evaluation github info. I now checked that, on average, we introduced 14.5 additional terms per target (with a median of 11 additional terms). To each parent, we would assign maximum confidence of his children's predictions. And only after the prediction propagation to parents, we sorted the predictions. Based on public LB, the prediction propagation to parents before sorting gave us a 0.02 boost. Regarding the overall number of unique GO terms, before the prediction propagation, we had ~32k. With the prediction propagation, we got ~30k, because the introduced parent terms might push out of the top 500 some low-confidence children predictions.
", "votes": null, "replies": [ { "id": 2403960, "author_name": "tilii7", "author_url": "", "post_date": "08/23/2023 01:16:43", "content": "\n\nThank you for all your posts
\n
I'll take the compliment, even though I honestly don't see that you have much to learn from me. Your ProFun package is excellent, and it was only mu aversion to using anything as a black-box tool that stopped me from using it. One of these days I will get to look under the hood and understand exactly what is happening, and I expect to be a happy user afterwards.
\n\n\nbut rather 500 terms per target x ontology pair
\n
If this is the case, your approach is correct and mine isn't. My understanding was 1500 terms per target total, regardless of category distribution. @iddux1 @predragradivojac @claradepaolis Could you provide a definitive answer on this issue?
\nI'll need a fresh brain to digest the last part of your post, and may have more to say tomorrow.
", "votes": null, "replies": [ { "id": 2403971, "author_name": "samusram", "author_url": "", "post_date": "08/23/2023 01:28:19", "content": "Dear Tilii,
\nThank you so much for your kind words! It means a lot coming from you. And your kind feedback motivates me to work on the ProFun package more, once I have time.
", "votes": null, "replies": [] }, { "id": 2404379, "author_name": "abaojiang", "author_url": "", "post_date": "08/23/2023 08:00:27", "content": "Hi @tilii7,
\nIf the official repo of CAFA-evaluator is used in the end, then I think the following statement is correct.
\n\nbut rather 500 terms per target x ontology pair
\n
I copy-paste an explanation of how the prediction file is parsed in the official evaluator here. Hope it helps.
", "votes": null, "replies": [] } ] } ] }, { "id": 2404347, "author_name": "impimp", "author_url": "", "post_date": "08/23/2023 07:38:03", "content": "Hi Tilii,
\nIt may not be necessary to submit so many predictions.
\nI tried to submit different number of predictions on MFO including top 50, top 100, top 200. But the LB scores for the last two are the same as the top 50.
\nSo to save time and memory, I am using top 100 prediction results on all the three types of GO.
", "votes": null, "replies": [ { "id": 2404393, "author_name": "abaojiang", "author_url": "", "post_date": "08/23/2023 08:11:08", "content": "Hi @impimp,
\nI think the reason why the score is the same is that tau_arr is used to derive the score. In all of the situations you mention (i.e., submit with top50, 100 and 200), the best tau might be the same one, and the function solidify_prediction will zero out the proba below this tau. Hence, it's possible that the solidified prediction matrix is exactly the same one in these 3 situations. Then, the final score is the same.
\nI just figured this out two days before the competition end, and used a simple method to optimize (to save time) the number of predicting terms for submission.
Hi, @abaojiang
\nIs this tau you speak of utilized in the calculation of max-F1 values?
Hi @impimp,
\nExactly. In evaluation.py, you can see there's a tau_arr used to derive the max-F1. And the logic behind the scene is as follows,
tau to solidify the prediction matrix (i.e., each entry is a probability).tau. Entries below tau are zeroed out.Note that for each tau, there exists a corresponding F metric. Then, the final max-F1 is the maximum among all these F1s at different taus. If you're interested in looking into the detailed implementation, the code snippet is at line 60 - 91 in evaluation.py.
\n\nIt may not be necessary to submit so many predictions.
\n
You are absolutely right, as for most proteins 50 categories is enough. Still, there are at least 100-200 proteins for which a larger number is required.
", "votes": null, "replies": [] } ] }, { "id": 2404468, "author_name": "alexandervc", "author_url": "", "post_date": "08/23/2023 09:15:58", "content": "That is an interesting question !
\nDo not have definitive answer and we tried different ways and for different models the best choice was different.
From the just theoretical point of view - many terms are very similar in Biological Process and Molecular function,
\nas well as some biologically inspired correlations should be with Cellar Component.
\nThat is why it is reasonable to think that training for the model on ALL subontologies at once,
\nshould benefit from correlations of tartgets in different subontologies.
\nThat is for for neural networks or \"pyboost\" for example which try to learn many targets at once.
\nThat is why that was main route pursued to us.
Yup, my approach was the same.
", "votes": null, "replies": [] }, { "id": 2404944, "author_name": "mtinti", "author_url": "", "post_date": "08/23/2023 15:41:56", "content": "I have been developing a model for each subontology since the start. I was afraid that a model trained on incomplete data might mistakenly assign a zero probability to a protein for a specific GO term, not because the protein doesn't have that function, process, or location, but because the annotation is absent from the training data. In retrospect, I'm thinking now that I could have used pseudo-labeling. For example, taking the highest confidence predictions from one model/subontology and add these predictions to the train data.
", "votes": null, "replies": [ { "id": 2405008, "author_name": "tilii7", "author_url": "", "post_date": "08/23/2023 16:15:57", "content": "Pseudo-labeling sounds like a good idea, though one would have to experiment with labeling thresholds.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2403803": "> Finally, to limit prediction file sizes, one target cannot be associated with more than 1500 terms for MF, BP, and CC subontologies combined.\n\nI think this line may have thrown many of us when formulating a strategy for final predictions. A couple of thoughts on that subject.\n\nFirst, I know that submitting more than 1500 terms per target doesn't invalidate the submission, as I have done it. I assume that they score only the first 1500 entries per target and ignore the rest. That still may be undesirable, as some confident predictions may not be included in the first 1500 entries.\n\nI have done 3 independent models of 2000 top entries for each category. Yes, a total of 6000 categories. I knew intuitively that for most proteins the number of predicted categories would not be anywhere near 1500, and that turned out to be true. If you are not confident about that, this thing works: simply clip out all the prediction p<0.03 and that will remove most of spurious subonthology predictions.\n\nStill, for about 50-100 proteins more than 1500 subontologies were predicted. As I said above, it makes not difference in scoring except that we have no control as to which 1500 are included. To get around that, I reverse-sorted all probabilities (highest on top), and did a `groupby()` like this:\n\n df = df.groupby(['ID_col']).head(1500)\n\nThis retains only 1500 predictions per target protein according to highest probabilities.\n\nIt is a separate issue whether predicting such a large number of subontologies is a good idea, as that could create specificity problems for our predictions. It didn't seem like a huge problem in my hands, and I'd rather do something like that than guess how many top groups to assign each category (800, 500 and 200, or whatever other partitioning schemes were used).", "2403944": "Dear Tilii,\n\nThank you for all your posts; I'm learning a lot from you! Finding your insightful comments feels to me like unpacking a Christmas gift.\n\n> I assume that they score only the first 1500 entries per target and ignore the rest.\n\nYour intuition is spot-on, as always, Tili! The hosts describe it in [their evaluation github](https://github.com/BioComputingUP/CAFA-evaluator/tree/master): \n> The -max_terms parameter determines the maximum number of terms that will be considered for each target. Parsing stops when the target limit for every ontology is reached. The score is not checked, meaning that terms are not sorted before the check, and the check is performed before propagation.\n\n\nAnd -max_terms is set to 500. So, to be precise, it's not \n>1500 terms per target\n\nbut rather 500 terms per target x ontology pair. That's why we, with Darek @thedrcat , did\n```\nnew_sub_df = (sub_df\n .sort_values(by=['EntryID', 'aspect', 'proba'], ascending=False)\n .groupby(['EntryID', 'aspect'])\n .head(500))\n```\n\nRegarding the number of terms for modeling, we didn't limit ourselves when possible. E.g., for PSI-BLAST or FoldSeek-based matching, we allowed all GO terms. Moreover, during the competition's final hours, we discovered that introducing additional predictions based on the ontological graph helps. Let's assume we predicted GO:0043232 (intracellular non-membrane-bounded organelle) for a target. From the ontological DAG, we have\n\n\nSo we'd propagate the predicted confidence of GO:0043232 to all its parents in the picture, introducing any parent terms not present in original predictions. In the ontological graph, we used \"is_a\" and \"part_of\" relationships, as per the [hosts' evaluation github](https://github.com/BioComputingUP/CAFA-evaluator/tree/master) info. I now checked that, on average, we introduced 14.5 additional terms per target (with a median of 11 additional terms). To each parent, we would assign maximum confidence of his children's predictions. And only after the prediction propagation to parents, we sorted the predictions. Based on public LB, the prediction propagation to parents before sorting gave us a 0.02 boost. Regarding the overall number of unique GO terms, before the prediction propagation, we had ~32k. With the prediction propagation, we got ~30k, because the introduced parent terms might push out of the top 500 some low-confidence children predictions.", "2403960": "> Thank you for all your posts\n\nI'll take the compliment, even though I honestly don't see that you have much to learn from me. Your ProFun package is excellent, and it was only mu aversion to using anything as a black-box tool that stopped me from using it. One of these days I will get to look under the hood and understand exactly what is happening, and I expect to be a happy user afterwards.\n\n> but rather 500 terms per target x ontology pair\n\nIf this is the case, your approach is correct and mine isn't. My understanding was 1500 terms per target total, regardless of category distribution. @iddux1 @predragradivojac @claradepaolis Could you provide a definitive answer on this issue?\n\nI'll need a fresh brain to digest the last part of your post, and may have more to say tomorrow.", "2403971": "Dear Tilii,\n\nThank you so much for your kind words! It means a lot coming from you. And your kind feedback motivates me to work on the ProFun package more, once I have time.", "2404347": "Hi Tilii,\n\nIt may not be necessary to submit so many predictions.\n\nI tried to submit different number of predictions on MFO including top 50, top 100, top 200. But the LB scores for the last two are the same as the top 50.\n\nSo to save time and memory, I am using top 100 prediction results on all the three types of GO.", "2404379": "Hi @tilii7,\n\nIf the official repo of `CAFA-evaluator` is used in the end, then I think the following statement is correct.\n\n> but rather 500 terms per target x ontology pair\n\nI copy-paste an explanation of how the prediction file is parsed in the official evaluator [here](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/433868#2404364). Hope it helps.", "2404393": "Hi @impimp,\n\nI think the reason why the score is the same is that `tau_arr` is used to derive the score. In all of the situations you mention (*i.e.,* submit with top50, 100 and 200), the best `tau` might be the same one, and the function `solidify_prediction` will zero out the proba below this `tau`. Hence, it's possible that the solidified prediction matrix is exactly the same one in these 3 situations. Then, the final score is the same.\nI just figured this out two days before the competition end, and used a simple method to optimize (to save time) the number of predicting terms for submission.", "2404468": "That is an interesting question ! \nDo not have definitive answer and we tried different ways and for different models the best choice was different.\n\nFrom the just theoretical point of view - many terms are very similar in Biological Process and Molecular function,\nas well as some biologically inspired correlations should be with Cellar Component.\nThat is why it is reasonable to think that training for the model on ALL subontologies at once,\nshould benefit from correlations of tartgets in different subontologies.\nThat is for for neural networks or \"pyboost\" for example which try to learn many targets at once. \nThat is why that was main route pursued to us.", "2404896": "> It may not be necessary to submit so many predictions.\n\nYou are absolutely right, as for most proteins 50 categories is enough. Still, there are at least 100-200 proteins for which a larger number is required.", "2404899": "Yup, my approach was the same.", "2404944": "I have been developing a model for each subontology since the start. I was afraid that a model trained on incomplete data might mistakenly assign a zero probability to a protein for a specific GO term, not because the protein doesn't have that function, process, or location, but because the annotation is absent from the training data. In retrospect, I'm thinking now that I could have used pseudo-labeling. For example, taking the highest confidence predictions from one model/subontology and add these predictions to the train data.", "2405008": "Pseudo-labeling sounds like a good idea, though one would have to experiment with labeling thresholds.", "2405541": "Hi, @abaojiang \n\nIs this `tau` you speak of utilized in the calculation of max-F1 values?", "2405742": "Hi @impimp,\n\nExactly. In `evaluation.py`, you can see there's a `tau_arr` used to derive the max-F1. And the logic behind the scene is as follows,\n1. Use the current `tau` to solidify the prediction matrix (*i.e.,* each entry is a probability).\n * Solidify in the code means binarization based on `tau`. Entries below `tau` are zeroed out.\n2. The binarized prediction matrix is used to derived the weighted precision and recall based on IA.\n3. The F metric is derived with the weighted precision and recall.\n\nNote that for each `tau`, there exists a corresponding F metric. Then, the final max-F1 is the maximum among all these F1s at different `tau`s. If you're interested in looking into the detailed implementation, the code snippet is at line 60 - 91 in [`evaluation.py`](https://github.com/BioComputingUP/CAFA-evaluator/blob/master/src/evaluation.py)." }, "source": "meta" }