I'm sure I'll end up face-palming when I get the answer to this, but I'm puzzled as to how my leaderboard score tends to increase when I submit fewer predictions. Let me explain. I have a large submission file generated by a neural net model (that doesn't really matter). Only predictions with probabilities >= 0.4 are included. I submit it and get a score of 0.48 - not bad not great. That score as I understand it is the average of the maximum F1 scores found for the 3 ontologies in my submitted predictions. OK, then I remove all the lowest scoring predictions (p < 0.5) just by using grep and resubmit that smaller file to the leaderboard. I then get a better LB score of 0.5 - still not great but better. How can that be? Surely the F1max process should have ignored the lower scoring predictions the first time around - how can removing the lower confidence predictions give me a better F1max score? I would expect the score to have stayed the same or gone down, but not up.
\nWhat am I missing here?
", "messages": [ { "id": "2390092", "postDate": "08/14/2023 11:27:13", "content": "I'm sure I'll end up face-palming when I get the answer to this, but I'm puzzled as to how my leaderboard score tends to increase when I submit fewer predictions. Let me explain. I have a large submission file generated by a neural net model (that doesn't really matter). Only predictions with probabilities >= 0.4 are included. I submit it and get a score of 0.48 - not bad not great. That score as I understand it is the average of the maximum F1 scores found for the 3 ontologies in my submitted predictions. OK, then I remove all the lowest scoring predictions (p < 0.5) just by using grep and resubmit that smaller file to the leaderboard. I then get a better LB score of 0.5 - still not great but better. How can that be? Surely the F1max process should have ignored the lower scoring predictions the first time around - how can removing the lower confidence predictions give me a better F1max score? I would expect the score to have stayed the same or gone down, but not up.
\nWhat am I missing here?
", "rawMarkdown": "I'm sure I'll end up face-palming when I get the answer to this, but I'm puzzled as to how my leaderboard score tends to increase when I submit fewer predictions. Let me explain. I have a large submission file generated by a neural net model (that doesn't really matter). Only predictions with probabilities >= 0.4 are included. I submit it and get a score of 0.48 - not bad not great. That score as I understand it is the average of the maximum F1 scores found for the 3 ontologies in my submitted predictions. OK, then I remove all the lowest scoring predictions (p < 0.5) just by using grep and resubmit that smaller file to the leaderboard. I then get a better LB score of 0.5 - still not great but better. How can that be? Surely the F1max process should have ignored the lower scoring predictions the first time around - how can removing the lower confidence predictions give me a better F1max score? I would expect the score to have stayed the same or gone down, but not up.\n\nWhat am I missing here?", "votes": null }, { "id": "2390536", "postDate": "08/14/2023 15:59:37", "content": "Can that be somehow caused by \"propagation\" ?
\nOtherwise if there is no bug in metric code of F1 it should not happen, as far as I understand at least for the moment.
", "rawMarkdown": "Can that be somehow caused by \"propagation\" ?\n\nOtherwise if there is no bug in metric code of F1 it should not happen, as far as I understand at least for the moment.", "votes": null }, { "id": "2390629", "postDate": "08/14/2023 16:54:24", "content": "Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?
", "rawMarkdown": "Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?", "votes": null }, { "id": "2390646", "postDate": "08/14/2023 17:05:02", "content": "May be propagation causes False positives, which are absent with cutoff 0.5, can it be ?
\n\n", "rawMarkdown": "May be propagation causes False positives, which are absent with cutoff 0.5, can it be ? \n\n> Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?", "votes": null }, { "id": "2390663", "postDate": "08/14/2023 17:11:55", "content": "Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?
\n
Surely any terms that are propagated would still have their original scores i.e. scores < 0.5, so when the F1 threshold is 0.5 then all of those extra terms would be excluded from the calculation? At least that's how I would code it.
", "rawMarkdown": "Surely any terms that are propagated would still have their original scores i.e. scores < 0.5, so when the F1 threshold is 0.5 then all of those extra terms would be excluded from the calculation? At least that's how I would code it.", "votes": null }, { "id": "2390715", "postDate": "08/14/2023 17:33:58", "content": "Yep, you are right , hmm
\nLet me think. In my notebooks there's local calculation,so can be checked there , but it is a bit tricky interfaced
I think it is possible that lower scores increase the performance. Imagine you only have 1 additional prediction term in one new protein (and then propagate it) at score 0.4 and that this prediction is exactly correct, so the whole path to the root is correct. Everything else is the same and at scores 0.5 or higher. Then, when you add this protein within the F_max formula it will be adding 1 for both precision and recall into the averaging formula over all proteins. Thus, it will increase F_max averaged over all proteins.
", "rawMarkdown": "I think it is possible that lower scores increase the performance. Imagine you only have 1 additional prediction term in one new protein (and then propagate it) at score 0.4 and that this prediction is exactly correct, so the whole path to the root is correct. Everything else is the same and at scores 0.5 or higher. Then, when you add this protein within the F_max formula it will be adding 1 for both precision and recall into the averaging formula over all proteins. Thus, it will increase F_max averaged over all proteins.", "votes": null }, { "id": "2391063", "postDate": "08/15/2023 00:54:06", "content": "I should that now, imagine the 0.4 branch was all wrong instead of all right. In this case, it will reduce the performance because both precision and recall will be 0. Basically, F_max can go either way.
", "rawMarkdown": "I should that now, imagine the 0.4 branch was all wrong instead of all right. In this case, it will reduce the performance because both precision and recall will be 0. Basically, F_max can go either way.", "votes": null }, { "id": "2391464", "postDate": "08/15/2023 07:17:54", "content": "Could also be related to the information content ic(f) of predictions?
", "rawMarkdown": "Could also be related to the information content ic(f) of predictions?", "votes": null }, { "id": "2391863", "postDate": "08/15/2023 11:05:44", "content": "I agree that the lower scoring predictions could affect the precision and recall in either direction if they are included. However, why would they be included in the calculated F1max score if they end up reducing the F1 score. In that case the F1max score would look at the F1 score taken at the higher threshold and automatically disregard those bad predictions. If they are good predictions, then the F1max calculation would look at the F1 score at the lower threshold and include them.
\nThe point here is that if I do the thresholding myself before I submit my predictions i.e. exclude all predictions with probabilities less than 0.5 I get a better LB score than if I leave it to the LB code to do the thresholding for me. I just can't see how that can be.
\nLooking at the evaluation code at first sight the very first thing it does is extract the predictions that are greater than the given threshold tau (which is varied from 0.01 to 0.99). All downstream calculations from there are based on that partitioning. So when it reaches 0.5, it should produce the same LB score as my pre-thresholded submission. But it doesn't seem to.
", "rawMarkdown": "I agree that the lower scoring predictions could affect the precision and recall in either direction **if** they are included. However, why would they be included in the calculated F1**max** score if they end up reducing the F1 score. In that case the F1max score would look at the F1 score taken at the higher threshold and automatically disregard those bad predictions. If they are good predictions, then the F1max calculation would look at the F1 score at the lower threshold and include them.\n\nThe point here is that if I do the thresholding myself before I submit my predictions i.e. exclude all predictions with probabilities less than 0.5 I get a better LB score than if I leave it to the LB code to do the thresholding for me. I just can't see how that can be.\n\nLooking at the evaluation code at first sight the very first thing it does is extract the predictions that are greater than the given threshold tau (which is varied from 0.01 to 0.99). All downstream calculations from there are based on that partitioning. So when it reaches 0.5, it should produce the same LB score as my pre-thresholded submission. But it doesn't seem to.", "votes": null }, { "id": "2391948", "postDate": "08/15/2023 11:40:25", "content": "Another try:
\nThere is cutoff of 1500 predictions.
\nIt seems it might be done WITHOUT ordering the predictions.
\nSo if you send say 10_000 predictions - per protein
\nit might that cutoff will select RANDOM 1500 out these 10_000
\nso scores - go down
Thus if you put 0.5 threshold by yourself it is very plausible that no protein will have more than 1500 predictions,
\nand that problem will not arise.
More precisely cutoff is 500 for each ontology, not overall 1500.
", "rawMarkdown": "Another try: \n\nThere is cutoff of 1500 predictions. \nIt seems it might be done WITHOUT ordering the predictions.\nSo if you send say 10_000 predictions - per protein\nit might that cutoff will select RANDOM 1500 out these 10_000\nso scores - go down \n\nThus if you put 0.5 threshold by yourself it is very plausible that no protein will have more than 1500 predictions,\nand that problem will not arise. \n\n-----\n\nMore precisely cutoff is 500 for each ontology, not overall 1500.", "votes": null }, { "id": "2392291", "postDate": "08/15/2023 15:21:50", "content": "Hmm. Admittedly that could create the observed effect. However, I don't think it's likely to be the case. The max number of terms I assign to a single protein at a threshold of 0.4 is 1145. Admittedly that doesn't mean that one of the ontologies couldn't be over 500. However only 296 of the 140k proteins were assigned more than 500GO terms in total, so it would be extreme bad luck if that is the explanation. I guess I'll have to give up two LB evaluations to repeating the test with those 296 terms excluded to be sure your explanation isn't the answer.
", "rawMarkdown": "Hmm. Admittedly that could create the observed effect. However, I don't think it's likely to be the case. The max number of terms I assign to a single protein at a threshold of 0.4 is 1145. Admittedly that doesn't mean that one of the ontologies couldn't be over 500. However only 296 of the 140k proteins were assigned more than 500GO terms in total, so it would be extreme bad luck if that is the explanation. I guess I'll have to give up two LB evaluations to repeating the test with those 296 terms excluded to be sure your explanation isn't the answer.", "votes": null }, { "id": "2392479", "postDate": "08/15/2023 17:34:59", "content": "OK experiment done. I used another smaller submission file this time around, so lower LB scores. However, no protein had more than 500 terms assigned in total. Results: 0.38125 with pre-applied 0.5 threshold, 0.37147 for same file but with pre-applied 0.4 threshold. Smaller difference for this file, but I really think this should not be possible at all. Still puzzled…
", "rawMarkdown": "OK experiment done. I used another smaller submission file this time around, so lower LB scores. However, no protein had more than 500 terms assigned in total. Results: 0.38125 with pre-applied 0.5 threshold, 0.37147 for same file but with pre-applied 0.4 threshold. Smaller difference for this file, but I really think this should not be possible at all. Still puzzled...", "votes": null }, { "id": "2392649", "postDate": "08/15/2023 19:29:35", "content": "Got it, I misunderstood the problem. I am not sure what the reason is, but there's another quirk about propagation that we have that could affect things (maybe). It is that if you propagate your scores up to the root of ontology yourself, the evaluation code does not change this, which allows for higher up terms to have lower scores (some legacy requirement we had from before). However, if you do not propagate all scores to the root, then the software will do this for you by picking the maximum score of a parent and all its children as the parent's score. I wonder if this could have some effect here.
", "rawMarkdown": "Got it, I misunderstood the problem. I am not sure what the reason is, but there's another quirk about propagation that we have that could affect things (maybe). It is that if you propagate your scores up to the root of ontology yourself, the evaluation code does not change this, which allows for higher up terms to have lower scores (some legacy requirement we had from before). However, if you do not propagate all scores to the root, then the software will do this for you by picking the maximum score of a parent and all its children as the parent's score. I wonder if this could have some effect here.", "votes": null }, { "id": "2392679", "postDate": "08/15/2023 19:52:13", "content": "Ah, light bulb moment I think. That's starting to sound like it could be the explanation.
\nIn my case I propagate the GO terms during training, but not during inference. I also predict the majority of GO terms (> 32000 of them). So it is possible that a term nearer to root could be scored less than a more specific term (it's hard to reason with large neural networks!).
\nIf I understand you correctly, when the lower scoring term is present in the submission file you don't overwrite the score. So when I remove the lower scoring terms, your script does then fill in a higher score for that label score with the higher score during propagation. Yes, I think that would do it.
\nProbably too late to fix it properly now, but I should either propagate during inference (might still be time for that) or constrain the GO term graph relationships as part of the training loss function (maybe for CAFA6!).
\nThanks for your input, Predrag!
", "rawMarkdown": "Ah, light bulb moment I think. That's starting to sound like it could be the explanation.\n\nIn my case I propagate the GO terms during training, but not during inference. I also predict the majority of GO terms (> 32000 of them). So it is possible that a term nearer to root could be scored less than a more specific term (it's hard to reason with large neural networks!).\n\nIf I understand you correctly, when the lower scoring term is present in the submission file you don't overwrite the score. So when I remove the lower scoring terms, your script does then fill in a higher score for that label score with the higher score during propagation. Yes, I think that would do it.\n\nProbably too late to fix it properly now, but I should either propagate during inference (might still be time for that) or constrain the GO term graph relationships as part of the training loss function (maybe for CAFA6!).\n\nThanks for your input, Predrag!", "votes": null }, { "id": "2392707", "postDate": "08/15/2023 20:14:19", "content": "Sure thing. We do not propagate scores if the submitter propagates them. In previous rounds of CAFA, some groups argued that this is what they wanted and we felt it made some sense (though not complete sense). In reality, there is some uncertainty inherent to prediction scores and leaving scores as they are resolves that uncertainty for the submitters without the need to reconcile parent-child inconsistencies themselves.
", "rawMarkdown": "Sure thing. We do not propagate scores if the submitter propagates them. In previous rounds of CAFA, some groups argued that this is what they wanted and we felt it made some sense (though not complete sense). In reality, there is some uncertainty inherent to prediction scores and leaving scores as they are resolves that uncertainty for the submitters without the need to reconcile parent-child inconsistencies themselves.", "votes": null }, { "id": "2392722", "postDate": "08/15/2023 20:38:37", "content": "Yes, I had the same behavior.
\nMy idea was to submit as many functions as possible even with small scores, and thus I have selected really small threshold. Recieved ~0.02 leaderboard score
\nAfterwards I have returned 0.1 threshold, launched the same script -- and the result was ~0.35 leaderboard score
\nI do not think that this behavior is a wanted one especially due to the fact that threshold-free metrics such as ROC-AUC or PRC-AUC can be nicely computed on the full dataset. At least I do that for 10% data without any problem every epoch for all the samples (140k) and all the classes (32k)
", "rawMarkdown": "Yes, I had the same behavior.\n\nMy idea was to submit as many functions as possible even with small scores, and thus I have selected really small threshold. Recieved ~0.02 leaderboard score\n\nAfterwards I have returned 0.1 threshold, launched the same script -- and the result was ~0.35 leaderboard score\n\nI do not think that this behavior is a wanted one especially due to the fact that threshold-free metrics such as ROC-AUC or PRC-AUC can be nicely computed on the full dataset. At least I do that for 10% data without any problem every epoch for all the samples (140k) and all the classes (32k)", "votes": null } ], "comments": [ { "id": 2390536, "author_name": "alexandervc", "author_url": "", "post_date": "08/14/2023 15:59:37", "content": "Can that be somehow caused by \"propagation\" ?
\nOtherwise if there is no bug in metric code of F1 it should not happen, as far as I understand at least for the moment.
", "votes": null, "replies": [ { "id": 2390629, "author_name": "biodav", "author_url": "", "post_date": "08/14/2023 16:54:24", "content": "Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?
", "votes": null, "replies": [ { "id": 2390646, "author_name": "alexandervc", "author_url": "", "post_date": "08/14/2023 17:05:02", "content": "May be propagation causes False positives, which are absent with cutoff 0.5, can it be ?
\n\n", "votes": null, "replies": [ { "id": 2390663, "author_name": "biodav", "author_url": "", "post_date": "08/14/2023 17:11:55", "content": "Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?
\n
Surely any terms that are propagated would still have their original scores i.e. scores < 0.5, so when the F1 threshold is 0.5 then all of those extra terms would be excluded from the calculation? At least that's how I would code it.
", "votes": null, "replies": [ { "id": 2390715, "author_name": "alexandervc", "author_url": "", "post_date": "08/14/2023 17:33:58", "content": "Yep, you are right , hmm
\nLet me think. In my notebooks there's local calculation,so can be checked there , but it is a bit tricky interfaced
I think it is possible that lower scores increase the performance. Imagine you only have 1 additional prediction term in one new protein (and then propagate it) at score 0.4 and that this prediction is exactly correct, so the whole path to the root is correct. Everything else is the same and at scores 0.5 or higher. Then, when you add this protein within the F_max formula it will be adding 1 for both precision and recall into the averaging formula over all proteins. Thus, it will increase F_max averaged over all proteins.
", "votes": null, "replies": [ { "id": 2391063, "author_name": "predragradivojac", "author_url": "", "post_date": "08/15/2023 00:54:06", "content": "I should that now, imagine the 0.4 branch was all wrong instead of all right. In this case, it will reduce the performance because both precision and recall will be 0. Basically, F_max can go either way.
", "votes": null, "replies": [ { "id": 2391464, "author_name": "mtinti", "author_url": "", "post_date": "08/15/2023 07:17:54", "content": "Could also be related to the information content ic(f) of predictions?
", "votes": null, "replies": [] }, { "id": 2391863, "author_name": "biodav", "author_url": "", "post_date": "08/15/2023 11:05:44", "content": "I agree that the lower scoring predictions could affect the precision and recall in either direction if they are included. However, why would they be included in the calculated F1max score if they end up reducing the F1 score. In that case the F1max score would look at the F1 score taken at the higher threshold and automatically disregard those bad predictions. If they are good predictions, then the F1max calculation would look at the F1 score at the lower threshold and include them.
\nThe point here is that if I do the thresholding myself before I submit my predictions i.e. exclude all predictions with probabilities less than 0.5 I get a better LB score than if I leave it to the LB code to do the thresholding for me. I just can't see how that can be.
\nLooking at the evaluation code at first sight the very first thing it does is extract the predictions that are greater than the given threshold tau (which is varied from 0.01 to 0.99). All downstream calculations from there are based on that partitioning. So when it reaches 0.5, it should produce the same LB score as my pre-thresholded submission. But it doesn't seem to.
", "votes": null, "replies": [ { "id": 2392649, "author_name": "predragradivojac", "author_url": "", "post_date": "08/15/2023 19:29:35", "content": "Got it, I misunderstood the problem. I am not sure what the reason is, but there's another quirk about propagation that we have that could affect things (maybe). It is that if you propagate your scores up to the root of ontology yourself, the evaluation code does not change this, which allows for higher up terms to have lower scores (some legacy requirement we had from before). However, if you do not propagate all scores to the root, then the software will do this for you by picking the maximum score of a parent and all its children as the parent's score. I wonder if this could have some effect here.
", "votes": null, "replies": [ { "id": 2392679, "author_name": "biodav", "author_url": "", "post_date": "08/15/2023 19:52:13", "content": "Ah, light bulb moment I think. That's starting to sound like it could be the explanation.
\nIn my case I propagate the GO terms during training, but not during inference. I also predict the majority of GO terms (> 32000 of them). So it is possible that a term nearer to root could be scored less than a more specific term (it's hard to reason with large neural networks!).
\nIf I understand you correctly, when the lower scoring term is present in the submission file you don't overwrite the score. So when I remove the lower scoring terms, your script does then fill in a higher score for that label score with the higher score during propagation. Yes, I think that would do it.
\nProbably too late to fix it properly now, but I should either propagate during inference (might still be time for that) or constrain the GO term graph relationships as part of the training loss function (maybe for CAFA6!).
\nThanks for your input, Predrag!
", "votes": null, "replies": [ { "id": 2392707, "author_name": "predragradivojac", "author_url": "", "post_date": "08/15/2023 20:14:19", "content": "Sure thing. We do not propagate scores if the submitter propagates them. In previous rounds of CAFA, some groups argued that this is what they wanted and we felt it made some sense (though not complete sense). In reality, there is some uncertainty inherent to prediction scores and leaving scores as they are resolves that uncertainty for the submitters without the need to reconcile parent-child inconsistencies themselves.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2392291, "author_name": "biodav", "author_url": "", "post_date": "08/15/2023 15:21:50", "content": "Hmm. Admittedly that could create the observed effect. However, I don't think it's likely to be the case. The max number of terms I assign to a single protein at a threshold of 0.4 is 1145. Admittedly that doesn't mean that one of the ontologies couldn't be over 500. However only 296 of the 140k proteins were assigned more than 500GO terms in total, so it would be extreme bad luck if that is the explanation. I guess I'll have to give up two LB evaluations to repeating the test with those 296 terms excluded to be sure your explanation isn't the answer.
", "votes": null, "replies": [ { "id": 2392479, "author_name": "biodav", "author_url": "", "post_date": "08/15/2023 17:34:59", "content": "OK experiment done. I used another smaller submission file this time around, so lower LB scores. However, no protein had more than 500 terms assigned in total. Results: 0.38125 with pre-applied 0.5 threshold, 0.37147 for same file but with pre-applied 0.4 threshold. Smaller difference for this file, but I really think this should not be possible at all. Still puzzled…
", "votes": null, "replies": [] } ] } ] }, { "id": 2391948, "author_name": "alexandervc", "author_url": "", "post_date": "08/15/2023 11:40:25", "content": "Another try:
\nThere is cutoff of 1500 predictions.
\nIt seems it might be done WITHOUT ordering the predictions.
\nSo if you send say 10_000 predictions - per protein
\nit might that cutoff will select RANDOM 1500 out these 10_000
\nso scores - go down
Thus if you put 0.5 threshold by yourself it is very plausible that no protein will have more than 1500 predictions,
\nand that problem will not arise.
More precisely cutoff is 500 for each ontology, not overall 1500.
", "votes": null, "replies": [] }, { "id": 2392722, "author_name": "tomorrowdawn", "author_url": "", "post_date": "08/15/2023 20:38:37", "content": "Yes, I had the same behavior.
\nMy idea was to submit as many functions as possible even with small scores, and thus I have selected really small threshold. Recieved ~0.02 leaderboard score
\nAfterwards I have returned 0.1 threshold, launched the same script -- and the result was ~0.35 leaderboard score
\nI do not think that this behavior is a wanted one especially due to the fact that threshold-free metrics such as ROC-AUC or PRC-AUC can be nicely computed on the full dataset. At least I do that for 10% data without any problem every epoch for all the samples (140k) and all the classes (32k)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2390092": "I'm sure I'll end up face-palming when I get the answer to this, but I'm puzzled as to how my leaderboard score tends to increase when I submit fewer predictions. Let me explain. I have a large submission file generated by a neural net model (that doesn't really matter). Only predictions with probabilities >= 0.4 are included. I submit it and get a score of 0.48 - not bad not great. That score as I understand it is the average of the maximum F1 scores found for the 3 ontologies in my submitted predictions. OK, then I remove all the lowest scoring predictions (p < 0.5) just by using grep and resubmit that smaller file to the leaderboard. I then get a better LB score of 0.5 - still not great but better. How can that be? Surely the F1max process should have ignored the lower scoring predictions the first time around - how can removing the lower confidence predictions give me a better F1max score? I would expect the score to have stayed the same or gone down, but not up.\n\nWhat am I missing here?", "2390536": "Can that be somehow caused by \"propagation\" ?\n\nOtherwise if there is no bug in metric code of F1 it should not happen, as far as I understand at least for the moment.", "2390629": "Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?", "2390646": "May be propagation causes False positives, which are absent with cutoff 0.5, can it be ? \n\n> Well, the propagation aspect was something that came to my mind also - mainly because it's the \"fuzziest\" part of the whole calculation. Still, I can't see an obvious mechanism (aside from a bug) that could reduce the F1max - surely any terms that are propagated upwards cannot have a score higher than the new lower threshold of 0.5, so how could the expanded terms have been counted as false positives to reduce the F1max scores?", "2390663": "Surely any terms that are propagated would still have their original scores i.e. scores < 0.5, so when the F1 threshold is 0.5 then all of those extra terms would be excluded from the calculation? At least that's how I would code it.", "2390715": "Yep, you are right , hmm \nLet me think. In my notebooks there's local calculation,so can be checked there , but it is a bit tricky interfaced", "2391048": "I think it is possible that lower scores increase the performance. Imagine you only have 1 additional prediction term in one new protein (and then propagate it) at score 0.4 and that this prediction is exactly correct, so the whole path to the root is correct. Everything else is the same and at scores 0.5 or higher. Then, when you add this protein within the F_max formula it will be adding 1 for both precision and recall into the averaging formula over all proteins. Thus, it will increase F_max averaged over all proteins.", "2391063": "I should that now, imagine the 0.4 branch was all wrong instead of all right. In this case, it will reduce the performance because both precision and recall will be 0. Basically, F_max can go either way.", "2391464": "Could also be related to the information content ic(f) of predictions?", "2391863": "I agree that the lower scoring predictions could affect the precision and recall in either direction **if** they are included. However, why would they be included in the calculated F1**max** score if they end up reducing the F1 score. In that case the F1max score would look at the F1 score taken at the higher threshold and automatically disregard those bad predictions. If they are good predictions, then the F1max calculation would look at the F1 score at the lower threshold and include them.\n\nThe point here is that if I do the thresholding myself before I submit my predictions i.e. exclude all predictions with probabilities less than 0.5 I get a better LB score than if I leave it to the LB code to do the thresholding for me. I just can't see how that can be.\n\nLooking at the evaluation code at first sight the very first thing it does is extract the predictions that are greater than the given threshold tau (which is varied from 0.01 to 0.99). All downstream calculations from there are based on that partitioning. So when it reaches 0.5, it should produce the same LB score as my pre-thresholded submission. But it doesn't seem to.", "2391948": "Another try: \n\nThere is cutoff of 1500 predictions. \nIt seems it might be done WITHOUT ordering the predictions.\nSo if you send say 10_000 predictions - per protein\nit might that cutoff will select RANDOM 1500 out these 10_000\nso scores - go down \n\nThus if you put 0.5 threshold by yourself it is very plausible that no protein will have more than 1500 predictions,\nand that problem will not arise. \n\n-----\n\nMore precisely cutoff is 500 for each ontology, not overall 1500.", "2392291": "Hmm. Admittedly that could create the observed effect. However, I don't think it's likely to be the case. The max number of terms I assign to a single protein at a threshold of 0.4 is 1145. Admittedly that doesn't mean that one of the ontologies couldn't be over 500. However only 296 of the 140k proteins were assigned more than 500GO terms in total, so it would be extreme bad luck if that is the explanation. I guess I'll have to give up two LB evaluations to repeating the test with those 296 terms excluded to be sure your explanation isn't the answer.", "2392479": "OK experiment done. I used another smaller submission file this time around, so lower LB scores. However, no protein had more than 500 terms assigned in total. Results: 0.38125 with pre-applied 0.5 threshold, 0.37147 for same file but with pre-applied 0.4 threshold. Smaller difference for this file, but I really think this should not be possible at all. Still puzzled...", "2392649": "Got it, I misunderstood the problem. I am not sure what the reason is, but there's another quirk about propagation that we have that could affect things (maybe). It is that if you propagate your scores up to the root of ontology yourself, the evaluation code does not change this, which allows for higher up terms to have lower scores (some legacy requirement we had from before). However, if you do not propagate all scores to the root, then the software will do this for you by picking the maximum score of a parent and all its children as the parent's score. I wonder if this could have some effect here.", "2392679": "Ah, light bulb moment I think. That's starting to sound like it could be the explanation.\n\nIn my case I propagate the GO terms during training, but not during inference. I also predict the majority of GO terms (> 32000 of them). So it is possible that a term nearer to root could be scored less than a more specific term (it's hard to reason with large neural networks!).\n\nIf I understand you correctly, when the lower scoring term is present in the submission file you don't overwrite the score. So when I remove the lower scoring terms, your script does then fill in a higher score for that label score with the higher score during propagation. Yes, I think that would do it.\n\nProbably too late to fix it properly now, but I should either propagate during inference (might still be time for that) or constrain the GO term graph relationships as part of the training loss function (maybe for CAFA6!).\n\nThanks for your input, Predrag!", "2392707": "Sure thing. We do not propagate scores if the submitter propagates them. In previous rounds of CAFA, some groups argued that this is what they wanted and we felt it made some sense (though not complete sense). In reality, there is some uncertainty inherent to prediction scores and leaving scores as they are resolves that uncertainty for the submitters without the need to reconcile parent-child inconsistencies themselves.", "2392722": "Yes, I had the same behavior.\n\nMy idea was to submit as many functions as possible even with small scores, and thus I have selected really small threshold. Recieved ~0.02 leaderboard score\n\nAfterwards I have returned 0.1 threshold, launched the same script -- and the result was ~0.35 leaderboard score\n\nI do not think that this behavior is a wanted one especially due to the fact that threshold-free metrics such as ROC-AUC or PRC-AUC can be nicely computed on the full dataset. At least I do that for 10% data without any problem every epoch for all the samples (140k) and all the classes (32k)" }, "source": "meta" }