Hi, everyone!
\nI'm new to kaggle and just started this competition today. Can someone please explain to me whether the precision in weighted F-measures (for MF, BP, and CC) is always going to be one since there is no negative label but absent label (Positive-unlabeled learning according to https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447)? I am just wondering if I should predict 1500 GO for every test protein sequence… Big thanks!
Hi, everyone!
\nI'm new to kaggle and just started this competition today. Can someone please explain to me whether the precision in weighted F-measures (for MF, BP, and CC) is always going to be one since there is no negative label but absent label (Positive-unlabeled learning according to https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447)? I am just wondering if I should predict 1500 GO for every test protein sequence… Big thanks!
In the Positive-Unlabeled learning setup for this competition, the precision will not always be one. Predicting all 1500 GO terms for each test protein sequence would increase false positives, lowering precision. Instead, focus on developing a model that accurately identifies relevant GO terms while balancing precision and recall to achieve a high F-measure score. Good luck!
", "rawMarkdown": "In the Positive-Unlabeled learning setup for this competition, the precision will not always be one. Predicting all 1500 GO terms for each test protein sequence would increase false positives, lowering precision. Instead, focus on developing a model that accurately identifies relevant GO terms while balancing precision and recall to achieve a high F-measure score. Good luck!", "votes": null }, { "id": "2248767", "postDate": "05/07/2023 07:01:10", "content": "Thank you so much for the reply, but I still don't know where does the false positive come from since we don't actually know if a protein is going to be labeled certain GO in the future. One of the discussion said it's not a negative label but an absence of label here =>(https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447).
", "rawMarkdown": "Thank you so much for the reply, but I still don't know where does the false positive come from since we don't actually know if a protein is going to be labeled certain GO in the future. One of the discussion said it's not a negative label but an absence of label here =>(https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447).", "votes": null }, { "id": "2248904", "postDate": "05/07/2023 10:30:28", "content": "The data itself is positive-unlabeled, i.e., there are certainly some positive labels we do not yet know about. However, for the evaluation each of those missing labels is assumed to be a negative label. Thus, even if you correctly predict an unknown positive label, it would count as a false positive as long as it is not annotated as such in the test set. At least that is how I understand the task.
", "rawMarkdown": "The data itself is positive-unlabeled, i.e., there are certainly some positive labels we do not yet know about. However, for the evaluation each of those missing labels is assumed to be a negative label. Thus, even if you correctly predict an unknown positive label, it would count as a false positive as long as it is not annotated as such in the test set. At least that is how I understand the task.", "votes": null }, { "id": "2249713", "postDate": "05/08/2023 04:00:31", "content": "Thank you I got it now! This is so fascinating and challenging!
", "rawMarkdown": "Thank you I got it now! This is so fascinating and challenging!", "votes": null }, { "id": "2256455", "postDate": "05/12/2023 13:55:39", "content": "does this mean we're actually \"predicting protein function + probability of someone actually researching this protein\"?
", "rawMarkdown": "does this mean we're actually \"predicting protein function + probability of someone actually researching this protein\"?", "votes": null }, { "id": "2256494", "postDate": "05/12/2023 14:22:44", "content": "\n\n\"predicting protein function + probability of someone actually researching this protein\"
\n
In a way, yes. Though it gets more complicated than that.
\nMany proteins in the data sets have already been researched. After all, many already have (some) GO annotations based on experiments. The ones that are crucial for the final evaluations are those proteins that are being researched \"right now\" and for which results will be published in the relevant time frame. Further, it would only be about the specific properties (CCO, BPO, MFO) being researched, which might only be a subset of those the protein actually has.
\nThat being said, trying to cram the part about \"probability of someone actually researching this protein\" into the model isn't necessarily helping the goal of CAFA; making a well performing general model is. And even though this might lead to incorrectly attributed \"false positives\", this issue would affect all the other participants, too.
", "rawMarkdown": "> \"predicting protein function + probability of someone actually researching this protein\"\n\nIn a way, yes. Though it gets more complicated than that.\n\nMany proteins in the data sets have already been researched. After all, many already have (some) GO annotations based on experiments. The ones that are crucial for the final evaluations are those proteins that are being researched \"right now\" and for which results will be published in the relevant time frame. Further, it would only be about the specific properties (CCO, BPO, MFO) being researched, which might only be a subset of those the protein actually has.\n\nThat being said, trying to cram the part about \"probability of someone actually researching this protein\" into the model isn't necessarily helping the goal of CAFA; making a well performing general model is. And even though this might lead to incorrectly attributed \"false positives\", this issue would affect all the other participants, too.", "votes": null } ], "comments": [ { "id": 2248507, "author_name": "thaweewatboy", "author_url": "", "post_date": "05/06/2023 23:17:17", "content": "In the Positive-Unlabeled learning setup for this competition, the precision will not always be one. Predicting all 1500 GO terms for each test protein sequence would increase false positives, lowering precision. Instead, focus on developing a model that accurately identifies relevant GO terms while balancing precision and recall to achieve a high F-measure score. Good luck!
", "votes": null, "replies": [ { "id": 2248767, "author_name": "andrewtsai0406", "author_url": "", "post_date": "05/07/2023 07:01:10", "content": "Thank you so much for the reply, but I still don't know where does the false positive come from since we don't actually know if a protein is going to be labeled certain GO in the future. One of the discussion said it's not a negative label but an absence of label here =>(https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447).
", "votes": null, "replies": [ { "id": 2248904, "author_name": "norixmb", "author_url": "", "post_date": "05/07/2023 10:30:28", "content": "The data itself is positive-unlabeled, i.e., there are certainly some positive labels we do not yet know about. However, for the evaluation each of those missing labels is assumed to be a negative label. Thus, even if you correctly predict an unknown positive label, it would count as a false positive as long as it is not annotated as such in the test set. At least that is how I understand the task.
", "votes": null, "replies": [ { "id": 2249713, "author_name": "andrewtsai0406", "author_url": "", "post_date": "05/08/2023 04:00:31", "content": "Thank you I got it now! This is so fascinating and challenging!
", "votes": null, "replies": [] }, { "id": 2256455, "author_name": "drorspeiser", "author_url": "", "post_date": "05/12/2023 13:55:39", "content": "does this mean we're actually \"predicting protein function + probability of someone actually researching this protein\"?
", "votes": null, "replies": [ { "id": 2256494, "author_name": "norixmb", "author_url": "", "post_date": "05/12/2023 14:22:44", "content": "\n\n\"predicting protein function + probability of someone actually researching this protein\"
\n
In a way, yes. Though it gets more complicated than that.
\nMany proteins in the data sets have already been researched. After all, many already have (some) GO annotations based on experiments. The ones that are crucial for the final evaluations are those proteins that are being researched \"right now\" and for which results will be published in the relevant time frame. Further, it would only be about the specific properties (CCO, BPO, MFO) being researched, which might only be a subset of those the protein actually has.
\nThat being said, trying to cram the part about \"probability of someone actually researching this protein\" into the model isn't necessarily helping the goal of CAFA; making a well performing general model is. And even though this might lead to incorrectly attributed \"false positives\", this issue would affect all the other participants, too.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2248197": "Hi, everyone! \nI'm new to kaggle and just started this competition today. Can someone please explain to me whether the precision in weighted F-measures (for MF, BP, and CC) is always going to be one since there is no negative label but absent label (Positive-unlabeled learning according to https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447)? I am just wondering if I should predict 1500 GO for every test protein sequence… Big thanks!", "2248507": "In the Positive-Unlabeled learning setup for this competition, the precision will not always be one. Predicting all 1500 GO terms for each test protein sequence would increase false positives, lowering precision. Instead, focus on developing a model that accurately identifies relevant GO terms while balancing precision and recall to achieve a high F-measure score. Good luck!", "2248767": "Thank you so much for the reply, but I still don't know where does the false positive come from since we don't actually know if a protein is going to be labeled certain GO in the future. One of the discussion said it's not a negative label but an absence of label here =>(https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/402696#2227447).", "2248904": "The data itself is positive-unlabeled, i.e., there are certainly some positive labels we do not yet know about. However, for the evaluation each of those missing labels is assumed to be a negative label. Thus, even if you correctly predict an unknown positive label, it would count as a false positive as long as it is not annotated as such in the test set. At least that is how I understand the task.", "2249713": "Thank you I got it now! This is so fascinating and challenging!", "2256455": "does this mean we're actually \"predicting protein function + probability of someone actually researching this protein\"?", "2256494": "> \"predicting protein function + probability of someone actually researching this protein\"\n\nIn a way, yes. Though it gets more complicated than that.\n\nMany proteins in the data sets have already been researched. After all, many already have (some) GO annotations based on experiments. The ones that are crucial for the final evaluations are those proteins that are being researched \"right now\" and for which results will be published in the relevant time frame. Further, it would only be about the specific properties (CCO, BPO, MFO) being researched, which might only be a subset of those the protein actually has.\n\nThat being said, trying to cram the part about \"probability of someone actually researching this protein\" into the model isn't necessarily helping the goal of CAFA; making a well performing general model is. And even though this might lead to incorrectly attributed \"false positives\", this issue would affect all the other participants, too." }, "source": "meta" }