https://www.kaggle.com/code/sergeifironov/duplicates-train-test/notebook?scriptVersionId=126699747
\nIt seems that half of the test IDs are present in the train. How to process it correctly? Was it meant to be like this?
", "messages": [ { "id": 2229848, "postDate": "2023-04-21T18:51:24.380Z", "content": "https://www.kaggle.com/code/sergeifironov/duplicates-train-test/notebook?scriptVersionId=126699747
\nIt seems that half of the test IDs are present in the train. How to process it correctly? Was it meant to be like this?
", "rawMarkdown": "https://www.kaggle.com/code/sergeifironov/duplicates-train-test/notebook?scriptVersionId=126699747\n\nIt seems that half of the test IDs are present in the train. How to process it correctly? Was it meant to be like this?", "votes": 16 }, { "id": 2230488, "postDate": "2023-04-22T12:12:59.780Z", "content": "So let's take, for example, this protein id found only in the test set: A0A1C3YKE0. The QuickGO annotation page (https://www.ebi.ac.uk/QuickGO/annotations?geneProductId=A0A1C3YKE0) reports two GO annotations, GO:0003824 and GO:0016740. These two annotations do not have experimentally assigned evidence. I'm correct in thinking that the GO:0003824 and GO:0016740 are not scored for A0A1C3YKE0, unless a curator adds experimental evidence for GO:0003824 or GO:0016740?
", "rawMarkdown": "So let's take, for example, this protein id found only in the test set: A0A1C3YKE0. The QuickGO annotation page (https://www.ebi.ac.uk/QuickGO/annotations?geneProductId=A0A1C3YKE0) reports two GO annotations, GO:0003824 and GO:0016740. These two annotations do not have experimentally assigned evidence. I'm correct in thinking that the GO:0003824 and GO:0016740 are not scored for A0A1C3YKE0, unless a curator adds experimental evidence for GO:0003824 or GO:0016740?", "votes": 3, "replies": [ { "id": 2230912, "postDate": "2023-04-22T22:00:19.987Z", "content": "Yes, you are correct. The scoring is only on experimentally validated GO-terms.
", "rawMarkdown": "Yes, you are correct. The scoring is only on experimentally validated GO-terms. ", "votes": 2, "replies": [ { "id": 2234045, "postDate": "2023-04-24T19:58:45.143Z", "content": "@iddux1 - two (hopefully quick to answer) questions from me to try to clarify this:
\n[1] Consider a protein that currently has a GO term annotation that appears in databases, but the annotation is for now inferred without direct experimental evidence. If during the scoring period that annotation acquires direct experimental support, will it count for scoring?
\n[2] What database or other data source will be considered definitive for providing the final new annotations for scoring?
", "rawMarkdown": "@iddux1 - two (hopefully quick to answer) questions from me to try to clarify this:\n\n[1] Consider a protein that currently has a GO term annotation that appears in databases, but the annotation is for now inferred without direct experimental evidence. If during the scoring period that annotation acquires direct experimental support, will it count for scoring?\n\n[2] What database or other data source will be considered definitive for providing the final new annotations for scoring?", "votes": 1 }, { "id": 2235075, "postDate": "2023-04-25T17:50:28.690Z", "content": "Thanks, that's now clear!
", "rawMarkdown": "Thanks, that's now clear!", "votes": 1 } ] } ] }, { "id": 2317606, "postDate": "2023-06-25T19:56:35.460Z", "content": "could there theoretically be a possibility that entry had a go term annotation in the train set, but some later experiments show that that annotation was spurious, and the annotation was removed? If so will such removals be included in the test set?
", "rawMarkdown": "could there theoretically be a possibility that entry had a go term annotation in the train set, but some later experiments show that that annotation was spurious, and the annotation was removed? If so will such removals be included in the test set?\n", "votes": 1, "replies": [ { "id": 2317643, "postDate": "2023-06-25T20:52:56.240Z", "content": "https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406914
", "rawMarkdown": "https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406914", "votes": 2 } ] }, { "id": 2229874, "postDate": "2023-04-21T19:25:54.910Z", "content": "It was, indeed. The idea is that only proteins that get new annotations will go into the final test set.
", "rawMarkdown": "It was, indeed. The idea is that only proteins that get new annotations will go into the final test set. ", "votes": 1, "replies": [ { "id": 2229878, "postDate": "2023-04-21T19:33:08.853Z", "content": "Ah, ok! Thanks
", "rawMarkdown": "Ah, ok! Thanks" }, { "id": 2229886, "postDate": "2023-04-21T19:40:24.180Z", "content": "What is the reason for duplicates ?
\nIt is quite unusual for data science. Usually called data leak.
Current leader board is calculated including these duplicates ?
\nOr excluding them?
Annotations for the duplicates are exactly the same jn the train and the test ?
\n", "rawMarkdown": "What is the reason for duplicates ?\nIt is quite unusual for data science. Usually called data leak. \n\nCurrent leader board is calculated including these duplicates ?\nOr excluding them?\n\nAnnotations for the duplicates are exactly the same jn the train and the test ?\n\n@iddux1\n" }, { "id": 2229887, "postDate": "2023-04-21T19:40:36.127Z", "content": "No, I'm sorry, I don't understand again. Do I understand correctly that we predict new properties for them? Or can we just copy the labels from the training set?
", "rawMarkdown": "No, I'm sorry, I don't understand again. Do I understand correctly that we predict new properties for them? Or can we just copy the labels from the training set?", "votes": 1, "replies": [ { "id": 2229920, "postDate": "2023-04-21T20:17:10.037Z", "content": "The final score will be on those sequences in the test_superset that will gain new experimentally validated annotations between the submission date (late August ) and the final evaluation date (December).
\nExample: a protein had no experimentally validated MFO annotations in August, but gained new annotations between August and December, the models will be evaluated on that protein, within the MFO sub-ontology.
\n\n\nDo I understand correctly that we predict new properties for them?
\n
You are evaluated only a what you called \"new properties\", which are new GO terms that are experimentally validated in one or more GO sub-ontologies (MFO, BPO, and CCO).
\n\n\nOr can we just copy the labels from the training set?
\n
You can, but you will not be evaluated on those.
", "rawMarkdown": "The final score will be on those sequences in the test_superset that will gain new [**experimentally validated**](http://geneontology.org/docs/guide-go-evidence-codes/) annotations between the submission date (late August ) and the final evaluation date (December). \n\nExample: a protein had no **experimentally validated** MFO annotations in August, but gained new annotations between August and December, the models will be evaluated on that protein, within the MFO sub-ontology. \n\n> Do I understand correctly that we predict new properties for them?\n\nYou are evaluated only a what you called \"new properties\", which are new GO terms that are experimentally validated in one or more GO sub-ontologies (MFO, BPO, and CCO). \n\n> Or can we just copy the labels from the training set?\n\nYou can, but you will not be evaluated on those. ", "votes": 5 }, { "id": 2229925, "postDate": "2023-04-21T20:21:25.603Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2229926, "postDate": "2023-04-21T20:21:42.613Z", "content": "Ok, It makes sense. Thanks, it's clear now.
", "rawMarkdown": "Ok, It makes sense. Thanks, it's clear now. " }, { "id": 2229930, "postDate": "2023-04-21T20:22:54.310Z", "content": "Think of it like predicting housing prices or sports teams' rankings: only the future gives us the ground-truth data.
", "rawMarkdown": "Think of it like predicting housing prices or sports teams' rankings: only the future gives us the ground-truth data. " }, { "id": 2229935, "postDate": "2023-04-21T20:35:10.237Z", "content": "But how the current leaderboard is calculated ?
", "rawMarkdown": "But how the current leaderboard is calculated ? ", "votes": 1 }, { "id": 2229944, "postDate": "2023-04-21T20:49:06.193Z", "content": "\n\nBut how the current leaderboard is calculated ?
\n
Confidential data provided to us by the Uniprot annotation team.
", "rawMarkdown": "> But how the current leaderboard is calculated ? \n\nConfidential data provided to us by the Uniprot annotation team. ", "votes": 2 }, { "id": 2230403, "postDate": "2023-04-22T10:40:27.290Z", "content": "Is it correct: that current leaderboard is evaluated ONLY on the PART of the provided test samples ?
\nThe train-test duplicates are NOT in that part ? Is it correct ?
", "rawMarkdown": "Is it correct: that current leaderboard is evaluated ONLY on the PART of the provided test samples ? \n\nThe train-test duplicates are NOT in that part ? Is it correct ? \n\n", "votes": 3 }, { "id": 2230565, "postDate": "2023-04-22T14:01:35.380Z", "content": "Same question, because it should be that way for it to make sense, unless the duplicate sequences have already acquired new experimentally validated annotations between the time the host received the train set and a later time when they had the entire test_super_set/(but If duplicates are included in the evaluation + have the same annotations as the train set, then roughly 50% of the current leaderboard (until August 2023) is data leakage)
", "rawMarkdown": "Same question, because it should be that way for it to make sense, unless the duplicate sequences have already acquired new experimentally validated annotations between the time the host received the train set and a later time when they had the entire test_super_set/(but If duplicates are included in the evaluation **+** have the same annotations as the train set, then roughly 50% of the current leaderboard (until August 2023) is data leakage)", "votes": 2 }, { "id": 2250848, "postDate": "2023-05-08T20:50:08.137Z", "content": "So is it possible that we have XXX unique GO ID values in the training set, but there may be new GO ID values in the test set (ones that were not present in the training set)?
", "rawMarkdown": "So is it possible that we have XXX unique GO ID values in the training set, but there may be new GO ID values in the test set (ones that were not present in the training set)?", "votes": 2 } ] } ] }, { "id": 2383523, "postDate": "2023-08-10T12:15:10.680Z", "content": "I'm a little bit confused by the \"new annotations\".
\nIf a protein already has some annotations on MF and CC, but after the submission deadline gains new annotations on both MF and CC, will it be included in the MF and CC as well? If so, we should directly include all the annotations with score 1 before the submission deadline to get a higher overall score.🤔
", "rawMarkdown": "I'm a little bit confused by the \"new annotations\". \n\nIf a protein already has some annotations on MF and CC, but after the submission deadline gains new annotations on both MF and CC, will it be included in the MF and CC as well? If so, we should directly include all the annotations with score 1 before the submission deadline to get a higher overall score.🤔", "replies": [ { "id": 2384114, "postDate": "2023-08-10T20:06:49.637Z", "content": "See https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406914
", "rawMarkdown": "See https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406914", "votes": 1 } ] } ], "comments": [ { "id": 2230488, "author_name": "MT", "author_url": "", "post_date": "2023-04-22T12:12:59.780000", "content": "So let's take, for example, this protein id found only in the test set: A0A1C3YKE0. The QuickGO annotation page (https://www.ebi.ac.uk/QuickGO/annotations?geneProductId=A0A1C3YKE0) reports two GO annotations, GO:0003824 and GO:0016740. These two annotations do not have experimentally assigned evidence. I'm correct in thinking that the GO:0003824 and GO:0016740 are not scored for A0A1C3YKE0, unless a curator adds experimental evidence for GO:0003824 or GO:0016740?
", "votes": 3, "replies": [ { "id": 2230912, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-22T22:00:19.987000", "content": "Yes, you are correct. The scoring is only on experimentally validated GO-terms.
", "votes": 2, "replies": [ { "id": 2234045, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-04-24T19:58:45.143000", "content": "@iddux1 - two (hopefully quick to answer) questions from me to try to clarify this:
\n[1] Consider a protein that currently has a GO term annotation that appears in databases, but the annotation is for now inferred without direct experimental evidence. If during the scoring period that annotation acquires direct experimental support, will it count for scoring?
\n[2] What database or other data source will be considered definitive for providing the final new annotations for scoring?
", "votes": 1, "replies": [] }, { "id": 2235075, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-25T17:50:28.690000", "content": "Thanks, that's now clear!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2317606, "author_name": "h4shbang", "author_url": "", "post_date": "2023-06-25T19:56:35.460000", "content": "could there theoretically be a possibility that entry had a go term annotation in the train set, but some later experiments show that that annotation was spurious, and the annotation was removed? If so will such removals be included in the test set?
", "votes": 1, "replies": [ { "id": 2317643, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-06-25T20:52:56.240000", "content": "https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406914
", "votes": 2, "replies": [] } ] }, { "id": 2229874, "author_name": "MT", "author_url": "", "post_date": "2023-04-21T19:25:54.910000", "content": "It was, indeed. The idea is that only proteins that get new annotations will go into the final test set.
", "votes": 1, "replies": [ { "id": 2229878, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2023-04-21T19:33:08.853000", "content": "Ah, ok! Thanks
", "votes": 0, "replies": [] }, { "id": 2229886, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-04-21T19:40:24.180000", "content": "What is the reason for duplicates ?
\nIt is quite unusual for data science. Usually called data leak.
Current leader board is calculated including these duplicates ?
\nOr excluding them?
Annotations for the duplicates are exactly the same jn the train and the test ?
\n", "votes": 0, "replies": [] }, { "id": 2229887, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2023-04-21T19:40:36.127000", "content": "No, I'm sorry, I don't understand again. Do I understand correctly that we predict new properties for them? Or can we just copy the labels from the training set?
", "votes": 1, "replies": [ { "id": 2229920, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-21T20:17:10.037000", "content": "The final score will be on those sequences in the test_superset that will gain new experimentally validated annotations between the submission date (late August ) and the final evaluation date (December).
\nExample: a protein had no experimentally validated MFO annotations in August, but gained new annotations between August and December, the models will be evaluated on that protein, within the MFO sub-ontology.
\n\n\nDo I understand correctly that we predict new properties for them?
\n
You are evaluated only a what you called \"new properties\", which are new GO terms that are experimentally validated in one or more GO sub-ontologies (MFO, BPO, and CCO).
\n\n\nOr can we just copy the labels from the training set?
\n
You can, but you will not be evaluated on those.
", "votes": 5, "replies": [] }, { "id": 2229925, "author_name": "", "author_url": "", "post_date": "2023-04-21T20:21:25.603000", "content": "", "votes": 0, "replies": [] }, { "id": 2229926, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2023-04-21T20:21:42.613000", "content": "Ok, It makes sense. Thanks, it's clear now.
", "votes": 0, "replies": [] }, { "id": 2229930, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-21T20:22:54.310000", "content": "Think of it like predicting housing prices or sports teams' rankings: only the future gives us the ground-truth data.
", "votes": 0, "replies": [] }, { "id": 2229935, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-04-21T20:35:10.237000", "content": "But how the current leaderboard is calculated ?
", "votes": 1, "replies": [] }, { "id": 2229944, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-21T20:49:06.193000", "content": "\n\nBut how the current leaderboard is calculated ?
\n
Confidential data provided to us by the Uniprot annotation team.
", "votes": 2, "replies": [] }, { "id": 2230403, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-04-22T10:40:27.290000", "content": "Is it correct: that current leaderboard is evaluated ONLY on the PART of the provided test samples ?
\nThe train-test duplicates are NOT in that part ? Is it correct ?
", "votes": 3, "replies": [] }, { "id": 2230565, "author_name": "", "author_url": "", "post_date": "2023-04-22T14:01:35.380000", "content": "Same question, because it should be that way for it to make sense, unless the duplicate sequences have already acquired new experimentally validated annotations between the time the host received the train set and a later time when they had the entire test_super_set/(but If duplicates are included in the evaluation + have the same annotations as the train set, then roughly 50% of the current leaderboard (until August 2023) is data leakage)
", "votes": 2, "replies": [] }, { "id": 2250848, "author_name": "Michalina Hulak", "author_url": "", "post_date": "2023-05-08T20:50:08.137000", "content": "So is it possible that we have XXX unique GO ID values in the training set, but there may be new GO ID values in the test set (ones that were not present in the training set)?
", "votes": 2, "replies": [] } ] } ] }, { "id": 2383523, "author_name": "Dewei Hu", "author_url": "", "post_date": "2023-08-10T12:15:10.680000", "content": "I'm a little bit confused by the \"new annotations\".
\nIf a protein already has some annotations on MF and CC, but after the submission deadline gains new annotations on both MF and CC, will it be included in the MF and CC as well? If so, we should directly include all the annotations with score 1 before the submission deadline to get a higher overall score.🤔
", "votes": 0, "replies": [ { "id": 2384114, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-08-10T20:06:49.637000", "content": "See https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/discussion/406914
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2229848": "https://www.kaggle.com/code/sergeifironov/duplicates-train-test/notebook?scriptVersionId=126699747\n\nIt seems that half of the test IDs are present in the train. How to process it correctly? Was it meant to be like this?", "2230488": "So let's take, for example, this protein id found only in the test set: A0A1C3YKE0. The QuickGO annotation page (https://www.ebi.ac.uk/QuickGO/annotations?geneProductId=A0A1C3YKE0) reports two GO annotations, GO:0003824 and GO:0016740. These two annotations do not have experimentally assigned evidence. I'm correct in thinking that the GO:0003824 and GO:0016740 are not scored for A0A1C3YKE0, unless a curator adds experimental evidence for GO:0003824 or GO:0016740?", "2317606": "could there theoretically be a possibility that entry had a go term annotation in the train set, but some later experiments show that that annotation was spurious, and the annotation was removed? If so will such removals be included in the test set?\n", "2229874": "It was, indeed. The idea is that only proteins that get new annotations will go into the final test set. ", "2383523": "I'm a little bit confused by the \"new annotations\". \n\nIf a protein already has some annotations on MF and CC, but after the submission deadline gains new annotations on both MF and CC, will it be included in the MF and CC as well? If so, we should directly include all the annotations with score 1 before the submission deadline to get a higher overall score.🤔" } }