{ "id": 406914, "title": "At the final evaluation - annotations can only be ADDED while existing (high-confident) will be the same, or some can also \"disappear\"(not-confirmed) ?", "url": "/competitions/cafa-5-protein-function-prediction/discussion/406914", "author_name": "", "post_date": "2023-05-04T11:13:29.180705300Z", "votes": 13, "comment_count": 2, "views": 0, "content": "

Thanks to organizers and Kaggle for the great challenge !

\n

May I clarify certain point:

\n

At the final evaluation - annotations can only be ADDED to existing high-confidence ones, or some \"high-confidence\" can also \"disappear\" (not confirmed by new experimental data ) ?

\n

By \"high-confidence\" I mean those mentioned in the rules of the competition:
\n\"validated by experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC). More information about evidence codes can be found here. \"

\n

In particular :
\nthere are duplicates between train and test - so as far as I understand the point for there existence - that some of them will be experimentally retested and annotations can change. From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ?

\n

How that question was resolved on the previous CAFA competitions ?

\n

@iddux1

", "messages": [ { "id": "2245428", "postDate": "05/04/2023 11:13:29", "content": "

Thanks to organizers and Kaggle for the great challenge !

\n

May I clarify certain point:

\n

At the final evaluation - annotations can only be ADDED to existing high-confidence ones, or some \"high-confidence\" can also \"disappear\" (not confirmed by new experimental data ) ?

\n

By \"high-confidence\" I mean those mentioned in the rules of the competition:
\n\"validated by experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC). More information about evidence codes can be found here. \"

\n

In particular :
\nthere are duplicates between train and test - so as far as I understand the point for there existence - that some of them will be experimentally retested and annotations can change. From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ?

\n

How that question was resolved on the previous CAFA competitions ?

\n

@iddux1

", "rawMarkdown": "Thanks to organizers and Kaggle for the great challenge ! \n\nMay I clarify certain point: \n\nAt the final evaluation - annotations can only be ADDED to existing high-confidence ones, or some \"high-confidence\" can also \"disappear\" (not confirmed by new experimental data ) ?\n\nBy \"high-confidence\" I mean those mentioned in the rules of the competition: \n\"validated by experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC). More information about evidence codes can be found here. \"\n\n\nIn particular : \nthere are duplicates between train and test - so as far as I understand the point for there existence - that some of them will be experimentally retested and annotations can change. From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ? \n\nHow that question was resolved on the previous CAFA competitions ? \n\n@iddux1", "votes": null }, { "id": "2263486", "postDate": "05/17/2023 16:21:21", "content": "

Dear organizers @predragradivojac @parnaljoshi @claradepaolis @damianopiovesan
\nor colleagues who participated in previous CAFA @coolmaksat @danofer

\n

Would you be so kind to comment on the question…

", "rawMarkdown": "Dear organizers @predragradivojac @parnaljoshi @claradepaolis @damianopiovesan \nor colleagues who participated in previous CAFA @coolmaksat @danofer \n\nWould you be so kind to comment on the question...", "votes": null }, { "id": "2263563", "postDate": "05/17/2023 17:26:33", "content": "

Theoretically, experimentally verified predictions may go away if the biocurator made an error in interpreting the paper, or the paper was redacted, or new evidence came up that contradicted previous evidence. The first two cases this needs to be discovered somehow down the line. I don't ever recall that happening. The third case may happen, but that would be a replacement of X by Y, and will be easily detected. I don't recall that happening either.

\n
\n

From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ?

\n
\n

Yes.

\n
\n

How that question was resolved on the previous CAFA competitions ?

\n
\n

It never came up.

", "rawMarkdown": "Theoretically, experimentally verified predictions may go away if the biocurator made an error in interpreting the paper, or the paper was redacted, or new evidence came up that contradicted previous evidence. The first two cases this needs to be discovered somehow down the line. I don't ever recall that happening. The third case may happen, but that would be a replacement of X by Y, and will be easily detected. I don't recall that happening either.\n\n>From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ?\n\n\n\nYes.\n\n>How that question was resolved on the previous CAFA competitions ? \n\nIt never came up.", "votes": null } ], "comments": [ { "id": 2263486, "author_name": "alexandervc", "author_url": "", "post_date": "05/17/2023 16:21:21", "content": "

Dear organizers @predragradivojac @parnaljoshi @claradepaolis @damianopiovesan
\nor colleagues who participated in previous CAFA @coolmaksat @danofer

\n

Would you be so kind to comment on the question…

", "votes": null, "replies": [] }, { "id": 2263563, "author_name": "iddux1", "author_url": "", "post_date": "05/17/2023 17:26:33", "content": "

Theoretically, experimentally verified predictions may go away if the biocurator made an error in interpreting the paper, or the paper was redacted, or new evidence came up that contradicted previous evidence. The first two cases this needs to be discovered somehow down the line. I don't ever recall that happening. The third case may happen, but that would be a replacement of X by Y, and will be easily detected. I don't recall that happening either.

\n
\n

From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ?

\n
\n

Yes.

\n
\n

How that question was resolved on the previous CAFA competitions ?

\n
\n

It never came up.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2245428": "Thanks to organizers and Kaggle for the great challenge ! \n\nMay I clarify certain point: \n\nAt the final evaluation - annotations can only be ADDED to existing high-confidence ones, or some \"high-confidence\" can also \"disappear\" (not confirmed by new experimental data ) ?\n\nBy \"high-confidence\" I mean those mentioned in the rules of the competition: \n\"validated by experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC). More information about evidence codes can be found here. \"\n\n\nIn particular : \nthere are duplicates between train and test - so as far as I understand the point for there existence - that some of them will be experimentally retested and annotations can change. From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ? \n\nHow that question was resolved on the previous CAFA competitions ? \n\n@iddux1", "2263486": "Dear organizers @predragradivojac @parnaljoshi @claradepaolis @damianopiovesan \nor colleagues who participated in previous CAFA @coolmaksat @danofer \n\nWould you be so kind to comment on the question...", "2263563": "Theoretically, experimentally verified predictions may go away if the biocurator made an error in interpreting the paper, or the paper was redacted, or new evidence came up that contradicted previous evidence. The first two cases this needs to be discovered somehow down the line. I don't ever recall that happening. The third case may happen, but that would be a replacement of X by Y, and will be easily detected. I don't recall that happening either.\n\n>From what I understand so far, that change can be only the ADDITION of new annotations, but let me clarify that question explicitly. Is it correct ?\n\n\n\nYes.\n\n>How that question was resolved on the previous CAFA competitions ? \n\nIt never came up." }, "source": "meta" }