The instructions state that \"If a protein ID is not listed in your submitted file, the organizers will assume that all predictions are 0.\" However, after some trial and error, and reading a comment by mtinti on the submissions guide notebook, it seems to be the case that submissions missing test rows can trigger a failure. Once I started filling GO:0000001 and 0.123 for the rows I was missing, a submission that had previously failed has started passing.
\nIt would be nice if the problem owners could check out the bug and tweak the instructions or the evaluation code.
", "messages": [ { "id": "2230114", "postDate": "04/22/2023 03:22:16", "content": "The instructions state that \"If a protein ID is not listed in your submitted file, the organizers will assume that all predictions are 0.\" However, after some trial and error, and reading a comment by mtinti on the submissions guide notebook, it seems to be the case that submissions missing test rows can trigger a failure. Once I started filling GO:0000001 and 0.123 for the rows I was missing, a submission that had previously failed has started passing.
\nIt would be nice if the problem owners could check out the bug and tweak the instructions or the evaluation code.
", "rawMarkdown": "The instructions state that \"If a protein ID is not listed in your submitted file, the organizers will assume that all predictions are 0.\" However, after some trial and error, and reading a comment by mtinti on the [submissions guide notebook](https://www.kaggle.com/code/inversion/submitting-to-the-cafa-competition), it seems to be the case that submissions missing test rows can trigger a failure. Once I started filling GO:0000001 and 0.123 for the rows I was missing, a submission that had previously failed has started passing.\n\nIt would be nice if the problem owners could check out the bug and tweak the instructions or the evaluation code.", "votes": null }, { "id": "2230947", "postDate": "04/22/2023 23:24:37", "content": "We are looking into this, thanks for the catch.
", "rawMarkdown": "We are looking into this, thanks for the catch.", "votes": null }, { "id": "2231390", "postDate": "04/23/2023 09:03:14", "content": "Hi, can you share the submission that fails? Alternatively, open an issue here https://github.com/BioComputingUP/CAFA-evaluator/issues
", "rawMarkdown": "Hi, can you share the submission that fails? Alternatively, open an issue here https://github.com/BioComputingUP/CAFA-evaluator/issues", "votes": null }, { "id": "2232410", "postDate": "04/24/2023 10:13:53", "content": "Hi, see for example this one https://www.kaggle.com/code/mtinti/fake-cafa-sub, where I submit the first 10 lines of inversion example. According to the rules, it should work, I guess. Probably it fails because the proteins used for the lb are not included.
", "rawMarkdown": "Hi, see for example this one https://www.kaggle.com/code/mtinti/fake-cafa-sub, where I submit the first 10 lines of inversion example. According to the rules, it should work, I guess. Probably it fails because the proteins used for the lb are not included.", "votes": null }, { "id": "2245396", "postDate": "05/04/2023 10:36:30", "content": "I confirm that if none of the targets overlap with the validation set, an exception is thrown by the program. Although the notebook doesn't report the exception, you can test it locally using the CAFA-evaluator standalone version.
", "rawMarkdown": "I confirm that if none of the targets overlap with the validation set, an exception is thrown by the program. Although the notebook doesn't report the exception, you can test it locally using the [CAFA-evaluator](https://github.com/BioComputingUP/CAFA-evaluator) standalone version.", "votes": null } ], "comments": [ { "id": 2230947, "author_name": "iddux1", "author_url": "", "post_date": "04/22/2023 23:24:37", "content": "We are looking into this, thanks for the catch.
", "votes": null, "replies": [] }, { "id": 2231390, "author_name": "damianopiovesan", "author_url": "", "post_date": "04/23/2023 09:03:14", "content": "Hi, can you share the submission that fails? Alternatively, open an issue here https://github.com/BioComputingUP/CAFA-evaluator/issues
", "votes": null, "replies": [ { "id": 2232410, "author_name": "mtinti", "author_url": "", "post_date": "04/24/2023 10:13:53", "content": "Hi, see for example this one https://www.kaggle.com/code/mtinti/fake-cafa-sub, where I submit the first 10 lines of inversion example. According to the rules, it should work, I guess. Probably it fails because the proteins used for the lb are not included.
", "votes": null, "replies": [] } ] }, { "id": 2245396, "author_name": "damianopiovesan", "author_url": "", "post_date": "05/04/2023 10:36:30", "content": "I confirm that if none of the targets overlap with the validation set, an exception is thrown by the program. Although the notebook doesn't report the exception, you can test it locally using the CAFA-evaluator standalone version.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2230114": "The instructions state that \"If a protein ID is not listed in your submitted file, the organizers will assume that all predictions are 0.\" However, after some trial and error, and reading a comment by mtinti on the [submissions guide notebook](https://www.kaggle.com/code/inversion/submitting-to-the-cafa-competition), it seems to be the case that submissions missing test rows can trigger a failure. Once I started filling GO:0000001 and 0.123 for the rows I was missing, a submission that had previously failed has started passing.\n\nIt would be nice if the problem owners could check out the bug and tweak the instructions or the evaluation code.", "2230947": "We are looking into this, thanks for the catch.", "2231390": "Hi, can you share the submission that fails? Alternatively, open an issue here https://github.com/BioComputingUP/CAFA-evaluator/issues", "2232410": "Hi, see for example this one https://www.kaggle.com/code/mtinti/fake-cafa-sub, where I submit the first 10 lines of inversion example. According to the rules, it should work, I guess. Probably it fails because the proteins used for the lb are not included.", "2245396": "I confirm that if none of the targets overlap with the validation set, an exception is thrown by the program. Although the notebook doesn't report the exception, you can test it locally using the [CAFA-evaluator](https://github.com/BioComputingUP/CAFA-evaluator) standalone version." }, "source": "meta" }