So yesterday, I took a bunch of proteins (51), and I added the quick-go annotations to my best submission. I saw a jump from 0.51 to 0.54. Does this mean that quick-go already has some (possibly correct) predictions for the proteins used in the LB?
", "messages": [ { "id": 2234363, "postDate": "2023-04-25T06:17:23.640Z", "content": "So yesterday, I took a bunch of proteins (51), and I added the quick-go annotations to my best submission. I saw a jump from 0.51 to 0.54. Does this mean that quick-go already has some (possibly correct) predictions for the proteins used in the LB?
", "rawMarkdown": "So yesterday, I took a bunch of proteins (51), and I added the quick-go annotations to my best submission. I saw a jump from 0.51 to 0.54. Does this mean that quick-go already has some (possibly correct) predictions for the proteins used in the LB?", "votes": 15 }, { "id": 2329798, "postDate": "2023-07-04T14:09:20.873Z", "content": "Hi Michele,
\nThanks again for sharing !
\nI wonder how did you choose these particular 51 proteins ? what was the rationale ? If it is not a secret
No secret. I think these proteins have been annotated in UniProt or moved from TrEMBL to Uniprot recently (03-May-2023). I could not find them when I parsed a uniprot_sprot.dat.gz file, but I could get a response using the web API. I can't be more precise because I did't investigate that further, and than I just started to add all the quickgo annotations anyway…
", "rawMarkdown": "No secret. I think these proteins have been annotated in UniProt or moved from TrEMBL to Uniprot recently (03-May-2023). I could not find them when I parsed a uniprot_sprot.dat.gz file, but I could get a response using the web API. I can't be more precise because I did't investigate that further, and than I just started to add all the quickgo annotations anyway...", "votes": 2 } ] }, { "id": 2234778, "postDate": "2023-04-25T13:35:53.983Z", "content": "It is very strange. Even if we change the predictions from completely wrong to absolutely correct for 51 out of 70K proteins, the score does not change that much. I experimented with calculating the metric (using code from the organizers repository), for 51 random samples this gives a maximum improvement less than 0.001. It may be, however, that I made a mistake somewhere.
", "rawMarkdown": "It is very strange. Even if we change the predictions from completely wrong to absolutely correct for 51 out of 70K proteins, the score does not change that much. I experimented with calculating the metric (using code from the organizers repository), for 51 random samples this gives a maximum improvement less than 0.001. It may be, however, that I made a mistake somewhere.", "votes": 1, "replies": [ { "id": 2234827, "postDate": "2023-04-25T14:11:41.800Z", "content": "How do you determine what is \"completely wrong?\"
", "rawMarkdown": "How do you determine what is \"completely wrong?\" ", "replies": [ { "id": 2234845, "postDate": "2023-04-25T14:22:14.693Z", "content": "Without any terms
", "rawMarkdown": "Without any terms" }, { "id": 2235008, "postDate": "2023-04-25T16:19:01.027Z", "content": "Now I see that this is not the worst case.
", "rawMarkdown": "Now I see that this is not the worst case.", "votes": 1 } ] }, { "id": 2234924, "postDate": "2023-04-25T15:11:49.783Z", "content": "Most likely, I stumbled across a few proteins used for the LB. Not that I wanted to hack the LB, but I was trying to understand inconsistency in the data.
", "rawMarkdown": "Most likely, I stumbled across a few proteins used for the LB. Not that I wanted to hack the LB, but I was trying to understand inconsistency in the data.", "votes": 2 } ] }, { "id": 2234378, "postDate": "2023-04-25T06:44:52.363Z", "content": "yup. There's a correlation between tentative/computational predictions and real functions/annotations
", "rawMarkdown": "yup. There's a correlation between tentative/computational predictions and real functions/annotations", "votes": 2, "replies": [ { "id": 2234430, "postDate": "2023-04-25T07:39:33.850Z", "content": "ok thanks, so possibly one could use just quick-go and get a decent place in the competition…
", "rawMarkdown": "ok thanks, so possibly one could use just quick-go and get a decent place in the competition...", "votes": 2, "replies": [ { "id": 2234594, "postDate": "2023-04-25T10:43:54.720Z", "content": "Maybe that explains why there are now 12 groups near the top with exactly the same score?
", "rawMarkdown": "Maybe that explains why there are now 12 groups near the top with exactly the same score?" }, { "id": 2234615, "postDate": "2023-04-25T11:00:45.950Z", "content": "na, i think that is Alexander Chervov's model. I still need to figure out how to download all the quick-go annotations. It seems the API works only with one id at a time.
", "rawMarkdown": "na, i think that is Alexander Chervov's model. I still need to figure out how to download all the quick-go annotations. It seems the API works only with one id at a time." }, { "id": 2234625, "postDate": "2023-04-25T11:11:32.223Z", "content": "The shared 0.4772 score is Alexander Chervov's binary model.
\nUsing existing annotations would give a good score for the public LB; however existing experimentally-based annotations will not be part of the scoring for the all-important future-based private LB.
\nI haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.
", "rawMarkdown": "The shared 0.4772 score is Alexander Chervov's binary model.\n\nUsing existing annotations would give a good score for the public LB; however existing experimentally-based annotations will not be part of the scoring for the all-important future-based private LB. \n\nI haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.", "votes": 3 }, { "id": 2234656, "postDate": "2023-04-25T11:35:02.200Z", "content": "Ah yes, I see it now. I guess that is the new baseline. Assuming they are following the standard CAFA procedure, then only evidence codes EXP, IDA, IPI, IMP, IGI, IEP, TAS, and IC are considered valid. Others (i.e. IEA) would just be ignored. it was more or less the case in CAFA3 that you would have been near the top with just InterPro GO labels if I recall, so clearly, existing IEA annotations were being ignored.
", "rawMarkdown": "Ah yes, I see it now. I guess that is the new baseline. Assuming they are following the standard CAFA procedure, then only evidence codes EXP, IDA, IPI, IMP, IGI, IEP, TAS, and IC are considered valid. Others (i.e. IEA) would just be ignored. it was more or less the case in CAFA3 that you would have been near the top with just InterPro GO labels if I recall, so clearly, existing IEA annotations were being ignored.", "votes": 4 }, { "id": 2234674, "postDate": "2023-04-25T11:53:24.853Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2234675, "postDate": "2023-04-25T11:53:52.603Z", "content": "I think they would be the best shot to augment the low frequency GO terms in train.
", "rawMarkdown": "I think they would be the best shot to augment the low frequency GO terms in train.", "votes": 4 }, { "id": 2234734, "postDate": "2023-04-25T12:57:40.480Z", "content": "@jbomitchell
\n\"Using existing annotations would give a good score for the public LB \"
I thought that organizers say that public LB is calculated ONLY on the PART of the \"test\"-samples,
\nand that part is precisely those which are NOT yet provided annotations in uniprot.
\nI.e. they say the manage to achieve agreement that uniprot delays a part of annotations release, for the sake of that challenge.
\nThus I thought that public annotations would NOT be helpful for the public LB.
\nBut what MT reports in the start of the topic - annotations from quick-go improve the score.
\nSo I am confused - may be annotations were delayed to release at uniprot, but appeared at quick-go - that would ruin the public LB ; or quick-go has computational annotations - which may be different from the real in public LB (but still may be good baseline) ?
If someone can comment - that would be very helpful.
\n@iddux1
\"I haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.\"
\nSecond on that question.
\n@iddux1
@alexandervc - \"Thus I thought that public annotations would NOT be helpful for the public LB.\"
\nYou may well be right, though there are other databases (than InterPro or QuickGO) that also have, sometimes different, annotatons. See for example nextprot.
\nI'm not clear what happens if new experimental evidence arises for existing non-experimental annotations either.
\nLike you, I'd like more clarity on how both public and private scoring work, I think even minor details matter a lot here.
", "rawMarkdown": "@alexandervc - \"Thus I thought that public annotations would NOT be helpful for the public LB.\" \n\nYou may well be right, though there are other databases (than InterPro or QuickGO) that also have, sometimes different, annotatons. See for example [nextprot](https://www.nextprot.org/entry/NX_A0A0A0MRZ8/).\n\nI'm not clear what happens if new experimental evidence arises for existing non-experimental annotations either.\n\nLike you, I'd like more clarity on how both public and private scoring work, I think even minor details matter a lot here.", "votes": 2 }, { "id": 2234771, "postDate": "2023-04-25T13:28:17.277Z", "content": "Any annotation labelled IEA (Inferred from Electronic Annotation) is a purely computational annotation. Most of the annotations in UniProt and Quick-GO are of that type. Because they are just predictions they are not considered ground truth in function prediction assessments. If a label was an IEA annotation but subsequently gets \"upgraded\" to experimentally validated, then it becomes available as a ground truth testing label. If you submit all of the IEA predictions as your entry, then some of those will turn up as true positives on the public LB.
", "rawMarkdown": "Any annotation labelled IEA (Inferred from Electronic Annotation) is a purely computational annotation. Most of the annotations in UniProt and Quick-GO are of that type. Because they are just predictions they are not considered ground truth in function prediction assessments. If a label was an IEA annotation but subsequently gets \"upgraded\" to experimentally validated, then it becomes available as a ground truth testing label. If you submit all of the IEA predictions as your entry, then some of those will turn up as true positives on the public LB.", "votes": 2 }, { "id": 2234776, "postDate": "2023-04-25T13:33:09.823Z", "content": "@mtinti
\nSo may we ask you - these labels for 51 protein - which you mention -
\nwas there any information that they were \"IEA (Inferred from Electronic Annotation)\" ?
\nOr quick-go does not provide information on evidences ?
It's a mixture, mainly IEA. I'll prepare a dataset and make available the ids I used with annotation this evening.
", "rawMarkdown": "It's a mixture, mainly IEA. I'll prepare a dataset and make available the ids I used with annotation this evening.", "votes": 3 }, { "id": 2235034, "postDate": "2023-04-25T16:44:30.560Z", "content": "QuickGO is based on the latest version of GO, you can just download that.
", "rawMarkdown": "QuickGO is based on the latest version of GO, you can just download that. ", "votes": 2 }, { "id": 2235038, "postDate": "2023-04-25T16:49:23.267Z", "content": "\n\nThus I thought that public annotations would NOT be helpful for the public LB.
\n
Why not? Public annotations are extremely useful, and can and should be used. A low hanging fruit would be using an IEA annotation that you have a high confidence in because, say, the protein has a high sequence similarity to another protein in which the queried function is experimentally validated.
\n\n\nSo I am confused - may be annotations were delayed to release at uniprot, but appeared at quick-go - that would ruin the public LB ; or quick-go has computational annotations - which may be different from the real in public LB (but still may be good baseline) ?
\n
QuickGO is based on the latest version of GO. There is some versioning difference between UniprotKB and GO. The exact terms we are holding out are not available, but similar terms (using the precision recall as described in the Evaluation section in the Rules) might be.
\n\n\n\"I haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.\"
\n
Yes, such \"upgraded from IEA to EXP (IDA, etc.)\" annotations would count towards the assessment. But only if the upgrade occurred after 2023-08-21. If it occurred before that date, then we will not use it for scoring.
", "rawMarkdown": ">Thus I thought that public annotations would NOT be helpful for the public LB.\n\nWhy not? Public annotations are extremely useful, and can and should be used. A low hanging fruit would be using an IEA annotation that you have a high confidence in because, say, the protein has a high sequence similarity to another protein in which the queried function is experimentally validated. \n\n> So I am confused - may be annotations were delayed to release at uniprot, but appeared at quick-go - that would ruin the public LB ; or quick-go has computational annotations - which may be different from the real in public LB (but still may be good baseline) ? \n\nQuickGO is based on the latest version of GO. There is some versioning difference between UniprotKB and GO. The exact terms we are holding out are not available, but similar terms (using the precision recall as described in the Evaluation section in the [Rules](https://www.kaggle.com/competitions/cafa-5-protein-function-prediction/rules)) might be. \n\n\n> \"I haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.\"\n\nYes, such \"upgraded from IEA to EXP (IDA, etc.)\" annotations would count towards the assessment. But only ~~of ~~ if the upgrade occurred after 2023-08-21. If it occurred before that date, then we will not use it for scoring. \n", "votes": 4 }, { "id": 2235072, "postDate": "2023-04-25T17:48:37.477Z", "content": "Thanks @iddux1 that is really helpful!
", "rawMarkdown": "Thanks @iddux1 that is really helpful!", "votes": 1 }, { "id": 2235136, "postDate": "2023-04-25T18:56:12.423Z", "content": "\n\nQuickGO is based on the latest version of GO, you can just download that.
\n
Is there a smart way of getting only the ids in train/test?
", "rawMarkdown": "> QuickGO is based on the latest version of GO, you can just download that.\n\nIs there a smart way of getting only the ids in train/test?", "votes": 2 }, { "id": 2235150, "postDate": "2023-04-25T19:05:23.510Z", "content": "\n\n@mtinti
\n
\n So may we ask you - these labels for 51 protein - which you mention -
\n was there any information that they were \"IEA (Inferred from Electronic Annotation)\" ?
\n Or quick-go does not provide information on evidences ?
So this is the dataset I downloaded manually for 51 proteins:
\nhttps://www.kaggle.com/datasets/mtinti/quick-go-annotations
Can you help me with how you acquired and added the quick GO annotations ?
", "rawMarkdown": "Can you help me with how you acquired and added the quick GO annotations ?", "replies": [ { "id": 2329935, "postDate": "2023-07-04T16:07:49.610Z", "content": "To cut a long story short, you need to download this file :https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz and parse it. You will need to decide which annotation you want to retain. As a bare minimum, you should remove the NOT annotations.
", "rawMarkdown": "To cut a long story short, you need to download this file :https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz and parse it. You will need to decide which annotation you want to retain. As a bare minimum, you should remove the NOT annotations.", "votes": 1 } ] } ], "comments": [ { "id": 2329798, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-07-04T14:09:20.873000", "content": "Hi Michele,
\nThanks again for sharing !
\nI wonder how did you choose these particular 51 proteins ? what was the rationale ? If it is not a secret
No secret. I think these proteins have been annotated in UniProt or moved from TrEMBL to Uniprot recently (03-May-2023). I could not find them when I parsed a uniprot_sprot.dat.gz file, but I could get a response using the web API. I can't be more precise because I did't investigate that further, and than I just started to add all the quickgo annotations anyway…
", "votes": 2, "replies": [] } ] }, { "id": 2234778, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2023-04-25T13:35:53.983000", "content": "It is very strange. Even if we change the predictions from completely wrong to absolutely correct for 51 out of 70K proteins, the score does not change that much. I experimented with calculating the metric (using code from the organizers repository), for 51 random samples this gives a maximum improvement less than 0.001. It may be, however, that I made a mistake somewhere.
", "votes": 1, "replies": [ { "id": 2234827, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-25T14:11:41.800000", "content": "How do you determine what is \"completely wrong?\"
", "votes": 0, "replies": [ { "id": 2234845, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2023-04-25T14:22:14.693000", "content": "Without any terms
", "votes": 0, "replies": [] }, { "id": 2235008, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2023-04-25T16:19:01.027000", "content": "Now I see that this is not the worst case.
", "votes": 1, "replies": [] } ] }, { "id": 2234924, "author_name": "MT", "author_url": "", "post_date": "2023-04-25T15:11:49.783000", "content": "Most likely, I stumbled across a few proteins used for the LB. Not that I wanted to hack the LB, but I was trying to understand inconsistency in the data.
", "votes": 2, "replies": [] } ] }, { "id": 2234378, "author_name": "Dan Ofer", "author_url": "", "post_date": "2023-04-25T06:44:52.363000", "content": "yup. There's a correlation between tentative/computational predictions and real functions/annotations
", "votes": 2, "replies": [ { "id": 2234430, "author_name": "MT", "author_url": "", "post_date": "2023-04-25T07:39:33.850000", "content": "ok thanks, so possibly one could use just quick-go and get a decent place in the competition…
", "votes": 2, "replies": [ { "id": 2234594, "author_name": "biodav", "author_url": "", "post_date": "2023-04-25T10:43:54.720000", "content": "Maybe that explains why there are now 12 groups near the top with exactly the same score?
", "votes": 0, "replies": [] }, { "id": 2234615, "author_name": "MT", "author_url": "", "post_date": "2023-04-25T11:00:45.950000", "content": "na, i think that is Alexander Chervov's model. I still need to figure out how to download all the quick-go annotations. It seems the API works only with one id at a time.
", "votes": 0, "replies": [] }, { "id": 2234625, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-04-25T11:11:32.223000", "content": "The shared 0.4772 score is Alexander Chervov's binary model.
\nUsing existing annotations would give a good score for the public LB; however existing experimentally-based annotations will not be part of the scoring for the all-important future-based private LB.
\nI haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.
", "votes": 3, "replies": [] }, { "id": 2234656, "author_name": "biodav", "author_url": "", "post_date": "2023-04-25T11:35:02.200000", "content": "Ah yes, I see it now. I guess that is the new baseline. Assuming they are following the standard CAFA procedure, then only evidence codes EXP, IDA, IPI, IMP, IGI, IEP, TAS, and IC are considered valid. Others (i.e. IEA) would just be ignored. it was more or less the case in CAFA3 that you would have been near the top with just InterPro GO labels if I recall, so clearly, existing IEA annotations were being ignored.
", "votes": 4, "replies": [] }, { "id": 2234674, "author_name": "", "author_url": "", "post_date": "2023-04-25T11:53:24.853000", "content": "", "votes": 0, "replies": [] }, { "id": 2234675, "author_name": "MT", "author_url": "", "post_date": "2023-04-25T11:53:52.603000", "content": "I think they would be the best shot to augment the low frequency GO terms in train.
", "votes": 4, "replies": [] }, { "id": 2234734, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-04-25T12:57:40.480000", "content": "@jbomitchell
\n\"Using existing annotations would give a good score for the public LB \"
I thought that organizers say that public LB is calculated ONLY on the PART of the \"test\"-samples,
\nand that part is precisely those which are NOT yet provided annotations in uniprot.
\nI.e. they say the manage to achieve agreement that uniprot delays a part of annotations release, for the sake of that challenge.
\nThus I thought that public annotations would NOT be helpful for the public LB.
\nBut what MT reports in the start of the topic - annotations from quick-go improve the score.
\nSo I am confused - may be annotations were delayed to release at uniprot, but appeared at quick-go - that would ruin the public LB ; or quick-go has computational annotations - which may be different from the real in public LB (but still may be good baseline) ?
If someone can comment - that would be very helpful.
\n@iddux1
\"I haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.\"
\nSecond on that question.
\n@iddux1
@alexandervc - \"Thus I thought that public annotations would NOT be helpful for the public LB.\"
\nYou may well be right, though there are other databases (than InterPro or QuickGO) that also have, sometimes different, annotatons. See for example nextprot.
\nI'm not clear what happens if new experimental evidence arises for existing non-experimental annotations either.
\nLike you, I'd like more clarity on how both public and private scoring work, I think even minor details matter a lot here.
", "votes": 2, "replies": [] }, { "id": 2234771, "author_name": "biodav", "author_url": "", "post_date": "2023-04-25T13:28:17.277000", "content": "Any annotation labelled IEA (Inferred from Electronic Annotation) is a purely computational annotation. Most of the annotations in UniProt and Quick-GO are of that type. Because they are just predictions they are not considered ground truth in function prediction assessments. If a label was an IEA annotation but subsequently gets \"upgraded\" to experimentally validated, then it becomes available as a ground truth testing label. If you submit all of the IEA predictions as your entry, then some of those will turn up as true positives on the public LB.
", "votes": 2, "replies": [] }, { "id": 2234776, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-04-25T13:33:09.823000", "content": "@mtinti
\nSo may we ask you - these labels for 51 protein - which you mention -
\nwas there any information that they were \"IEA (Inferred from Electronic Annotation)\" ?
\nOr quick-go does not provide information on evidences ?
It's a mixture, mainly IEA. I'll prepare a dataset and make available the ids I used with annotation this evening.
", "votes": 3, "replies": [] }, { "id": 2235034, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-25T16:44:30.560000", "content": "QuickGO is based on the latest version of GO, you can just download that.
", "votes": 2, "replies": [] }, { "id": 2235038, "author_name": "Iddo Friedberg", "author_url": "", "post_date": "2023-04-25T16:49:23.267000", "content": "\n\nThus I thought that public annotations would NOT be helpful for the public LB.
\n
Why not? Public annotations are extremely useful, and can and should be used. A low hanging fruit would be using an IEA annotation that you have a high confidence in because, say, the protein has a high sequence similarity to another protein in which the queried function is experimentally validated.
\n\n\nSo I am confused - may be annotations were delayed to release at uniprot, but appeared at quick-go - that would ruin the public LB ; or quick-go has computational annotations - which may be different from the real in public LB (but still may be good baseline) ?
\n
QuickGO is based on the latest version of GO. There is some versioning difference between UniprotKB and GO. The exact terms we are holding out are not available, but similar terms (using the precision recall as described in the Evaluation section in the Rules) might be.
\n\n\n\"I haven't yet got clarity on whether annotations that currently exist but are only inferred from bioinformatics and homology methods would count for scoring if they subsequently acquired full experimental justification in the scoring period.\"
\n
Yes, such \"upgraded from IEA to EXP (IDA, etc.)\" annotations would count towards the assessment. But only if the upgrade occurred after 2023-08-21. If it occurred before that date, then we will not use it for scoring.
", "votes": 4, "replies": [] }, { "id": 2235072, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-04-25T17:48:37.477000", "content": "Thanks @iddux1 that is really helpful!
", "votes": 1, "replies": [] }, { "id": 2235136, "author_name": "MT", "author_url": "", "post_date": "2023-04-25T18:56:12.423000", "content": "\n\nQuickGO is based on the latest version of GO, you can just download that.
\n
Is there a smart way of getting only the ids in train/test?
", "votes": 2, "replies": [] }, { "id": 2235150, "author_name": "MT", "author_url": "", "post_date": "2023-04-25T19:05:23.510000", "content": "\n\n@mtinti
\n
\n So may we ask you - these labels for 51 protein - which you mention -
\n was there any information that they were \"IEA (Inferred from Electronic Annotation)\" ?
\n Or quick-go does not provide information on evidences ?
So this is the dataset I downloaded manually for 51 proteins:
\nhttps://www.kaggle.com/datasets/mtinti/quick-go-annotations
Can you help me with how you acquired and added the quick GO annotations ?
", "votes": 0, "replies": [ { "id": 2329935, "author_name": "MT", "author_url": "", "post_date": "2023-07-04T16:07:49.610000", "content": "To cut a long story short, you need to download this file :https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz and parse it. You will need to decide which annotation you want to retain. As a bare minimum, you should remove the NOT annotations.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2234363": "So yesterday, I took a bunch of proteins (51), and I added the quick-go annotations to my best submission. I saw a jump from 0.51 to 0.54. Does this mean that quick-go already has some (possibly correct) predictions for the proteins used in the LB?", "2329798": "Hi Michele, \nThanks again for sharing ! \nI wonder how did you choose these particular 51 proteins ? what was the rationale ? If it is not a secret \n\n", "2234778": "It is very strange. Even if we change the predictions from completely wrong to absolutely correct for 51 out of 70K proteins, the score does not change that much. I experimented with calculating the metric (using code from the organizers repository), for 51 random samples this gives a maximum improvement less than 0.001. It may be, however, that I made a mistake somewhere.", "2234378": "yup. There's a correlation between tentative/computational predictions and real functions/annotations", "2326779": "Can you help me with how you acquired and added the quick GO annotations ?" } }