{
  "id": 613138,
  "title": "# 🧬 From zero to ~0.269 LB using GOA UniProt (GAF) with “NOT”-free annotations",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/613138",
  "author_name": "MT",
  "post_date": "2025-10-24T12:11:10.545000",
  "votes": 22,
  "comment_count": 31,
  "views": 0,
  "content": "<p>I built a simple baseline by mapping UniProt proteins to Gene Ontology (GO) terms directly from <strong>GOA UniProt</strong> and removing negated (“NOT …”) annotations. Submitting the remaining terms for IDs that overlap with the <strong>test superset</strong> yielded <strong>~0.269 on the public leaderboard</strong> for me. I’ll share a Polars-based parser for the large GAF file in a follow-up post.</p>\n<h2>What is a GAF?</h2>\n<p><strong>GAF (Gene Association File)</strong> is a tab-separated standard used to distribute GO annotations. Each row links a database object (e.g., a UniProt accession) to a GO term, alongside evidence and metadata. In GAF v2.2, the key fields are:</p>\n<ul>\n<li><strong>DB_Object_ID</strong> (column 2): the stable identifier for the entity (e.g., UniProt ID like <code>P12345</code>).  </li>\n<li><strong>Qualifier</strong> (column 4): may include relationship labels (e.g., <code>enables</code>, <code>involved_in</code>) and <strong>negations</strong> using <code>NOT</code>.  </li>\n<li><strong>GO_ID</strong> (column 5): the Gene Ontology term (e.g., <code>GO:0005515</code>).  </li>\n<li>Comment lines start with <code>!</code> and should be ignored.</li>\n</ul>\n<p>Why care about <strong>NOT</strong>? A <code>NOT</code> qualifier explicitly states that the gene/product is <strong>not</strong> associated with that term/relationship. For leaderboard alignment, I removed these negated associations before building the submission.</p>\n<hr>\n<h2>Data source</h2>\n<p>Download the comprehensive GOA UniProt file from EBI’s FTP:</p>\n<ul>\n<li><strong><code>goa_uniprot_all.gaf</code></strong> (compressed as <code>.gz</code>)  </li>\n</ul>\n<h2>  Path: <code>https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/</code></h2>\n<h2>What exactly did I remove?</h2>\n<p>From the <strong>Qualifier</strong> field, I excluded rows containing any of the following negated qualifiers (and, more generally, anything with <code>NOT</code> in it):</p>\n<ul>\n<li><code>NOT|involved_in</code>  </li>\n<li><code>NOT|is_active_in</code>  </li>\n<li><code>NOT|colocalizes_with</code>  </li>\n<li><code>NOT|enables</code>  </li>\n<li><code>NOT|acts_upstream_of_or_within</code>  </li>\n<li><code>NOT|acts_upstream_of_or_within_negative_effect</code>  </li>\n<li><code>NOT|contributes_to</code>  </li>\n<li><code>NOT|part_of</code>  </li>\n<li><code>NOT|acts_upstream_of</code>  </li>\n<li><code>NOT|located_in</code>  </li>\n<li><code>NOT|acts_upstream_of_or_within_positive_effect</code></li>\n</ul>\n<p>This removes explicit negations so we only submit positive associations.</p>\n<hr>\n<h2>Building the submission</h2>\n<ol>\n<li><p><strong>Load the test superset IDs.</strong>  <br>\nUse the competition’s provided test set/superset to obtain the list of UniProt accessions.</p></li>\n<li><p><strong>Parse GOA UniProt (GAF).</strong>  </p>\n<ul>\n<li>Skip comment lines (<code>!</code>).  </li>\n<li>Extract <strong>DB_Object_ID</strong> (col 2), <strong>Qualifier</strong> (col 4), and <strong>GO_ID</strong> (col 5).</li></ul></li>\n<li><p><strong>Filter to test overlap.</strong>  <br>\nKeep only rows where <strong>DB_Object_ID</strong> is in the test superset.</p></li>\n<li><p><strong>Drop negations.</strong>  <br>\nExclude rows where <strong>Qualifier</strong> contains <code>NOT</code> or matches any of the specific negated patterns listed above.</p></li>\n<li><p><strong>Write <code>submission.tsv</code>.</strong>  </p></li>\n</ol>\n<pre><code>       go_term \n</code></pre>\n<p>Submitting the above mapping (no model, no ranking) gave me <strong>~0.269 LB</strong>. </p>",
  "messages": [
    {
      "id": 3306368,
      "postDate": "2025-10-24T12:11:10.547Z",
      "content": "<p>I built a simple baseline by mapping UniProt proteins to Gene Ontology (GO) terms directly from <strong>GOA UniProt</strong> and removing negated (“NOT …”) annotations. Submitting the remaining terms for IDs that overlap with the <strong>test superset</strong> yielded <strong>~0.269 on the public leaderboard</strong> for me. I’ll share a Polars-based parser for the large GAF file in a follow-up post.</p>\n<h2>What is a GAF?</h2>\n<p><strong>GAF (Gene Association File)</strong> is a tab-separated standard used to distribute GO annotations. Each row links a database object (e.g., a UniProt accession) to a GO term, alongside evidence and metadata. In GAF v2.2, the key fields are:</p>\n<ul>\n<li><strong>DB_Object_ID</strong> (column 2): the stable identifier for the entity (e.g., UniProt ID like <code>P12345</code>).  </li>\n<li><strong>Qualifier</strong> (column 4): may include relationship labels (e.g., <code>enables</code>, <code>involved_in</code>) and <strong>negations</strong> using <code>NOT</code>.  </li>\n<li><strong>GO_ID</strong> (column 5): the Gene Ontology term (e.g., <code>GO:0005515</code>).  </li>\n<li>Comment lines start with <code>!</code> and should be ignored.</li>\n</ul>\n<p>Why care about <strong>NOT</strong>? A <code>NOT</code> qualifier explicitly states that the gene/product is <strong>not</strong> associated with that term/relationship. For leaderboard alignment, I removed these negated associations before building the submission.</p>\n<hr>\n<h2>Data source</h2>\n<p>Download the comprehensive GOA UniProt file from EBI’s FTP:</p>\n<ul>\n<li><strong><code>goa_uniprot_all.gaf</code></strong> (compressed as <code>.gz</code>)  </li>\n</ul>\n<h2>  Path: <code>https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/</code></h2>\n<h2>What exactly did I remove?</h2>\n<p>From the <strong>Qualifier</strong> field, I excluded rows containing any of the following negated qualifiers (and, more generally, anything with <code>NOT</code> in it):</p>\n<ul>\n<li><code>NOT|involved_in</code>  </li>\n<li><code>NOT|is_active_in</code>  </li>\n<li><code>NOT|colocalizes_with</code>  </li>\n<li><code>NOT|enables</code>  </li>\n<li><code>NOT|acts_upstream_of_or_within</code>  </li>\n<li><code>NOT|acts_upstream_of_or_within_negative_effect</code>  </li>\n<li><code>NOT|contributes_to</code>  </li>\n<li><code>NOT|part_of</code>  </li>\n<li><code>NOT|acts_upstream_of</code>  </li>\n<li><code>NOT|located_in</code>  </li>\n<li><code>NOT|acts_upstream_of_or_within_positive_effect</code></li>\n</ul>\n<p>This removes explicit negations so we only submit positive associations.</p>\n<hr>\n<h2>Building the submission</h2>\n<ol>\n<li><p><strong>Load the test superset IDs.</strong>  <br>\nUse the competition’s provided test set/superset to obtain the list of UniProt accessions.</p></li>\n<li><p><strong>Parse GOA UniProt (GAF).</strong>  </p>\n<ul>\n<li>Skip comment lines (<code>!</code>).  </li>\n<li>Extract <strong>DB_Object_ID</strong> (col 2), <strong>Qualifier</strong> (col 4), and <strong>GO_ID</strong> (col 5).</li></ul></li>\n<li><p><strong>Filter to test overlap.</strong>  <br>\nKeep only rows where <strong>DB_Object_ID</strong> is in the test superset.</p></li>\n<li><p><strong>Drop negations.</strong>  <br>\nExclude rows where <strong>Qualifier</strong> contains <code>NOT</code> or matches any of the specific negated patterns listed above.</p></li>\n<li><p><strong>Write <code>submission.tsv</code>.</strong>  </p></li>\n</ol>\n<pre><code>       go_term \n</code></pre>\n<p>Submitting the above mapping (no model, no ranking) gave me <strong>~0.269 LB</strong>. </p>",
      "rawMarkdown": "I built a simple baseline by mapping UniProt proteins to Gene Ontology (GO) terms directly from **GOA UniProt** and removing negated (“NOT …”) annotations. Submitting the remaining terms for IDs that overlap with the **test superset** yielded **~0.269 on the public leaderboard** for me. I’ll share a Polars-based parser for the large GAF file in a follow-up post.\n\n\n## What is a GAF?\n\n**GAF (Gene Association File)** is a tab-separated standard used to distribute GO annotations. Each row links a database object (e.g., a UniProt accession) to a GO term, alongside evidence and metadata. In GAF v2.2, the key fields are:\n\n- **DB_Object_ID** (column 2): the stable identifier for the entity (e.g., UniProt ID like `P12345`).  \n- **Qualifier** (column 4): may include relationship labels (e.g., `enables`, `involved_in`) and **negations** using `NOT`.  \n- **GO_ID** (column 5): the Gene Ontology term (e.g., `GO:0005515`).  \n- Comment lines start with `!` and should be ignored.\n\nWhy care about **NOT**? A `NOT` qualifier explicitly states that the gene/product is **not** associated with that term/relationship. For leaderboard alignment, I removed these negated associations before building the submission.\n\n---\n\n## Data source\n\nDownload the comprehensive GOA UniProt file from EBI’s FTP:\n\n- **`goa_uniprot_all.gaf`** (compressed as `.gz`)  \n  Path: `https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/`\n---\n\n## What exactly did I remove?\n\nFrom the **Qualifier** field, I excluded rows containing any of the following negated qualifiers (and, more generally, anything with `NOT` in it):\n\n- `NOT|involved_in`  \n- `NOT|is_active_in`  \n- `NOT|colocalizes_with`  \n- `NOT|enables`  \n- `NOT|acts_upstream_of_or_within`  \n- `NOT|acts_upstream_of_or_within_negative_effect`  \n- `NOT|contributes_to`  \n- `NOT|part_of`  \n- `NOT|acts_upstream_of`  \n- `NOT|located_in`  \n- `NOT|acts_upstream_of_or_within_positive_effect`\n\nThis removes explicit negations so we only submit positive associations.\n\n---\n\n## Building the submission\n\n1. **Load the test superset IDs.**  \n   Use the competition’s provided test set/superset to obtain the list of UniProt accessions.\n\n2. **Parse GOA UniProt (GAF).**  \n   - Skip comment lines (`!`).  \n   - Extract **DB_Object_ID** (col 2), **Qualifier** (col 4), and **GO_ID** (col 5).\n\n3. **Filter to test overlap.**  \n   Keep only rows where **DB_Object_ID** is in the test superset.\n\n4. **Drop negations.**  \n   Exclude rows where **Qualifier** contains `NOT` or matches any of the specific negated patterns listed above.\n\n6. **Write `submission.tsv`.**  \n   ```\n   protein_id    go_term 1\n   ```\n\nSubmitting the above mapping (no model, no ranking) gave me **~0.269 LB**. ",
      "votes": 22
    },
    {
      "id": 3306385,
      "postDate": "2025-10-24T13:08:44.503Z",
      "content": "<p>Hi Michele<br>\nGreat work !<br>\nWhy can it work ?<br>\nNon experimental annotations became experimental ?<br>\nHave you used all anonstations or only experimental ? <br>\nOr data leakage ? And some proteins in test are already in unprot?</p>",
      "rawMarkdown": "Hi Michele\nGreat work !\nWhy can it work ?\nNon experimental annotations became experimental ?\nHave you used all anonstations or only experimental ? \nOr data leakage ? And some proteins in test are already in unprot?",
      "votes": 4,
      "replies": [
        {
          "id": 3306405,
          "postDate": "2025-10-24T13:42:29.810Z",
          "content": "<p>Hi Alexander, great to see you here! I used all annotations regardless of evidence code, and only removed rows with NOT in the Qualifier. So I think it works because some electronic annotations have since been curated into experimental ones.</p>",
          "rawMarkdown": "Hi Alexander, great to see you here! I used all annotations regardless of evidence code, and only removed rows with NOT in the Qualifier. So I think it works because some electronic annotations have since been curated into experimental ones.",
          "votes": 2,
          "replies": [
            {
              "id": 3306598,
              "postDate": "2025-10-24T21:09:22.320Z",
              "content": "<p>Great to see you too ! <br>\nYes, I think you are right !</p>\n<p>But a bit surprising:<br>\ngoa_nonexp_predictions.tsv 0.182<br>\nthere is submit from ogranizers - it scores 0.182, much less than 0.26<br>\nwhich seems only electronic annotations.</p>\n<p>Would it mean that proteins in test include some annotations which are experimantal and already are in uniprot ? <br>\nIt seems it might be so, at least for some sub-ontlogies. </p>",
              "rawMarkdown": "Great to see you too ! \nYes, I think you are right !\n\nBut a bit surprising:\ngoa_nonexp_predictions.tsv 0.182\nthere is submit from ogranizers - it scores 0.182, much less than 0.26\nwhich seems only electronic annotations.\n\nWould it mean that proteins in test include some annotations which are experimantal and already are in uniprot ? \nIt seems it might be so, at least for some sub-ontlogies. \n\n",
              "votes": 4
            },
            {
              "id": 3306600,
              "postDate": "2025-10-24T21:30:47.603Z",
              "content": "<p>I updated the baseline with the current UniProt release and I got 0.207 for \"goa_nonexp_predictions.tsv\" (all non-experimental evidence codes, not just electronic) and 0.269 if using all evidence codes (similar to MT's post). I was also surprised to see that just using all annotations made such a big improvement. Right now we are checking to see if any annotations in the leaderboard hold-out set (which should not be in the public database) were accidentally published. </p>\n<p>That being said, any annotations that are public before the submission deadline will not be included in the final evaluation. </p>",
              "rawMarkdown": "I updated the baseline with the current UniProt release and I got 0.207 for \"goa_nonexp_predictions.tsv\" (all non-experimental evidence codes, not just electronic) and 0.269 if using all evidence codes (similar to MT's post). I was also surprised to see that just using all annotations made such a big improvement. Right now we are checking to see if any annotations in the leaderboard hold-out set (which should not be in the public database) were accidentally published. \n\nThat being said, any annotations that are public before the submission deadline will not be included in the final evaluation. ",
              "votes": 5
            },
            {
              "id": 3306713,
              "postDate": "2025-10-25T04:59:10.953Z",
              "content": "<p>Does that mean this GOA submission would be 0 in the final private leaderboard?</p>",
              "rawMarkdown": "Does that mean this GOA submission would be 0 in the final private leaderboard?",
              "votes": 2
            },
            {
              "id": 3306726,
              "postDate": "2025-10-25T05:42:03.963Z",
              "content": "<blockquote>\n  <p>any annotations that are public before the submission deadline will not be included in the final evaluation.</p>\n</blockquote>\n<p>Any <em>experimental</em> annotation, right?</p>",
              "rawMarkdown": ">any annotations that are public before the submission deadline will not be included in the final evaluation.\n\nAny *experimental* annotation, right?",
              "votes": 2
            },
            {
              "id": 3307024,
              "postDate": "2025-10-26T01:23:04.933Z",
              "content": "<p>Yes, that is correct</p>",
              "rawMarkdown": "Yes, that is correct",
              "votes": 2
            },
            {
              "id": 3307039,
              "postDate": "2025-10-26T02:18:29.140Z",
              "content": "<p>Before the submission deadline, we will collect the most updated public GO experimental annotations. These GO experimental annotations are considered \"known\" and will be excluded from the final private leaderboard (which will be assessed around in June 2026). So, if your submission only consists of GO experimental annotations, it will score 0. </p>",
              "rawMarkdown": "Before the submission deadline, we will collect the most updated public GO experimental annotations. These GO experimental annotations are considered \"known\" and will be excluded from the final private leaderboard (which will be assessed around in June 2026). So, if your submission only consists of GO experimental annotations, it will score 0. ",
              "votes": 3
            },
            {
              "id": 3307961,
              "postDate": "2025-10-28T08:41:15.533Z",
              "content": "<p>So we might have both <strong>new</strong> experimental and non-experimental annotations in private leaderboard? Or the evaluation is done only on the experimental annotations that are discovered over time? </p>\n<p>EDIT: Read again the overview and the answer is that only <strong>experimentally-validated</strong> annotations will be in private leaderabaord.</p>\n<p>Follow-up question: What about <strong>NOT</strong> quantifier? Does it still count as \"scientific discovery\" or there will be only positive associations in private leaderboard? </p>",
              "rawMarkdown": "So we might have both **new** experimental and non-experimental annotations in private leaderboard? Or the evaluation is done only on the experimental annotations that are discovered over time? \n\nEDIT: Read again the overview and the answer is that only **experimentally-validated** annotations will be in private leaderabaord.\n\nFollow-up question: What about **NOT** quantifier? Does it still count as \"scientific discovery\" or there will be only positive associations in private leaderboard? ",
              "votes": 2
            },
            {
              "id": 3308089,
              "postDate": "2025-10-28T14:40:11.460Z",
              "content": "<p>Annotations with \"NOT\" qualifier will not be included in the ground truth for evaluation</p>",
              "rawMarkdown": "Annotations with \"NOT\" qualifier will not be included in the ground truth for evaluation",
              "votes": 3
            },
            {
              "id": 3308866,
              "postDate": "2025-10-30T12:33:27.870Z",
              "content": "<blockquote>\n  <p>Yes, that is correct.\n  so, the final code should not include the pulic annotations like gaf?And this may lead to the 0 point in final evaluation?</p>\n</blockquote>",
              "rawMarkdown": "> Yes, that is correct.\nso, the final code should not include the pulic annotations like gaf?And this may lead to the 0 point in final evaluation?\n\n\n"
            },
            {
              "id": 3308871,
              "postDate": "2025-10-30T12:45:51.107Z",
              "content": "<p>\"Yes, that is correct. so, the final code should not include the pulic annotations like gaf?And this may lead to the 0 point in final evaluation?\"</p>\n<p>I wouldn't reach that conclusion here. Including non-scored annotations would be neutral, not harmful. There may be benefits to including electronic annotations (i.e. bioinformatics predictions) as these have the potential to be confirmed by experiment.</p>",
              "rawMarkdown": "\"Yes, that is correct. so, the final code should not include the pulic annotations like gaf?And this may lead to the 0 point in final evaluation?\"\n\nI wouldn't reach that conclusion here. Including non-scored annotations would be neutral, not harmful. There may be benefits to including electronic annotations (i.e. bioinformatics predictions) as these have the potential to be confirmed by experiment."
            },
            {
              "id": 3308882,
              "postDate": "2025-10-30T13:21:35.580Z",
              "content": "<p>Note that if you <em>only</em> have <strong>public experimental</strong> annotations from GAF, it will score 0. Otherwise, if you include non-experimental annotations (and these happen to be experimentally verified after submission deadline), you will score &gt;0. Like John said, including non-scored annotations would be neutral and not harmful.</p>",
              "rawMarkdown": "Note that if you *only* have **public experimental** annotations from GAF, it will score 0. Otherwise, if you include non-experimental annotations (and these happen to be experimentally verified after submission deadline), you will score >0. Like John said, including non-scored annotations would be neutral and not harmful.",
              "votes": 2
            },
            {
              "id": 3308971,
              "postDate": "2025-10-30T16:18:13.730Z",
              "content": "<blockquote>\n  <p>Note that if you <em>only</em> have <strong>public experimental</strong> annotations from GAF, it will score 0. Otherwise, if you include non-experimental annotations (and these happen to be experimentally verified after submission deadline), you will score &gt;0. Like John said, including non-scored annotations would be neutral and not harmful.</p>\n</blockquote>\n<p>Okey， so if the cafa5 use the submit result of cafa4 as reference. Will this be helpful to get better score in final result. I am wondering this because a code is using this.</p>",
              "rawMarkdown": "> Note that if you *only* have **public experimental** annotations from GAF, it will score 0. Otherwise, if you include non-experimental annotations (and these happen to be experimentally verified after submission deadline), you will score >0. Like John said, including non-scored annotations would be neutral and not harmful.\n\nOkey， so if the cafa5 use the submit result of cafa4 as reference. Will this be helpful to get better score in final result. I am wondering this because a code is using this.\n\n"
            },
            {
              "id": 3309002,
              "postDate": "2025-10-30T17:21:07.563Z",
              "content": "<blockquote>\n  <p>Okey， so if the cafa5 use the submit result of cafa4 as reference. Will this be helpful to get better score in final result. I am wondering this because a code is using this.</p>\n</blockquote>\n<p>Yes, if a sequence is in the test sets of both CAFA 5 and CAFA 6, then a reused prediction originally made for CAFA 5 might end up scoring towards CAFA 6 if the experimental annotation happens to be published during the CAFA 6 scoring period.</p>",
              "rawMarkdown": "> Okey， so if the cafa5 use the submit result of cafa4 as reference. Will this be helpful to get better score in final result. I am wondering this because a code is using this.\n\nYes, if a sequence is in the test sets of both CAFA 5 and CAFA 6, then a reused prediction originally made for CAFA 5 might end up scoring towards CAFA 6 if the experimental annotation happens to be published during the CAFA 6 scoring period.",
              "votes": 1
            },
            {
              "id": 3320329,
              "postDate": "2025-11-12T12:56:47.773Z",
              "content": "<p>Sir, I want to doublecheck the experimental annotations, is the BASIC_EXP_CODES = {'EXP', 'IDA', 'IPI', 'IMP', 'IGI', 'IEP'}\nHTP_EXP_CODES = {'HTP', 'HDA', 'HMP', 'HGI', 'HEP'} are experimental and these are not experimental:   'IEA', 'IBA', 'ISO', 'ISS', 'IC', 'ISA', 'ISM', 'IKR', 'IGC', \n'TAS', 'NAS', 'RCA' , 'IBD', 'IRD'.</p>",
              "rawMarkdown": "Sir, I want to doublecheck the experimental annotations, is the BASIC_EXP_CODES = {'EXP', 'IDA', 'IPI', 'IMP', 'IGI', 'IEP'}\nHTP_EXP_CODES = {'HTP', 'HDA', 'HMP', 'HGI', 'HEP'} are experimental and these are not experimental:   'IEA', 'IBA', 'ISO', 'ISS', 'IC', 'ISA', 'ISM', 'IKR', 'IGC', \n'TAS', 'NAS', 'RCA' , 'IBD', 'IRD'."
            },
            {
              "id": 3320526,
              "postDate": "2025-11-12T15:45:57.657Z",
              "content": "<blockquote>\n  <p>experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC)</p>\n</blockquote>\n<p>We used all the BASIC_EXP_CODES and HTP_EXP_CODES you listed, plus \"IC\" and \"TAS\" since these are high quality annotations. In the discussion comments, I just referred to them all as experimental annotations.</p>",
              "rawMarkdown": ">experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC)\n\nWe used all the BASIC_EXP_CODES and HTP_EXP_CODES you listed, plus \"IC\" and \"TAS\" since these are high quality annotations. In the discussion comments, I just referred to them all as experimental annotations."
            },
            {
              "id": 3320527,
              "postDate": "2025-11-12T15:46:23.703Z",
              "content": "<p>Hi，sir, are the IC and TAS experimental or non-experimental? IC:Inferred by Curator\nTAS:Traceable Author Statement</p>",
              "rawMarkdown": "Hi，sir, are the IC and TAS experimental or non-experimental? IC:Inferred by Curator\nTAS:Traceable Author Statement\n"
            },
            {
              "id": 3320530,
              "postDate": "2025-11-12T15:48:49.087Z",
              "content": "<p>IC and TAS are not considered experimental annotations. But note that we use experimental and IC and TAS annotations to build the train_terms.tsv and to build the evaluation ground truth.</p>",
              "rawMarkdown": "IC and TAS are not considered experimental annotations. But note that we use experimental and IC and TAS annotations to build the train_terms.tsv and to build the evaluation ground truth."
            },
            {
              "id": 3320591,
              "postDate": "2025-11-12T16:29:08.367Z",
              "content": "<p>And will you remove the annotation based on TAS and IC. In other side, regard them as experimental evidence. Or the experimenal annotation are only included in what the uniprot database says</p>",
              "rawMarkdown": "And will you remove the annotation based on TAS and IC. In other side, regard them as experimental evidence. Or the experimenal annotation are only included in what the uniprot database says\n"
            },
            {
              "id": 3320609,
              "postDate": "2025-11-12T16:47:38.363Z",
              "content": "<p>As I said, all of these codes will be used {'EXP', 'IDA', 'IPI', 'IMP', 'IGI', 'IEP', 'HTP', 'HDA', 'HMP', 'HGI', 'HEP', 'IC', 'TAS'}</p>",
              "rawMarkdown": "As I said, all of these codes will be used {'EXP', 'IDA', 'IPI', 'IMP', 'IGI', 'IEP', 'HTP', 'HDA', 'HMP', 'HGI', 'HEP', 'IC', 'TAS'}"
            },
            {
              "id": 3322699,
              "postDate": "2025-11-13T16:46:57.633Z",
              "content": "<p>sir, I have another question, for the known label（experimenal go annotation） in training.set, will the prediction of them be score in the public lb? In other word, would they will be excluded in the calculation of public lb. Also, when use the evalutor, the ground truth file should exclude the training label? I think in private lb, they will not. </p>",
              "rawMarkdown": "sir, I have another question, for the known label（experimenal go annotation） in training.set, will the prediction of them be score in the public lb? In other word, would they will be excluded in the calculation of public lb. Also, when use the evalutor, the ground truth file should exclude the training label? I think in private lb, they will not. "
            },
            {
              "id": 3322921,
              "postDate": "2025-11-13T20:42:52.047Z",
              "content": "<p>The labels in the training data are currently excluded in the calculation of public lb. It does not matter if your ground truth file excludes the training labels or not, they will be propagated up to the subontology root. To exclude the training labels from the evaluation, you must pass the training labels in <code>-known</code> argument. Check the example here: <a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK/tree/main/example\" target=\"_blank\">https://github.com/claradepaolis/CAFA-evaluator-PK/tree/main/example</a> </p>",
              "rawMarkdown": "The labels in the training data are currently excluded in the calculation of public lb. It does not matter if your ground truth file excludes the training labels or not, they will be propagated up to the subontology root. To exclude the training labels from the evaluation, you must pass the training labels in `-known` argument. Check the example here: https://github.com/claradepaolis/CAFA-evaluator-PK/tree/main/example "
            },
            {
              "id": 3330568,
              "postDate": "2025-11-16T09:24:29.263Z",
              "content": "<p>Hi, sir. I have another question about where you get the seq(acid) and the taxonomy id of training and test set. I found an idea in my experiments. </p>",
              "rawMarkdown": "Hi, sir. I have another question about where you get the seq(acid) and the taxonomy id of training and test set. I found an idea in my experiments. "
            },
            {
              "id": 3332288,
              "postDate": "2025-11-16T16:52:55.747Z",
              "content": "<p>The sequence and the taxonomy id were obtained from the uniprot_sprot.fasta file which you can find inside this tar archive on <a href=\"https://ftp.ebi.ac.uk/pub/databases/uniprot/previous_releases/release-2025_03/knowledgebase/uniprot_sprot-only2025_03.tar.gz\" target=\"_blank\">UniProt FTP site</a>.</p>",
              "rawMarkdown": "The sequence and the taxonomy id were obtained from the uniprot_sprot.fasta file which you can find inside this tar archive on [UniProt FTP site](https://ftp.ebi.ac.uk/pub/databases/uniprot/previous_releases/release-2025_03/knowledgebase/uniprot_sprot-only2025_03.tar.gz)."
            },
            {
              "id": 3371255,
              "postDate": "2025-12-11T11:30:16.917Z",
              "content": "<p>Thank you.. as I see 'IC' is there, but what about \"ND\"??</p>",
              "rawMarkdown": "Thank you.. as I see 'IC' is there, but what about \"ND\"??"
            },
            {
              "id": 3371618,
              "postDate": "2025-12-11T15:25:48.197Z",
              "content": "<p>\"ND\" (stands for \"No biological Data available\") is not included in the training data. If you check the GAF file, most (if not all) ND annotations are just annotating the 3 subontology roots (\"GO:0008150\", \"GO:0003674\", \"GO:0005575\") so they won't be informative anyway. </p>",
              "rawMarkdown": "\"ND\" (stands for \"No biological Data available\") is not included in the training data. If you check the GAF file, most (if not all) ND annotations are just annotating the 3 subontology roots (\"GO:0008150\", \"GO:0003674\", \"GO:0005575\") so they won't be informative anyway. "
            },
            {
              "id": 3377820,
              "postDate": "2025-12-17T02:01:46.403Z",
              "content": "<p>Do this mean that what in the gaf has some experimental overlap with the holdout experimental annotation</p>",
              "rawMarkdown": "Do this mean that what in the gaf has some experimental overlap with the holdout experimental annotation"
            }
          ]
        }
      ]
    },
    {
      "id": 3308046,
      "postDate": "2025-10-28T13:19:50.573Z",
      "content": "<p>I think my take-home from this is that while any extant experimental GO annotation won't contribute to the private score, it is potentially worth including \"Inferred from Electronic Annotation (IEA)\" data (essentially bioinformatics predictions) in our submissions. These could contribute to the score if they happen to be experimentally confirmed at the right time. </p>",
      "rawMarkdown": "I think my take-home from this is that while any extant experimental GO annotation won't contribute to the private score, it is potentially worth including \"Inferred from Electronic Annotation (IEA)\" data (essentially bioinformatics predictions) in our submissions. These could contribute to the score if they happen to be experimentally confirmed at the right time. ",
      "votes": 2,
      "replies": [
        {
          "id": 3394958,
          "postDate": "2026-01-22T00:40:54.893Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 3371350,
      "postDate": "2025-12-11T12:28:40.240Z",
      "content": "<p>CAFA 6 Protein Function Prediction</p>",
      "rawMarkdown": "CAFA 6 Protein Function Prediction"
    }
  ],
  "comments": [
    {
      "id": 3306385,
      "author_name": "Alexander Chervov",
      "author_url": "",
      "post_date": "2025-10-24T13:08:44.503000",
      "content": "<p>Hi Michele<br>\nGreat work !<br>\nWhy can it work ?<br>\nNon experimental annotations became experimental ?<br>\nHave you used all anonstations or only experimental ? <br>\nOr data leakage ? And some proteins in test are already in unprot?</p>",
      "votes": 4,
      "replies": [
        {
          "id": 3306405,
          "author_name": "MT",
          "author_url": "",
          "post_date": "2025-10-24T13:42:29.810000",
          "content": "<p>Hi Alexander, great to see you here! I used all annotations regardless of evidence code, and only removed rows with NOT in the Qualifier. So I think it works because some electronic annotations have since been curated into experimental ones.</p>",
          "votes": 2,
          "replies": [
            {
              "id": 3306598,
              "author_name": "Alexander Chervov",
              "author_url": "",
              "post_date": "2025-10-24T21:09:22.320000",
              "content": "<p>Great to see you too ! <br>\nYes, I think you are right !</p>\n<p>But a bit surprising:<br>\ngoa_nonexp_predictions.tsv 0.182<br>\nthere is submit from ogranizers - it scores 0.182, much less than 0.26<br>\nwhich seems only electronic annotations.</p>\n<p>Would it mean that proteins in test include some annotations which are experimantal and already are in uniprot ? <br>\nIt seems it might be so, at least for some sub-ontlogies. </p>",
              "votes": 4,
              "replies": []
            },
            {
              "id": 3306600,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-10-24T21:30:47.603000",
              "content": "<p>I updated the baseline with the current UniProt release and I got 0.207 for \"goa_nonexp_predictions.tsv\" (all non-experimental evidence codes, not just electronic) and 0.269 if using all evidence codes (similar to MT's post). I was also surprised to see that just using all annotations made such a big improvement. Right now we are checking to see if any annotations in the leaderboard hold-out set (which should not be in the public database) were accidentally published. </p>\n<p>That being said, any annotations that are public before the submission deadline will not be included in the final evaluation. </p>",
              "votes": 5,
              "replies": []
            },
            {
              "id": 3306713,
              "author_name": "Bianco Chiu",
              "author_url": "",
              "post_date": "2025-10-25T04:59:10.953000",
              "content": "<p>Does that mean this GOA submission would be 0 in the final private leaderboard?</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3306726,
              "author_name": "MT",
              "author_url": "",
              "post_date": "2025-10-25T05:42:03.963000",
              "content": "<blockquote>\n  <p>any annotations that are public before the submission deadline will not be included in the final evaluation.</p>\n</blockquote>\n<p>Any <em>experimental</em> annotation, right?</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3307024,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-10-26T01:23:04.933000",
              "content": "<p>Yes, that is correct</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3307039,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-10-26T02:18:29.140000",
              "content": "<p>Before the submission deadline, we will collect the most updated public GO experimental annotations. These GO experimental annotations are considered \"known\" and will be excluded from the final private leaderboard (which will be assessed around in June 2026). So, if your submission only consists of GO experimental annotations, it will score 0. </p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 3307961,
              "author_name": "Mateusz Pieniak",
              "author_url": "",
              "post_date": "2025-10-28T08:41:15.533000",
              "content": "<p>So we might have both <strong>new</strong> experimental and non-experimental annotations in private leaderboard? Or the evaluation is done only on the experimental annotations that are discovered over time? </p>\n<p>EDIT: Read again the overview and the answer is that only <strong>experimentally-validated</strong> annotations will be in private leaderabaord.</p>\n<p>Follow-up question: What about <strong>NOT</strong> quantifier? Does it still count as \"scientific discovery\" or there will be only positive associations in private leaderboard? </p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3308089,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-10-28T14:40:11.460000",
              "content": "<p>Annotations with \"NOT\" qualifier will not be included in the ground truth for evaluation</p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 3308866,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-10-30T12:33:27.870000",
              "content": "<blockquote>\n  <p>Yes, that is correct.\n  so, the final code should not include the pulic annotations like gaf?And this may lead to the 0 point in final evaluation?</p>\n</blockquote>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3308871,
              "author_name": "John Mitchell",
              "author_url": "",
              "post_date": "2025-10-30T12:45:51.107000",
              "content": "<p>\"Yes, that is correct. so, the final code should not include the pulic annotations like gaf?And this may lead to the 0 point in final evaluation?\"</p>\n<p>I wouldn't reach that conclusion here. Including non-scored annotations would be neutral, not harmful. There may be benefits to including electronic annotations (i.e. bioinformatics predictions) as these have the potential to be confirmed by experiment.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3308882,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-10-30T13:21:35.580000",
              "content": "<p>Note that if you <em>only</em> have <strong>public experimental</strong> annotations from GAF, it will score 0. Otherwise, if you include non-experimental annotations (and these happen to be experimentally verified after submission deadline), you will score &gt;0. Like John said, including non-scored annotations would be neutral and not harmful.</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 3308971,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-10-30T16:18:13.730000",
              "content": "<blockquote>\n  <p>Note that if you <em>only</em> have <strong>public experimental</strong> annotations from GAF, it will score 0. Otherwise, if you include non-experimental annotations (and these happen to be experimentally verified after submission deadline), you will score &gt;0. Like John said, including non-scored annotations would be neutral and not harmful.</p>\n</blockquote>\n<p>Okey， so if the cafa5 use the submit result of cafa4 as reference. Will this be helpful to get better score in final result. I am wondering this because a code is using this.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3309002,
              "author_name": "John Mitchell",
              "author_url": "",
              "post_date": "2025-10-30T17:21:07.563000",
              "content": "<blockquote>\n  <p>Okey， so if the cafa5 use the submit result of cafa4 as reference. Will this be helpful to get better score in final result. I am wondering this because a code is using this.</p>\n</blockquote>\n<p>Yes, if a sequence is in the test sets of both CAFA 5 and CAFA 6, then a reused prediction originally made for CAFA 5 might end up scoring towards CAFA 6 if the experimental annotation happens to be published during the CAFA 6 scoring period.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 3320329,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-11-12T12:56:47.773000",
              "content": "<p>Sir, I want to doublecheck the experimental annotations, is the BASIC_EXP_CODES = {'EXP', 'IDA', 'IPI', 'IMP', 'IGI', 'IEP'}\nHTP_EXP_CODES = {'HTP', 'HDA', 'HMP', 'HGI', 'HEP'} are experimental and these are not experimental:   'IEA', 'IBA', 'ISO', 'ISS', 'IC', 'ISA', 'ISM', 'IKR', 'IGC', \n'TAS', 'NAS', 'RCA' , 'IBD', 'IRD'.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3320526,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-11-12T15:45:57.657000",
              "content": "<blockquote>\n  <p>experimental or high-throughput evidence, traceable author statement (evidence code TAS), or inferred by curator (IC)</p>\n</blockquote>\n<p>We used all the BASIC_EXP_CODES and HTP_EXP_CODES you listed, plus \"IC\" and \"TAS\" since these are high quality annotations. In the discussion comments, I just referred to them all as experimental annotations.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3320527,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-11-12T15:46:23.703000",
              "content": "<p>Hi，sir, are the IC and TAS experimental or non-experimental? IC:Inferred by Curator\nTAS:Traceable Author Statement</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3320530,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-11-12T15:48:49.087000",
              "content": "<p>IC and TAS are not considered experimental annotations. But note that we use experimental and IC and TAS annotations to build the train_terms.tsv and to build the evaluation ground truth.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3320591,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-11-12T16:29:08.367000",
              "content": "<p>And will you remove the annotation based on TAS and IC. In other side, regard them as experimental evidence. Or the experimenal annotation are only included in what the uniprot database says</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3320609,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-11-12T16:47:38.363000",
              "content": "<p>As I said, all of these codes will be used {'EXP', 'IDA', 'IPI', 'IMP', 'IGI', 'IEP', 'HTP', 'HDA', 'HMP', 'HGI', 'HEP', 'IC', 'TAS'}</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3322699,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-11-13T16:46:57.633000",
              "content": "<p>sir, I have another question, for the known label（experimenal go annotation） in training.set, will the prediction of them be score in the public lb? In other word, would they will be excluded in the calculation of public lb. Also, when use the evalutor, the ground truth file should exclude the training label? I think in private lb, they will not. </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3322921,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-11-13T20:42:52.047000",
              "content": "<p>The labels in the training data are currently excluded in the calculation of public lb. It does not matter if your ground truth file excludes the training labels or not, they will be propagated up to the subontology root. To exclude the training labels from the evaluation, you must pass the training labels in <code>-known</code> argument. Check the example here: <a href=\"https://github.com/claradepaolis/CAFA-evaluator-PK/tree/main/example\" target=\"_blank\">https://github.com/claradepaolis/CAFA-evaluator-PK/tree/main/example</a> </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3330568,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-11-16T09:24:29.263000",
              "content": "<p>Hi, sir. I have another question about where you get the seq(acid) and the taxonomy id of training and test set. I found an idea in my experiments. </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3332288,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-11-16T16:52:55.747000",
              "content": "<p>The sequence and the taxonomy id were obtained from the uniprot_sprot.fasta file which you can find inside this tar archive on <a href=\"https://ftp.ebi.ac.uk/pub/databases/uniprot/previous_releases/release-2025_03/knowledgebase/uniprot_sprot-only2025_03.tar.gz\" target=\"_blank\">UniProt FTP site</a>.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3371255,
              "author_name": "CavdadHun - VBECZ",
              "author_url": "",
              "post_date": "2025-12-11T11:30:16.917000",
              "content": "<p>Thank you.. as I see 'IC' is there, but what about \"ND\"??</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3371618,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-12-11T15:25:48.197000",
              "content": "<p>\"ND\" (stands for \"No biological Data available\") is not included in the training data. If you check the GAF file, most (if not all) ND annotations are just annotating the 3 subontology roots (\"GO:0008150\", \"GO:0003674\", \"GO:0005575\") so they won't be informative anyway. </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 3377820,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-12-17T02:01:46.403000",
              "content": "<p>Do this mean that what in the gaf has some experimental overlap with the holdout experimental annotation</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3308046,
      "author_name": "John Mitchell",
      "author_url": "",
      "post_date": "2025-10-28T13:19:50.573000",
      "content": "<p>I think my take-home from this is that while any extant experimental GO annotation won't contribute to the private score, it is potentially worth including \"Inferred from Electronic Annotation (IEA)\" data (essentially bioinformatics predictions) in our submissions. These could contribute to the score if they happen to be experimentally confirmed at the right time. </p>",
      "votes": 2,
      "replies": [
        {
          "id": 3394958,
          "author_name": "",
          "author_url": "",
          "post_date": "2026-01-22T00:40:54.893000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3371350,
      "author_name": "Khushi Yadav",
      "author_url": "",
      "post_date": "2025-12-11T12:28:40.240000",
      "content": "<p>CAFA 6 Protein Function Prediction</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3306368": "I built a simple baseline by mapping UniProt proteins to Gene Ontology (GO) terms directly from **GOA UniProt** and removing negated (“NOT …”) annotations. Submitting the remaining terms for IDs that overlap with the **test superset** yielded **~0.269 on the public leaderboard** for me. I’ll share a Polars-based parser for the large GAF file in a follow-up post.\n\n\n## What is a GAF?\n\n**GAF (Gene Association File)** is a tab-separated standard used to distribute GO annotations. Each row links a database object (e.g., a UniProt accession) to a GO term, alongside evidence and metadata. In GAF v2.2, the key fields are:\n\n- **DB_Object_ID** (column 2): the stable identifier for the entity (e.g., UniProt ID like `P12345`).  \n- **Qualifier** (column 4): may include relationship labels (e.g., `enables`, `involved_in`) and **negations** using `NOT`.  \n- **GO_ID** (column 5): the Gene Ontology term (e.g., `GO:0005515`).  \n- Comment lines start with `!` and should be ignored.\n\nWhy care about **NOT**? A `NOT` qualifier explicitly states that the gene/product is **not** associated with that term/relationship. For leaderboard alignment, I removed these negated associations before building the submission.\n\n---\n\n## Data source\n\nDownload the comprehensive GOA UniProt file from EBI’s FTP:\n\n- **`goa_uniprot_all.gaf`** (compressed as `.gz`)  \n  Path: `https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/`\n---\n\n## What exactly did I remove?\n\nFrom the **Qualifier** field, I excluded rows containing any of the following negated qualifiers (and, more generally, anything with `NOT` in it):\n\n- `NOT|involved_in`  \n- `NOT|is_active_in`  \n- `NOT|colocalizes_with`  \n- `NOT|enables`  \n- `NOT|acts_upstream_of_or_within`  \n- `NOT|acts_upstream_of_or_within_negative_effect`  \n- `NOT|contributes_to`  \n- `NOT|part_of`  \n- `NOT|acts_upstream_of`  \n- `NOT|located_in`  \n- `NOT|acts_upstream_of_or_within_positive_effect`\n\nThis removes explicit negations so we only submit positive associations.\n\n---\n\n## Building the submission\n\n1. **Load the test superset IDs.**  \n   Use the competition’s provided test set/superset to obtain the list of UniProt accessions.\n\n2. **Parse GOA UniProt (GAF).**  \n   - Skip comment lines (`!`).  \n   - Extract **DB_Object_ID** (col 2), **Qualifier** (col 4), and **GO_ID** (col 5).\n\n3. **Filter to test overlap.**  \n   Keep only rows where **DB_Object_ID** is in the test superset.\n\n4. **Drop negations.**  \n   Exclude rows where **Qualifier** contains `NOT` or matches any of the specific negated patterns listed above.\n\n6. **Write `submission.tsv`.**  \n   ```\n   protein_id    go_term 1\n   ```\n\nSubmitting the above mapping (no model, no ranking) gave me **~0.269 LB**. ",
    "3306385": "Hi Michele\nGreat work !\nWhy can it work ?\nNon experimental annotations became experimental ?\nHave you used all anonstations or only experimental ? \nOr data leakage ? And some proteins in test are already in unprot?",
    "3308046": "I think my take-home from this is that while any extant experimental GO annotation won't contribute to the private score, it is potentially worth including \"Inferred from Electronic Annotation (IEA)\" data (essentially bioinformatics predictions) in our submissions. These could contribute to the score if they happen to be experimentally confirmed at the right time. ",
    "3371350": "CAFA 6 Protein Function Prediction"
  }
}