{
  "id": 657230,
  "title": "Inquiry regarding the benchmark entries on the LB (e.g., \"goa_predictions_Oct2025.tsv\")",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/657230",
  "author_name": "Emiri Watanabe",
  "post_date": "2025-12-10T07:57:48.469000",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hello everyone,</p>\n<p>I noticed some entries on the leaderboard that appear to be benchmarks or baselines, specifically files like \"goa_predictions_Oct2025.tsv\".</p>\n<p>I am interested in understanding how these predictions were generated. Could anyone kindly point me to where the methodology or description for these submissions is documented?</p>\n<p>Thank you in advance for your help.</p>",
  "messages": [
    {
      "id": 3371613,
      "postDate": "2025-12-11T15:21:41.183Z",
      "content": "<p>The \"goa_nonexp_predictions_Oct2025.tsv\" were created using non-experimental evidence codes in the GAF file (release 2025-09-04, downloaded <a href=\"https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.227.gz\" target=\"_blank\">here</a>). You can re-create this baseline submission with the following steps:</p>\n<ul>\n<li>Retain only UniProt IDs that appear in the test_superset.fasta</li>\n<li>Filter out \"NOT\" annotations in the GAF file</li>\n<li>Retain only annotations with all non-experimental <a href=\"https://geneontology.org/docs/guide-go-evidence-codes/\" target=\"_blank\">evidence codes</a> {\"ISS\", \"ISO\", \"ISA\", \"ISM\", \"IGC\", \"RCA\", \"IBA\", \"IBD\", \"IKR\", \"IRD\", \"NAS\", \"ND\", \"IEA\"}</li>\n<li>Assign a score of 1.0 to all the remaining annotations</li>\n</ul>",
      "rawMarkdown": "The \"goa_nonexp_predictions_Oct2025.tsv\" were created using non-experimental evidence codes in the GAF file (release 2025-09-04, downloaded [here](https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.227.gz)). You can re-create this baseline submission with the following steps:\n\n- Retain only UniProt IDs that appear in the test_superset.fasta\n- Filter out \"NOT\" annotations in the GAF file\n- Retain only annotations with all non-experimental [evidence codes](https://geneontology.org/docs/guide-go-evidence-codes/) {\"ISS\", \"ISO\", \"ISA\", \"ISM\", \"IGC\", \"RCA\", \"IBA\", \"IBD\", \"IKR\", \"IRD\", \"NAS\", \"ND\", \"IEA\"}\n- Assign a score of 1.0 to all the remaining annotations",
      "votes": 2,
      "replies": [
        {
          "id": 3372542,
          "postDate": "2025-12-12T04:26:41.663Z",
          "content": "<p>Thank you for the clear instructions! I understand how the \"goa_nonexp_predictions_Oct2025.tsv\" baseline was created now. This information helps me a lot!</p>\n<p>I also noticed other entries such as \"goa_predictions_Oct2025.tsv\", \"naive_predictions.tsv\", \"goa_nonexp_predictions.tsv, \"blast_predictions.tsv\" on the Leaderboard. Could you please share how these were generated as well?</p>",
          "rawMarkdown": "Thank you for the clear instructions! I understand how the \"goa_nonexp_predictions_Oct2025.tsv\" baseline was created now. This information helps me a lot!\n\nI also noticed other entries such as \"goa_predictions_Oct2025.tsv\", \"naive_predictions.tsv\", \"goa_nonexp_predictions.tsv, \"blast_predictions.tsv\" on the Leaderboard. Could you please share how these were generated as well?",
          "votes": 2,
          "replies": [
            {
              "id": 3377041,
              "postDate": "2025-12-15T15:44:52.707Z",
              "content": "<p>\"goa_nonexp_predictions.tsv\" was similarly created, but with GAF file release on 2025-05-03 (most updated file before competition launches, downloaded <a href=\"https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.226.gz\" target=\"_blank\">here</a>).</p>\n<p>\"naive_predictions.tsv\" predicts every GO term for every protein in the \"testsuperset.fasta\" file; the score of each prediction is the frequency of the GO term in the GAF file. For example, if \"GO:0000001\" was annotated to 50% of all proteins in the \"train_terms.tsv\" file, then \"naive_predictions.tsv\" will have \"GO:0000001\" predictions for all proteins with score 0.5 for each prediction.</p>\n<p>\"blast_predictions.tsv\" was created using two steps:</p>\n<ul>\n<li>Do a BLASTp for every protein in the test superset against a BLAST database (BLAST db contains sequences from train_sequences.fasta) and get BLAST \"pident\" score (percent sequence identity) for each query sequence-database sequence pair</li>\n<li>For each test sequence, transfer all GO terms of proteins that are its BLAST similarity hits, with the BLAST pident score as the confidence score</li>\n</ul>",
              "rawMarkdown": "\"goa_nonexp_predictions.tsv\" was similarly created, but with GAF file release on 2025-05-03 (most updated file before competition launches, downloaded [here](https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.226.gz)).\n\n\"naive_predictions.tsv\" predicts every GO term for every protein in the \"testsuperset.fasta\" file; the score of each prediction is the frequency of the GO term in the GAF file. For example, if \"GO:0000001\" was annotated to 50% of all proteins in the \"train_terms.tsv\" file, then \"naive_predictions.tsv\" will have \"GO:0000001\" predictions for all proteins with score 0.5 for each prediction.\n\n\"blast_predictions.tsv\" was created using two steps:\n- Do a BLASTp for every protein in the test superset against a BLAST database (BLAST db contains sequences from train_sequences.fasta) and get BLAST \"pident\" score (percent sequence identity) for each query sequence-database sequence pair\n- For each test sequence, transfer all GO terms of proteins that are its BLAST similarity hits, with the BLAST pident score as the confidence score",
              "votes": 5
            }
          ]
        }
      ]
    },
    {
      "id": 3369638,
      "postDate": "2025-12-10T07:57:48.470Z",
      "content": "<p>Hello everyone,</p>\n<p>I noticed some entries on the leaderboard that appear to be benchmarks or baselines, specifically files like \"goa_predictions_Oct2025.tsv\".</p>\n<p>I am interested in understanding how these predictions were generated. Could anyone kindly point me to where the methodology or description for these submissions is documented?</p>\n<p>Thank you in advance for your help.</p>",
      "rawMarkdown": "Hello everyone,\n\nI noticed some entries on the leaderboard that appear to be benchmarks or baselines, specifically files like \"goa_predictions_Oct2025.tsv\".\n\nI am interested in understanding how these predictions were generated. Could anyone kindly point me to where the methodology or description for these submissions is documented?\n\nThank you in advance for your help.",
      "votes": 2
    },
    {
      "id": 3372540,
      "postDate": "2025-12-12T04:26:14.553Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 3371613,
      "author_name": "An Phan",
      "author_url": "",
      "post_date": "2025-12-11T15:21:41.183000",
      "content": "<p>The \"goa_nonexp_predictions_Oct2025.tsv\" were created using non-experimental evidence codes in the GAF file (release 2025-09-04, downloaded <a href=\"https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.227.gz\" target=\"_blank\">here</a>). You can re-create this baseline submission with the following steps:</p>\n<ul>\n<li>Retain only UniProt IDs that appear in the test_superset.fasta</li>\n<li>Filter out \"NOT\" annotations in the GAF file</li>\n<li>Retain only annotations with all non-experimental <a href=\"https://geneontology.org/docs/guide-go-evidence-codes/\" target=\"_blank\">evidence codes</a> {\"ISS\", \"ISO\", \"ISA\", \"ISM\", \"IGC\", \"RCA\", \"IBA\", \"IBD\", \"IKR\", \"IRD\", \"NAS\", \"ND\", \"IEA\"}</li>\n<li>Assign a score of 1.0 to all the remaining annotations</li>\n</ul>",
      "votes": 2,
      "replies": [
        {
          "id": 3372542,
          "author_name": "Emiri Watanabe",
          "author_url": "",
          "post_date": "2025-12-12T04:26:41.663000",
          "content": "<p>Thank you for the clear instructions! I understand how the \"goa_nonexp_predictions_Oct2025.tsv\" baseline was created now. This information helps me a lot!</p>\n<p>I also noticed other entries such as \"goa_predictions_Oct2025.tsv\", \"naive_predictions.tsv\", \"goa_nonexp_predictions.tsv, \"blast_predictions.tsv\" on the Leaderboard. Could you please share how these were generated as well?</p>",
          "votes": 2,
          "replies": [
            {
              "id": 3377041,
              "author_name": "An Phan",
              "author_url": "",
              "post_date": "2025-12-15T15:44:52.707000",
              "content": "<p>\"goa_nonexp_predictions.tsv\" was similarly created, but with GAF file release on 2025-05-03 (most updated file before competition launches, downloaded <a href=\"https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.226.gz\" target=\"_blank\">here</a>).</p>\n<p>\"naive_predictions.tsv\" predicts every GO term for every protein in the \"testsuperset.fasta\" file; the score of each prediction is the frequency of the GO term in the GAF file. For example, if \"GO:0000001\" was annotated to 50% of all proteins in the \"train_terms.tsv\" file, then \"naive_predictions.tsv\" will have \"GO:0000001\" predictions for all proteins with score 0.5 for each prediction.</p>\n<p>\"blast_predictions.tsv\" was created using two steps:</p>\n<ul>\n<li>Do a BLASTp for every protein in the test superset against a BLAST database (BLAST db contains sequences from train_sequences.fasta) and get BLAST \"pident\" score (percent sequence identity) for each query sequence-database sequence pair</li>\n<li>For each test sequence, transfer all GO terms of proteins that are its BLAST similarity hits, with the BLAST pident score as the confidence score</li>\n</ul>",
              "votes": 5,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3372540,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-12-12T04:26:14.553000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3371613": "The \"goa_nonexp_predictions_Oct2025.tsv\" were created using non-experimental evidence codes in the GAF file (release 2025-09-04, downloaded [here](https://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/goa_uniprot_all.gaf.227.gz)). You can re-create this baseline submission with the following steps:\n\n- Retain only UniProt IDs that appear in the test_superset.fasta\n- Filter out \"NOT\" annotations in the GAF file\n- Retain only annotations with all non-experimental [evidence codes](https://geneontology.org/docs/guide-go-evidence-codes/) {\"ISS\", \"ISO\", \"ISA\", \"ISM\", \"IGC\", \"RCA\", \"IBA\", \"IBD\", \"IKR\", \"IRD\", \"NAS\", \"ND\", \"IEA\"}\n- Assign a score of 1.0 to all the remaining annotations",
    "3369638": "Hello everyone,\n\nI noticed some entries on the leaderboard that appear to be benchmarks or baselines, specifically files like \"goa_predictions_Oct2025.tsv\".\n\nI am interested in understanding how these predictions were generated. Could anyone kindly point me to where the methodology or description for these submissions is documented?\n\nThank you in advance for your help.",
    "3372540": ""
  }
}