{
  "id": 671902,
  "title": "Approaches for this competition ",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/671902",
  "author_name": "",
  "post_date": "2026-02-04T20:38:09.838331500Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Now that the competition is over, I am curious to know everyone's method for achieving a high score on the public leaderboard (besides ensembling with GOA). Did you notice any discrepancies between your validation performance and the public leaderboard score? And what were the best performances on your validation set and the public leaderboard?</p>\n<p>My approach involves implementing a hierarchical multi-label neural network. It predicts scores both globally and at each level separately, and the final result is the average of these two predictions.  For my evaluation, I performed a simple train-validation split, using a random sample of 10,000 for the validation set. During validation, I achieved a weighted f-max score of approximately 0.58. However, I could only obtain a weighted f-max score of 0.30 on the public leaderboard. I am unsure which one would be a better representation of the test performance. </p>",
  "messages": [
    {
      "id": "3401917",
      "postDate": "02/04/2026 20:38:09",
      "content": "<p>Now that the competition is over, I am curious to know everyone's method for achieving a high score on the public leaderboard (besides ensembling with GOA). Did you notice any discrepancies between your validation performance and the public leaderboard score? And what were the best performances on your validation set and the public leaderboard?</p>\n<p>My approach involves implementing a hierarchical multi-label neural network. It predicts scores both globally and at each level separately, and the final result is the average of these two predictions.  For my evaluation, I performed a simple train-validation split, using a random sample of 10,000 for the validation set. During validation, I achieved a weighted f-max score of approximately 0.58. However, I could only obtain a weighted f-max score of 0.30 on the public leaderboard. I am unsure which one would be a better representation of the test performance. </p>",
      "rawMarkdown": "Now that the competition is over, I am curious to know everyone's method for achieving a high score on the public leaderboard (besides ensembling with GOA). Did you notice any discrepancies between your validation performance and the public leaderboard score? And what were the best performances on your validation set and the public leaderboard?\n\nMy approach involves implementing a hierarchical multi-label neural network. It predicts scores both globally and at each level separately, and the final result is the average of these two predictions.  For my evaluation, I performed a simple train-validation split, using a random sample of 10,000 for the validation set. During validation, I achieved a weighted f-max score of approximately 0.58. However, I could only obtain a weighted f-max score of 0.30 on the public leaderboard. I am unsure which one would be a better representation of the test performance.",
      "votes": null
    },
    {
      "id": "3402100",
      "postDate": "02/05/2026 09:07:45",
      "content": "<p>HI!,</p>\n<p>I evaluated a custom version of <strong>FANTASIA</strong>, an embedding-proximity transfer tool. In this iteration I replaced the usual truncation strategy with a <strong>sliding-window matching</strong> approach, i.e., I match <em>slices</em> of embeddings rather than cutting sequences, aiming to reduce truncation bias for long proteins.</p>\n<p>In my pipeline, the attribute that consistently provided the strongest signal was <strong>sequence identity</strong> (used as the primary transfer criterion / weighting feature).</p>\n<p>For evaluation I used a <strong>GAF time-split from GAF224 → GAF228</strong> over <strong>reviewed proteins (~500k proteins)</strong>, and obtained the following:</p>\n<table>\n<thead>\n<tr>\n<th>Split</th>\n<th>Namespace</th>\n<th>tau</th>\n<th>f_w</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>LK</strong></td>\n<td>biological_process</td>\n<td><strong>0.980</strong></td>\n<td><strong>0.288</strong></td>\n</tr>\n<tr>\n<td><strong>LK</strong></td>\n<td>cellular_component</td>\n<td><strong>0.940</strong></td>\n<td><strong>0.384</strong></td>\n</tr>\n<tr>\n<td><strong>LK</strong></td>\n<td>molecular_function</td>\n<td><strong>0.910</strong></td>\n<td><strong>0.323</strong></td>\n</tr>\n<tr>\n<td><strong>NK</strong></td>\n<td>biological_process</td>\n<td><strong>0.520</strong></td>\n<td><strong>0.345</strong></td>\n</tr>\n<tr>\n<td><strong>NK</strong></td>\n<td>cellular_component</td>\n<td><strong>0.530</strong></td>\n<td><strong>0.511</strong></td>\n</tr>\n<tr>\n<td><strong>NK</strong></td>\n<td>molecular_function</td>\n<td><strong>0.550</strong></td>\n<td><strong>0.522</strong></td>\n</tr>\n<tr>\n<td><strong>PK</strong></td>\n<td>biological_process</td>\n<td><strong>0.810</strong></td>\n<td><strong>0.296</strong></td>\n</tr>\n<tr>\n<td><strong>PK</strong></td>\n<td>cellular_component</td>\n<td><strong>0.910</strong></td>\n<td><strong>0.432</strong></td>\n</tr>\n<tr>\n<td><strong>PK</strong></td>\n<td>molecular_function</td>\n<td><strong>0.910</strong></td>\n<td><strong>0.354</strong></td>\n</tr>\n</tbody>\n</table>\n<p>I set up the validation environment too late, so I didn’t have much room to properly tune parameters. For the final production submission I used a richer reference table (fed with updated <strong>QuickGO</strong>) and tried to calibrate FANTASIA together with additional ML predictions to improve recall, but I wasn’t able to obtain a consistent gain.</p>\n<p>I also experimented with weighting the <strong>evidence codes</strong> in the biological ground truth to reshape the ranking (making it smoother) so that my teammates’ ML predictions could be incorporated more safely, but I didn’t achieve much success.</p>\n<p>By the way, one thing I computed that might be particularly interesting is an evidence-code analysis of “conversion” into new successful annotations (I’ll paste the table below).</p>\n<table>\n<thead>\n<tr>\n<th>evidence_code</th>\n<th>aspect</th>\n<th>N_T0</th>\n<th>N_T0_to_SUCCESS_T1</th>\n<th>conversion_rate</th>\n<th>support_ratio</th>\n<th>effective_weight</th>\n<th>w_norm</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>IEA</td>\n<td>C</td>\n<td>1779720</td>\n<td>28681</td>\n<td>0.016115</td>\n<td>0.769703</td>\n<td>0.11137375179896992</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>IBA</td>\n<td>C</td>\n<td>102092</td>\n<td>4575</td>\n<td>0.044813</td>\n<td>0.044153</td>\n<td>0.04448168745392747</td>\n<td>0.519513</td>\n</tr>\n<tr>\n<td>ISO</td>\n<td>C</td>\n<td>30913</td>\n<td>883</td>\n<td>0.028564</td>\n<td>0.013369</td>\n<td>0.019541867163925707</td>\n<td>0.34037</td>\n</tr>\n<tr>\n<td>NAS</td>\n<td>C</td>\n<td>9306</td>\n<td>881</td>\n<td>0.09467</td>\n<td>0.004025</td>\n<td>0.019519723397987458</td>\n<td>0.340211</td>\n</tr>\n<tr>\n<td>ISS</td>\n<td>C</td>\n<td>64010</td>\n<td>627</td>\n<td>0.009795</td>\n<td>0.027683</td>\n<td>0.01646718920382097</td>\n<td>0.318284</td>\n</tr>\n<tr>\n<td>ISA</td>\n<td>C</td>\n<td>1107</td>\n<td>55</td>\n<td>0.049684</td>\n<td>0.000479</td>\n<td>0.004877160522364997</td>\n<td>0.235033</td>\n</tr>\n<tr>\n<td>ISM</td>\n<td>C</td>\n<td>632</td>\n<td>38</td>\n<td>0.060127</td>\n<td>0.000273</td>\n<td>0.004053941742674189</td>\n<td>0.22912</td>\n</tr>\n<tr>\n<td>RCA</td>\n<td>C</td>\n<td>972</td>\n<td>20</td>\n<td>0.020576</td>\n<td>0.00042</td>\n<td>0.002941038453639716</td>\n<td>0.221126</td>\n</tr>\n<tr>\n<td>ND</td>\n<td>C</td>\n<td>6667</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.002883</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IKR</td>\n<td>C</td>\n<td>9</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.000004</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IGC</td>\n<td>C</td>\n<td>43</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.000019</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IEA</td>\n<td>F</td>\n<td>2732877</td>\n<td>16897</td>\n<td>0.006183</td>\n<td>0.799986</td>\n<td>0.07032923462609807</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>IBA</td>\n<td>F</td>\n<td>95385</td>\n<td>7293</td>\n<td>0.076459</td>\n<td>0.027922</td>\n<td>0.04620449159177866</td>\n<td>0.725579</td>\n</tr>\n<tr>\n<td>ISO</td>\n<td>F</td>\n<td>24114</td>\n<td>788</td>\n<td>0.032678</td>\n<td>0.007059</td>\n<td>0.015187776510689283</td>\n<td>0.372762</td>\n</tr>\n<tr>\n<td>ISS</td>\n<td>F</td>\n<td>54542</td>\n<td>639</td>\n<td>0.011716</td>\n<td>0.015966</td>\n<td>0.013676706217920765</td>\n<td>0.355573</td>\n</tr>\n<tr>\n<td>NAS</td>\n<td>F</td>\n<td>3086</td>\n<td>259</td>\n<td>0.083927</td>\n<td>0.000903</td>\n<td>0.008707249141087028</td>\n<td>0.299046</td>\n</tr>\n<tr>\n<td>RCA</td>\n<td>F</td>\n<td>1346</td>\n<td>86</td>\n<td>0.063893</td>\n<td>0.000394</td>\n<td>0.00501741836046824</td>\n<td>0.257073</td>\n</tr>\n<tr>\n<td>ISA</td>\n<td>F</td>\n<td>1241</td>\n<td>70</td>\n<td>0.056406</td>\n<td>0.000363</td>\n<td>0.004526683279715323</td>\n<td>0.251491</td>\n</tr>\n<tr>\n<td>ISM</td>\n<td>F</td>\n<td>1059</td>\n<td>66</td>\n<td>0.062323</td>\n<td>0.00031</td>\n<td>0.004395447092455655</td>\n<td>0.249999</td>\n</tr>\n<tr>\n<td>IGC</td>\n<td>F</td>\n<td>69</td>\n<td>2</td>\n<td>0.028986</td>\n<td>0.00002</td>\n<td>0.0007651491267319679</td>\n<td>0.208704</td>\n</tr>\n<tr>\n<td>IKR</td>\n<td>F</td>\n<td>117</td>\n<td>1</td>\n<td>0.008547</td>\n<td>0.000034</td>\n<td>0.0005410421361311397</td>\n<td>0.206154</td>\n</tr>\n<tr>\n<td>ND</td>\n<td>F</td>\n<td>13810</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.004043</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IEA</td>\n<td>P</td>\n<td>1904841</td>\n<td>10786</td>\n<td>0.005662</td>\n<td>0.744451</td>\n<td>0.06492603939817149</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>IBA</td>\n<td>P</td>\n<td>115813</td>\n<td>8675</td>\n<td>0.074905</td>\n<td>0.045262</td>\n<td>0.058226869944677836</td>\n<td>0.917455</td>\n</tr>\n<tr>\n<td>ISO</td>\n<td>P</td>\n<td>68384</td>\n<td>1240</td>\n<td>0.018133</td>\n<td>0.026726</td>\n<td>0.022014029573268416</td>\n<td>0.471251</td>\n</tr>\n<tr>\n<td>ISS</td>\n<td>P</td>\n<td>115249</td>\n<td>769</td>\n<td>0.006673</td>\n<td>0.045042</td>\n<td>0.017336118738209134</td>\n<td>0.413611</td>\n</tr>\n<tr>\n<td>NAS</td>\n<td>P</td>\n<td>15317</td>\n<td>696</td>\n<td>0.04544</td>\n<td>0.005986</td>\n<td>0.01649275937526058</td>\n<td>0.403219</td>\n</tr>\n<tr>\n<td>ISA</td>\n<td>P</td>\n<td>751</td>\n<td>16</td>\n<td>0.021305</td>\n<td>0.000294</td>\n<td>0.002500625723120532</td>\n<td>0.230812</td>\n</tr>\n<tr>\n<td>IGC</td>\n<td>P</td>\n<td>129</td>\n<td>14</td>\n<td>0.108527</td>\n<td>0.00005</td>\n<td>0.0023391211771177168</td>\n<td>0.228822</td>\n</tr>\n<tr>\n<td>ISM</td>\n<td>P</td>\n<td>284</td>\n<td>11</td>\n<td>0.038732</td>\n<td>0.000111</td>\n<td>0.0020734093161754886</td>\n<td>0.225548</td>\n</tr>\n<tr>\n<td>RCA</td>\n<td>P</td>\n<td>1508</td>\n<td>7</td>\n<td>0.004642</td>\n<td>0.000589</td>\n<td>0.0016540084463569972</td>\n<td>0.22038</td>\n</tr>\n<tr>\n<td>ND</td>\n<td>P</td>\n<td>8327</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.003254</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IKR</td>\n<td>P</td>\n<td>16</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.000006</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n</tbody>\n</table>\n<p>Anyway, my performance on the public leaderboard was always quite low. I couldn’t consistently reach 0.40 without merging/ensembling signals; on its own, my most curated submission was only around 0.35.</p>",
      "rawMarkdown": "HI!,\n\nI evaluated a custom version of **FANTASIA**, an embedding-proximity transfer tool. In this iteration I replaced the usual truncation strategy with a **sliding-window matching** approach, i.e., I match *slices* of embeddings rather than cutting sequences, aiming to reduce truncation bias for long proteins.\n\nIn my pipeline, the attribute that consistently provided the strongest signal was **sequence identity** (used as the primary transfer criterion / weighting feature).\n\nFor evaluation I used a **GAF time-split from GAF224 → GAF228** over **reviewed proteins (~500k proteins)**, and obtained the following:\n\n| Split  | Namespace          | tau       | f_w       |\n| ------ | ------------------ | --------- | --------- |\n| **LK** | biological_process | **0.980** | **0.288** |\n| **LK** | cellular_component | **0.940** | **0.384** |\n| **LK** | molecular_function | **0.910** | **0.323** |\n| **NK** | biological_process | **0.520** | **0.345** |\n| **NK** | cellular_component | **0.530** | **0.511** |\n| **NK** | molecular_function | **0.550** | **0.522** |\n| **PK** | biological_process | **0.810** | **0.296** |\n| **PK** | cellular_component | **0.910** | **0.432** |\n| **PK** | molecular_function | **0.910** | **0.354** |\n\nI set up the validation environment too late, so I didn’t have much room to properly tune parameters. For the final production submission I used a richer reference table (fed with updated **QuickGO**) and tried to calibrate FANTASIA together with additional ML predictions to improve recall, but I wasn’t able to obtain a consistent gain.\n\nI also experimented with weighting the **evidence codes** in the biological ground truth to reshape the ranking (making it smoother) so that my teammates’ ML predictions could be incorporated more safely, but I didn’t achieve much success.\n\nBy the way, one thing I computed that might be particularly interesting is an evidence-code analysis of “conversion” into new successful annotations (I’ll paste the table below).\n\n| evidence_code | aspect | N_T0    | N_T0_to_SUCCESS_T1 | conversion_rate | support_ratio | effective_weight      | w_norm   |\n| ------------- | ------ | ------- | ------------------ | --------------- | ------------- | --------------------- | -------- |\n| IEA           | C      | 1779720 | 28681              | 0.016115        | 0.769703      | 0.11137375179896992   | 1.0      |\n| IBA           | C      | 102092  | 4575               | 0.044813        | 0.044153      | 0.04448168745392747   | 0.519513 |\n| ISO           | C      | 30913   | 883                | 0.028564        | 0.013369      | 0.019541867163925707  | 0.34037  |\n| NAS           | C      | 9306    | 881                | 0.09467         | 0.004025      | 0.019519723397987458  | 0.340211 |\n| ISS           | C      | 64010   | 627                | 0.009795        | 0.027683      | 0.01646718920382097   | 0.318284 |\n| ISA           | C      | 1107    | 55                 | 0.049684        | 0.000479      | 0.004877160522364997  | 0.235033 |\n| ISM           | C      | 632     | 38                 | 0.060127        | 0.000273      | 0.004053941742674189  | 0.22912  |\n| RCA           | C      | 972     | 20                 | 0.020576        | 0.00042       | 0.002941038453639716  | 0.221126 |\n| ND            | C      | 6667    | 0                  | 0.0             | 0.002883      | 0.0                   | 0.2      |\n| IKR           | C      | 9       | 0                  | 0.0             | 0.000004      | 0.0                   | 0.2      |\n| IGC           | C      | 43      | 0                  | 0.0             | 0.000019      | 0.0                   | 0.2      |\n| IEA           | F      | 2732877 | 16897              | 0.006183        | 0.799986      | 0.07032923462609807   | 1.0      |\n| IBA           | F      | 95385   | 7293               | 0.076459        | 0.027922      | 0.04620449159177866   | 0.725579 |\n| ISO           | F      | 24114   | 788                | 0.032678        | 0.007059      | 0.015187776510689283  | 0.372762 |\n| ISS           | F      | 54542   | 639                | 0.011716        | 0.015966      | 0.013676706217920765  | 0.355573 |\n| NAS           | F      | 3086    | 259                | 0.083927        | 0.000903      | 0.008707249141087028  | 0.299046 |\n| RCA           | F      | 1346    | 86                 | 0.063893        | 0.000394      | 0.00501741836046824   | 0.257073 |\n| ISA           | F      | 1241    | 70                 | 0.056406        | 0.000363      | 0.004526683279715323  | 0.251491 |\n| ISM           | F      | 1059    | 66                 | 0.062323        | 0.00031       | 0.004395447092455655  | 0.249999 |\n| IGC           | F      | 69      | 2                  | 0.028986        | 0.00002       | 0.0007651491267319679 | 0.208704 |\n| IKR           | F      | 117     | 1                  | 0.008547        | 0.000034      | 0.0005410421361311397 | 0.206154 |\n| ND            | F      | 13810   | 0                  | 0.0             | 0.004043      | 0.0                   | 0.2      |\n| IEA           | P      | 1904841 | 10786              | 0.005662        | 0.744451      | 0.06492603939817149   | 1.0      |\n| IBA           | P      | 115813  | 8675               | 0.074905        | 0.045262      | 0.058226869944677836  | 0.917455 |\n| ISO           | P      | 68384   | 1240               | 0.018133        | 0.026726      | 0.022014029573268416  | 0.471251 |\n| ISS           | P      | 115249  | 769                | 0.006673        | 0.045042      | 0.017336118738209134  | 0.413611 |\n| NAS           | P      | 15317   | 696                | 0.04544         | 0.005986      | 0.01649275937526058   | 0.403219 |\n| ISA           | P      | 751     | 16                 | 0.021305        | 0.000294      | 0.002500625723120532  | 0.230812 |\n| IGC           | P      | 129     | 14                 | 0.108527        | 0.00005       | 0.0023391211771177168 | 0.228822 |\n| ISM           | P      | 284     | 11                 | 0.038732        | 0.000111      | 0.0020734093161754886 | 0.225548 |\n| RCA           | P      | 1508    | 7                  | 0.004642        | 0.000589      | 0.0016540084463569972 | 0.22038  |\n| ND            | P      | 8327    | 0                  | 0.0             | 0.003254      | 0.0                   | 0.2      |\n| IKR           | P      | 16      | 0                  | 0.0             | 0.000006      | 0.0                   | 0.2      |\n\n\nAnyway, my performance on the public leaderboard was always quite low. I couldn’t consistently reach 0.40 without merging/ensembling signals; on its own, my most curated submission was only around 0.35.",
      "votes": null
    },
    {
      "id": "3402186",
      "postDate": "02/05/2026 12:04:35",
      "content": "<p>I think your line of thinking is probably correct. For a more robust score improvement, I believe we could either train the model with a larger dataset or apply ensemble learning. And I reckon overfitting is an unavoidable factor behind my current score. 😭</p>",
      "rawMarkdown": "I think your line of thinking is probably correct. For a more robust score improvement, I believe we could either train the model with a larger dataset or apply ensemble learning. And I reckon overfitting is an unavoidable factor behind my current score. 😭",
      "votes": null
    },
    {
      "id": "3402629",
      "postDate": "02/06/2026 12:45:29",
      "content": "<p>I tried a go-embedding / protein-embedding matching via dot product, and learned smoothing/refinement over the GO hierarchy afterwards. My idea was that there should be a lot of untapped potential in having the go-nodes communicate via learned weights with each other. </p>\n<p>Unfortunately, I did not get to a proper validation pipeline done in the 1 month I had. First pipeline was simple train-test split of proteins, with a severe data leakage (I guess from near-identical proteins) -&gt; Still got somehow to 0.3 on the leaderboard. \nFor the  second and last validation pipeline, I did a proper timesplit (train until mid 2024, validation after). This (mostly) got rid of the train-val leak… but now everything was just bad ;-) . Never got close to 0.3 on the leaderboard from before\n(maybe leaving out 1.5y for validation was too much. I tested much fewer entries than with the \"leaking\" validation pipeline, the whole point of revising the leaderboard was to rely on the annoying large-file export to .tsv and upload to the leaderboard so much)</p>\n<p>Another issue in this joint model across go hierarchies: The different CC/MF/BP hierarchies seemed to need its own network respectively. More epochs, more layers for CC, less of both for MF (which was far better in fMax than the other 2 for me. Likely also needed more and different features)</p>\n<p>I tried a weighted Average Precision score as a substitute to the jiggly f-max, but that went terrible for some reason.</p>\n<p>Happy to hear if anyone tried similar ideas, and what the results were!</p>",
      "rawMarkdown": "I tried a go-embedding / protein-embedding matching via dot product, and learned smoothing/refinement over the GO hierarchy afterwards. My idea was that there should be a lot of untapped potential in having the go-nodes communicate via learned weights with each other. \n\nUnfortunately, I did not get to a proper validation pipeline done in the 1 month I had. First pipeline was simple train-test split of proteins, with a severe data leakage (I guess from near-identical proteins) -> Still got somehow to 0.3 on the leaderboard. \nFor the  second and last validation pipeline, I did a proper timesplit (train until mid 2024, validation after). This (mostly) got rid of the train-val leak... but now everything was just bad ;-) . Never got close to 0.3 on the leaderboard from before\n(maybe leaving out 1.5y for validation was too much. I tested much fewer entries than with the \"leaking\" validation pipeline, the whole point of revising the leaderboard was to rely on the annoying large-file export to .tsv and upload to the leaderboard so much)\n\nAnother issue in this joint model across go hierarchies: The different CC/MF/BP hierarchies seemed to need its own network respectively. More epochs, more layers for CC, less of both for MF (which was far better in fMax than the other 2 for me. Likely also needed more and different features)\n\nI tried a weighted Average Precision score as a substitute to the jiggly f-max, but that went terrible for some reason.\n\nHappy to hear if anyone tried similar ideas, and what the results were!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3402100,
      "author_name": "xaxipirulazo",
      "author_url": "",
      "post_date": "02/05/2026 09:07:45",
      "content": "<p>HI!,</p>\n<p>I evaluated a custom version of <strong>FANTASIA</strong>, an embedding-proximity transfer tool. In this iteration I replaced the usual truncation strategy with a <strong>sliding-window matching</strong> approach, i.e., I match <em>slices</em> of embeddings rather than cutting sequences, aiming to reduce truncation bias for long proteins.</p>\n<p>In my pipeline, the attribute that consistently provided the strongest signal was <strong>sequence identity</strong> (used as the primary transfer criterion / weighting feature).</p>\n<p>For evaluation I used a <strong>GAF time-split from GAF224 → GAF228</strong> over <strong>reviewed proteins (~500k proteins)</strong>, and obtained the following:</p>\n<table>\n<thead>\n<tr>\n<th>Split</th>\n<th>Namespace</th>\n<th>tau</th>\n<th>f_w</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>LK</strong></td>\n<td>biological_process</td>\n<td><strong>0.980</strong></td>\n<td><strong>0.288</strong></td>\n</tr>\n<tr>\n<td><strong>LK</strong></td>\n<td>cellular_component</td>\n<td><strong>0.940</strong></td>\n<td><strong>0.384</strong></td>\n</tr>\n<tr>\n<td><strong>LK</strong></td>\n<td>molecular_function</td>\n<td><strong>0.910</strong></td>\n<td><strong>0.323</strong></td>\n</tr>\n<tr>\n<td><strong>NK</strong></td>\n<td>biological_process</td>\n<td><strong>0.520</strong></td>\n<td><strong>0.345</strong></td>\n</tr>\n<tr>\n<td><strong>NK</strong></td>\n<td>cellular_component</td>\n<td><strong>0.530</strong></td>\n<td><strong>0.511</strong></td>\n</tr>\n<tr>\n<td><strong>NK</strong></td>\n<td>molecular_function</td>\n<td><strong>0.550</strong></td>\n<td><strong>0.522</strong></td>\n</tr>\n<tr>\n<td><strong>PK</strong></td>\n<td>biological_process</td>\n<td><strong>0.810</strong></td>\n<td><strong>0.296</strong></td>\n</tr>\n<tr>\n<td><strong>PK</strong></td>\n<td>cellular_component</td>\n<td><strong>0.910</strong></td>\n<td><strong>0.432</strong></td>\n</tr>\n<tr>\n<td><strong>PK</strong></td>\n<td>molecular_function</td>\n<td><strong>0.910</strong></td>\n<td><strong>0.354</strong></td>\n</tr>\n</tbody>\n</table>\n<p>I set up the validation environment too late, so I didn’t have much room to properly tune parameters. For the final production submission I used a richer reference table (fed with updated <strong>QuickGO</strong>) and tried to calibrate FANTASIA together with additional ML predictions to improve recall, but I wasn’t able to obtain a consistent gain.</p>\n<p>I also experimented with weighting the <strong>evidence codes</strong> in the biological ground truth to reshape the ranking (making it smoother) so that my teammates’ ML predictions could be incorporated more safely, but I didn’t achieve much success.</p>\n<p>By the way, one thing I computed that might be particularly interesting is an evidence-code analysis of “conversion” into new successful annotations (I’ll paste the table below).</p>\n<table>\n<thead>\n<tr>\n<th>evidence_code</th>\n<th>aspect</th>\n<th>N_T0</th>\n<th>N_T0_to_SUCCESS_T1</th>\n<th>conversion_rate</th>\n<th>support_ratio</th>\n<th>effective_weight</th>\n<th>w_norm</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>IEA</td>\n<td>C</td>\n<td>1779720</td>\n<td>28681</td>\n<td>0.016115</td>\n<td>0.769703</td>\n<td>0.11137375179896992</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>IBA</td>\n<td>C</td>\n<td>102092</td>\n<td>4575</td>\n<td>0.044813</td>\n<td>0.044153</td>\n<td>0.04448168745392747</td>\n<td>0.519513</td>\n</tr>\n<tr>\n<td>ISO</td>\n<td>C</td>\n<td>30913</td>\n<td>883</td>\n<td>0.028564</td>\n<td>0.013369</td>\n<td>0.019541867163925707</td>\n<td>0.34037</td>\n</tr>\n<tr>\n<td>NAS</td>\n<td>C</td>\n<td>9306</td>\n<td>881</td>\n<td>0.09467</td>\n<td>0.004025</td>\n<td>0.019519723397987458</td>\n<td>0.340211</td>\n</tr>\n<tr>\n<td>ISS</td>\n<td>C</td>\n<td>64010</td>\n<td>627</td>\n<td>0.009795</td>\n<td>0.027683</td>\n<td>0.01646718920382097</td>\n<td>0.318284</td>\n</tr>\n<tr>\n<td>ISA</td>\n<td>C</td>\n<td>1107</td>\n<td>55</td>\n<td>0.049684</td>\n<td>0.000479</td>\n<td>0.004877160522364997</td>\n<td>0.235033</td>\n</tr>\n<tr>\n<td>ISM</td>\n<td>C</td>\n<td>632</td>\n<td>38</td>\n<td>0.060127</td>\n<td>0.000273</td>\n<td>0.004053941742674189</td>\n<td>0.22912</td>\n</tr>\n<tr>\n<td>RCA</td>\n<td>C</td>\n<td>972</td>\n<td>20</td>\n<td>0.020576</td>\n<td>0.00042</td>\n<td>0.002941038453639716</td>\n<td>0.221126</td>\n</tr>\n<tr>\n<td>ND</td>\n<td>C</td>\n<td>6667</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.002883</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IKR</td>\n<td>C</td>\n<td>9</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.000004</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IGC</td>\n<td>C</td>\n<td>43</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.000019</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IEA</td>\n<td>F</td>\n<td>2732877</td>\n<td>16897</td>\n<td>0.006183</td>\n<td>0.799986</td>\n<td>0.07032923462609807</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>IBA</td>\n<td>F</td>\n<td>95385</td>\n<td>7293</td>\n<td>0.076459</td>\n<td>0.027922</td>\n<td>0.04620449159177866</td>\n<td>0.725579</td>\n</tr>\n<tr>\n<td>ISO</td>\n<td>F</td>\n<td>24114</td>\n<td>788</td>\n<td>0.032678</td>\n<td>0.007059</td>\n<td>0.015187776510689283</td>\n<td>0.372762</td>\n</tr>\n<tr>\n<td>ISS</td>\n<td>F</td>\n<td>54542</td>\n<td>639</td>\n<td>0.011716</td>\n<td>0.015966</td>\n<td>0.013676706217920765</td>\n<td>0.355573</td>\n</tr>\n<tr>\n<td>NAS</td>\n<td>F</td>\n<td>3086</td>\n<td>259</td>\n<td>0.083927</td>\n<td>0.000903</td>\n<td>0.008707249141087028</td>\n<td>0.299046</td>\n</tr>\n<tr>\n<td>RCA</td>\n<td>F</td>\n<td>1346</td>\n<td>86</td>\n<td>0.063893</td>\n<td>0.000394</td>\n<td>0.00501741836046824</td>\n<td>0.257073</td>\n</tr>\n<tr>\n<td>ISA</td>\n<td>F</td>\n<td>1241</td>\n<td>70</td>\n<td>0.056406</td>\n<td>0.000363</td>\n<td>0.004526683279715323</td>\n<td>0.251491</td>\n</tr>\n<tr>\n<td>ISM</td>\n<td>F</td>\n<td>1059</td>\n<td>66</td>\n<td>0.062323</td>\n<td>0.00031</td>\n<td>0.004395447092455655</td>\n<td>0.249999</td>\n</tr>\n<tr>\n<td>IGC</td>\n<td>F</td>\n<td>69</td>\n<td>2</td>\n<td>0.028986</td>\n<td>0.00002</td>\n<td>0.0007651491267319679</td>\n<td>0.208704</td>\n</tr>\n<tr>\n<td>IKR</td>\n<td>F</td>\n<td>117</td>\n<td>1</td>\n<td>0.008547</td>\n<td>0.000034</td>\n<td>0.0005410421361311397</td>\n<td>0.206154</td>\n</tr>\n<tr>\n<td>ND</td>\n<td>F</td>\n<td>13810</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.004043</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IEA</td>\n<td>P</td>\n<td>1904841</td>\n<td>10786</td>\n<td>0.005662</td>\n<td>0.744451</td>\n<td>0.06492603939817149</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>IBA</td>\n<td>P</td>\n<td>115813</td>\n<td>8675</td>\n<td>0.074905</td>\n<td>0.045262</td>\n<td>0.058226869944677836</td>\n<td>0.917455</td>\n</tr>\n<tr>\n<td>ISO</td>\n<td>P</td>\n<td>68384</td>\n<td>1240</td>\n<td>0.018133</td>\n<td>0.026726</td>\n<td>0.022014029573268416</td>\n<td>0.471251</td>\n</tr>\n<tr>\n<td>ISS</td>\n<td>P</td>\n<td>115249</td>\n<td>769</td>\n<td>0.006673</td>\n<td>0.045042</td>\n<td>0.017336118738209134</td>\n<td>0.413611</td>\n</tr>\n<tr>\n<td>NAS</td>\n<td>P</td>\n<td>15317</td>\n<td>696</td>\n<td>0.04544</td>\n<td>0.005986</td>\n<td>0.01649275937526058</td>\n<td>0.403219</td>\n</tr>\n<tr>\n<td>ISA</td>\n<td>P</td>\n<td>751</td>\n<td>16</td>\n<td>0.021305</td>\n<td>0.000294</td>\n<td>0.002500625723120532</td>\n<td>0.230812</td>\n</tr>\n<tr>\n<td>IGC</td>\n<td>P</td>\n<td>129</td>\n<td>14</td>\n<td>0.108527</td>\n<td>0.00005</td>\n<td>0.0023391211771177168</td>\n<td>0.228822</td>\n</tr>\n<tr>\n<td>ISM</td>\n<td>P</td>\n<td>284</td>\n<td>11</td>\n<td>0.038732</td>\n<td>0.000111</td>\n<td>0.0020734093161754886</td>\n<td>0.225548</td>\n</tr>\n<tr>\n<td>RCA</td>\n<td>P</td>\n<td>1508</td>\n<td>7</td>\n<td>0.004642</td>\n<td>0.000589</td>\n<td>0.0016540084463569972</td>\n<td>0.22038</td>\n</tr>\n<tr>\n<td>ND</td>\n<td>P</td>\n<td>8327</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.003254</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>IKR</td>\n<td>P</td>\n<td>16</td>\n<td>0</td>\n<td>0.0</td>\n<td>0.000006</td>\n<td>0.0</td>\n<td>0.2</td>\n</tr>\n</tbody>\n</table>\n<p>Anyway, my performance on the public leaderboard was always quite low. I couldn’t consistently reach 0.40 without merging/ensembling signals; on its own, my most curated submission was only around 0.35.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3402186,
      "author_name": "scutlipengyu",
      "author_url": "",
      "post_date": "02/05/2026 12:04:35",
      "content": "<p>I think your line of thinking is probably correct. For a more robust score improvement, I believe we could either train the model with a larger dataset or apply ensemble learning. And I reckon overfitting is an unavoidable factor behind my current score. 😭</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3402629,
      "author_name": "jrsilicon",
      "author_url": "",
      "post_date": "02/06/2026 12:45:29",
      "content": "<p>I tried a go-embedding / protein-embedding matching via dot product, and learned smoothing/refinement over the GO hierarchy afterwards. My idea was that there should be a lot of untapped potential in having the go-nodes communicate via learned weights with each other. </p>\n<p>Unfortunately, I did not get to a proper validation pipeline done in the 1 month I had. First pipeline was simple train-test split of proteins, with a severe data leakage (I guess from near-identical proteins) -&gt; Still got somehow to 0.3 on the leaderboard. \nFor the  second and last validation pipeline, I did a proper timesplit (train until mid 2024, validation after). This (mostly) got rid of the train-val leak… but now everything was just bad ;-) . Never got close to 0.3 on the leaderboard from before\n(maybe leaving out 1.5y for validation was too much. I tested much fewer entries than with the \"leaking\" validation pipeline, the whole point of revising the leaderboard was to rely on the annoying large-file export to .tsv and upload to the leaderboard so much)</p>\n<p>Another issue in this joint model across go hierarchies: The different CC/MF/BP hierarchies seemed to need its own network respectively. More epochs, more layers for CC, less of both for MF (which was far better in fMax than the other 2 for me. Likely also needed more and different features)</p>\n<p>I tried a weighted Average Precision score as a substitute to the jiggly f-max, but that went terrible for some reason.</p>\n<p>Happy to hear if anyone tried similar ideas, and what the results were!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3401917": "Now that the competition is over, I am curious to know everyone's method for achieving a high score on the public leaderboard (besides ensembling with GOA). Did you notice any discrepancies between your validation performance and the public leaderboard score? And what were the best performances on your validation set and the public leaderboard?\n\nMy approach involves implementing a hierarchical multi-label neural network. It predicts scores both globally and at each level separately, and the final result is the average of these two predictions.  For my evaluation, I performed a simple train-validation split, using a random sample of 10,000 for the validation set. During validation, I achieved a weighted f-max score of approximately 0.58. However, I could only obtain a weighted f-max score of 0.30 on the public leaderboard. I am unsure which one would be a better representation of the test performance.",
    "3402100": "HI!,\n\nI evaluated a custom version of **FANTASIA**, an embedding-proximity transfer tool. In this iteration I replaced the usual truncation strategy with a **sliding-window matching** approach, i.e., I match *slices* of embeddings rather than cutting sequences, aiming to reduce truncation bias for long proteins.\n\nIn my pipeline, the attribute that consistently provided the strongest signal was **sequence identity** (used as the primary transfer criterion / weighting feature).\n\nFor evaluation I used a **GAF time-split from GAF224 → GAF228** over **reviewed proteins (~500k proteins)**, and obtained the following:\n\n| Split  | Namespace          | tau       | f_w       |\n| ------ | ------------------ | --------- | --------- |\n| **LK** | biological_process | **0.980** | **0.288** |\n| **LK** | cellular_component | **0.940** | **0.384** |\n| **LK** | molecular_function | **0.910** | **0.323** |\n| **NK** | biological_process | **0.520** | **0.345** |\n| **NK** | cellular_component | **0.530** | **0.511** |\n| **NK** | molecular_function | **0.550** | **0.522** |\n| **PK** | biological_process | **0.810** | **0.296** |\n| **PK** | cellular_component | **0.910** | **0.432** |\n| **PK** | molecular_function | **0.910** | **0.354** |\n\nI set up the validation environment too late, so I didn’t have much room to properly tune parameters. For the final production submission I used a richer reference table (fed with updated **QuickGO**) and tried to calibrate FANTASIA together with additional ML predictions to improve recall, but I wasn’t able to obtain a consistent gain.\n\nI also experimented with weighting the **evidence codes** in the biological ground truth to reshape the ranking (making it smoother) so that my teammates’ ML predictions could be incorporated more safely, but I didn’t achieve much success.\n\nBy the way, one thing I computed that might be particularly interesting is an evidence-code analysis of “conversion” into new successful annotations (I’ll paste the table below).\n\n| evidence_code | aspect | N_T0    | N_T0_to_SUCCESS_T1 | conversion_rate | support_ratio | effective_weight      | w_norm   |\n| ------------- | ------ | ------- | ------------------ | --------------- | ------------- | --------------------- | -------- |\n| IEA           | C      | 1779720 | 28681              | 0.016115        | 0.769703      | 0.11137375179896992   | 1.0      |\n| IBA           | C      | 102092  | 4575               | 0.044813        | 0.044153      | 0.04448168745392747   | 0.519513 |\n| ISO           | C      | 30913   | 883                | 0.028564        | 0.013369      | 0.019541867163925707  | 0.34037  |\n| NAS           | C      | 9306    | 881                | 0.09467         | 0.004025      | 0.019519723397987458  | 0.340211 |\n| ISS           | C      | 64010   | 627                | 0.009795        | 0.027683      | 0.01646718920382097   | 0.318284 |\n| ISA           | C      | 1107    | 55                 | 0.049684        | 0.000479      | 0.004877160522364997  | 0.235033 |\n| ISM           | C      | 632     | 38                 | 0.060127        | 0.000273      | 0.004053941742674189  | 0.22912  |\n| RCA           | C      | 972     | 20                 | 0.020576        | 0.00042       | 0.002941038453639716  | 0.221126 |\n| ND            | C      | 6667    | 0                  | 0.0             | 0.002883      | 0.0                   | 0.2      |\n| IKR           | C      | 9       | 0                  | 0.0             | 0.000004      | 0.0                   | 0.2      |\n| IGC           | C      | 43      | 0                  | 0.0             | 0.000019      | 0.0                   | 0.2      |\n| IEA           | F      | 2732877 | 16897              | 0.006183        | 0.799986      | 0.07032923462609807   | 1.0      |\n| IBA           | F      | 95385   | 7293               | 0.076459        | 0.027922      | 0.04620449159177866   | 0.725579 |\n| ISO           | F      | 24114   | 788                | 0.032678        | 0.007059      | 0.015187776510689283  | 0.372762 |\n| ISS           | F      | 54542   | 639                | 0.011716        | 0.015966      | 0.013676706217920765  | 0.355573 |\n| NAS           | F      | 3086    | 259                | 0.083927        | 0.000903      | 0.008707249141087028  | 0.299046 |\n| RCA           | F      | 1346    | 86                 | 0.063893        | 0.000394      | 0.00501741836046824   | 0.257073 |\n| ISA           | F      | 1241    | 70                 | 0.056406        | 0.000363      | 0.004526683279715323  | 0.251491 |\n| ISM           | F      | 1059    | 66                 | 0.062323        | 0.00031       | 0.004395447092455655  | 0.249999 |\n| IGC           | F      | 69      | 2                  | 0.028986        | 0.00002       | 0.0007651491267319679 | 0.208704 |\n| IKR           | F      | 117     | 1                  | 0.008547        | 0.000034      | 0.0005410421361311397 | 0.206154 |\n| ND            | F      | 13810   | 0                  | 0.0             | 0.004043      | 0.0                   | 0.2      |\n| IEA           | P      | 1904841 | 10786              | 0.005662        | 0.744451      | 0.06492603939817149   | 1.0      |\n| IBA           | P      | 115813  | 8675               | 0.074905        | 0.045262      | 0.058226869944677836  | 0.917455 |\n| ISO           | P      | 68384   | 1240               | 0.018133        | 0.026726      | 0.022014029573268416  | 0.471251 |\n| ISS           | P      | 115249  | 769                | 0.006673        | 0.045042      | 0.017336118738209134  | 0.413611 |\n| NAS           | P      | 15317   | 696                | 0.04544         | 0.005986      | 0.01649275937526058   | 0.403219 |\n| ISA           | P      | 751     | 16                 | 0.021305        | 0.000294      | 0.002500625723120532  | 0.230812 |\n| IGC           | P      | 129     | 14                 | 0.108527        | 0.00005       | 0.0023391211771177168 | 0.228822 |\n| ISM           | P      | 284     | 11                 | 0.038732        | 0.000111      | 0.0020734093161754886 | 0.225548 |\n| RCA           | P      | 1508    | 7                  | 0.004642        | 0.000589      | 0.0016540084463569972 | 0.22038  |\n| ND            | P      | 8327    | 0                  | 0.0             | 0.003254      | 0.0                   | 0.2      |\n| IKR           | P      | 16      | 0                  | 0.0             | 0.000006      | 0.0                   | 0.2      |\n\n\nAnyway, my performance on the public leaderboard was always quite low. I couldn’t consistently reach 0.40 without merging/ensembling signals; on its own, my most curated submission was only around 0.35.",
    "3402186": "I think your line of thinking is probably correct. For a more robust score improvement, I believe we could either train the model with a larger dataset or apply ensemble learning. And I reckon overfitting is an unavoidable factor behind my current score. 😭",
    "3402629": "I tried a go-embedding / protein-embedding matching via dot product, and learned smoothing/refinement over the GO hierarchy afterwards. My idea was that there should be a lot of untapped potential in having the go-nodes communicate via learned weights with each other. \n\nUnfortunately, I did not get to a proper validation pipeline done in the 1 month I had. First pipeline was simple train-test split of proteins, with a severe data leakage (I guess from near-identical proteins) -> Still got somehow to 0.3 on the leaderboard. \nFor the  second and last validation pipeline, I did a proper timesplit (train until mid 2024, validation after). This (mostly) got rid of the train-val leak... but now everything was just bad ;-) . Never got close to 0.3 on the leaderboard from before\n(maybe leaving out 1.5y for validation was too much. I tested much fewer entries than with the \"leaking\" validation pipeline, the whole point of revising the leaderboard was to rely on the annoying large-file export to .tsv and upload to the leaderboard so much)\n\nAnother issue in this joint model across go hierarchies: The different CC/MF/BP hierarchies seemed to need its own network respectively. More epochs, more layers for CC, less of both for MF (which was far better in fMax than the other 2 for me. Likely also needed more and different features)\n\nI tried a weighted Average Precision score as a substitute to the jiggly f-max, but that went terrible for some reason.\n\nHappy to hear if anyone tried similar ideas, and what the results were!"
  },
  "source": "meta"
}