{
  "id": 657539,
  "title": "[UPDATE:  Rescoring Complete] Fixes to GO Graph Processing & IA File — Leaderboard Rescoring in Progress",
  "url": "/competitions/cafa-6-protein-function-prediction/discussion/657539",
  "author_name": "MartynaPlomecka",
  "post_date": "2025-12-10T14:19:57.434000",
  "votes": 14,
  "comment_count": 9,
  "views": 0,
  "content": "<p><strong>Update: Rescoring has been completed</strong></p>\n<p>Hi everyone,</p>\n<p>We want to let you know about two important fixes that were identified over the past few days and will require a leaderboard rescoring:</p>\n<p><strong>1. GO Graph Topological Sorting Bug:</strong></p>\n<p>A few GO term pairs contained duplicated parent–child edges, which caused the topological sorting step to behave incorrectly in rare cases. The fix (now merged in the public repository) deduplicates all parent/child edges before performing the topological sort. This issue is not expected to significantly affect leaderboard rankings.</p>\n<p><strong>2. Incorrect Values in IA.tsv:</strong></p>\n<p>The original IA.tsv file contained 6,216 values that were mistakenly set to 0.0. \nWe have now replaced it with the correct file.</p>\n<p>Because the evaluation pipeline depends on both the GO graph and the IA statistics, we are rescoring the leaderboard. During this time, the leaderboard may look unstable or partially updated.\nWe expect things to be a bit fluctuating for the next couple of hours, and we'll post another update once rescoring is complete.</p>\n<p>Thanks so much for your patience and to the community members who helped identify the issues!\n— Martyna</p>",
  "messages": [
    {
      "id": 3370014,
      "postDate": "2025-12-10T14:19:57.433Z",
      "content": "<p><strong>Update: Rescoring has been completed</strong></p>\n<p>Hi everyone,</p>\n<p>We want to let you know about two important fixes that were identified over the past few days and will require a leaderboard rescoring:</p>\n<p><strong>1. GO Graph Topological Sorting Bug:</strong></p>\n<p>A few GO term pairs contained duplicated parent–child edges, which caused the topological sorting step to behave incorrectly in rare cases. The fix (now merged in the public repository) deduplicates all parent/child edges before performing the topological sort. This issue is not expected to significantly affect leaderboard rankings.</p>\n<p><strong>2. Incorrect Values in IA.tsv:</strong></p>\n<p>The original IA.tsv file contained 6,216 values that were mistakenly set to 0.0. \nWe have now replaced it with the correct file.</p>\n<p>Because the evaluation pipeline depends on both the GO graph and the IA statistics, we are rescoring the leaderboard. During this time, the leaderboard may look unstable or partially updated.\nWe expect things to be a bit fluctuating for the next couple of hours, and we'll post another update once rescoring is complete.</p>\n<p>Thanks so much for your patience and to the community members who helped identify the issues!\n— Martyna</p>",
      "rawMarkdown": "**Update: Rescoring has been completed**\n\nHi everyone,\n\nWe want to let you know about two important fixes that were identified over the past few days and will require a leaderboard rescoring:\n\n**1. GO Graph Topological Sorting Bug:**\n\nA few GO term pairs contained duplicated parent–child edges, which caused the topological sorting step to behave incorrectly in rare cases. The fix (now merged in the public repository) deduplicates all parent/child edges before performing the topological sort. This issue is not expected to significantly affect leaderboard rankings.\n\n**2. Incorrect Values in IA.tsv:**\n\nThe original IA.tsv file contained 6,216 values that were mistakenly set to 0.0. \nWe have now replaced it with the correct file.\n\n\nBecause the evaluation pipeline depends on both the GO graph and the IA statistics, we are rescoring the leaderboard. During this time, the leaderboard may look unstable or partially updated.\nWe expect things to be a bit fluctuating for the next couple of hours, and we'll post another update once rescoring is complete.\n\nThanks so much for your patience and to the community members who helped identify the issues!\n— Martyna",
      "votes": 14
    },
    {
      "id": 3370068,
      "postDate": "2025-12-10T14:59:49.693Z",
      "content": "<p>Note that the correct IA.tsv file has been updated in the Data tab and can be downloaded now. The correct IA values were generated using this <a href=\"https://github.com/claradepaolis/InformationAccretion\" target=\"_blank\">GitHub repo</a>.</p>",
      "rawMarkdown": "Note that the correct IA.tsv file has been updated in the Data tab and can be downloaded now. The correct IA values were generated using this [GitHub repo](https://github.com/claradepaolis/InformationAccretion).",
      "votes": 2
    },
    {
      "id": 3370128,
      "postDate": "2025-12-10T15:33:22.517Z",
      "content": "<p>Thank you for your effort. I want to mention that in our case, some of the reruns crashed with an <em>Internal scoring error</em>.</p>",
      "rawMarkdown": "Thank you for your effort. I want to mention that in our case, some of the reruns crashed with an *Internal scoring error*.",
      "replies": [
        {
          "id": 3370130,
          "postDate": "2025-12-10T15:35:39.783Z",
          "content": "<p>Please wait until the rescore is complete. </p>",
          "rawMarkdown": "Please wait until the rescore is complete. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 3372534,
      "postDate": "2025-12-12T04:16:07.773Z",
      "content": "<p>Hi，I want to ask if all of the submissions have been rescored. I just see that you said the rescore has been completed and found the large difference between the old score and new score for a resubmission</p>",
      "rawMarkdown": "Hi，I want to ask if all of the submissions have been rescored. I just see that you said the rescore has been completed and found the large difference between the old score and new score for a resubmission",
      "replies": [
        {
          "id": 3373357,
          "postDate": "2025-12-12T12:31:02.567Z",
          "content": "<p>yes, all submissions have been rescored.</p>",
          "rawMarkdown": "yes, all submissions have been rescored.",
          "replies": [
            {
              "id": 3374026,
              "postDate": "2025-12-12T16:13:26.863Z",
              "content": "<p>very strange, for some score trained with old ia.tsv are not scored lower, but remain same. I changed to new ia.tsv file, but the score dropped……</p>",
              "rawMarkdown": "very strange, for some score trained with old ia.tsv are not scored lower, but remain same. I changed to new ia.tsv file, but the score dropped......"
            }
          ]
        }
      ]
    },
    {
      "id": 3370590,
      "postDate": "2025-12-11T00:30:41.743Z",
      "content": "<p>Is this the correct expectations for the new IA.tsv file?</p>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>Previous (buggy)</th>\n<th>New (corrected)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Terms with 0.0</td>\n<td>~11,828</td>\n<td>~5,612 (11,828 - 6,216)</td>\n</tr>\n<tr>\n<td>Terms with &gt; 0.0</td>\n<td>~28,294</td>\n<td>~34,510</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "Is this the correct expectations for the new IA.tsv file?\n\nMetric                   | Previous (buggy) | New (corrected)            |\n| ----------------  | ----------------- | ----------------------- |\nTerms with 0.0     | ~11,828                 | ~5,612 (11,828 - 6,216)  |\nTerms with > 0.0  | ~28,294                | ~34,510                         |\n",
      "replies": [
        {
          "id": 3370592,
          "postDate": "2025-12-11T00:38:00.913Z",
          "content": "<p>It should be like this (the current IA.tsv file in the Data tab is the new corrected file): </p>\n<table>\n<thead>\n<tr>\n<th><strong>Metric</strong></th>\n<th><strong>Previous (buggy)</strong></th>\n<th><strong>New (corrected)</strong></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Terms with IA = 0.0</td>\n<td>18,044</td>\n<td>11,828 (18,044 - 6,216)</td>\n</tr>\n<tr>\n<td>Terms with IA &gt; 0.0</td>\n<td>22,078</td>\n<td>28,294</td>\n</tr>\n</tbody>\n</table>",
          "rawMarkdown": "It should be like this (the current IA.tsv file in the Data tab is the new corrected file): \n| **Metric** | **Previous (buggy)** | **New (corrected)** |\n| --- | --- | --- |\n| Terms with IA = 0.0 | 18,044 | 11,828 (18,044 - 6,216) |\n| Terms with IA > 0.0 | 22,078 | 28,294 |\n",
          "votes": 3
        }
      ]
    },
    {
      "id": 3370827,
      "postDate": "2025-12-11T05:57:15.820Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 3370068,
      "author_name": "An Phan",
      "author_url": "",
      "post_date": "2025-12-10T14:59:49.693000",
      "content": "<p>Note that the correct IA.tsv file has been updated in the Data tab and can be downloaded now. The correct IA values were generated using this <a href=\"https://github.com/claradepaolis/InformationAccretion\" target=\"_blank\">GitHub repo</a>.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 3370128,
      "author_name": "Tony Hauptmann",
      "author_url": "",
      "post_date": "2025-12-10T15:33:22.517000",
      "content": "<p>Thank you for your effort. I want to mention that in our case, some of the reruns crashed with an <em>Internal scoring error</em>.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3370130,
          "author_name": "MartynaPlomecka",
          "author_url": "",
          "post_date": "2025-12-10T15:35:39.783000",
          "content": "<p>Please wait until the rescore is complete. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3372534,
      "author_name": "common-people",
      "author_url": "",
      "post_date": "2025-12-12T04:16:07.773000",
      "content": "<p>Hi，I want to ask if all of the submissions have been rescored. I just see that you said the rescore has been completed and found the large difference between the old score and new score for a resubmission</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3373357,
          "author_name": "MartynaPlomecka",
          "author_url": "",
          "post_date": "2025-12-12T12:31:02.567000",
          "content": "<p>yes, all submissions have been rescored.</p>",
          "votes": 0,
          "replies": [
            {
              "id": 3374026,
              "author_name": "common-people",
              "author_url": "",
              "post_date": "2025-12-12T16:13:26.863000",
              "content": "<p>very strange, for some score trained with old ia.tsv are not scored lower, but remain same. I changed to new ia.tsv file, but the score dropped……</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3370590,
      "author_name": "Young",
      "author_url": "",
      "post_date": "2025-12-11T00:30:41.743000",
      "content": "<p>Is this the correct expectations for the new IA.tsv file?</p>\n<table>\n<thead>\n<tr>\n<th>Metric</th>\n<th>Previous (buggy)</th>\n<th>New (corrected)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Terms with 0.0</td>\n<td>~11,828</td>\n<td>~5,612 (11,828 - 6,216)</td>\n</tr>\n<tr>\n<td>Terms with &gt; 0.0</td>\n<td>~28,294</td>\n<td>~34,510</td>\n</tr>\n</tbody>\n</table>",
      "votes": 0,
      "replies": [
        {
          "id": 3370592,
          "author_name": "An Phan",
          "author_url": "",
          "post_date": "2025-12-11T00:38:00.913000",
          "content": "<p>It should be like this (the current IA.tsv file in the Data tab is the new corrected file): </p>\n<table>\n<thead>\n<tr>\n<th><strong>Metric</strong></th>\n<th><strong>Previous (buggy)</strong></th>\n<th><strong>New (corrected)</strong></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Terms with IA = 0.0</td>\n<td>18,044</td>\n<td>11,828 (18,044 - 6,216)</td>\n</tr>\n<tr>\n<td>Terms with IA &gt; 0.0</td>\n<td>22,078</td>\n<td>28,294</td>\n</tr>\n</tbody>\n</table>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 3370827,
      "author_name": "",
      "author_url": "",
      "post_date": "2025-12-11T05:57:15.820000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3370014": "**Update: Rescoring has been completed**\n\nHi everyone,\n\nWe want to let you know about two important fixes that were identified over the past few days and will require a leaderboard rescoring:\n\n**1. GO Graph Topological Sorting Bug:**\n\nA few GO term pairs contained duplicated parent–child edges, which caused the topological sorting step to behave incorrectly in rare cases. The fix (now merged in the public repository) deduplicates all parent/child edges before performing the topological sort. This issue is not expected to significantly affect leaderboard rankings.\n\n**2. Incorrect Values in IA.tsv:**\n\nThe original IA.tsv file contained 6,216 values that were mistakenly set to 0.0. \nWe have now replaced it with the correct file.\n\n\nBecause the evaluation pipeline depends on both the GO graph and the IA statistics, we are rescoring the leaderboard. During this time, the leaderboard may look unstable or partially updated.\nWe expect things to be a bit fluctuating for the next couple of hours, and we'll post another update once rescoring is complete.\n\nThanks so much for your patience and to the community members who helped identify the issues!\n— Martyna",
    "3370068": "Note that the correct IA.tsv file has been updated in the Data tab and can be downloaded now. The correct IA values were generated using this [GitHub repo](https://github.com/claradepaolis/InformationAccretion).",
    "3370128": "Thank you for your effort. I want to mention that in our case, some of the reruns crashed with an *Internal scoring error*.",
    "3372534": "Hi，I want to ask if all of the submissions have been rescored. I just see that you said the rescore has been completed and found the large difference between the old score and new score for a resubmission",
    "3370590": "Is this the correct expectations for the new IA.tsv file?\n\nMetric                   | Previous (buggy) | New (corrected)            |\n| ----------------  | ----------------- | ----------------------- |\nTerms with 0.0     | ~11,828                 | ~5,612 (11,828 - 6,216)  |\nTerms with > 0.0  | ~28,294                | ~34,510                         |\n",
    "3370827": ""
  }
}